2024 Cloud Service Provider API Report

Introduction

As part of our API Directory operations, in 2023 APIContext made more than 650 million API calls to more than 10,000 different API endpoints from more than 100 geographically diverse cloud data center locations across major public cloud service providers including AWS, Azure, Google, and IBM. This proprietary dataset gives us unique insights into the cloud API landscape.

In this report, we analyze the API quality data generated by multiple APIContext services, including the APImetrics platform, our API Directory and our Supplier Index. Now in its sixth year, the result is an unbiased, industry-wide baseline for API quality scoring.

As the volume of API calls across the internet continues to grow, our data continues to get both broader and deeper. To create this analysis, we leverage aggregated, anonymized data from leading API services, including infrastructure providers, financial services institutions, social networks, search engines, and other key services.

2023 Executive Summary

This is the sixth year of the APIContext Cloud API Performance Report. With years of historical data, we can now identify longer-term trends in addition to our annual analysis.

While cloud services become even more important to our personal and professional lives and now comprise the vast majority of internet traffic, cloud providers continue to struggle to maintain speed and quality in all circumstances. Baseline performance continues to be good, but corner cases and edge cases with reduced quality are more prevalent, at a time when user expectations of reliability and availability have never been higher.

We can now say definitively that the 2020s have seen an increase in latency. This is likely due to two trends that accelerated during the global pandemic and continued across 2023: Remote Work and APIs Eating Software. Remote work requirements have increased cloud loads and forced cloud providers to expand edge infrastructure. At the same time, both public and internal APIs have continued to expand, running more applications than ever.

Although there have been year-on-year increases in capital expenditure by the main cloud service providers, demand for cloud computing is outstripping supply in large part because of the phenomenal growth of artificial intelligence and machine learning in 2023 and its associated requirements for immense amounts of compute. This has put pressure on cloud services and may lead to some degradations in network quality.

Quality is stable

Most services are rated as excellent, with a CASC score of 9.00 or more. Overall quality is similar to 2020, 2021 and 2022, which means improvements in performance might be at a plateau. In 2024 there is no excuse for not having a highly stable and consistently performant API. You cannot blame your infrastructure provider for everything; only for network performance. Once that has been optimized with the most suitable cloud for your API service, the ball is in your court to offer the fastest and most reliable service possible.

Key conclusions

1. API availability is getting worse: only 7% of observed services achieved 99.99% availability in 2023, compared with 18% in 2022.
2. The most popular cloud data center in the world has degraded: AWS US-East moved from 1.23 ms Time to Connect in 2022 to 2.50 ms in 2023.
3. Four of the top five locations for Time to Connect are AWS locations. Google Europe-West 3 in Frankfurt was fourth-fastest at 2.81 ms.
4. IBM was the fastest cloud by Total Time, averaging 443 ms. AWS was second at 450 ms, and Azure was slowest at 529 ms.
5. AWS was once again totally dominant on DNS Time at 2 ms; no other managed cloud performed better than 13 ms.

Availability

Five 9s is a tough target, but we believe that 99.99% is a goal that should be achievable for most APIs. Only 7% of the services we studied managed to reach this level, down from 18% in 2022 and up from 6% in 2021.

PagerDuty was again the API service with the highest availability and the only one to achieve four 9s in both 2022 and 2023. No API had previously managed to reach the 99.99% mark two years in a row, so this was an exceptional performance by PagerDuty.

In all of 2023, PagerDuty had 30.0 minutes of measurable downtime on their APIs. The worst-performing API had more than 8.9 days of downtime. If an API is exercised at an average rate of 50 calls per second, that would mean nearly 39 million attempted calls were lost.

Cloud performance varies

In 2023, Azure was consistently more than 75 ms slower globally than AWS. A typical phone app operation such as checking in for a flight might use six to nine APIs or more, which means more than 400 ms of latency owing to cloud choice.

All regions were slower in terms of both DNS Time and Connect Time in 2023 compared to 2022. In the case of Connect Time, South America was slower by 3500%. North America has only improved by 1 ms for DNS Time since 2018 and is no longer the fastest region for that latency component.

The API Supply Chain

In 2023, API performance was good across a wide range of popular services. In contrast to 2022, no APIs in the study rated as being of concern. But the problem of the API Supply Chain, as Founder and CEO of ProgrammableWeb John Musser calls it, remains significant.

There are meaningful geographic differences, such as physical distances across oceans and continents; and cloud performance variations, such as the amount of bandwidth available through fiber optic cables and the capacity of network equipment. DNS lookup times, which have always been a problem, seem to be getting worse, as do Connect Times.

Using an API is not just relying on a black box. The API you provide or use exists in a universe of components including their own cloud service, a CDN provider, probably a gateway of their own, a backend server architecture, and potentially a security and identity service. Each of those components has its own configuration and cloud dependencies - and a failure could end up costing $200,000 per incident.

DNS deterioration

DNS resolution times slowed in two of the four clouds, Google and IBM, and across all regions in 2023 compared to 2022. If AWS can have a median DNS Time of 2 ms, so can the other clouds. Individual services can have hundreds of milliseconds of extra latency friction created through non-optimized DNS.

DNS is still a problem

The overall average rose to 19 ms in 2023, compared with 10 ms in 2022. The best DNS Time was 3.4 ms from Box. In contrast, DocuSign had a DNS Time of 224 ms, up from 199 ms in 2022, and Capital One had a DNS Time of 420 ms, up from 339 ms in 2022. Such large differences cost money and add unacceptable friction to the user experience.

Pass Rates

API calls used in this analysis were made from 82 data centers around the world using APIContext observer agents running on application servers provided by Amazon (AWS), Google, IBM, and Microsoft (Azure). The 82 cloud locations are locations that have been used since reporting began in 2020.

The sample sizes for each API are roughly the same and are equivalent to a call from each cloud location made to each endpoint every five minutes throughout the year. We logged the data using the APImetrics platform. Latency, pass rates, and quality scores were recorded in the same way for all APIs.

In calculating the pass rate, we define failures to include the following:

• 5xx server-side errors: Errors returned by the server.
• Network errors: No response is returned.
• Content errors: The API did not return the correct content, such as an empty JSON body or incorrect data.
• Slow errors: A response is received after an overly long period.
• Redirect errors: A 3xx redirect HTTP status code is returned.

We ignored call-specific application errors such as issues with returned content and client-side HTTP status code 4xx warnings caused by authentication problems such as expired tokens. If an API fails, it may pass if called again immediately and succeed if the outage is transitory. Our methodology still gives a general indication of availability issues.

Top achievers

PagerDuty had 30.0 minutes of measurable downtime on their APIs in 2023. PagerDuty was also first in 2022 with 21.9 minutes of downtime, so while it increased a bit, this was another superb performance. The DNS host and service host for PagerDuty is AWS.

Quality

APIContext uses CASC (Cloud API Service Consistency), our patented quality scoring system to compare the quality of different APIs. CASC blends multiple factors.

It is important to note that CASC scores do not fall on a normal curve. The scores are absolute, and we see no engineering reasons why prominent APIs should not consistently reach a CASC score of 8.00+. Most of the services we studied have a CASC score of 9.00+, which indicates excellent performance.

• Over 9.00: Exceptional quality of operation.
• Over 8.00: Healthy, well-functioning APIs that will give few problems to users.
• 6.00-8.00: Some significant issues that will lead to a degraded user experience and increased engineering support costs.
• Below 6.00: Urgent attention is required.

n-9s Reliability

The traditional telecommunications standard for service availability is five 9s - at least 99.999% uptime, or just five minutes of downtime in a year. Of the 27 services analyzed in this study, no API managed to achieve five 9s. Two services achieved four 9s, down from six in 2022. In 2023, 19% of major corporate services scored less than three 9s, slightly up from 2022.

Latency

Some calls will be faster than others because of backend processing, so total call duration, even over a sample size of tens of millions of calls, can only give a partial view of API behavior.

Over the past three years, the overall trend has broadly been an increase in Total Time. IBM was the fastest cloud provider in 2023 with an average Total Time of 443 ms, slightly ahead of AWS at 450 ms and 86 ms faster than Microsoft.

All clouds are slower for Total Time in 2023. IBM may benefit from being the least heavily loaded cloud. Azure has been consistently the slowest cloud by median time since 2018.

AWS's median DNS Lookup Time has continued to fall and is now just 2 ms, down from 3 ms in 2022. The next best cloud, Google, had a DNS Time of 13 ms and Azure 18 ms. All regions had slower DNS in 2023; Oceania was twice as slow at 16 ms compared with 8 ms in 2022.

Recommendations

1. Are you actively monitoring the availability and latency of all APIs you expose and consume? If you are not, you do not know how your APIs are performing right now for users in the real world.
2. Are you benchmarking the performance and quality of your APIs against those of your peers or competitors? You would not want to find out that they outperform you.
3. Do you know the differences between cloud locations and user locations? Your service might be hosted in Virginia, but your users might be in Vienna or Vietnam.
4. Do you know that 70 ms or more of latency can be down to your choice of cloud? Your API users should not wait tens or hundreds of milliseconds simply because of a decision made years ago.
5. Do you want to rely on a single cloud service provider? Build in resilience by using multiple cloud service providers.
6. Have you ensured no specific issues affect the DNS Lookup Time for your domain? It should be 12 ms or less.
7. Do you understand what factors impact call latency and where to focus your efforts?
8. Are you tracking performance outliers and determining their causes? Slow outliers can greatly impact user experience.
9. Is your organization aware of the impact of API failures and errors on user experience and business costs? Bad APIs mean lost customers.

2023 Results

Historical comparisons

From mid-2018 to mid-2021, DNS Time was quantized for all four clouds. Improvements to the APIContext observer network in 2021 have allowed a more granular analysis over the last two years. AWS has had by far the best performance in this period, with the other clouds at least 10 ms slower.

Of the six regions, Europe and North America have consistently been the fastest over the seven-year period. In 2023, East Asia and South America comprised an intermediate group, with Oceania and South Asia a slow group. All regions show large increases in Connect Time in 2023 compared to 2021, with South America reaching 106 ms, likely caused by issues in a specific data center.

Methodology

API calls used in this analysis were made from 82 data centers around the world using APIContext observer agents running on application servers provided by Amazon (AWS), Google, IBM, and Microsoft (Azure). New locations come online and old ones are taken offline; the 82 cloud locations are locations used since reporting began in 2020. For the 2025 report, the number will increase as a significant number of new locations have recently been added.

The sample sizes for each API are roughly the same and are equivalent to a call from each cloud location made to each endpoint every five minutes throughout the year. We logged the data using the APImetrics platform. Latency, pass rates, and quality scores were recorded in the same way for all APIs. For most APIs, data is available for the whole period.

High-level data for 2023 is provided for free at the APIContext Directory. If you would like to dive deeper into the details, please contact us for licensing access.

Learn More About Your Own APIs

APIContext analyzes APIs to report on API performance, availability, security, and conformance. Our results are used to improve the quality of hundreds of digital applications, including B2B, B2C, large and hyperscale technology products, IoT applications, banking and financial services, healthcare, and more.

Whether you need to respond immediately to customer issues, align internal stakeholders, or prepare for regulatory reporting, our platform has your APIs covered.

• Synthetic API monitoring from more than 100 locations worldwide.
• Observed API traffic analysis from within your application or IT infrastructure.
• Performance and reliability analysis, including full call details and auditable results.
• Analysis for API security misconfigurations and sensitive data exposure.
• Measurement of API conformance against your specifications and open standards.

Glossary

The measurement terms used throughout this report, defined for teams comparing API quality across clouds, regions and providers.

CASC

Cloud API Service Consistency, APIContext's benchmarked quality score from 0.00 to 10.00.

DNS Time

The time required to resolve a domain name before a connection can be established.

Connect Time

The time required to establish a network connection to the service.

Total Time

The full measured duration of an API call, including network setup, server processing and response delivery.

Five 9s

99.999% availability, allowing approximately five minutes of downtime in a year.

API Supply Chain

The connected set of cloud services, CDNs, gateways, backends, security services and other dependencies that an API relies on.

Pass Rate

The proportion of observed calls that avoid server, network, content, slow-response and redirect failures.

About APIContext

APIContext eliminates blind spots for enterprises across the digital delivery chain with proactive synthetic monitoring, performance analytics, and automated conformance validation. Our platform delivers actionable insights so connected systems perform and conform—ensuring every interaction is trusted, secure, and compliant.

What are your APIs saying to AI? Contact us to find out.