The Cloud for APIs: 2022 in Review

Introduction

In 2022, APImetrics made over a billion API calls to more than 8,400 different API endpoints from 70 geographically diverse cloud data centers across AWS, Azure, Google, and IBM Cloud.

In this report, we build on the unique and ever-expanding API dataset generated by the APImetrics platform and API.expert portal to establish an unbiased, industry-wide baseline for API quality scoring.

The report focuses on data from leading API services, including those from prominent corporate infrastructure providers, financial services institutions, social networks, and search engines.

Summary

Cloud API quality has been stable over 2020–2022, albeit with evidence of deterioration in certain aspects of performance, particularly DNS lookup time. This period has likely been impacted by COVID-19 and geopolitical uncertainties, and performance does still vary between clouds, regions, and locations.

Quality is stable

Most services are rated as excellent with a CASC (quality) score of 9.00 or more. Overall quality is similar to 2020 and 2021, suggesting that improvements in performance might be tending to plateau. There is no excuse for not having a highly stable and consistently performant API.

Availability

Five 9s is a tough target, but 99.99% is a goal that should be achievable for most APIs and six of the services studied managed to reach this level, up from two in 2021. This indicates an area of future focus for quality improvements.

Cloud performance

We see significant differences in performance between clouds. In 2022, Azure was consistently more than 70 ms slower than AWS. In an API-first economy where every millisecond increasingly counts, can you and your customers afford to be using a slow cloud?

DNS deterioration

DNS resolution times have slowed across clouds except AWS, and across all regions except South America. If AWS can have a median DNS time of 3 ms, so can the other clouds. In 2023, we want to see DNS times getting better again across all clouds and regions.

Absolute regional differences have decreased

In our first report for 2017 there was a 10x difference between South America and Europe in median TCP connect time. South America is now 40% quicker than Europe and over 3x faster than North America. This is excellent and shows the APIs and the cloud providers continue to strive for improvement.

That said, you will still pay geographic penalties depending on how the APIs you call are architected and where they are hosted. Interestingly, the relative regional differences have remained more or less the same over the six-year period. Oceania and South Asia are still around 3x slower than North America for total time even though total time is down across the board; only Europe has significantly improved relative to North America.

Many services are hosted in North America and further performance improvements would be helped by multi-hosting services in additional regions to reduce distance from the end user, and thus download time.

The API supply chain

API performance is good across a wide range of popular services, with more and better-quality availability than in 2021. But the problem of the API supply chain, as John Musser calls it, remains significant. There are meaningful geographic differences, such as physical distances across oceans and continents; and cloud performance variations, such as the amount of bandwidth available through fiber optic cables and the capacity of network equipment. DNS lookup times, which have always been a problem, seem to be getting worse.

Using an API isn't just relying on a black box. The API you provide or use exists in a universe of components including their own cloud service, a CDN provider, probably a gateway of their own, a backend server architecture, and potentially a security and identity service. And each of those components has its own configuration and cloud dependencies. A failure could end up costing $200,000 per incident.

DNS is still a problem

DNS lookup time has increased across most cloud services and regions, suggesting there are issues with network infrastructure and configuration for Azure, Google, and IBM Cloud, particularly in Europe and North America.

For example, in 2022, DocuSign had a DNS time of 199.35 ms compared to the average of 10.05 ms. Capital One had a DNS time of 339.2 ms. In 2021, DocuSign was slowest at 252.8 ms compared to an average of 16.975 ms; Capital One was 143.75 ms.

From our analysis, most services use a CDN provider — Akamai, Amazon, Automattic, Cloudflare, Fastly, Google, Incapsula and Microsoft all appear in the dataset. However, a CDN doesn't just work out of the box and needs to be configured. If you are a user or a provider of any services, you should focus on this.

Top achievers

From January 2022 to December 2022, PagerDuty had 21.9 minutes of measurable downtime on their APIs. The best performer in 2021 was DocuSign, with 26.5 minutes of downtime. Congratulations to the DevOps team and everyone else at PagerDuty involved.

The next-best performer had a failure equivalent of just 31.6 minutes of downtime over 12 months, which is still a pretty good effort. To put that in perspective, the worst-performing API had over 3.2 days of downtime. In other words, if an API is being exercised at an average rate of 50 calls per second, that would mean more than 13 million attempted calls were lost.

Look for yourself

High-level data for 2022 is provided for free through the API directory and ranking lists. If you would like to be able to dive deeper into the details, please get in touch about licensing access.

Methodology

API calls were made on a regularly scheduled basis from 70 data centers around the world using APImetrics observer agents running on application servers provided by the cloud computing services of Amazon (AWS), Google, IBM Cloud, and Microsoft (Azure).

The sample sizes for each API called are roughly the same and are equivalent to a call from each cloud location made to each endpoint every five minutes throughout the year.

• AWS
• Azure
• Google
• IBM Cloud

APIContext Engine

• 70 data centers worldwide
• Fully authenticated API calls
• Scheduled every five minutes
• Six years of historical records

• Insights and analysis
• CASC quality scoring
• SLA and SLO reporting
• Alerts and workflows

Observer agents run on application servers in each cloud, so every measurement is taken from outside the API.

We logged the data using our API performance and quality monitoring and management platform. Latency, pass rates, and quality scores were recorded in the same way for all APIs. For most APIs, data is available for the whole of the period. For analysis, we have grouped the APIs and endpoints we monitored into the services they represented.

Pass rates

In calculating the pass rate, we define failures to include the following:

• 5xx server-side errors: An actual reported error from the application server hosting the API.
• Network errors: Cases in which no response is returned at all.
• Content errors: Cases where the API did not return the correct content — for example, an empty JSON body or incorrect data returned.
• Slow errors: Cases in which a response is received only after an exceptionally long period.
• Redirect errors: Cases in which a 3xx redirect HTTP status code is returned.

We ignored call-specific application errors such as issues with the returned content, and client-side HTTP status code 4xx warnings caused by authentication problems such as tokens that have expired.

If an API fails, it may pass if called again immediately and succeed if the outage is transitory. However, our methodology still gives a general indication of availability issues.

n-9s reliability

The traditional telecommunications standard for service availability is five 9s — at least 99.999% uptime, or just five minutes of downtime in a year. Of 34 services analyzed in this study, no API managed to achieve five 9s. Six services achieved four 9s, up from two in 2021.

In 2022, 18% of major corporate services measured scored less than three 9s. There was a nearly 18-hour difference in unscheduled downtime observed between two leading file management services, compared to just 28 minutes between them in 2021.

Quality

APImetrics uses CASC, our patented quality scoring system, to compare the quality of different APIs. CASC (Cloud API Service Consistency) blends multiple factors to derive a "credit rating" for an API, benchmarked against our unmatched historical dataset of API test call records.

• 9.00+: Evidence of exceptional quality of operation.
• 8.00+: A healthy, well-functioning API that will give few problems to users.
• 6.00–8.00: Some significant issues that will lead to a degraded user experience and increased engineering support costs.
• Below 6.00: Considered poor. Urgent attention is required.

It is important to note that CASC scores do not fall on a normal curve. The scores are absolute, and we see no engineering reasons why prominent APIs should not consistently reach an 8.00+ CASC score.

Most services studied are of very good quality with a CASC score of 9.00 or over, which indicates excellent performance over 2020–2022. Performance in 2022 was very similar to 2020. In both years, only a small minority of services had CASC scores between 7.00 and 7.99, which indicates significant issues that need attention. The consistency of quality over three years might indicate that a plateau in API service performance has been reached, or that investment in infrastructure has been curtailed because of uncertainties around the global COVID-19 pandemic and the geopolitical situation.

Latency

Some calls will be faster than others because of the nature of the backend processing involved, so total call duration — even over a sample size of tens of millions of calls — can only give a partial view of the behavior of the APIs.

In 2022, we added a huge number of fast calls for the Serinus Cloud Watch service. This had the effect of reducing the total time average, but it doesn't affect DNS time, TCP connect time or SSL handshake time, which are determined by the cloud, region and location rather than by the call being made. Those three components are involved in the network setup of the call and are the same for all calls: they vary with cloud, region and location and over time, but not by type of call.

The average total time has gone down, but the trends between clouds and regions remain. Remember, we are dealing with a snapshot from the data we have, from which we can extract broad trends; we are not claiming that everything is like-for-like over a period of six years. We can only really do that for UK Open Banking.

• AWS was the fastest cloud by median total time from mid-2021 and the whole of 2022.
• Azure has been consistently the slowest cloud by median time since 2018.
• AWS's median DNS lookup time has continued to fall and is now about 3 ms.
• The DNS lookup times for Azure and IBM Cloud have increased from mid-2021, bucking the long-term trend and no longer demonstrating the marked quantized behavior seen since mid-2018. This suggests that there was some fundamental change made by Azure and IBM Cloud in 2021 to the configuration of their DNS services.
• Google's DNS lookup time has only increased slightly in this period, but like Azure and IBM Cloud shows more variance and is no longer as obviously quantized, suggesting that they too made configuration changes.
• DNS lookup time by region also showed a slight increase in 2022 and more variance, which reflects the changes made by Azure, Google, and IBM Cloud.

DNS matters

From mid-2021, DNS lookup times increased whether measured by cloud (except for AWS) or by region. They have also shown more short-time variation and, in most cases, are less obviously quantized than was the case in the three-year period from mid-2018 through mid-2021.

South America was the fastest region for median DNS lookup time in 2022 with a time of around 8 ms, down from about 9 ms in 2021, and South Asia the slowest at around 16 ms, up from 10 ms in 2021. Europe was the fastest region in 2021 at about 6 ms but was up to around 9 ms in 2022.

South America was the only region to have faster DNS in 2022 than 2021. Given that faster performance has been achieved in the past, this is an area for focus in improving service quality for all cloud network engineering teams, who should monitor DNS times on an ongoing basis and ensure performance criteria from all regions and locations are not becoming longer or less stable.

Recommendations

1. Actively monitor the availability and latency of all APIs you expose and consume: If you're not, you don't know how your APIs are performing right now for users in the real world. And if you're not actively monitoring your APIs, you're not managing them.
2. Benchmark the performance and quality of your APIs against those of your peers and competitors: Because you really don't want to discover that they are so much better than you.
3. Know the differences between cloud locations and user locations: Your service might be hosted in Virginia, but your users might be in Vienna and Vietnam. Make sure your choice of cloud isn't wasting valuable milliseconds for your users.
4. Not all clouds are the same, and they change over time: 70 ms or more of latency can be down to your choice of cloud. Your API users shouldn't pay tens or hundreds of milliseconds of latency just because of a decision made years ago.
5. Ensure no specific issues affect the DNS lookup time for your domain: It should be 12 ms or less. DNS should always be fast. If it isn't, do something about it, because slow DNS is just money down the drain.
6. Understand what factors impact call latency and where to focus improvements: What's the latency component most impacting user experience, and what can you do to improve it?
7. Track performance outliers and determine their causes: Slow outliers can greatly impact user experience. Are some calls taking 30 seconds or more to complete? How can you stop that?
8. Be aware of the impact of API failures and errors on user experience and business costs: All API performance and quality issues cost money. Bad APIs mean lost customers. Can your organization afford not to have the best possible APIs providing the best possible user experience?

Detailed Findings by Cloud

Failure rates by cloud and region

Failures here include 5xx server-side errors, network errors in which no response is returned, slow errors in which a response is received after an exceptionally long period, and redirect errors in which a 3xx redirect HTTP status code is returned. 4xx client-side warnings are excluded.

Latency data per cloud

The variation in median total time shows a roughly similar pattern between clouds. This is to be expected, as the variation is primarily driven by changes on the server side for API providers. The median is a better metric here as it excludes the effect of outliers.

The fall in median latency at the beginning of 2022 is owing to the introduction of the Serinus Cloud Monitor tests. These tests are both fast and high frequency, so have the effect of reducing average total time.

The tables below show the median total times, DNS times and connect times for the four clouds. Median connect time has decreased for all four clouds over the last three years, but DNS time stayed the same or increased for three clouds. Only AWS managed a close to optimal median DNS time for 2022, at 3 ms, down from 4 ms in 2021.

From mid-2018 to mid-2021, DNS time is quantized for all four clouds, but improvements to the APImetrics observer network implemented in 2021 have allowed for a more granular analysis. Azure and IBM Cloud in particular show marked increases in DNS time. Google only increases slightly, but best-in-class AWS manages to decrease.

Detailed Findings by Region

Of the six regions, Europe and North America have consistently been the fastest over the four-year period. This is likely because of a combination of relative geographical compactness and infrastructure investment. In 2022, East Asia and South America comprise an intermediate group, with Oceania and South Asia a slow group.

All regions show a decrease in connect time in 2022 compared to 2021, although there is likely still room for improvement for Oceania and South America locations. Only South America decreased DNS time in 2022 — Oceania was constant and other regions increased — so this is clearly an area where improvements are possible to return to the optimal performance seen in 2021.

We see the same behavior as noted for clouds: a slight upward trend in DNS time from mid-2021, with the clear quantized pattern seen from mid-2018 no longer evident.

Fastest and slowest cloud locations for time to connect

Glossary

The measurement terms used throughout this report, defined for teams comparing API performance across clouds, regions and providers.

4xx client-side warning

4xx HTTP status codes are generated when a request is made to an endpoint that does not exist or for which the user lacks the appropriate authorization. Because these types of issues indicate that the web server receiving the request is behaving as expected, 4xx warnings generally should not be included when determining the performance and quality of an API endpoint.

5xx server error

A 5xx server error is an actual reported error from the application server hosting the APIs.

95th percentile

The latency of the slowest 5% of calls, expressed in milliseconds.

99th percentile

The latency of the slowest 1% of calls, expressed in milliseconds.

Agent

The software agent runs at various cloud locations around the world, enabling synthetic calls to be generated as if they were being made by an end user or partner.

API

Although an Application Programming Interface is a general concept in computer systems, in the current context we are concerned only with web APIs. A user makes an HTTP request to a published API endpoint, which causes the web server to return a payload in a specified format or change the state of some remote resource.

API call

A single HTTP request made to a particular endpoint. Details of the request and the response are stored for further analysis to determine the performance and quality of the endpoint.

Authentication and authorization

Access to a particular API endpoint may depend on validating the identity of the requesting party and whether it has been granted the appropriate authorization. This might involve encrypted passwords or tokens managed through a protocol such as OAuth 2.0, supplemented by a specification such as FAPI.

Availability

Closely linked to pass rate. Strictly, availability should always be higher than the pass rate: calls may fail because of authentication issues or a malformed request while the endpoint is still available.

AWS

Amazon Web Services — observer agents running on AWS servers in different regions.

Azure

Azure from Microsoft — observer agents running on Microsoft servers in different regions.

CASC score

Cloud API Service Consistency is a patented technology that combines measures of API performance — availability, latency, reliability and number of outliers — benchmarked against a historical collection of API call records to give a single blended metric, much like a credit rating. The CASC score lets you see at a glance the quality of an API endpoint, whether it is getting better or worse, and how it compares to other endpoints.

CDN

Content Delivery Network — used by IT systems to optimize the performance of their APIs and websites.

Cloud provider

An organization that provides a commercial service hosting applications at a server. Well-known cloud providers include Google Cloud Platform, Amazon Web Services, Microsoft Azure and IBM Cloud, all of which have many locations around the world.

CMA9 banks

The nine large UK banks mandated by the Competition and Markets Authority to expose certain Open Banking APIs and regularly report on their performance: Allied Irish Bank, Bank of Ireland, Barclays, Danske, HSBC, Lloyds Group, Nationwide, NatWest Group and Santander.

Configuration

Internal and external network configuration, such as load balancers at the API gateway that direct requests to specific IP addresses, can have a significant impact on API performance and quality. Problems with external configuration such as routing tables can cause requests to be misdirected, and load balancers can direct requests to IP addresses that do not support a particular service.

DNS

Domain Name Service — the global system that allows the internet to identify the location of a specific internet service via a human-readable name.

DNS lookup time

The time the cloud provider's Domain Name Service host takes to resolve the URI for the API. This report recommends a target of 12 ms or less; AWS achieved a median of 3 ms in 2022.

Download time

The time taken for a request to be downloaded from the web server to the agent.

Endpoint

The Universal Resource Indicator that is called when you make an API call. For the call to work you will need the URI plus the parameters of the call plus security.

Failure rate

The proportion of calls made to an API endpoint that return an unexpected response.

FAPI

Financial-grade API is a technical specification developed by the OpenID Foundation. It uses OAuth 2.0 and OpenID Connect as its base and defines additional technical requirements for industries that require higher API security.

Handshake time

The time to complete the process that sets up an HTTP connection, which is called a handshake.

HTTP status code

A standard set of codes used by computer services to identify what is happening. 200 codes generally indicate a passing result; 400 codes indicate something has gone wrong but the call has not necessarily failed; 500 codes indicate a failure.

IBM Cloud

Observer agents running on IBM servers in different regions.

Latency

The time a specific action takes to complete, usually measured in milliseconds. A normal camera flash lasts one millisecond, and light travels about 300 km in that time.

Median total time

The median latency of calls from observer agents, from the point at which the call is initiated to the point at which the last byte is received. The median is used rather than the mean because it excludes the distorting effect of outliers.

Metric

A measure of some aspect of API endpoint performance, such as availability or the median length of a latency component.

Network infrastructure

The totality of the physical network elements that make up the internet, including switches, routers, and connectors such as fiber and microwave links.

Non-conformance

An API endpoint that does not respond according to its published specification. Typically this means the return payload has missing fields, contains incorrect information, or the endpoint generates errors despite the call being made to specification.

OBIE

Open Banking Implementation Entity — the UK entity managing standards for Open Banking within the United Kingdom.

Open Banking

A global paradigm for banking, financial and payment services that enables new products and user experiences powered by data exchange through APIs.

Outliers

The percentage of calls over time that were determined to be a performance outlier.

Pass rate

The percentage of calls in a given period that passed, either with an HTTP 200 code or by meeting the criteria set for a successful pass.

Performance

The set of metrics — availability, latency, reliability and number of outliers — that define how an API endpoint has behaved over time.

Processing time

How long the target server takes to handle the API call.

PSD2

Payment Services Directive 2 is a pan-European agreement to open payments and banking services, applicable to all financial service providers doing business in the EU and United Kingdom.

Quality

How good an API endpoint is from the end-user perspective. Blended metrics such as the CASC score provide a quantitative benchmark for comparing an endpoint over time, or two endpoints at a glance.

Region

The global region where the call originates.

Reliability

A reliable API endpoint tends to respond within a narrow range of time. A reliable endpoint may not necessarily be fast, but the variance in its latency will be relatively small.

Speed

The rate at which data is passed along a connection such as an intercontinental undersea fiber link. The more traffic, the slower the speed of the connection.

SSL handshake time

The time taken for the Secure Sockets Layer handshake to take place.

TCP connect time

The time it takes for the observer agent to connect over TCP to the server hosting the API.

Total time

The time between a request being made to an API endpoint and the whole of the response being received, including the DNS lookup time.

Upload time

How long the observer agent takes to upload the parameters of the call to the target server.

Version

APIs are often updated to change the way endpoints are invoked or the content of the payload returned. It is important to ensure the endpoint for the correct version is invoked; often the URI will contain the version.

About APIContext

APIContext eliminates blind spots for enterprises across the digital delivery chain with proactive synthetic monitoring, performance analytics, and automated conformance validation. Our platform delivers actionable insights so connected systems perform and conform—ensuring every interaction is trusted, secure, and compliant.

What are your APIs saying to AI? Contact us to find out.