A Standard for Cloud API Performance Measurement

The Challenge of API Measurement

For many organizations, APIs constitute a vital layer that exposes business capabilities in an easily consumable manner. Yet, many organizations struggle to achieve situational awareness of the behavior of APIs at runtime when they are actively consumed by customers and partners.

In contrast to traditional application performance management (APM), measuring API performance, consistency, and overall quality in a meaningful way is notoriously difficult. Understanding the quality of an API is a complex problem, particularly when trying to quantify whether you have functioning and reliable cloud-based APIs or SaaS products.

In today's increasingly cloud- and SaaS-reliant environments, poor performance impacts service providers in two critical ways:

1. Higher user churn and poor perception.
2. Loss of time, money, and organizational focus.

Historically, there has been no agreed methodology or standard to benchmark the performance of services from different vendors and identify end-to-end run-time issues with APIs.

Not all clouds or service hosting environments are equal. Cloud-to-cloud interactions, DNS server performance, and geographical distance can all impact API functionality, often in ways that are not obvious within traditional DevOps APM-based tooling and practices.

Meanwhile, cloud and SaaS providers often measure what they deliver—but not their users' consumption experience.

To address this gap in API measurement, APImetrics has created CASC (Cloud API Service Consistency) scoring. Through extensive research and testing, we have identified key measurements for the performance of a given API.

Overview

CASC scoring uses a Machine Learning algorithm that blends multiple metrics intelligently and identifies trends over time. The result is a simple score of out 10 that can be used as a holistic indicator of the overall health of an API.

APImetrics' goal is to enable API providers and consumers to:

• Make data-driven benchmark comparisons of service quality
• Provide transparency in SLA compliance discussions
• Identify trends and anomalies in all the services used

Measuring Cloud API Service Consistency: The CASC Standard

CASC takes a base set of metrics and analyzes them in the context of a massive historical dataset of API performance. The result is a contextual measurement of consistent behavior as well as changes over time, both of which might be overlooked by other algorithms.

Latency

The latency of an API call or sequence is a key indicator of overall service health and end-user satisfaction. High latency (delays of 500ms or more) can lead to users experiencing a “context shift” that disrupts their activity.

Consistency of Service

To claim confidently and truthfully that your data is accessible, whether to the public or to paying customers, you must provide a consistent, stable experience when accessing an API. A healthy API consistently returns data in a timely manner regardless of the time of day, location of the data, or location of the end user.

While a slow API can be designed around, an inconsistent experience is much harder for an application developer to work with.

Sample Size Depth

To maintain a healthy API, multiple call sequences must be tested with high frequency. Simulation of actual end-user activity and reporting the results are critical. This is often overlooked or intentionally disregarded, as setting up realistic, run-time monitoring that mirrors an end-user can be a complex process.

Along with setting up the right call sequences, they must be run frequently to understand the health of your API fully. Further, these calls should be made from multiple locations to inspect whether an API is performing consistently in all regions.

The CASC Score Explained

CASC is a score out of 10. By analyzing a multi-terabyte dataset of production API calls, APImetrics has been able to identify the following:

Meaningful Comparisons

The APImetrics CASC score is generated in the same way across all services, allowing for both the generation of blended metrics across multiple endpoints on the same domains AND like-for-like comparison between endpoints.

Real-World Impact of Unhealthy APIs

Practically speaking, if your services are performing inconsistently, your customers will look elsewhere for a similar competing service. Consider this real-world use case:

Holistic monitoring, which encompasses the end-user experience and overall function, instead of industry expectations, would have saved customers and revenues.

AN API WITH A GOOD SCORE will still experience between four and six incidents a month with no obvious root cause. By monitoring complex API sequences often and from multiple geographical locations, APImetrics minimizes the high cost of maintenance and repair. And in conjunction with our CASC Score, a provider can narrow the search for the cause of a problem.

AN API WITH A FAIR SCORE will generate, on average, twice as many issues as an API with a good score. As an example, APImetrics has been running calls on over two dozen of the most popular federally published APIs from data.gov. We found that over 50% of these fell into the range of fair. This indicates that multiple incidents will disrupt an end user's activity every month and will cost a provider substantial time and money to fix.

AN API WITH A POOR SCORE will have dozens of incidents each month, rendering it almost completely unusable for any given end user. The costs associated with maintenance and issue resolution will quickly eat away at even a sizable IT budget as headcount is diverted from core tasks to impact investigation and resolution.

To resolve problems that crop up, you must know the scale of the problem. Is the problem affecting only a certain area or all users? Where on the service is the issue being experienced, or is it service-wide?

Continuous monitoring of APIs in the runtime environment enables IT teams across an organization to coordinate response to issues affecting a service rapidly. By automating call sequences to mirror the activity of end users, engineers can rapidly narrow the potential problem areas in an API, saving time and money.

Factoring the Cost of an Unhealthy API

Through analysis of millions of API calls and consultation with customers over time, we have developed a system for highlighting the potential costs in terms of human engineering hours associated with an unhealthy API.

We have found that the number of incidents per API endpoint per month, which covers all problems requiring investigation, can be estimated for each level of the CASC scale.

• An API with a poor rating will experience 12+ incidents per month, with each incident lasting approximately 28 hours from the time the incident starts to its resolution.
• An API with a fair rating will experience about 9 incidents per month, with each incident lasting approximately 24 hours.
• An API with a good rating will experience about 4 or fewer incidents per month, with each incident lasting approximately 14 hours. With these numbers combined with a standardized pricing for a network engineer, we can estimate the cost of repairs to an organization per annum.

That's 1,200–2,000 hours of additional engineering work per year compared to an API with a good score, which might cost 400 hours or less per year.

Poorly performing APIs can tie up significant full-time engineering resources across multiple facets of the organization—from DevOps to network engineering to customer success/support.

Assuming a fully loaded cost of $100 per hour for an onshore engineer in the U.S., the difference between a good CASC score and a poor one can add up to $200,000 per year per service endpoint.

Summary

With API reliability becoming an increasingly vital business concern and a critical component of good API lifecycle governance, it is essential to establish standards that identify what is considered healthy and acceptable for an API and what is not. More importantly, all stakeholders in the service delivery chains need to have a basis for agreement on what the metrics are telling them.

CASC scoring lays a foundation for an industry-wide standard. With an easy-to-read numerical measurement, This “credit score” for APIs provides a simple, convenient way to understand immediately the health of an API as well as that API's health compared to others. This information can be then used to identify and correct any issues before they cause damage to an organization's revenue or reputation, ultimately resulting in happier end-users and more efficient organizations.

About APImetrics

APImetrics provides an API monitoring and run-time lifecycle governance platform for organizations offering API services across the Financial Services, Open Banking, Telecoms, Software and IoT sectors.

Our patented technology automates the process of producing regulator-ready reports for financial services providers around the world including the UK, Europe with PSD2 and beyond.

Our active monitoring platform integrates to many of the leading developer operations suites and provides API centric views of:

• Real-time API performance from more than 80 locations worldwide on four clouds
• Fully integrated security monitoring designed and built for the needs of the financial services industry and providers of complex secure token-based OAuth
• Machine learning based analysis driven by a database of more than a billion, real API calls
• Integrated reporting, analysis, and alerting
• 360-degree visibility with CASC (Cloud API Service Consistency) scoring, allowing for at-a-glance service and competitor comparisons

For more information, visit apimetrics.io, email [email protected], or call +1 833 274 6389.

Glossary

The terms used in this white paper, defined for teams that measure the performance and consistency of cloud APIs.

CASC (Cloud API Service Consistency)

The APImetrics scoring method that analyzes a base set of API metrics against a historical dataset and produces a simple score out of 10 as a holistic indicator of the overall health of an API.

APM (application performance management)

Traditional application performance management, which the paper contrasts with measuring API performance, consistency and overall quality.

Context shift

The disruption to a user's activity that can result from high latency, defined in the paper as delays of 500ms or more.

About APIContext

APIContext eliminates blind spots for enterprises across the digital delivery chain with proactive synthetic monitoring, performance analytics, and automated conformance validation. Our platform delivers actionable insights so connected systems perform and conform—ensuring every interaction is trusted, secure, and compliant.

What are your APIs saying to AI? Contact us to find out.