OpenAPI Specifications in the Real World

Executive Summary

APIs are now an essential part of all software operations, and key to building an API is having a machine-readable description and design for them. However, as this paper will show, there is still significant work to be done in ensuring that API descriptions and the APIs deployed are fit for purpose.

In our analysis of hundreds of leading APIs from some of the largest providers in the field, we discovered that almost 30% of published APIs did not match the APIs in operational use. Furthermore, only 10% of API specifications have been updated this year, with 30% still using an OpenAPI Specification version replaced in 2017.

With governance becoming a critical part of the lifecycle process and more services coming under regulatory scrutiny, this paper provides analysis and recommendations on the emerging gap and how best to fill it.

Introduction

APIs make the modern internet possible; in fact more than 80% of all internet traffic is over APIs. As APIs become more reliable, all digital applications improve, and the API ecosystem generates more value for all stakeholders. Consequently, giving API users an accurate description is critical for all API providers.

Starting at the dawn of the API era in 2010, the industry gravitated toward formally describing APIs, documenting API specifications using the OpenAPI Specification (OAS). The OpenAPI Initiative shepherds the specification, which has gone through numerous revisions and expansions. The goal of the project is to create a standard interface description for APIs, thus reducing friction by helping API designers to design better APIs and API users to use and understand APIs better.

With such a crucial goal, it is clear why building, finding, and using OpenAPI Descriptions (OADs) is important to API industry professionals. They rely on the accuracy, availability, and quality of OADs to ensure they integrate APIs correctly, leverage them for appropriate use cases, and keep their applications updated with APIs on which they rely. When OADs are inaccurate, unavailable, or of low quality, they cannot be used for their intended purpose.

Because of their importance to the API ecosystem, APIContext has analyzed OpenAPI Descriptions in the real world. This investigation had two parts.

First, for a large number of well-known API providers in our API Directory, we looked to see if we could find a publicly available description and, if we could, what type of description it was: whether official or unofficial, which version of the OpenAPI Specification, and when it was last updated.

Then, where we could find publicly available OpenAPI Descriptions, we used our Conformance Analysis product to determine where production APIs deviated from their descriptions.

We call this deviation API drift. API drift is a major source of friction in the API ecosystem. Once we measure API drift, we can eliminate it, making APIs more reliable, more performant, easier to use, and more valuable for customers, partners, and API owners alike.

Why Publish an OpenAPI Description?

There are several reasons why it is good practice to publish an OAD for your API, and also some reasons to keep your OAD private. Depending on the product or use case, business considerations could weigh in either direction. In general, however, we recommend that most application owners publish an OAD because typically the benefits outweigh the costs. In particular, any application where the API is publicly available for developers to self-service should have a public OAD.

Here are some reasons to consider making your OAD public:

• Clearer documentation: An OAD makes your API's documentation easier to understand for developers. It spells out exactly what functions the API offers, how to use them, what kinds of data the API expects, and what kinds of data the API returns. A well-documented API will be more attractive to developers and thus help boost the adoption of your API and services.
• Easier integration: With a clear description, developers can more easily integrate your API into their applications. They can use the OAS to generate code libraries, test cases, and test calls for monitoring, saving time, money, and effort, and reducing bugs.
• Standardized approach: OADs are language-agnostic, so they work regardless of the programming language used to develop the API. This creates a consistent approach that makes it easier for developers familiar with the OpenAPI Specification to work with your API.
• Leverage AI and other tools to generate code from a description: Many large language models can read OADs and use them to build working API endpoints. This accelerates development efforts and allows customer teams to leverage your APIs faster. ChatGPT, Google Gemini, Llama, LangChain, and many others support reading OAS natively.
• Accelerate documentation by generating a description from code: Additional tools work in the opposite direction — they read production APIs and auto-generate OAS documentation. Flask-RESTX is a Python Flask extension that provides a coherent collection of decorators and tools to describe an API and expose its documentation using an OAD. TypeSpec, incubated in Microsoft, is an evolved approach to building functional OAS-compliant descriptions from code. And the Optic platform automatically reviews your existing code and builds OAS specs.

Drawbacks of Publishing Public OpenAPI Descriptions

Despite the clear benefits of providing publicly available OpenAPI Descriptions, several reasons cause many DevOps and engineering teams to keep their specs private:

• Increased attack surface: Publicly exposing API descriptions can reveal sensitive information about the system's architecture and endpoints, making it easier for malicious actors to discover and exploit vulnerabilities. However, vulnerabilities are discovered in production APIs every day, often by bad actors. Often, publishing a description can allow vulnerabilities to be discovered and fixed before they are exploited.
• Misuse and abuse: Public APIs can be misused or abused by unauthorized users, potentially leading to unexpected behavior or resource depletion. However, this is better addressed with robust access management and common sense rate limits.
• Unnecessary complexity: At first, it may seem that exposing internal APIs designed for internal organization use would create unnecessary confusion for external users who don't require specific functionalities. However, detailed information about the API allows other users to use the API in novel ways that create value for all stakeholders.
• Competitive advantage: In some cases, the unique functionalities and features of an API might provide a competitive advantage, and keeping the descriptions private can help maintain that edge. However, any successful service will be well-studied through the production endpoints, so any competitive advantage from documentation will be fleeting.
• Versioning challenges: DevOps teams often work in an environment of rapid iteration and continuous deployment. Public API descriptions might need frequent updates, which could be cumbersome and create versioning challenges and potential compatibility issues. But these issues exist in fast-moving environments whether specs are published or not. Better that the published documentation always reflects the behavior of the exposed API.
• Maintenance overhead: Creating and maintaining comprehensive, up-to-date public API documentation can be resource-intensive, especially for smaller teams with limited bandwidth. However, tools reduce this burden, and the friction caused by inaccurate API documentation inevitably destroys value for all stakeholders.
• Customization: Some API providers use their own in-house tooling, optimized for their own specific requirements. For instance, Intuit explicitly states that its APIs are custom and are not OpenAPI/Swagger compliant.

Of course teams should consider all the pros and cons when deciding whether to publish their API descriptions. But, on balance, we find that openness enables the ecosystem, and recommend that most teams should publish their OADs. Certainly, as our findings show, many teams miss opportunities to make their APIs more usable by sharing their descriptions. This makes their applications less resilient.

Investigation Findings: Methodology

API descriptions are only useful if developers can find them. Our landscape investigation started with searching for the leading API providers we examined with the search terms “<name of API provider> OpenAPI Specification” (acknowledging that “specification” is often used where “description” would be technically correct) and “<name of leading API provider> Swagger”. Because OpenAPI Specifications should be easily discoverable, we limited the amount of time searching the API provider's developer pages to less than 10 minutes.

Our intention was to mimic the way a developer would investigate an API. Our search was not designed to be exhaustive, and may have missed some available OADs. However, if API descriptions are not easily found, we consider them effectively undiscoverable.

Some OpenAPI Descriptions were discoverable. Sometimes our search led us to the API developer pages of the API provider from where it was possible to download the OAD. In other cases, we were led to a GitHub repository for the API and OAD from where it was possible to download the description.

Sometimes it was clear that the spec provided was an official company resource. Other times it was not possible to determine if a description on a GitHub repository was official; and at times, it was evident that the description was one provided by users of the API. In these cases, we classified the OAD as an unofficial public description.

We supplemented this primary research with informal conversations with experts in the industry on topics related to OpenAPI Descriptions and other aspects of API definition and governance to validate our interpretation of our findings.

We used the APImetrics Conformance product to determine how closely the endpoint descriptions in the OADs matched the observed behavior of the exercised endpoints in synthetic test calls made from the system. This involved uploading an OAD into the APImetrics project for a particular API and matching the endpoint definitions against the already defined tests.

The tool identifies variance between OADs and production endpoints, against a wide range of description expectations. It documents the type and category of variance on each endpoint monitored, including endpoint behavior, content in the payload, and definitions. From there, we drew our analysis.

Key Observations

Availability of descriptions

More than a third (43%) of the APIs we examined had no discoverable public OpenAPI Description. We find this surprising, given the level of public consumption of these APIs.

The majority (57%) of APIs we examined had an OpenAPI Description that was readily available. Discoverable specs broke down into two types: 48% had an official spec, and 9% had an unofficial spec.

We find it notable that most UK banks don't provide a version of the Open Banking OpenAPI Description that is customized to their hostname. Apart from being an opportunity to reinforce their branding, it is also a best practice to provide a customized description that contains the specific paths for the Open Banking API. Even though UK regulations require a specific Open Banking API description, proper hostname definitions reassure users that the API is being properly and actively maintained.

Specification versions

Adoption of the latest versions of the OpenAPI Specification is dramatically behind where we would expect. OAS version 3.0.0 was released in July 2017. But version 2.0 (Swagger) is still used by nearly one third of applications, despite it being ten years old.

Some teams may not be able to update their spec even if they want to, because they are limited by some of their key software. For example, AWS Gateway does not support OAS v3.1, and Google's API gateway only supports OAS v2.0. These key infrastructure providers risk alienating API teams by not supporting current standards.

Frequency of description updates

Where we found OADs in GitHub repositories, it is generally possible to see when the description was last worked on. We found a number of dormant descriptions, and more than half (52%) have not been updated in the previous six months. Where OADs are updated, they are updated regularly, as often as every few days.

More than any other single indicator, this data gives us strong insight into differing priorities for API product teams. Some teams update their OADs as part of their standard operating process, indicating good hygiene and coordination between architecture and engineering. Ultimately, this leads to better adoption by developers and customers, as users of a description will understand it to be authoritative.

For the majority that have not updated their OADs in the past six months, our discussions indicate that at issue is whether the description is a definitive source of truth, or merely a guiding document. Some teams feel that a JSON schema should not be prescriptive by design. Instead, they see OpenAPI Descriptions as tools that are useful for designing, building, and supporting APIs, but which must always be supplemented by other sources of information and truth. While additional documents can always supplement descriptions, starting from a well-maintained OAD ensures that other documentation is complementary, and does not cause confusion with multiple documentation sources in conflict.

Other teams see OADs as working documents that are useful as part of the overall software development process of an organization, and should be managed as internal resources. In those cases, we would recommend not publishing a public description at all. If the description is public, API providers should ensure its accuracy.

For some, the behavior of an endpoint is what it is, and if the description does not reflect that behavior, so much the worse for the OAD — it can always be altered. We agree with this view, as long as the OAD is altered to match production endpoints. We often see important conversations between architects and engineers when these differences are highlighted. Sometimes the description ends up getting changed, and sometimes the API needs to be modified, but the end goal is always the same: alignment between documentation and production.

Observed Drift of APIs Against OpenAPI Descriptions

After reviewing production API endpoints, as well as OpenAPI Descriptions, for a wide range of API products, we found that there is a wide range of conformance between OADs and production. API drift ran the gamut from none — where the OAD and the API were perfectly in sync — to completely drifted, where there were issues in every endpoint we tested. Selected results are discussed in detail below.

A regression analysis indicated that the correlation factor between time and conformance rate is 0.46, indicating a weak positive correlation between the two variables. There's a noticeable trend that more recent specifications have higher conformance rates, but there's still a lot of variability in the data.

Banking, fintech, and bulk email management APIs showed no API drift, with 100% conformance. It's laudable that all the UK Open Banking APIs examined, both for Open Data and Production, are fully compliant with the appropriate description. In the case of UK Open Banking solutions, it's not a surprising finding because financial services companies are required to test their APIs at least annually against the regulations using a compliance suite. This bodes well for the success of Open Banking efforts around the world. It is possible to go from spec to code as well as code to spec.

So, it is clearly possible to build business processes to ensure API conformance. Which makes it all the more disappointing that several selected vendors did not have even one API endpoint that was conformant to its description in our sample endpoint testing. The three APIs with 0% conformance are aimed at enterprise users. We expect this is a fundamental issue with API supply chain accountability for business users. Customers' lack of analysis leads to a lack of priority from API product owners. With less urgency and pressure from business-to-business users, there is more leeway for things to go wrong in non-mission-critical services.

Common Error Patterns

There are a number of common error patterns we have observed from the examples we reviewed. These error patterns make the API difficult to work with, make it challenging to debug in use, or lead to a degraded experience. These include:

1. 01

   Wrong data format

   The API returned the wrong data format.

2. 02

   Wrong data content

   The API returned the wrong data content.

3. 03

   Ambiguous paths

   The API uses ambiguous paths, and therefore resolves ambiguously.

4. 04

   Undefined pass/fail

   The API has poorly defined pass/fail criteria.

Reviewing the results, we saw some commonly repeated specific errors:

1. 01

   Missing “type”

   The “type” field must be defined when the “nullable” field is used.

2. 02

   No 2XX response

   Operation must have at least one 2XX response.

3. 03

   Unresolved $ref

   “Can't resolve $ref.” Here “$ref” refers to external specifications that are not directly linked. As more programmatic solutions continue to be deployed, references become more common, and this problem will become more significant.

Invalid type errors

An invalid type is marked where the description calls for a particular type of content to be returned, typically a string or integer. This can cause issues when a number is expected, but the backend returns the number as a text string. The resulting value may not be easily used for calculations or mathematical functions. This scenario can also cause issues where a value is expected, but instead an empty object is returned with a “null” value. The “null” value can break downstream applications that are expecting a number, even if that number is zero.

In some cases, this is observed where the call was not configured correctly; however, in a large number of cases, the expected behavior of the API was simply not returning data correctly. This was observed in many APIs where fields relating to currency, type of service provided, and related data fields would frequently return “null” when the description clearly provided an expected result.

The best practice would be to return a structured object with the appropriate fields marked as null or empty. Specific use cases need further analysis, as this behavior will sometimes create additional work for engineering (the structured object has to be created) or it may create security issues (if this were deliberately implemented as security through obscurity — although if the description is publicly available, the expected structure of the returned response will typically be known to the potential attacker).

Such errors need careful analysis as they can also show an underlying problem with the API, which would need to be updated.

Invalid value errors

Where a value is specifically called out in the API — for example, a list of currency or supplier codes, or an expected string for a mandatory field — we observed that there was a degree of deviation in many cases that often revealed fundamental differences in the API behavior from the description.

For instance, a pattern seen with a number of APIs including Box and Shippo is that the OAD calls for a string to be returned, but the response is in fact empty because the queried object does not exist. Either the OAD should take the empty case into account or — likely a better solution from the user perspective — a well-constructed object should be returned with empty or null values for the fields and a warning message to the effect that the queried object did not exist or was not found.

A related example occurred with the Intercom API where the returned object was not one of the expected enumerated types because an empty or null response is not one of those types.

In most cases, a better solution should be to return a 4xx HTTP status code (404, 403, or 400) and define such behavior in the specification to make handling errors in use easier to detect. It is entirely possible that such behavior — where the API is passing incorrect data, but still returning an HTTP 200 status code — is designed to simplify work for developers, but the approach makes it much more difficult to troubleshoot problematic behavior or unexpected interactions.

Missing or incorrect hostnames

The hostname is the base URL of a service. Without knowledge of the hostname, it is not possible to programmatically make calls to the defined endpoints. Not all specifications contain a hostname, and some were detected to contain a name that was clearly part of an internal test server name.

Hostnames in public descriptions should either be the actual root for the API call, i.e. https://api.example.com, or clearly specify how the hostname is structured if there will be a customer or regional use case involved. For instance, the hostname for a particular user might contain a user ID or a server name/location. If that is hard to find or unclear, it will lead to friction with developers and make it difficult to configure the API.

The OpenAPI Specification makes it easy to include more than one hostname in the OpenAPI Description, which is useful when using the same OAD for both sandbox and production environments. Because of these OAD limitations in practice, APImetrics automatically detects the hostname and allows it to be edited or other hostnames added. However, wherever possible, it is good practice to provide the hostname.

For instance, for the UK Open Banking description, the specific hostname for the particular bank should be added to the specification rather than using the generic specification provided by the regulator. We noted that the CMA9 banks typically do this, whereas legacy banks and neobanks do not, which might indicate issues resourcing API DevOps teams in smaller organizations.

Governance and Management of Descriptions

From the analysis, we have been able to identify a number of clear governance and management requirements that are essential to the process of driving OpenAPI Descriptions.

It is vitally important that the GitHub repository and the OAD on the developer pages are the same, ideally with direct links to one consistent location. If you must maintain the same OAD in GitHub and separately in a developer portal, take extra care to ensure that they do not accidentally become misaligned. Moreover, we see that API drift occurs frequently, so it is critical to test your endpoints against their descriptions continuously in production.

Many publicly available OpenAPI and Swagger descriptions are several years old and may not be maintained assiduously, and thus often no longer reflect the behavior of the exposed endpoints. It is critical to ensure that externally and internally available specifications conform with the monitored behavior of the endpoints.

Issues with the backend and third-party systems can often lead to non-conformances. Even for systems you do not directly control, it is important to know it conforms to the description because your clients, customers, partners, suppliers, and users depend on you to resolve any problems with the API.

Many OpenAPI Descriptions are worked on actively. This ensures that the description matches the behavior of the implemented endpoints, but it can be a problem for users as descriptions may differ from a past analysis after they are updated. Ongoing monitoring with a tool like APImetrics can alleviate some of these issues by allowing you to maintain current and previous versions of the OAD.

Some styles of OAD do not state the version used. These styles should typically be avoided, and there should always be a direct indication of the OAS version used and whether or not it is current.

Recommendations

Based on our review, and on working with hundreds of APIs from some of the largest providers in the world, we would make the following recommendations for all API owners and product managers:

• Make your OpenAPI Descriptions public and easy to discover.
• Share public OADs via a direct link to a file or GitHub repository that is updated regularly.
• Establish a governance policy and process to maintain and update the description in line with the production APIs.
• Test production APIs against the published description to ensure there is no solution drift and that endpoint behavior matches the design.
• Treat clear version and date controls as an essential part of governance, validated and tested if backward compatibility is required.
• Create a dynamic “description → code → description” loop for validation of your documentation.
• Provide a hostname that is easy to discover in the documentation, to simplify making test API calls.
• When providing a documentation portal, do not list endpoint data in alphabetical order — a default behavior of many tools that makes understanding how to use an API extremely hard.
• Ensure error codes and error messages are useful and correct in the description.

While API drift is an issue for most teams in the industry, it does not need to be difficult or time consuming to guard against it. And please share your feedback with us on how you protect against API drift in your own products.

Glossary

The terms used throughout this white paper, defined for teams new to OpenAPI governance and conformance testing.

API drift

The deviation of a production API's observed behavior from its published description or contract. Drift accumulates whenever endpoints change without the description being updated, or the description changes without the endpoints following.

Conformance

The degree to which an API's live responses match what its description says they will be, measured endpoint by endpoint through synthetic test calls against production.

OpenAPI Specification (OAS)

The standard, maintained by the OpenAPI Initiative, that defines how to describe an HTTP API in a machine-readable way. Current major versions are 3.0 and 3.1; version 2.0 is the former Swagger specification.

OpenAPI Description (OAD)

The actual document describing one specific API, written to conform to the OpenAPI Specification. The specification is the standard; the description is the artifact you publish.

Swagger

The original name of the specification donated to the OpenAPI Initiative, now OAS 2.0, and also the name of a family of tooling. OAS 2.0 was superseded by 3.0 in July 2017.

Official vs unofficial description

An official description is published by the API provider itself. An unofficial description is maintained by users of the API, often in a community GitHub repository, and carries no guarantee of accuracy.

$ref

A JSON Reference pointer used inside an OpenAPI Description to reuse a schema defined elsewhere. When the target is external and unlinked, tools report “Can't resolve $ref” and validation fails.

nullable

An OpenAPI 3.0 schema keyword marking that a value may be null. It is only valid alongside a declared type, which is why “type must be defined when nullable is used” is such a common validation error.

Hostname (server URL)

The base URL of the API declared in the description's servers block. Without it, a description cannot be used to make real calls; the specification allows multiple entries so sandbox and production can share one document.

CMA9

The nine largest UK current-account providers required by the Competition and Markets Authority to implement Open Banking APIs, and to test them against the regulator's specification at least annually.

About APIContext

APIContext eliminates blind spots for enterprises across the digital delivery chain with proactive synthetic monitoring, performance analytics, and automated conformance validation. Our platform delivers actionable insights so connected systems perform and conform—ensuring every interaction is trusted, secure, and compliant.

What are your APIs saying to AI? Contact us to find out.