UK Open Banking API Performance 2021–2022

Introduction

The UK continues to be at the forefront of the global Open Banking revolution thanks to the proactive attitude of the regulators. These regulators helped create an Open Banking ecosystem that encourages and facilitates smaller banks and new entries, including fintechs and neobanks, to participate in the Open Banking market. As the most advanced Open Banking market in the world, the UK provides an example of best practices in the implementation of API-based Open Banking.

We studied the performance of the large CMA9 UK banks, traditional High Street banks, credit card providers and building societies, and new entrant banks (neobanks). The endpoints were provided by the banks and measured using our patented APImetrics quality scoring system, CASC (Cloud API Service Consistency).

This report is generated from real API calls made using the FAPI compliant consent process with the partnership of tomato pay, a leading open banking provider in the UK. All calls were made between July 1, 2021 and June 30, 2022.

Key Findings

• Most, but by no means all, applications are hosted in the UK.
• While many are hosted on AWS cloud, at least 25% of banks self-host.
• There are significant differences in the performance of the three groups: Neobanks performed well, the CMA9 provided good service overall, and performance issues are evident in the traditional banking providers.
• Most providers maintained a FAPI consent journey with high quality scores: Neobanks are by far the fastest; traditional banks are the slowest with a 300 ms difference. Some traditional banks suffered from periods of poor availability.
• Where major TPPs are hosted impacts a bank's API performance: UK locations are fastest (under 400 ms for Azure, Google, and IBM), but some non-UK locations can be faster. When every millisecond counts, choosing a cloud and location with superior DNS and TCP connect time can help maintain a performant API.
• When benchmarked against general IT services using the same measurement system, the neobanks perform well, but CMA9 and traditional banks lag.
• UK-hosted applications perform better than other locations; due to distance and other factors we would not recommend hosting in Nordic data centers.

We find it remarkable that there is a significant difference between the performance of the different providers given that all types of providers had to implement this infrastructure in the last three years and none of the metrics we analyzed are overly dependent on historical technical debt.

Our analysis of Open Banking API services extends our established methodology and benchmarks for monitoring and ranking API performance and reinforces the importance of actively monitoring APIs.

Honorable mentions

Hosting patterns in UK Open Banking

In compiling the data for this report, we were able to analyze the hosting providers and locations for UK Open Banking where possible. Across all banking providers, AWS was the leading option, used by 30% of the providers; approximately 25% of the banks self-hosted and another 25% were unable to determine their core hosting service. The remaining banks were split evenly between Azure and Google services.

Where we were able to determine a hosting location, 35% were hosted inside the United Kingdom and 25% in the Republic of Ireland. The remaining hosting was distributed evenly across Belgium, Denmark, France, Germany, and the Netherlands.

As will be shown in this report, the distance from the hosting solution can play a significant role in performance. Traditional bank solutions had more hosting outside of the United Kingdom (70% of providers, compared to 60% for neobanks and CMA9 banks). If we combine numbers for the UK and Ireland, we see that 70% of the CMA9 are hosted in the UK and Ireland, 60% of neobanks, and just 50% of traditional banks. This may further explain some of the performance differences.

Performance differences

Financial-grade API (FAPI) consent journey endpoints are not reliant on legacy technology, technical infrastructure, or business processes, so we didn't expect such significant differences in total time. These differences can be attributed to several factors:

• Whether the bank uses an in-house or third-party solution for consent endpoint implementation.
• Infrastructure used for implementation: Including the length of time public keys are cached and stored for validation, and the performance of the infrastructure serving the consent journey.
• Human resources made available for API implementation, testing, monitoring and maintenance: Neobanks have a strong cultural focus on API quality and no legacy systems to maintain. CMA9 banks have significant human resources, but need to maintain legacy systems. Traditional banks tend to focus on off-the-shelf solutions that need more tuning and maintenance than they always have available to deliver the highest quality service.

Scope of Report

We made ~17 million API calls to Open Banking endpoints and measured availability and latency components including DNS (name look up), connection, handshake, and backend processing times.

We categorized the UK Open Banking landscape into groups of similar types of banks: CMA9 banks — the largest incumbent banks; traditional banks — established smaller banks and building societies; and neobanks — innovative entrants to the UK banking services market.

APImetrics monitored FAPI-compliant journey endpoints for 28 Open Banking brands, up from 16 the previous year, classified as follows.

Methodology

APImetrics used our active API performance and quality monitoring system with a Software Statement from our partner tomato pay, a UK third-party provider (TPP), to make standardized end-to-end consent calls for the financial-grade API (FAPI) journey at 28 bank brands approximately every five minutes from APImetrics software agents hosted at 22 cloud locations in Europe.

From the raw metrics we obtained, we used our patented Cloud API Service Consistency (CASC) technology to generate a quality score each month for each brand.

Each API call consists of a sequence of steps that take place before data is exchanged with the target server. These are Name Lookup (DNS), TCP Connect (the digital connection with the remote server), and TLS Handshake (the secure handshake between the two services).

As a rule of thumb, the TLS handshake is, at a minimum, double the TCP connect time. In practice, it can be even longer as there might be server analysis required to ensure that the incoming connection is valid. Pulling these together shows why it is important to consider both DNS and TCP connect times as part of your overall expected latency.

Once the connection is established, the call request is uploaded to the target server. The target server then handles the query and a response is sent. The total time elapsed is then logged. No APImetrics customer data was used in the generation of the reporting data; all data points were developed independently.

Detailed Results

In interpreting the meaning of our results, we have considered context and compared the performance of UK Open Banking consent endpoints to a collection of the top Enterprise IT APIs we monitor. The value derived from this exercise is that these are highly stable, high-quality APIs from some of the largest IT providers in the world that run focused, API-first engineering teams. This gives us an insight into how well, or not, the banking providers are doing.

Endpoints

Neobanks tend to be technology-first organizations, so it's not surprising that they perform at a higher level than business-first organizations such as the CMA9 and traditional banks. Both types performed consistently in the CASC Green Zone. Traditional banks are always the least performant, spending just as much time in the Green Zone as in the Yellow Zone — meaning that some additional work is needed for better consistency.

There is an established latency metric of ~450 ms for a delay to become perceptible to a human. Neobanks all perform better, the CMA9 are close, and traditional banks are significantly outside this interval of human latency perception.

The overall availability of traditional banks was impacted by the performance of two brands, one of which suffered major issues in October and November 2021. We would generally consider Enterprise IT APIs from vendors such as Microsoft, Google, Dropbox and Box as being the target quality and performance standard for all API providers.

Cloud datacenter performance

Most applications built to call Open Banking APIs are hosted in a cloud data center from one of the major providers. With APImetrics we have made the same calls, using the same infrastructure and configuration from AWS, Azure, Google, and IBM Cloud data centers. Latency is known to become perceptible to end users as call durations exceed 450 ms — a data point established by IBM with network based computing. With this in mind, we can see that cloud datacenter choice for an application can have a significant effect.

In terms of the total time taken for the entire API call sequence to complete including networking factors, the difference between the CMA9, traditional, and neobanks is stark. Neobanks are typically ~150 ms faster than CMA9 banks and 350 ms faster than traditional banks. Such a large difference in total time is likely to have a noticeable impact on user perception of service quality. Calls made from AWS data centers were generally faster by the end of the year ending June 2022 compared to calls made from Azure data centers, and IBM Cloud showed the most improvement.

• Three cloud locations in the UK have an average total time less than 400 ms: In 2020-21, no cloud locations had an average total time less than 400 ms, indicating an overall improvement in network infrastructure. Azure UK is on average slower than IBM UK and AWS Ireland.
• p99 times are >1,100 ms from all cloud locations: An improvement on 2020-21, when every cloud location was above 1,200 ms. In 2020-21, the four UK cloud locations had the four lowest p99 times; this was not the case in 2021-22.
• Nordic data centers should be avoided for hosting UK Open Banking applications: Calls made from Nordic locations were almost twice as slow as ones made from the UK or Ireland. Distance plus poor DNS resolution times make any data centers outside of proximity to the UK and Ireland a bad choice.

DNS and TCP connect time

• There is a significant variation in DNS lookup time between clouds, with IBM Cloud and AWS far ahead of Azure.
• From our global monitoring, we've found that a well-configured setup should have a DNS lookup time of ~14 ms or less.
• AWS and IBM make good choices for DNS resolution from the cloud data center; Azure is negatively impacted by slow DNS lookup times.

• Google Finland is known to implement strong security on inbound and outbound traffic, which explains the higher DNS resolution.
• Azure Ireland is interesting as Ireland is a common hosting location, and this represents a significant latency overhead for that data center.
• IBM Cloud provides best overall DNS performance across all data centers.

Pulling the data together and removing the impact of Nordic datacenters, we can see that the best performant data centers for Open Banking applications in the period 2021-22 were those provided by IBM and Azure in the UK and Ireland. The significant performance differences seen between the different types of providers were consistent across all cloud providers.

Traditional banks all have p99 times longer than 1,200 ms regardless of cloud, whereas neobanks have p99 times under 1,000 ms. Slow calls can impact user experience, and neobanks can provide not only faster endpoints than traditional banks, but more reliable ones with fewer outliers.

Recommendations

1. 01

   Monitor from the end-user perspective

   Only by monitoring API quality from the end-user perspective can an organization understand how its APIs behave. Internal monitoring will not expose issues for rapid escalation and resolution.

2. 02

   Mind the distance

   Be aware of the location you are planning to call your application from. UK banking providers are generally hosted in the United Kingdom, Ireland, and the Netherlands. The farther you are from those locations, the slower your call latency will be.

3. 03

   Design for the traditional-bank overhead

   Traditional banking providers are significantly slower than neobanks. If you are providing service to an audience using or integrating with traditional banks, design this overhead into your applications.

4. 04

   Choose the cloud carefully

   Best performance currently comes from applications built on AWS or IBM datacenters. Azure currently has DNS resolution issues in conventional use and may need additional work to bring performance up to acceptable levels.

About the partnership

APImetrics

APImetrics — now APIContext — provides run-time API governance solutions for organizations offering API services across the Financial Services, Open Banking, Telecoms, Software, and IoT sectors. By enabling a holistic, end-to-end view of performance, quality, and functional issues across the API surface, we allow organizations to better serve their customers and end users. Our patented technology automates the process of producing regulator-ready reports for financial services providers around the world.

• Real-time API performance from more than 80 locations worldwide on four clouds and six continents.
• Fully integrated security monitoring designed and built for the needs of the financial services industry.
• Machine learning based analysis driven by a database of more than a billion real API calls.
• Integrated reporting, analysis, and alerting.
• 360-degree visibility with Cloud API Service Consistency scoring (CASC), allowing for at-a-glance service and competitor comparisons.

tomato pay

The tomato pay API platform powers partner propositions focused on supporting small and medium enterprises, including its own simple, QR-code based payments and invoice app used by businesses and sole traders who want to receive payments in a fairer, cheaper and more ethical way.

Businesses and sole traders can benefit from a low-cost solution with no hidden fees, saving money and time compared to current payment systems. It offers instant access to money as cash settlement happens almost immediately, and access to all bank accounts and transactions in one place. Everyone can support their local communities and help them thrive by paying their neighbourhood businesses in a cashless, faster, cheaper, hassle-free way.

Glossary

The measurement and Open Banking terms used throughout this report, defined for teams comparing API performance across banks, clouds and locations.

4xx HTTP status codes

Generated when a request is made to an endpoint that does not exist or for which the user lacks the appropriate authorization. Because these issues indicate that the web server receiving the request is behaving as expected, 4xx client-side warnings should not generally be included when determining the performance and quality of an API endpoint.

5xx server error

An actual reported error from the application server hosting the APIs.

Agent

The software agent run at various cloud locations around the world, enabling synthetic calls to be generated as if they were being made by an end user or partner.

API

In the current context we are concerned only with web APIs. A user makes an HTTP request to a published API endpoint, and the web server returns a payload containing information in a specified format or changes the state of some remote resource.

API call

A single HTTP request made to a particular endpoint. Details of the request and the response are stored for further analysis to determine the performance and quality of the endpoint.

Authentication and authorization

Access to an API endpoint may depend on validating the identity of the requesting party and that it has been granted the appropriate authorization. This might involve encrypted passwords or tokens managed through a protocol such as OAuth 2.0, supplemented by a specification such as FAPI.

Availability

Closely linked to pass rate. Strictly, availability should always be higher than the pass rate: calls may not pass because of authentication and authorization issues or because the request is malformed, while the endpoint is still available.

CASC score

Cloud API Service Consistency (CASC) is a patented technology that blends availability, latency, reliability, consistency of response and number of outliers into a single score out of 10, benchmarked against a historical collection of API call records. Green is 8.00 or greater, yellow is 6.00–7.99, and red is below 6.00 and needs urgent remedial attention.

CMA9 banks

The nine large UK banks mandated by the Competition and Markets Authority to expose certain Open Banking APIs and regularly report on their performance: Allied Irish Bank, Bank of Ireland, Barclays, Danske, HSBC, Lloyds Group, Nationwide, NatWest Group and Santander.

Cloud provider

An organization that provides a commercial service hosting applications at a server. Well-known cloud providers include Google Cloud Platform, Amazon Web Services, Microsoft Azure and IBM Cloud.

Configuration

Internal and external network configuration, such as load balancers at the API gateway that direct requests to specific IP addresses, can have a significant impact on API performance and quality.

DNS latency

DNS is the global service that identifies where a particular service is located on the internet. The lookup time is the time taken for the cloud service making the API call to identify where the target server is and route the request.

Download time

The time taken for a request to be downloaded from the web server to the agent.

Endpoint

The web address that is called when you make an API call. For the call to work you need the URI plus the parameters of the call plus security.

Failure rate

The proportion of calls made to an API endpoint that return an unexpected response.

FAPI

Financial-grade API (FAPI) is a technical specification developed by the Financial-grade API Working Group of the OpenID Foundation. It uses OAuth 2.0 and OpenID Connect as its base and defines additional technical requirements for industries that require higher API security.

GET

The simplest HTTP verb, sending a request to an API endpoint that gets a resource such as a list of account transactions. Parameters and headers allow complex requests to be made with a GET.

Handshake time

The time to complete the process that sets up an HTTP connection, which is called a handshake.

Latency

In general, latency is the same as total time. It consists of several components including name lookup (DNS) time, handshake time, upload time, processing time and download time. In this report, latency is reported in milliseconds.

Metric

A measure of some aspect of API endpoint performance, such as the availability or median length of a latency component.

Network infrastructure

The totality of the physical network elements that make up the systems that together comprise the internet, including switches, routers, and connectors such as fiber and microwave links.

Non-conformance

An API endpoint that does not respond according to its published specification. Typically the return payload has missing fields or incorrect information, or the endpoint generates errors and warnings despite the call being made according to specification.

OBIE

The Open Banking Implementation Entity, the UK entity managing standards for Open Banking within the United Kingdom.

Open Banking

A global paradigm for banking, financial and payment services that enables innovative new products and user experiences powered by data and information exchange through APIs.

Performance

The set of metrics such as availability, latency, reliability and number of outliers that define how an API endpoint has behaved over a period of time.

Processing time

One of the components of latency: the time the server takes to process a received request before sending the response back to the end user.

PSD2

Payment Services Directive 2 is a pan-European agreement on payment services applicable to all financial services providers doing business in the EU and United Kingdom. Responsibility for implementation lies with each country.

Quality

How good an API endpoint is from the end-user perspective. Blended metrics such as the CASC score provide a quantitative benchmark for comparing an endpoint over time, or two endpoints at a glance.

Reliability

A reliable API endpoint tends to respond within a narrow range of times. A reliable endpoint may not necessarily be fast, but the variance in its latency will be relatively small.

Speed

The rate at which data is passed along a connection such as an intercontinental undersea fiber link. The more traffic, the slower the speed of the connection.

Total time

The time between a request being made to an API endpoint and the whole of the response being received, including the name lookup (DNS) time, TCP connection time and handshake time.

Upload time

The time taken for a request to be uploaded from the agent to the web server.

Version

APIs are often updated to change the way endpoints are invoked or the content of the payload returned. It is important to ensure the endpoint for the correct version is invoked; often the URI will contain the version.

About APIContext

APIContext eliminates blind spots for enterprises across the digital delivery chain with proactive synthetic monitoring, performance analytics, and automated conformance validation. Our platform delivers actionable insights so connected systems perform and conform—ensuring every interaction is trusted, secure, and compliant.

What are your APIs saying to AI? Contact us to find out.