Monitoring the Untestable

Abstract

Who should read this report?

This white paper discusses the challenges for banks, financial institutions and other organizations to monitor PSD2 APIs in ways that meet both their security requirements and the operational ones that support Open Banking. We explore the need for a change in API practices, and strategies that can be used if you cannot test against actual scenarios.

PSD2 (Payment Services Directive 2) came into force in the European Single Market on 13 January 2018. Banks are obligated to implement open banking solutions that allow properly authorized third parties to access customer accounts. To satisfy regulators, banks must prove that the exposed APIs are working according to specification. Banks will also want to know if there are problems with the performance and quality of the third-party PSD2 APIs they consume.

Banks can carry out passive monitoring with API gateways and other internal stack monitoring tools to measure some characteristics of the APIs and detect certain types of errors. However, this type of passive monitoring only gives a partial understanding of API behavior. It can only monitor what is being used when it is being used, missing less frequently used APIs or failing APIs during a period of low use.

Additionally, users outside your network will access services through third parties, and any external failure or access issue will be missed by internal passive monitoring. For third-party PSD2 APIs exposed by other banks, active monitoring is the only mechanism through which any reliable knowledge of the APIs can be obtained. To gain a full understanding of API behavior from the end-user perspective and detect the full range of problems, active periodic monitoring using transactions from external locations is essential.

PSD2 and APIs

PSD2 came into force in the European Single Market (EU, including the UK, Iceland, Liechtenstein, Norway and Switzerland) on 13 January 2018. PSD2 is designed to increase competition in the payment services market by obligating banks to implement open banking solutions that allow properly authorized third parties, both incumbent institutions and new entrants, to access customer accounts to provide a wide variety of account and payment services. Although PSD2 does not mandate specific technological methods for banks to facilitate this access, in practice web APIs (Application Programming Interfaces) are invariably used.

Regulators require that PSD2 APIs meet certain specified quality standards, and banks must prove that their APIs meet these criteria. Furthermore, it will be necessary for banks to understand the performance and quality of the PSD2 APIs of other banks, as these APIs affect customer experience. It will thus be vital for banks to monitor continuously the performance and quality of both the PSD2 APIs they themselves expose, and those of the third-party PSD2 APIs they consume.

Why passive monitoring is not enough

To understand the behavior of APIs and prove that they are performing as expected to regulators, managers and other banks, it is vitally important to monitor them continuously. Passive monitoring with API gateway and stack monitoring tools is helpful for information about load and throughput on APIs. It can also be used to detect non-200 HTTP code responses.

However, for many reasons it is not possible to rely solely on passive monitoring.

• A server-side error should lead to a HTTP 5XX response: In practice, an API may return a pass even when there has been a server-side error. Legacy service installations may return a pass code with a text-based error message, leaving the end user with an error that cannot be detected from gateway logs.
• The end user is interested in the content of the returned payload: Payload content is not stored in the logs, so an empty, incomplete or missing payload may be detected only by the end user.

It is therefore important to carry out active monitoring using synthetic API calls made periodically from locations that reflect those of the API end users. Active monitoring tools are a vital complement to API gateway monitoring tools.

Why active monitoring is vital

Passive monitoring only tells you about one side of the gateway (internal). But the user is on the other side of the gateway and the network (external). To understand the end user experience fully, it is necessary to understand the external side.

Even though the external side might appear to be outside the bank's control, it is only by taking it into account that a true picture can be formed of how users experience the APIs. Performance might be much lower from certain geographical locations or cloud services because of network architecture issues that can be resolved directly by the bank or in conjunction with a cloud service or internet service provider.

• You can see how the end-user experience varies between API calls made from different cloud services and geographical locations: External vantage points reveal network, DNS and routing issues that internal tools cannot see.
• Regular periodic testing means you always have a real-time picture of API behavior: Less frequently used APIs still have a heartbeat and issues can be detected before users experience deterioration in service.
• The end user's experience ultimately defines API performance and quality: A request might never reach the gateway, or the gateway may appear healthy while the returned availability, latency or payload is wrong.

Why testing against the live environment is important

Bank security teams are understandably reluctant to allow testing against live production environments with real accounts, or creating test accounts in these environments. This results in a Catch-22 at the heart of API measurement: not knowing whether APIs are working to specification in the live environment is an unacceptable risk to the bank, but making synthetic transactions against test accounts in live environments may also be an unacceptable risk.

Testing against sandbox environments is often substituted, but there are significant issues with this approach.

• Banks have complex, heterogeneous software ecosystems: It may not be possible to deploy a test version of every system used within the bank, and a testbed cannot reproduce the exact scale and nature of production behavior.
• Backend services in testbeds may be stubbed off and replaced by mocked APIs: A mocked API may not demonstrate the same bugs as the real API, while production databases contain a wider variety of data and real-world errors.

Examples of what you can miss

External performance varies significantly by cloud and location

Internal passive monitoring may show a median server response time of 600 ms per transaction. Looking at the entire end-to-end experience from different cloud services can show median response time increasing to almost 1.5 seconds, with some calls hosted on Microsoft Azure taking more than 30 seconds.

Content errors

In many cases, the content returned by the API might not be what was expected. The same API may return the expected content string in one case and a passing code with no content in another. In a banking context, this could be a failed transaction or missing data in a comparison or similar service.

HTTP codes do not tell the whole story

A 4XX might be a symptom of a more serious problem with a consent flow or authentication server. It could be that a legacy system is down, with the failure apparent only to end users and third-party providers. Conversely, a successful 200 response is not always a functional pass.

What to do next

Define your active monitoring and real functional testing strategy

In this brave new world of Open Banking, banks of all sizes are only going to be as good as the performance and quality of their APIs. A move into the PSD2 world must ensure that the critical services under offer are monitored and tested rigorously. Although passive monitoring and sandboxes are important in understanding API behavior and risk, it is only through active test account monitoring through periodic synthetic transactions in the production environment that a true picture can be formed, and the risks associated with it properly mitigated.

To build trust between organizations that use PSD2 APIs on both a bilateral and multilateral basis, banks should explore opportunities in 360° monitoring - in which Bank A actively monitors the APIs of itself and Bank B, and Bank B actively monitors the APIs of itself and Bank A.

APIContext and active monitoring

APIContext provides the state-of-the-art active API performance and quality monitoring solution. APIContext allows you to undertake active monitoring of APIs from locations in public cloud services on five continents, and from secure, dedicated servers controlled by customers.

Our Insights technology leverages a historical dataset of API call records to produce a single credit-rating-like number that lets API managers see at a glance the quality of any API against any other API, and whether it is improving or getting worse.

The analytics suite lets you dive into API results to see how they are really behaving. You can create workflows and back-to-back tests that use result parameters extracted earlier as input later to simulate complex business logic. APIContext can also define custom error and warning conditions, capturing API behavior from the user's point of view and raising problems to the higher-level management system.

Glossary

The terms used in this white paper, defined for teams responsible for the quality and compliance of Open Banking APIs.

API

Application Programming Interface - a way of exchanging data and information between different computer systems.

REST

Representational State Transfer - a common standard for exchanging data between websites, mobile applications and more.

SOAP

Simple Object Access Protocol, an older protocol often used in enterprise and backend banking systems.

API Gateway

A software system designed to manage the connection between backend services and the public internet, including security, load and traffic data.

Stack-based monitoring tools

Tools used inside infrastructure and backend systems to monitor different parts of internal IT infrastructure.

Backend services

The services an organization owns or controls internally.

Cloud servers

Publicly available hosting and infrastructure commonly used by third-party developers and services.

HTTP codes

The code returned after an API call is made to the server. 200 is a successful response but is not always indicative of a functional pass; 400 signals a security issue or similar problem; 500 is an actual server error.

Sandbox environment

An environment set up for test purposes but detached from actual services, often used for low-risk training or testing.

Production environment

Actual live software being used by customers and third-party providers.

About APIContext

APIContext eliminates blind spots for enterprises across the digital delivery chain with proactive synthetic monitoring, performance analytics, and automated conformance validation. Our platform delivers actionable insights so connected systems perform and conform—ensuring every interaction is trusted, secure, and compliant.

What are your APIs saying to AI? Contact us to find out.