Monitoring the Untestable

Abstract

Who should read this report?

This white paper discusses the challenges for banks, financial institutions and other organizations to monitor PSD2 APIs in ways that meet both their security requirements and the operational ones that support open banking. We explore the need for a change in API practices, and strategies that can be used if you can’t test against actual scenarios.

PSD2 (Payment Services Directive 2) came into force in the European Single Market on 13 January 2018. Banks are obligated to implement open banking solutions that allow properly authorized third parties to access customer accounts. To satisfy regulators, banks must prove that the exposed APIs are working according to specification. Banks will also want to know if there are problems with the performance and quality of the third-party PSD2 APIs they consume.

Banks can carry out passive monitoring with API gateways and other internal stack monitoring tools to measure some characteristics of the APIs and detect certain types of errors. However, this type of passive monitoring only gives a partial understanding of the behavior of APIs. For instance, it can only monitor what is being used when it is being used, therefore missing failures of less frequently used APIs or failing APIs during a period of low use.

Additionally, users outside your network will access services through third parties, and any external failure or access issue will be missed by internal passive monitoring. For third-party PSD2 APIs exposed by other banks, active monitoring is the only mechanism through which any reliable knowledge of the APIs can be obtained. To gain a full understanding of API behavior from the end-user perspective and detect the full range of problems, active periodic monitoring using transactions from external locations is essential.

PSD2 allows authorized third parties to access customer accounts to provide services, including making and receiving payments. This involves access to data and information in banks’ live production environments. Traditionally, bank security teams have been reluctant to allow access to production environments, even for testing.

PSD2 and APIs

PSD2 came into force in the European Single Market (EU, including the UK, Iceland, Lichtenstein, Norway and Switzerland) on 13 January 2018. PSD2 is designed to increase competition in the payment services market by obligating banks to implement open banking solutions that allow properly authorized third parties, both incumbent institutions and new entrants, to access customer accounts to provide a wide variety of account and payment services. Although PSD2 does not mandate specific technological methods for banks to facilitate this access, in practice web APIs (Application Programming Interfaces) are invariably used.

Regulators require that PSD2 APIs meet certain specified quality standards, and banks must prove that their APIs meet these criteria. Furthermore, it will be necessary for banks to understand the performance and quality of the PSD2 APIs of other banks, as these APIs affect customer experience. It will thus be vital for banks to monitor continuously the performance and quality of both the PSD2 APIs they themselves expose, and those of the third-party PSD2 APIs they consume.

Why passive monitoring is not enough

To understand the behavior of APIs and prove that they are performing as expected to regulators, managers and other banks, it’s vitally important to monitor them continuously. Passive monitoring with API gateway and stack monitoring tools are helpful for information about load and throughput on APIs. It can also be used to detect non-200 HTTP code responses (see Glossary).

However, for many reasons it’s not possible to rely solely on passive monitoring.

When properly implemented, a server-side error should lead to a HTTP 5XX response (a response explicitly stating that there has been an error) being returned.

• In some cases, whether because of a bug or design oversight, an API may return a pass, even when there has been a server-side error.
• A particular issue with legacy services installations is that these will typically return a pass code, but with a message containing a text-based error message.
• In these cases, the end user of the API will experience an error, but it will not be possible to detect it from examining gateway logs.

The end user is interested in the content of the returned payload.

• The payload content is not stored in the logs, so if there is an issue with the payload (for instance, if the payload is empty, or data is incomplete or missing), it will be detected only by the end user.

If is therefore important to carry out active monitoring using synthetic API calls made periodically from locations that reflect those of the API end users. Active monitoring tools are a vital complement to API gateway monitoring tools.

Why active monitoring is vital

Passive monitoring only tells you about one side of the gateway (internal). But the user is on the other side of the gateway and the network (external). To understand the end user experience fully, it is necessary to understand the external side.

Even though the external side might appear to be outside the bank’s control, it’s only by taking it into account that a true picture can be formed of how users experience the APIs. For instance, performance might be much lower from certain geographical locations or cloud services. This can be caused by network architecture issues that can be resolved either directly by the bank or in conjunction with the cloud service or internet service provider.

Active monitoring allows a complete picture of the behavior of an API.

• You can see how the end-user experience varies between API calls made from different cloud services and geographical locations.

Regular periodic testing of an API means you always have a real-time picture of its behavior.

• Some important APIs might only be used infrequently by external users.
• With API gateway monitoring tools, if an API is not being used, it’s not being monitored.
• With periodic synthetic transactions, for less used APIs, you have a record of an API’s heartbeat and can detect issues with the API, even before users experience any deterioration in service.

The end user’s experience ultimately defines the performance and quality of an API.

• The API might appear to be working according to passive monitoring at the gateway, but if the end user is not getting what is expected (availability, latency, payload), the API is not functioning properly.
• In some cases, a request might not even reach the gateway and therefore cannot appear in logs; this kind of failure is detected by active monitoring.

Why testing against the live environment is important

Bank security teams are understandably reluctant to allow testing against live production environments with real accounts, or creating test accounts in these environments.

This results in a Catch-22 style conflict at the heart of API measurement. Not knowing whether APIs are working to spec in the live environment is an unacceptable risk to the bank. But making synthetic transactions against test accounts in live environments may also be an unacceptable risk. Which risk is more important?

Testing against sandbox environments is often substituted, but there are significant issues with this approach.

Banks have complex, heterogeneous software ecosystems.

• Even in a comprehensively resourced sandbox environment, it may not always be possible to deploy a test version of every system used within the bank.
• It is intrinsically difficult for a testbed to reproduce the exact scale and nature of production environment behavior.
• It would only be possible to stimulate the production environment fully if the sandbox were the same size in terms of users, load, backend databases.

In practice, backend services in testbeds may be stubbed off and replaced by mocked APIs

• Although this can be useful for certain kinds of testing purposes, many of the issues encountered by real-world users will not be observed.
• A mocked API may not demonstrate the same bugs as the real API.
• Production environment backend systems have to deal with the actual information in the databases they use, which might be quite different (for instance, containing a wider variety of types of data or much more data that is incorrect or incomplete) to those in testbed databases.

Examples of what you can miss

External performance varies significantly by cloud and location

Internal passive monitoring shows a median server response time of 600ms per transaction. However, when we look at the entire end to end experience when made from different clouds services we see that median response time increasing to almost 1.5 seconds. However, if we review the graph of results from external clouds we also see that calls made from servers hosted on Microsoft Azure that 5% of all calls take more than 30 seconds.

Content errors

In many cases, the content returned by the API might not be what was expected. Here we have the same API, one returned the expected content string; the other also returns a passing code, but with no content returned. In a banking context, this could be a failed transaction or missing data in a comparison or similar service.

Resolving the dilemma

Negative testing

This can be a useful technique for detecting security issues. Some API calls should fail. For instance, any call that lacks the proper authentication and authorization should return a specific type of error code (HTTP 4XX) – this demonstrates that the security is working as expected. A pass in this instance suggests a potentially serious security flaw, exposing end-user accounts and data to bad actors.

The active monitoring system will immediately raise an alarm, allowing the issue to be dealt with straightaway. Testing for failures in the live system, i.e., test payments, should fail, and the fact it does proves everything in the security chain is working as expected. A disadvantage is that the actual flow is not tested explicitly.

Sandbox testing

This is acceptable to internal regulators and security teams because of the lower risk associated with it. However, it is important to ensure that the testbed reflects the production environment and end user experience as closely as possible to ensure that sandbox API behavior is close to that of live production. Testing only against the sandbox can lead to serious issues not being identified. For instance, there is no guarantee that the sandbox will behave in exactly the same way as the production environment, even if no stubbed or mocked APIs are used. The size and velocity of the live system will invariably be different from the testbed, which will present user-affecting issues not seen in the sandbox.

Test accounts on production environments

Ultimately, the only way of capturing the actual quality and performance of APIs from the end user perspective is to use active testing of a range of representative test accounts in the live environment. These tests can be undertaken by trusted personnel from dedicated systems, such as active monitoring software running on a Raspberry Pi or a security-locked VM running in an external cloud environment, with additional measures taken to limit access, such as specific security certificates. This minimizes the possibility malicious use of these test accounts. Precautions, such as the use of time-limited tokens, can also be put in place that allow only the test accounts to be accessed by the active monitoring system. Furthermore, negative testing should be used in conjunction with the test accounts to ensure that neither the test accounts nor other (dummy) accounts can be accessed by the APIs without the proper authentication and authorization being in place.

What to do next

Define your active monitoring and real functional testing strategy

In this brave new world of open banking, banks of all sizes are only going to be as good as the performance and quality of their APIs. A move into the PSD2 world must ensure that the critical services under offer are monitored and tested rigorously. Although passive monitoring and sandboxes are important in understanding API behavior and risk, it is only through active test account monitoring through periodic synthetic transactions in the production environment that a true picture can be formed, and the risks associated with it properly mitigated.

To build trust between organizations that use PSD2 APIs on both a bilateral and multilateral basis, banks should explore the opportunities in 360° monitoring – in which Bank A actively monitors the APIs of itself and Bank B, and Bank B actively monitors the APIs of itself and Bank A.

APImetrics and active monitoring

APImetrics provides the state-of-the-art active API performance and quality monitoring solution. APImetrics allows you to undertake active monitoring of APIs from locations in public cloud services on five continents, and from secure, dedicated servers controlled by our customers.

Our proprietary patent for Insights technology leverages our unrivalled historical dataset of API call records to produce the CASC score, which blends different measures of API performance to give a single credit rating-like number that lets API managers see at a glance the quality of any APIs against any other API, and whether it is improving or getting worse.

Our analytics suite lets you dive into API results to see how they are really behaving. You can create workflows, and back-to-back tests that use result parameters extracted earlier as input later to simulate complex business logic.

APImetrics can be used to define custom error and warning conditions, capturing API behavior from the user’s point of view and problems raised to the higher level management system.

To find out how APImetrics can help you resolve the dilemma of API monitoring in the coming open banking revolution, contact us today through our website or email.

Glossary

API

Application Programming Interface – a way of exchanging data and information between different computer systems

REST

REST (Representational State Transfer) – the most common standard for exchanging data between websites, mobile applications and more

SOAP

SOAP (Simple Object Access Protocol) an older protocol often used in enterprise and backend banking systems

API Gateway

A software system designed to manage the connection between backend services and the public internet. The gateway manages security for the API calls, manages the load and provides data on traffic

Stack based monitoring tools

Tools used inside infrastructure and backend to monitor different parts of internal IT infrastructure

Back end services

The services you own or control inside your organization

Cloud servers

Publicly available hosting and infrastructure commonly used by third-party developers and services to host their technology. Common cloud providers include AWS (Amazon Web Services), Azure (Microsoft Cloud), IBM and Google.

HTTP codes

The code returned after an API call is made to the server. 200: A successful response from the server – however, it’s not always indicative of a functional pass. 400: A security issue or similar with the API call – the action is not allowed. 500: An actual error returned from the server –something has failed.

Sandbox Environment

An environment set up for test purposes but detached from actual services. Often used for testing purposes or for low-risk training purposes such as hackathons

Production Environment

Actual live software being used

About APIContext

APIContext eliminates blind spots for enterprises across the digital delivery chain with proactive synthetic monitoring, performance analytics, and automated conformance validation. Our platform delivers actionable insights so connected systems perform and conform—ensuring every interaction is trusted, secure, and compliant.

What are your APIs saying to AI? Contact us to find out.