Finance API Ratings, Nov 10-17: Open Banking Sandboxes

Something of interest

Sandboxes for financial APIs are crucial and sometimes don’t get the respect they deserve. That’s partly because some banks have been known to just stick up a bank-in-a-box that bears no resemblance to the production environment.

Creating a test environment that accurately reflects live behavior is no easy task. But it is a necessary one if you want third parties to be able to easily integrate with your APIs, which will help increase the number of Open Banking products and drive consumer adoption.

Having people expending time, money, and energy integrating to a sandbox only to find that the production environment APIs work completely differently is a recipe for frustration. This is a known problem in the industry and we hope that most banks recognize the importance of a performant and representative sandbox.

That’s why we monitor sandboxes in the Open Banking – UK – Sandboxes category.

It’s interesting how great a variation in performance we see from the superb Royal Bank of Scotland – with a CASC score of 9.62 – down to the dismal ForgeRock, with both of their Auth and non-Auth APIs firmly in the Red Zone.

The issue with ForgeRock is problems in authentication workflows then having a knock-on effect on other calls because of not being able to obtain valid credentials. But as can be seen for the other banks, once you have got the tests set up and running, they should continue to run smoothly.

Authentication for Open Banking is tricky to get working and we at APImetrics have a lot of experience with it. It’s because it is fiddly that it is so vital to monitor whether your authentication endpoints are working properly as they are the key to the rest of the functionality. If you can’t reliably get credentialed or authentication unexpectedly stops working, you are in trouble.

API Performance: overall quality

Week Ending 16 November 2020

Category	Organization	CASC score
Cryptocurrency Exchanges	Coinpaprika	9.34
Fintech	PayPal (Sandbox)	9.33
Open Banking: PSD2	ABN AMRO Bank (Sandbox)	9.60
Open Banking UK: Sandbox	Royal Bank of Scotland (Sandbox)	9.62
Open Banking UK: Production	Danske Bank	9.32
Open Banking UK: Open Data	Bank of Ireland	9.63

Bank of Ireland in Open Banking – UK – Open Data takes the overall title again this week with a CASC score of 9.63, down from 9.67 last week. A CASC score of over 9.00 is very good and one of 9.50 or more exceptional. Three of the six categories are headed by organizations with a CASC score of 9.50 or more this week. Sustaining a CASC score of >9.25 over a period of several weeks is a good showing and congratulations to those organizations that achieved it. All categories are headed by an API with CASC score of 9.33 or more, which is an acceptable level of performance.

Top Performers: latency

Week Ending 16 November 2020

Category	Organization	Median Latency
Cryptocurrency Exchanges	Kraken	228 ms
Fintech	Stripe	307 ms
Open Banking: PSD2	Railsbank	110 ms
Open Banking UK: Sandbox	Starling Bank (Sandbox)	141 ms
Open Banking UK: Production	Tide (Auth)	153 ms
Open Banking UK: Open Data	HSBC	85 ms

HSBC in Open Banking – UK – Open Data again takes the overall title again with a median latency of 85 ms, up from 83 ms last week. An important caveat: medians can be misleading! An Open Banking API might have a fast median latency but produce many slow outliers. These won’t affect the median, but they mean that users can experience many calls that were unacceptably slow. So just being fast isn’t everything. You have to be reliable too if you want to have good APIs and get a high CASC score!

Worst quality across all Open Banking categories

Week Ending 16 November 2020

Category	Organization	CASC score
Cryptocurrency Exchanges	Binance	7.05
Fintech	Nothing under 8.00!
Open Banking: PSD2	DNB PSD2 (Sandbox)	5.59
Open Banking UK: Sandbox	ForgeRock (Sandbox Auth)	3.89
Open Banking UK: Production	Creation Cards (Auth)	6.06
Open Banking UK: Open Data	HSBC	7.84

ForgeRock (Sandbox Auth) in the Open Banking – UK - Production category takes the overall wooden spoon with a CASC score of 3.89. We have now split out the financial APIs  into separate weekly and monthly reports. This allows us to give you a more focused view on the different classes of APIs you care about. In particular, we have been seeing increased adoption of and interest in APIs in the financial sector as uptake of Open Banking accelerates and APImetrics has over the last few weeks been able to add many UK Open Banking APIs (we now monitor all nine of the CMA9 banks) to the ones we monitor and the performance of which can be seen at our API.expert site. See you again in a week as we head ever deeper in November. And don’t forget to look out for our October financial API report and the weekly and monthly non-financial ones.

Do more with finance API monitoring!

If you're new to API monitoring, you can learn about our comprehensive features for testing, monitoring, security and performance and sign up for a free trial account.