API Ratings | Sept 14-21, 2020: Playing in Sandboxes

We are now in the final third of September(!) and that time of the week has already arrived again for us to look at some key API ratings over the past seven days.

We recently launched API.expert as a simple way to provide everybody with insights into the API economy and the leading API ratings in different categories in particular.

The basic service is free and will remain free. But other, related API ratings services will be built on top of it, including your very own API.expert dashboards if you want them!

Each week, month, and year we’ll look at API ratings in a variety of sectors for their quality CASC score using our patented system (US Patent 10,644,962, if you fancy a little light bedtime reading) that allows you to see the API ratings and compare them to other ones, as well as factors like uptime and availability.

If something is missing in our API ratings that you think should be here, don’t hesitate to drop us a line!

Now, onto what the past week tells us about the state of API ratings during this insane week of an unending year.

API Performance Headlines

To compile our API ratings, we look at more than 200 APIs. Then we pull all the metrics together to give you a general feeling for API quality in a particular category.
We do see that certain providers consistently vie for top spot in their API ratings category, including GitHub and Google. Others do less well, such as Halifax and the NHS.

Top API Ratings – CASC Score

Sept 14-21, 2020

Category	Organization	CASC score
Corporate Infrastructure	Github	9.68
COVID-19	COVID-10 Finnhub API	9.24
Cryptocurrency Exchanges	Coinpaprika	9.35
Fintech	Stripe	9.50
Search	Google	9.43
Social Networks	Twitter	9.69
Open Banking – PSD2	Nordea Bank	9.89
Open Banking – UK Sandbox	Royal Bank of Scotland (Sandbox)	9.85
Open Banking – UK Production	Royal Bank of Scotland	8.76
Open Banking – UK Open Data	Allied Irish Bank (GB)	9.80
UK Government	GOV.UK	9.86
US Government	Department of Justice	9.72

Three changes in API ratings this week:

• Coinpaprika swapping with OKEx in Cryptocurrency Exchanges
• Royal Bank of Scotland topping the new category of Open Banking – UK Production
• Allied Irish Bank (GB) replacing Bank of Ireland in Open Banking – UK Open Data

Nordea retains the overall title for the private sector with a CASC score of 9.89.

A CASC score of over 9.00 is very good and one of 9.50 or more exceptional for an API rating. Six of the eleven categories are headed by organization with a CASC score of 9.50 or more this week, the same as last week. Sustaining a CASC score of >9.25 over a period of several weeks is a good showing for API ratings. Congrats to those organizations that achieved it.

All API rating categories but one are headed by an API with CASC score of 9.20, which is an acceptable level of performance.

Top API Ratings – Latency

Sept 14-21, 2020

Category	Organization	Median Latency
Corporate Infrastructure	Microsoft Office	234 ms
COVID-19	CDC Tools	163 ms
Cryptocurrency Exchanges	FTX	218 ms
Fintech	Stripe	302 ms
Search	Facebook	322 ms
Social Networks	Google	57 ms
Open Banking - PSD2	Railsbank	94 ms
Open Banking - UK Sandbox	Royal Bank of Scotland (Sandbox)	148 ms
Open Banking - UK Production	Cashplus	137 ms
Open Banking - UK Open Data	HSBC	81 ms
UK Government	Police.UK	90 ms
US Government	Department of Justice	51 ms

A stable week, with just one change in the API ratings: Cashplus topping the new category of Open Banking – UK Production. Department of Justice takes the overall title with a median latency of 51 ms, up from 50 ms last week. An important caveat for these API ratings: medians can be misleading! An API might have a fast median latency but produce many slow outliers. These won’t affect the median, but they mean that users can experience many calls that were unacceptably slow. So just being fast isn’t everything. You have to be reliable too if you want to have good APIs and get a high CASC score! As so often, FTX tops the Cryptocurrency Exchange API rating category by median latency. This week they are down to 14th out of 18 overall with a CASC score of 8.29, placing them in the Green Zone, but not as solidly as we'd like to see it.

Worst API Rating Overall

Sept 14-21, 2020

Category	Organization	CASC Score
Corporate Infrastructure	Cisco Spark	7.93
COVID-19	Coronavirus Data API (thevirustracker.com)	6.52
Cryptocurrency Exchanges	RadarRelay	6.97
Fintech	OBIE Directory (Prod Auth)	7.36
Search	Nobody below 8.00!
Social Networks	Nobody below 8.00!
Open Banking - PSD2	BNP Parribas (Sandbox)	7.69
Open Banking - UK Sandbox	Barclays (Sandbox Auth)	1.00
Open Banking - UK Production	Creation Cards (Auth)	4.05
Open Banking - UK Open Data	HSBC	7.98
UK Government	NHS	6.99
US Government	Federal Communications Commission	7.52

Five changes this week in the API ratings!

• Cisco Spark replacing DocuSign in Corporate Infrastructure
• OBIE Directory (Production Auth) a new entry in Fintech
• BNP Paribas (Sandbox) a new entrant in Open Banking – PSD2
• Creation Cards (Auth) taking the bottom spot in the new category of Open Banking – UK Production
• Halifax replacing HSBC in Open Banking – UK Open Data

Barclays (Sandbox Auth) again takes the overall API rating loser spot for the week with a CASC score of 1.00.

Something of interest

As you can see in our revised API ratings, APImetrics has recently been able to start monitoring not just the sandbox environments of some of the banks in the UK that are offering open banking, but also certain APIs in the production environments too. Generally, we separate the functionality the bank provides into an auth piece and a operational piece for convenience. It is possible and useful to test the auth piece without having credentials to test the operational piece. We can see that Royal Bank of Scotland tops both API rating categories. (We don’t currently monitor the NatWest test environment.) Royal Bank of Scotland and NatWest are brands of the same bank, so we would expect them to have similar performance. Interestingly, the auth pieces have similar latency for both the sandbox and production environments. We see a lot of outliers for the auth endpoints in both the sandbox and production environments. For the operational piece, we see that the number of outliers in the production environment is over 30 times as many as for the sandbox. This might be because of heavier load on the production environment. Note that the median latency is higher for the production environment by 63 ms. Using an active API performance and quality monitoring system like APImetrics allows you to determine exactly where the rate determining steps are with your APIs, and where the issues that might affect user experience could be. It also lets you compare yourself with your peers. Both Lloyds Banking and Allied Irish Bank (GB) have pretty solid sandboxes, but they are much slower than Royal Bank of Scotland. Now with APIs and API ratings, sometimes you might be comparing apples and oranges. Two corporate infrastructure or even banking APIs might be doing quite different things. But this isn’t the case with open banking where all the banks are implementing the same APIs. So it is absolutely crucial to benchmark yourself to get true API ratings. If your competitor has a median latency of 148 ms for their operational piece, why is your latency 850 ms? See you again in a week as we look once more at API ratings in the last week of September. Where did this month go?