API Ratings: Who won the week? | March 9-16, 2020

We are now well into March (and sadly the time of COVID-19) and once again it is time to talk of the State of the APIs. 

We recently launched API.expert as a simple way to provide everybody with insights into the API economy and the leading APIs in different categories in particular. The basic service is free and will remain free, but other, related services will be built on top of it, including your very own API.expert dashboards if you want them! 

Each week, month and year we’ll look at the APIs we track in a variety of sectors for their quality (using our patent pending CASC score) and other factors like uptime and availability. 

If something is missing that you think should be here, don’t hesitate to drop us a line! 

Now, onto what the last seven days and last month can tells us about where the world of APIs is in early 2020.

API Performance Headlines 

We look at over 200 APIs and pull all the metrics together to give you a general feeling for the service quality for an organization’s APIs in a particular category.  

We do see that certain providers consistently vie for top spot in their category including Twitter, Slack and Github with others doing less well such as Cisco’s Spark services. 

Top Performers based on overall quality in each category were: 

Two changes this week with Railsbank replacing ABN AMRO at the top of PSD2 Banks and Twitter and Google swapping places in Social Networks. The Department of Justice once again takes the overall title keeping the it in the public sector.

A CASC score of over 900 is very good and one of 950 or more exceptional. Six of the eight categories are headed by organization with a CASC score of 950 or more this week, which is a good showing. Sustaining a CASC score of >925 over a period of several weeks is a good showing and congratulations to those organizations that have achieved it. 

No changes at all yet again this week, which shows that APIs are more consistent in latency than they are in overall performance. 

An important caveat: medians can be misleading! An API might have a fast median latency but produce many slow outliers. These won’t affect the median, but they mean that users can experience many calls that were unacceptably slow. So just being fast isn’t everything. You have to be reliable too if you want to have good APIs and get a high CASC score! 

As so often, FTX might have a low enough latency to top the Cryptocurrency Exchange category, but they also have a lot of outliers and a high variance on their latency, which is why they come out 11th out of 18 in terms of the CASC score this week, up from 9th last week. 

They do though have a CASC score of 859, which is a reasonable CASC score. It’s no good being fast if you are flaky, although being fast does help. 

Just one change this week with the Department of Commerce swapping places with the Bureau of Labor Statistics. Oasis Dex takes the overall Wooden Spoon for the second week in a row and has somehow managed to “top” last week’s ghastly CASC score of 289 with one of 241.

Something of interest

Halifax has been languishing at or close to the bottom of UK Open Banking (Open Data) for some time. This week is had been joined by the Bank of Scotland at 15th (out of 17) and Lloyds Bank at 16th. Now, of course, the Bank of Scotland, Halifax and Lloyds are all part of the same organization, Lloyds Group. So they are all using the same (or very similar) backend infrastructure. Generally, though Halifax has been the worst of the three.

But what is up with these APIs? After all, they are only doing the exact same thing as the Bank of Ireland and they have a CASC score of over 950 most weeks.

The three sets of APIs are the three slowest in their category. With a median latency of 640 ms, Bank of Scotland is only slightly slower than Santander (UK) (640 ms), but Santander (UK) has fewer outlier. In fact, there are several brands that have more outliers than the Lloyds Group three (Adam Bank, Coutts, NatWest, Esme Loans, Royal Bank of Scotland and HSBC). The problem is a combo of fairly high number of outliers and being slow (in the case of Halifax, at 1116 ms, really slow).

The main problem are the pass rate of just 85.42% and a outlier rate of 15.03%. These are both indicative of very poor performance and are going to really tank the CASC score.

For all three, the rate-determining step is the process time, which is pretty constant between clouds, so this is definitely a Lloyds Group issue.

Bank of Scotland and Lloyds Bank are almost identical in process time performance, suggesting they may be hosted on the same hardware. 

Halifax though is more than 300 ms slower. So it is presumably not co-hosted with Bank of Scotland and Lloyds Bank. But why are they all so slow in terms of process time compared to the rest of the CMA9? And why is Halifax even slower? 

See you again in a week to see how things are going as we power our way through the spring (in the Northern hemisphere) equinox.