API Ratings | Sept 7-14, 2020: DocuSign Plunges

How did we already arrive at the middle of September?! Let’s take a look at some key API ratings over the past seven days.

We recently launched API.expert as a simple way to provide everybody with insights into the API economy and the leading APIs in different categories in particular.

The basic service is free and will remain free, but other, related services will be built on top of it, including your very own API.expert dashboards if you want them!

Each week, month, and year we’ll look at API ratings in a variety of sectors for their quality CASC score using our patented system (US Patent 10,644,962, if you fancy a little light bedtime reading) that allows you to see the API ratings and compare them to other ones, as well as factors like uptime and availability.

If something is missing in our API ratings that you think should be here, don’t hesitate to drop us a line!

Now, onto what the past week tells us about the state of API ratings during this insane week of an unending year.

API Performance Headlines

To compile our API ratings, we look at more than 200 APIs. Then we pull all the metrics together to give you a general feeling for API quality in a particular category.
We do see that certain providers consistently vie for top spot in their API ratings category, including GitHub and Google. Others do less well, such as Halifax and the NHS.

Top API Ratings – CASC Score

Sept 7-14, 2020

Category	Organization	CASC score
Corporate Infrastructure	Github	9.62
COVID-19	COVID-10 Finnhub API	9.33
Cryptocurrency Exchanges	OKEx	9.20
Fintech	Stripe	9.41
Search	Google	9.44
Social Networks	Twitter	9.70
Open Banking – PSD2	Nordea Bank	9.79
Open Banking – UK	Royal Bank of Scotland (Sandbox)	9.81
Open Banking – UK Open Data	Bank of Ireland	9.78
UK Government	GOV.UK	9.44
Us Government	Department of Justice	9.62

Another volatile week!

• GitHub replacing Box in Corporate Infrastructure
• COVID-19 Finnhub API replacing ECDC in Covid-19
• OKEx replacing Coinpaprika in Cryptocurrency Exchanges
• Twitter replacing Google in Social Network
• Bank of Ireland replacing Allied Irish Bank in UK – Open Banking

Nordea takes the overall title back for the private sector with a CASC score of 9.79.

A CASC score of over 9.00 is very good and one of 9.50 or more exceptional for an API rating. Six of the eleven categories are headed by organization with a CASC score of 9.50 or more this week, the same as last week. Sustaining a CASC score of >9.25 over a period of several weeks is a good showing and congratulations to those organizations that achieved it.

All API rating categories are headed by an API with a CASC score of 9.20, which is certainly a highly acceptable performance.

Top API Ratings – Latency

Sept 7-14, 2020

Category	Organization	Median Latency
Corporate Infrastructure	Microsoft Office	227 ms
COVID-19	CDC Tools	168 ms
Cryptocurrency Exchanges	FTX	209 ms
Fintech	Stripe	298 ms
Search	Facebook	326 ms
Social Networks	Google	58 ms
Open Banking - PSD2	Railsbank	98 ms
Open Banking - UK	Royal Bank of Scotland (Sandbox)	148 ms
Open Banking - UK Open Data	HSBC	78 ms
UK Government	Police.UK	89 ms
Us Government	Department of Justice	50 ms

A stable week, with just one change: Microsoft Office replacing Dropbox in Corporate Infrastructure. Department of Justice takes back the overall title from Google in Social Networks with a median latency of just 50 ms. An important caveat: medians can be misleading! An API might have a fast median latency but produce many slow outliers. These won’t affect the median, but they mean that users can experience many calls that were unacceptably slow. So just being fast isn’t everything. You have to be reliable too if you want to have good APIs and get a high CASC score! As so often, FTX tops the Cryptocurrency Exchange category by median latency. This week they are up to seventh out of 18 overall with a CASC score of 8.72, placing them very solidly in the Green Zone.

Worst API Rating Overall

Sept 7-14, 2020

Category	Organization	CASC Score
Corporate Infrastructure	DocuSign	7.80
COVID-19	Coronavirus Data API (thevirustracker.com)	7.04
Cryptocurrency Exchanges	RadarRelay	6.97
Fintech	Nobody below 8.00!
Search	Nobody below 8.00!
Social Networks	Nobody below 8.00!
Open Banking - PSD2	Nobody below 8.00!
Open Banking - UK	Barclays (Sandbox Auth)	1.02
Open Banking - UK Open Data	HSBC	7.98
UK Government	NHS	6.73
Us Government	Federal Communications Commission	7.52

A stable week this week with DocuSign Cisco Spark in Corporate Infrastructure, NHS a new entry in UK Government, HSBC replacing Halifax in Open Banking – UK – Open Data. Barclays (Sandbox Auth) again takes the overall wooden spoon for the week with a CASC score of 1.02.

Something of interest

DocuSign is normally a well-behaved API. Yet this week it has plunged to the bottom of the Corporate Infrastructure. OK, its CASC score is still 7.80, placing it high in the Amber Zone, but it is the only API of 16 in Corporate Infrastructure outside the Green Zone. It needs a looking at. What has gone wrong? The pass rate isn’t that low, but the API does have the most outliers in the category and is second slowest. But here’s the interesting thing about pass rates: they are not all equivalent. We are currently working on a white paper on API and cloud performance. Spoiler alert: DocuSign is the only API to get a name check because in 2019 and the first eight months of 2020, they returned no 5xx HTTP status codes at all. That is very impressive performance. Out of the nearly major APIs we examined, DocuSign were the only one to come even close to that. And yet here we DocuSign with a pass rate of 99.95%. What gives? There are a couple of reasons that are related. Sometimes the response to an API call takes so long that APImetrics marks it as a warning or, if it is sufficiently slow, as a failure, even if the call responds eventually correctly. A slow failure is counted as a failure for the calculation of the CASC score, although a 2xx HTTP status code is returned. On other occasions, the call timeouts altogether and no response or HTTP code is returned at all.

DocuSign Failure: Reasons

Now, a thing is and the reason why we only looked at 5xxs in the whitepaper is that slow failures/warnings are not necessarily the fault of the API. Sometimes requests or responses get lost in the network. This is an issue with the network. However, what we saw over the last week with DocuSign was a cluster of slow failures across all of the DocuSign networks we monitor. We didn’t see similar failures with other APIs, so it is likely that the slow failures were due to something inside the DocuSign setup. Sometimes the backend was so slow that the call timed out. As far as the end user is concerned that’s a failure. Sometimes, it might not be your fault, but the internet’s. Here, it clearly looks like it is DocuSign’s fault. This is one reason why you need to active external monitoring. Perhaps the calls are never getting to API gateway and it is a network or cloud issue. But sometimes they might be getting through and it is your backend being slow. You need to know which is which so that you can provide the best service possible to your users. And this is yet another reminder that 200 does not always mean OK. What brought down DocuSign’s CASC score was not pass rate, but number of outliers. There are a lot of exceptionally long calls, some of which timeout. These kinds of problems aren’t necessarily easy to detect by passive monitoring alone, but are much more obvious when you are doing active monitoring that lets you understand how the API is behaving from the end user’s perspective. As our Insights data from our main product (sign up link) shows there was a cluster of outliers for DocuSign on 11 September. Something out of the ordinary was happening with DocuSign on that day. So many outliers will affect the user experience. APImetrics lets you see what has happened to your API and you can use this actionable intelligence to understand what went wrong and stop it happening again. See you again in a week as we look once more at API ratings. Also, don’t forget to look out for our August monthly report that we recently published.

Do more with API monitoring!

If you are new to APImetrics, sign up for a free trial so see how they could work for you with our free API set. If you have any questions, please don't hesitate to reach out.

If you're new to API monitoring, you can learn about our comprehensive features for testing, monitoring, security and performance and sign up for a free trial account.