API Ratings, July 2020: What exactly happened on July 28?

We have staggered further into the second half of the year and another API ratings report for the month of July 2020. We recently launched API.expert as a simple way to provide everybody with insights into the API economy and the leading APIs in different categories in particular. The basic service is free and will remain free, but other, related services will be built on top of it, including your very own API.expert dashboards if you want them! Each week, month and year we’ll look at the APIs we track in a variety of sectors for their quality CASC score (using our patented (US Patent 10,644,962, if you fancy a little light bedtime reading), which allows you at a glance to see the performance of an API and compare it to other ones as well as factors like uptime and availability. If something is missing that you think should be here in our API ratings, don’t hesitate to drop us a line! Now, onto what the past month tells us about the state of the world of APIs at the beginning of the second half of 2020.

API Ratings: The Headlines

We look at over 200 APIs for our API ratings but pull all the metrics together to give you a general feeling for the service quality for an organization’s APIs in a particular category. We do see that certain providers consistently vie for the top spot in their category including GitHub and Google with others doing less well such as Cisco’s Spark services.

Best API Ratings - CASC

July 2020

Category   

Organization   

CASC score

Corporate Infrastructure

GitHub

9.51

Covid-19

COVID-19 API (nubentos.com)

8.99

Cryptocurrency Exchanges

Coinpaprika

9.11

PSD2 Banks

Nordea Bank

9.89

Search

Google

9.47

Social Networks

Google

9.68

UK Open Banking (Open Data)

Danske Bank

9.55

UK Government

Police.UK

9.29

US Government

Federal Communications Commission

9.52

  A somewhat less volatile month in July for these API ratings, with four categories swapping hands:

• Nordea Bank swapping with ABN AMRO Bank in PSD2 Banks
• Danske Bank replacing Bank of Ireland in UK Open Banking (Open Data)
• Police.UK replacing GOV.UK in UK Government
• Department of Commerce replacing Federal Communications Commission in US Government

As in April,  May, and June, in 5 of 9 categories, the top API has managed to sustain as CASC score of over 9.50, which is an excellent showing on their parts. Nordea Bank takes the overall top spot with a first-class CASC score of 9.89.

Best API Ratings - Latency

July 2020

Category   

Organization   

Median latency

Corporate Infrastructure

Microsoft Office

228 ms

COVID-19

CDC tools (Centers for Disease Control and Prevention)

166 ms

Cryptocurrency Exchanges

Coinbase

133 ms

PSD2 Banks

Nordea Bank

170 ms

Search

Facebook

284 ms

Social Networks

Google

57 ms

UK Open Banking (Open Data)

HSBC

82 ms

UK Government

Police.UK

89 ms

US Government

Department of Justice

74 ms

  No changes this month. Google in Social Networks is again fastest overall with a median latency of 57 ms.

Worst API Ratings

July 2020

 

Category

Organization

CASC score

Corporate Infrastructure

Cisco Spark

7.93

Covid-19

Coronavirus Data API (thevirustracker.com)

6.79

Cryptocurrency Exchanges

Oasis Dex

5.41

PSD2 Banks

Open Bank Project

6.83

Search

Nobody below 8.00!

Social Networks

Nobody below 8.00!

UK Government

NHS

6.18

UK Open Banking (Open Data)

Barclays (but see below)

7.49

US Government

Department of Commerce

6.05

  A fairly stable month in July!

• Open Bank Project swapped with BBVA in PSD2 Banks
• Nobody below 8.00 in Search and Social Networks
• Barclays replacing Halifax in UK Open Banking (Open Data)

Note that Nationwide Building Society in UK Open Banking (Open Data) had a CASC score of 4.16, but this was owing to changing the URL for one of their endpoints causing the deprecated to fail. We are therefore allowing them a let. The overall API ratings loser trophy for July goes to Oasis Dex yet again albeit with another improved CASC score of 5.41.

Something of interest

An outage can ruin your day, your week, or even your month. And SendGrid had a bad in late July. It ruined the day of a lot of people at SendGrid and probably that of a lot of their customers too. The outage began around midday UTC on 28 July. That’s four o’clock in the morning on the West Coast of the US, just about the kind of time that you might cut over to a new software build. The outage lasted until 19:21 UTC. That’s a 7-hour outage. It sent SendGrid into the Amber Zone for the week and tumbling for the month, but it still managed to finish in the Green for July with a CASC score of 8.12, putting it 14th out of 15 in Corporate Infrastructure, although it has the lowest pass rate for any of the APIs in that category. APImetrics provides many tools that can be used to identify and analyze issues with APIs. Here we see the heatmap for the SendGrid v3 - Get Bounce endpoint and we can the hole were the outage on 28 July was. A giant white hole! Here we see the percentage of pass, fails, and warnings for the endpoint. Something bad happened on 28 July. Interestingly, we observed an eclectic mix of HTTP status code errors:

• 500 Internal Server Error
• 502 Bad Gateway
• 503 Service Unavailable
• 504 Gateway Timeout

We also saw some 401 Unauthorized warnings, which since the calls were authorized means that there were even more problems because the backend was returning an inappropriate status code. There was a lot of work going on to get the endpoints back up and running, which is probably what caused the various status codes to be returned as different bits of the system came up and down. APImetrics saves the results of all the calls made to an endpoint. Here we see the start of the outage. We see three 500s before two 401s and then the 500s resuming, You can dig down into each result including looking at the returned payload, which can provide useful information about the cause of the problem. This is the end of the outage where we see a 503 followed four 502 before normal service is resumed. You can refresh the results page in realtime so you can follow the progress of an outage as it happens via them as well as looking back to see exactly what went wrong. To make things worse, we saw a number of fails on 29 July with no HTTP status code returned with the client-side message “Unable to connect to endpoint” (these are the fails you can see on the chart above for 29 July). This is one reason why it is absolutely vital to actively monitor from endpoints from external locations. Your users might not even be able to access your service because of network issues. Often they will be nothing to do with you but want to know about them so you don’t get the blame, but sometimes it can be because you have dropped off the internet for some reason. SendGrid knew about the outage straightaway because people were tweeting them. But you shouldn’t rely on your customers to tell you about outages. You should be actively monitoring your APIs from a variety of cloud locations and depending solely on internal monitoring or gateways logs. That way when your users tweet you to say that the service is down, you can “Yes, we know, it will be back up as soon as possible” because you can see there is an outage on your live dashboard and you are receiving alerts from APImetrics directly into your higher-level management systems. And know exactly how the outage evolved (where the results pages come in) and be really useful in finding and fixing the bugs in your APIs and backend software including the one that caused that 401 Unauthorized. APImetrics Insights allows you to dive deep into the performance of an endpoint in a particular period. The summary provides you immediately with actionable intelligence about the endpoint was behaving. We can see at a glance that there were big problems and get an overview of what they were – in this case, 7 hours of unscheduled downtime. The outlier's charts provide an immediate visual indication that there was something badly out of kilter on 28 July. So if you had identified the problem earlier, you would certainly be able to see it from looking at the Insights report. Another organization that had a bad month was Bank of Ireland. It was on the king of the world or least the UK Open Banking (Open Data) category in June but is down to 14th out of 18 in July. This was for the same reason as SendGrid. An outage, a serious 28-hour intermittent outage in late July. That ruined their CASC score for that week, but for the month they still come in with a respectable enough CASC score of 8.12. The CASC score that we use for our API ratings is all about consistency. Which is why Nationwide Building Society has a higher CASC score than Bank of Ireland even though they have exactly the same pass rate and Bank of Ireland has fewer outliers, Bank of Ireland is slower and their standard deviation of the latency is likely lower. As we have seen in our API ratings, one bad day can knock you down the table, but you can still be in the Green Zone if you have been performing at a high level for the other 30 days of the month. And you will bounce back again the next month – provided there are no repetitions of the outages. We expect to see Bank of Ireland challenging for the title again in our August API ratings, but who knows. APIs are the toughest game in the world. You can’t rest on your laurels. You have to run to stand still. That’s why you get to make sure you are actively monitoring the performance and quality of your APIs. Then you can see immediately that there is a problem. You don’t have to wait until someone in DevOps notices or, worse, your customers are tweeting you about it. Talk to APImetrics today about we can help you understand what your APIs are doing in realtime. Because if you aren’t monitoring your APIs, you aren’t managing them. See you again in a month as we assess how matters have gone as move into August and deeper into the second half of this extraordinary year. Also, don’t forget to check out our weekly API ratings reports.