API Call Usage & Statistics

As we’ve now been “live” for about 50 days now, with a sizeable user base that’s provided much better real-world usage than we were able to simulate in testing, we’ve implemented a number of things in order to regulate how the service is used.

This article may get slightly technical, so a brief summary before we start: if your API call usage gets too high, you’ll receive an e-mail warning to that effect.  An increase much beyond that level of usage will result in another e-mail, and the system will temporarily block you.  We’ll also be notified, and will work with you to figure out the reason for the high usage.  To date, we’ve had a handful of users who’ve hit the “warning” threshold, but in each case, we were able to bring them down to “safe” levels without too much trouble.

For Premium users, you can view your recent API call usage and click-thru statistics via the “Your Stats” page, off the main menu.

So now the more technical parts:

The true “load” on the system is actually not how many calls we’re sending and receiving to the eBay API, but how much RAM and CPU resources are consumed on our server to process a single call from a user.  This was a bit of a surprise to us, and necessitated some extensive code optimization in order to minimize disk access and server database queries.  We’re comfortable with the limits we have in place now, the server load has calmed down, and thankfully we’ve been able to assist the (very few) users who had overly high throughput.  We’re also very happy with the performance of the WP eBay Product Feeds plugin, but users who have either modified the plugin, are using a different CMS solution, or most importantly, are utilizing a feature where an end-user can search eBay through our service (which ends up getting exploited by bots), have in some cases been causing issues.  As mentioned, should this occur, we’ll work diligently with you to examine your usage case and hopefully implement some code additions on your side to help mitigate and get you below that critical threshold.

When viewing your Stats screen, some things to keep in mind.  Because we have a cache system in place (a 5-minute cache for Premium users, and a 30-minute cache for free users), identical calls received by our system within that cache window will be served the cache result stored here, without needing to query the eBay API.  Any of these “cache hits” don’t count towards your API call usage.  Additionally, most error messages you get were caught by us before querying eBay, and they won’t count either.  Lastly, we have bot-filtering algorithms in place to cut down on the number of search-engine bots, or crawlers, that are viewing your feeds and causing API calls.  All of these strategies have dramatically reduced the actual API calls you’re making, and may be quite a bit less than what you would expect.

(Incidentally, this experience in monitoring usage real-time has revealed what I believe is the primary reason eBay shut down the RSS service.  Without any ability to filter bots, regulate usage, or serve a cache, their RSS would have been hammered by literally tens of millions of queries per day.  The resources to handle that would have been quite formidable, and costly.)

The usage metric we’re looking at is your calls to the API over the last thirty-minute period.  We check this every five minutes, in order to better catch a sudden massive spike.  The API call stats you see on your Stats screen will be five minutes or less “behind live”, so if you make an API call and then run over to check the stats screen, it may take up to five minutes to reflect.  The click stats are live.  The click stats also require you to have enabled Link Cloaking in your Profile, as that’s the only way we’re able to track your clicks.

Leave a Reply

Your email address will not be published. Required fields are marked *