First-Class APIs
APIs are commonly an afterthought, like a hot tub awkwardly attached to a house — a shoehorned approach that produces a suboptimal app with scarce support that lacks documentation. In effect, APIs are the ugly stepchild of the Web.
This is a sad reality that we are faced with, because many companies make their living consuming third-party APIs and mixing in their own data to create amazing and interesting mashups.
In the initial phases of development, there is rarely enough money to develop the app and its API. By the time there’s both demand and money, it can be hard to fit an API on top of the architecture in such a way that the whole thing won’t fall over.
APIs should be first class citizens of the Web. Inconceivable? Possimpible? Not at all!
What can be done to solve this problem?
Is there a silver bullet? I’m afraid not.
My company has done a lot of API work where we have encountered and solved a myriad of different API problems for customers and our own apps. We even released an API framework along the way. That’s not to say that we haven’t made our share of mistakes on that journey — in fact, we have made every single mistake in the book, but that’s how you grow and learn. I firmly believe that you cannot solve a problem unless you’ve failed at least once or twice before.
In the past, we have often been involved in bolting an API on top of legacy apps, and on top of startup code we were directly involved with from the beginning. Most of the time, neither situation is ideal for attempting to use the current code base (or to bring in behemoths like Zend Framework or Symfony) to write an API service layer. We put together our own lightweight API framework which can be dropped into a code base without much fuss, and it can reuse the data models. As a side bonus, it will automagically generate API documentation.
Now we are getting somewhere! We had solved some of the pain points of building an API, even if it’s bolted on afterward. That was not enough for us, though.
The solution we came up with is fairly simple, but not glamorous at all, I’m afraid.
A traditional web site has data models that interact with the database and any other data source. Add an API into the mix, and it will either be interacting with the data sources directly or potentially using the same data models as the web site.
Our approach shakes things up a little bit. The API is built first, and it interacts with the database and any other data source (e.g., Oracle, MySQL, or Solr). The web site then consumes the API just as a client would. The web site is still built with a MVC framework, but it’s completely oblivious to where the API is pulling the information from, so the API becomes a middle layer, a data layer.
The benefit is that the API becomes core to the system, is built from the start, and will be maintained properly. This way you can make an iPad app, and the web site use exactly the same API.
I also find APIs to be easier to scale, but that’s a topic for another article.
The downside is that this development model does cost you more money in the short term, and it means a tad longer time to market, but if you compare it to adding the API at a later stage and maintaining to separate systems, then you will clearly see big financial and operational gains.
Story time
An example may explain this better. This is the tale of the elusive Twitter API.
Back in 2007, Twitter took off at SXSW with a roughly 300% increase in tweets per day. Everyone was happy, but many people felt like the web interface was not all that great. People asked for an API, so they could create their own apps and mashups. Lo and behold, an API was bolted on top of Twitter. It became evident that the initial API was not enough for people, and thus began the organic growth of the monster that is the Twitter API. It was a fairly under-maintained and under-staffed API in my opinion, given it was the lifeline of Twitter. It is what kept millions of people using the service.
This is a classic example of a bolt-on API that was not accounted for properly in the infrastructure, and it quite possibly caused a majority of the fail whales. The web site and the API (to my knowledge) were separate.
A shift happened in 2010, and #NewTwitter was released in all its glory. A new web interface with many bells, whistles, and, quite possibly, a lawnmower. However, what was most exciting about the release was that the web site started consuming; it became a client of the API. A JavaScript client uses the API just as Tweetie or Echofon would. What that means is Twitter focused on maintaining the API and making sure everything ran smoothly with it.
I applaud the Twitter people for taking this step. As I mentioned before, the API is their lifeline. As such, keeping it in top shape is important. With the main web site relying on the API, it is imperative that it never fails and that bugs are taken care of with the utmost priority. This is a similar approach to what we do in my company, but taken even a step further! A gutsy step at that. Now the API is truly a first-class citizen, an important core component which runs the product.
Imagine what would happen if Facebook did a similar thing. Would we stop seeing people agonize for hours over a problem with one of the Facebook APIs or services? I think it would be interesting to see how much Facebook would enjoy eating their own dog food and how much things would improve if they took the steps Twitter did. One can dream. :-)
Final thoughts
Increasingly, I’m hearing developers say that when evaluating a service for their company or personal use, they give the API a good look, sometimes going as far as saying that a product is not worthwhile if there isn’t any API at all.
This only reaffirms my belief that companies need to be proactive and offer APIs from the start, APIs that are an integral part of the infrastructure and architecture, along with abundant documentation and support.
Sounds like nirvana, an unreachable dream, but I believe that, with a bit of foresight and planning on our products, we can be a few steps closer to that dream. Very zen, no? :-)
© Helgi Þormar Þorbjörnsson. First published in PHP Advent 2010 under the CC BY-SA 3.0 US license.