Steve Klabnik — REST & Hypermedia

Hypermedia FizzBuzz

Fri, 02 May 2014 00:00:00 GMT

I read a really great blog post last week: Solving FizzBuzz with Hypermedia. In it, Stephen Mizell builds a FizzBuzz service using Siren, and then shows how client code evolves. I wanted to explain exactly why I think this example is amazing, because I’m not sure it’s exactly obvious.

FizzBuzz? Really?

The first thing that makes this post brilliant is that it uses FizzBuzz. While FizzBuzz has historically been a simple interview question to make sure that you can actually write some basic code, in this case, it works well because it’s a very, very simple programming problem. When writing examples, there’s always tension between real-world and straw-man examples. If you make it too real-world, all sorts of incidental details creep in and distract from your main point. If you make a trivial example, it can be claimed that it works for your simple example, but not for ‘real’ examples. Now, this may be true of FizzBuzz, but what I like about it is that it’s an example that everyone is already familiar with. It has just enough actual computing that it straddles that line.

Haters gonna HATEOAS

The real insight of this post, however, is demonstrating one of the hardest concepts to grok about the hypermedia approach: the way in which hypermedia reduces duplication, by moving application entirely to the server. This is the core of that dreaded “HATEOAS” concept, and it’s where people get tripped up. Here’s the sample client in the hypermedia style:

from siren import SirenResource as Hyperclient

BASE_URL = "http://fizzbuzzaas.herokuapp.com"

def fizzbuzz(resource):
    """
    Prints the fizzbuzz value and follows "next" links
    """
    print resource.properties["value"]

    if resource.has_link("next"):
        fizzbuzz(resource.follow_link("next"))

def begin_fizzbuzz(resource):
    """
    Follows the first link, then hands off to fizzbuzz
    """
    if resource.has_link("first"):
        fizzbuzz(resource.follow_link("first"))

root_resource = Hyperclient(BASE_URL, path="/")
begin_fizzbuzz(root_resource)

Let’s talk about what this client’s goal is: to print a list of numbers. This is a classic recursive list traversal algorithm: we start at the head of the list (first), and then follow the links until we get to the end. (next) This client does not actually calculate fizzbuzz. It just so happens that the service we’re pointing it at calculates FizzBuzz. If we pointed it at, say, a list of search results, where the same link relations and Siren were used, it would still work!

Compare this to the non-hypermedia client:

import requests

BASE_URL = "http://fizzbuzzaas.herokuapp.com"

def fizzbuzz(params):
    url = BASE_URL + "/fizzbuzz"
    response = requests.get(url, params=params)
    response_json = response.json()
    return response_json["properties"]["value"]

for number in range(1, 101):
    print fizzbuzz({"number": number })

This client doesn’t fully calculate FizzBuzz on its own, but it does have some of the logic for doing so embedded inside. We explicitly loop over the range FizzBuzz needs, and we print out each value.

Why does this matter? It seems like a pretty subtle distinction. Well, this logic already exists on the server. We’re duplicating that logic here. This duplication creates coupling across the client/server boundary. For example, if we pointed this client at a “search results” resource, we would get 100 search results, even if there are less. We’re not letting the server guide our interactions, and we’re coding business logic into the client.

This comes back to bite us when we try to change the client. As Stephen’s next example shows:

import requests

BASE_URL = "http://fizzbuzzaas.herokuapp.com"
STARTS_AT = 4
ENDS_AT = 20
ADD = 2

def fizzbuzz(params):
    url = BASE_URL + "/fizzbuzz"
    response = requests.get(url, params=params)
    response_json = response.json()
    print response_json["properties"]["value"]

    params["number"] += ADD

    if params["number"] <= ENDS_AT:
        fizzbuzz(params)

fizzbuzz({ "number": STARTS_AT })

This version uses different start, end, and skip parameters. And so we had to throw out our entire old client, because it was too tightly coupled to the notion of how to calculate FizzBuzz. We’re now manually doing the skip and end calculations, even further coupling this client. Our fundamental algorithm even changed: the last client was iterative, but this client is recursive. We could have still made it iterative, but that still would have been a lot of change.

Now , the hypermedia version:

from siren import SirenResource as Hyperclient

BASE_URL = "http://fizzbuzzaas.herokuapp.com"

def fizzbuzz(resource):
    """
    Prints the fizzbuzz value and follows "next" links
    """
    print resource.properties["value"]

    if resource.has_link("next"):
        fizzbuzz(resource.follow_link("next"))

def begin_fizzbuzz(resource):
    """
    Follows the first link, then hands off to fizzbuzz
    """
    if resource.has_link("first"):
        fizzbuzz(resource.follow_link("first"))

def custom_fizzbuzz(root_resource, params):
    """
    Submits actions for custom fizzbuzz
    """
    resource = root_resource.take_action("custom-fizzbuzz", params)
    begin_fizzbuzz(resource)

root_resource = Hyperclient(BASE_URL, path="/")
params = { "startsAt": 4, "endsAt": 20, "add": 2 }
custom_fizzbuzz(root_resource, params)

Very little has changed between the last client and this one. We just added an extra method to submit our custom start point, but the rest of the code is all identical: we’re still traversing the list, with the same code. This worked because we chose the right behavior for our client, and let the server handle all of the business logic.

Traversal from the root

The last thing this post does is demonstrate how an actual hypermedia API starts from the API root and progresses from there. Lots of people hear this, and assume that there will be way more API calls than in an API in a different style. That’s because they assume that the client will be written in the same way as they’re used to: function calls over HTTP. As you can see here, we don’t start each call from the root of the API, we start our initial interaction with the service from the root of the API. Each step is just one more call after that, just like in any other API.

This is the other part of the hypermedia constraint that trips people up. Navigating a state machine of your API’s business process has a very different feel than making function calls over HTTP. It requires a different kind of mindset and approach to building clients. This is the area of hypermedia research that’s been least published about. The server-side story has been fleshed out, but the real frontier in hypermedia theory and practice is client building guidelines, and that’s why I like this example so much.

The profile link relation and you

Mon, 06 May 2013 00:00:00 GMT

I was quite pleased when RFC 6906 was finalized. It’s a really useful pattern that people are using to enhance documentation of their APIs.

Let’s start with an example. I tweet something like this:

“Oh man, the example.com API is super awesome. You should check it out!” - @steveklabnik seconds ago from web

You don’t know anything about this API or what it offers. So you fire up curl:

$ curl -i http://example.com
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 273

{
  "wtl": "MjAxMy0wNS0wNiAxMjo1Nzo1MyAtMDcwMA==\n",
  "grobb34s": [
    {
      "flog": "Top 100 foobars",
      "zilch": "http://example.com/foo/bar?baz=qux"
    },
    {
      "flog": "Mega Troll Title",
      "zilch": "http://example.com/exploit.exe/foobar"
    }
  ]
}

To be blunt, this makes absolutely no sense. You tell me so, and the next day, I tell you to check it out again. You grumble, and get back to the curl:

$ curl -i http://example.com
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 273
 Link: <http://example.com/profile>; rel="profile"

{
  "wtl": "MjAxMy0wNS0wNiAxMjo1Nzo1MyAtMDcwMA==\n",
  "grobb34s": [
    {
      "flog": "Top 100 foobars",
      "zilch": "http://example.com/foo/bar?baz=qux"
    },
    {
      "flog": "Mega Troll Title",
      "zilch": "http://example.com/exploit.exe/foobar"
    }
  ]
}

Oh, wait. “Profile”. Let’s see what that’s about:

$ curl -i http://example.com/profile
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 548

The Example.com API
===================

Example.com provides access to our blog through an API.

In the API, you'll see two major things of interest: `wtl` and `grobb34s`.

## wtl

The value provided under the `wtl` key is the time the latest blog post
was posted, in "%Y-%m-%d %H:%M:%S %z" format. This value is then Base64
encoded.

## grobb34s

The `grobb34s` key will hold an array of blog posts. These posts are
represented by a JSON object with two keys. `flog` has the title, suitable for
display, and `zilch` contains a link to the post.

Oh. You don’t care about some blog with such terrible titles, so you go about your day.

self-descriptive-ness

You may consider this a highly contrived example, but it’s not as contrived as you think. In Fielding’s thesis, he describes a ‘self-described messages’ constraint, which means that any API call should be able to be understood alone, without additional context.

In HTTP, this information is generally provided via the Content-Type header. It describes a media type (such as application/json in this example) that describes the rules for processing the response. When writing a HTTP client, you fetch the response, check the Content-Type, and then invoke the correct parser based on the type provided.

But many times are very general. Take application/json, for example. It says nothing about blog posts. A user-agent that only knows about application/json is very much like you as a human seeing gibberish keys and values; it’s not smart enough to make assumptions based on the text of keys and values it may see in JSON. However, with the added context of a profile, we have enough information to make sense of this strange API. And any user-agent that sees a profile that it recognizes can act on those new semantics, too.

JSON API

This is one of the reasons that we’re working on JSON API, a standard media type for APIs that use JSON. Generic JSON has no standard semantics that are useful to API authors, since it’s a very generic format. By declaring a new media type with common features to many APIs, we can write generic tools that handle the 80% of cases that come up during API development.

And if the generic case doesn’t suit your requirements, you can always extend it with profiles. In fact, it’d be nice if many people just added a Link header to their existing API documentation; that’d start alleviating this problem.

If you enjoyed this article, you might want to check out my in-progress book on building APIs that respect standards and HTTP, Designing Hypermedia APIs. This post also serves as its first blog post.

The next iteration of "Designing Hypermedia APIs"

Tue, 12 Feb 2013 00:00:00 GMT

I sent out an email today to everyone who’d previously purchased Designing Hypermedia APIs. Here’s the text:

Hey there,

First of all, I want to apologize for sending you an email. I try to
keep these to a minimum. But this one is important.

I haven't been uploading new content lately for a few reasons.
Holidays are complicated, I wasn't feeling inspired... but this is the
real reason: I've totally moved the site to something different. I
just updated it, so the DNS might need an update, but
http://www.designinghypermediaapis.com/ is all-new. Here's the deal:

1. Gone is logging in and reading it online. PDF/ePUB/MOBI now.
2. Because you purchased a copy before, you get the big bundle. Thank
you for your support over the last year.
3. You'll be getting another email with a link to download the files.
When I send an update, you'll get a new email with a new link.
4. The current written content is basically an appendix: I'm going to
be writing a more traditional book, from start to finish. It'll walk
you through building an application, both client and server.
5. I want this to be of the highest quality, so I'm not going to
commit to a schedule.

If you aren't a part of the mailing list, we're doing a reading club
over the next few months, "Building Hypermedia APIs with HTML5 and
Node." You can join by emailing hypermedia@librelist.com.

Thank you once again for your support of this project, I really
appreciate it. As always, any and all feedback welcome: I bet I can do
some nicer formatting of the PDF/ePUB/MOBI now that I have them
generating.

Wooo!

I invented hypermedia APIs by accident

Fri, 21 Dec 2012 00:00:00 GMT

Long long ago, I got an internship in college. Back then, I didn’t know anything about web development. My college professor said “GET and POST are the same thing, it’s just that GET requests show parameters in the URL bar.” But a friend of mine was working at this post-acquisition startup. They still operated as an autonomous unit within the parent company, so they were still pretty fun to work for. People would upload us MP3s and we’d give them a podcast. They’d get a blog, the necessary RSS, we’d help get them listed in iTunes, all that jazz.

Then the iPhone came out, and you could make apps for it.

Bizdev decided that we should sell apps to our clients, who could then sell them to their audience and help finance the show. We’d do all the dev work, we’d all make money, it’d be a good time. So they asked me to write an iPhone app generator. We picked a big feature set, and people could give us some images, pick their individual features, and set some other config options. We’d take that file, and compile in them as defaults. All seemed good.

Then the first app got rejected: it turns out that Apple has some restrictions on the bitrate of MP3s that you could download. I suspected that AT&T wasn’t very happy with you using lots of data back then. Anyway, we had to get around this issue: we wanted to serve up quality audio. What to do?

I don’t remember who came up with it, but we decided that the first thing the app would do would be to fetch the config file from the server, to see if any of the defaults were changed. It was just XML, so the iPhone could parse it easily, and then change whatever was different. So we’d have something like this:

<?xml version="1.0" encoding="UTF-8"?><config>  <link href="http://example.com/low.rss" rel="podcast_url" /></config>

I don’t remember the exact schema, but you get the idea.

Anyway, so we’d link to the low quality feed while the app was in review, and then later, we’d change the response to this:

<?xml version="1.0" encoding="UTF-8"?><config>  <link href="http://example.com/high.rss" rel="podcast_url" /></config>

Now the client would fetch the high quality feed. No client code needed to change! We’d managed to sneak around a restriction in the review process.

But why stop here? Once we saw this flexibility, we started taking full advantage. People could set up a bunch of different configuration options, and that would change the UI based on their choices. So, for example, they could choose to be contacted by email, they could put in a website or two, and the page would change. Here’s a mockup of the web page:

This would cause our app to serve up the following XML:

<?xml version="1.0" encoding="UTF-8"?><config>  <link href="http://example.com/high.rss" rel="podcast_url" />  <about>    <links>      <link href="http://example.com">Homepage</link>      <link href="http://example.com/about">About Us</link>    </links>    <phone>5558675309</phone>    <email>foo@example.com</email>  </about></config>

And that would make this appear in the “about” section of the app:

Neat, eh? We’d turn the email into a mailto: link and the phone into a tel: one, as well.

Anyway, maybe later they’d go back and unset their phone number, and the about us link. So then we’d generate this XML:

<?xml version="1.0" encoding="UTF-8"?><config>  <link href="http://example.com/high.rss" rel="podcast_url" />  <about>    <links>      <link href="http://example.com">Homepage</link>    </links>    <email>foo@example.com</email>  </about></config>

But check out what the app would do:

Whoah! Isn’t that rad? The UI would change based on the config.

Here’s what’s even rad-er: because the app would read the config on each load, changes would get propagated very quickly. If you took your email away, everyone’s apps would no longer have it, as if by magic. Next time they loaded up the app, it just wouldn’t be there any more.

Here’s what’s even rad-er: You wouldn’t need to go through the App Store approval process to push this change out to the users. It Just Worked. If you’re an app developer, and you’ve been forced to sit through long waits to push out a release, you know how painful it can be. I don’t know how fast it is these days, but back then, it could take a month. With this approach, we could, say, remove a feature, and it’d be gone immediately. No oversight. If we added a new feature, older apps would still work, because they’d just ignore the new part of the config, and people who got the new version would get the newest features.

Years later, after doing tons of research on Hypermedia APIs, I realized that that thing I did long ago was a form of it, and it provided us with a pretty massive benefit. So of course I wasn’t the first one to come up with it; my point is that I implemented a system in line with hypermedia principles because it made sense to do so, even though I had no idea that Roy Fielding was a PhD candidate. And I certainly had no way of articulating it as a ‘style,’ it was just a neat hack.

If you liked this story, you may care to learn about what else I’ve learned along the way with Hypermedia APIs. I wrote an e-‘book’ about it called Designing Hypermedia APIs. It’s only $20, and you get updates forever! I’m actively working on adding new content, and I’ll be doing a total redux in the new year.

Hypermedia API reading list

Mon, 27 Feb 2012 00:00:00 GMT

Originally, this post was titled “A RESTful Reading List,” but please note that REST is over. Hypermedia API is the new nomenclature.

I’ve been doing an intense amount of research on Hypermedia APIs over the last few months, and while I didn’t save every resource I found, I’ve made a list here of the most important.

I’ll be updating this post as I get new resources, so check back!

The book list

If you want to go from ‘nothing to everything,’ you can do it by reading just a few books, actually. I’m going to make all of these links affiliate. I purchase a lot of books myself, maybe suggesting things to you can help defray the cost of my massive backlog. All are easily searchable on Amazon if you’re not comfortable with that.

Start off with Restful Web Services by Leonard Richardson and Sam Ruby. This book is fantastic from getting you from zero knowledge to “I know how Rails does REST by default,” which is how most people do REST. But as you know, that’s flawed. However, understanding this stuff is crucial, not only for when you interact with REST services, but also for understanding how Hypermedia APIs work differently. This baseline of knowledge is really important.It also comes in cookbook form. You really only need one or the other; pick whichever format you like.

Next up, read REST in Practice: Hypermedia and Systems Architecture. This book serves as a great bridge to understanding Hypermedia APIs from the RESTful world. Chapters one through four read like Richardson & Ruby; yet they start slipping in the better hypermedia terminology. Chapter five really starts to dig into how Hypermedia APIs work, and is a great introduction. Chapter six covers scaling, chapter seven is an introduction to using ATOM for more than an RSS replacement, nine is about security, and eleven is a great comparison of how Hypermedia and WS-* APIs differ. All in all, a great intermediate book.

To really start to truly think in Hypermedia, though, you must read Building Hypermedia APIs with HTML5 and Node. Don’t let the title fool you, as Mike says in the introduction:

[HTML5, Node.js, and CouchDB] are used as tools illustrating points about hypermedia design and implementation.

This is not a Node.js book. I find Node slightly distasteful, but all the examples were easy to follow, even without more than a cursory working knowledge.

Anyway, the book: Mike says something else that’s really important in the intro:

While the subject of the REST architectural style comes up occasionally, this book does not explore the topic at all. It is true that REST identifies hypermedia as an important aspect of the style, but this is not the case for the inverse. Increasing attention to hypermedia designs can improve the quality and functionality of many styles of distributed network architecture including REST.

And, in the afterward:

However, the aim of this book was not to create a definitive work on designing hypermedia APIs. Instead, it was to identify helpful concepts, suggest useful methodologies, and provide pertinent examples that encourage architects, designers, and developers to see the value and utility of hypermedia in their own implementations.

I think these two statements, taken together, describe the book perfectly. The title is “Building Hypermedia APIs,” not “Designing.” So why recommend it on an API design list? Because understanding media types, and specifically hypermedia-enabled media types, is the key to understanding Hypermedia APIs. Hence the name.

Mike is a great guy who’s personally helped me learn much of the things that I know about REST, and I’m very thankful to him for that. I can’t recommend this book highly enough.

However, that may leave you wondering: Where’s the definitive work on how to actually build and design a Hypermedia API? Did I mention, totally unrelated of course, that I’m writing a book? ;)

Yes, it still has REST in the title. Think about that for a while, I’m sure you can see towards my plans. I’m planning on a beta release as soon as I’m recovered from some surgery this week, but I’m not sure how long that will take, exactly. So keep your eyes peeled.

REST API Design Rulebook, while I haven’t actually read it, seems quite terrible. Let me copy an Amazon review:

The first chapters give a good feel for the vocabulary, and some good techniques for implementing REST. A lot of the ‘rules’, especially those related to basic CRUD operations, are clean and simple with useful examples.Unfortunately, the later chapters get more and more focused on specifying something called ‘WRML’, which is a concept/language newly introduced in this book as far as I can tell.Personally I would recommend ignoring the sections dealing with WRML (or keep them in mind as a detailed example of one possible way of handling some of the REST issues).As to WRML itself: yuck. It appears to be an attempt to drag in some of the unnecessary complexity of SOAP with little added benefit. Not recommended.

Looking up information about WRML, I can agree, 100%. Ugh. So nasty. So this gets a big fat downvote from me.

Books I want to read

There aren’t any in this category. Should there be? You tell me!

Web resources

There are so many, this will just be a partial list for now.

Of course, Fielding’s dissertation is essential.

Roy has also written REST APIs Must be Hypertext Driven. This post is central for understanding why “Hypermedia API” is a much better name than REST, and why hypermedia in general is so essential.

I’ve written this post about HATEOAS. It’s a pretty basic, simple introduction to the topic.

I gave a talk called Everything you know about REST is wrong.

This talk by Jon Moore was instrumental in giving me a mental breakthrough about HATEAOS. He was kind enough to share some code with me that he used in the presentation as well. This is also an earlier example of the usage of “Hypermedia APIs.”

A classic: How I explained REST to my wife. A great story, simple and easy to explain.

Another classic is How to GET a cup of coffee. It does a great job of explaining how to model your business processes as state machines, and then convert them to HTTP.

In which Mike Mayo has a realization that HATEOAS is not simply academic: http://mikemayo.org/2012/how-i-learned-to-stop-worrying-and-love-rest

A recent resource that’s popped up is Designing a RESTful Web API. It’s a nice, basic overview of lots of things.

APIs: A Strategy Guide seems really interesting. This isn’t about REST or Hypermedia APIs specifically, but more making a case for why you’d want an API in the first place. Which is a related topic for all of us API enthusiasts, for sure. I haven’t read it yet, but it’s on the related list.

Protocol: How Control Exists after Decentralization is one of my favorite books ever. This book manages to be both a hard computer science book as well as referencing a broad range of philosophy, history, and other fields as well.

If sentences like

A perfect exmaple of a distributed network is the rhizome described in Deleuze and Guattari’s A Thousand Plateaus. Reacting specifically to what they see as the totalitarianism inherent in centralized and even decentralized networks, Deleuze and Guattari instead describe the rhizome, a horizontal meshwork derived from botany. The rhizome links many autonomous nodes together in a manner that is neither linear nor hierarchical. Rhizomes are heterogeneous and connective, that is to say, “Any point of a rhizome can be connected to anything other.”

immediately followed by a hardcore, low-level diagram of the four levels of networking: application layer, transport layer, internet layer, and link layer. With full (and accurate) descriptions of TCP, IP, DNS, the SYN-ACK/SYN-ACK handshake, and HTTP following gets you all hot and bothered, you need to read this book. Hell, if you don’t like literature stuff, the politics in this book are amazing. This book draws the connections between why the APIs we’re building matter, and for that matter, the systems that we programmers create. Seriously, I can’t recommend this book enough.

Hypertext 3.0: Critical Theory and New Media in an Era of Globalization is related, and absolutely interesting. Here’s the eight main sections:

Hypertext: An introduction
Hypertext and Critical Theory
Reconfiguring the Text
Reconfiguring the Author
Reconfiguring Writing
Reconfiguring Narrative
Reconfiguring Literary Education
The Politics of Hypertext: Who Controls the Text?

While not directly useful for those designing APIs, those of us who like to draw broad connections between disciplines will find that this book has lots of interesting parallels. Especially around the politics angle, as well as reconfiguring narrative.

REST is over

Thu, 23 Feb 2012 00:00:00 GMT

REST is

Yep. Sorry to have to inform you. REST is totally over. The cool kids are moving on. We’re building “Hypermedia APIs” now. Such is life.

A lesson from the anti-globalization movement

Way back in the day, COINTELPRO was at the forefront of America’s fight against “subersive” organizations and individuals. One goal of COINTELPRO was to create tension and division amongst radical groups, in order to disrupt their operations. Techniques such as Concern Trolling are really effective at this kind of thing.

In 2008, for the Republican National Convention in St. Paul, a document was passed around:

At an anti-RNC conference held over the weekend of February 9th and 10th, a broad spectrum of groups revealed what are being called the “St. Paul Principles” of unity for resisting the 2008 Republican National Convention (RNC).

This is a departure from the sectarian squabbles that have plagued past years’ anti-convention organizing. Pitting groups of differing political beliefs against each other has been a frequent tactic of state repression since the days of COINTELPRO.

By drafting the principles together, the co-signing organizations are taking historic steps to actively extinguish divisiveness from their respective groups. The principles will ensure respect for the soon-to-be-permitted march on September 1 by people planning non-permitted activities, and in turn, participants in the September 1 march will adhere to the principles and do nothing to sow division among the many activists coming to the Twin Cities to protest the RNC.

The principles are:

Our solidarity will be based on respect for a diversity of tactics and the plans of other groups.
The actions and tactics used will be organized to maintain a separation of time or space.
Any debates or criticisms will stay internal to the movement, avoiding any public or media denunciations of fellow activists and events.
We oppose any state repression of dissent, including surveillance, infiltration, disruption and violence. We agree not to assist law enforcement actions against activists and others.

Please draw your attention to the third principle. The reasons behind this rule are interesting:

‘Solidarity’ kinda goes out the window when you’re busy arguing about each other’s drama.
Every second you have in the media is precious. Why waste it talking about each other when you could be talking about your issue?
Media will jump on any kind of debate as a weakness. The only way to make these sort of self-reflective discussions productive is to keep them internal.

So, keeping that all in mind, why are we arguing about what REST means all the time? Yes, it’s annoying that the common usage of REST is not actually REST. Yes, it’s really hard when someone is wrong on the Internet. Yes, words do matter. However, it’s a question of what’s most productive with our time. Every moment we waste arguing over what REST means could have been spent discussing how to properly build APIs instead. But why bother being productive when we can be critical?

‘Hypermedia API’ is more clear

The real problem is that REST is just bad branding. This isn’t Roy’s fault, he wasn’t thinking about such things when trying to write his thesis. But really, from an outside perspective, a ‘real RESTful API’ does two things:

Uses HTTP correctly.
Serves hypermedia responses.

HTTP does most of the heavy lifting in terms of bringing us into REST compliance. So why are we talking about how to transfer representations of state?

The phrase “Hypermedia API” is much more direct: it’s an API where hypermedia is at the center. Those RESTish APIs could never be called ‘Hypermedia APIs,’ as it’s quite obvious that they don’t use hypermedia. It is not quite clear that they don’t transfer state and representations, though. ;)

We just really shouldn’t fight

Ultimately, it really comes back to the first point, though. Arguing about this is just wasting everyone’s time. It’s time to take a deep breath, step back, and just let REST go. Language changes. It happens. It might be a little bit sad, but life will move on. Let’s build fantastic APIs instead. <3 <3 <3

Oh, and I didn’t actually kick this off, credit for that goes to Mike Amundsen and O’Reilly with Building Hypermedia APIs with HTML5 and Node. Once something has an O’Reilly book about it, it’s legit. ;) Additionally, the term has been bandied about in the past, in various presentations and talks, but I feel that now’s the time to really step forward and start calling a spade a spade.

An API ontology

Mon, 13 Feb 2012 00:00:00 GMT

NOTE: The alpha of my book on APIs is out! Check it out at http://designinghypermediaapis.com.

As I’ve done research on APIs for Designing Hypermedia APIs, I’ve become increasingly interested in different styles of API. I currently see most real-world deployed APIs fit into a few different categories. All have their pros and cons, and it’s important to see how they relate to one other.

You may find this amusing if you’ve read some of the literature on the topic, but I’ve created this list in a top-down way: APIs as black boxes, rather than coming up with different aspects of an API and categorizing them based on that. I also decided to look at actually deployed APIs, rather than theoretical software architectures.

If you have an API that doesn’t fit into one of these categories, I’d love to hear about it. I’d also like to further expand these descriptions, if you have suggestions in that regard, please drop me a line, too.

HTTP GET/POST

Synopsis:

Provide simple data through a simple GET/POST request.

Examples:

Description:

Simple data is made available via an HTTP GET or POST request. The vast majority of these services seem to return images, but data is possible as well.

These API are technically a sub-type of *-RPC, but I feel that their lack of business process makes them feel different. It’s basically just one specific remote procedure, available over HTTP.

*-RPC

Synopsis:

Remote procedure call; call a function over the web.

Examples:

Description:

Similiar to how structured programming is built around functions, so is RPC. Rather than call functions from your own programs, RPC is a way to call functions over the Internet.

All calls are made through some sort of API endpoint, and usually sent over HTTP POST.

Major flavors include XML-RPC and JSON-RPC, depending on what format data is returned in.

Note that while Flickr’s API says REST, it is very clearly RPC. Yay terminology!

WS-* (or SOAP)

Synopsis:

Serialize and send objects over the wire.

Examples:

Description:

SOAP stands for “Simple Object Access Protocol,” and that describes it pretty well. The idea behind these APIs is to somehow serialize your objects and then send them over the wire to someone else.

This is usually accomplished by downloading a WSDL file, which your IDE can then use to generate a whole ton of objects. You can then treat these as local, and the library will know how to make the remote magic happen.

These are much more common in the .NET world, and have fallen out of favor in startup land. Many larger businesses still use SOAP, though, due to tight integration with the IDE.

“REST”

Synopsis:

Ruby on Rails brought respect for HTTP into the developer world. A blending of RPC, SOAP, and hypermedia API types.

Examples:

Description:

Originally, REST was synonymous with what is now called “Hypermedia APIs.” However, after large amounts of misunderstanding, REST advocates are rebranding REST to “Hypermedia APIs” and leaving REST to the RESTish folks. See ’REST is over’ for more.

REST is basically “RPC and/or SOAP that respects HTTP.” A large problem with RPC and SOAP APIs is that they tunnel everything through one endpoint, which means that they can’t take advantage of many features of HTTP, like auth and caching. RESTful APIs mitigate this disadvantage by adding lots of endpoints; one for each ‘resource.’ The SOAPish ones basically allow you to CRUD objects over HTTP by using tooling like ActiveResource, and the RPC ones let you perform more complicated actions, but always with different endpoints.

Hypermedia

Synopsis:

Hypermedia is used to drive clients through various business processes. The least understood and deployed API type, with one exception: the World Wide Web.

Examples:

Description:

Originally called REST, Hypermedia APIs take full advantage of HTTP. They use hypermedia formats to drive business processes, providing the ultimate decoupling of clients and servers.

You can tell an API is Hypermedia by providing only one API endpoint, but which accepts requests at other endpoints that are provided by discovery. You navigate through the API by letting the server’s responses guide you.

Implementing HATEOS with presenters

Fri, 06 Jan 2012 00:00:00 GMT

I’m a big fan of using the presenter pattern to help separate logic from presentation. There’s a great gem named Draper that can help facilitate this pattern in your Rails apps. When doing research for my book about REST, I realized that the presenter pattern is a great way to create responses that comply with the hypermedia constraint, a.k.a. HATEOAS. I wanted to share with you a little bit about how to do this.

Please note that ’REST is over’.

Note: We’ll be creating HTML5 responses in this example, as HTML is a hypermedia format, and is therefore conducive to HATEOAS. JSON and XML don’t cut it.

First, some setup

I fired up a brand new Rails app by doing this:

$ rails new hateoas_example
$ cd hateoas_example
$ cat >> Gemfile
gem "draper"
^D
$ bundle
$ rails g resource post title:string body:text
$ rake db:migrate
$ rails g draper:decorator Post

Okay, now we should be all set up. We’ve got a Rails app, it’s got draper in the Gemfile, we have a Post resource, and our PostDecorator.

The View

I like to do the view first, to drive our what we need elsewhere. Here it is:

<h2>Title</h2>
<p><%= @post.title %></p>

<h2>Body</h2>
<p><%= @post.body %></p>

<h2>Links</h2>
<ul>
  <% @post.links.each do |link| %>
    <li><%= link_to link.text, link.href, :rel => link.rel %></li>
  <% end %>
</ul>

We’re displaying our title and body, but we also want to spit out some links. These links should have a few attributes we need. I might even (shhhhhhh) extract this link thing out into a helper to add the rel stuff every time. It just depends. For this example, I didn’t feel like it.

The Controller

Well, we know we’re gonna need a @post variable set, so let’s get that going in our controller:

class PostsController < ApplicationController
  def show
    @post = PostDecorator.find(params[:id])
  end
end

Super simple. Yay Draper!

The Presenter

We know we need a links method that returns some links, and those links need to have rel, href, and text attributes. No problem!

class Link < Struct.new(:rel, :href, :text)
end

class PostDecorator < ApplicationDecorator
  decorates :post

  def links
    [self_link, all_posts_link]
  end

  def all_posts_link
    Link.new("index", h.posts_url, "All posts")
  end

  def self_link
    Link.new("self", h.post_url(post), "This post")
  end
end

Now, we could have just returned an array of three-element arrays, but I really like to use the Struct.new trick to give us an actual class. It makes error messages quite a bit better, and reminds us that we don’t happen to have an array, we have a Link.

We construct those links by taking advantage of the ‘index’ and ‘self’ rel attributes that are defined in the registry.

The output

That gives us this:

<!DOCTYPE html>
<html>
<head>
  <title>HateoasSample</title>
  <link href="/assets/application.css?body=1" media="screen" rel="stylesheet" type="text/css" />
<link href="/assets/posts.css?body=1" media="screen" rel="stylesheet" type="text/css" />
  <script src="/assets/jquery.js?body=1" type="text/javascript"></script>
<script src="/assets/jquery_ujs.js?body=1" type="text/javascript"></script>
<script src="/assets/posts.js?body=1" type="text/javascript"></script>
<script src="/assets/application.js?body=1" type="text/javascript"></script>
  <meta content="authenticity_token" name="csrf-param" />
<meta content="0k+SQVv6yr0d12tGWYx7KNXUWaf6f+wgUUNITsAOnHI=" name="csrf-token" />
</head>
<body>

<h2>Title</h2>
<p>A post, woo hoo!</p>

<h2>Body</h2>
<p>this is some text that's the body of this post</p>

<h2>Links</h2>
<ul>
    <li><a href="http://localhost:3000/posts/1" rel="self">This post</a></li>
    <li><a href="http://localhost:3000/posts" rel="index">All posts</a></li>
</ul>

</body>
</html>

You’ll probably want to make a layout that ignores all of the JS stuff, but for this example, I just left it as-is. It’s just that easy. Happy linking!

Write better cukes with the rel attribute

Tue, 20 Dec 2011 00:00:00 GMT

The other day, I was working on some Cucumber features for a project, and I discovered a neat technique that helps you to write better Cucumber steps.

Nobody wants to be cuking it wrong, but what does that really mean? Here’s Jonas’ prescription:

A step description should never contain regexen, CSS or XPath selectors, any kind of code or data structure. It should be easily understood just by reading the description.

Great. Let’s pop the why stack a few times, shall we?

Q1: Why do we want to have descriptions not use regexen, CSS selectors, or code? A1: To give it a simpler language.

Q2: Why do we want it to be in a simpler language? A2: So that it’s easily understandable for stakeholders.

Q3: Why do we want it to be easily understandable for stakeholders? A3: Because then we can share a common language.

Q4: Why is a common language important? A4: A shared language assists in making sure our model matches the desires of our stakeholders.

Q5: Why do we want to match the desires of our stakeholders? A5: That’s the whole reason we’re on this project in the first place!

Anyway, that’s what it’s really all about: developing that common language. Cukes should be written in that common language so that we can make sure we’re on track. So fine: common language. Awesome. Let’s do this.

Write some cukes in common language

Time to write a cuke:

Scenario: Editing the home page
  Given I'm logged in as an administrator
  When I go to the home page
  And I choose to edit the article
  And I fill in some content
  And I save it
  Then I should see that content

Basic CMS style stuff. I’m not going to argue that this is the best cuke in the world, but it’s pretty good. What I want to do is examine some of these steps in more detail. How would you implement these steps?

When /^I choose to edit the article$/ do
  pending
end

When /^I fill in some content$/ do
  pending
end

When /^I save it$/ do
  pending
end

Then /^I should see that content$/ do
  pending
end

Go ahead. Write them down somewhere. I’ll wait.

… done yet?

Implementing a step

Done? Okay! Before I show you my implementation, let’s talk about this step:

When /^I choose to edit the article$/

When writing this step, I realized something. When trying to write steps like these, there’s a danger in tying them too closely to your specific HTML. It’s why many people don’t write view tests: they’re brittle. I actually like view tests, but that’s another blog post. Point is this: we know we’re going to follow a link, and we know that we want that link to go somewhere that will let us edit the article. We don’t really care where it is in the DOM, just that somewhere, we’ve got an ‘edit article’ link. How to pull this off?

First idea: id attribute

You might be thinking “I’ll give it an id attribute!” Here’s the problem with that: ids have to be unique, per page. With article editing, that might not be a problem, but it’s certainly not a general solution. So that’s out.

Second time: class attribute

“Okay, then just use a class. Your blog sucks.” Well, let’s check out what the HTML5 spec says about classes.

Basically nothing about semantics.

Okay, so that’s paraphrased. But still, the spec basically says some stuff about the details of implementing classes, but absolutely nothing about the semantics of a class. In practice, classes are largely used for styling purposes. We don’t want to conflate our styling with our data, so overloading class for this purpose might work, but feels kinda wrong.

What about the text?

We could match on the text of the link. After all, that’s what people use to determine what links to click on. The link with the text “Edit this article” lets us know that that link will let us edit a article.

Matching on the text is brittle, though. What happens when marketing comes through and changes the text to read “Edit my article”? Our tests break. Ugh.

There’s got to be a better way. Otherwise, I wouldn’t be writing this blog post.

The best way: the rel attribute

When doing research for my book on REST, I’ve been doing a lot of digging into various standards documents. And one of the most important attributes from a REST perspective is one that nobody ever talks about or uses: the rel attribute. From the HTML5 spec:

The rel attribute on a and area elements controls what kinds of links the elements create. The attribue’s value must be a set of space-separated tokens. The allowed keywords and their meanings are defined below.

Below? That’s here:

The following table summarizes the link types that are defined by this specification. This table is non-normative; the actual definitions for the link types are given in the next few sections.alternate: Gives alternate representations of the current document. author: Gives a link to the current document’s author. bookmark: Gives the permalink for the nearest ancestor section.

Hey now! Seems like we’re on to something. There’s also RFC 5988, “Web Linking”. Section four talks about Link Relation Types:

In the simplest case, a link relation type identifies the semantics of a link. For example, a link with the relation type “copyright” indicates that the resource identified by the target IRI is a statement of the copyright terms applying to the current context IRI.Link relation types can also be used to indicate that the target resource has particular attributes, or exhibits particular behaviours; for example, a “service” link implies that the identified resource is part of a defined protocol (in this case, a service description).

Bam! Awesome! This is exactly what we want!

So how do we use rel attributes?

I’ll be going into more depth about these kinds of topics in my book, but here’s the TL;DR:

There are a set of official types. Try to use those if they’re applicable, but they’re quite general, so that’s often not the case.
You can put whatever else you want. Space delineated. *
The best way is to use a URI and then make a resource at that URI that documents the relation’s semantics.

We’ll go with option two for now, for simplicity. In a real application, make it a URI.

(*) Technically, this isn’t true. Extension relations are required to be URIs, or something that can be serialized to a URI. Again, details are outside of the scope of this post.

Making our link, with semantics.

Here’s what a link with our newly minted relation looks like:

<a href="/articles/1/edit" rel="edit-article">Edit this article</a>

Super simple. Just that one little attribute. Now we can write a step to match:

When /^I choose to edit the article$/ do
  find("//a[@rel='edit-article']").click
end

This code matches what we’d do as a person really well. “Find the link that edits an article, and click on it.” We’ve not only made the title of our step match our idea of what a person would do, but the code has followed suit. Awesome. We can move this link anywhere on the page, our test doesn’t break. We can change the text of the link, and our test doesn’t break. So cool.

What about stuff that’s not links? What about data attributes?

That’s what my book is going to talk about, sorry. These kinds of practical examples are one of the reasons I decided to write it in the first place, and I don’t want to publish all the content on my blog…

Better tests through web standards

Turns out that diving around in standards has some practical benefits after all, eh? Think about the relationship between your cukes, your tests, and your API clients: Cucumber, through Selenium, is an automated agent that interacts with your web service. API clients are automated agents that interact with your web service. Hmmmm…

If you want to know more about this, that’s what my book is for. I’ll be covering topics like this in depth, and explaining standards in simple language.

Seriously. Did you sign up for my book yet? ;)

Some people understand REST and HTTP

Sun, 07 Aug 2011 00:00:00 GMT

This is a follow-up post to my post here. You probably want to read that first.

UPDATE: Please note that ‘REST is over’. ’Hypermedia API’ is the proper term now.

A few words on standards versus pragmatism

When I wrote my first post on this topic, I tried to take a stance that would be somewhat soft, yet forceful. Engineering is the art of making the proper trade-offs, and there are times when following specifications is simply not the correct decision. With that said, my motivation for both of these posts is to eradicate some of the ignorance that some developers have about certain areas of the HTTP spec and Fielding’s REST paper. If you understand the correct way, yet choose to do something else for an informed reason, that’s absolutely, 100% okay. There’s no use throwing out the baby with the bathwater. But ignorance is never a good thing, and most developers are ignorant when it comes to the details of REST.

Secondly, while I think that REST is the best way to develop APIs, there are other valid architectural patterns, too. Yet calling non-REST APIs ‘RESTful’ continues to confuse developers as to what “RESTful” means. I’m not sure what exactly we should call “RESTish” APIs (hey, there we go, hmmm…) but I’m also not under the illusion that I personally will be able to make a huge dent in this. Hopefully you, humble reader, will remember this when dealing with APIs in the future, and I’ll have made a tiny dent, though.

So who does understand REST?

As it turns out, there are two companies that you’ve probably heard of who have APIs that are much more RESTful than many others: Twilio and GitHub. Let’s take a look at GitHub first.

GitHub: logically awesome

GitHub’s developer resources are not only beautiful, but thorough. In addition, they make use of lots more of REST.

The good

GitHub uses custom MIME types for all of their responses. They’re using the vendor extensions that I talked about in my post, too. For example:

application/vnd.github-issue.text+json

Super cool.

Their authentication works in three ways: HTTP Basic, OAuth via an Authentication Header, or via a parameter. This allows for a maximum amount of compatibility across user agents, and gives the user some amount of choice.

Their Pagination uses a header I didn’t discuss in part I: the Link header. Here’s a link to the reference. Basically, Link headers enable HATEOAS for media types which aren’t hypertext. This is important, especially regarding JSON, since JSON isn’t hypermedia. More on this at the end of the post. Anyway, so pagination on GitHub:

$ curl -I "https://api.github.com/users/steveklabnik/gists"
HTTP/1.1 200 OK
Server: nginx/1.0.4
Date: Sun, 07 Aug 2011 16:34:48 GMT
Content-Type: application/json
Connection: keep-alive
Status: 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4994
Link: <https://api.github.com/users/steveklabnik/gists?page=2>; rel="next", <https://api.github.com/users/steveklabnik/gists?page=33333>; rel="last"
Content-Length: 29841

The Link header there shows you how to get to the next page of results. You don’t need to know how to construct the URL, you just have to parse the header and follow it. This, for example, is a great way to connect a resource that’s not text-based, such as a PNG, to other resources.

The bad

There’s really only one place that GitHub doesn’t knock it out of the park with their new API, and that’s HATEOAS. GitHub’s API isn’t discoverable, because there’s no information at the root:

$ curl -I https://api.github.com/
HTTP/1.1 302 Found
Server: nginx/1.0.4
Date: Sun, 07 Aug 2011 16:44:02 GMT
Content-Type: text/html;charset=utf-8
Connection: keep-alive
Status: 302 Found
X-RateLimit-Limit: 5000
Location: http://developer.github.com
X-RateLimit-Remaining: 4993
Content-Length: 0

Well, at least, this is how they present it. If you ask for JSON:

$ curl -I https://api.github.com/ -H "Accept: application/json"
HTTP/1.1 204 No Content
Server: nginx/1.0.4
Date: Sun, 07 Aug 2011 16:45:32 GMT
Connection: keep-alive
Status: 204 No Content
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4991
Link: <users/{user}>; rel="user", <repos/{user}/{repo}>; rel="repo", <gists>; rel="gists"

You do get Links, but you have to construct things yourself. As a user, you get the same thing. It doesn’t change the links to point to your repos, it doesn’t give you links to anything else that you can do with the API.

Instead, the root should give you a link to the particular resources that you can actually view. Maybe something like this:

$ curl -I https://api.github.com/ -H "Accept: application/json" -u "username:password"
HTTP/1.1 204 No Content
Server: nginx/1.0.4
Date: Sun, 07 Aug 2011 16:45:32 GMT
Connection: keep-alive
Status: 204 No Content
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4991
Link: </gists/public>; rel="public_gists", </user/repos>; rel="repos", <gists>; rel="gists"

And a bunch more, for all of the other resources that are available. This would make the API truly discoverable, and you wouldn’t be forced to read their gorgeous documentation. :)

Twilio

I’ve always really enjoyed Twilio. Their API is incredibly simple to use. I once hooked up a little “Text me when someone orders something from my site” script, and it took me about fifteen minutes. Good stuff.

The good

Twilio has got the HATEOAS thing down. Check it out, their home page says that the base URL is “https://api.twilio.com/2010-04-01”. Without looking at any of the rest of their docs, (I glanced at a page or two, but I didn’t really read them fully yet), I did this:

$ curl https://api.twilio.com/2010-04-01
<?xml version="1.0"?>
<TwilioResponse>
  <Version>
    <Name>2010-04-01</Name>
    <Uri>/2010-04-01</Uri>
    <SubresourceUris>
      <Accounts>/2010-04-01/Accounts</Accounts>
    </SubresourceUris>
  </Version>
</TwilioResponse>

I introduced some formatting. Hmm, okay, Accounts. Let’s check this out:

$ curl https://api.twilio.com/2010-04-01/Accounts<?xml version="1.0"?>
<TwilioResponse><RestException><Status>401</Status><Message>Authenticate</Message><Code>20003</Code><MoreInfo>http://www.twilio.com/docs/errors/20003</MoreInfo></RestException></TwilioResponse>

Okay, so I have to be authenticated. If I was, I’d get something like this:

<TwilioResponse>
  <Account>
    <Sid>ACba8bc05eacf94afdae398e642c9cc32d</Sid>
    <FriendlyName>Do you like my friendly name?</FriendlyName>
    <Type>Full</Type>
    <Status>active</Status>
    <DateCreated>Wed, 04 Aug 2010 21:37:41 +0000</DateCreated>
    <DateUpdated>Fri, 06 Aug 2010 01:15:02 +0000</DateUpdated>
    <AuthToken>redacted</AuthToken>
    <Uri>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d</Uri>
    <SubresourceUris>
      <AvailablePhoneNumbers>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/AvailablePhoneNumbers</AvailablePhoneNumbers>
      <Calls>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/Calls</Calls>
      <Conferences>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/Conferences</Conferences>
      <IncomingPhoneNumbers>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/IncomingPhoneNumbers</IncomingPhoneNumbers>
      <Notifications>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/Notifications</Notifications>
      <OutgoingCallerIds>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/OutgoingCallerIds</OutgoingCallerIds>
      <Recordings>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/Recordings</Recordings>
      <Sandbox>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/Sandbox</Sandbox>
      <SMSMessages>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/SMS/Messages</SMSMessages>
      <Transcriptions>/2010-04-01/Accounts/ACba8bc05eacf94afdae398e642c9cc32d/Transcriptions</Transcriptions>
    </SubresourceUris>
  </Account>
</TwilioResponse>

Awesome. I can see my all of the other resources that I can interact with. Other than knowing how to authenticate, I can follow the links from the endpoint, and discover their entire API. Rock. This is the way things are supposed to be.

The bad

This:

$ curl https://api.twilio.com/2010-04-01/Accounts -H "Accept: application/json"
<?xml version="1.0"?>
<TwilioResponse><RestException><Status>401</Status><Message>Authenticate</Message><Code>20003</Code><MoreInfo>http://www.twilio.com/docs/errors/20003</MoreInfo></RestException></TwilioResponse>

Versus this:

$ curl https://api.twilio.com/2010-04-01/Accounts.json
{"status":401,"message":"Authenticate","code":20003,"more_info":"http:\/\/www.twilio.com\/docs\/errors\/20003"}

:/ Returning JSON when your resource ends with ‘.json’ isn’t bad, but not respecting the Accept header, even when you return the right MIME type, is just unfortunate.

… and a little Announcement

It seems that this is a topic that people are really interested in. Part I of this article was pretty well-received, and I got lots of great email and feedback from people. It was also made pretty clear by a few people that they want more content from me on this topic.

So I decided to write a book about it. You can check out the site for “Get Some REST”, and put in your email address. Then you’ll get updated when I start pre-release sales.

So what’s in “Get Some REST”? It’s going to be a full description of how to build RESTful web applications, from the ground up. Designing your resources, laying out an API, all the details. I’m going to try to keep most of the content language-agnostic, but provide code samples in Rails 3.1, as well.

I plan on writing a bunch of content, and then releasing the book at half-price in beta. Early adopters will be able to get their two cents in, and I’ll cover things they still have questions on. It’ll be available under a liberal license, in PDF, ePub, all that good stuff.

I’ve also set up a Twitter account at @hypermediaapis. I’ll be tweeting updates about the book, and also other good content related to RESTful design.

Nobody understands REST or HTTP

Sun, 03 Jul 2011 00:00:00 GMT

HI HN, PLEASE READ THIS!!!

Since I’ve posted this, I’ve refined a few of my positions on things. Everyone learns and grows, and while I still stand by most of what I said, I specifically don’t agree that versioning the media type is how to properly version APIs. Hypermedia APIs should not actually use explicit versioning, but I’d rather see a version in the URI with HATEOAS than no HATEOAS and versioned media types. I’ve been meaning to update this post and write more, but alas, my work on Get some REST has taken priority. I don’t have a HN account, so feel free to email me with any thoughts or questions!

Furthermore, everything in engineering is ALWAYS a trade-off. I primarily wish that more people understood the tools that HTTP provides them with, and made an informed choice, rather than cargo-culting what they’ve seen others do.

Update: Part II of this post is here. Check it out, and there’s an announcement at the end!

Update: Please note that REST is over. Hypermedia API is the new nomenclature.

The more that I’ve learned about web development, the more that I’ve come to appreciate the thoroughness and thoughtfulness of the authors of the HTTP RFC and Roy Fielding’s dissertation. It seems like the answers to most problems come down to “There’s a section of the spec for that.” Now, obviously, they’re not infallible, and I’m not saying that there’s zero room for improvement. But it really disappoints me when people don’t understand the way that a given issue is supposed to be solved, and so they make up a partial solution that solves their given case but doesn’t jive well with the way that everything else works. There are valid criticisms of the specs, but they have to come from an informed place about what the spec says in the first place.

Let’s talk about a few cases where either REST or HTTP (which is clearly RESTful in its design) solves a common web development problem.

I need to design my API

This one is a bit more general, but the others build off of it, so bear with me.

The core idea of REST is right there in the name: “Representational State Transfer” It’s about transferring representations of the state… of resources. Okay, so one part isn’t in the name. But still, let’s break this down.

Resources

From Fielding’s dissertation:

The key abstraction of information in REST is a resource. Any information that can be named can be a resource: a document or image, a temporal service (e.g. “today’s weather in Los Angeles”), a collection of other resources, a non-virtual object (e.g. a person), and so on. In other words, any concept that might be the target of an author’s hypertext reference must fit within the definition of a resource. A resource is a conceptual mapping to a set of entities, not the entity that corresponds to the mapping at any particular point in time.

When we interact with a RESTful system, we’re interacting with a set of resources. Clients request resources from the server in a variety of ways. But the key thing here is that resources are nouns. So a RESTful API consists of a set of URIs that map entities in your system to endpoints, and then you use HTTP itself for the verbs. If your URLs have action words in them, you’re doing it wrong. Let’s look at an example of this, from the early days of Rails. When Rails first started messing around with REST, the URLs looked like this:

/posts/show/1

If you use Rails today, you’ll note that the corresponding URL is this:

/posts/1

Why? Well, it’s because the ‘show’ is unnecessary; you’re performing a GET request, and that demonstrates that you want to show that resource. It doesn’t need to be in the URL.

A digression about actions

Sometimes, you need to perform some sort of action, though. Verbs are useful. So how’s this fit in? Let’s consider the example of transferring money from one Account to another. You might decided to build a URI like this:

POST /accounts/1/transfer/500.00/to/2

to transfer $500 from Account 1 to Account 2. But this is wrong! What you really need to do is consider the nouns. You’re not transferring money, you’re creating a Transaction resource:

POST /transactions HTTP/1.1
Host: <snip, and all other headers>

from=1&to=2&amount=500.00

Got it? So then, it returns the URI for your new Transaction:

HTTP/1.1 201 OK
Date: Sun, 3 Jul 2011 23:59:59 GMT
Content-Type: application/json
Content-Length: 12345
Location: http://foo.com/transactions/1

{"transaction":{"id":1,"uri":"/transactions/1","type":"transfer"}}

Whoah, HATEOS! Also, it may or may not be a good idea to return this JSON as the body; the important thing is that we have the Location header which tells us where our new resource is. If we give a client the ID, they might try to construct their own URL, and the URI is a little redundant, since we have one in the Location. Regardless, I’m leaving that JSON there, because that’s the way I typed it first. I’d love to hear your thoughts on this if you feel strongly one way or the other.

EDIT: I’ve since decided that yes, including the URI is a bad idea. The Location header makes much more sense. More on this in Part ii, yet to come.

Anyway, so now we can GET our Transaction:

GET /transactions/1 HTTP/1.1
Accept: application/json

and the response:

HTTP/1.1 blah blah blah

{"id":1,"type":"transfer","status":"in-progress"}

So we know it’s working. We can continue to poll the URI and see when our transaction is finished, or if it failed, or whatever. Easy! But it’s about manipulating those nouns.

Representations

You’ll notice a pair of headers in the above HTTP requests and responses: Accept and Content-Type. These describe the different ‘representation’ of any given resource. From Fielding:

REST components perform actions on a resource by using a representation to capture the current or intended state of that resource and transferring that representation between components. A representation is a sequence of bytes, plus representation metadata to describe those bytes. Other commonly used but less precise names for a representation include: document, file, and HTTP message entity, instance, or variant.A representation consists of data, metadata describing the data, and, on occasion, metadata to describe the metadata (usually for the purpose of verifying message integrity).

So /accounts/1 represents a resource. But it doesn’t include the form that the resource takes. That’s what these two headers are for.

This is also why adding .html to the end of your URLs is kinda silly. If I request /accounts/1.html with an Accept header of application/json, then I’ll get JSON. The Content-Type header is the server telling us what kind of representation it’s sending back as well. The important thing, though, is that a given resource can have many different representations. Ideally, there should be one unambiguous source of information in a system, and you can get different representations using Accept.

State and Transfer

This is more about the way HTTP is designed, so I’ll just keep this short: Requests are designed to be stateless, and the server holds all of the state for its resources. This is important for caching and a few other things, but it’s sort of out of the scope of this post.

Okay. With all of that out of the way, let’s talk about some more specific problems that REST/HTTP solve.

I want my API to be versioned

The first thing that people do when they want a versioned API is to shove a /v1/ in the URL. THIS IS BAD!!!!!1. Accept solves this problem. What you’re really asking for is “I’d like the version two representation of this resource.” So use accept!

Here’s an example:

GET /accounts/1 HTTP/1.1
Accept: application/vnd.steveklabnik-v2+json

You’ll notice a few things: we have a + in our MIME type, and before it is a bunch of junk that wasn’t there before. It breaks down into three things: vnd, my name, and v2. You can guess what v2 means, but what about vnd. It’s a Vendor MIME Type. After all, we don’t really want just any old JSON, we want my specific form of JSON. This lets us still have our one URL to represent our resource, yet version everything appropriately.

I got a comment from Avdi Grimm about this, too:

Here’s an article you might find interesting: http://www.informit.com/articles/article.aspx?p=1566460The author points out that MIMETypes can have parameters, which means you can actually have a mimetype that looks like this:vnd.example-com.foo+json; version=1.0Sadly, Rails does not (yet) understand this format.

I’d like my content to be displayed in multiple languages

This is related, but a little different. What about pages in different languages? Again, we have a question of representation, not one of content. /en/whatever is not appropriate here. Turns out, there’s a header for that: Accept-Language. Respect the headers, and everything works out.

Oh, and I should say this, too: this doesn’t solve the problem of “I’d like to read this article in Spanish, even though I usually browse in English.” Giving your users the option to view your content in different ways is a good thing. Personally, I’d consider this to fall out in two ways:

It’s temporary. Stick this option in the session, and if they have the option set, it trumps the header. You’re still respecting their usual preferences, but allowing them to override it.
It’s more permanent. Make it some aspect of their account, and it trumps a specific header. Same deal.

I’d like my content to have a mobile view

Sounds like I’m beating a dead horse, but again: it’s a representation question. In this case, you’d like to vary the response by the User-Agent: give one that’s mobile-friendly. There’s a whole list of mobile best practices that the w3c recommends, but the short of it is this: the User-Agent should let you know that you’re dealing with a mobile device. For example, here’s the first iPhone UA:

Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3

Then, once detecting you have a mobile User-Agent, you’d give back a mobile version of the site. Hosting it on a subdomain is a minor sin, but really, like I said above, this is really a question of representation, and so having two URLs that point to the same resource is kinda awkward.

Whatever you do, for the love of users, please don’t detect these headers, then redirect your users to m.whatever.com, at the root. One of my local news websites does this, and it means that every time I try to follow a link from Twitter in my mobile browser, I don’t see their article, I see their homepage. It’s infuriating.

I’d like to hide some of my content

Every once in a while, you see a story like this: Local paper boasts ultimate passive-agressive paywall policy. Now, I find paywalls distasteful, but this is not the way to do it. There are technological means to limit content on the web: making users be logged-in to read things, for example.

When this was discussed on Hacker News, here’s what I had to say:

nkurz:

I presume if I had an unattended roadside vegetable stand with a cash-box, that I’d be able to prosecute someone who took vegetables without paying, certainly if they also made off with the cash-box. Why is this different on the web? And if a written prohibition has no legal standing, why do so many companies pay lawyers to write click-through “terms of service” agreements?

me:

Why is this different on the web?Let’s go through what happens when I visit a web site. I type a URL in my bar, and hit enter. My web browser makes a request via http to a server, and the server inspects the request, determines if I should see the content or not, and returns either a 200 if I am allowed, and a 403 if I’m not. So, by viewing their pages, I’m literally asking permission, and being allowed.It sounds to me like a misconfiguration of their server; it’s not doing what they want it to.

I’d like to do some crazy ajax, yet have permalinks

This is an example of where the spec is obviously deficient, and so something had to be done.

As the web grew, AJAXy ‘web applications’ started to become more and more the norm. And so applications wanted to provide deep-linking capabilities to users, but there’s a problem: they couldn’t manipulate the URL with Javascript without causing a redirect. They could manipulate the anchor, though. You know, that part after the #. So, Google came up with a convention: Ajax Fragments. This fixed the problem in the short term, but then the spec got fixed in the long term: pushState. This lets you still provide a nice deep URL to your users, but not have that awkward #!.

In this case, there was a legitimate technical issue with the spec, and so it’s valid to invent something. But then the standard improved, and so people should stop using #! as HTML5 gains browser support.

In conclusion

Seriously, most of the problems that you’re solving are social, not technical. The web is decades old at this point, most people have considered these kinds of problems in the past. That doesn’t mean that they always have the right answer, but they usually do have an answer, and it’d behoove you to know what it is before you invent something on your own.

Steve Klabnik — REST & Hypermedia

Hypermedia FizzBuzz

FizzBuzz? Really?

Haters gonna HATEOAS

Traversal from the root

The profile link relation and you

self-descriptive-ness

JSON API

The next iteration of "Designing Hypermedia APIs"

I invented hypermedia APIs by accident

Hypermedia API reading list

The book list

Books I don’t recommend

Books I want to read

Web resources

Related resources

REST is over

REST is

A lesson from the anti-globalization movement

‘Hypermedia API’ is more clear

We just really shouldn’t fight

An API ontology

HTTP GET/POST

Synopsis:

Examples:

Description:

*-RPC

Synopsis:

Examples:

Description:

WS-* (or SOAP)

Synopsis:

Examples:

Description:

“REST”

Synopsis:

Examples:

Description:

Hypermedia

Synopsis:

Examples:

Description:

Implementing HATEOS with presenters

First, some setup

The View

The Controller

The Presenter

The output

Write better cukes with the rel attribute

Write some cukes in common language

Implementing a step

First idea: id attribute

Second time: class attribute

What about the text?

The best way: the rel attribute

So how do we use rel attributes?

Making our link, with semantics.

What about stuff that’s not links? What about data attributes?

Better tests through web standards

Some people understand REST and HTTP

A few words on standards versus pragmatism

So who does understand REST?

GitHub: logically awesome

The good

The bad

Twilio

The good

The bad

… and a little Announcement

Nobody understands REST or HTTP

HI HN, PLEASE READ THIS!!!

I need to design my API

Resources

A digression about actions

Representations

State and Transfer

I want my API to be versioned

I’d like my content to be displayed in multiple languages

I’d like my content to have a mobile view

I’d like to hide some of my content