Friday, August 24, 2012

REST – Epic Semantic Fail

Roy Fielding writes a PhD dissertation describing the architectural style of the World Wide Web. He coins the term ‘Representational State Transfer’ (REST) to describe it – after all, if you’re going to talk about something, you need to give it a name. Somehow, in a semantic shift of epic fail proportions, the term REST has also come to mean something completely different to most people, a ‘style’ of writing web services. This style has no agreed protocol.

The result? The internet is ablaze with an out of control REST flame war. It seems that many people think there’s a REST protocol when in fact there’s no such thing. Looking for a protocol in Roy Fielding’s dissertation will get you nowhere because it’s an academic paper describing an architectural style, there’s no protocol to be had. The only contribution Mr Fielding makes to the debate is to tell almost anyone who describes their API as RESTful, that it is not.

Writing RESTful web services, in practice – in the real world – means that you are on your own. You have to write your own protocol (probably implicitly, because you don’t even realise that’s what you’re doing). Now the whole thing about a protocol – TCP/IP, HTTP, SMTP, SOAP – is that everyone agrees on a set of (reasonably) unambiguous rules for communication. These can then be coded into libraries, toolkits, servers, what have you, and my Linux web server written in PHP can communicate with your .NET client running on Windows because the TCP/IP, HTTP and HTML specs are unambiguous enough to ensure that if you follow them stuff will work. If you write your own protocol and nobody else adopts it, it’s not very useful. If I want to write a client to communicate with your REST API I’m in a world of pain; there’s no serviceutil I can point at your API to generate me a proxy, instead I have to carefully read your documentation (if it exists) and then lovingly handcraft my client using low level HTTP API calls.

Now don’t get me wrong, I think a web service protocol based on a RESTful architectural style would be a wonderful thing, but let’s not kid ourselves that such a thing exists.

Show me the metadata.

The core missing pieces of any RESTful web service protocol are agreed standards on media type and link metadata. Everyone seems to agree that the Content-Type header should describe the structure and purpose of the resource, but currently it’s up for grabs how you might navigate from a media type description (like ‘application/vnb.sutekishop.customer+json’) to a machine readable schema definition for the customer – should it be XSD? JSON schema? . The same goes for hyperlinks. Although there’s an established HTML standard (the A tag), how links should be encoded in an XML or JSON representation is up to the individual API author. Similarly although there seems to be agreement that the link metadata should live at URI described by the ‘rel’ attribute, what that metadata should look like is also undefined.

Sure there are some valiant attempts to come up with a protocol – I quite liked HAL, and the HAL browser is an interesting take on how RIA UIs might be purely metadata driven at runtime – these are all still just proposals.

I think we’ll know when we have an established RESTful web service protocol, or protocols. It will be when we stop using the term ‘REST’ to describe what we are doing. When we’re writing SOAP based services we call them just that. “I’m writing a SOAP web service.” Not, “I’m writing ‘XML based RPC tunnelled over HTTP’”, which of course could be open to interpretation about exactly how we’re doing it. When we have an established protocol, ‘REST’ will be retuned to its rightful place, and the only people who will use the term will be software architecture academics like Mr Fielding.

Evolution works for me

So far the tone of this rant has been somewhat negative, It seems like I’ve been rubbishing the whole ‘REST’ project. Actually I think the current situation is healthy. Monolithic ‘enterprise’ protocols like SOAP usually end up being a bad idea. The internet is a soup of small layered protocols, each one simple in itself, but that work together to make a much larger whole. The debate around REST has reminded us that much of the infrastructure to create hugely scalable services already exists. If the community can build on this, and fill in the missing pieces, preferably with small protocols that solve each problem individually, then I think we will arrive at a much better place. My reason for writing this piece is simply a warning to the unwary, regular ‘Morts’ like myself, that when someone says, “Mike, can you write us a REST API?” to be aware that the rule book has not yet been written and that you will be making much of it up as you go along.

22 comments:

Anonymous said...

Handcraft HTTP API calls? You are fired

Anonymous said...

"If you write your own protocol and nobody else adopts it, it’s not very useful."

Why not? Seems like it fulfills its goal perfectly if it works for your service.

"If I want to write a client to communicate with your REST API I’m in a world of pain;"

Then you're using the wrong programming language, maybe. Writing to a REST API is one of the easiest things in the world.

"there’s no serviceutil I can point at your API to generate me a proxy, instead I have to carefully read your documentation (if it exists)"

The endpoint should contain the documentation itself, as "REST APIs must be hypertext-driven". There's no reason in theory you couldn't have tools that generate proxy objects, if that's what gets you off, except Microsoft is in bed with SOAP and seems to have little interest in supporting REST.

But also, creating a proxy object means coupling, and that kind of goes against the loosely-coupled nature of both REST and the web itself, so it's going to be less robust.

"and then lovingly handcraft my client using low level HTTP API calls."

I'd say HTTP calls are pretty high-level. I can put them in 1-line shell scripts (unlike SOAP-based or other protocols). I get the whole HTTP stack for free -- show me how to do caching with SOAP. If a REST call is too low-level, that sounds like a problem with the particular REST API you're using, not REST in particular.

jb said...

For the soapy guys :P

http://wanderingbarque.com/nonintersecting/2006/11/15/the-s-stands-for-simple/

Rob G said...

Actually, REST does have a protocol - that protocol is well documented in the HTTP Standard.

The entire point of Roy Fielding's original dissertation was that HTTP already has appropriate verbs. The crucial aspect is using URI's to represent resources, and HTTP verbs for actions on those resources.

It really is quite easy in practice.

Anonymous said...

Well i still like my 'protocols' and 'datastructures' the best.
Don't laugh it is just a csv send as plain/text over http.

Sure i started with the fancy easy stuff just to find out that almost anyway who would have to consume it has no idea how to do it.
So i juest give them an url with a few querystring values they can change and the result they just split on a linefeed to have the records and on a seperator to get the fields.
Since then most of the people can write some code for that in 5 minutes.

Stivz said...

"I think a web service protocol based on a RESTful architectural style would be a wonderful thing, but let’s not kid ourselves that such a thing exists"

...so, what about Open Data (OData)?

Anonymous said...

I have spent the last few weeks implementing web services clients provided by Googke, Amazon, MS, Apple, among many others - (Yes Im a 10X coder), and I had to implement my own client for a major company that used a REST API. It turned out being very simple. I hadnt implemented a REST client before and I thought hey this is pretty easy.

As far as "implementing your own protocol", um if you have ever designed a web service for a big company, your um downstream customers will always think your doing that...because you are making them eat your dogfood.

Whether it be SOAP or REST or whatever, we are all on our own to some degree so I dont understand the controversy.

I think that if anything REST APIs feel a bit more intuitive than SOAP RPC type APIs.

Give it a try, dont cling to shitty standards like SOAP, they dont matter.

RTigger said...

I think one of the arguments the article is making that a lot of the things that SOAP clients are used to, like metadata and generated proxy classes, aren't there in REST.

While you can argue that the metadata was needlessly verbose and the proxy clients created a strongly-typed association that causes more problems than its worth, for someone who just wanted to use an API it was the easiest way to do so.

Compare this to a REST API where you have to rely on documentation that may or may not be good/updated, or the fact you have to manage your own HTTP Requests - there's no "easy" way to do this in .NET at least, unless you bring in a library like RESTSharp.

tec-goblin said...

Have you checked OData? It's a REST "protocol" which permits to create proxies for many common scenarii and using many common conventions. .NET proxies are actually extremely powerful (thanks to LINQ) and to create a full read/write proxy for your database is quite trivial, but what's really good is that it's just a protocol for the meaning of the messages, you don't have to buy the whole "WCF Data Services" thing. You can do your own back-end in whichever way you want and there are proxy implementations for php and java.

Anonymous said...

RESTful services use the HTTP protocol. Have you read Fielding's paper?

Scott said...

Agreed with some above. REST is a principle, one that was used in designing HTTP. When some asks for a RESTful service, whether they know it or not, are asking for services following the HTTP spec. SOAP's problem was that it re-implemented much of what HTTP could do on top of HTTP. Replicating functionality at different layers of the stack is ugly and inefficient. I agree that in practice it's quiet simple and from job to job and team to team I have not seen much disagreement. As far as your metadata issue... again refer to the spec, if you are defining a content type you must use one defined in the HTTP spec. The one you use in your example is not. Also take a look at Restlet for java. Not the best code ever but they follow all the specs and principles to the T. Resources, Representations, Verbs, Content Negotiation etc ....

Anonymous said...

"and the only people who will use the term will be software architecture academics like Mr Fielding."

Surely you mean Dr Fielding?

Anonymous said...

check HATEOAS buddy

Ken said...

REST isn't a protocol. In Dr. Fielding's own terms, it's an "architectural style". I.e., (paraphrasing here, bear with me) it's a collection of constraints intended to produce specific properties in a system. In this case those properties are meant to help meet the original goals of the web.

So once you understand the style, you could apply it using any number of different protocols. HTTP &c. would be a logical first choice, I would think.

As regards metadata: Self-descriptive services are handy in some cases, but it isn't a requirement of distributed/service-based architectures. In order for two systems to interact you need to have an agreed upon "interface" (using that term broadly); metadata is only one of several ways of exchanging that information. I personally haven't used metadata to generate clients for years now.

Unknown said...

I very much agree with the spirit of your post. If you expose a web service (via SOAP, REST, XML-RPC, etc) then you will have a de facto interface. The question is how strongly typed it should be.

SOAP appeals to C#/Java developers who use static languages with good IDE tooling. WSDL lets you forward engineer clients. You know exactly what data to pass when calling a service.

Dynamic languages have always had problems with SOAP. The packages that do work feel non-idiomatic. I think this is the reason folks jumped on the REST bandwagon -- REST is dynamic typing for web services.

I wrote Barrister RPC to try to explore a middle ground. Would it be useful to have a RPC system with a declarative grammar that supports code generation for static languages, but runtime type verification for dynamic languages?

SOAP has scared folks away from anything with an interface definition file, but I hope that in time the pendulum will swing back a bit.

Cheers

has said...

REST isn't a protocol, it's an interaction model. (Vacuous phrases like 'architectural style' really aren't helpful, doctorate or no. People need to appreciate the What and Why if they're ever to grasp the How.) HTTP is the protocol. TCP/IP is the transport mechanism. REST describes *how* an interface should look and feel; it just happens to be a bit vague on exactly what web devs should - and shouldn't - do to achieve it. That's REST's real failing: not being exhaustively prescriptive, but instead assuming all web devs can think for themselves (protip: they really can't).

SOAP is a pseudo-protocol tunnelled over HTTP for and by people too thick or lazy to learn the HTTP protocol and network programming in general. "Hey, it's just like local OOP message-passing; what could possibly go wrong?" Try reading Peter Deutsch's "Eight Fallacies of Network Computing" you fools, and get back to us when you're done. RPC freaking sucks. I'm not saying you should _never_ use it, but if you can't enumerate all of the many ways it blows chunks then you really shouldn't be let near distributed computing of any sort.

SOAP/RPC/RMI/CORBA are message-centric (aka behaviour-centric) IPC. REST is data-centric (state-centric) IPC. RPC sends exact instructions on what to do next; REST says 'this is what your current state should be; figure out what to do to be in that position yourself' (for static web servers, it's a no-brainer; for dynamic systems, it's very big on state machines). If you are so much as *thinking* in message-passing terms, as >99.999% of OO developers do, you will never grok REST. (Other red-flag buzzwords include: API, protocol, endpoint, message/call, payload/arguments/parameters, and anything else that betrays a message-centric OO mindset.)

80% of web devs couldn't give a good golly crap about REST - the same 80% who don't understand squat about the internet, but earn the same salary whether they know what they're doing or not. Hey, at least they're honest.

Of the remainder, 90% _think_ they're doing REST (closer to 100% in PHP/Java/.Net/Rails circles), but really they're just doing ad-hoc RPC over HTTP with the message name in the URL/headers/body and arguments serialized as XML payload - essentially SOAP without all the heavyweight baggage. They really should stop calling this 'REST' for their own sakes: just call it HTTP-RPC and avoid embarrassing yourselves with buzzwords you have no business using. (Even big Web 2.0 names like Twitter do this; preening foolishness loves company.)

For the final 2% there is perhaps hope. But that's an even longer essay. :)

Anonymous said...

LOL I bet, what you really make you feel weird is the lack of RPCness hahahahaha

Unknown said...

I like where you are going with this. It reminds me of Restful Web Services, by Sam Ruby, David Hansson and Leonard Richardson. You shouldn't have to "learn" how to use an API, just as you shouldn't have to "learn" how to use a website.

When an API is written alongside the site, using your tools like HTTP to their fullest, I think that it is amazing to stand back and look at how simple and organized your system actually ended up. I think that is actually more work to ignore these principles, and will end up requiring twice the amount of code. Plus your API is now as navigable as your website, if not more so. You are also in a much easier position to scale your service with this type of architecture. When the time comes to write a mobile app, you will be thankful you did this.

Ruby does a great job of building this into their standard conventions, but it still requires a little work to get the REST of the way there. Even then you end up implementing your own interpretation of what seems to be correct. I am all for a standard being developed here, as I really think it is a beautiful way to build applications.

Dave Gauer said...

I completely agree with Conar Welsh that "You shouldn't have to 'learn' how to use an API, just as you shouldn't have to 'learn' how to use a website."

And I'm probably as wrong about REST as anyone, but I truly believe the hypermedia aspect is the most interesting and least explored/implemented aspect.

The open-source hm-json Browser allows you to completely discover a JSON-based API with absolutely no documentation and only one starting URI.

I'd love to see people start to use tools like this out in the wild.

Nick Chan Abdullah said...

"Of the remainder, 90% _think_ they're doing REST (closer to 100% in PHP/Java/.Net/Rails circles), but really they're just doing ad-hoc RPC over HTTP with the message name in the URL/headers/body and arguments serialized as XML payload - essentially SOAP without all the heavyweight baggage. They really should stop calling this 'REST' for their own sakes: just call it HTTP-RPC and avoid embarrassing yourselves with buzzwords you have no business using. (Even big Web 2.0 names like Twitter do this; preening foolishness loves company.)"


I completely agree. Most REST services are not REST-ful at all.
many programmers for go so-called REST WebAPI because they think they can't transport JSON over SOAP or XML-RPC, which actually they can, by just embedding it in an xml node value. they are all strings anyway !

Tadzik said...

I agree!

The definition of metadata is always subject to interpretation. In REST, the only thing you've really got is HTTP Headers, which forces a lot of 'conventional metadata' into the body of the request. I don't think that's a bad thing.

Personally, I think the best answer to the hypermedia question is to provide a OpenAPI / Swagger spec that identifies where the URLs are and what they could mean, and to link to it in the headers of the response. The best reason to have a convention (like HAL) around links is to make hypermedia content more obvious to readers of the client code consuming the API - who are not necessarily able to see a sample API response.

Warren said...

You want to see some ugly-ass REST+actual-metadata implementations? Check out the healthcare world, DICOM-RS and HL7 FHIR in particular. I think that REST as a name sucks. Also HATEOAS. I think HTTP-JSON-HYPERMEDIA lacks a certain je-ne-sais-quoi. So let's just call it REST even though it means nothing.

Now let's talk about identification systems. What a mess that is.