REST API Evolution

posted by Décio Sousa on
tags: ,

In one way or another, every developer has come in touch with an API. Either integrating a major system for a big corporation, producing some fancy charts with the latest graph library, or simply by interacting with his favorite programming language. The truth is that APIs are everywhere! They actually represent a fundamental building block of the nowadays Internet, playing a fundamental role in the data exchange process that takes place between different systems and devices. From the simple weather widget on your mobile phone to a credit card payment you perform on an online shop, all of these wouldn’t be possible if those systems wouldn’t communicate with each other by calling one another’s APIs.

So with the ever growing eco-system of heterogeneous devices connected to the internet, APIs are put a new set of demanding challenges. While they must continue to perform in a reliable and secure manner, they must also be compatible with all these devices that can range from a wristwatch to the most advanced server in a data-center.

REST to the rescue

One of the most widely used technologies for building such APIs are the so called REST APIs. These APIs aim to provide a generic and standardize way of communication between heterogeneous systems. Because they heavily rely on standard communication protocols and data representation – like HTTP, XML or JSON – it’s quite easy to provide client side implementations on most programming languages, thus making them compatible with the vast majority of systems and devices.

So while these REST APIs can be compatible with most devices and technologies out there, they also must evolve. And the problem with evolution is that you sometimes have to maintain retro-compatibility with old client versions.

Let’s build up an example.

Let’s imagine an appointment system where you have an API to create and retrieve appointments. To simplify things let’s imagine our appointment object with a date and a guest name. Something like this:

A very simple REST API would look like this:

Let’s assume this plain simple API works and is being used on mobile phones, tablets and various websites that allow for booking and displaying appointments. So far so good.

At some point, you decide it would be very interesting to start gathering some statistics about your appointment system. To keep things simple you just want to know who’s the person who booked most times. For this you would need to correlate guest between themselves and decide you need to add an unique identifier to each guest. Let’s use Email. So now your object model would look like something like this:

So our object model changed slightly which means we will have to adapt the business logic on our api.

The Problem

(R)evolution!

While adapting the API to store and retrieve the new object types should be a no brainer, the problem is that all your current clients are using the old model and will continue to do so until they update. One can argue that you shouldn’t have to worry about this, and that customers should update to the newer version, but the truth is that you can’t really force an update from night to day. There will always be a time window where you have to keep both models running, which means your api must be retro-compatible.

This is where your problems start.

So back to our example, in this case it means that our API will have to handle both object models and be able to store and retrieve those models depending on the client. So let’s add back the guestName to our object to maintain compatibility with the old clients:

Remember a good thumb rule on API objects is that you should never delete fields. Adding new ones usually won’t break any client implementations (assuming they follow a good thumb rule of ignoring new fields), but removing fields is usually a road to nightmares.

Now for maintaining the API compatible, there are a few different options. Let’s look at some of the alternatives:

  • Duplication: pure and simple. Create a new method for the new clients and have the old ones using the same one.
  • Query parameters: introduce a flag to control the behavior. Something like useGuests=true.
  • API Versioning: Introduce a version in your URL path to control which method version to call.

So all these alternatives have their pros and cons. While duplication can be plain simple, it can easily turn your API classes into a bowl of duplicated code.

Query parameters can (and should) be used for behavior control (for example to add pagination to a listing) but we should avoid using them for actual API evolutions, since these are usually of a permanent kind and therefore you don’t want to make it optional for the consumer.

Versioning seems like a good idea. It allows for a clean way to evolve the API, it keeps old clients separated from new ones and provides a generic base from all kinds of changes that will occur during your API lifespan. On the other hand it also introduces a bit of complexity, specially if you will have different calls at different versions. Your clients would end up having to manage your API evolution themselves by upgrading a call, instead of the API. It’s like instead of upgrading a library to the next version, you would upgrade only a certain class of that library. This can easily turn into a version nightmare…

To overcome this we must ensure that our versions cover the whole API. This means that I should be able to call every available method on /v1 using /v2. Of course that if a newer version on a given method exists on v2 it should be run on the /v2 call. However, if a given method hasn’t changed in v2, I expect that the v1 version would seamlessly be called.

Inheritance based API Versioning

In order to achieve this we can take advantage of Java objects polymorphic capabilities. We can build up API versions in a hierarchical way so that older version methods can be overridden by newer, and calls to a newer version of an unchanged method can be seamlessly fallen back to it’s earlier version.

So back to our example we could build up a new version of the create method so that the API would look like this:

So now we have 2 working versions of our API. While all  the old clients that didn’t yet upgrade to the new version will continue to use v1 – and will see no changes – all your new consumers can now use the latest v2. Note that all these calls are valid:

CallResult
GET /api/v1/appointments/123Will run getAppointment on the v1 class
GET /api/v2/appointments/123Will run getAppointment on the v1 class
POST /api/v1/appointmentsWill run createAppointment on the v1 class
POST /api/v2/appointmentsWill run createAppointment on the v2 class

This way any consumers that want to start using the latest version will only have to update their base URLs to the corresponding version, and all of the API will seamlessly shift to the most recent implementations, while keeping the old unchanged ones.

Caveat

For the keen eye there is an immediate caveat with this approach. If your API consists of tenths of different classes, a newer version would imply duplicating them all to an upper version even for those where you don’t actually have any changes. It’s a bit of boiler plate code that can be mostly auto-generated. Still annoying though.

Although there is no quick way to overcome this, the use of interfaces could help. Instead of creating a new implementation class you could simply create a new Path annotated interface and have it implemented in your current implementing class. Although you would sill have to create one interface per API class, it is a bit cleaner. It helps a little bit, but it’s still a caveat.

Final thoughts

API versioning seems to be a current hot topic. Lot of different angles and opinions exists but there seems to be a lack of standard best practices. While this post doesn’t aim to provide such I hope that it helps to achieve a better API structure and contribute to it’s maintainability.

A final word goes to Roberto Cortez for encouraging and allowing this post on his blog. This is actually my first blog post so load the cannons and fire at will. 😉

Share with others!
  • Twitter
  • Facebook
  • LinkedIn
  • Google Plus
  • Reddit
  • Add to favorites
  • Email
  • RSS

Comments ( 13 )

  1. Replyrealpestano

    Hi Roberto,
    really nice post!

    Another option I remember is content negotiation.

    Also note that Roy Fielding itself consider versioning a bad practice but I respectfully don’t care 😉

    • ReplyRoberto Cortez

      Thank you for reading Rafael and for your links. Very interesting content.

      Just a remark: the post was written by Décio Sousa :)

    • ReplyDécio Sousa

      Hi Rafael,

      tks for your kind comment and feedback.

      Content negotiation per-se I don’t think helps. Afaik it only allows for specifying the format of the response as a mime type. By reading a bit from your link I just came across HATEOAS which I wasn’t aware of. Although this could be interesting I don’t see an easy way to maintain and specially to consume an API in this way. But I’ll sure give a deeper look. :)

      Again, thanks for reading and for the very interesting feedback! :)

      Cheers,
      Décio

  2. Replyrmpestano

    Hi Décio and Roberto,

    IMHO Hateoas is too complex for both clients and server (maybe at long term it justifies it) so I prefer simple approaches.

    For content negotiation versioning you put version info in custom media types.

    Congratulations for the post.

  3. Replyserverdude

    Had the guest been a resource from the start, there should have been no need – naturally, this defeats the purpose of evolving API.

    One thing to consider though is that the first version should be complete, and following the Open Closed Principle – it should not be modified. It can be extended though, e.g. by wrapping parts in a different call transparent for all users of the API.

    • ReplyDécio Sousa

      Hi,
      thanks for reading and commenting :)

      The guest is just an example to illustrate a need for evolving. Unfortunately when building systems and APIs we never have all the requirements at hand, so there’s always the need for change and adaption.

      Extending the object model is also a good idea. Though it will add significant complexity overtime since you’ll have a new object every time you have a new field. I guess it’s always something to consider case by case.

  4. ReplyMwanji Ezana

    Re: versioning with mime types, see Steve Klabnik’s simple example: http://blog.steveklabnik.com/posts/2011-07-03-nobody-understands-rest-or-http#i_want_my_api_to_be_versioned

    In your Jax-Rs examole, you wouldn’t duplicate each API method, but return a different response entity.

  5. ReplyDécio Sousa

    Hi Mwanji,

    Thanks for reading and for sharing a very nice link.

    I understand that versioning in the accept header is a cleaner approach but I also see some difficulties:
    – For clients, having to include this kind of parameters in the headers might not be straightforward. Depending on the setup you might have to include these versions manually on every call which can easily turn into pain.
    – On the server side, handling these versions is also not that straightforward. I can picture interceptors for retrieving object versions and re-routing, methods would still need versioning when business logic changes, and objects would also need to be versioned…

    I’ve never actually tried a real implementation of this but at first-glance it looks like a troubled road. :)

  6. ReplyMwanji

    I haven’t had the chance to do a full versioned API implementation, either, but:

    “– For clients, having to include this kind of parameters in the headers might not be straightforward. Depending on the setup you might have to include these versions manually on every call which can easily turn into pain.”

    Clients need to include an accept header in every single request, so I don’t think it’s an extra burden.

    “– On the server side, handling these versions is also not that straightforward. I can picture interceptors for retrieving object versions and re-routing, methods would still need versioning when business logic changes, and objects would also need to be versioned…”

    I don’t think it would be more work than your URL-based implementation. Ideally, the changes would be contained to the serialisation phase. In JAX-RS, that might be a MessageBodyWriter with an @Produces(“application/vnd…..”). Also, versioning becomes more modular (the appointment MIME type can be versioned independently of other MIME types), which may be useful.

    Also, re-reading the article, I think it might be easier or more idiomatic to use sub-resources. For example:

    @Path(“/api/{version}/appointments”)
    public class AppointmentsAPI {

    @PathParam(“version”)
    private String version;

    @GET
    @Path(“/{id}”)
    public AppointmentDTO getAppointment(@PathParam(“id”) String id) { … }

    @Path(“/”)
    public AppointmentsAPIVersion getVersion(AppointmentDTO appointment) {
    return “v2”.equals(version) ? new AppointmentsAPIVersion2() : new AppointmentsAPIVersion1();
    }
    }

    This way, methods that change are located in the AppointmentsAPIVersion interface and selecting the sub-resource is more flexible. Eg. versions 1-5 might all point to the same sub-resource, but v6 uses a different one.

  7. Replydbembibre

    Congrats, a very good approach for me !

Leave a reply

Your email address will not be published.

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>