Playing Nice

Designing and implementing well-behaved web APIs

Daniel Westheide | @kaffeecoder | https://innoq.com

Play Framework User Group Berlin Brandenburg

March 25, 2015

Recipe: building a REST API with Play

Step #1: some trivial JSON serialization code

implicit val saleFormat = Json.format[Sale]

Step #2: Some simple controller code

def get(saleId: String) = Authenticated.async {
  saleRepository.findById(saleId) map {
    case Some(sale) => Ok(Json.toJson(sale))
    case None       => NotFound
  }
}
def post() = Authenticated.async { request =>
  salesService
    .startSale(request.user.id)
    .map(sale => Created(Json.toJson(sale)))
}

Step #3: Some routes

GET         /sales/:id           @controllers.Sales.get(id)
POST        /sales               @controllers.Sales.post()
PUT         /sales/:id           @controllers.Sales.put(id)

...et voilà

curl -u me:secret -XPOST localhost:9000/sales

HTTP/1.1 201 Created
Content-Length: 87
Content-Type: application/json; charset=utf-8

{
    "id": "2a345261-8243-4559-8f90-e6dfd8f63312",
    "items": [],
    "sellerId": "1",
    "status": "Open"
}

So what's wrong with this?

`HTTP && !SOAP == REST`, right?

Bon Jovi by Marco Maas (CC BY-NC-ND 2.0)

Shot through the heart
And you're to blame
You give REEEEEEST a bad name!

- Jon Bon Jovi, 1986

A well-behaved web API...

...helps other citizens of the web do their jobs

Clients (browsers, command-line clients, other servers)
Caches (Squid, Varnish...)
Proxies
Client developers

To achieve that, it...

uses the protocol semantics of HTTP
returns appropriate status codes and response headers
defines cacheability of responses
supports conditional requests
participates in content negotiation
drives the business process instead of exposing CRUD operations
is tolerant towards clients
is extensible
is self-descriptive and discoverable

Piece of cake with Play?

Expiration caching

Cache-Control and Expires response headers
Specify if and how long a representation may be cached
Use-case-specific for each (type of) resource
stale-if-error directive: trade availability for correctness if necessary
Tremendous simplification of cache invalidation

Expiration caching with Play

trait DefaultController extends Controller {

  def cacheDirectives: `Cache-Control` = `Cache-Control`(Seq.empty)

  object CacheControlling extends ActionBuilder[Request] {
    override def invokeBlock[A](
        request: Request[A], 
        block: (Request[A]) => Future[Result]) =
      block(request) map (_.withHeaders(cacheExpirationHeaders: _*))
  }

  private def cacheExpirationHeaders: Seq[(String, String)] = 
    List(cacheControlHeader, expiresValue).flatten

  private def cacheControlHeader: Option[(String, String)] =
    if (cacheDirectives.directives.isEmpty) None
    else Some(CACHE_CONTROL -> cacheDirectives.value)

  private def expiresValue: Option[(String, String)] = 
    cacheDirectives.directives.collect {
      case `max-age`(seconds) => 
        EXPIRES -> formatExpires(seconds.toInt)
    }.headOption

}

Expiration caching with Play

class Articles(articleRepository: ArticleRepository) 
  extends DefaultController {

  override def cacheDirectives = 
    `Cache-Control`(`max-age`(3600), `stale-if-error`(900))

  def get(id: String) = CacheControlling.async {
    articleRepository.findById(ArticleId(id)) map {
      case Some(article) => Ok(Json.toJson(article))
      case None => NotFound
    }
  }

}

Expiration caching response headers

curl localhost:9000/articles/1
HTTP/1.1 200 OK

Cache-Control: max-age=3600, stale-if-error=900
Expires: Tue, 24 Mar 2015 11:32:48 GMT

Server-driven content negotiation

Variant property	Request header	Play support
Language	`Accept-Language`	implicit `Lang` in controllers
Character set	`Accept-Charset`	none
Compression	`Accept-Encoding`	`GzipFilter` on a global level
Format	`Accept`	request extractors

Support for these in Play is inconsistent and incomplete
Beware of interaction with ETags!

Server-driven content format negotiation

curl localhost:9000/sales/1 -H"Accept: text/*;q=0.1,application/json"
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Vary: Accept

{
    "id": "1",
    "items": [],
    "sellerId": "1",
    "status": "Open"
}

»I would really like to get a JSON representation, but any text representation is fine, too.«

Server-driven content format negotiation

def get(saleId: String) = Authenticated.async { implicit req =>
  saleRepository.findById(SaleId(saleId)) map {
    case Some(sale) => render {
      case Accepts.Html() => Ok(views.html.sale(sale))
      case Accepts.Json() => Ok(Json.toJson(sale))
    }
    case None => NotFound
  }
}

Building blocks:

partial function from MediaRange to Result
Accepting boolean extractors

Server-driven content format negotiation

Play's content negotiation support cannot automatically create a Not Acceptable with hints about supported media types:

curl localhost:9000/sales/1 -H"Accept: application/xml"

HTTP/1.1 406 Not Acceptable
Content-Length: 0
Vary: Accept

Server-driven content format negotiation

404 Not Found instead of a 406 Not Acceptable
unnecessary database lookup
even worse for unsafe HTTP methods that cause state changes on the server

curl localhost:9000/sales/doesnotexist -H"Accept: application/xml"

HTTP/1.1 404 Not Found
Content-Length: 0

Server-driven content format negotiation

Improving this results in either lots of duplication or some extra work:

def get(saleId: String) = Authenticated.async { implicit req =>
  def respond[A : Writeable](f: Sale => A) = {
    saleRepository.findById(SaleId(saleId)) map {
      case Some(sale) => Ok(f(sale))
      case None => NotFound
    }
  }
  render.async {
    case Accepts.Html() => respond(views.html.sale(_))
    case Accepts.Json() => respond(Json.toJson(_))
  }
}

play-content-negotiation

resolvers += 
  "restful-scala" at "https://dl.bintray.com/restfulscala/maven"

libraryDependencies += 
  "org.restfulscala" %% "play-content-negotiation" % "0.2.0"

experimental micro-library
declarative content negotiation for Play
built on top of Play's content negotiation

play-content-negotiation

class Sales(saleRepository: SaleRepository) extends 
  Controller with ContentNegotiation {
  
  val negotiate = represent[Sale](
    as(Accepts.Html, views.html.sale(_)),
    as(Accepts.Json, Json.toJson(_))
  )

  def get(saleId: String) = Authenticated.async { implicit req =>
    negotiate.async { variant =>
      saleRepository.findById(SaleId(saleId)) map {
        case Some(sale) => variant.respond(sale, 200)
        case None => NotFound
      }
    }
  }

  def post() = Authenticated.async { implicit req =>
    negotiate.async { variant =>
      salesService
        .startSale(SellerId(req.user.id))
        .map(sale => variant.respond(sale, 201)
        .withHeaders(
          LOCATION -> location(sale.id),
          CONTENT_LOCATION -> location(sale.id)))
    }
  }

}

play-content-negotiation

Client developers have an easier time exploring the API:

curl localhost:9000/sales/1 -H"Accept: application/xml"

HTTP/1.1 406 Not Acceptable
Content-Length: 51
Content-Type: text/plain; charset=utf-8
Vary: Accept

Acceptable media types: text/html, application/json

Robustness principle

(aka Postel's Law)

Be liberal in what you accept, and conservative in what you send

- RFC 1122

...applied to your web API

Ignore unknown fields in JSON payload
Evolve by adding optional fields
No need for versioning your API

Unknown fields in request payload

implicit val articleReads: Reads[Article] = Json.reads[Article]

// is actually:

implicit val articleReads = (
  (__ \ 'articleId).read[ArticleId] and
  (__ \ 'name).read[String] and
  (__ \ 'price).read[Money]
)(Article)

New fields in request payload

case class Article(
  id: ArticleId,
  name: String,
  price: Money,
  description: Option[String])

  implicit val articleReads: Reads[Article] = (
    (__ \ 'articleId).read[ArticleId] and
    (__ \ 'name).read[String] and
    (__ \ 'price).read[Money] and
    (__ \ 'description).readNullable[String]
  )(Article)

New fields in request payload

case class Article(
  id: ArticleId,
  name: String,
  price: Money,
  description: String)

  implicit val articleReads: Reads[Article] = (
    (__ \ 'articleId).read[ArticleId] and
    (__ \ 'name).read[String] and
    (__ \ 'price).read[Money] and
    (__ \ 'description).readNullable[String]
                       .map(_ getOrElse "No description specified")
  )(Article)

Support for hypermedia APIs

What can I do with your API?
Where can I go from here?
What can I do now?

What can I do with your API?

{
    "http://api.wetakeyourstuff.com/rels/articles" : { 
        "href" : "/articles"
    },
    "http://api.wetakeyourstuff.com/rels/sales" : { 
        "href" : "/sales",
        "hints" : { "allow" : ["GET", "POST"]}
    },
    "http://api.wetakeyourstuff.com/rels/sale" : { 
        "href-template" : "/sales/{saleId}",
        "href-vars" : {
            "saleId" : "http://api.wetakeyourstuff.com/saleId"
        }
    }
}

A JSON Home document

Where can I go from here?

{
    "_embedded": {
        "articles": [...]
    },
    "_links": {
        "next": {
            "href": "/articles?offset=2"
        },
        "self": {
            "href": "/articles"
        }
    },
    "total": 10,
    "selectedmax": 2
}

A HAL+JSON representation of a collection resource of articles

HAL

Hypertext Application Language
XML and JSON
Links
Embedded Resources
State
Specification: https://tools.ietf.org/html/draft-kelly-json-hal-06

The Hoff to the rescue!

David Hasselhoff at re:publica 2014 by re:publica (CC BY-SA 2.0)

HALselhof

val halselhof = 
  RootProject(uri("git://github.com/restfulscala/HALselhof.git"))

lazy val root = (project in file("."))
  .enablePlugins(PlayScala)
  .dependsOn(halselhof)

A HAL micro-library based on Play-JSON
Play Framework integration
Disclaimer: developed by my co-worker Tobias Neef

HALselhof

Very simple HAL AST (HalResource, HalLink)
Seamless integration into Play-JSON
Leverages existing Play-JSON Writes instances

Creating a HAL Resource consisting only of state

implicit val saleWrites: Writes[Sale] = Json.writes[Sale]

import play.api.hal.Hal

def saleToHal(sale: Sale): HalResource = Hal.state(sale)

Creates a HalResource whose state is the JSON produced by the Writes[Sale]

Creating a HAL Resource consisting only of state

{
  "id": "548a95cb-d975-4945-b432-43560593df1a",
  "items": [],
  "sellerId": "1",
  "status": "Open"
}

Creating a HAL Resource consisting only of links

import play.api.hal.Hal

def saleToHalLinks(sale: Sale)(implicit req: RequestHeader) = {
  Hal.links(
    HalLink(
      rel = "self",
      href = routes.Sales.get(sale.id.value).absoluteURL(),
      `type` = Some(halMimeType)
    ),
    HalLink(
      "http://example.org/rels/sale-items",
      routes.Sales.items(sale.id.value).absoluteURL(),
      `type` = Some(halMimeType)
    )
  )
}

Creating a HAL Resource consisting only of links

{
    "_links": {
        "http://example.org/rels/sale-items": {
            "href": "http://localhost:9000/sales/1/items",
            "type": "application/hal+json"
        },
        "self": {
            "href": "http://localhost:9000/sales/1",
            "type": "application/hal+json"
        }
    }
}

Combining two HAL resources

import play.api.hal.Hal._

implicit val saleWrites: Writes[Sale] = Json.writes[Sale]

def saleToHal(sale: Sale)(implicit req: RequestHeader): HalResource = {
  Hal.state(sale) ++ saleToHalLinks(sale)
}

Combining two HAL resources

{
  "_links": {
      "http://example.org/rels/sale-items": {
          "href": "http://localhost:9000/sales/1/items",
          "type": "application/hal+json"
      },
      "self": {
          "href": "http://localhost:9000/sales/1",
          "type": "application/hal+json"
      }
  },
  "id": "20a39883-3d41-4dd9-9ef8-31b252bec05f",
  "items": [],
  "sellerId": "1",
  "status": "Open"
}

Responding with HAL representations

def get(saleId: String) = Authenticated.async { implicit req =>
  import play.api.mvc.hal._
  saleRepository.findById(SaleId(saleId)) map {
    case Some(sale) => Ok(saleToHal(sale))
    case None => NotFound
  }
}

Content-type response header will be application/hal+json.

HAL in content negotiation

import play.api.mvc.hal._
def negotiate()(implicit req: RequestHeader) = represent[Sale](
  as(Accepts.Html, views.html.sale(_)),
  as(AcceptHal, saleToHal)
)

Also works with Play's standard content negotiation API

What can I do now?

Resources are state machines
Inform the client about the state of the resource
Provide templated links for possible state transitions
Use a media type featuring hypermedia controls (Siren, HTML...)
Put application semantics into a documented profile

Siren

One of the most expressive hypermedia formats
JSON and potentially XML
Entities
Actions for state transitions
Links for client navigation
Sub entities as embedded links or representations
Specification: http://sirenspec.org

Example: A used-stuff aquisition portal

The business logic:

A sale can be cancelled before it is in delivery
An open sale can be committed when its price exceeds EUR 10.00
Items can only be added before the sale is committed

Presenting possible state transitions

curl "localhost:9000/sales/1

HTTP/1.1 200 OK
Content-Type: application/vnd.siren+json

{
  "class": [ "sale" ],
  "properties": { 
      "id": "1",
      "status": "Open",
      "total": "EUR 2.32"
  },
  "entities": [
    { 
      "class": [ "items", "collection" ], 
      "rel": [ "http://example.org/rels/sale-items" ], 
      "href": "http://localhost:9000/sales/1/items"
    }
  ],
  "actions": [
    {
      "name": "add-item",
      "method": "POST",
      "href": "http://localhost:9000/sales/1/items",
      "fields": [
        { "name": "articleId", "type": "text" }
      ]
    },
    {
      "name": "cancel",
      "method": "DELETE",
      "href": "http://localhost:9000/sales/1"
    }
  ]
}

Presenting possible state transitions

curl "localhost:9000/sales/1

HTTP/1.1 200 OK
Content-Type: application/vnd.siren+json

{
  "class": [ "sale" ],
  "properties": { 
      "id": "1",
      "status": "Open",
      "total": "EUR 12.32"
  },
  "entities": [
    { 
      "class": [ "items", "collection" ], 
      "rel": [ "http://example.org/rels/sale-items" ], 
      "href": "http://localhost:9000/sales/1/items"
    }
  ],
  "actions": [
    {
      "name": "add-item",
      "method": "POST",
      "href": "http://localhost:9000/sales/1/items",
      "fields": [
        { "name": "articleId", "type": "text" }
      ]
    },
    {
      "name": "cancel",
      "method": "DELETE",
      "href": "http://localhost:9000/sales/1"
    }
    {
      "name": "commit",
      "method": "PUT",
      "href": "http://localhost:9000/sales/1/commital"
    }
  ]
}

Presenting possible state transitions

curl "localhost:9000/sales/1

HTTP/1.1 200 OK
Content-Type: application/vnd.siren+json

{
  "class": [ "sale" ],
  "properties": { 
      "id": "1",
      "status": "Committed",
      "total": "EUR 12.32"
  },
  "entities": [
    { 
      "class": [ "items", "collection" ], 
      "rel": [ "http://example.org/rels/sale-items" ], 
      "href": "http://localhost:9000/sales/1/items"
    }
  ],
  "actions": [
    {
      "name": "cancel",
      "method": "DELETE",
      "href": "http://localhost:9000/sales/1"
    }
  ]
}

Presenting possible state transitions

curl "localhost:9000/sales/1

HTTP/1.1 200 OK
Content-Type: application/vnd.siren+json

{
  "class": [ "sale" ],
  "properties": { 
      "id": "1",
      "status": "InDelivery",
      "total": "EUR 12.32"
  },
  "entities": [
    { 
      "class": [ "items", "collection" ], 
      "rel": [ "http://example.org/rels/sale-items" ], 
      "href": "http://localhost:9000/sales/1/items"
    }
  ]
}

play-siren

resolvers += 
  "restful-scala" at "https://dl.bintray.com/restfulscala/maven"

libraryDependencies += "org.restfulscala" %% "play-siren" % "0.2.0"

A micro-library for integrating siren-scala with Play
Rich model of Siren
Serialization to play-json
Play Framework integration

Providing a SirenRootEntityWriter

implicit def saleWriter
  (implicit req: RequestHeader): SirenRootEntityWriter[Sale] =
  new SirenRootEntityWriter[Sale] {
   override def toSiren(sale: Sale): RootEntity = {
           RootEntity(
        classes = Some(List("sale")),
        properties = Some(
          List(
            Property("id", StringValue(sale.id.value)),
            Property("seller_id", StringValue(sale.sellerId.value)),
            Property("status", StringValue(sale.status.toString))
          )
        ),
        entities = embeddedLinks(sale),
        actions = Some(actions(sale)).filter(_.nonEmpty)
      )

  }

Typeclass for types that can be represented as Siren root entities
Properties must be serialized manually

Actions for the current resource state

override def toSiren(sale: Sale): RootEntity = {

  val addItem = Action(
    name = "add-item",
    method = Some(Action.Method.POST),
    href = routes.Sales.addItem(sale.id.value).absoluteURL(),
    fields = Some(List(Action.Field("articleId", Action.Field.Type.text)))
  ) -> Sale.canAddItems _
  ...
  def actions(sale: Sale): List[Action] = {
    List(addItem, cancelSale, commit) collect {
      case (action, condition) if condition(sale) => action
    }
  }

Responding with Siren representations

def get(saleId: String) = Authenticated.async { implicit req =>
  import com.yetu.siren._
  import play.api.mvc.hal._
  saleRepository.findById(SaleId(saleId)) map {
    case Some(sale) => Ok(Siren.asRootEntity(sale))
    case None => NotFound
  }
}

Content-type response header is application/vnd.siren+json

Extending Play with new content types

case class Writeable[-A](
  transform: (A) => Array[Byte], 
  contentType: Option[String])

class Status(status: Int) extends Result {
  def apply[C](content: C)(implicit w: Writeable[C]): Result = ???

case class ContentTypeOf[-A](mimeType: Option[String])

Extending Play with new content types

val sirenJsonMediaType = "application/vnd.siren+json"

def sirenJsonContentType(implicit codec: Codec): String = 
  ContentTypes withCharset sirenJsonMediaType

implicit def contentTypeOfSirenJson
    (implicit codec: Codec): ContentTypeOf[RootEntity] =
  ContentTypeOf[RootEntity](Some(sirenJsonContentType))

implicit def writableOfSirenJson
    (implicit codec: Codec): Writeable[RootEntity] =
  Writeable(e =>  codec.encode(Json.toJson(e).toString()))

Siren in content negotiation

import org.restfulscala.playsiren._
import com.yetu.siren._

def negotiate()(implicit req: RequestHeader) = represent[Sale](
  as(Accepts.Html, views.html.sale(_)),
  as(AcceptsSirenJson, Siren.asRootEntity(_))
)

Also works with Play's standard content negotiation API

Custom request extractors

val AcceptsSirenJson = Accepting(sirenJsonMediaType)

But the router is all wrong!

GET           /articles/:id              @controllers.Articles.get(id)
GET           /articles                  @controllers.Articles.list()

GET           /sales/:id                 @controllers.Sales.get(id)
GET           /sales                     @controllers.Sales.list()
POST          /sales                     @controllers.Sales.post()
PUT           /sales/:id                 @controllers.Sales.put(id)

POST          /sales/:id/items           @controllers.Sales.addItem(id)
DELETE        /sales/:id                 @controllers.Sales.cancel(id)
PUT           /sales/:id/commital        @controllers.Sales.commit(id)
GET           /sales/:id/items           @controllers.Sales.items(id)

The router can be replaced!

trait GlobalSettings {
  def onRouteRequest(request: RequestHeader): Option[Handler]
}

play-machine

Resource-oriented, declarative approach
inspired by Erlang's Web Machine
replaces the default router
Caution: Result of a 1-day hackathon
incomplete, dirty, untested proof-of-concept

Hooking in the machine

object Global extends GlobalSettings {

  val cells    =  "cells" / Param("cellId")       :~> CellResource
  val switches =  "switches" / Param("switchId")  :~> SwitchResource

  val myDispatcher = Dispatcher.from(cells, switches)

  override def onRouteRequest(request: RequestHeader) = {
    myDispatcher(request)
  }
}

An example resource

object SwitchResource extends Resource[Switch, SwitchId] {

  override def allowedMethods = Set(HEAD, GET, POST, OPTIONS)

  override def extractRequestParams(
      request: Request[_], pathParams: Seq[PathParam]) = {
    extractPathParam("switchId", pathParams) map SwitchId
  }

  override def isResourceExists(
      request: Request[_], switchId: SwitchId) = 
    SwitchRepository findById switchId

  override def handleGet(resource: Switch) = {
    case Accepts.Html()     => Ok(views.html.switch(resource))
    case AcceptsSirenJson() => Ok(Siren.asRootEntity(resource))
  }

  override def handlePost(
      request: Request[_], switchId: SwitchId) = 
    isResourceExists(request, switchId) map {
      case Some(switch) =>
        val updated = switch.flip()
        SwitchRepository.save(updated)
        Right(updated)
      case None => Left(404)
    }

  override implicit def executionContext = defaultContext
}

Thanks for your attention!

Questions?

Twitter: @kaffeecoder

Blog/Website: http://danielwestheide.com

Email: daniel.westheide@innoq.com

Playing Nice

Designing and implementing well-behaved web APIs

Recipe: building a REST API with Play

Step #1: some trivial JSON serialization code

Step #2: Some simple controller code

Step #3: Some routes

...et voilà

So what's wrong with this?

HTTP && !SOAP == REST, right?

A well-behaved web API...

...helps other citizens of the web do their jobs

To achieve that, it...

Piece of cake with Play?

Expiration caching

Expiration caching with Play

Expiration caching with Play

Expiration caching response headers

Server-driven content negotiation

Server-driven content format negotiation

Server-driven content format negotiation

Server-driven content format negotiation

Server-driven content format negotiation

Server-driven content format negotiation

play-content-negotiation

play-content-negotiation

play-content-negotiation

Robustness principle

...applied to your web API

Unknown fields in request payload

New fields in request payload

New fields in request payload

Support for hypermedia APIs

What can I do with your API?

What can I do with your API?

Where can I go from here?

HAL

The Hoff to the rescue!

HALselhof

HALselhof

Creating a HAL Resource consisting only of state

Creating a HAL Resource consisting only of state

Creating a HAL Resource consisting only of links

Creating a HAL Resource consisting only of links

Combining two HAL resources

Combining two HAL resources

Responding with HAL representations

HAL in content negotiation

What can I do now?

Siren

Example: A used-stuff aquisition portal

The business logic:

Presenting possible state transitions

Presenting possible state transitions

Presenting possible state transitions

Presenting possible state transitions

play-siren

Providing a SirenRootEntityWriter

Actions for the current resource state

Responding with Siren representations

Extending Play with new content types

Extending Play with new content types

Siren in content negotiation

Custom request extractors

But the router is all wrong!

The router can be replaced!

play-machine

Hooking in the machine

An example resource

Thanks for your attention!

Questions?

`HTTP && !SOAP == REST`, right?