Mundane Clojure for Mortals: Let's build a Web API

1 Introduction

This project illustrates the basic of building a web API with Clojure. Emphasis is on using simple approaches, allowing Clojure beginners to get started easily. It's not about illustrating the best, state-of-the-art way of building such an API with Clojure.

You can find all relevant code and documentation on github: https://github.com/nblumoe/mundane-clojure-for-mortals

For a more sophisticated, advanced and structured approach have a look at duct for example.

2 Setting up a new project

You need to have Java installed. Things should also work with the .Net CLR, I don't have any experience or interest in that. Should you try the CLR, please let me know how it goes.

2.1 Project automation and dependecy Management with Leiningen

To manage the project dependencies, build the project and handle various tasks during development we are going to use Leiningen. Leiningen is a de-facto standard in the Clojure world and very mature.

To get Leiningen simply follow the installation instruction on the homepage: http://leiningen.org/

Recently, Boot was introduced as an interesting alternative, especially when working on an application (in contrast to library). If you are interested, please have a look at Boot, too!

In the following sections we are going to use Leiningen and you will learn about how to use it on the go.

2.2 Create your new project

To get started, create a new project using Leiningen: 

$ lein new api-for-mortals

This will create a new project in a subdirectory called api-for-mortals. The project name is also used for the Clojure namespaces being created.

2.3 Basic project setup

The central point for the declarative leiningen project configuration is project.clj which can be found in the project's root directory. It should look a lot like this: 

(defproject api-for-mortals "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [ring/ring-core "1.3.2"]
                 [ring/ring-jetty-adapter "1.3.2"]
                 [ring/ring-json "0.3.1"]
                 [compojure "1.3.1"]
                 [metosin/compojure-api "0.17.0"]
                 [metosin/ring-swagger-ui "2.1.1-M1"]
                 [buddy "0.4.1"]
                 [com.datomic/datomic-free "0.9.5130"]]
  :plugins [[lein-ring "0.9.1"]]
  :ring {:handler api-for-mortals.server/api-app})

This defines your project. The first two parameters to defproject are the project name and the version, :description, :url and :license should be self explaining. :dependencies is a list of lists definining the project's dependencies. As you can see, Clojure itself is a dependency of the project. This makes it very easy to use a specific version of Clojure in your project (and another version in any other project).

3 HTTP handling: One Ring to rule them all!

Ring is a Clojure library to handle HTTP requests with similarities to Python's WSGI and Ruby's Rack. Ring is another de-facto standard in the Clojure world. Many other libraries build on top of Ring.

To use it in our project, we need to add it as a dependency in project.clj. Add these two vectors to the dependencies: 

[ring/ring-core "1.3.2"]
[ring/ring-jetty-adapter "1.3.2"]

We added the core ring module and the HTTP server http-kit.

The next time we run any lein task, it will automatically pull the libraries from Clojars.

3.1 Handling our first HTTP request

Nothing more is needed to start building our web API. To actually handle requests, we need to define Ring handlers. Handlers are functions which take a map representing the request as input and return a map representing the response.

Let's write our first handler in src/api_for_mortals/core.clj: 

(ns api-for-mortals.simple)

(defn hello-handler [request]
  {:status 200
   :headers {"Content-Type" "text/plain"}
   :body (str "Hello " (:body request))})

This simple handler is a pure function and their at no dependencies for the namespace at all! It simply transforms the input map (request) to an output map (response), operating on simple, basic data structures.

3.2 Convenience: Ring plugin for Leiningen

We can make using Ring a bit more comfortable by using the Lein-Ring plugin. Add this map to project.clj: 

:plugins [[lein-ring "0.9.1"]]

The entry point for ring gets defined like this: 

:ring {:handler api-for-mortals.simple/hello-handler}

Now run lein ring server in the project root directory to start your web application. Head over to http://localhost:3000/ to see the response from your ring handler.

4 Better responses: Ring middlewares

[ring/ring-json "0.3.1"]

Creating a new handler which returns the request map as json data: 

(require '[ring.middleware.json :as json])
(def hello-json-handler
  (json/wrap-json-response
    (fn [req] {:status 200 :body (dissoc req :body)})))

request :body is an InputStream which cannot be parsed by the json middleware, thus we remove it from the response.

Let's also update the ring config to use the new handler when running lein ring server: 

:ring {:handler api-for-mortals.simple/hello-json-handler}

5 Convenient routing with Compojure

  • COMMENT Put this before ring middlewares?

    Compojure is a popular library to handle routing in Ring applications.

    To use it, simply add the dependecy to project.clj: 

    [compojure "1.3.1"]

5.1 Namespace setup

Setup a new file and namespace for the compojure handlers: 

(ns api-for-mortals.compojure
  (:require [compojure.core :refer :all]
            [compojure.route :as route]
            [clojure.edn :as edn]
            [ring.middleware.json :as json]
            [ring.middleware.params :as params]))
compojure
compojure.core has the main functions we need to define routes with compojure and compojure.route offers a convenient, premade route to handle missing routes with an 404 HTTP error.
clojure.edn
EDN (Extensible Data Notation) is a data format, a subset of the Clojure data notation. It's worth having a look at it. Here we are only using it to coerce string parameters from HTTP requests to integers via clojure.edn/read-string.
ring.middleware
ring.middleware.json we already know, we use it to build proper JSON responses. ring.middleware.params is used to extract parameters from the request's query string.

5.2 Some functions to produce "meaningful" output

(defn rnd-int-in-range [min max]
  (+ min (rand-int (+ (- max min) 1))))

(defn randomizer [min-str max-str]
  (let [min (edn/read-string min-str)
        max (edn/read-string max-str)]
    (rnd-int-in-range min max)))

5.3 Defining routes and their handlers

A small collection of routes. 

(defroutes api-routes
  (GET "/bounce-request" request {:body (dissoc request :body)})
  (GET "/users/:id" [id] {:body {:name "foo" :id id :role "admin"}})
  (GET "/randomizer" [min max] {:body {"randomNumber" (randomizer min max)}})
  (route/not-found {:body {:error "Page not found"}}))

Wrapping the routes with ring middlewares to get parameters from requests and produce JSON output in the responses. 

(def api
  (-> api-routes
    params/wrap-params
    json/wrap-json-response))

5.4 Complete namespace

(ns api-for-mortals.compojure
  (:require [compojure.core :refer :all]
            [compojure.route :as route]
            [clojure.edn :as edn]
            [ring.middleware.json :as json]
            [ring.middleware.params :as params]))

(defn rnd-int-in-range [min max]
  (+ min (rand-int (+ (- max min) 1))))

(defn randomizer [min-str max-str]
  (let [min (edn/read-string min-str)
        max (edn/read-string max-str)]
    (rnd-int-in-range min max)))

(defroutes api-routes
  (GET "/bounce-request" request {:body (dissoc request :body)})
  (GET "/users/:id" [id] {:body {:name "foo" :id id :role "admin"}})
  (GET "/randomizer" [min max] {:body {"randomNumber" (randomizer min max)}})
  (route/not-found {:body {:error "Page not found"}}))

(def api
  (-> api-routes
    params/wrap-params
    json/wrap-json-response))

5.5 Update ring entry point

Update the entry point for the ring server in project.clj to use our new, improved handler: 

:ring {:handler api-for-mortals.compojure/api-app}

6 Sweet APIs: Compojure-Swagger

https://github.com/metosin/compojure-api#sample-minimalistic-swaggered-app

Add the dependencies for the API library and the API testing UI to project.clj: 

[metosin/compojure-api "0.17.0"]
[metosin/ring-swagger-ui "2.1.1-M1"]

(ns api-for-mortals.swagger
  (:require [ring.util.http-response :refer [ok]]
            [compojure.api.sweet :refer :all]))

(defroutes* api
  (context "/api" []
    (GET* "/jolts/:id" [id]
      (ok {:id id}))
    (POST* "/jolts" []
      :body-params [body :- Long]
      (ok {:data body}))))

(require '[api-for-mortals.swagger :as swagger])

(def api-app
  (let [old-routes (-> api-routes
                     params/wrap-params
                     json/wrap-json-response)
        new-routes swagger/api]
    (routes new-routes old-routes)))

7 Authentication and Authorization: Buddy

Buddy is a small set of libraries for authentication and authorization including the needed cryptography, hashing and message signing. If needed these libs can be used independently.

Add dependency for 'buddy' meta package, including all these libraries: 

[buddy "0.4.1"]

Buddy supports multiple auth mechanisms, including HTTP basic auth, sessions and tokens, as well as stateless authentication. We are going to use the latter one.

7.1 Stateless authentication

Buddy's stateless authentication is based on JSON Web Signatures (JSW). 

(ns api-for-mortals.auth
  (:require [buddy.sign.jws :as jws]
            [buddy.auth :refer [authenticated?]]
            [buddy.auth.middleware :refer [wrap-authentication]]
            [buddy.auth.backends.token :refer [jws-backend]]
            [ring.util.http-response :refer [ok unauthorized]]
            [compojure.api.sweet :refer :all]))

Some dummy data to work with. 

(def secret "abc123")

(def user {:id 3
           :username "foo"
           :password "bar"})

(def users {(:id user) user})

When submitting valid credentials to the login endpoint, the user receives a token with his encrypted id. 

(defn login [username password]
  (if (and (= username (:username user))
        (= password (:password user)))
    {:token (jws/sign {:id (:id user)} secret)}
    nil))

The auth-backend checks the request data for an authorization header and tries to decode it. If successful, it writes the decoded data to the key :identity in the request map. 

(def auth-backend (jws-backend {:secret secret}))

A very simple middleware to check if the user is authenticated. For authenticated users is calls the given handler, othwerise it returns an HTTP error. authenticated? checks for an existing (non-nil) value of :identity in the request map (see auth-backend above). 

(defn auth-mw [handler]
  (fn [request]
    (if (authenticated? request)
      (handler request)
      (unauthorized {:error "Invalid Token"}))))

(defroutes* auth-api
  (context "/auth-api" []

    (wrap-authentication
      (GET* "/users/:id" {:as request}
        :middlewares [api-for-mortals.auth/auth-mw]
        :header-params [authorization :- String]
        :path-params [id :- Long]
        (if (= (:id (:identity request)) id)
          (ok (users id))
          (ok {:error "not auth"})))
      auth-backend)

    (POST* "/login" []
      :body-params [username :- String
                    password :- String]
      (ok (login username password)))))

7.2 Updated ring entry point

(ns api-for-mortals.server
  (:require [compojure.api.sweet :refer :all]
            [api-for-mortals.compojure :as comp]
            [api-for-mortals.swagger :as swagger]
            [api-for-mortals.auth :as auth]))

(defapi api-app
  (swagger-ui)
  (swagger-docs)
  (swaggered "No Authentication" swagger/api)
  (swaggered "Authentication" auth/auth-api)
  comp/api)

:ring {:handler api-for-mortals.server/api-app}

further reading:

7.3 Complete file

(ns api-for-mortals.auth
  (:require [buddy.sign.jws :as jws]
            [buddy.auth :refer [authenticated?]]
            [buddy.auth.middleware :refer [wrap-authentication]]
            [buddy.auth.backends.token :refer [jws-backend]]
            [ring.util.http-response :refer [ok unauthorized]]
            [compojure.api.sweet :refer :all]))

(def secret "abc123")

(def user {:id 3
           :username "foo"
           :password "bar"})

(def users {(:id user) user})

(defn login [username password]
  (if (and (= username (:username user))
        (= password (:password user)))
    {:token (jws/sign {:id (:id user)} secret)}
    nil))

(def auth-backend (jws-backend {:secret secret}))
(defn auth-mw [handler]
  (fn [request]
    (if (authenticated? request)
      (handler request)
      (unauthorized {:error "Invalid Token"}))))
(defroutes* auth-api
  (context "/auth-api" []

    (wrap-authentication
      (GET* "/users/:id" {:as request}
        :middlewares [api-for-mortals.auth/auth-mw]
        :header-params [authorization :- String]
        :path-params [id :- Long]
        (if (= (:id (:identity request)) id)
          (ok (users id))
          (ok {:error "not auth"})))
      auth-backend)

    (POST* "/login" []
      :body-params [username :- String
                    password :- String]
      (ok (login username password)))))

8 Datomic Database

8.1 Namespace outline

<<db-setup>>
<<db-init-and-seed>>
<<db-queries>>

8.2 Database setup

Using a database will be shown with Datomic, an innovative database solution, close to the Clojure way of doing things. 

[com.datomic/datomic-free "0.9.5130"]

(ns api-for-mortals.datomic
  (:require [datomic.api :as d]))

We need to create the database and connect to it. We will use an in-memory database. 

(def uri "datomic:mem://jolts")

(d/create-database uri)

(def conn (d/connect uri))

8.3 Initialize DB and seed data

We need a schema for our data. 

(def schema-tx [;; jolts
                {:db/id #db/id[:db.part/db]
                 :db/ident :jolt/sender
                 :db/valueType :db.type/string
                 :db/cardinality :db.cardinality/one
                 :db.install/_attribute :db.part/db}])

Submit the schema transaction 

@(d/transact conn schema-tx)

Define and submit seed data 

(def seed-tx [{:db/id #db/id[:db.part/user -1000001]
               :jolt/sender "nblu@futurice.com"}])

@(d/transact conn seed-tx)

8.4 Query database

Get all jolts with a sender: 

(def results (q '[find ?j :where [?j :jolt/sender]] (db conn)))

Get id of first result and make it an entity map. 

(def id (ffirst results))
(def entity (-> conn db (d/entity id)))

Get the sender of the entity. 

(:jolt/sender entity)

9 Integrating other HTTP services

10 Deployment

11 Automated Testing

12 Appendix

12.1 Important Ring Concepts

Handlers
Ring handlers are Clojure functions taking a map representing the HTTP requests as an argument and return a map representing the HTTP response. Thus handlers transform requests into repsonses. (Request Map Reference, Response Map Reference).
Middleware

Middleware are higher order functions taking a handler as an argument and returning a new, transformed handler. Middleware extends the handling of raw HTTP requests to add functionalities like request parameters, sessions, file uploading etc.

Middleware example: 

;; defining the middleware
(defn wrap-content-type [handler content-type]
  (fn[request]
    (let [response (handler request)]
      (assoc-in response [:headers "Content-Type"] content-type))))

;; wrapping the middleware around a handler
(def app (wrap-content-type handler "text/html"))
Adapters
Adapters connect Ring to web servers like Jetty, http-kit, etc.

12.2 Frameworks / Boilerplates

13 Always respond properly: Liberator

should I include Liberator? or maybe just mention it? I suspect it not to work (well) with swagger