IaaS-like RESTfull API based on microservices
Abstracts
Recently, I’ve been looking at the principles of a middleware layer and especially on how a RESTFULL API could glue different IT services together.
I am reading more and more about the “API economy”
I’ve also seen this excellent video made by Mat Ryer about how to code an API in GO and why go would be the perfect language to code such a portal.
The problem I’m facing is that in the organization I’m working for, the developments are heterogeneous and therefore you can find ruby teams as well as python teams and myself as a go team (That will change in the future anyway) The key point is that I would like my middleware to serve as an entry point to the services provided by the department.
We (as an “ops” team) would then be able to present the interface via, for example, a swagger like interface, take care of the API and do whatever RPC to any submodule.
An example: a IAAS like interface
Let’s consider a node compute lifecycle.
What I’d like to be able to do is:
- to create a node
- to update a node (maybe)
- to delete a node
- to get the status of the node
The backend
The backend is whatever service, able to create a node, such as openstack, vmware vcac, juju, … Thoses services usually provide RESTfull API.
I’ve seen in my experience, that usually, the API are given with a library in a so called “modern language”. This aim to simplify the development of the clients. Sometimes this library may also be developed by an internal team that will take care of the maintenance.
The library
In my example, we will consider that the library is a simple gem file developed in ruby. Therefore, our service will be a simple server that will get RPC calls, call the good method in the gemfile and that will, in fine transfer it to the backend.
The RestFull API.
I will use the example described here as a basis for this post. Of course there are many other examples and excellent go packages that may be used, but according to Mat Ryer, I will stick to the idiomatic approach.
The glue: MSGPACK-RPC
There are several methods for RPC-ing between different languages. Ages ago, there was xml-rpc; then there has been json-rpc; I will use msgpack-rpc which is a binary, json base codec. The communication between the Go client and the ruby server will be done over TCP via HTTP for example.
Later on, outside of the scope of this post, I may use ZMQ (as I have already blogged about 0MQ communication between those languages).
The implementation of the Client (the go part)
I will describe here the node creation via a POST method, and consider that the other methods could be implemented in a similar way.
The signature of the node creation
Here is the expected signature for creating a compute element:
1{
2 "kind":"linux",
3 "size":"S",
4 "disksize":20,
5 "leasedays":1,
6 "environment_type":"dev",
7 "description":"my_description",
8}
The corresponding GO structure is:
1type NodeRequest struct {
2 Kind string `json:"kind"` // Node kind (eg linux)
3 Size string `json:"size"` // size
4 Disksize int `json:"disksize"`
5 Leasedays int `json:"leasedays"`
6 EnvironmentType string `json:"environment_type"`
7 Description string `json:"description"`
8}
The route
The Middleware is using the gorilla mux package. According the description, I will add an entry in the routes array (into the routes.go file):
1Route{
2 "NodeCreate",
3 "POST",
4 "/v1/nodes",
5 NodeCreate,
6},
Note : I am using a prefix /v1 for my API, for exploitation purpose.
I will then create the corresponding handler in the file with this signature
1func NodeCreate(w http.ResponseWriter, r *http.Request){
2 var nodeRequest NodeRequest
3 body, err := ioutil.ReadAll(io.LimitReader(r.Body, 1048576))
4 if err != nil {
5 panic(err)
6 }
7 if err := r.Body.Close(); err != nil {
8 panic(err)
9 }
10 if err := json.Unmarshal(body, &nodeRequest); err != nil {
11 w.Header().Set("Content-Type", "application/json; charset=UTF-8")
12 w.WriteHeader(http.StatusBadRequest) // unprocessable entity
13 if err := json.NewEncoder(w).Encode(err); err != nil {
14 panic(err)
15 }
16 }
17}
That’s in this function that will be implemented RPC (client part). To keep it simple at the beginning, I will instantiate a TCP connection on every call. Don’t throw things at me, that will be changed later following the advice of Mat Ryer.
The implementation of the handler
The effective remote procedure call
To use msgpack I need to import the go implementation github.com/msgpack-rpc/msgpack-rpc-go/rpc.
This library will take care of the encoding/decoding of the messages.
Let’s dial the RPC server and call the NodeCreate method with, as argument, the information we had from the JSON input
1 conn, err := net.Dial("tcp", "127.0.0.1:18800")
2 if err != nil {
3 fmt.Println("fail to connect to server.")
4 return
5 }
6 client := rpc.NewSession(conn, true)
7 retval, err := client.Send("NodeCreate", nodeRequest.Kind, nodeRequest.Size, nodeRequest.Disksize, nodeRequest.Leasedays, nodeRequest.EnvironmentType, nodeRequest.Description)
8 if err != nil {
9 fmt.Println(err)
10 return
11 }
12 fmt.Println(retval)
The RPC server (the ruby part)
This part is written in ruby, and will take care of the effective node creation.
At first, we should install the GEM file with the command gem install msgpack-rpc.
1require 'msgpack/rpc'
2class MyHandler
3 def NodeCreate(kind, size, disksize, leasedays, environmenttype, description)
4 print "Creating the node with parameters: ",kind, size, disksize, leasedays, environmenttype, description
5 return "ok"
6 end
7end
8svr = MessagePack::RPC::Server.new
9svr.listen('0.0.0.0', 18800, MyHandler.new)
10svr.run
let’s test it
Launch the RPC server:
ruby server.rb
Then launch the API rest server
go run *go
Then perform a POST request
1curl -X POST -H 'Content-Type:application/json' -H 'Accept:application/json' -d '{"kind":"linux","size":"S","disksize":20,"leasedays":1,"environment_type":"dev","description":"my_description"}' -k http://localhost:8080/v1/nodes
It should write something like this:
2015/11/10 13:56:51 POST /v1/nodes NodeCreate 2.520673ms
ok
And something like this in the output of the ruby code:
Creating the node with parameters: linux S 20 1 dev my_description
That’s all folks! What’s left:
- To implement the other methods to be “CRUD” compliant
- To implement an authentication and accreditation mechanism (JWT, Oauth, ?)
- To change the implementation of the RPC client to use a pool instead of a single connection
- To implement the swagger interface and documentation of the API
- Whatever fancy stuff you may want from a production ready interface.
You can find all the codes in the github repository here in the branch iaas-like-restfull-api-based-on-microservices