This post is a little different from the last ones. As usual, the introduction tries to be open, but it quickly goes deeper into a go implementation. Some explanations may be tricky from time to times and therefore not very clear. As usual, do not hesitate to send me any comment via this blog or via twitter @owulveryck.

TL;DR: This is a step-by-step example that turns a golang cli utility into a webservice powered by gRPC and protobuf. The code can be found here.

About the cli utilities

I come from the sysadmin world… Precisely the Unix world (I have been a BSD user for years). Therefore, I have learned to use and love “the cli utilities”. Cli utilities are all those tools that make Unix sexy and “user-friendly”.

From a user perspective, cli tools remains a must nowadays because:

• there are usually developed in the pure Unix philosophy: simple enough to use for what they were made for;
• they can be easily wrapped into scripts. Therefore, it is easy to automate cli actions.

The point with cli application is that they are mainly developed for an end-user that we call “an operator”. As Unix is a multi-user operating system, several operators can use the same tool, but they have to be logged onto the same host.

In case of a remote execution, it’s possible to execute the cli via ssh, but dealing with automation, network interruption and resuming starts to be tricky. For remote and concurrent execution web-services are more suitable.

Let’s see if turning a cli tool into a webservice without re-coding the whole logic is easy in go?

Hashicorp’s cli

For the purpose of this post, and because I am using Hashicorp tools at work, I will take @mitchellh’s framework for developing command line utilities. This package is used in all of the Hashicorp tools and is called……………. “cli”!

This library provides a Command type that represents any action that the cli will execute. Command is a go interface composed of three methods:

• Help() that returns a string describing how to use the command;
• Run(args []string) that takes an array of string as arguments (all cli parameters of the command) and returns an integer (the exit code);
• Synopsis() that returns a string describing what the command is about.

Note: I assume that you know what an interface is (especially in go). If you don’t, just google, or even better, buy the book The Go Programming Language and read the chapter 7 :).

The main object that holds the business logic of the cli package is an implementation of Cli. One of the elements of the Cli structure is Commands which is a map that takes the name of the action as key. The name passed is a string and is the one that will be used on the command line. The value of the map is a function that returns a Command. This function is named CommandFactory. According to the documentation, the factory is needed because we may need to setup some state on the struct that implements the command itself. Good idea!

Example

First, let’s create a very simple tool using the “cli” package. The tool will have two “commands”:

• hello: will display hello args…. on stdout
• goodbye: will display goodbye args… on stderr

 1func main() {
 2      c := cli.NewCLI("server", "1.0.0")
 3      c.Args = os.Args[1:]
 4      c.Commands = map[string]cli.CommandFactory{
 5            "hello": func() (cli.Command, error) {
 6                      return &HelloCommand{}, nil
 7            },
 8            "goodbye": func() (cli.Command, error) {
 9                      return &GoodbyeCommand{}, nil
10            },
11      }
12      exitStatus, err := c.Run()
13      ... 
14}

Now, let’s create the HelloCommand structure that will fulfill the cli.Command interface:

 1type HelloCommand struct{}
 2
 3func (t *HelloCommand) Help() string {
 4      return "hello [arg0] [arg1] ... says hello to everyone"
 5}
 6
 7func (t *HelloCommand) Run(args []string) int {
 8      fmt.Println("hello", args)
 9      return 0
10}
11
12func (t *HelloCommand) Synopsis() string {
13      return "A sample command that says hello on stdout"
14}

The GoodbyeCommand is similar, and I omit it for brevity.

After a simple go build, here is the behavior of our new cli tool:

 1~ ./server help
 2Usage: server [--version] [--help] <command> [<args>]
 3
 4Available commands are:
 5    goodbye    synopsis...
 6    hello      A sample command that says hello on stdout
 7
 8~ ./server hello -help
 9hello [arg0] [arg1] ... says hello to everyone
10
11~ ./server/server hello a b c
12hello [a b c]

So far, so good! Now, let’s see if we can turn this into a webservice.

Micro-services

There is, according to me, two options to consider turning our application into a webservice:

• a RESTish communication and interface;
• an RPC based communication.

SOAP is not an option anymore because it does not provide any advantage over the REST and RPC methods.

Rest?

I’ve always been a big fan of the REST “protocol”. It is easy to understand and to write. On top of that, it is verbose and allows a good description of “business objects”. But, its verbosity, that is a strength, quickly become a weakness when applied to machine-to-machine communication. The “contract” between the client and the server have to be documented manually (via something like swagger for example). And, as you only transfer objects and states, the server must handle the request, understand it, and apply it to any business logic before returning a result. Don’t get me wrong, REST remains a very good thing. But it is very good when you think about it from the beginning of your conception (and with a user experience in mind).

Indeed, it may not be a good choice for easily turning a cli into a webservice.

RPC!

RPC, on the other hand, may be a good fit because there would be a very little modification of the code. Actually, the principle would be to:

1. trigger a network listener
2. receive a procedure call with arguments,
3. execute the function
4. send back the result

The function that holds the business logic does not need any change at all.

The drawbacks of RPCs are:

• the development language need a library that supports RPC,
• the client and the server must use the same communication protocol.

Those drawbacks have been addressed by Google. They gave to the community a polyglot RPC implementation called gRPC.

Let me quote this from the chapter “The Production Environment at Google, from the Viewpoint of an SRE” of the SRE book:

All of Google’s services communicate using a Remote Procedure Call (RPC) infrastructure named Stubby; an open source version, gRPC, is available. Often, an RPC call is made even when a call to a subroutine in the local program needs to be performed. This makes it easier to refactor the call into a different server if more modularity is needed, or when a server’s codebase grows. GSLB can load balance RPCs in the same way it load balances externally visible services.

Sounds cool! Let’s dig into gRPC!

gRPC

We will now implement a gRPC server that will trigger the cli.Commands.

It will receive “orders”, and depending on the expected call, it will:

• Implements a HelloCommand and trigger its Run() function;
• Implements a GoodbyeCommand and trigger its Run() function

We will also implement a gRPC client.

For the server and the client to communicate, they have to share the same protocol and understand each other with a contract. Protocol Buffers (a.k.a., protobuf) are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data Even if it’s not mandatory, gRPC is usually used with the Protocol Buffer.

So, first, let’s implement the contract with/in protobuf!

The protobuf contract

The protocol is described in a simple text file and a specific DSL. Then there is a compiler that serializess the description and turns it into a contract that can be understood by the targeted language.

Here is a simple definition that matches our need:

 1syntax = "proto3";
 2
 3package myservice;
 4
 5service MyService {
 6    rpc Hello (Arg) returns (Output) {}
 7    rpc Goodbye (Arg) returns (Output) {}
 8}
 9
10message Arg {
11    repeated string args = 1;
12}
13
14message Output {
15    int32 retcode = 1;
16}

Here is the English description of the contract:

Let’s take a service called MyService. This service provides to actions (commands) remotely:

• Hello
• Goodbye

Both takes as argument an object called Arg that contains an infinite number of string (this array is stored in a field called args).

Both actions return an object called Output that returns an integer.

The specification is clear enough to code a server and a client. But the string implementation may differ from a language to another. You may now understand why we need to “compile” the file. Let’s generate a definition suitable for the go language:

protoc --go_out=plugins=grpc:. myservice/myservice.proto

Note the definition file has been placed into a subdirectory myservice

This command generates a myservice/myservice.pb.go file. This file is part of the myservice package, as specified in the myservice.proto.

The package myservice holds the “contract” translated in go. It is full of interfaces and holds helpers function to easily create a server and/or a client. Let’s see how.

The implementation of the “contract” into the server

Let’s go back to the roots and read the doc of gRPC. In the gRPC basics - go tutorial is written:

To build and start a server, we:

1. Specify the port we want to use to listen for client requests…
2. Create an instance of the gRPC server using grpc.NewServer().
3. Register our service implementation with the gRPC server.
4. Call Serve() on the server with our port details to do a blocking wait until the process is killed or Stop() is called.

Let’s decompose the third step.

“service implementation”

The myservice/myservice.pb.go file has defined an interface for our service.

1type MyServiceServer interface {
2      // Sends a greeting
3      Hello(context.Context, *Arg) (*Output, error)
4      Goodbye(context.Context, *Arg) (*Output, error)
5}

To create a “service implementation” in our “cli” utility, we need to create any structure that implements the Hello(…) and Goodbye(…) methods. Let’s call our structure grpcCommands:

 1package main
 2
 3...
 4import "myservice"
 5...
 6
 7type grpcCommands struct {}
 8
 9func (g *grpcCommands) Hello(ctx context.Context, in *myservice.Arg) (*myservice.Output, error) {
10    return &myservice.Output{int32(0)}, err
11}
12func (g *grpcCommands) Goodbye(ctx context.Context, in *myservice.Arg) (*myservice.Output, error) {
13    return &myservice.Output{int32(0)}, err
14}

Note: *myservice.Arg is a structure that holds an array of string named Args. It corresponds to the proto definition exposed before.

“service registration”

As written in the doc, we need to register the implementation. In the generated file myservice.pb.go, there is a RegisterMyServiceServer function. This function is simply an autogenerated wrapper around the RegisterService method of the gRPC Server type.

This method takes two arguments:

• An instance of the gRPC server
• the implementation of the contract.

The 4 steps of the documentation can be implemented like this:

1listener, _ := net.Listen("tcp", "127.0.0.1:1234")
2grpcServer := grpc.NewServer()
3myservice.RegisterMyServiceServer(grpcServer, &grpcCommands{})
4grpcServer.Serve(listener)

So far so good… The code compiles, but does not perform any action and always return 0.

Actually calling the Run() method

Now, let’s use the grpcCommands structure as a bridge between the cli.Command and the grpc service.

What we will do is to embed the c.Commands object inside the structure and trigger the appropriate objects’ Run() method from the corresponding gRPC procedures.

So first, let’s embed the c.Commands object.

1type grpcCommands struct {
2      commands map[string]cli.CommandFactory
3}

Then change the Hello and Goodbye methods of grpcCommands so they trigger respectively:

• HelloCommand.Run(args)
• GoodbyeCommand.Run(args)

with args being the array of string passed via the in argument of the protobuf.

as defined in myservice.Arg.Args (the protobuf compiler has transcribed the repeated string args argument into a filed Args []string of the type Arg.

 1func (g *grpcCommands) Hello(ctx context.Context, in *myservice.Arg) (*myservice.Output, error) {
 2      runner, err := g.commands["hello"]()
 3      if err != nil {
 4            return int32(0), err
 5      }
 6      ret = int32(runner.Run(in.Args))
 7      return &myservice.Output{int32(ret)}, err
 8}
 9func (g *grpcCommands) Goodbye(ctx context.Context, in *myservice.Arg) (*myservice.Output, error) {
10      runner, err := g.commands["goodbye"]()
11      if err != nil {
12            return int32(0), err
13      }
14      ret = int32(runner.Run(in.Args))
15      return &myservice.Output{int32(ret)}, err
16}

Let’s factorize a bit and create a wrapper (that will be useful in the next section):

 1func wrapper(cf cli.CommandFactory, args []string) (int32, error) {
 2      runner, err := cf()
 3      if err != nil {
 4            return int32(0), err
 5      }
 6      return int32(runner.Run(in.Args)), nil
 7}
 8
 9func (g *grpcCommands) Hello(ctx context.Context, in *myservice.Arg) (*myservice.Output, error) {
10      ret, err := wrapper(g.commands["hello"])
11      return &myservice.Output{int32(ret)}, err
12}
13func (g *grpcCommands) Goodbye(ctx context.Context, in *myservice.Arg) (*myservice.Output, error) {
14      ret, err := wrapper(g.commands["goodbye"])
15      return &myservice.Output{int32(ret)}, err
16}

Now we have everything needed to turn our cli into a gRPC service. With a bit of plumbing, the code compiles and the service runs. The full implementation of the service can be found here.

A very quick client

The principle is the same for the client. All the needed methods are auto-generated and wrapped by the protoc command.

The steps are:

1. create a network connection to the gRPC server (with TLS)
2. create a new instance of myservice’client
3. call a function and get a result

for example:

1conn, _ := grpc.Dial("127.0.0.1:1234", grpc.WithInsecure())
2defer conn.Close()
3client := myservice.NewMyServiceClient(conn)
4output, err := client.Hello(context.Background(), &myservice.Arg{os.Args[1:]})

Note: By default, gRPC requires some TLS. I have specified the WithInsecure option because I am running on the local loop and it is just an example. Don’t do that in production.

Going further

Normally, Unix tools should respect a certain philosophy such as:

Anyway, we all know that tools are verbose, so let’s add a feature that sends the content of stdout and stderr back to the client. (And anyway, we are implementing a service greeting. It would be useless if it was silent :))

stdout / stderr

What we want to do is to change the output of the commands. Therefore, we simply add two more fields to the Output object in the protobuf definition:

1message Output {
2    int32 retcode = 1;
3    bytes stdout = 2;
4    bytes stderr = 3;
5}

The generated file contains the following definition for Output:

1type Output struct {
2      Retcode int32  `protobuf:"varint,1,opt,name=retcode" json:"retcode,omitempty"`
3      Stdout  []byte `protobuf:"bytes,2,opt,name=stdout,proto3" json:"stdout,omitempty"`
4      Stderr  []byte `protobuf:"bytes,3,opt,name=stderr,proto3" json:"stderr,omitempty"`
5}

We have changed the Output type, but as all the fields are embedded within the structure, the “service implementation” interface (grpcCommand) has not changed. We only need to change a little bit the implementation in order to return a completed Output object:

1func (g *grpcCommands) Hello(ctx context.Context, in *myservice.Arg) (*myservice.Output, error) {
2    var stdout, stderr []byte
3    // ...
4    return &myservice.Output{ret, stdout, stderr}, err
5}

Now we have to change the wrapper function that has been defined previously to return the content of stdout and stderr:

1func wrapper(cf cli.CommandFactory, args []string) (int32, []byte, []byte, error) {
2    // ...
3}
4func (g *grpcCommands) Hello(ctx context.Context, in *myservice.Arg) (*myservice.Output, error) {
5    var stdout, stderr []byte
6    ret, stdout, stderr, err := wrapper(g.commands["hello"], in.Args)
7    return &myservice.Output{ret, stdout, stderr}, err
8}

All the job of capturing stdout and stderr is done within the wrapper function (This solution has been found on StackOverflow:

• first, we backup the standard stdout and stderr
• then, we create two times, two file descriptors linked with a pipe (one for stdout and one for stderr)
• we assign the standard stdout and stderr to the input of the pipe. From now on, every interaction will be written to the pipe and will be received into the variable declared as output of the pipe
• then, we actually execute the function (the business logic)
• we get the content of the output and save it to variable
• and then we restore stdout and stderr

Here is the implementation of the wrapper:

 1func wrapper(cf cli.CommandFactory, args []string) (int32, []byte, []byte, error) {
 2	var ret int32
 3	oldStdout := os.Stdout // keep backup of the real stdout
 4	oldStderr := os.Stderr
 5
 6	// Backup the stdout
 7	r, w, err := os.Pipe()
 8        // ...
 9	re, we, err := os.Pipe()
10        //...
11	os.Stdout = w
12	os.Stderr = we
13
14	runner, err := cf()
15        // ...
16	ret = int32(runner.Run(args))
17
18	outC := make(chan []byte)
19	errC := make(chan []byte)
20	// copy the output in a separate goroutine so printing can't block indefinitely
21	go func() {
22		var buf bytes.Buffer
23		io.Copy(&buf, r)
24		outC <- buf.Bytes()
25	}()
26	go func() {
27		var buf bytes.Buffer
28		io.Copy(&buf, re)
29		errC <- buf.Bytes()
30	}()
31
32	// back to normal state
33	w.Close()
34	we.Close()
35	os.Stdout = oldStdout // restoring the real stdout
36	os.Stderr = oldStderr
37	stdout := <-outC
38	stderr := <-errC
39	return ret, stdout, stderr, nil
40}

Et voilà, the cli has been transformed into a grpc webservice. The full code is available on GitHub.

Side note about race conditions

The map used for cli.Command is not concurrent safe. But there is no goroutine that actually writes it so it should be ok. Anyway, I have written a little benchmark of our function and passed it to the race detector. And it did not find any problem:

1go test -race -bench=.      
2goos: linux
3goarch: amd64
4pkg: github.com/owulveryck/cli-grpc-example/server
5BenchmarkHello-2             200          10483400 ns/op
6PASS
7ok      github.com/owulveryck/cli-grpc-example/server   4.130s

The benchmark shows good result on my little chromebook, gRPC seems very efficient, but actually testing it is beyond the scope of this article.

Interactivity

Sometimes, cli tools ask questions. Another good point with gRPC is that it is bidirectional. Therefore, it would be possible to send the question from the server to the client and get the response back. I let that for another experiment.

Terraform ?

At the beginning of this article, I have explained that I was using this specific cli in order to derivate Hashicorp tools and turned them into webservices. Let’s take an example with the excellent terraform.

We are going to derivate terraform by changing only its cli interface, add some gRPC powered by protobuf…

$$\frac{\partial terraform}{\partial cli} + grpc^{protobuf} = \mu service(terraform)$$ ¹

About concurrency

Terraform uses backends to store its states. By default, it relies on the local filesystem, which is, obviously, not concurrent safe. It does not scale and cannot be used when dealing with webservices. For the purpose of my article, I won’t dig into the backend principle and stick to the local one. Hence, this will only work with one and only one client. If you plan to do more work around terraform-as-a-service, changing the backend is a must!

What will I test?

In order to narrow the exercise, I will partially implement the plan command.

My test case is the creation of an EC2 instance on AWS. This example is a copy/paste of the example Basic Two-Tier AWS Architecture.

I will not implement any kind of interactivity. Therefore, I have added some default values for the ssh key name and path.

Let’s check that the basic cli is working:

 1localhost two-tier [master*] terraform plan | tail
 2      enable_classiclink_dns_support:   "<computed>"
 3      enable_dns_hostnames:             "<computed>"
 4      enable_dns_support:               "true"
 5      instance_tenancy:                 "<computed>"
 6      ipv6_association_id:              "<computed>"
 7      ipv6_cidr_block:                  "<computed>"
 8      main_route_table_id:              "<computed>"
 9
10Plan: 9 to add, 0 to change, 0 to destroy.

Ok, let’s “hack” terraform!

hacking Terraform

Creating the protobuf contract

The contract will be placed in a terraformservice package. I am using a similar approach as the one used for the greeting example described before:

 1syntax = "proto3";
 2
 3package terraformservice;
 4
 5service Terraform {
 6    rpc Plan (Arg) returns (Output) {}
 7}
 8
 9message Arg {
10    repeated string args = 1;
11}
12
13message Output {
14    int32 retcode = 1;
15    bytes stdout = 2;
16    bytes stderr = 3;
17}

Then I generate the go version of the contract with:

protoc --go_out=plugins=grpc:. terraformservice/terraform.proto

The go implementation of the interface

I am using a similar structure as the one defined in the previous example. I only change the methods to match the new ones:

1type grpcCommands struct {
2      commands map[string]cli.CommandFactory
3}
4
5func (g *grpcCommands) Plan(ctx context.Context, in *terraformservice.Arg) (*terraformservice.Output, error) {
6      ret, stdout, stderr, err := wrapper(g.commands["plan"], in.Args)
7      return &terraformservice.Output{ret, stdout, stderr}, err
8}

The wrapper function remains exactly the same as the one defined before because I didn’t change the Output format.

Setting a gRPC server in the main function

The only modification that has to be done is to create a listener for the grpc like the one we did before. We place it in the main code, just before the execution of the Cli.Run() call:

 1if len(cliRunner.Args) == 0 {
 2        log.Println("Listening on 127.0.0.1:1234")
 3        listener, err := net.Listen("tcp", "127.0.0.1:1234")
 4        if err != nil {
 5                log.Fatalf("failed to listen: %v", err)
 6        }
 7        grpcServer := grpc.NewServer()
 8        terraformservice.RegisterTerraformServer(grpcServer, &grpcCommands{cliRunner.Commands})
 9        // determine whether to use TLS
10        grpcServer.Serve(listener)
11}

Testing it

The code compiles without any problem. I have triggered the terraform init and I have a listening process waiting for a call:

~ netstat -lntp | grep 1234
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:1234          0.0.0.0:*               LISTEN      9053/tfoliv     

Let’s launch a client:

 1func main() {
 2      conn, err := grpc.Dial("127.0.0.1:1234", grpc.WithInsecure())
 3      if err != nil {
 4            log.Fatal("Cannot reach grpc server", err)
 5      }
 6      defer conn.Close()
 7      client := terraformservice.NewTerraformClient(conn)
 8      output, err := client.Plan(context.Background(), &terraformservice.Arg{os.Args[1:]})
 9      stdout := bytes.NewBuffer(output.Stdout)
10      stderr := bytes.NewBuffer(output.Stderr)
11      io.Copy(os.Stdout, stdout)
12      io.Copy(os.Stderr, stderr)
13      fmt.Println(output.Retcode)
14      os.Exit(output.Retcode)
15}

1~ ./grpcclient
2~ echo $?
3~ 0

Too bad, the proper function has been called, the return code is ok, but all the output went to the console of the server… Anyway, the RPC has worked.

I can even remove the default parameters and pass them as an argument of my client:

1~ ./grpcclient -var 'key_name=terraform' -var 'public_key_path=~/.ssh/terraform.pub'
2~ echo $?
3~ 0

And let’s see if I give a non existent path:

1~ ./grpcclient -var 'key_name=terraform' -var 'public_key_path=~/.ssh/nonexistent'
2~ echo $?
3~ 1

about the output: I have been a little optimistic about the stdout and stderr. Actually, to make it work, the best option would be to implement a custom UI (it should not be difficult because Ui is also an interface). I will try an implementation as soon as I will have enough time to do so. But for now, I have reached my first goal, and this post is long enough :)

Conclusion

Transforming terraform into a webservice has required a very little modification of the terraform code itself which is very good for maintenance purpose:

 1diff --git a/main.go b/main.go
 2index ca4ec7c..da5215b 100644
 3--- a/main.go
 4+++ b/main.go
 5@@ -5,14 +5,18 @@ import (
 6        "io"
 7        "io/ioutil"
 8        "log"
 9+       "net"
10        "os"
11        "runtime"
12        "strings"
13        "sync"
14 
15+       "google.golang.org/grpc"
16+
17        "github.com/hashicorp/go-plugin"
18        "github.com/hashicorp/terraform/helper/logging"
19        "github.com/hashicorp/terraform/terraform"
20+       "github.com/hashicorp/terraform/terraformservice"
21        "github.com/mattn/go-colorable"
22        "github.com/mattn/go-shellwords"
23        "github.com/mitchellh/cli"
24@@ -185,6 +189,18 @@ func wrappedMain() int {
25        PluginOverrides.Providers = config.Providers
26        PluginOverrides.Provisioners = config.Provisioners
27 
28+       if len(cliRunner.Args) == 0 {
29+               log.Println("Listening on 127.0.0.1:1234")
30+               listener, err := net.Listen("tcp", "127.0.0.1:1234")
31+               if err != nil {
32+                       log.Fatalf("failed to listen: %v", err)
33+               }
34+               grpcServer := grpc.NewServer()
35+               terraformservice.RegisterTerraformServer(grpcServer, &grpcCommands{cliRunner.Commands})
36+               // determine whether to use TLS
37+               grpcServer.Serve(listener)
38+       }
39+
40        exitCode, err := cliRunner.Run()
41        if err != nil {
42                Ui.Error(fmt.Sprintf("Error executing CLI: %s", err.Error()))

Of course, there is a bit of work to setup a complete terraform-as-a-service architecture, but it looks promising.

Regarding grpc and protobuf: gRPC is a very nice protocol, I am really looking forward an implementation in javascript to target the browser (Meanwhile it is possible and easy to set up a grpc-to-json proxy if any web client is needed).

But it reminds us that the main target of RPC is machine-to-machine communication. This is something that the ease-of-use-and-read of json has shadowed…