Twirp -- Build Simple, High Performance Go HTTP Protobuf Services

Posted at — Nov 28, 2023

Overview

Twirp is an RPC / protobuf framework developed by Twitch.tv. It offers some very powerful features:

JSON or protobuf responses, depending on the request Content-Type header (application/protobuf or application/json)
Hooks sprinkled throughout the request / response lifecycle
Interceptors, Twirp’s middleware
Auto-generated clients in in all popular languages (Go, Rust, Ruby, Node, Python, etc.)
Built on top of Go’s net/http package.

How it works

We’re going to walk through the code in some detail, and there is a lot to absorb. But keep in mind that Twirp implementation is actually very simple:

Make .proto files containing request / response messages and endpoint definitions
Run Twirp’s generator
Implement hooks and/or interceptors
Implement the methods listed in the generated code’s interface
Fire up your server

Define your protobuf services in your .proto file

rpc/charlie_go.go

 1syntax = "proto3";
 2
 3package charlie_go;
 4option go_package = "rpc/charlie-go";
 5
 6// This "service" represents the endpoints, and all the
 7// related code, that Twirp will generate
 8
 9service CharlieGo {
10  rpc CreateItem(CreateItemRequest) returns (None);
11  rpc GetItem(GetItemRequest) returns (Item);
12}
13
14// These messages represent the requests and responses
15// Twirp's generator will transform into Go structs
16
17message None {}
18
19message CreateItemRequest {
20  string name = 1; // required
21}
22
23message GetItemRequest {
24  string id = 1;
25}
26
27message Item {
28  string id = 1;
29  string name = 2;
30  string created_at = 3;
31  string updated_at = 4;
32}

Run Twirp’s generator. See here. It generates our client and server, along with some utility functions. We’re going to focus on the server and client code. Included in it:
- The NewCharlieGoServer function that creates the HTTP handler we’ll pass to our mux
- An interface we’ll implement via our provider. Our provider handles our business logic; it accepts a request struct, creates an item, and hands back a response which our handler will pass back to the client.
- Interceptor and hook method calls, that will allow us to hook into various points in the HTTP request / response lifecycle
- A Go protobuf client

Note that Twirp is also able to generate clients in JS, Ruby, Python, and a number of other languages. We’re going to generate only the Go client.

Code Crawl

At a high level, Twirp’s generated server code works just as you’d expect from any HTTP framework, with a few slick features thrown in:

It allows for only the POST HTTP method
It requires that we implement only the business logic, which it buries inside all the needed HTTP machinery, thus allowing for surrounding our business logic with hooks and interceptors
It exposes the required business logic as an interface we must implement
It switches on request paths to fire off requests to the proper code paths
It also switches on request header, providing different paths for application/json and application/protobuf requests/responses

Let’s crawl through the code, from the beginning to the end of the CreateItem request / response, and see how it works.

Our Run() function creates provider and handler instances, and starts our http server:

 1func Run() {
 2
 3// The "provider" is our code's base object. It implements the
 4// interface Twirp generated for us in 
 5// rpc/charlie_go/twirp_server.go, as we'll see below.
 6
 7  provider := twirp_server.NewProvider()
 8
 9// We can create as many twirp.ChainHooks objects as we want.
10// Each can implement any or all the hook functions that are
11// sprinkled throughout Twirp's generated code in twirp_server.go
12
13  chainHooks := twirp.ChainHooks(
14    provider.AuthHooks(),
15  )
16
17// We instantiate the http package's muxer:
18
19  mux := http.NewServeMux()
20
21// Here are the paths Twirp creates for our endpoints. Note below
22// that we are setting a PathPrefix api/v1; Without it, our paths
23// would be like twirp/charlie_go/CharlieGo instead of
24// api/v1/charlie_go/CharlieGo
25
26  // POST http(s)://<host>/api/v1/charlie_go/CharlieGo/CreateItem
27  // POST http(s)://<host>/api/v1/charlie_go/CharlieGo/GetItem
28
29// In Go's http package, any object that implements the
30// http.Handler interface, ie any object that as a
31// ServeHTTP method with the right signature, can be
32// passed into a mux
33
34  handler := charlie_go.NewCharlieGoServer(provider,
35    twirp.WithServerPathPrefix("/api/v1"), chainHooks)
36
37  mux.Handle(
38    handler.PathPrefix(), twirp_server.AddJwtTokenToContext(
39      handler,
40    ),
41  )
42// Start our server
43	http.ListenAndServe(":8080", mux)
44}

Notice the AddJwtTokenToContext middleware. We require a valid JWT in the Authorization header for requests to CreateItem:

 1// This function adds whatever is in the Authorization header
 2// to the request context.
 3
 4func AddJwtTokenToContext(next http.Handler) http.Handler {
 5	return http.HandlerFunc(
 6		func(w http.ResponseWriter, r *http.Request) {
 7			auth := r.Header.Get(Authorization)
 8			if auth != "" {
 9				ctx := r.Context()
10				ctx = context.WithValue(ctx, contextJWT, auth)
11				r = r.WithContext(ctx)
12			}
13			next.ServeHTTP(w, r)
14		},
15	)
16}

Once the request makes it through the middleware, it passes to our Twirp server’s ServeHTTP method:

rpc/charlie_go/twirp.server.go

1func (s *charlieGoServer) ServeHTTP(resp http.ResponseWriter,
2	req *http.Request) {
3...
4
5// callRequestReceived is the first Twirp hook that gets called,
6// near the top of the ServeHTTP function:
7
8  ctx, err = callRequestReceived(ctx, s.hooks)

Twirp has a concept of ChainHooks, collections of ServerHooks which Twirp will call in the order they are passed into our NewCharlieGoServer handler constructor. Each ServerHook can implement any or all of Twirp’s five ServerHook methods:

 1// ServerHooks is a container for callbacks that can instrument a
 2// Twirp-generated server. These callbacks all accept a context
 3// and return a context. They can use this to add to the request
 4// context as it passes through the code, appending values or
 5// deadlines to it.
 6//
 7// The RequestReceived and RequestRouted hooks are special: they
 8// can return errors. If they return a non-nil error, handling
 9// for that request will be stopped at that point. The Error hook
10// will be triggered, and the error will be sent to the client.
11// This can be used for stuff like auth checks before deserializing
12// a request.
13//
14// The RequestReceived hook is always called first, and it is
15// called for every request that the Twirp server handles. The
16// last hook to be called in a request's lifecycle is always
17// ResponseSent, even in the case of an error.
18//
19// Details on the timing of each hook are documented as comments on
20// the fields of the ServerHooks type.
21type ServerHooks struct {
22	// RequestReceived is called as soon as a request enters the
23	// Twirp server at the earliest available moment.
24	RequestReceived func(context.Context) (context.Context, error)
25
26	// RequestRouted is called when a request has been routed to a
27	// particular method of the Twirp server.
28	RequestRouted func(context.Context) (context.Context, error)
29
30	// ResponsePrepared is called when a request has been handled
31	// and a response is ready to be sent to the client.
32	ResponsePrepared func(context.Context) context.Context
33
34	// ResponseSent is called when all bytes of a response
35	// (including an error response) have been written.
36	// Because the ResponseSent hook is terminal, it
37	// does not return a context.
38	ResponseSent func(context.Context)
39
40	// Error hook is called when an error occurs while handling a
41	// request. The Error is passed as argument to the hook.
42	Error func(context.Context, Error) context.Context
43}

(see here)

We’ve created only one ServerHook (see here) for Auth, which checks whether each request has a valid JWT. It implements only the RequestReceived method.

All the hook functions are called in the same way. They iterate over all ServerHooks, and call the hook function if it’s implemented. The way this is implemented is very clever. Recall that in cmd/twirp_server.go, we create ChainHooks, and pass them into our handler:

1chainHooks := twirp.ChainHooks(
2	provider.AuthHooks(),
3)

The below code shows Twirp’s code iterates over each hook, and calls the RequestReceived method on each, if it exists:

 1// ChainHooks creates a new *ServerHooks which chains the callbacks
 2// in each of the constituent hooks passed in. Each hook function
 3// will be called in the order of the ServerHooks values passed in.
 4//
 5// For the erroring hooks, RequestReceived and RequestRouted, any
 6// returned errors prevent processing by later hooks.
 7func ChainHooks(hooks ...*ServerHooks) *ServerHooks {
 8	if len(hooks) == 0 {
 9		return nil
10	}
11	if len(hooks) == 1 {
12		return hooks[0]
13	}
14	return &ServerHooks{
15		RequestReceived: func(ctx context.Context) (context.Context,
16				error) {
17			var err error
18
19      // iterate over each ServerHook and call each
20
21			for _, h := range hooks {
22				if h != nil && h.RequestReceived != nil {
23					ctx, err = h.RequestReceived(ctx)
24					if err != nil {
25						return ctx, err
26					}
27				}
28			}
29			return ctx, nil
30		},
31		RequestRouted: func(ctx context.Context) (context.Context,
32				error) {
33			var err error
34			for _, h := range hooks {
35				if h != nil && h.RequestRouted != nil {
36					ctx, err = h.RequestRouted(ctx)
37					if err != nil {
38						return ctx, err
39					}
40				}
41			}
42			return ctx, nil
43		},
44		ResponsePrepared: func(ctx context.Context) context.Context {
45			for _, h := range hooks {
46				if h != nil && h.ResponsePrepared != nil {
47					ctx = h.ResponsePrepared(ctx)
48				}
49			}
50			return ctx
51		},
52		ResponseSent: func(ctx context.Context) {
53			for _, h := range hooks {
54				if h != nil && h.ResponseSent != nil {
55					h.ResponseSent(ctx)
56				}
57			}
58		},
59		Error: func(ctx context.Context, twerr Error) context.Context {
60			for _, h := range hooks {
61				if h != nil && h.Error != nil {
62					ctx = h.Error(ctx, twerr)
63				}
64			}
65			return ctx
66		},
67	}
68}

rpc/charlie_go/twirp.server.go cont’d

 1// The code switches on each implemented RPC method, which matches
 2// 1:1 with the rightmost part of the request URL:
 3
 4  switch method {
 5  case "CreateItem":
 6      s.serveCreateItem(ctx, resp, req)
 7...
 8
 9func (s *charlieGoServer) serveCreateItem(ctx context.Context,
10	resp http.ResponseWriter, req *http.Request) {
11...
12// Switch based on whether the ContentType == application/json
13// or application/protobuf:
14
15  switch strings.TrimSpace(strings.ToLower(header[:i])) {
16  case "application/json":
17    s.serveCreateItemJSON(ctx, resp, req)
18  case "application/protobuf":
19    s.serveCreateItemProtobuf(ctx, resp, req)
20...

serveCreateItemProtobuf and serveCreateItemJSON are nearly identical. We’ll look at serveCreateItemProtobuf. Here is where the remaining hooks methods are called, and where the response is returned:

  1func (s *charlieGoServer) serveCreateItemProtobuf(ctx context.Context,
  2	resp http.ResponseWriter, req *http.Request) {
  3	var err error
  4	ctx = ctxsetters.WithMethodName(ctx, "CreateItem")
  5
  6// callRequestRouted hook function
  7
  8	ctx, err = callRequestRouted(ctx, s.hooks)
  9	if err != nil {
 10		s.writeError(ctx, resp, err)
 11		return
 12	}
 13
 14	buf, err := ioutil.ReadAll(req.Body)
 15	if err != nil {
 16		s.handleRequestBodyError(ctx, resp,
 17			"failed to read request body", err)
 18		return
 19	}
 20	reqContent := new(CreateItemRequest)
 21	if err = proto.Unmarshal(buf, reqContent); err != nil {
 22		s.writeError(ctx, resp, malformedRequestError("the protobuf
 23			request could not be decoded"))
 24		return
 25	}
 26
 27  // Twirp calls our CreateItem function a "handler", which frankly
 28  // is a little misleading
 29
 30  handler := s.CharlieGo.CreateItem
 31	if s.interceptor != nil {
 32		handler = func(ctx context.Context, req *CreateItemRequest)
 33			(*None, error) {
 34			resp, err := s.interceptor(
 35				func(ctx context.Context, req interface{}) (interface{},
 36					error) {
 37					typedReq, ok := req.(*CreateItemRequest)
 38					if !ok {
 39						return nil, twirp.InternalError("failed type assertion
 40							req.(*CreateItemRequest) when calling interceptor")
 41					}
 42					return s.CharlieGo.CreateItem(ctx, typedReq)
 43				},
 44			)(ctx, req)
 45			if resp != nil {
 46				typedResp, ok := resp.(*None)
 47				if !ok {
 48					return nil, twirp.InternalError("failed type assertion
 49						resp.(*None) when calling interceptor")
 50				}
 51				return typedResp, err
 52			}
 53			return nil, err
 54		}
 55	}
 56
 57	// Call service method
 58	var respContent *None
 59	func() {
 60		defer ensurePanicResponses(ctx, resp, s.hooks)
 61
 62// Here is where our CreateItem function gets called ...
 63
 64		respContent, err = handler(ctx, reqContent)
 65	}()
 66
 67	if err != nil {
 68
 69// ... and here is where our response, , or an error, is returned
 70
 71		s.writeError(ctx, resp, err)
 72		return
 73	}
 74	if respContent == nil {
 75		s.writeError(ctx, resp, twirp.InternalError("received a nil
 76			*None and nil error while calling CreateItem. nil responses
 77			are not supported"))
 78		return
 79	}
 80
 81	ctx = callResponsePrepared(ctx, s.hooks)
 82
 83	respBytes, err := proto.Marshal(respContent)
 84	if err != nil {
 85		s.writeError(ctx, resp, wrapInternal(err, "failed to marshal
 86			proto response"))
 87		return
 88	}
 89
 90	ctx = ctxsetters.WithStatusCode(ctx, http.StatusOK)
 91	resp.Header().Set("Content-Type", "application/protobuf")
 92	resp.Header().Set("Content-Length", strconv.Itoa(len(respBytes)))
 93	resp.WriteHeader(http.StatusOK)
 94	if n, err := resp.Write(respBytes); err != nil {
 95		msg := fmt.Sprintf("failed to write response, %d of %d bytes
 96			written: %s", n, len(respBytes), err.Error())
 97		twerr := twirp.NewError(twirp.Unknown, msg)
 98		ctx = callError(ctx, s.hooks, twerr)
 99	}
100	callResponseSent(ctx, s.hooks)
101}

Summary

Twirp allows for creating highly scalable protobuf / JSON services, with built-in powerful features. At heart, it bears little difference from any Go HTTP server you’d write yourself, or any of the frameworks out there.

Once you have it set up in your project, the 1-2-3 of 1) creating new .proto request / response messages and services, 2) generating code, and 3) implementing business logic, is quite simple and fast.

In addition, there are terrific insights in Twirp’s code, including the code it generates for us, into proper Go idioms. I hope this blog post inspires you to create a Twirp service, generate some code, and do an even more detailed crawl through it than we’ve done together.

Addendum – Run the charlie-go app

We use Cobra, a Go CLI utility that allows us to create one binary, but run different parts of it, depending on what flags we pass to our binary.

Note below, that we’ll generate one binary, and we’ll run our server, JWT generator, and db seeds, from that one binary, by passing flags to it.

Copy .env.sample to .env
- My local settings are in there, so you get a concrete idea of what yours should look like, but you’ll have to replace my settings with yours
Pull the charlie-go code: git clone [email protected]:charliemcelfresh/charlie-go.git
Install and run Postgres locally, and set your local DATABASE_URL in .env
Get dbmate and Create your database and run migrations: dbmate up
Run seeds: go run main.go seed_database or go build && ./go-by-charlie seed_database
Look in the database, and make sure you have one record each in the users, items, and user_items table
Run our Twirp server: go run main.go twirp-server or go build && ./go-by-charlie twirp_server
Generate JWTs:
- Generate a keypair, set their file permissions, and update your .env:
  - ssh-keygen -t rsa -b 4096 -m PEM -f jwtRS256.key && chmod 600 jwtRS256.key
  - openssl rsa -in jwtRS256.key -pubout -outform PEM -out jwtRS256.key.pub && chmod 644 jwtRS256.key.pub
  - set JWT_PRIVATE_KEY and JWT_PUBLIC_KEY in your .env file
- Generate a user JWT:
  - Grab a user’s id from the users table
  - go run main.go generate_jwt 10000h <users.id> or go build && ./go-by-charlie generate_jwt 10000h <users.id>

Now you should be able to paste your token in curl requests:

Create an item: __curl –location ‘http://localhost:8080/twirp/charlie_go.CharlieGo/CreateItem’
–header ‘Content-Type: application/json’
–header ‘Authorization: Bearer ’
–data ‘{ “name”: “Widget 02” }’
Get an item:

curl –location ‘http://localhost:8080/twirp/charlie_go.CharlieGo/GetItem’
–header ‘Content-Type: application/json’
–header ‘Authorization: Bearer ’
–data ‘{ “id”: “<items.id in your database connected to the user in the above JWT>” }’

McElfresh Blog

Go, PostgreSQL, MySQL, Ruby, Rails, Sinatra, etc.