#Programming | #Golang

August 16, 2018

Twisk - Golang RPC starter kit

Twisk, an acronym for Twirp starter kit, helps you get started with a simple Golang RPC framework with protobuf service definitions - Twirp. It features everything from authorization, implemented CRUD on a single entity, logging, configuration and more. Using minimal dependencies, idiomatic code and best practices, it helps you get started with Golang backend API development - both JSON and Protobuf.

About four to five months ago our company adopted Twirp for building some of our APIs with it. Since there are barely any blog posts and tutorials mentioning it, I decided to give a lightning talk on how we use it at Gophercon. To further improve my knowledge and have a working example for the audience, I made Twisk.

Previously I made something similar (Gorsk - using Echo, and Gorsk-gin using Gin - of course) so making Twisk wasn’t that much complicated. It does feature some different concept and I’ll mostly focus on them in this blog post.

What is Twirp

From Twirp’s README file - “Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API definition files and lets you focus on your application’s logic instead of thinking about folderol like HTTP methods and paths and JSON.

Define your service in a Protobuf file and then Twirp autogenerates Go code with a server interface and fully functional clients. It’s similar to gRPC, but without the custom HTTP server and transport implementations: it runs on the standard library’s extremely-well-tested-and-high-performance net/http Server. It can run on HTTP 1.1, not just http/2, and supports JSON clients for easy integrations across languages

Twirp handles routing and serialization for you in a well-tested, standardized, thoughtful way so you don’t have to. Serialization and deserialization code is error-prone and tricky, and you shouldn’t be wasting your time deciding whether it should be “POST /friends/:id/new” or “POST /:id/friend” or whatever. Just get to the real work of building services!”

Why Twirp

Prior to Twirp, we built our services using net/http and gorilla/mux. We have plenty of them still running in production - but writing those services had a few inconveniences.

If you’re going to stick to the stdlib handleFunc, you’re probably going to have lots of duplicated code (unmarshaling requests, validations …)
API consumers need to wait until APIs are ready to be able to develop unless you’re mocking your APIs with a tool like Apiary.
Writing handler tests.
Writing swagger specs using go-swagger (not manual, but takes some time)

Twirp solves these problems in an elegant way. Our validations (most of them) and swagger docs are automatically generated.

Once we all agree on the proto definition file, we generate clients in our CI/CD pipeline for Typescript and Java, so our frontend and mobile developers can start working on the new feature - immediately.

There’s no need to worry about JSON serialization or HTTP verbs/routes! Twirp routing and serialization handles that for you, reducing the risk of introducing bugs. Twirp supports both JSON and Protobuf.

Thanks to proto definition, Twirp supports ENUMs which Go does not.

Functions generated by Twirp have the following signature:

func funcName(context.Context, req *twisk.Request) (*twisk.Response, error)

With this, you get the context (from Go 1.7+), marshaled and validated request in req variable. The signature might draw some away, but it’s the consistency that I like about it. No more tinkering with request/response format.

Our application services remained unchanged. Instead, we create a transport.go file that implements Twirp’s service interface, and maps Twirp’s request/response to our models. This helped us speed up transition from net/http and I prefer it that way.

If you’re building a large set of microservices, with Twirp you can start using protobufs for calling the services instead of JSON. You could use gRPC with gRPC-gateway as well, but WWwirp does have few advantages over it:

Twirp has a much simpler protocol, so hand-written clients are pretty easy
Twirp supports http 1.1
Twirp supports JSON-encoded messages (available with gRPC-gateway in case of gRPC)
gRPC supports unbounded streaming RPCs, twirp is only request-response (although the proposal for streaming RPCs is open)

Twirp Hooks

Unlike middleware used for routers, Twirp features hooks. These hooks provide a framework for side-effects at important points while a request gets handled. You can do things like log requests, record response times in statsd, authenticate requests, and so on. ServerHooks struct contains five methods with self-explanatory names:

type ServerHooks struct {
    // RequestReceived is called as soon as a request enters the Twirp
    // server at the earliest available moment.
    RequestReceived func(context.Context) (context.Context, error)

    // RequestRouted is called when a request has been routed to a
    // particular method of the Twirp server.
    RequestRouted func(context.Context) (context.Context, error)

    // ResponsePrepared is called when a request has been handled and a
    // response is ready to be sent to the client.
    ResponsePrepared func(context.Context) context.Context

    // ResponseSent is called when all bytes of a response (including an error
    // response) have been written. Because the ResponseSent hook is terminal, it
    // does not return a context.
    ResponseSent func(context.Context)

    // Error hook is called when an error occurs while handling a request. The
    // Error is passed as argument to the hook.
    Error func(context.Context, Error) context.Context
}

An example of this is present in Twisk - WithJWTAuth.

Things I don’t like about Twirp

It introduces a bit of complexity to start with.
Even though it’s promoted as a feature, and I can see some of its benefits, if you’re exposing your APIs to end user (being visible in the network console), having something such as domain.com/twirp/google.user.User/CreateUser seems a bit ugly. This will slightly change with Twirp v6, where twirp prefix is gone (new route will be domain.com/google.user.User/CreateUser). There was a discussion on changing the routing scheme from package.serviceName to package/serviceName (e.g. iam.Password/ChangePassword to iam/password/change) but it was rejected for now.
Manually mapping response, which is usually a database model, into protobuf one. Converting Go’s time.Time into google.protobuf.Timestamp, concrete values into ENUMs etc.
Protobuf 3 doesn’t support nillable fields. There are wrappers that support this, but they introduce more complexity and you have to manually check for val != nil. Since nillable fields are not supported, update (PATCH) request receiving only changed fields becomes impossible. Yet again, there is FieldMask, but with Go, you have to use reflection to parse the elements and it feels like a rushed-out workaround.

Conclusion

As everything else, Twirp has its good and bad sides. I think some services are a great fit for Twirp, especially if you need to prototype fast and support backwards compatibility - Twirp is great. If your microservice is going to get invoked a lot by other microservices, generated JSON/Protobuf clients are a great asset too.