Made by
Made by
Tristan Sokol speaking at DevXcon 2017
So before we begin, one of the main things I wanted to cover in this talk was, you know, let’s say you have a rest API and you developed these little bundles of software that allow end users to make API calls more easily. Do you call those things SDKs? Do you call those things client libraries? Could we get a show of hands for SDKs? Nobody. Show of hands for client libraries? Also nobody. So yeah. Very split field.
So I’ve always wanted to do a good, the bad, the ugly thing presentation and so you get to see what 30 minutes of keynote animations look like. Imagine some yesteryear world of western ideals I guess. The thing that I’m supposed to be describing here is that we went through a process of identifying how we want to build our SDKs not too long ago and figuring out what sort of strategies other companies use. And this is the learnings from that. I’ll be diving deeper into what we end up choosing which was SDK generation.
So in case you guys didn’t hear it in the beginning, I work at Square. If you’re not familiar with our org chart, I’m gonna give you a little rundown. We have a bunch of other like big parts of the company and then we have developers where I work. And one of the interesting things about our developer platform team which is what we’re called is that we don’t actually make any of the APIs that our end users consume. All these guys do. So our team handles things like, you know, authentication, doing the documentation, the developer portal, handling OAuth. Anything that an end user that’s external from our company might want to use for APIs but maybe it isn’t as applicable for internal use.
So one of those things is our SDKs. I hope you guys love this sound because you’re gonna hear it a lot. So one of the reasons I’m harping on SDKs so much is that, you know, looking from the end user experience, right, an end user…like as a product owner or a company, you might thing that API is like what you produce. Like I’m creating this restful service that other people are gonna use but in reality, whatever people can pull down from their favorite package manager is like the end user experience. And so, if you have a bad experience there, like welcome to their competitor. Like, you know, somebody else is going to move along to find someone else who can give you a good experience there.
So that brings me into my first SDK generation technique called HOPE. So companies of all sizes…and this is something that we are very guilty of at Squares. You know, our development team for the APIs is fairly small and we want to release new features. We wanna release new functionality. We have new endpoints we’re working on. We love doing that. We don’t love doing so much tooling around our APIs or, you know, maintenance around our sample apps. And so, you know, any company of any size is sort of guilty of that and so one technique that we saw people using is, you know, HOPE. Building a little stub of an SDK, hoping that somebody in the world of GitHub or your open source or existing developer community jumps on that and just like writes all that code for you. You know, in reality, this causes your developers not really have any experience with an SDK, right. You can get an SDK that somebody wrote 60% for their application and then the other 60%, you know, or 40% is just empty, you know. The next person who comes along writes another 60%. It’s a different overlapping 60%, uses different idioms, different language. Your experience just becomes super fragmented. The company I was previously at…if you search for product PHP SDK in GitHub, there’s four pages of results. Only two of them are forks of each other. And so, you have a just wide berth of experiences that you aren’t in control of and as an end user, you know, you’re not really helping them find the right experience for them.
So there are ways you can do this properly. You know, you can make the people who make these open source contributions, your SDKs sort of, you know, show them on your documentation, turn them into evangelists themselves, you know, highlight one open source SDK and like make everyone else contribute to that one. You can do this correctly but that was something that we decided was not gonna work for us.
The next technique I like to call BESPOKE. Handcrafted, artisanal SDKs. Great logo there. So the solution seems to be why don’t you just start making all these SDKs yourself, right? Like just, you know, hire somebody to write your Java SDK. That’s great. But, you know, this person maybe doesn’t know Ruby, right. So like well, that’s easy. One more head count. But, you know, if you’re creating a bespoke, awesome experience in each language, you know, it’s hard to have super great depth of knowledge in each programming language and so maybe if you wanna do, you know, Python or PHP, you’re gonna need a couple of other people but like well, now…you know, you have a core dev team that’s already working on new features and new endpoints for your APIs and so, you know, you have all these people releasing SDKs. Like what about somebody to sort of align all those documentation pieces or maybe somebody to write the GO SDK or Haskell. You know, what about a community manager, handle the issues that people are filling on GitHub for your SDK? It’s sort of like a population explosion when, you know, perhaps your team is not that big to start with.
So this was something that we would’ve loved to have done but we do not have enough money to hire all these smiling faces. And so that was something that we decided not to do either. So how do you get the best of both worlds, how do you get an experience that is both good for your end users but also doesn’t require you to have sort of the resourcing of a gazillion dollar company? Leading us to our third sound clip.
The generated. So we use SDK generation. It all sort of starts with our specification file. Here you can see it in its entirety as it loops through for 15 seconds. It’s a few thousand lines long. And so, the specification file sort of describes all aspects of your API and, you know, from what the endpoints look like to some metadata. We actually use the Open API specification. It’s formerly known as Swagger. So if you look around on the internet, Swagger has like really good SEO for it. Open API is a little bit harder. So you might have to look for both terms as you explore. There’s other specification formats out there such as RAML or API Blueprint. Definitely explore those options if this is something that you’re interested in because the tooling around each ecosystem is a little bit different. We chose Open API specifically for some tooling that I’ll mention in the future but there’s other options out there, for sure.
Just to give you an example of what some of that specification looks like, it’s a JSON file file. Here you can see some metadata about our API, you’re gonna have some versioning, some contact information, you can see what license or terms of service you need to be using. This is an actual part of our endpoint and so at the top, you can see this is actually the URL. So it’s, you know, connect.scrub.com/b1/me/employees. That’s the URL and you can see the HTTP verb that you use underneath. It’s a post request. And then there’s some metadata there around what the input does. That gets exposed to the SDKs. Also, to some documentation. And then you can see it’s sort of what you need to pass into it and so here we’re giving it employees object. So this is for creating an employee.
There’s other stuff that you can stick in there as well. We use OAuth but API Key is also supported. You can see our external documentation. So think of how nice it would be if when you’re in SDK and you’re like, “Oh, man. Like where can I look up more info about this endpoint?” Well, you can just stick that right into the SDK itself. That’s nice.
And then you can see here what kind of responses you get in the back. So you get an employee object back after you create an employee. Isn’t that great? You can put a bunch of more stuff in here. It’s like example responses, what happens when you get an error back. It’s really pretty expressive depending on if it matches how you designed your API which is one of the challenges.
And so, we actually publish this publicly. You can go on GitHub, see our spec. And then also on GitHub we have a bunch of reposts for all of our SDKs. Currently we have five SDKs. And that’ll be more important later. So remember our org chart. I’m sure you guy studied it very carefully. We don’t make the APIs, right. These guys do. And we actually use prickle buffers to communicate with all of our internal services. And so, it’s a really useful toolset for us but each of our internal services communicate with each other and that contract is defined in proto files and so we have these files that sort of innately define how our services communicate with each other, right.
And so, we actually take those proto files and we actually just have another tool that we’ve cleverly named Java Tool and turn those into our API specification. And so, this is really important for us because we have a lot of people upstream from us making these proto files and so whenever changes occur in any way…let’s say employee has changed, right. We can just sorta automatically change that or reflect that change in our specification without having to do any manual tooling, right. So we’re not a bottleneck to other services upstream creating any changes which is very important to us.
And so, then we get into the star of the show which is a tool called Swagger Codegen. Swagger Codegen is an open source project that takes your specification and turns it into SDKs. Basically. How it works, you have your specification like I mentioned. Then you have a series of templates. And so, these templates are made with a framework called Mustache. And Swagger Codegen just basically…it takes your specification, applies it to these templates and out pops an SDK, you know. So do some configuration right there. And I’ll show you a little bit more about how that actually looks and so in our case, we make SDKs with that. And so, what we used to do is that we…every time we saw a change in our upstream protos, we’d regenerate the specification and then somebody would just download Swagger Codegen on the laptop and just start popping out SDKs that we pushed. As you can imagine, that manual process right there was bad. It was actually terrible. Swagger Codegen’s a very active project and so a lot of things change very quickly on it. So if you don’t generate all the time, it’s gonna be an issue for you.
So the costar of our show is Travis CI. So Travis…you know, I’m sure most of you guys are pretty familiar with it. It’s a continuous integration framework, works well with GitHub. And so, all of our work is actually done by Travis now. So we actually take our specification repo and we actually check in these templates and config files as well and so in the same repo you can see all the templates that we use, all the configuration files and one thing…like I mentioned, Swagger Codegen is a pretty active project. If you don’t check in your templates, things are gonna break unexpectedly because Swagger Codegen just uses the latest and greatest templates that are out there. So if you don’t sort of manually make sure that those things work, you’re gonna have an issue there.
This is sort of one of our repos. And so, the config files…they’re just little JSON files that talk about…you know, for Ruby you can specify your Ruby Gem name or we can talk a little bit more about versioning or if you have like a special model name for something, you can stick that in there too. They’re pretty…very small JSON files.
So what does Travis do for us? Well, quite a bit. I’m gonna break out each little section here. And so, the first part is actually just installing Swagger Codegen to the Travis machine so…and you can actually see all this in our GitHub repo. Travis files are pretty public. So we take the script to just download the latest version of Swagger from the Swagger Codegen project, the Git repo. Then we compile it with Java and, you know, build it there. This line is a section about decrypting some secrets. And so, Travis CI has some built-in infrastructure for encrypting secrets. It’s kinda tricky because you can never see your secrets again. So it’s a little bit harder to use on a team, right, because only one person really knows the secrets. But so basically it keeps on deploy keys in a zip file or a tar file and then we decrypt those on the build machine.
And then we have a script that generates all of our SDKs. Here you can see the actual script. It’s pretty easy. We just have a list of all the languages that we have. And, you know, the…this is like Swagger Codegen. Generate. You tell what your spec is so it’s our API specification file. You describe what language you’re using, whether it’s PHP or Java or Node, Ruby, whatever. The config files, the templates. Those are all in that same repo. And then, you know, we just stick it in a folder. And so, Travis CI then goes to and builds all of the SDKs for us.
And then the next part is just taking these SDKs and Travis CI actually pushes the outputs to each of our other Git repos. And so here we’ve gone through a process of, you know, somebody changed some upstream proto for us. We generate an API specification and then as soon as we like make the pull request for that, we have Travis CI pushing out all of these new complete SDKs to like your repos. And then we do…we have a process around, you know, making sure that these look correct. So one of the awesome things that we’ve done is that we actually…so we generate…we run our tests on pull requests too and so let’s say I have a new idea for a change to the template. And so, I wanna make like a little template change that pushes, you know, a different type of link in or something. And so, I make a pull request for that. Travis CI then generates all the SDKs, pushes them, makes those a new branch in all the SDK repos and I can actually just use the Git compare features to see like, “Oh, wow. I made the change in this template and like here’s what the actual SDK looks like or, you know…” You actually test that SDK out by just pulling down the branch.
And so that’s been very helpful for us. That’s been the most exciting part of this whole process is that you can make a change to the specification or template that’s minor and see how it propagates throughout the whole system. So that’s been great.
And so, after we, you know, we see these new SDKs, we say, “Hey, man, that looks great.” We merge them to the master. We also use Travis CI to do a bunch of the testing on our SDKs and so you can imagine once the SDKs get pushed to their individual repos, then Travis CI runs some more tests on those repos about, “Oh, did this new SDK break any of the integrations? Like do all the endpoints work like they used to?” You know, if you’re changing the templates and changing the specifications, they might not. Each language template works a little bit differently. They all have different contributors. It’s a very active project. You can contribute to them. We’ve made a few contributions back to the project. And so, it’s important to have these tests to make sure like, “Oh, I made this change to the specification and my templates are still working.”
Because a lot of times there’ll be changes lock step between new features and specifications to new Swagger Codegen versions and so if you try to use some unsupported feature, then if you don’t upgrade your templates, then it’s not gonna work anyway. So an important thing to sorta keep in mind there.
So it’s kind of the general overview of everything. You know, one last sound. And so some of the pros to this is that, you know, our team acts as a thin layer across many other teams that are actually working on the APIs. And so, we don’t have to worry about becoming a blocker for any of these other teams, you know. Almost all the work is done automatically and so if, you know, you’re on some upstream team and like, “I wanna release a new endpoint.” Okay. Like well, you updated your proto, right? And so, you know, that goes through Travis CI, hits our specification or then goes through Travis CI. It goes to all the repos and become a new SDK for everyone to use.
So it’s great if you don’t have a bunch of people you want to dedicate to this kind of work. Not so great if you want that bespoke handcrafted experience, right. You know, sort of…if you can imagine generating an SDK, it’s not gonna be the most idiomatic code you’re ever gonna see, right. It’s really a template file that gets mashed over and over again. So there’s some complaints about like, “Oh, man, this Ruby SDK looks a lot like a Java SDK.” It’s like well, yeah. Think about how it was made I guess. But it’s better than no SDK in our minds.
So yep. Our develop relations team is hiring. My name’s Tristan. Feel free to reach out to me if you are interested in more and hearing how we do it or if, you know, you wanna get a job. But yeah. Thank you.