The UX of DX: user testing in the invisible world of APIs


Jenny Wanger

Jenny Wanger

If you want to cultivate loyal developers, you should treat the products you create for them the same you would any other user-facing offering. And that, according to Jenny Wanger in her DevRelCon London 2017 talk, means applying a UX design focus to your APIs. In this session, Jenny outlines some strategies for making this lateral leap, and suggests some important questions you should be asking.


Thanks for so much. Really appreciate being here. Very excited. We’ve got about 25 minutes in which I am going to teach you some basics of UX for those of you without a UX background as well as how to apply them to APIs. And what’s really going on here is all about API usability, right? I’ve been told in my DevEx role things like, “Oh, well, APIs don’t have an interface, so they don’t need designers.” Or, “Well, we’re hiring developers, and APIs are used by developers, so shouldn’t they be able to build things that are easy to use?” And well, API design as a practice in the UX world is still emerging. I would like in the next couple of minutes to turn all of you into API designers instead. So, we’re gonna go through a little bit more about UX, we’re gonna go through the techniques that you can use in order to apply the UX strategies to API design, and we’ll jump in from there.

If you are curious, by the way, APIs do have an interface. It’s just not visual. That’s what I always try and explain to the stakeholders who say, “Oh, they don’t need design.” It just happens to be that it’s a code-based interface, but people are still asking you for information, you’re still having to give information back. And as for the second thing about developers knowing how to build great and easy-to-use things for developers, you guys are all here, so I hope you’ve all been through that experience, and know that it’s still really good to get outside of your own room, outside of the conference room in the office in order to see how other people are looking at your stuff because the more you start at the same set of code overtime, the more you think it’s totally obvious and easy to use, and the less that’s ever true.

Putting the focus on usability

So, API Usability has become the differentiator for many API programs, and many developer experience programs because there is no longer the case where you are going to be the only program out there doing what you do. There’s always going to be somebody else offering a very similar product to you, and that thing that’s going to set you apart from them is going to be really easy to use APIs. Things where developers don’t need to bother reading the docs because it’s just that clear. And so, how do you get there? UX design.

UX designers have been doing this for a long time. I approached my job as a product manager for developer experience, with a very strong user experience background, right? And the framework we’re gonna work through today has these four steps, right? It goes discovery, the one end which is the lowest fidelity, and then all the way to usability where you’re going to be building something that looks a whole heck of a lot exactly like the API except that it isn’t actually functioning code. At no point during the UX process do you actually need to code a single line. You don’t need a backend, you don’t need anything, pretty much except for a computer, and a piece of paper.

Think about what you’re building

Discovery is all about building something useful. Are you actually making something with your API that people want to use? And the first question you should ask is why does it need to be an API, right? It could very well be that it actually should be a chatbot, or a visual UI, or a helpline that people call, right? And API is a very specific thing. It’s filling a specific need. Use your discovery phase to figure out why the API is the right solution, and what actually needs to be a part of it. Taxonomy is all about navigation, right? So, if you think about, like, the navigation bar on a website or the menu within an app, that’s where the taxonomy really comes into play. What do you call things? How do you organise it? Your API is going to have the same navigation clues in it, and it’s just a very different way of phrasing things. Then there’s mocking and prototyping. So, this is where you’re starting to actually build stuff out. This is where you’re trying out different ideas. You’re saying, “Oh, well, wouldn’t it be cool if we had everything in this way or that way?” And you can try out a whole bunch of different solutions very quickly, do things rapidly, change your mind every 10 minutes, and it’s not a big deal because you’re not having to rip out any code. And then, finally, once you think you know you’ve got the right direction, you’re going to usability, which is those final details, right? So, in a visual interface, mocking and prototyping will be like wire-framing, usability is then like do we need this button to be a little bit bigger so that people can click on it more easily or is it good the way it is?

So, now that you are all experts in UX theory, we’re going to jump into the API part, and how to take this, and translate it from that world of visual UIs, and other interfaces, and into API specifically. So, discovery, right? Why does this need to be an API? What needs are you filling? What’s important to remember here is a couple of different things, right? Collaborate with your end users to develop your use cases. So, don’t just talk to developers and think, “Oh, we know what developers want, so we know exactly the API we need to build.” Right? Go talk to the people who the developers are building something for because they’ve always got a customer, too. If you understand that customer’s needs, you are going to be able to meet your developer’s needs so much better. If you expect that your developers know everything there is, if your direct customers know everything there is about this API, and what’s necessary, unless it’s a very technical backend thing, you’re probably gonna be missing one of the major points. So, get out there and actually talk to the end users, and think a lot about information structures, right? APIs are exchange of information. You ask me one question, I give you an answer.

What is the information that needs to be included at the broadest level that’s going to actually satisfy the need for your customers? So, these are sort of…these are the two really big things to keep in mind for discovery. With that, techniques that you can use, interviews, site visits, ethnography. It’s all gonna be very qualitative. It’s all going to be conversations with people. It’s going to be discovery in a very loose sense. You’re sort of… You’re going on a safari to try and see what’s out there, and you don’t actually know what you’re going to discover quite yet. So, we’re gonna… For each part of the UX framework, we’re going to go through sort of the basic idea of what it is. We’ll do a little bit of more, like, what you’re trying to learn or a little example, and then go through that into an actual example of a real instance of it.

“Observation is greater than conversation”

So, for discovery, we’ve got three math equations to remember. The first is observation is greater than conversation, right? It’s all nice and well to have somebody tell you what they want, but if you can actually see it in action, that’s better. It’s also very easy to have a conversation face-to-face with somebody where you think you’re talking about the same thing. If you can then draw it out, make it visual, you will often discover that there were things that you thought you were on the same page about that you’re not. So, observation, visual better than conversation. The second thing, developers plus end users, right? Make sure you’re talking to both groups of people. And then, finally, a lot of people say, “Well, when do I know that I’ve done enough discovery? When do I get to move on to the next phase?” And the answer to that is, to really look for that point of marginal return, right? So, you look for the point where you’ve talked to…the next person you talked to you, and you’re like, “I learned a lot less in this conversation than I learned in the last one.” Then you talk to another person and you’re like, “I learned almost nothing.” If you ever say that, the answer is you’re done with discovery, you’ve done a great job, and you should probably be moving on to the next phase. So, what should this look like, right? Get everybody around the table, be visual, make sure that you’re all working on the same thing, have these conversations about what people are looking for, really talk to them, try and understand what’s happening, and if you do this through your discovery phase, hopefully then, you know why you’re building the API, and what success looks like once you have built it.


For taxonomy, this is back to that idea of navigation, right? So, you really wanna be focusing here on methods, on method naming, resource naming, right? What do you call things, how do you structure it, what’s the hierarchies? So, one of the important things to keep in mind here is personas, right? You may have in the discovery phase learned that you’ve got two different kinds of people who are trying to achieve two different things with your API, and that is totally okay, but what you need to do is not then try and give both of them a one-size-fits-all experience. And taxonomy is really where you start to hammer out how to separate these lines apart. And so, what I really like to do with taxonomy is things like card sorts and tree testing, and just to sort of give you an example of what’s the kind of thing you’re trying to figure out, right? What’s the kind of challenge that really needs a taxonomy approach to it? This is something that I was building last year with a team where they had this concept of a user, and to that user, you could have a couple of all sorts of different information, like, what phone they use, how old they are, whether they’ve got a cat or if they’re more of a dog person. There is pretty much a whole variety of different stuff you could attach to this identity as a user, and we sat around, and we were like, “Okay. We’re gonna call them characteristics.” And we did that. And then, we were trying even just around the office like, “So, what characteristics do you need to be able to attach the user?” And somebody go, “What’s a characteristic?” All right. So, we’re like, “Okay, fine. Characteristics, it’s not working.” But attribute, attribute is very clear, right? We came up with a whole bunch of different names, and the answer was we could walk around the office, and try these out as many times as we wanted to, and it wasn’t going to make it clear whether we were actually designing something useful that a developer who wasn’t totally familiar with our API can understand.

So, with taxonomy, index cards are the best thing you can possibly use. Take an index card, and write down each of these words, and pass them off to the developer, and say, you know, “What does this word mean to you?” And see if they can get to that concept, right? And sometimes, with like these characteristics, attributes, whatever it was, it turned out there was not one ideal word that in everybody’s mind could bring up the same thing because it was so abstract, but tags made right, and it depends also on the system you’re using. But the point is that you get it into somebody else’s hands. What I was doing with Ben here in this particular photo is I actually wrote down on an index card the verb and the endpoint name for every single method in our API, and I passed it over to him, and then I said, “What does our API do?” I said, “What do each of these endpoints do and what order are you going to call them in?” And if he could not actually sort that out and explain to me the answers to all of those questions with just the index cards, I went back. I would take another card, I’d write something else, and I’d pass it over. “Okay. How about this one?” You can iterate so quickly with index cards, you can get things to names where people understand it so well, and it really ends up being a powerful way to structure you APIs, so that you’re making sure people know how to get around it, so that they don’t read the documentation.

Mocking and prototyping

The third thing, mocking and prototyping, right? So, this is like wire frames. This is where you say, “Okay, we’ve got sort of the basic structure, we know what the API is going to do, we know sort of how we’re going to organise things, and now we need to get into a space where we can handle what the request bodies are gonna be, what the responses are going to be, what units we’re using.” The goal is to really get a basic outline of each endpoint, right? So, with your mocking and prototyping, you’re not going to necessarily know exactly what scope names you’re gonna use, or what owe of, but you’re going to know in this method, we need to include this information. Over here, we need to include that information. For this, there’s two main techniques I like to use. One is paper prototype which we’re going to jump into in a second, the other is instant messenger APIs. And this one is really fun because what you do is you stick your developer in a room with a moderator and a computer, and you put some sort of instant messenger tool, you know, Slack work is great for this or whatever else you want. And you give them that stack of index cards, and you say here are the APIs, you know, with the methods and all of the resource names, can you piece it together however it makes sense to you, and start just making some calls, and see what the API returns. And what you actually do is in the other room, you have a developer sitting at keyboard, and every time you’re test subject types in an endpoint, and types in some sort of call, then the other person has to type a response, right? So, you’re actually and essentially, verballing this API back and forth. What happens though is that you’re forcing your test subject to express what his expectations are, what her expectations are of each endpoint, right? And she will say something like, “Well, why did you return this,” right? Or, “What’s going on here?”

So, for instance, I love to hate on Weather Undergrad. It’s both a brilliant API, and yet, at the same time, it drives me absolutely mad. And I’ve just pulled up the GET forecast. This is for London, I guess. And, this is a little bit modified to fit on the screen. But in terms of, you know, if you’re doing this kind of mock and prototype test, what the information you’re looking for is like, okay. Here’s this response and my first thing is, well, isn’t the title of Wednesday night the most important piece of information? So, I would actually like that first because that’s gonna be the first thing I display in a forecast, and that’s just how I think. And then, precipitation is zero. So, does that mean is it percent, is it inches, is it centimeters? Like, what does that actually mean, right? I don’t know. And finally, there’s FCT text which I figured out is forecast text, and then it’s just forecast text, but then, it’s forecast text metric. And I was wondering, you know, why is it that metric is called out as metric, but imperial is not called out as imperial? These are the sorts of things you discover through mocking and prototyping.

The other great strategy to use, at this point, where you’re trying to get answers to these sorts of questions is paper prototypes. Ugly is the best way to do this, and this is actually something I was using a while back. You type out your APIs on sheets of paper, you print them out, and you pass them over to somebody, and you say, “Would you edit this?” Why do you need to make it ugly? Because people are willing to tell you the truth if they don’t think that you’ve put a lot of time into it. And the good news is this doesn’t take a lot of time. You could type up most of an API in less than an hour, and hand it to somebody, and get edits back. And then you can go and build it, and it works so much better than trying to build something, and get the edits afterwards. Paper prototypes are great. You can have somebody write on them, or if you wanna reuse them, you pass somebody post it notes, you have them put posted notes on it, you take photos, then you take the posted notes off, and you can pass it to somebody else, or you can print out multiple copies if you don’t like the environment. Whatever the case maybe, it’s a really great strategy to get feedback, to get clarity. You can also…This is another example where you can try 10 different things out, right? So, you use the instant messenger strategy when you want to understand what’s in somebody’s head, and what mental models they’re using. You use this if you’ve got an idea of what model you want to use or you’ve got… you’re trying to decide between two or three, and you want to get some information from people about whether this make sense to them or not.


And, finally, usability, right? So, this is now where you’ve gone through all of these other stuff and you’re like, “Okay. I think I really know what needs to happen, I know how to design this API, but I need to make sure like when I return something, is it very clear, what it does, can people handle everything, what kind of OS do I need, what kind of headers do I need, right?” This is where you go into that level of detail. You use standard UX usability testing strategies at this point, and the moderator’s guide is really important. For this, the best thing you can do Postman Swagger pick your tool of choice, stub out the entire thing. Type it all up, have it there in a documentation format, and use that. But again, still, no backend because you’re still going to be making a lot of changes and learning things at this phase. So, an example of moderator guides. What you’re gonna do is you’re gonna have a task that your developer needs to do, and then you’re going to say whether or not they’ve succeeded at doing that task, right? So, if you’re doing a payments API, charge a credit card for $12.50, apply a discount, and create a pricing schema. And during the process of this usability test, they’re either going to succeed on each of these things or they’re going to fail. If they fail, that means that you need to go back and iterate once more. At this point, maybe, is where you start to add in a little bit of documentation, right? You might have learned in some of the previous phases, like with that user attribute example, that you just need to explain what something is, that there is no way to have it exist without a bit of explanation. So, insert that here, but try and keep it minimal because you’re still wanting to make sure that things are usable without docs.

And another trick for this one is if you put the developer in a conference room, put her at the head of the table with her laptop in front of her, and the screen right behind her because then as you’re using the camera of the laptop to record the session, you get both the face, you get the screen. Also then, as you’re doing the interview itself, you can sit across the table, so you have the face-to-face interaction, but you still know what’s going on on the screen. It works really well to help make sure that you’ve got that communication going throughout the entire test. So, testing is complete. Great. You’ve got this wonderful, beautiful API. Now, you can actually go build the backend.

This whole cycle can take one to two weeks if you do it quickly, if you dedicate time to it. It’s really not a lot of investment to put into your product ahead of time, to help you succeed later on. But I’m sure we all wish that we were lucky enough to be able to design every single API from scratch, right? And that every single team we worked with has built their API with all of this usability in mind, and that it’s perfect. And I’m sure that you guys, just like me, have had to deal in situations where that has not been the case. Where you’ve been handed an API, and despite the fact that you work at the company, that you have called it before, that you know all about it, you still are completely confused by the product and you’re sort of stuck. I wanna tell you, don’t despair. There is still a way out.

So, the first thing is that all of these same techniques can be used on an API that’s already built, right? The only difference is that you’re going to have to be identifying then what you’re vision is going to be for the API, right? You might not be able to have it at perfect usability today, but perhaps, an iteration or two down the line, you can. And that’s why you need to create a versioning strategy. You need to think really hard, what are the things that are heinous, that really need to get out now, that need to be fixed, and what are the things we can live with? Don’t think that because the API is already built, you’re out of luck.

Creative fixes

And there’s also all sorts of creative fixes. So, this is an example, a slight modification on an API that I had to deal with at one point in time for my team. It was GET reports, and the first thing you had to supply was the type. So, they asked. So, in the body of the GET, you had to add…You had to supply whether it was a periodic, or a summary, or a… There are all sorts of different report types, and depending on which report type you are asking for, you then had to provide different information. The only thing that was consistent about all of them was that you have to provide both the start and the end date, but otherwise, it was sort of this hodgepodge mad libs strategy. And so, this wasn’t working very well for us. We couldn’t do it, our users couldn’t do it. But at the same time, we had helped a couple of them execute on it, and we didn’t want any breaking changes for this issue. And so, what we did is we actually used our docs to hide the issues with this API. So, the first thing we did is we set up an abstraction layer, right? So, we set up GET reports/periodic. And then we made the docs say, for this endpoint, putting the start date, and the end date, and the interval length, and this, and that, and the other, and supply this information, and we’ll return the report. What we did on the backend is when they called this endpoint, we were actually calling the previous endpoint that I just mentioned, but inserting the type periodic in there for them. So, all of a sudden, our documentation looked like it was actually the right information at the right time, and it was, if not, well-designed, at least, okay designed as opposed to just completely ridiculous. It hid things front the customer, no breaking changes, the old endpoint was still up, so anybody who had the old system could still use it. And this was one of the things we said, we’re gonna make this compromise, in terms of getting to a useable strategy.

So, if you do have stuff that’s already built, be creative, find fixes, but the only way you can even figure out what it is that needs to be fixed is if you go through this sort of testing strategy. So, my philosophy, users first. I’ll be around during the breaks. Would love to talk to you then. You can also tweet questions at me. Thank you, guys, so much, and I hope that all of your APIs are forevermore usable.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.