Made by
Made by
Andrew Baker
Developer training often follows a familiar model, but with TwilioQuest, Twilio’s Developer Education team have tried a very different approach. In this talk from DevRelCon San Francisco 2019, Andrew Baker talks through the history of the project and shares what’s coming next.
Really excited to be here today to share with you some of what we’ve learned building TwilioQuest and the ideas we’re thinking about next. So, first, a little bit more about me, my name is Andrew Baker, I’m our director of developer education at Twilio. Started my career doing Python web development around Washington, D.C. where I grew up, and then I’ve been at Twilio about four years now on our developer education team.
And this photo on the right here is a photo that we took with a drone after the first training event that my team put on, after I started leading the team two and a half years ago. And it can actually be a little tough to spot me in this one, so my co-worker, Kyle, made a version of it that’s easier for you.
I was super stoked that day, and I can kind of trace my path to dev rel, all the way back to 2014 when Docker was getting really hot. I just started teaching developers around me in D.C. about it, because I learned about it myself, and saw how voracious the appetite was to learn about Docker there.
And that’s actually how I met Carissa Karcher, who’s here, who was working at Docker at the time, and I was like, “Can you send me Docker swag for these events I’m putting on?” And that was, kind of, my path to dev rel.
Serving developers, teaching them skills that can be career-changing. And for folks who are new to our industry, you know, we know how life-changing working in tech can be. It has been really, really fulfilling for me, and these past four years have been incredible.
So, our agenda today, what is TwilioQuest? We’re going to talk about exactly what we built and why we built it. And then, the version we have out there today has been there for about a year and a half, so I want to share some data with you all that we’ve collected, and do some reflection on what we think is working and not working, and then I’m going to talk about where we’re headed next.
But before we get into TwilioQuest, it probably bears mentioning what is Twilio. Twilio makes it easy for developers to put communications in their apps, and the company has been around for about eleven years now. Most folks know us for a phone call and text message APIs, but as I know from our work on the documentation oh-too-well, the product portfolio has really expanded in the past few years.
So, these days with Twilio, you’ve got APIs you can use for video, email, 2FA, IoT, and then one product that we launched at Signal last October, which we call Autopilot, our conversational bot machine learning framework. That’s my favorite new product from Twilio, and I’m pretty sure that’s the case for Lizzie too, one of our evangelists who’s going to be on the stage here later this afternoon.
And the next question you might ask is, “What does developer education mean?” It was a new idea to me when I joined this team in 2015. So, kind of, the short history lesson there is, about five years ago, Twilio decided to found a team focused on documentation and education. So, two of our evangelists spun off to create that team, Kevin Whinnery and Jarod Reyes, and then I was the first outside hire for that team in 2015.
And I want to take a moment to acknowledge that there have been some kind words said about our documentation at this conference this week. We definitely don’t consider them perfect, I know we have a long ways to go before we’re serving every developer who comes to Twilio.
But if you want to learn more about our philosophy on documentation, Kat King, who leads our documentation program, gave a really great talk about our philosophy at Write the Docs last year. So, if you go on YouTube and just search for “Empathy driven developer documentation,” you’ll find Kat’s talk, and it’s a really great one.
But our path to our training program actually started a little bit earlier. So, winding the clock all the way back to 2013, Twilio puts on a developer conference every year, and the evangelism team decided that they wanted to offer some training as part of the conference. And when they sat down to think about what that training experience might look like, they probably thought about something like this, and certainly, that’s what I thought about too when I started putting together trainings for Twilio.
Based on my own personal experience, usually you show up in some hotel conference center, there’s a speaker at the front of the room, they plug in their laptop, go through, like, 15 minutes of slides, then turn everybody loose on a code challenge, then 15 minutes later they say, “Okay, is everybody ready to move on?” Maybe, like, if they’re lucky, half the room raises their hands, and then they just repeat that process, slides, code, slides, code for the next four hours.
And if you’re like me, when early in my career I attended a ton of workshops like this, you rarely walked away feeling satisfied. It was either way too fast and you knew you were over your head in the first 20 minutes, or it felt way too slow and you walked away feeling like you’d wasted four hours.
So, that’s what led us to the first tenet of our training philosophy at Twilio, is we are all in on the concept of self-directed and hands-on training. And there may be reasons why you’d want to choose a different, you know, technique for your own subject matter, but for us, it seemed essential.
Twilio wants its audience to be as broad as possible. We want to serve developers at all levels of experience and with all kinds of backgrounds in the programming languages that they use.
And increasingly, when developers come to our training events, they’re interested in learning about more than just voice and SMS. The only way to provide a successful training experience for all those folks is to make it self-directed and hands-on.
But if you’re going to go down that path, you could imagine just creating, like, a table of contents for the docs. Or some kind of glorified e-book, and spending four hours looking at that doesn’t sound too fun.
So, that’s when the team started looking for other ideas that could make this sort of training work, and how we landed on video game concepts. So, the idea of using points to recognize players’ progress, like, leveling up as a way to celebrate what they’ve learned, and to motivate them to keep going further.
These were the ideas that were, kind of, the second founding tenet of our training program, which is to layer in RPG video game mechanics, so role-playing game video game mechanics, to propel players to learn more and celebrate their success. And when you put those two together, that’s where you get TwilioQuest. So, back in 2013 at that Twilio Con, the evangelism team put on the first-ever TwilioQuest event. Kevin Whinnery from the evangelism team led creating this experience.
And from what I understand, this was before I joined Twilio, it was definitely a success, but the evangelism team after the show got back to evangelizing. You know, training was not a core part of their mission. So, the idea of TwilioQuest actually, kind of, just gathered dust for a couple of years, until 2017, when the education team would pick it up again.
So, what I’d like to do next is, kind of, walk you through the core experience of TwilioQuest as we’ve built it today, explain a few of our decisions, and then we’ll start reflecting on what’s working and what’s not.
So, when you first sign into TwilioQuest, there’s actually step zero, which is signing up for a Twilio account. In today’s version of the experience, you have to have a Twillio account to get into TwilioQuest, and that’s how we keep track of your identity.
Once you’re in TwilioQuest, you’re setting up your profile. So, you’re putting your name in for your avatar, customizing your appearance, and if you’re playing TwilioQuest at one of our live training events, this is where you select which event you’re at, so that on the leaderboard we have, at the front of the room, your name and your avatar pop up every time you complete an exercise.
After that, we drop you in this mission that we call “Getting started.” This is a collection of six exercises which teach you the fundamentals of using Twilio. And once you finish those, then you land on our main mission screen where the world is your oyster.
And as you can imagine, most folks are reaching for those SMS and voice missions, but like I said, increasingly, and especially folks who’ve been using Twilio for a long time, the reason why they’re showing up to our training events is to learn about all the other stuff that we have to offer.
Once you pick one of those missions, you’re going to see about 8 to 12 individual exercises, which walk you through the basics of using that API, all the way up to more advanced concepts.
And to give you an example of what one of these exercises looks like, it’s kind of like this. On the left side, you see a description of the exercise, on the right side, those victory conditions, those are, kind of, like, completion criteria for us. And as we scroll down the page, you’ll see there’s actually not a ton of content in the middle.
One of the decisions we made with TwilioQuest 2.0, which is what we call this version, is that we thought we wanted to teach developers how to fish. We wanted to show them how to find the answers they needed in our documentation and with Twilio’s other developer tools, as opposed to giving them all the information inside TwilioQuest. That’s an assumption we’ll check in on it a little bit.
The last text box you see here at the bottom is how our self verification logic works. So, it’s really important to us to provide an experience that was a little smarter than just a typical quiz.
So, in this particular exercise, we’re asking you to acquire your first Twilio phone number. And so, when you paste in that phone number and hit submit, we use an internal API within Twilio to verify, yes, that phone number is a Twilio phone number, and, yes, it’s associated with this account, so the player completed the exercise.
And that’s the version of TwilioQuest that we’ve had live for about a year and a half. So, I’d like to tell you some data points that we’ve collected along the way, and start that reflection.
The first time that we deployed this version of TwilioQuest was at something we called Superclass. So, Superclass was the training event that the education team put on in May 2017 for Twilio’s annual conference, and it was a really, really fun day.
TwilioQuest was the heart of it, hands-on training was the heart of the event, but it was not the only portion of the event. We also had a talk track, which had one-on-one style talks about Twilio products, and we had this feature that we called “Office Hours.”
So, if you were an experienced Twilio developer, you just wanted to, like, corner an engineer and ask them your toughest questions, we had a way for you to sit down with them for 20 minutes and get that time.
And this was a super, super fun and memorable day, and Gene Chorba is in the audience somewhere. Gene was actually the top point-getter at this first inaugural Superclass, and will forever hold the title of inaugural Superclass champion. I’m pretty sure I was the one who got to put the crown and the cape on Gene at the end of the day, which was definitely a career highlight for me.
So, this event also looked good in the numbers, we had about 120 students there, and our NPS, or Net Promoter Score, for the event clocked in around plus 64, which was a score that we were really happy with for a live training day.
But one limitation with that version of the platform is that you had to flag down a TA every time you wanted to mark your progress in TwilioQuest. So, for the rest of 2017, we invested in the engineering work to make that self-verification logic, and then launched TwilioQuest online to the world in October 2017.
And we were pretty happy with the reception here too, and one of the most astonishing parts of it was, even here on the Orange website, it is one of the harshest places in tech, but we actually had a really constructive comment thread that day. And I share that with you not just because it made the team feel good, but because I think there’s actually a really interesting dev rel story here.
So, when you look at the top comment on this page, it’s from this guy named Leon, and he’s describing how he came to the conference in May with the intention of coming to a whole bunch of talks and doing a whole bunch of networking, but as soon as he found TwilioQuest, he actually just spent the whole time hacking on TwilioQuest and leveling up his own skills.
Leon was one of the first comments on this post, he remained the top comment on that post as long as we were, you know, seeing activity on Hacker News, and I think it’s a really great dev rel story, because sometimes we talk about the difference and the tension between work offline and work online.
When you’re choosing how to spend your time in dev rel, do you do a lot of events, meet a lot of people one-on-one, or do you try and engage in online communities and ship along content?
This was a really cool example of where we’d spent some really good time with Leon, providing a really great experience for him at our annual conference, and he ended up being the person who set the tone in the comment thread on my team’s most important online evangelism day of the year.
So, what is working about the current version of TwilioQuest? Something which is definitely a big worked is its role supporting our live training program.
So, after that initial one in May 2017, we decided to take Superclass on the road. In 2018, we did about 15 smaller Superclass events all around the world, generally with about 30 to 60 developers in each location. This clip is from our event in New York City in that fall, and since then we’ve reached a whole bunch more. So, about 1,200 students around the world over the past year and a half, and we’ve maintained that average NPS of plus 64.
So, TwilioQuest is definitely working for our live training program. It’s also been a great impact on Twilio’s onboarding experience. So, later last year, we decided to do an analysis, where we compared the onboarding stats for accounts who would sign up for Twilio and played TwilioQuest, and those accounts that had signed up and didn’t play TwilioQuest.
And we found some really great stats, where those who played TwilioQuest were 10% more likely to achieve 10 API calls, and that’s, kind of, our threshold for, if you sign up for Twilio and you make 10 API calls, then you’ve probably onboarded to Twilio, and we’re confident that we succeeded in onboarding you.
And on top of that, depending on which product the player was working with, they achieved that milestone 15% to 25% faster.
The last way that TwilioQuest has been working for us is just as a really novel and fascinating addition to our brand. So, these are what we call Dev boxes, kind of, really nice gift boxes that we send out to our favorite developers and our community over the past year and a half.
You can see it’s got some socks in there, there’s a box of Cheerios, which we rebranded to Twilios. Underneath that there’s, like, a TwilioQuest coloring book and a messenger bag. This particular tweet is from Carter Rabasa, who’s a former Twilio evangelist, but we’ve seen these boxes be a really really great way to reward the most engaged members of our community.
But even at a smaller scale, for swag, we ran a promotion in 2018 that’s still live today, where, if you complete that getting started mission in TwilioQuest will give you a form to order yourself one of these blue TwilioQuest T-shirts.
And this remains the only piece of TwilioQuest swag which can only be earned, even employees don’t get this one unless they do the same work as everybody else, and I will hold the line on that and make it the hill I die on.
A few other ways that it’s been useful is for our evangelists. So, Layla, one of our evangelists in the UK, has started hosting these events called “TwilioQuest and tea.” Basically, a tea party where they also hack on TwilioQuest. And you can see it looks like a super classy affair, there’s like a really nice custom cake on there. I am super bummed that I’ve not been able to attend one yet.
And it’s also been useful for our evangelists as they go to third party events. So, Margaret Staples, who sent this tweet, is the evangelist who’s done the most work here, but this particular one is talking about Sam Agnew and Brent Schooley, two of our other evangelists, who took TwilioQuest to try! Swift in Tokyo, and basically ran a little arcade game with it.
So, if you did some TwilioQuest during the show, came back and showed them that you had 200 or 300 points, you could pick a couple of small prizes off the table.
But it’s not all roses, of course, we’re always trying to challenge ourselves to do better, so what isn’t working out about the current iteration of TwilioQuest?
The first thing is usage. So, this chart represents the average player’s progress through that getting started mission, those first six exercises which teach you Twilio fundamentals.
And you’ll see, probably the first thing you notice on this one, is that there’s a pretty big drop between step one and step two. So, of all the folks who complete our first exercise, there’s much fewer who actually complete the second exercise.
And that first exercise is teaching you how to use TwilioQuest, the second one is where we actually ask you to go into the Twilio console, acquire your first phone number, and actually get started with the service.
After that, there’s definitely some more attrition, but it’s not too bad. And you see the next big dip at the end there, that is where we introduce the tool called ngrok, which is a tool we recommend for developers to manage webhooks in their local development environment.
Any of you who have APIs which use webhooks know that it is just a really tricky concept for developers of all shapes and sizes, we see it in the docs work too, and no surprise that it’s causing problems here in TwilioQuest.
The other way that we haven’t done a great job is integrating TwilioQuest into other parts of Twilio’s developer experience. So, this screenshot here is actually one of the few places you’ll see TwilioQuest outside of twilio.com/quest.
You’ll see this if you sign up for a Twilio account and choose learn and explore as your use case. Otherwise, actually, you really don’t see TwilioQuest anywhere in our console, or in any of the emails that we send you after you sign up.
And that might be part of the reason why, my third point, that we actually haven’t seen that much traffic to TwilioQuest, we haven’t seen as many players come and use it as we would’ve liked. The stat I always like to use here is that our traffic for 2019 in TwilioQuest is only 1.6% of the traffic that comes to our documentation.
So, when I sum up TwilioQuest internally these days and I look at the opportunity ahead of us, I use two GIFs. First, when it comes to TwilioQuest, developers pretty much say they don’t know her, and second, even if they do, they don’t make it too far.
But I think we can say that we validate our initial hypothesis, that self-directed, hands-on training and gamification is effective.
So, where are we headed next? In late 2018, we were able to dedicate a person on the team full-time to working on TwilioQuest, that’s when Kevin Whinnery came back to the team. And he and I decided to focus on two problem areas.
First, we wanted to propel players to learn more, and then we wanted to reach a much bigger audience. And tackling those problems is what led us to TwilioQuest 3.0.
So, in TwilioQuest 3.0, you will assume the role of the operator, and your mission is to protect the free people of the cloud from a tyrannic Kabal known as the Legacy Systems.
But fear not, adventurer, for you are not alone on this quest, you’re going to have a whole bunch of characters to help you out, not least of all Cedric, your robotic companion and guide to the cloud. And you’re also going to do a lot of exploration along the way.
So, your home base is the Fog Owl, a ship based in the cloud, but you’re going to go and explore all sorts of new worlds as you complete those coding challenges and expand your skills.
So, you can see in TwilioQuest 3.0, we’re wrapping the concept in more of a story, we’re layering in more of those videogame concepts, adding in characters, and we’re hoping that’s going to really propel players to keep going in a way that the current experience doesn’t.
And if you were looking closely on that last screen, you saw this. So, we’re not just adding new videogame concepts to TwilioQuest, but we also want to level up the learning experience too.
And I was in one of the bird-of-a-feather discussions yesterday morning, the concept of development environments came up, how much they can derail any workshop or hackathon. We wanted to come up with a really satisfying solution to that problem, which is why we’re bundling an IDE in the next version of TwilioQuest.
So, if you want to work with JavaScript, you’ll be able to do that without any additional environment set up. And it’s this feature that actually led us to the key tech stack decisions in the new version of TwilioQuest. We’re still using web technologies, but we’re actually going to a downloadable client with Electron. Most of the UI you saw in those clips was built with React, and then Phaser JS is what we use for the game world. And I’m really excited to see how these tools work out for us.
When I joined Twilio, especially my experience with Docker before, I was sure I was going to build cloud-based development environments for our audience. I think we were all surprised at this path to go down for a downloadable client, but it makes it possible for us to create much more complex exercises and much more complex validation scenarios, because we have full access to the player’s development environment. We can just peek at their code instead of trying to black box it through their API usage.
So, a couple of quick stats on TwilioQuest 3.0, we’ve got about 70 players in early access today, and the thing we’re really excited about is that, of those players, they complete, on average, 90% of the available content in the new version of TwilioQuest.
Now, I totally expect that number is going to go down as we start opening up the platform and bringing on more folks, but it’s really, really exciting to see that, at least, it seems like we can be reasonably confident the new version of TwilioQuest is going to make an impact on that anemic funnel that you saw for TwilioQuest 2.0.
But there’s definitely some risks. A rich, powerful, downloadable experience doesn’t work if the developer doesn’t click download. So, we’re going to have to provide some pretty convincing argument and make our case to developers who come and look at Twilio for why they should click “Download,” and wait for that 200-megabyte electron client to show up.
The other thing, of course, is that most developers, you know, overlap in a healthy way with the Venn diagram of people who enjoy video games, but some don’t. And for the developers who aren’t into video games, are they going to be turned off by the fact that this experience looks a lot more like a game than it does a traditional training platform?
So, our rough timeline, we’re here in June for early access, this is where we’re inviting people to the platform one at a time. In July, we’re going to go live with our invite system. So, every player who’s in the early access program is going to get five invitations that they can share with friends and co-workers. In August, we’re launching at Signal, Twilio’s annual developer conference, that’s where we’re going to go worldwide.
And then in Q4, we’re going to get cracking on Chapter 2. So, lots more technical content inside TwilioQuest, and the next chapter of the operator’s adventure. And this is where I’d love to invite you all to join us. There is so much experience in this room, equipping and inspiring developers, and we would love your feedback on the next version of TwilioQuest as we prepare it for the world.
So, this is a DevRelCon exclusive. If you’d like to join the early access program for TwilioQuest and help us refine it and give us some feedback, I would love for you to participate, just go to twil.io/devrelcon, drop your email in that form, and then I will email you your access code in a week or two, and here’s my contact info too. Thank you, folks.