Developers love webhooks, you should too

Send some data in, get some data out, this is how most developers interact with API nowadays. But as we are making progress to build more API driven products, we tend to forget about the simplest interaction: webhooks.

In this talk, Typeform’s Nicolas Grenie makes the case that webhooks should be a central piece of an API strategy. He makes recommendations to deliver the best webhook experience inspired by services that are doing it right, both from a technical perspective and from a DX (developer experience) point of view.

Transcript

Nicolas: Hello everybody, pretty excited to be here in London, all the way on the other side of the pond. I live in San Francisco, but I’m French, so it’s the other side of two ponds for me. Today we’re here to talk about webhook, but I know we are waiting also for the food, you’re like the last talk before the food, so I will show you some food to get your attention right now. And hopefully you will stick with me for the content that’s coming after.

As Elise mentioned, I work for Typeform. We are, what we decided to call ourselves now, it’s an interface company. We are mainly known for our forms. So one-question-at-a-time forms, making them interactive, making them look cool. If you want to try a live form you should stop by the Hoopy Booth, they have a developer relationship survey, and they used Typeform for it. So you get a chance to see how it looks like. But we also just launched VideoAsk, which is a video interaction app, so I really encourage you to give it a try. Especially in the community space, it’s a good way to get feedback on the stuff you’re working on.

We’re here today to talk about webhooks, with those lovely characters. And you don’t know yet, maybe, but webhooks are actually a natural thing. I think they can be inspired by spiders, and what the spiders are doing when they’re trying to eat. If I understood well how it works, they make a web and just wait for someone to get caught on the web, usually it’s a bug. And then when the bug gets catch, they just receive a notification, on the web, directly, like do-doo to-doo. There is someone they get caught, you should have “the dinner is served”, almost. So there is a simple notification of when it happens, the webhooks, go get your lunch or dinner. You do that also when you’re a student. You have your own webhook interface, that’s called a microwave. You put the food inside a microwave, and when it gets ready you have a ding notification that something has happened. So, we are kind of surrounded by webhook-like interfaces. You should be familiar with this mechanism. We can all agree on something, on what webhook means in this specific use case. The idea is – something happens on a platform, usually it’s your platform, and you’ll get notified through a HTTP request. Do we all agree on this? Awesome.

I’ll tell you what’s our challenge at Typeform. For us, we collect a lot of data, but people don’t really consume the data on our platform. The data moves away. We’re usually the first thing that people use to collect leads, to ask for feedback, but we’re not where they consume and analyze this data. So, it goes away. The main interaction for those things to go away is through a lot of no-code solutions, a lot of workflow apps, where this is an example of someone doing sentiment analysis. You receive a lot of feedback, you want to do the sentiment analysis of if people like your product or not. You connect that to an IBM Watson, you connect that to AWS similar APIs or I’m sure Google has something like this and then it gets stored at somewhere else, so, we’re in the phase of workflows and we fit into those, those workflows.

To empower those workflows, we use all the webhooks, we started by having webhook first, we didn’t have APIs. It’s only really recently that we started to have APIs, but thanks to those webhooks, we managed to keep our users active and come up with a lot of use cases on how people can connect to Typeform’s. If we didn’t have those webhooks, I don’t think we were be able to get the same groups and I don’t think that we’ll be able to have the same fan base kind of, people that were hacking around. So, if you’re wondering what you should build now if you’re a company you’re like, Oh, should we do API first or we should do webhook, I really encourage you to go webhook first. Specifically, now that there is like this whole no-code movement and if you want to be part of it, if you want to be an actor of it, if you want to be involved in it, you have to get webhooks to be integrated with Zapier, Parabola, all the other codes like this.

Also the thing that if we look at for the developers, for them to, it’s pretty easy to get started. They almost don’t have to get a buy in from any of the DevOps team. It’s as simple as doing a Lambda function or a Google Cloud function or a Glitch project. It just an end point somewhere. They will receive data and do some actions on it. They can get started very quickly, they get to see the first result really quickly, and they will adopt your product in a more faster way. You have a lot of bunch of reasons to use webhooks. As I said, try to use the webhooks and implement those webhooks early beginning because you hope, I hope you see some results.

Now that you’ve decided to use the webhooks and you decided to make your product webhook friendly. Let’s look at what would be the experience for developers. I will tell you what we did first. When I jumped I found about a year and a half ago, we decided to be consistent because for some reason, we were not sending the same data in the right order all the time. If you fill a form once with your name last name and all the things and you fill this and form of second time, the webhook events were totally different and totally mixed up.

The first thing we did, we said we’ll be consistent. So, you have the same event in the same order all the single time. It’s pretty easy to get started. Then we gave the power to our developers and that was fully one of the first things we implemented to test webhooks. Especially for something like our step, is connected to other workflows, people might not want to put real data at the beginning. They wanted to get a try of what would be the payload and what would be what I get to see and whether we’ll be dealing with, if I have real data, just send me a test and I will see what I can do with this, and I will eventually do some manipulation around this. If you are able to do this in our case, just mimic the structure of your form and we send you random data.

The second thing we implemented, and we get inspired by other companies in the industry (mainly GitHub around that, they really had that for a long time), we give you an interface to inspect both sides, what we sent you, the request and what you sent us back. So, like this you have very clear feedback of how did this event go and you can look at what we sent you and you can tell us if you went well or not. So, we also added some visual stuff. My example was speed about it because we were saying is green, but in case it’s not green, we’ll tell you it will give you a red and we’ll give you also the response codes that you sent us. So, when there’s something happening, you can also look into this. We also give you a trace, like go back into the history so you can see where you need to feel the issues and if there was any issues with this.

Something that we also got from GitHub, and we got inspired to use this is redelivering the webhook. When you deal with a lot of events, when you deal with real data and especially long forms like us that can be complicated, it’s painful to go one by one filling all the single fields and once you’ve done this, if you have to do that every single time you make changes to your code, it’s even more cumbersome. We have a simple redeliver button where we just send the same events and you have the same payload. Like this, you don’t mess up with your real data either.

I will say that if you implement webhooks, you should be really careful about your retry policy and be really straightforward to your users about this. If the delivery doesn’t work, how many times are you going to try to send it again? How many times will you try to be up in that time? But, for us we tried to do a logarithmic version of it. A number of times after the first fail, we spaced it out. That’s how you decide to implement it later on, not in your backend. You can decide that if a webhook has failed so many times you can turn it off so we won’t send any events until they turn it back on on their own. That’s something you do. You just say if you do it, we don’t do it for now. These are examples from Shopify and Stripe. I encourage you to have that specificly in your documentation as well.

Something that I really like, also and that’s something I’ve seen from Nylas. Nylas is an API that integrates with your emails. They send you an email notification when your Webhook is not feeling really good. So, as you’re working on your webhook integration with them, you’re doing a lot of failures is you try an errors, they don’t send you anything, but one is a production level when they estimate that, “okay, you’re done” and is receiving real traffic, if it’s failing, they will send you an email saying “Oh by the way, this whole thing over there is not working. Probably you should pay attention to it. There’s something needs to be fixed” So, I’ll say that’s good attention to have. It’s probably something harder to integrate, but you get to build a feedback loop with these users and let them know when something wrong is happening.

Something that has become a standard that has been inspired by Slack, GitHub and others, is signing your webhook payload. We started to do that Typeform also, the idea is for a user to identify that the request that’s coming on their webhook and point is legitimate it comes from you as a provider. So when, because everybody can discover a URL and hit it and eventually try to harm your system. With this in place, you can make sure that the API call comes from Typeform. This is the secret that you have put in place. You decrypt it and have authenticity on this. It’s a bit harder to integrate for users. We usually recommend when they do trials to not use it, so they go faster and they get to the success quicker. But it’s also a better practice when it goes into production and it’s a challenge for you when you do the documentation. So, you make sure that all your documentations integrate examples in shortcuts, so it’s almost integrated and they don’t have to think about it.

Something that I find interesting also was an interesting example from Twitch. So Twitch, when you add a webhook event and a webhook URL to their app, they do what they call a challenge. They want to verify that this endpoint is a URL exists. So, what do you do? You have your Twitch server on the site and you have your webhook endpoints. They will send you (and I think Slack does this as well) they will send you a first post request and say, “okay, we sent you a challenge”. This is the payload of the challenge. Just answer back and tell us what you got and you just do a response from it and you send like the payload. Pretty simple thing to integrate. They are sure that the URL belongs to the user, that the URL exists, and then they’re not going to send requests in the world. They do that regularly to check the the health of the URL. The main issue here for you, if you implement this, it just like another step in the setup process. So, it can also be complicated for a user and you may lose another option over there.

Something that we did and as we’re really webhook driven that really helped us, we decided to put this whole webhook functionality into an API that’s accessible to our partners. When you use Typeform for Zapier, it actually creates a Webhook on the form for you on your behalf, tagged as a Zapier Webhook. Like this, we can easily get integrated with other types of partners and this is for them very easy to integrate. A lot of integration starts from the webhook point. If we did not have an API it would have been a manual process. Everybody will have to say, “Okay, hit your end points here, copy this end point into your interface”. It’s really, really hard to do and also a way to drop off users. So, instead we have an API, we can automate this process. We added the corresponding OAuth scope. So, anybody that has the permissions to do it can create a webhooks on your behalf. And as well, we added delivery. So, if you want to check the health of your webhook endpoint, and your relationship with Typeform, you can check how many of the deliveries went well and how many didn’t go well.

Something that’s very important also for us on the workflow use case, we added the possibility to add more than one webhook to a form. So, if you have this possibility, I encourage you to do that. For a while our interface was only showing you one webhook, while our API will let you do more. So, we decided to put that directly into the interface because we have a lot of users that say, “Okay, now when someone submits Typeform, I want to add that to Mailchimp, I want to add that to Airtable, and Slack”, all this at the same time. So, the old three different webhooks. So, if you’re able to give that possibility, I will really encourage you to do so, and specifically because you can also have a Dev or a production environment when you can switch on very quickly. Name spacing was interesting in that, in that case. So people can, we don’t restrict that specifically, but people can restrict a name. So, you know that there’s a webhook called Zapier. No one should be scratching this workspace and so you can always retrieve easily if a form is specifically integrated with your app as you have tagged a webhook with your name.

Again, an example from GitHub, something we don’t do yet and this is something I wish we were doing better. If you have built your whole infrastructure around microservices and even driven stuff, should be easier to build, is the possibility to send more events than, in our case just submission. So, if you look at this, those are a couple of examples from from GitHub, they let you subscribe to many, many events that are happening on a repo that’s happening on a user, When they change, when they updated, when it deletes. like this, you’re a bit more reactive and again that’s happening that’s helps you power all workflow apps. Because if we were able to update some one when form was changed and eventually there’s a policy that needs to be changed somewhere, there’s someone that needs to approve it. If the form was changed, maybe we should send a notification somewhere in Slack. At the moment it’s not possible because our only webhook event is when someone submits a form. So, if you’re redesigning your webhook mindset, if you’re redesigning your webhook strategy, think about all the different events that people can subscribe to. That will give you more power and they will give you more powers to users.

Something that we haven’t done, and I think it’s a great thing to do, that we are looking at eventually doing. Our webhook payloads is kind of obscure and not easy to get at the beginning the way we structured it, it will less likely change because legacy, we’re not going to change that. But we can help our users to understand how to deal with it easily. So, eventually we’ll create, and they’re already some created by the community, handlers to get to the data quicker. So in our case, if you want to filter all the responses that someone filled into a form and you just want to extract the email, we should be able to give you a tool like this instead of you having to parse the Json and find this information and do all the hard work. The idea will abstract that for you and you will find it for yourself. So, if you webhook an event and you webhook payload is not really straight forward, adding that to your SDK could be a huge win also for adaptations of users.

When we talk about webhooks, we also talk about a lot of HTTP requests and right now I feel that serverless and all those serverless functions are taking over the HTTP world and their servers definitely. What we’ve seen, and I’ve listed a couple of examples here is companies like Twilio, PubNub, or Netlify that are a powering directly into their own interface a serverless environment.You can attach this to a specific event, not an HTTP end points, but a function directly and you’re staying in their own environment to write that function. In the end is still an HTTP request, it’s still an HTTP attached to the function you’ve built, but they stay on your own, on your own company, on your own environments. They used to be a project powered by Auth0 and I think it was called Auth0 Extend where their whole idea was to provide this as a tool so all of us can implement that on our own and we can be as powerful as tool function. Sadly, I think it died, but there are many other ways to do this and I think you if you implement this you also reduce the friction for your users. They don’t have to think about where they host the code and what you could do with it.

Other alternatives are things like Standard Library and Glitch. So, Standard Library, they also provide a serverless infrastructure and eventually they have an API where you can automatically create a function on behalf of the user and have a template for this function. For now, what we’ve been using and what has been working for us is Glitch. So, Glitch if you’re not familiar with it, is like an online ID mainly for the Node, JavaScript, HTML world and with the potential, the possibility to do remix. So remix, is like a fork of your project. The good thing about this is as you launch a project, it already has a URL. It’s already available on the internet. So, the time to action for a developer that’s remixing your project and they want to interact with your webhook is reduced a lot and we have a lot of people using this. I encourage you to look at those solutions.

So, pretty simple takeaways. If you don’t have webhooks yet, please build webhooks. It’s a way to communicate and build better integration between each other. And, if you build webhooks straight to have a dedicated developer experience effort on this, your users will thank you. There are definitely experiences that we can build. So, everybody has informed data about what’s going on with those webhooks. And eventually if you have the engineering power to do so, add some native serverless experience around that. That’s probably what’s, what’s happening and what’s coming for all of us. And eventually some of those providers will white label it for us. So, it will make it easier. I think that’s all for me. I hope you liked it, and if you have any questions, let’s talk afterwards.