Deploying HTTPS at scale using Let’s Encrypt – Aloria
Articles,  Blog

Deploying HTTPS at scale using Let’s Encrypt – Aloria


[Music] about my experience deploying HTTP or TLS as the current technology at a very large infrastructure organization using let’s encrypt and we’ll get into what that is for anyone who isn’t familiar with the tool just to give a little background about myself Who I am why you might be interested in what the hell I have to say is that can I get aa diablos I’ve been working in security for about thirteen fourteen years now I started out doing vulnerability research for the US Army and then moved into applications security for a number of financial organizations and kind of decided that I didn’t want to be a corporate sellout anymore so I went to work for tumblr it’s how many people in here are tumblr users anyone know that’s all right I don’t work there anymore so I’m not offended and that job was actually a very kind of big change for me because it wasn’t just looking at other people’s code and telling them hey you suck you suck at writing code you need to do things differently here we at tumblr and also at my present job at Spotify we’re working to not only assist the developers in writing better products but also taking the imperative upon ourselves to implement security tooling that basically makes it easier for the organization to do security correctly and to do it easily and quickly so I am currently working at Spotify how many people here like Spotify yeah yeah way better than tumblr right but this is going to be talking about my experience at tumblr and using that let’s encrypt tool so basically two years of my life two and a half years of my life was all about encryption all about getting tumblr and all of its services as much as possible over HTTP so the problem statement now if you’ve ever are familiar with tumblr there’s kind of two components to it one is the dashboard which is sort of like if you’re familiar with a Twitter or an Instagram feed is a collection of all of the accounts that you follow all of the content that they’re publishing that’s the thing that you log into to see all your stuff on the second hand of it we have the blog Network which is basically the private or personal websites that that you would set up so for example we want or not for example but we want tumblr users not only to have a sort of integrated experience with all the content that they’re interested in being consolidated into one place but we also want the users of tumblr to be able to treat their accounts as their own personal website similar to something you might set up on WordPress or Squarespace for example so we’ve had the logged in part of tumblr over HTTPS before I even started it’s been over there for a very long time so our next goal our next problem that we wanted to solve was to get as much of those personal websites those blogs get as much of that network over HTTP we also want to do this with the least amount of money you know tumblr is still kind of startup B I imagine that many of you working for security companies know that your overlords can be a little stingy when giving you budget to do security stuff so we wanted to make it as inexpensive for us as possible and also to make the experience user-friendly users are especially in an environment like Tumblr there is a wide kind of variety of how technical or non-technical they can be and we want to provide the service to everyone even if you may not know what a certificate is even if you know hey I heard that we should have things over HTTP but maybe not necessarily know why so this is just kind of showing you the two different challenges for the blog network on the what is that the left-hand side we have the default when you sign up and create a blog it would be your username tumblr comm we also because we want this also to be an experience for people you know to have a website to give them the options you have a custom domain name assigned to the site so on the Left we have security reactions that tumblr.com which I don’t know if any of you have ever visited but I haven’t updated it in a couple years so whatever and on this side we have clients from hell dotnet if any of you are consultants or do freelance work I definitely recommend visiting the site it is hilarious so the security reactions that tumblr.com use case is very easy we can just make a wild card domain you know asterisk tumblr com there’s our certificate scoped to that and then anyone who wants to turn on a secure encrypted communications for their site just has to flip a switch and not really a lot has to happen on the other case it’s a little bit more difficult right because each domain needs its own cert you can’t do a wild-card cert 4 ComNet it’s just would break the whole purpose of having certificates in the first place right so clients from hell dotnet we would need to generate a certificate for them getting it installed in our infrastructure so as I mentioned before some of the challenges is that users are not often very tech savvy may not may know that they want a fee TPS but may not know the advantages and disadvantages of having a site served that way the certificate generation process for anyone who’s tried to do it I mean all of us in this room are probably quite technical so it is probably easy for us to understand but explaining to you know my 10 year old niece or my 60 year old grandfather had to do it they’re not very technical and that’s not really something that we want to put the onus or the responsibility on the user to have to figure out the other reason for this was a little bit of a selfish one is that we have limited resources in terms of our support staff we really did not want to do any hand-holding for them we didn’t want to have customer support requests coming in every single time somebody wanted to install a certain couldn’t figure out how to do it the other thing that was a challenge is that if you have a site that wasn’t designed to be served over HTTP or a what we would call protocol agnostic it can make your site look like a piece of crap maybe I should just throw out a few swear words and see see which ones get bleep so if you have let’s say all the things that make your site look pretty you may have your JavaScript break because it’s been hard coded to be served over HTTP the same thing with your cascading style sheets and even with maybe some images that you’re loading they don’t necessarily will not load but you instead of getting that beautiful green lock you get like the yellow mixed content warning and of course with chrome being very kind of anal retentive we we really wanted to make this also an opt-in process just so that the user could turn it on and then if they realize that it breaking their sight and couldn’t figure out why they could turn it back off we didn’t want to turn this on for everybody and then have a bunch of really shitty looking sights I see alright I’m sorry I’m having too much fun with this so a little bit about let’s encrypt this is a wonderful tool that came out I forget how long ago but it was at least three four years ago it is a certificate authority where you can get a certificate for no cost which is really maybe not doesn’t seem like a big deal but it was sort of a little bit of a breakthrough when it came out normally you would have to go to like Verisign or any other certificate authority give them a bunch of money and like $30 maybe is not a big deal to to a lot of people but still the fact that you could get a inexpensive or rather free certificate from the certificate authority have it trusted not have to rely on a self-signed certificate anymore was really a huge deal and really what made us able again inexpensively to offer encrypted communications for our users when let’s encrypt came out there were a lot of cool tools that you know you could just run a script or an application and it would go through the process of generating their certificate and installing it on your web server but could we find the one that works with our setup as you can imagine a organization like tumblr or maybe other web hosting companies have multiple servers you know we need to have distribution we need to have ways to kind of load balance so we needed to find out if one would work with our setup and unfortunately maybe there are now but back three years ago when we started looking at this we really there really wasn’t a lot out there and I’m going to blame PHP for that sorry I love PHP but it sucks so again our design decision so you wanted the certificates the lot opted in because we could break a site and we wanted the user to be able to recover from that well they figured out what was going on maybe not so much of a big deal for someone like me my personal website doesn’t look pretty for a day or two not a big deal but for some of their bigger customers like we had a chocolate company that was using tumblr to promote their brand we had a luxury car company I won’t name names but if their site looks crappy that’s really embarrassing for them and for their personal brand or their corporate brands so we really wanted that to be a switch that you could flip so you could figure things out instead of having a busted looking site for however long the other thing is that let’s encrypt does have a rate limit you can have it increased but since we were starting out we didn’t want to go to let’s encrypt we didn’t know what sorry I’m talking fast we didn’t know what the rate limit would need to be increased to so really as we rolled this out we wanted to do it in a way that would be a little bit more graceful not overload let’s encrypt servers again one so that we could continue generating certificates and also so that we didn’t ruin the experience for other consumers of let’s encrypt so just trying to be good citizens here I’ve talked to other people other organizations that have leveraged let’s encrypt to do this exact same thing and one of the things that I found interesting one or two companies basically just got a list of all the custom domains that they were hosting and then went out and talked to let’s encrypt and generated certificates for every single one of them so they basically had a huge pile of certificates and then if somebody wanted to turn it on they could just go out and grab it from their data store again rate limits also we didn’t really feel tumblr is a little bit of a social justice conscious company you’ve probably seen it being made fun of before we were kind of uncomfortable generating certificates on somebody’s when a user or a organizations behalf without getting permission it just seemed like the wrong thing to do we didn’t want to be sitting on a pile of unused private public keys it won because that’s a huge attractive thing for somebody to steal and also it just we wanted to you have users be in control of their own information and whether or not that information exists so just to go into how let’s encrypting works and it works very similar to how any other request for a certificate from a paid certificate authority works but just for people who aren’t familiar with that process is you have a domain so you have to generate your public and private keys and then you tell let’s encrypt hey I want a certificate and you sign that with the private key let’s encrypt response back basically saying okay you want a domain for limp Biskit rules com prove that you own that domain you know they don’t want to just Jenner they don’t take your word for it as they shouldn’t otherwise I could say hey give me a certificate a valid certificate for google.com and now I can do bad things so they give you a piece of challenge information and they ask you to host it somewhere within the scope of that domain so that can either be hosted as a on your web server as a challenge or a the challenge can be put actually into your DNS record so it supports both there are reasons that one might prefer one or the other we went with the the web hosted challenge one because that was something that we could easily control since we do own the web servers and two just because again we don’t want to the users have control of their domain names we don’t want to ask them hey update your DNS record two popes this random weird-looking string so once we have the challenge hosted then we go back to lesson care can say hey we proved the thing please give us a certificate the certificate signing request usually in a generation process comes first where comes at the beginning of the request but in our case it really didn’t matter we give the certificate signing request to lessen krypt after validation has occurred and then we have our shiny new cert there comes in the problem of how to get that certificate installed on our servers but we’ll come to that in a little bit and just for the visual people in the audience you can see the organization here would be tumblr the root certificate authority would be let’s encrypts you’re generating your key pair and a CSR on the root certificate authority side they have their own public/private keys that they use for signing the certificate to prove that they did issue it and then the certificate is created and returned so that doesn’t seem so hard right that’s basically certificate generation 101 but you have to talk to the let’s encrypt servers right this is a automates a semi-automated sort of thing their servers I can’t remember what Acme stands for I apologize it’s something like automated certificate management ectoplasm I don’t know so we didn’t really know how to talk to the let’s encrypt servers and there really wasn’t like a straightforward how-to guide let’s encrypt was more of a hey you don’t need to you just run in one of these fancy tools and you’ll have a certificate so we read does anyone here like reading RFC’s okay me neither but we had to read a lot of code basically reverse engineering the existing tools to figure out how they worked and also reading the RFC s for the Acme server implementation so it was basically a lot of reading documentation and a little better reverse engineering maybe 30 basically I’d say it was 30 percents reading the RFC and saying it I’m just gonna sniff some traffic and see how this works the let’s encrypt API gives you a bit of information if you go to their directory path so we did see that there are five difference of things to talk to of there’s one for a keychain there’s one for requesting a new authorization there is one for requesting a new certificate and there is one for registering the certificate there’s also a path for certificate revocation which is super awesome we actually did not revoke certificates if people were turning it off unless they had it turned off for a certain length of time just because there is that use case of somebody wanting to turn it off fix their site layout and turn it back on so just another design decision what you’ll notice here and no this is not enough medical order is it maybe I don’t know I don’t know Aang I don’t this is not the list of these end points is not the order in which you actually talk to let’s encrypt another thing that we had to figure out the way it actually works is you talk to the new registration endpoint where your public account key you let them know what your public account key is and then you sign that communication with your private key just to prove that it’s you what happens is let’s encrypt responds back and says okay again prove who you are it’s a little small here so I apologize but basically they say put this challenge at this URL it usually goes in the format of example.com slash dot well-known acne challenge that’s and they say here is the information that you’re going to sign subsequent requests with so after we write the challenge then we go and request a challenge check and then we can receive our certificate again I still have not talked about how we’re actually going to get these things on the hundreds and hundreds of server I don’t know if it’s hundred but and on maybe I was talking too fast buffer overflow can I be able to talk like I’ll do some rapping at the end and see if we can crash whistling um if we have time so let’s get to the best part we have a lot of servers which means that there are a lot of places that the certificate information leet needs to live hitting one website could lead to any one of hundreds of web servers actually being the one to serve up that content so there certificate information needed to live on all of those we also didn’t want to expose sensitive data such as private keys you know we do have a storage in the in our back-end where we’re keeping those keys but we don’t want to have that database just sitting in our DMZ for anyone to try to break into so what we did is we wrote a trusty middleware service in Scala who here likes Scala ok good me neither oh I’m sorry sir for if but first let’s talk about the backends because that’s really where most of the gears are turning so basically the actions are triggered by the user requesting to activate HTTPS on his or her block again it’s a little toggle so then we generate the keys and start the process to talk to let’s encrypt we host the challenge and we pull let’s encrypt for this challenge validation this usually takes under an hour it’s not a one after another thing there usually is a little bit of waiting time between these steps one because of respecting rate limits two because the certificate generation or the challenge validation usually can take a little bit of time for the let’s encrypt side to go out and figure out that you actually did what it wanted to do and then again certificate generation can also take a little bit of time usually we could get this well under an hour an hour was usually the maximum that it take and keep in mind that we will be generating certificates for a lot of different domains at once so we really have to again kind of throttle this process so on the front ends if a request comes in for a domain we have to figure out whether or not it has HTTPS enabled if it does and the certificate is not already installed in the web server then we pull our middleware which I’ll talk about in the next slide we ask one whether the site supports HTTP will we do that pretty much for every request and then request the certificate information and load it into our web servers if the if HTTPS has been disabled or was never turned on in the first place then we redirect back to plaintext if a plaintext request came in then we automatically forward it to HTTPS meaning that you can’t hit the site on both protocols it’s always going to be one or another because again what’s the point of having an HTTPS enabled site if people are going to try to hit it on the old URL this was also done because we want to there may be hard-coded links somewhere so we want to basically forward that those requests that may still be coming in over the old outdated unencrypted link cool I’m doing good on time but so yeah I’m sorry let’s talk about the middleware so the middleware does a few things it saves us from directly or constantly pulling the database for certificate information if a sir if a certificate has already been requested then it’s going to cash it in that middleware so that we’re not constantly taking the computational expense of talking to whatever our database servers were in I think they were MySQL you were they yes sorry so what it’s going to do if the cached information doesn’t already exist it is going to talk to the back end and first determine has HTTP been enabled for the site if not then it really has nothing else to do but let the front end know and then if it does have HTTP enabled there may be a sort of what’s a good way to put this limbo state where the person has turned on HTTP for their site but the certificate generation process has still not completed so that’s the other piece of information that we need to know and then it will return the certificate generation information to the front ends so that it can be installed we were using Apache with a no I’m sorry nginx with a open rusty module and what that does is allow you to dynamically install certificates without having to reboot the server or reload the server which is as you can imagine incredibly important because you can’t be reloading your server every single time a new site turns HTTPS on cool so we’re getting kind of I’m actually kind of running a little bit early so let’s just talk a little bit about our conclusions observations and then we can do questions and if we don’t have any questions you can either leave early or I can do rapgod okay the one thing that I want to say is this is not there is no one-size-fits-all solution for deploying HTTPS some of the I talked about some of the other approaches of generating all their certificate information some people will just use their content distribution network and have them proxy those requests for you so I believe that’s a service for example CloudFlare offers where they will deal with the certificate generation process we one didn’t want to do that because again we wanted the option of users turning it on or turning it off at their leisure oh there’s water here I forgot about that no wonder I’m sweating and also as I said if you have the money for CloudFlare to enable the service good we did not we wanted to make this as cheap as possible to keep our Yahoo overlords from getting upset with us I think their oath now which is a really silly name but tumblr is still a great company so no problems with them again the more complicated your infrastructure is the more moving parts the more things talking to each other it’s a little bit more difficult than just say Oh dump of dump a certificate in there and I think that’s really something to keep in mind when trying to do a project like this is it is somewhat ambitious and I think that’s one thing to take into consideration when organizations are when you are asking why haven’t you turned HTTPS on yet why haven’t you offered it there’s a lot of background engineering work that needs to happen and so maybe to anyone in here who is a hiring manager get more people please because I did this all by myself and it sucked it didn’t suck it was just a lot work the one thing that I also wanted to point out and sort of the thing that I was griping about earlier was that there weren’t a lot of tools and there certainly wasn’t one that would work for our infrastructure which was PHP nginx and a bunch of other weird technology technology decisions but as people mourn people have been getting interested in the service as adoption has increased more people are writing tools to interact with let’s encrypts so really automates and make it less painful to get a certificate from them so that’s I don’t want to say if you loot usually decide to use less encrypt it’s just going to be a whole lot of blood sweat and tears there may be a solution out there for you I haven’t looked into it recently but there is a high possibility that there is something out there now that didn’t exist before that we could have used if we had a time machine and also just always take into consideration the user experience I know we’re really all super passionate and gung-ho about making things more secure and getting as much of the internet as possible over encrypted connections but just keep in mind that one your user may not understand and it may sour them on your products if you kind of try to ram it down their throats and also you know there are maybe some things that you don’t necessarily think about right off the top of your head so for example I didn’t really consider the fact that turning HTTPS on for a website could make it look like a piece of crap from 1997 hosted on Angel Fire just something I didn’t really think of until a designer pointed it out to me I’m beginning to feel like a rap god rap god all my people from the front to the back so I’ll open it up to questions again I’m really honored and a very thank you so much for having me hopefully this was informative of two maybe one or two people in the audience if you have really want to dive in so let’s encrypt how it works there the top is sort of a high-level view of how would you know again how the system makes the things happen and then this is actually the RFC if you really like reading RFC’s which I assume you just all really really love I’ll be around for questions later if we don’t get to them here I again I am a Lauria on Twitter and this Lauria at traverse dot al yes that is an albanian domain name don’t ask me why is my email address cool so what we have like 10 minutes left to see if anyone has any otherwise and if I could wrap like a beautifui in my genes i got a laptop my back pocket my pen’ll go off when i half-cock again living in and kiln off it ever since bill clinton was still in office with monica lewinsky soon know okay [Applause] [Music] you

2 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *