This article will be about a deployment scheduling technique which I call operationally cautious deployment. (I didn't pick the name so the acronym would end up being OCD! Honest! I swear!).
- Distribution packages (debs, rpms...)
- Language's native packages (eggs, gems, ...)
- The development environment
- The test environment
Hi, and welcome to the second part to my ranting about development stacks. In my previous post, I talked about the issues with some of the popular hosted solutions (Github, Bitbucket and Launchpad). In this post, I'll try to roll my own dev stack that hopefully doesn't suck too much.
- version control
- issue tracking
- preferably some form of code review
- You want other people to use your system.
- You want to be able to use different things, now or in the future.
After a Django pony, testing goat, packaging pig, there is now also a deployment duck.
- A version control system. Preferably not CVS, Visual SourceSafe, ClearCase, Perforce, RCS, SCCS...
- A ticketing system. Preferably not Trac.
- A code review system, possibly just informal code review as part of the ticketing system.
- Launchpad with Bazaar.
- Bitbucket with Mercurial
- Github with git (and hub)
This post is just about the process of moving a dev shop to continuous deployment. If you're just interested in the things I'd like to build, feel free to skip this one.
Despite the ominous messages in my last post about continuous deployment changing your dev shop dramatically, I do think it's possible to prepare properly. That means that you get up to the point where you could be doing continuous deployment, but you just aren't. Or, to put it differently, imagining you had to release right now, and getting trunk up to the point where that doesn't raise your blood pressure.
The first step is getting your testing up to an adequate level. I can't stress this enough -- I'm hoping it won't be a tough sale considering the amount of people who've hammered it before me -- given today's tools, there is very little excuse left for not having an extensive suite of things that check your code. I'm deliberately not saying "unit tests", because it encompasses a lot more than just unit tests:
- unit tests (These still go up as number one because they're what you're dealing with most. Plus, out of the things in your code that are just plain broken, unit tests will help you find at least 90%.)
- static analysis (pyflakes is great, but Python is quite hard to do sensible static analysis on -- statically typed languages, for example, can expect to reap much greater benefits)
- integration tests (although these are generally part of my normal test suite and are ran together with unit tests, they're really quite different)
- user interface tests (if you're building a webapp, Selenium is wonderful)
Testing is very important, but it's hardly the whole story. Testing (with the minor exception of some forms of static analysis) make sure your code works -- it makes no guarantees about that code being any good. For now, computers aren't too great at judging code in such as subtle fashion, and you really want humans to do it for you. Especially in languages that let you do pretty much anything, code review is very important if you want to keep your code sane. Like test-driven development, a lot of people much smarter than me have driven this one home for a long time, so I won't try to convince you too much.
Another advantage of code review is collective code ownership and a resulting increase in bus number. High bus numbers are very important, especially so when you're doing continuous deployment. You can't, or at least shouldn't, delay a release. Once something does go wrong, fixing the situation should be everyone's immediate priority. Only one person understanding how a particular part of the project works slows things down. That sounds like a restriction, and in some ways it is, but it's the kind of restriction that forces you to do the right thing.
I have found that a lot of people do one kind of code review, and contrast them. I've had good experiences with doing extensive and multiple kinds of code review. Most people believe that to be a waste of time and resources. I'm not convinced. First of all, there's Linus' Law arguing for many reviewers. Secondly, I've found that different kinds of review (and different kinds of reviewers) spot different problems, because they care about different things:
- A pair programmer will spot nitpicks about code quality and save you a bunch of "duh" moments
- A reviewer working on the same project reviewing a feature branch will pay more attention to interoperability issues with the rest of the code
- Someone who's not even a programmer (yep, "everybody writes code" works out great) will spot documentation issues
- ... (this is not, by any means, an exhaustive list)
Adding a form of code review takes a load off for all of the other forms of code review. The curve is sublinear: adding more code review doesn't make code review take proportionally longer (it won't make it shorter either, of course).
A lot of people dislike being criticized about their code. Stamp this behavior out as soon as possible. Everyone's only human, and plenty of ugly, broken or otherwise inadequate code gets written. The point is to get it out, not berating the person that wrote it.
Pressing the big red button once you're done will still be scary. You might even have a hiccup or two. But ideally, nothing will blow up. After a while (and probably in less time than you think it will), releases will become, as Timothy Fitz puts it, a non-event.
Lately, I've been thinking a lot about building a continuous deployment system.
I was really glad to find out that Timothy Fitz was doing continuous deployment over at IMVU (partially because Timothy and I like some of the same software). Eric Ries, former CTO and co-founder of IMVU, has had a lot of interesting things to say about it as well. I used to do this at my old day job, and it's been an absolute eye-opener -- yes, it does require quite a bit of discipline, but I'm convinced it pays off handsomely, especially in the long run.
Despite my newfound conviction, the system we used there wasn't all that fancy. That has its upsides, the less complex it is, the less can go wrong. Unfortunately, being a horrendous hodgepodge (I'd call it an unholy alliance, but there's no VB or COBOL) of bash, perl and Python distributing software with rsync, it's not very suitable for showing off.
The goal of this series of blog posts is to end up with a continuous deployment system people can agree on. (If you're laughing, yes, I realize that's probably a little naive -- I'm hoping to come out relatively unscathed and with a continuous deployment system I can agree on.)
Hi! This is part two of my continuous deployment rantseries which started here. In this part, I'll try to explain what I'm talking about when I say continuous deployment.
Strictly speaking, continuous deployment just means that you push code out to production servers all the time, practically as it gets written.
Think about that for a second. Are you scared? If so, why? Don't feel too bad if you are. Most people I've talked to about this are somewhere halfway between intrigued and terrified, and are mostly glad it's not happening to their code. A lot of those people also produce great software anyway. The people that don't feel anxious at all either haven't produced enough production software yet, or are already busy doing continuous deployment.
People who already practice continuous integration, with all sorts of tests and code quality metrics, often talk about how trunk is always ready for production. Continuous deployment is about putting your money where your mouth is and actually doing just that. The basic idea is similar to that of continuous integration: test as many things as you sensibly can and fail as quickly as possible once things go awry. Hopefully, that'll contain minor hiccups before they become real problems (generally defined as "the kind that costs revenue").
However, I don't think the strict definition is a very useful thing to do or talk about purely by itself. I've learned that continuous deployment doesn't come alone. Much more so than continuous integration, which feels like a natural extension of TDD, continuous deployment pretty much turns your development process upside down. Doing continuous deployment without a disciplined team to back it up and a testing infrastructure to constantly check your work is a bit like having a transmission without an engine and a steering wheel. (I apologize for the car analogy, and promise to try and not make a habit out of it.)
From what I've read and my own experience, teams who have successfully applied continuous deployment see these two things come back inseparably. If you want to make sure your code isn't plain broken, you better have a wide array of tests. If you want to make sure the code you're now responsible for in production isn't terrible, you better be doing code review (pair programming and/or tool-assisted code review is great -- we've had success doing both). There's nothing quite like the first time you commit a file and then watching your shiny new piece of code do something on a production server less than a minute later.
So, I guess the series is partially about continuous deployment proper, and partly about greenfielding a development process that rocks.