Hello internet, remember me from 7 months ago? What’s that, you ask? Yes, I’m still alive but it’s taken me until now to recover from a veritable barrage of poor excuses as to why I should stop spending time writing Stuff. There’s another 2 or 3 interesting recipes on their way soon, but what better way to resume from hiatus but with a rambling piece of vanity editorial opinion.
Last autumn was taken over by the idea that I had time to do two evening classes in a single term. It turns out that I could, but that it didn’t leave much time to write home about it afterwards. The dinner party cooking class was as good as ever, but the real wake up call was doing Stanford University’s Introduction to Artificial Intelligence class. I’ve never had much of a computer science background and yet I’ve made a career out of bending computers to my will, so this was wonderful opportunity to find out if I could still learn university level stuff after 12 years. The answer was yes I could, but it helped that I was properly interested in learning it and it was taught by two experts both in their subject matter and in teaching it to people. Whilst I am proud of getting an “84%” for my efforts, there were enough other people in the class of dozens of thousands who were distraught at the idea of not getting 100%, so that was a different idea of ‘normal’ I hadn’t met before. I could write pages and pages about my experiences, and perhaps I will, but this isn’t yet the time.
What has really rekindled my energy and focus was a game changing conference I attended in London last weekend, PHP UK 2012. I’ve been to a few techy conferences before, organised by The IET, vendors, or by the community of customers they serve. Scotch on the Rocks has been, and will probably continue to be my primary conference simply because it’s based on the language I’m most practiced in, but CFML will never have the same sized audience as PHP.
The London PHP user group has retained enough critical mass to host an increasingly awesome conference for 7 years now, attracting some top tier speakers and a good sized audience, but not so big that you can’t move or get close to the speakers or exhibitors. Who do I call a top tier speaker? You’ve probably come across names like eBay, the BBC, WordPress.com, Etsy.com and Rasmus Lerdorf, but there were some excellent talks from engineers closer to those I would consider my peers. I took home nearly as many points on presentation skills as I did technical ideas. I was able to speak with a few of speakers and many had spent weeks of their spare time preparing, practicing and refining their 40 minute presentations for this event, which is an inspiring amount of effort.
I realise that we were in the capital city, but I was surprised by how many people wrote on their delegate badges, or on their presentation slides “we’re hiring”. Some people even trolled the twitterfall (more on that in another post) to say things like ‘I’m hiring, DM me’. Thinking about it, it’s a great way to meet like minded people who might want to help you do whatever it is you’re trying to do, but I hadn’t encountered it before.
One of the more interesting debates was on release controls. How often should one release code and how easy should it be to release code? I’ve always been lucky enough to think that you should trust your developers, that you should have a team who’s not afraid to learn, and that you learn most quickly when you’re fixing a bug that you’ve just deployed to your application 🙂
Did you take more than 10 seconds to revert the change once you realised something was wrong? Did anybody die? Did your company lose a significant amount of trust, goodwill or money? If the answer to all of the above was ‘no’, then no harm was done and aforementioned developer has just learned some very valuable lessons and won’t be doing that again in a hurry. Clearly this approach only applies to a certain category of applications and user bases, you wouldn’t want to bork something like Amazon’s checkout system, or Google’s landing page for even a couple of seconds, but I believe that the person who best understands the implications of releasing some code is the person who’s just written it.
This approach works in practice if you force yourself to make lots of small releases, rather than fewer enormous ones, so it only really works for web-based projects. To quote Mike Williams (one of Erlang’s inventors): “Make mistakes on a small scale, not in a production project.”
I think the audience was split 50/50 between those that thought this approach was normal and why would you take the time to do it another way, and those that couldn’t believe somebody could even contemplate such a thing and how could they get away with such a cowboy attitude, but it can’t be all that terrible with services like Flickr and WordPress doing it. Some people call this continuous deployment, for me this falls into the mantra of ‘minimum necessary to get the job done well’.
From a personal point of view, the biggest ‘whoa’ moment came out of a panel discussion on scalability. Like most of the panel members, I think scalability is a function of systems architecture and how inspired your engineers are, not the programming language you use (assuming one’s code isn’t crap of course), so the conversation was widely applicable. It was a very good discussion and well worth the time to re-watch.
I’ll have to wait for the videos to be edited and published to check the exact words, but one question was about fixing technical problems when a bug only manifests for 1/4% of your 10^7 hits. Hugh E Williams from eBay said that if you hired engineers who were good at learning then they’d figure out almost anything, but it helped to be able to wheel out ‘that one guy’ who had that rare blend of curiosity, enthusiasm and doggedness who’s highly skilled in system operations and software engineering who could meditate on the problem (Amiga style) for a week and then fix it. Rasmus weighed in with a comment along the lines of ‘hire him. just hire him’.
A few people in the audience weren’t certain what they meant, isn’t this a really big single point of failure and anyway the split between programming and sysops exists for a reason. I think halfway through the explanation, Rasmus said something like ‘yep, we get it’. I don’t know whether it’s the weight of culture or if devops has taken sufficient root in The Valley for this to be normal, but whatever the understanding, it was like they’d just described me and what I do in just a few words. Whoa.