Tuesday, April 23, 2013

In the coal mine.

In a recent episode of the Ship Show podcast, Paul Reed interviewed Damon Edwards in which they made the analogy to Release Engineers being the "canaries in the coal mine." Meaning that when there's problems in the release cycle, the release engineers are usually the first to notice. Then they made the astute observation that traditional development shops don't push the detection upstream to the developers, but instead blame the Release Engineers, which is in essence saying, "send more canaries". Hence the name of this blog. 

Before joining Netflix, I was a full-time programmer, doing J2EE things, for a site with real customers. While technically I'm still a "Senior Software Engineer" my current role focuses on the tools used by engineers. It's not a Release Engineer position, since I don't actually release anything, but many of the tools used by engineers are for releasing, so I play in that space. 

If I directly help other teams 40% of the time and program the other 60%, I consider it a good day. Our team is lucky in how it's structured as a service organization. This is only possible because it's a core Netflix tenant that teams take responsibility for their releases. Developers are expected to be the ones that code, build, release and deploy. Teams might structure themselves into more traditional Dev/QA/RelEng roles, but that's up to them. Engineering Tools is there to help, but if we can't do it, the team has to take responsibility for what ever they're missing or having trouble with. Effectively meaning that we're not there to gate their release. If we do gate/block their release, we've done something wrong.

Our value comes in being domain experts (Ant, Ivy, Gradle, Jenkins, AWS), evangelizing good practices used by other teams and automating anything we can. (This means we do end up supporting our own work, e.g. Asgard, Monkeys, Aminator, Gradle builds. But we do our best to either contribute back to OSS, or just OSS ourselves, so that's its not unique to our team.) I hadn't ever seen this organization structure before, I wonder how common it is. It's amazingly effective, and I believe it's why the concept of DevOps has succeeded at Netflix. It's balances the line between making things easy enough for teams to deploy and not abstracting everything away.

I guess I'm trying to say that let's save the canaries and make great tools, so developers (like my past self) can control the entire deployment pipeline.


  1. Replies
    1. IEEE Final Year Project centers make amazing deep learning final year projects ideas for final year students Final Year Projects for CSE to training and develop their deep learning experience and talents.

      IEEE Final Year projects Project Centers in India are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation.

      corporate training in chennai corporate training in chennai

      corporate training companies in india corporate training companies in india

      corporate training companies in chennai corporate training companies in chennai

      I have read your blog its very attractive and impressive. I like it your blog. Digital Marketing Company in Chennai

  2. Everything is unguarded with an exact explanation of the difficulties. It was truly instructive. Your site is very useful. Much obliged for sharing! best interiors

  3. Brilliant post! We are connecting to this extraordinary post on our site. Keep up the extraordinary composition.best interiors

  4. You should be a piece of a challenge for probably the best website on the web. I will suggest this site!
    best interiors