beforeEvaluate

In Gradle, I've come upon a development pattern. It usually follows this pattern:

1. Write plugin
2. Write Extension, so that the user can customize the behavior of the plugin
3. Wish I could use Convention Mappings, but then resist since they're not public
4. Realize that I can't look at the Extension until afterEvaluate, which is after the user's code has had a chance to set values on the Extension
5. Put all my plugins logic in project.afterEvaluate
6. Swear at any @DeferredConfigurable plugins
7. Fight the ordering of my plugins

When I say @DeferredConfigurable, I'm pretty specifically looking at the PublishingPlugin's DefaultPublishingExtension, which will throw an exception if you try to configure it after it's been configured. Honestly this make sense, but it doesn't make it any easier to use. The publish plugin adds it's own afterEvaluate which forces the extension to be configured, since it needs to create tasks before the task graph is built. 

This means that if I have a Plugin that uses an Extension which is going alter a Software Component, it HAS to run before the publish plugin is applied, so that my afterEvaluate is run before the PublishingExtension's afterEvaluate is, keeping in mind that afterEvaluate is evaluated in insertion order. But that's impossible since I can't configure the PublishingExtension until after the PublishingPlugin has added the PublishingExtension to the project. I think I've also seen issues where the IvyPublication is resolved before I can add in my generated/imported dependencies, which is another reason I need to run before the publish plugin's afterEvaluate.

Technically the publish plugin is @Incubating, so I'm not faulting the Gradle guys and don't even expect this to be a bug in its own right, since they're still figuring it out. In my cause, I require the ability to modify the module descriptor, and that's only really possible with the publish plugin, so I'll live with it. Though I'm starting to get the feeling that there's some fundamental ordering issues cropping up, which will eventually grow out of control.

My solution is implement a beforeAfterEvaluate method that lets me jump to the front of the dispatcher LinkedHashMap. Caveat, there's lot of internal classes being used, so I know that it can break in future versions of Gradle.

Loading ....

This doesn't answer the general ordering issue, but it can unblock a stalemate in the order of plugins. It's worth mentioning that Convention Mappings aren't the solution either, they're just an opportunistic lazy evaluation that "typically" works in your favor.

Ahoy Matie!

At the last Java Posse Roundup, I gave a lighting talk on "Thinking with box" where I outlined some fun I have had with cardboard with my daughter. I'll link to the video once it's uploaded (I'm looking at you +Carl Quinn) . Here are some of the steps, with some tips and tricks, that I did to make it. (I've meant to write this up for a while, but I finally got a little push from @Chris Cobb which put me over the edge.)

Step 0: Surf the web

My wife was on Pinterest looking at pirate ideas for our little one's 4th birthday, and looking over her shoulder I saw a cardboard pirate ship that looked like a lot of fun. So I dug deeper by searching Google for "cardboard pirate ship", and wow, there's a lot of stuff out there that adventurous parents had done. I zero'd in on this picture pretty quickly.

Step 1: Build a Model

I'm fortunate to work at a company that has lots of spare TV boxes around, but I didn't necessarily know how to get them for myself, so I filed a Helpdesk ticket. They were fantastic in helping, they brought by samples to see exactly I was interested in. They even took a huge whiteboard out of a box, just to give to me. I had some great building materials, but I didn't want to waste any and I wasn't even sure if I had enough, so I built a model. 

This proved helpful in designing the large straight pieces, and more importantly the angle for the bow.

Step 2: Cutting

I'd say cutting cardboard is best done like drywall, where you score one side with subtle pressure, folding it, then cutting from behind. Before you score, I'd recommend getting a very long straight-edge and drawing a line where you're going to score, even if you're going to score against the straight-edge. I didn't have any rulers long enough, so I ended up using some wood scrap, which I'll talk about later. The cutting was primarily done with a utility knife.

Step 3: Attaching

The model showed me that I had to cut the bow as a trapezoid  but once I got to real cardboard I learned that there more complex angles going on in three dimensional space. But I was close enough that I thought I'd just start attaching everything. The original idea was to just staple gun two pieces of cardboard, that was a horrible idea. I don't even know why I thought that would work. The additional stress on the bow make it clear that I'd need more support, so I went back to Home Depot and picked up some wood scraps which I could screw into. The best way to describe it was 1/2" x 1". It's no more than a $1 per 8 foot piece (wood is cheap in California). You can seen an example in the picture for the mast.

I'd cut the strip to length, letting the little one pull down the chop-saw. To attach the cardboard, I put washers on a drywall screw then threw the cardboard into the wood behind it. The washers were necessary to prevent the screwhead from going through the cardboard. It was quite common for the screw to come through the wood, in which case I used a Dremil to cut it off the end sticking out. It wasn't perfectly smooth, but it was good enough for 4 year olds. If you zoom into the picture you might be able to  see the washers and wood at the intersections between pieces.

Step 4: Laying out

I worked from the bow to the back, and I honestly didn't know how each part was going to work out until I tried. The model really helped. 

I'd sometimes staple gun in between the screws to get a really good seal between pieces, but this probably wasn't needed.

Step 5: Cutting Windows

I cut a single window, then used it as a template for the other windows. I found that the utility knife had a tendency to go straight, so I used an X-Acto knife for corners. The thickness of cardboard meant I had to make a few passes before it went through, but tracing a previous cutout made it easy to follow the path. I'd usually duct tape a previous cut where I wanted the next window, so that it wouldn't drift as I cut around it.

Step 6: Painting

At most home improvement stores, they have previously mixed paint which has been returned and is now on clearance. You can get a gallon of decent paint really cheap. We did find that Home Depot was selling these clearance items at full price, our theory is that people were abusing the system so they needed to stop selling them so cheap, so we went to Lowe's instead. We let our daughter pick any color they had there, and we picked something to complement it. She wanted purple, but they only had pink. Painting cardboard proved to be really straightforward.

Step 7: Finishing touches

At this point it was looking fantastic, but we had already pretty much taken the day off of work, so we thought to fancy it up a little. My wife cut a large H and I put duct tape around the edges. And anchor was made by stapling loops of paper together. A sail was just an old sheet, which had to be cut or else the ship would literally sail away.

Seeing the rings

Right now is a great time to see the rings of Saturn, just follow the moon down and to the left, past Spica and the next bright thing is Saturn. Hmm. Let me rephrase that, it's a great time to see the rings of Saturn through a telescope. A few months ago we bought a telescope for the parents who took astronomy classes together the little one. We kept the budget manageable enough that we could get a nice telescope sometime in the future if this works out. We've learned some lessons, which might have been different if we had a $3,000 telescope:

• Jupiter and Saturn are a rush to see
• Stars are just dots in the sky, which you make big and blurry with a telescope
• Four year olds have the patience to see one celestial object a night
• Four year olds "get it" and are excited to see a blurry star even its just orange in color
• The moon never gets old

I do think there's a limit to viewing. Once we saw Jupiter and the moon, there wasn't much more we could get that wouldn't be more enjoyable by just surfing images.google.com. But then we waited and saw Saturn. I think we're going to find that we bring it out every 6 months, see some new object in the sky and get a renewed interest for a week. And that's perfectly fine.

We're lucky that Orion is a local company, with a store in Cupertino. The whole family went over one Sunday and we brought home a very reasonably priced telescope. There's an argument to be had that we should have just bought really nice binoculars, especially given my above statement about bursty interest. There's just something special about having a real telescope in front of you. Our budget was <$400. To extend our budget, we opted to go without any tracking mechanism, so that we can could spend more on the optics. (It's fun to just talk about the optics, I understand optics better now than I ever did in Physics class, but I'll save that discussion for another post.) The other balancing act was portability, we wanted to be able to bring it to viewing parties and just whip it out from the garage when it's nice out.

In the end, we got an Orion SpaceProbe 130mm EQ, with a 900mm focal length. And after some initial fun we bought the Orion Telescope Accessory Kit for a few more length and some filters, including a moon filter. In both cases they were on sale in the store, so the total was around $300.

We like it a lot. It's not too big that I can't just leave it setup in the garage. It's advanced enough that I can play with different lenses. Maybe in the future we'll be able to justify $3,000 for a sweet Cassegrain, but for right now this works for us.

Don't dare prompt me

At Netflix, we've been trying to come up with an effective way to distribute the Gradle wrapper (more on that in another post), and the solution we're going for is to check it into a git repo, which can then be cloned locally or referenced as a submodule for CI builds. So, when we're constructing our wrapper and pointing it to a custom distribution of Gradle we need to perform git commands. Luckily Andrew Oberstar created a eponymous Gradle plugin called gradle-git, which wraps JGit. We already started to use it to publish javadocs to gh-pages in the Netflix OSS projects. In this case, I started to use it to check the new wrapper into a repo, only to find out that I was being prompted for my passphrase.


It's easy to forget that one's SSH private key has a passphrase on it, because most apps indirectly ask ssh-agent or Pageant for the key. (Note: I almost forgot my passphrase, please go and check that you still remember yours.) When I say most apps, I mean every possible app except JGit.  When JGit was written it appears that it was abstracting out the underlying implementation of SSH, but generally uses JSch to keep everything in the JVM. The problem is the JGit seems to use some defaults that bring a Swing UI to prompt a user for their passphrase every time it instantiates a session. This might make sense in an IDE, but in a CI server that is ridiculous.


A quick Google search shows that the creators of JSch figured this out and wrote jsch-agent-proxy. Great, all I have to do is wire that in and prest-o change-o, I'll no longer be prompted. Nope. I searched high and low and couldn't find an example of someone configuring jsch-agent-proxy with JGit. Some more research showed that JGit in all it's lovely abstraction layers hides the parts of JSch that I need to plug in jsch-agent-proxy.

Here's my implementation that connects the JSch under JGit to use jsch-agent-proxy:

Loading ....

Most of this code comes from jsch-agent-proxy's JSchWithAgentProxy example. jsch-agent-proxy has an impressive set of implementations. It can use Unix Sockets or JNA, or talk to ssh-agent or Pageant. Given how easy the JGit API is to use, I expect some interesting Java-based utilities to come out in the next year.

In the coal mine.

In a recent episode of the Ship Show podcast, Paul Reed interviewed Damon Edwards in which they made the analogy to Release Engineers being the "canaries in the coal mine." Meaning that when there's problems in the release cycle, the release engineers are usually the first to notice. Then they made the astute observation that traditional development shops don't push the detection upstream to the developers, but instead blame the Release Engineers, which is in essence saying, "send more canaries". Hence the name of this blog. 

Before joining Netflix, I was a full-time programmer, doing J2EE things, for a site with real customers. While technically I'm still a "Senior Software Engineer" my current role focuses on the tools used by engineers. It's not a Release Engineer position, since I don't actually release anything, but many of the tools used by engineers are for releasing, so I play in that space. 

If I directly help other teams 40% of the time and program the other 60%, I consider it a good day. Our team is lucky in how it's structured as a service organization. This is only possible because it's a core Netflix tenant that teams take responsibility for their releases. Developers are expected to be the ones that code, build, release and deploy. Teams might structure themselves into more traditional Dev/QA/RelEng roles, but that's up to them. Engineering Tools is there to help, but if we can't do it, the team has to take responsibility for what ever they're missing or having trouble with. Effectively meaning that we're not there to gate their release. If we do gate/block their release, we've done something wrong.

Our value comes in being domain experts (Ant, Ivy, Gradle, Jenkins, AWS), evangelizing good practices used by other teams and automating anything we can. (This means we do end up supporting our own work, e.g. Asgard, Monkeys, Aminator, Gradle builds. But we do our best to either contribute back to OSS, or just OSS ourselves, so that's its not unique to our team.) I hadn't ever seen this organization structure before, I wonder how common it is. It's amazingly effective, and I believe it's why the concept of DevOps has succeeded at Netflix. It's balances the line between making things easy enough for teams to deploy and not abstracting everything away.

I guess I'm trying to say that let's save the canaries and make great tools, so developers (like my past self) can control the entire deployment pipeline.