Build It Like You Mean It

Sunday, May 29, 2016

Disclaimer

2016-05-29

I have just started working at JP Morgan Chase, and I want to make it perfectly clear that my opinionations in this blog before this point in time are not flavored in any way by that position or experience: the ant-patterns I described, for example, are not from my time at JPMC, and I don't know whether they need any of the advice in my earlier posts.

I will not, in general, put content here in my blog that reflects on JPMC or how things are done there; I have a large pool of experience, most of it elsewhere, to draw on for errors companies need to learn not to make.

The only thing I can state with confidence so far about JPMC is that their security standards and practices are the best I have seen, though the bar is not set very high by the places I have been in the past. These people take their security seriously and practice it with discipline and care.

Agile Anti-patterns

2016-05-29

Agile is another term that everybody uses for whatever they're doing, in some cases with good reason, in other cases with nothing like it. I have seen a lot of companies trying to be Agile, and I have this advice for some of the anti-patterns I've seen:

Managers using scrum as a status meeting

A scrum is supposed to be about the team self-directing and helping each other. If the manager dominates the discussion and uses the meeting for their own purposes, this can often be lost and the unique energizing effect of proper scrum is lost. If you haven't tried doing this by the book, try it for a sprint or two and you'll see the difference.

Management drivers that don't keep track of their stories

Another really major point of Agile process is that management and business minds stay engaged in the process, making sure what's getting done is what they understood was getting done - that's the most important point that makes Agile worth doing. If the managers that own a narrative decide they're too busy to keep tabs on it, this key element dies out and the project can head off track.

Stories that don't motivate the management

One good description of how to write a good story is: keep asking "why do you want that" until you finally reach a business driver: cost savings, profit, something like that. If you let stories be too abstract they don't have enough punch to keep business people motivated and you get the problem immediately above.

Agile is flexible enough that you can benefit from pretending you're doing it, without actually having any good idea what it is, but it's better when you know why you're doing it and make sure you get the benefits it is good at delivering.

Continuous Integration: First Principles

2016-05-10

Continuous Integration is a term used by a great many software development organizations, with a very broad spectrum of possible meanings. It's hard to imagine more that about 20% of the people who claim they know what Continuous Integration is have actually ever read any articles about it to make sure they're not just guessing. In some organizations I've been in, Continuous Integration was defined as having a nightly build and heaping scorn on whoever makes it break. That's certainly better than not having an official build that often, since it implied that there is a periodic build that is:

Done on a separate, controlled system that has known tool sets on it and
Run strictly from what is checked in to Version Control

Those are the first principles of Continuous Integration, and practicing them is good even if it's not really the whole enchilada. However, it's not Continuous Integration, any more than acceptance to the University of your choice is a diploma.

The next principle that some places accept is to do these builds often, even several times a day. There are projects where there is not very much activity, and nightly is often enough, but if there will be a dozen checkins picked up in a single build, it can make it a lot harder to pinpoint the cause of a failure.

However, the next principle that a lot less projects have in place is testing the new builds. You should have unit tests, ones that developers can run before they check in, and that should be part of the Continuous Integration build too; those should be written so they run fast and report clearly, so they encourage developers to use them and they don't slow down the CI build. However, you should also have some deeper, more broad ranging tests to run after the CI build was successful and passed all its unit tests. Once you have that, you have the real deal, Continuous Integration. So our list of First Principles of Continuous Integration now adds:

Run the builds often enough that only a few changes are picked up in each build.
Unit tests must pass 100%
Once that's accomplished, some function and integration tests should be run, and the build is only good if they also pass.

The reason Continuous Integration is so good that it's become as popular as it has is this: every project is going to have some developers who understand some parts of the design a little bit differently than the others, and when they code according to their idea of the design, errors can crop up that were very hard to expect. If you have enough testing soon enough, you get two advantages:

The problems come to light soon enough after they were created that everything is fresh in the minds of the developers;
and the design misunderstanding may be clear enough that the perpetrator of the problem gets a new, better understanding of that part of the design.

Back in the bad old days, we didn't have this to keep errors from being overlooked until the end of a project, and as a result 80% of all projects were tragic failures. Don't miss the opportunity to improve those odds to the moderns level - it's not as hard to do as trying to fix bugs weeks after they've been buried under layers of dependent code.

Test-driven development: Get done on time with confidence

2016-05-09

Test-driven development in its original extreme form is very hard to sell to developers, because it feels like it's slower than traditional style developing.

It's not. It's way faster.

The problem is, that doesn't show up on the first couple hours, it takes a couple of weeks for it to start taking hold.

Now, this probably sounds too good to be true. Kind of like when people say that programming in a functional language without variables is more productive and satisfying - you don't believe it until you give it an honest try, and an honest try requires overcoming some pretty deep-seated habits.

However, you also don't have to go all the way to "write a test first for every feature and run the test to prove it fails when you haven't even written the code to implement the feature it tests". You can make it a little less uncomfortable if you write your first test at the point where you would normally be testing your freshly minted code to make sure you did everything right.

And that's where the big gain starts to appear. Every job requirement for a developer contains a phrase like "very detail-oriented". What that really means is "ery, very careful to not write stuff that won't work correctly the first time". Nonetheless, most of us don't get code that works perfectly the first time even 50% of the times we try, because it's terribly hard. There are so many things that can go wrong.

So if you write a test for each basic functional requirement of the code you're writing, that runs fast and reports sensibly, so you can run it after every time you save a file on disk, you get something you didn't really think about: confidence without that stress of trying to be perfect in your first draft coding. A huge source of stress is disarmed!

Think about it. We all know that the Waterfall method doesn't work, and has never worked, because there is no such thing as a perfect specification - you will discover half the true spec during the project, always. Well, that also applies to coding. You take a good shot at it, and then you test it until it's solid when tested against whatever you can think of that may throw it off.

Now extend this over the period of a project. The old fashioned way, your confidence follows a pattern like this:

You start out with however much confidence you naturally have in your first draft, but since testing is a long time down the schedule, your confidence will erode steadily over time, especially as the specs become more clear and you go back to rewrite portions that were written under the original wrong assumptions. Then, way later, you throw the thing over the wall to QA and start to feel a little confident.

QA sends back a raft of bug reports and the causes of the bugs may be buried very deep in the code you wrote a long time ago, and may be subtle side effects of very basic classes in your design. Troubleshooting and unearthing the root causes is a long, hard, subtle process that takes a long time and is not certain to be completely successful.

Now let's compare that with your confidence when the functionality of your classes is tested every few minutes while you're developing:

Here you start with higher confidence because you don't write as much before you start having automated, repeatable, lightning-fast testing to use every time you touch anything, and every change you make from then on can be run through it again. Now you have no deeply rooted mistakes causing trouble in later times, because you see every error immediately, while it's still fresh in your mind.

QA can take this code and test for functionality and integration and Acceptance and performance as soon as there is enough function to make any given path work, and if you find a design problem, you can change the tests to reflect what you learned too, and make sure design changes go in exactly the right place and don't have any unexpected side-effects, which is a common problem in projects done the old way.

So bottom line: if you want to get done on time, and sleep well at night, and be proud of your work, you write tests along with the basic classes and all the later stages as well, and run them constantly, so you don't have to be perfect. Instead, you can just be productive.

Yeah, this is theoretical, and your mileage may vary, but once you try using detail testing early and often in your cycle, you'll never go back. It would be too scary.

Introduction

2016-05-05

I am a long time Build/Release/Automation/Devops engineer. When I started doing this, it was rare to find a company that thought that this role would be worth paying somebody for, and nobody thought there was an organized engineering discipline involved. I convinced them differently by learning about Software Development Life Cycle ideas and patterns and finding ways to apply what I learned in the real world.

Over time, it has become more accepted that this is a role most software development projects will benefit from a solid practitioner of, but it's a lot like being a bass player. If you play perfectly, nobody knows you're there, but if you stop or make a mistake, the spotlight comes on you, and it's very damn hot.

There have been various currents during my time in this field, that have seriously changed how things were done:

Agile development: At heart, this is a way to get business people and developers to work out how to communicate in spite of the fact that, up to that time, they had hated and feared the people they now have to work with (not usually in such stark terms, but that was how it felt)

Continuous Integration.: The idea that, if a bunch of programmers are working on a project, it will be a good idea to build and test it frequently, so that when somebody breaks something it's still fresh in their mind. Seems obvious, now that we know how much it helps.

The cloud: It's taken longer than it probably should have to start, for the obvious reason that a business needs to be able to be confident that putting their Intellectual Property on somebody else's storage is not exposing it to theft. At this point, files in AWS are probably safer than ones in your company's in-house data center, if my experience of security practices in software firms is any indication.

Test-driven development: Also, acceptance or behavior driven development, etc: the integration of testing earlier than before in the development cycle. This has the potential to be the biggest win of all, but isn't getting as much attention as the others. Ruby has a language community around it where this is well developed, and wow, what a difference it makes. Your confidence in your product starts much higher than in traditional approaches, and then stays there while traditional approaches erode it to nothing.

Distributed version control: Actually, this is not the revolution people think it is, because for a project to be done by a bunch of developers, you have to have a central master copy, but what this does give you is much, much better branching, and some thought has been applied to this that can make things work much better than they used to.

Infrastructure as code: People have wanted to be able to define what a developer's workstation would be like in a way that could be duplicated without effort or strain, and the same for oceans of servers, and now we have Chef, and Puppet, and Ansible, and lots of other ways to make this dream come true.

Virtual machines and Containers: ... like Docker. Now you can run many clean, lightweight, but perfectly sufficient environments on hardware that could never do so much before, and combine their efforts to get great results. This way lies scalability; nothing else that has come along has as much promise as this.

Functional programming: It's not all the rage now, but trust me, it will be everything. Why? Because Moore's Law. You see how CPUs stopped getting higher gigahertz rates? It's because it's not physically possible to go faster, so we are getting more cores. With Functional programming, you can use all those cores without having to do locking and other fragile, unreliable tricks. Doing that with the languages we have been using is a nightmare for even the best of us, which most of us are not. Get used to Amdahl's law: if you divide a job up equally among 4 processors, you get 16 times more done in the same time.