I was notified the other day of a discussion on estimates that flicked a few nerves.

The discussion surrounded eliminating points from the estimation process, attempting to break cards down to equal sizes.

I can understand the appeal, assuming everything averages out, however I don’t think it’s possible really to break down the cards to equal size chunks, and if you have varied sized cards, you lose the best thing to come out of agile methodology.

The separation between COMPLEXITY estimation, and TIME estimation. (I’ve ranted about that before)

There are two primary arguments that come out of developers who don’t want to estimate.

1 - I don’t want to commit to an estimate, and if I give you one I am committed

2 - If you want a number from me you don’t trust me to do the work

Both of these arguments are demonstrably false.  Additionally, they are somewhat contradictory, but often held together.  

If you don’t trust your ability to hit an estimate, then why do you expect your client to trust your ability to do all the work they require in the budget they have?

It’s a response to the fear of contract heavy development where planners don’t trust developers that swings the pendulum to the complete opposite direction, where developers don’t trust planners and refuse to allow them to plan.

It’s fairly self evident that it won’t work.  Any significant development will need some planning to happen around release dates.  This planning will be done whether you chose to engage in it or not.

By delivering accurate estimates, which are regularly checked, you are actually making it less likely that unreasonable delivery dates will be targeted, and making the discussion around moving them a whole lot more accessible and scientific.

Personally, I’ve lately been on a project where estimation was largely dropped for about 3 months.  

While we did deliver _something_ along dates which were presented to us, the process was not pleasant for anyone involved.  

There were many overtime hours and weekends worked (no way of measuring their positive or negative impact) there were many hurt feelings and complaints every time a new date or requirement came down.

The team had a fantastic level of skill, and the delivered product was a success, however there was not the sense of achievement and satisfaction you might expect after doing so.  In general people seemed to think we could have done a lot better.

Contrast this with my latest project, where estimation and regular analysis has been kept up for the 9 month start to launch.

The scope was bumped up by about 50/60% twice, and the team was doubled each time.

The original launch date was pushed out by 3-6 months.

There was of course, pressure to increase velocity and discussions were had, however they were accompanied by scientific data, the tone was never “you are not doing enough” because what we were doing was clearly visible in the burn up chart.

Once the planners moved the release date to account for the new scope and the achieved velocity, the project continued, as planned, and delivered a smooth release without any over time and while some people got a bit stressed in the final week, every single person I spoke to afterwards was satisfied with the result and happy with their achievements.

Now both of those situations were environments where the client already bought into the “agile” mindset, they were both reasonably flexible and had the ability to break away from “planned work” and focus on “delivering value”

I have been on at least two projects previously with pushy, borderline hostile clients, with less ability to shift from the established plan towards delivering real value.

In both projects I can think of, where estimations and velocity were tracked, the conversations over what gets delivered were framed in scientific reality, which goes a great length to cut through the political argy bargy of ‘you were contracted to deliver x dot points by y date’.

The problem seems to boil down to people believing that reasonable project planning, is on one side of an axis, with accurate developer predictions being on the other side.

It’s a false dichotomy, the two are in no way mutually exclusive.

Disclaimer:  I would probably start most new projects these days in rails, the amount of current problems that have been solved and the almost seamless way of integrating those solutions is an incredibly compelling offer.  The community is really rails and ruby’s strength.

Having done quite a bit in ruby so far, I’m finding more and more that I miss my type system and get annoyed by the kludginess of everything I’m using.

Hashes, Symbols, OpenStructs

<pre>

structure = OpenStruct.new({“foo” => 1, :foo => 1, :children => {:tim => “mah”}})

structure.children.tim

</pre>

Mixing and matching symbols is a royal PITA.  
If you’re working with a web framework, at some point it will convert all your symbols to strings, then you’ll think about symbolize keys, but that won’t do the whole tree if you’ve got a nested hash.

Addressing nested structures in hash format is ugly hash[“a”][“b”][“c”] and although OpenStruct makes this easier, converting between OpenStructs and Hashes is painful with nesting, not seamless.

I find hash rockets to be clunky and annoying syntax.  The newer {symbol: value} format is much nicer, but what if I don’t want to use symbols for my keys?

IMO Maps are a really important part of every language, and it bothers me greatly that in ruby they are not terribly fun to use

I don’t really like git submodules.

It makes sense for sharing modules across teams / Open Source groups, but within a single project, even bigger ones, I’m overwhelmingly in favour of single repos with sub folders.

I sometimes see people in a setup where they have already got lots of little repos though, and they don’t want to lose their commit history.

Here’s how you merge them but keep the history:

mkdir target
cd target
git init
touch .gitignore
git add .gitignore
git commit -m "initial commit"

git remote add -f first ssh://dius@git-server/git/lots_of_repos/first
git merge -s ours --no-commit first/master
git read-tree --prefix=first/ -u first/master
git commit -m "Subtree merged in first"

git remote add -f second ssh://dius@git-server/git/lots_of_repos/second
git merge -s ours --no-commit second/master
git read-tree --prefix=second/ -u second/master
git commit -m "Subtree merged in second"

Today I found myself in a discussion about technology choice and estimates.

Given the current nature of the market, frequent small projects, lots of push for new technologies, this is a common situation to be in.

We had looked at the problem, come up with a solution, done a bit of a story by story point by point break down and come up with a proposal.

As is a common story, we chose what we feel is the best tech stack for the job, however this popular tech stack is not what’s used in the clients organisation currently.

The client wants to try new technologies, has heard of the one we suggest, but is nervous and wants it compared against their existing tech stack in every way possible.

The suggestion came up to supply a percentage increase in cost for using the existing, slightly older tech stack.

This is where things get a little murky.  I had quite a bit of difficulty explaining why I thought just adding a percentage and running with it was a bad idea.

I’ve been playing with OmniGraffle a bit lately, so during the meeting a bit of a diagram popped into my head to help me explain it.

The problem is in relative complexity.

Here’s my first awesome diagram

Pretend frameworks, pretend stories, pretend everything.  I’m illustrating a point here, not presenting evidence.  Certainly not buying into the group think of bashing one particular technology and glorifying another (undeservedly, but that’s another rant)

Pretend blue is a 1, yellow a 3 and red an 8.

Note how you could take a 1 from the emerald framework, drop it in sunacle, and it could be a 1, it could be a 3 or it could be an 8.

Why is this so?  Well, there’s tons of reasons.  A particularly compelling, and easy to understand one, is the footprint of changes on the system.

Different graph of the same fake stories, this time, showing the layers of the system that need to be touched to achieve each user story.

Notice how, in the upper framework, there’s lots of layers, and little is left to convention in the layers.  Lots of things need doing to affect even small changes.

In the other system, some layers aren’t used, and for many changes, only a few layers need to be touched just to specify some unusual departure from convention.

The important thing to note though, is that it’s not linear across the system.
To make up an example, changing a bit of static text on the view templating layer is likely to remain fairly similar across both frameworks.

Add a field to the database and send an email however… well, in one system it’s close to as simple as the text change, in the other it’s a major endeavour.

I like to call this a technology profile.  Some tech stacks are simply better at some things than others.  This comes out when you do proper estimates across both systems.  If you try to take the estimate from one technology, and linearly extrapolate it onto another, you’re going to come unstuck.  

You can take a %ge number across the project, and come out with approximately the correct total.  

However if you do that, there’s a reasonable chance that the relative complexities you came up with with your original framework will be used to plan the new project.  Your velocity will look all kinds of weird, and you’ll generally look like you don’t know what you’re doing

Even worse, you might end up having the client accept the more expensive technology, then try to cut costs by trimming stories.  

They’ll use the original numbers you gave them, completely unaware that the original relative complexity is likely to be fairly useless now.

You can try putting a disclaimer in your quote, but that’s fairly certain to be ignored by the client.

My advice, would be:
firstly, don’t give two numbers
secondly, if you have to, do a full re-estimation
thirdly, if you can’t do that, then only give them total numbers, not relative complexity break down. 

script.rb

at_exit {
  puts "nothing to see here!
  exit
}

puts "error encountered closing system"
exit 1

execute:

ruby script.rb && echo $?

result

error encountered closing system
nothing to see here!
0

So this gets pretty messy when you’re trying to run rspec via rake in a rails project with a bazillion gems around all with their own at_exit hooks (capybara and the web-driver implementations were our problem) It can result in you having failing tests that your CI server don’t report on because it gets a success return code. Our first hack solution was to put our own at_exit block to check for errors and set the return code in spec_helper.rb:

at_exit {
  if check_for_errors
    exit 1
  end
}

Unfortunately some gems are loaded lazily and may append their at_exit hooks after our check and blow away our return code. So our second hack was to nest the at_exit in another at_exit just to try and make sure it’s the last thing that executes:

at_exit {
  at_exit {
    if check_for_errors
      exit 1
    end
  }
}

We’re entering some serious WTF territory there.

It looks odd and it’s still possible that some crazy gem down the path loads a new gem within its at_exit block which causes additional at_exit hooks to be appended. So it’s a fragile solution. Technically we can just keep nesting at_exits in a sort of arms race scenario to try to keep our check at the end of the cleanup code.

The alternative I suppose would be to write the status to a file or other persistance store somewhere and write some specialised code in our rake task to check that store when the tests are finished. I’m not happy with the idea of either solution.

Ultimately it just seems wrong that a call to exit with an error return code gets wiped out by some other cleanup code. I don’t think it quite qualifies as a bug, but it’s definitely a design decision I don’t agree with.

This is another reason why I like Scala more than ruby :P

Today I made a disturbing discovery around the performance of the watir-webdriver framework on firefox.

I inherited some tests that were asserting values on a fairly large html page (12000 lines) within a fairly nested and rigid structure

It looked something like:

div(:class, "outer").ul(:class, "results").li(:class, "result").div(:class, "header").span(:class, "value").text.should_be("7")

Which is apparently the correct and recommended way to query the dom.

When I run this test however, it took 270 seconds to execute!

My belief is that watir is re-building and re-interrogating the dom tree for each element invocation (or causing web-driver to do so, which is equivalent so far as I am concerned)

I migrated this particular test to use capybara and replaced it with the equivalent selector:

find("div.outer ul.results li.result div.header span.value").should have_content("7")

The test execution time dropped to 12 seconds.

Now I wouldn’t expect most tests to experience such a severe difference, the html I was working with is quite a lot of data.

However I think it’s worth knowing that those chained selectors are actually costing you a reasonable amount in test execution time.

Even if it’s only a second or two per test, if you’ve got a high test coverage that’s going to add quite a lot to your test runs. 

Functional tests are already painfully slow, so when you write them, pay attention to doing what you can to speed them up

Ok, so I’ve been complaining to my friends about Maven a LOT recently.  It has taken me an embarrassingly long time to grok it even the small amount that I have.

So, basically you get phases in which you can invoke goals, which are where the work happens.  Unusual goals are generally provided by plugins, for example, the maven-exec plugin lets you execute arbitrary commands.

A common thing to want to do, is execute the equivalent of exec:exec right after the compile phase, but before the test phase, and have this happen automagically when you run mvn install.

So to do the execution bit, you declare an execution for the plugin within the build.  I like to think of this as an augmentation to some existing structure.  in your pom.xml it looks like this:

<project>
  <!—blah blah —>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.maven</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <executions>
          <execution>
            <id>whateverYouWant</id>
            <phase>compile</phase> 
            <configuration>
              <executable>dir</executable>
            </configuration>
            <goals>
              <goal>exec</goal> 
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

Yeah I know, verbose right?  

I’m not a big xml fan either, but if we look at it there, you’re basically saying that our project has a build structure, in that build structure some plugins will be involved, this specific exec-maven-plugin is involved at that compile phase, where we want it to run the exec goal with the given configuration.

My first mental flip was having to recognise that the build doesn’t just have phases and goals grouped together, but spread out between plugins.

So another common thing, is to only want this exec thing to happen under certain conditions.  That’s achievable using a profile.  Profiles can be combined, so there are multiple of them, and can be switched on and off either with the -P flag when invoking mvn or using an <activation> block which lets you check for environment variables etc.

within the profile block, you can put more or less anything you put in the project block and it works kind of like an overrides/extending piece.

<project>
  <!— blah blah —>
  <profiles>
    <profile>
      <id>whateverYouLike</id>
      <build>
        <!— insert the plugins block from above —>
      </build>
    </profile>
  </profiles>
</project> 

So that’s the basics of it… to invoke the build with the above profile, because there’s no activation block, you’d have to invoke mvn install -P whateverYouLike

Getting into activations and lots of profiles, some turned on and some turned off is where things get a bit more complicated, but with the above starting blocks I’ve found the rest a lot more approachable.

It’s actually described in the maven docs at:
http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
however for some reason I found that to be fairly difficult to grok… maybe it’s the higher level build construct with overrides that I needed to get to make it all fit.

Anyway, hopefully this helps someone else out there as confused as I was 

Ok, so this is a half baked Idea I’m just jotting down…

Business programming world uses a lot of lazy loading ORM fanciness, this is convenient for programmers but bad for performance, and also clearly bad for statelessness

Haskell has inferred types

Can we define a set of types that is the O model part of the ORM, that can be used to produce the necessary set of queries to retrieve all the data that will be observed in the course of execution based on inference at compile time?

That would lead to something like:

//in request handler

save(joinNames(loadData(databaseConnectionDetails)))

public UpdatedModel joinNames(initialModel) {

  val allNames = initialModel.names.join(” “)

  return initialModel.withAllNames(allNames)

}

So that’s probably not as descriptive as I’d like, but the general idea being that you’re reading a bunch of name fields, which have been mapped to database tables in the definition of InitialModel.person you’re joining them together and using the result to update another field similarly mapped.

The InitialModel and UpdatedModel types theoretically (and this is the really hard bit I imagine) decompose and infer enough to figure out that you only care about the names field and the allNames field, and therefore the program generates only the required SQL for the loadData and save functions.

I can’t imagine this hasn’t occurred to people before, so if anyone has some links for me to read on it I’d greatly appreciate it.

It occurs to me that dynamic languages make easy stuff easier and hard stuff harder.

It’s tolerable in the same way being bi-polar is