Why Maths Matters

A few months ago I was working on a piece of code that polls a service for a set of data, then for each item in the set, sends a request to a second service.  This was just one part of a pipeline of data processing.  My first naive solution was to just take a guess at the polling interval and the buffer size of the sets and hope for the best.  Then, I made these two parameters configurable.  Now, someone else is responsible for the guesses.

The actual problem is that I was irresponsible.  The situation called for precision in my design and I was being casual.  I focused on the plumbing (i.e. HTTP requests) and not the underlying questions that needed answers.  For example,

  • How big a buffer do I need for the data set?
  • How frequently must I poll for new data?
  • How frequently can I push data out? 
  • How can I reduce the push rate if the receiver cannot accept data faster than I can give it?

One very precise model comes ready made and is known as the leaky bucket problem.  The leaky bucket solution is applied very often in low level networking solutions, but is just as applicable in my problem space.  Now before you roll your eyes and fake a yawn at the maths behind this, hear me out.

The moment we go into higher order control systems, we need higher order maths to build precise mathematical models of these systems. That is not the maths that I’m suggesting we chase.  Instead, with a little bit of creativity and some strategic design constraints, we may be able to reduce many classes of systems to first and second order systems. In other words, introduce constants until you have one or two variables that you can tweak.

This is exactly the situation that I had.  Sure, there were more than two buffers to be managed, but I was in a position to fix the size of some buffers or the flow rates.  Then, the buffers and flow rates that were critical to me could be modeled with straight forward linear equations.

Had I done this, then I would have been in a position to have built a system that could have adjusted itself to a steady state, without frequent reconfiguration.  The fact that I was casual about my design led directly to a case of the pipeline being in a non-deterministic state.  This problem was highlighted early when the users started asking for different kinds of “status reports” for data flowing through the pipeline.  Of course, being casual about it, I treated it as a feature request and implemented a few reports.

This is when maths and science make a difference in software development.  Unsurprisingly, mathematical models are generally deterministic, self contained (i.e. highly cohesive), and at the right level of abstraction for the problem.  And all of those characteristics lead to highly testable models that you can do in wee little steps, test first.

That’s why it matters to have a maths background.  And if you came into software development through some other route, then do some layman studying of algorithms, control systems and simple higher order maths.  It will serve you well forever. It will certainly give you a design advantage when you need it most.  Right now, I’ll take any advantage because design is just so darn difficult.

What’s the point in Scrum?

Scrum people like to use points for estimating and measuring velocity.  I won’t go into detail about how points work and how to play those poker estimation games.  Just search around and you will find a ton of stuff.  So, back to this points stuff.  I have a divided relationship with the humble point.  I like it when a team switches to using points for the first time, because it gives them a chance to think a little bit deeper about what they want to do.  I don’t like it when we start inventing rules around points (and you can lump guidelines and best practices into the rules pot too).  When the rules appear, the thinking disappears.
In every team trying Scrum, there is bound to be a rule about points.  I dare you to put up a hand and say you have none.  These rules are things like “We can’t take anything over 13 points into a sprint”, “Our epics are 100 points”, “The login screen is our baseline of 3 points”, “Anything over 40 points must be broken down”.  So, I double dare you 🙂

Sprint backlog shape with high shared understanding

I have different view of the humble point. A point may seem like a one dimensional thing, but it has a some facets built into it.  One facet is the “amount of effort to build something”.  Another facet is “amount of ignorance” and this has an inverse – “amount of shared knowledge”.  Sometimes I find it useful to make a judgement based on what I don’t know as opposed to what I do know.  Regardless of whether I choose to view the cup as half full or half empty, I cannot estimate effort to build something based upon what I don’t know.  So, effort tends to track the amount of knowledge, not ignorance.  As knowledge increases, my ignorance decreases and each point starts representing more and more of pure effort.

However, if I am in a state of complete ignorance, then it is completely impossible for me to make any judgement on effort to build.  I’d be simply speculating.  What I can do, though, is create a time box to explore the unknown so that I can start moving out of my state of ignorance.  This is also an estimate and I am not making an excuse for non-delivery either.  I need to understand some things and also show my understanding in some code.  Yes, the code that I produce may not have a visible user interface or some other convenient demo-friendly stuff, but I need to carefully plan my sprint review to express my understanding.

It’s all about gaining a SHARED understanding. This understanding is body of knowledge that I have learned which I need to confirm with others.  This act of confirmation can happen in several ways.  I can have a conversation and explain what I understand, I can draw a blocks and lines picture, or show a spreadsheet, and so on.  Regardless of the method of communication, I still use the opportunity of discovery to express my understanding in code as tests.  Another powerful way of expressing my understanding is to write out a story and a few scenarios.  Using BDD style grammar can be a great way of concisely expressing some things, that can be easily shared.  Yes, you heard me correctly – as a developer, I write the stories and scenarios.  When I am given a story and scenario by someone and asked to estimate, then I am attempting to estimate based on  another person’s expression of their understanding and my assumed understanding.

In a recent discussion with Jimmy Nilsson, he said that he prefered to call scenarios “examples”.  That really resonated with me.  I also do a lot of discovery by example, and then gradually introduce more a more into the examples, as I get more and more confident of my knowledge.

How do I know how much I don’t know? That’s a tough question.  What I do comes straight out of my TDD habits.  I create a list of questions – my test list.  For some questions, I will know the answer easily, some not all, and some are debatable.  The more that I can answer, the better I can estimate effort.  I can then turn the questions that I can answer into statements of fact.  The more facts I have, the less ignorant I am.

Recently, I worked with a team that wanted to get TDD going, and the most significant change that I introduced was in backlog grooming and sprint planning.  During these two ceremonies, we (as a team) threw questions madly at a requirement, regardless of whether we knew the answer or not.  We then worked through the questions (as a team) to establish how much we could answer.  The trend that emerged was that the original estimates where either half of the new estimate or double of the new estimate.  When they where halved, it was generally because we were able to negotiate some of the unknowns (the ignorant areas) to a future sprint with the product owner.  In some cases, the product owner was equally ignorant, and was reacting to the “business wants the feature” pressure.  When they were doubled, it was so much more was discovered than originally assumed.  At the end of the session, we always asked the meta-question “If we answer all these questions sufficiently, will we be done?”.  I call this style of working “test first backlog grooming” or “test first sprint planning”.

Often I discover more things I don’t know. Annoyingly, this happens in the middle of a sprint, but if it did not happen in that phase of work, then perhaps I was not digging deep enough.  When this happens, I just keep on adding them to my list of questions.  These new questions are raised at any time with others on the team, the customer or with whoever can help me understand a bit more.  Sometimes, it’s put on the table for negotiation to be dealt with at another time.  Nevertheless, standups still seem to be a good time to put new questions on the table, for discussion later.

There are several ripple effects of thinking about points in this manner – this notion of ignorance and shared knowledge gauges.

The first is about the possible shape of your sprint backlog. If you have deep understanding, then it is likely that you will be able to decompose complex problems into simple solutions, that take less effort.  The effect is that low point stories are in greater number in a sprint.

If you are highly ignorant, then the estimation points reflect that and there are more medium to high point stories in the sprint.

The second is about what you value in a story. You will find less value in the ontology of epics, themes and stories.  It is no longer about size of effort but degree of understanding or ignorance.  Instead, the shape of the product backlog is something that is constantly shifting from high uncertainty (big point numbers) to high certainty (low point numbers).  That’s what test first backlog grooming gives you.

The third is about continuous flow that is the nature of discovery.  When you work steadily at reducing your degree of ignorance, then you are steadily answering questions through answers expressed in code, and steadily discovering new questions that need answering.  This process of discovery is one of taking an example based on what you know in this moment and modeling it.  Then expanding that example with one or two more additional twists, and modeling that, and so it goes.

It also touches product ownership and software development. When you work in this way, then explicit estimation of effort becomes less significant.  Moments that have been earmarked as important  points in the life of the product become more significant.  Call them milestones.  These milestones are strategically and tactically defined, and become a dominant part of product ownership.  Software development becomes the act of having long running conversations with the customer.  Those milestones give context for the content of those conversations.  Ultimately, those conversations are then expressed as a set of organised thoughts in code.  If your code is not organised well, then perhaps you also don’t understand the problem or solution or both.

This is a long story for a short message. A high priority is to resolve the tension that exists in an estimation in the form of knowlege/ignorance fighting against effort.  When you release that tension through shared understanding, then you can deal with the tension that exists in the act of creating those significant milestones.  In my opinion, that’s the real wicked problem.

Modeling out Loud

I will be running a 6 hour long session at the Scrum Gathering in Cape Town in September titled Modeling Out Loud.  I’m now convinced that the Scrum tribe are weird.  They call these sessions Deep Dives.  Presumably, you need to carry enough oxygen to survive the session.
I think I’m going out on limb here because I will be challenging the value of Product Owners writing stories.  I’m also suggesting that when Product Owners write stories riddled with behavior then developers are disconnected from domain experts and you regress into a waterfall mode of execution fronted by a Scrum Board.  So be prepared to experiment with me and turn up your self-reflection to maximum level because we will challenge many assumptions.

DDD Reference Card

I know it’s absolutely insane to try to reduce Eric Evans’ amazing book into just a few pages, but stupidity won.  I think it’s still useful as a “next to the coffee mug on your desk thing” if you’re just starting off with DDD.  So download the free Domain Driven Design Reference Card at http://refcardz.dzone.com. Small warning: it’s not useful unless you’ve read Eric’s and/or Jimmy’s book or have attended a DDD course.
I’ve tried to keep it true to the book.  I’ve aslo added a reference to a couple of additional patterns right at the end, after some quick chats with Eric and Jimmy.  Given the space constraints, I decided to leave out the CQRS work.  It feels better anyway, since this meant as a cheat sheet for people starting out.

Readability is the real (re)usability

Last week on the factor10 DDD course in Cape Town, the question of reusability came up again.  It’s the same old object orientation promise of “Just do OO and you get phenomenal reuse for free”.  Today, I was refactoring some code with another developer at a client and I extracted some lines into a few private methods just to clean up a really fat loop.  The initial reaction from the other developer was “That’s an overkill because you won’t reuse that method”.  My spontaneous reaction was “Readability is the real reusability”.
It’s true, the method won’t be reused.  It’s also true that most us were taught in some Programming 101 course that you should create a method, function, procedure only if you are going to call it more than once, otherwise just leave it all inline.  I value ubuntu coding, and so I have learned to unlearn that naive rule.  When I make my code more readable, I get more reuse out of it.  The reuse I value is not really about the number of repeated method calls or number of inherited classes.  I value the increased reusability that is achieved when more developers are able to read my code and walk away understanding my intention, clearly and unambiguously.

Let me put it another way.  Your code is a representation of your model.  Your model should be used to drive all collaborative discussions about the solution.  That’s where you get the real reuse in your model.  If people can’t understand your model, then your model can’t be re-used for further discussions.

Are you coming to OOPSLA?

In a couple of weeks I will be at the OOPSLA conference in Orlando, USA.  I am absolute OOPSLA nOOb but am already excited about it.  I’ve heard lots of nice things from the OOPSLA “veterans” at factor10 and now I can’t really wait to get there.
I will be giving a tutorial on using AOP to solve some domain problems, not just removing the infrastructural noise from your domain models.  Also, I’ve been invited to be part of a panel on my best-loved-hated subject … modularity.  I will also take part in the Cloud Computing Design workshop.

There’s also an amazing line up for the other tutorials and OOPSLA still has a “Pay for 3 and attend 4” promotion going on.  Take advantage of it.  If you already signed up for 3, then just sign up for the 4th.  If you’ve signed up for 2, then pay for the third and register for the 4th too.

So much happening in just a short week.  But, it will be lot’s of fun and worth the 24 hour travel time from Cape Town.

Fast Track to Domain Driven Design

I finally got out of neutral and pulled together the first public offering our Domain Driven Design courses in Cape Town, South Africa.  Normally we give these courses on-site with people on the same development team but I thought it may be fun and inspiring to open it up to everyone for a change.  Now I’m all excited again and really looking forward to a diverse mixture of people. Hopefully, I will see some old faces and lots of new people.
The one thing I can tell you is that the course is a very immersive experience.  I really hate lecturing but I enjoy probing conversations and that’s how I give the course. I don’t have answers to the practical work and concerns are addressed as we go along.  As a result, the day takes unexpected turns and routes.  But in the end I get you to the right destination.  Come along; you will leave exhausted, but inspired!

Take the Fast Track to Domain Driven Design

about the course /  course contents / should you attend? / register for the course

factor10 has expanded its services in South Africa to include our advanced and
expert level courses aimed for the software professional.  On September 8-9, 2009,
we will be offering a fast track to DDD for Architects at the BMW Pavilion in
Cape Town.

zz485d34cf

Who should attend?

This course is for software professionals that want to take the right steps towards
advanced and expert levels in their careers.  Register for this course if you want to …

  • learn more than just another syntax and set of tools
  • write software for large, long living systems
  • increase the quality and maintainability of your design
  • design high quality models and use code to represent those models effectively
  • develop applications with a good APIs
  • add a design edge to your skill set

zz3caeb696

Why should you learn DDD?

More and more developers and architects realise that learning every detail of a new
API just isn’t the way to deliver the best business value. It’s such a tough balancing
act; focus on the solving the business problem and focus on building working software
with your frameworks.

One way of taking a big leap in the right direction is to learn and apply domain driven
design. It is definitely not abstract and fluffy; it deals a lot with the code also. DDD
leads us to focus on understanding and to communicate that understanding very well;
in language, in design and in code. You will shift your focus away from designing for a
technology, and you will learn to design for the business domain; to the core of the
problems and solutions. Those are the most interesting parts and what your users
and customers really care about.

about the course /  course contents / should you attend? / register for the course

Ubuntu Coding for your Friends

Last week I gave a domain driven design course and one slide I put up was titled “Coding for Friends” with a single message “Avoid conceptual corruption”. In other words “Code so I can understand you and you don’t screw up my mind”.  I did not realise the significance until I started working through the practical exercises with the groups and kept on referring back to this simple idea.
So often we write code in isolation and the code reflects our personal interpretation of the problem and a personalised solution too.  We easily forget that other people will execute this code, modify it, and, at the very least, read it.  Coding is a social exercise first, then a technical exercise.  We have a responsibility towards increasing the probability of success for the next person or team that will work this code.

We can write code in isolation that is of high quality, focusing on self and metaphysics of quality, etc.  That’s a zen view and it is about you.  I like to think (believe?) that Ubuntu is zen for a group, not for an individual.

In Zulu, the Ubuntu philosophy is summed up as

Umuntu ngumuntu ngabantu

which roughly translates to

A person is a person through (other) persons

And in geek-speak it is

A developer is a developer through (other) developers

I get better because you make me better through your good actions.  And the opposite also holds true: You get worse at what you do when I am bad at what I do.

How do we write ubuntu code?  It’s not hard.  It’s based on “old” principles and wise words of many people.  It’s old material but worth revisiting with ubuntu glasses on: when you think about what the effect is on other developers and what the reciprocal effect will be, back onto yourself.

Reveal your intention, not your implementation. If you have designed a project management app with tasks and decided to implement it as a graph, call the class Task and give it a method called resolveCyclicDependencies().  Don’t create a class Node with a method called topologicalSort().

Avoid side effects. Let a method do one thing, and one thing only.  It’s frightening how many developers don’t do this.  For example, if you add a line item to an invoice, don’t update the total.  Have something else that calculates the total.  Basically, be boring and predictable.  I long time ago a SQL guru looked at my code and said “The optimizer does not like fancy code”.  Same thing.

Broken metaphors. Metaphors exist for conceptual understanding.  Once you get the conceptual break through, leave it.  It’s ok! Trying to build an economic model with a metaphoric model of fluid flows (or bears and bulls!) will just create havoc downstream.

Where am I? Figure out where you are in the bigger picture.  Are you upstream or downstream?  Is someone going call my code to create that Book object? Am I going to call someone else’s code to get the Customer object?  In other words, know what you supply and what you consume.

Fail fast. Assert! It’s allowed in production code too.  I would rather fail quickly and dramatically than delay the effect until it is obscure.  More importantly, when I read an assert then I know that the particular condition is critical.  It is an invariant that must be honored.

Pairing. Programming in pairs is an active exercise, not a case of a second syntax checker.  I see many teams that pair mechanically and it does nothing for increasing code quality at all.  If you practice active pairing, you are closer to ubuntu coding.

There are other techniques which increase the collective responsibility of your design and code such as context maps in strategic design, values and attitudes such as responsibility and feedback.  I’ll deal with those on another day.

But for now, I think that

Code is code through (other) code!

Do you have any other ubuntu coding techniques? Attitudes?

Great Quotes for Domain Driven Design

I am revisiting the material for the factor10 Domain Driven Design course as part of a “freshen up” for the when I give it to a development team on July 1, 2009.  So I started hunting for some quotes to liven it up a bit.
Here are some choice bites for ubiquitous language.

“They have been at a great feast of languages and stolen the scraps” — William Shakespear

and

“Thought is the blossom; language the bud; action the fruit behind it” — Ralph Waldo Emerson

And on the need for simplicity, I really liked this one.

“There is never any justification for things being complex when they could be simple” — Edward de Bono

Robert Bravery commented on an old blog post of mine and I re-discovered this one

“Don’t indulge in any unnecessary, sophisticated moves.  You’ll get clobbered if you do” — Bruce Lee

Lastly, and the one I’m definitely going to use to stress the importance of working directly with the domain expert.

“Never hold discussions with the monkey when the organ grinder is in the room” — Winston Churchill

What a diverse group of people but such collective value.  Do you have any classic quotes?  Obscure quotes?

My Event Horizon

It’s already halfway through the year, so let’s see what events are is in store for the rest of the year.
June: Next week is SPIN week.  Join us after the June 16 public holiday.  Same time, same place.  Last month we had 40 people attend and we just about ran out of chairs.  Lots of new faces.  Please come along.  The talks are normally good and the conversations are great.

July: Really need to get my act together and get to the next SA Developer‘s talk.  Hilton Giesenow tells me it’s a great local event.  And On July 27, Lia turns 5!  Simply amazing!

August: Hmmm, seems quiet?  I think I’m going to submit a tutorial proposal for the ICSE conference happening in Cape Town in May 2010.

September: Looking forward to the Scrum User Group‘s conference in early September.  I may be giving a talk on the techie track.  I think it’s going to be a great event.  Carlo and Peter tell me so.  But on September 3, Fiona turns … older 😉

October: My tutorial on using DDD and AOP to create clean, rich domain models has been accepted by OOPSLA.  So, October is OOPSLA time for me.

November: I will miss Oredev.  The competition to get in was so much tougher.  That just means it’s going to be better than last year.  If you’re a S.African developer, you should make a plan to get there.  It’s a great event that is very high on value for money.  But my November highlight is Khaleel turning 9 on the 22nd!  Boy, how did that happen so fast?

December: Crazy season again.  I really hope it’s a quiet, relaxing one this year.