Why am I poking SUGSA?

I’ve been taking a poke or few at SUGSA on twitter in the last few weeks around the2013 South African Scrum Gathering.  It’s time to put it into perspective.  My proposal to present a session at the conference been accepted and I have declined this acceptance.

This is not news to the SUGSA committee.  I have communicated my
decision and my reasons to the SUGSA committee.  I have also had
one-on-one discussions with a few committee members.  I am satisfied
that the committee has received and digested my reasons.  Individuals on
the committee have assured me that my feedback is in line with their
values and that they aim to address it in the future. 

My poking is just me pressuring SUGSA to recognize a few issues which I believe are important to our community, and to force the committee to address us on their position.  Tell all of us, not just me, of your position and we shall listen.  That is all, no more, no less.

These are the issues that I raised with SUGSA.

  • African conferences need to be representative of our community.  One way is to nominate an African keynote.  It will take a courageous decision but it can still happen.   Agile Africa responded symbolically very late and it was fantastic.
  • We must embrace our diversity and find ways of breaking the polarised community that exists in Cape Town.  I think the people from Cape Town that attended Agile Africa in Johannesburg were surprised by the diversity of speakers and audience at that conference.
  • Be transparent with the review process.  The committee needs to maintain its independence and having committee members occupying several speaking slots sends the wrong message.  I’ve seen that in other communities where the committee became the soapbox for individuals.  If I was on the committee, I would withdraw my proposal on principal.  I know of one committee member that did this.  Thank you for taking a courageous step.
  • I also questioned the expertise within the committee, especially for technical submissions.  That is my perception and there is every chance that I might be wrong.  I welcome the correction. 

This is not a boycott.   I will still attend as a paying conference attendee.  I appreciate the
learning I receive from this community and look forward to more of the
same.

I believe my issues have been heard and my actions, seemingly harsh to some, was necessary.  I still want SUGSA to take a position publicly and I leave it them to figure out when and how.  The SUGSA committee have invited me to discuss this further and  I look forward to that.

Thanks go out to Austin Fagan, Sam Laing and Karen Greaves who gave me perspective to resolve a few internal conflicts, and shape my thoughts on why I am doing this, and what I expect out of it.  Had they not given input, this could have been unnecessarily ugly.

Why Maths Matters

A few months ago I was working on a piece of code that polls a service for a set of data, then for each item in the set, sends a request to a second service.  This was just one part of a pipeline of data processing.  My first naive solution was to just take a guess at the polling interval and the buffer size of the sets and hope for the best.  Then, I made these two parameters configurable.  Now, someone else is responsible for the guesses.

The actual problem is that I was irresponsible.  The situation called for precision in my design and I was being casual.  I focused on the plumbing (i.e. HTTP requests) and not the underlying questions that needed answers.  For example,

  • How big a buffer do I need for the data set?
  • How frequently must I poll for new data?
  • How frequently can I push data out? 
  • How can I reduce the push rate if the receiver cannot accept data faster than I can give it?

One very precise model comes ready made and is known as the leaky bucket problem.  The leaky bucket solution is applied very often in low level networking solutions, but is just as applicable in my problem space.  Now before you roll your eyes and fake a yawn at the maths behind this, hear me out.

The moment we go into higher order control systems, we need higher order maths to build precise mathematical models of these systems. That is not the maths that I’m suggesting we chase.  Instead, with a little bit of creativity and some strategic design constraints, we may be able to reduce many classes of systems to first and second order systems. In other words, introduce constants until you have one or two variables that you can tweak.

This is exactly the situation that I had.  Sure, there were more than two buffers to be managed, but I was in a position to fix the size of some buffers or the flow rates.  Then, the buffers and flow rates that were critical to me could be modeled with straight forward linear equations.

Had I done this, then I would have been in a position to have built a system that could have adjusted itself to a steady state, without frequent reconfiguration.  The fact that I was casual about my design led directly to a case of the pipeline being in a non-deterministic state.  This problem was highlighted early when the users started asking for different kinds of “status reports” for data flowing through the pipeline.  Of course, being casual about it, I treated it as a feature request and implemented a few reports.

This is when maths and science make a difference in software development.  Unsurprisingly, mathematical models are generally deterministic, self contained (i.e. highly cohesive), and at the right level of abstraction for the problem.  And all of those characteristics lead to highly testable models that you can do in wee little steps, test first.

That’s why it matters to have a maths background.  And if you came into software development through some other route, then do some layman studying of algorithms, control systems and simple higher order maths.  It will serve you well forever. It will certainly give you a design advantage when you need it most.  Right now, I’ll take any advantage because design is just so darn difficult.

An African Keynote

Now that we are attracting international speakers, does it bother anyone that we have never had an African keynote in any of our conferences.  I can understand that people want to learn from people to whom they don’t normally have access.  But why don’t we have an African give the keynote?

Before I go off bleating about how this bothers me, I thought I’d understand this thing called a keynote.  It originates from music, a capella in particular.  At the start of a performance, someone sang the key note and others followed, calibrating to that note.  In a similar vein, a conference keynote is meant to set the tone for the conference.  It lays the foundation for the other speakers to build upon.  It calibrates the conference for the speakers and attendees.  If the keynote inspires, all the better.

What should we expect of anyone that is giving a keynote?   The first criteria is that the person must have good insights into the subject of the conference.  The person does not need to be an expert, but should have a good understanding the subject and the context thereof for the audience.  Second is to have the ability to deliver a decent talk, one that shapes the conference.  Being a great speaker is either a talent or a learned skill. Either way, a person gets better the more they speak.  Lastly, the person must speak with passion, from the heart and create an energy that inspires, even if it only lasts for the duration of the conference.

What, then, does it take for someone to give an African keynote?  I think it is the most important to understand Africa, her people, her history, her challenges and her future.  Without this, an African software conference lacks context for our the challenges and constraints.

It’s great to hear someone from Amazon or Facebook about their grand ultra-scalable, cloud infrastructure.  Sure, now try that in Africa when a break in an undersea cable is the equivalent of severing your carotid artery.  Then try that in Africa where the minimum living wage is equivalent or less to the bandwidth costs for a month for an average middle class home.  How do you build a scalable cloud infrastructure that reaches into the tiniest village in, literally, dark Africa.

Forget technical challenges, let’s take agile conferences where trust, collaboration and self-organising teams are common themes.  And magic and transformation too.  If you grew up in Africa, then you will understand that trust is not a commodity item.  When an entire continent of people have been exploited as slaves for centuries, cultures systematically eradicated for the wealth of privileged few, establishing trust and collaboration is a different ball game.  Try transforming a team with distinct divisions, each with deeply entrenched prejudices for each other.  This is Africa. She is not your average, homogeneous nanny state that tells you what to think.

I believe that we do have people in Africa that easily meet all these criteria, and we are certain to have more in coming years.  A couple or years ago, ICSE was held in Cape Town and Archbishop Tutu gave the keynote address.  Here is a person of Africa addressing the academic elite of the software world, and he is not even a geek.  We don’t need to have instantly recognisable names.  From the top of my head, here are three names that you may or may not have heard that are more than qualified to keynote African software development conferences.

  • Enyo Kumahor
  • Herman Chinery-Hess
  • Jonathan Jansen

Some technical topics transcend culture.  For example, object orientated or functional programming can be learnt regardless of who we are and where we come from.  And in a localised way, a few refactoring steps to clean out a piece of smelly code doesn’t impact on our culture or being.  That I can accept.   What I cannot accept is when this reasoning is abused to deflate the situation of under-representation of our people. Let technical things be technical, but we also need Africans to be technical leaders. That is not the topic under discussion here.

This is important. Whether they like it or not, every African keynote speaker creates the path for many to walk.  Even if it is not a keynote, an African that shares a room with a few people gathered around, listening and engaging with what is being shared is cutting down brush and laying a new path.  These will become well worn pathways.  And each traveler walking on it will see virgin brush that needs to be cut, and a new path will be laid.  An African keynote can have this effect, if we allow it to happen.

I have a challenge.  Next up is Agile Africa which will happen in August 2013.  This is the keynote lineup:

  • Martin Fowler
  • Mitch Lacy
  • David Hussman
  • Amr Noaman
  • Ivar Jacobsen

Amr, from Egypt, is the one African in the keynote lineup.  That is fantastic. I challenge the JCSE to rather reduce the number of keynotes to just two or three and nominate other Africans to stand alongside Amr Noaman. 

In October is the Scrum Gathering in Cape Town.  Here’s the keynote lineup.

  • Geoff Watts
  • Dave Snowden
  • Alexander Kjerulf

I challenge the SUGSA committee to offer these speakers a regular slot and give our African community African keynotes.  I am not asking for the current of the keynotes to be rejected.  I want these experts at our conferences.  I want to attend their sessions and learn.  I also want us to realign them to an African context, that broadens their perspective.  We share, we learn, we teach, we grow – it’s all the same.

But  I cannot accept that there are no African keynote speakers to be found.

Get up, Stand up.

I first got paid to write code in 1992.  The few years before were the heavy days of mass protest against apartheid.  Having a common enemy was not what drove people to change.  It was having a common belief that we are all equal and deserve to be treated equally. It is that simple.

In the years that followed, my career always surfaced a blatant fact.  I was always the minority in a white, male dominated industry.  In my first job I was hired by people that also despised apartheid.  For that I am grateful because it gave me a chance to learn without fighting for place.  But the places in which I plied my trade were dominated by white, male developers and managers with an attitude of superiority by entitlement, birth, law and any other decree.

And today I am still the minority in a white, male dominated industry.  I don’t need hard stats and precise surveys when it is patently obvious from the demographics at community events, conferences, start-ups and cubicle farms that black software development has not gained a foothold in our industry.  It is especially so in Cape Town, and only marginally better in Johannesburg.  I can’t speak for Durban because I haven’t worked there since 2005 but it was not any better then.  It is not just South Africa.  It is the same in Europe and North America.

 There are at least two sides to any story. Here, the one side is with corporate South African software development as a conduit for opportunity or suppression thereof, and the other side lies with the black software developer with messed up priorities.  By the way, when I say “black” I mean “not white”, and I everything I say is a generalization because I believe it is in the majority.

In my experience corporate South African software development perpetuates apartheid.  There are official policies and frameworks for employment of people and for furthering their careers.  There is nothing wrong with these frameworks. But like all rules, how we use them matters.

A few weeks ago I had another black developer share his entrance into a prominent consulting company.  He was offered an internship whilst a white male of equal qualification was offered a full contract.  When his internship was over, he was not offered a contract on the grounds that his line manager did not like his “work ethic”.  Really? Who’s work ethic needs to be questioned?  Today, this developer is employed as an equal in more progressive organisation with a highly respected brand that is determined to breakdown apartheid in South African software.

I am yet to meet a black software developer that is leading a team, making the call on design decisions.  Along the way, I have met many blacks in leadership positions. Many are tokens and lack the leadership and technical skill to justify their position.   This is also the fault of these individuals, exploiting the rules for self gain.

The majority of black software developers are either doing “maintenance” or being instructed on how to write to specification or requirement with a full blown design being given to them.  I have met many black developers who I’ve seen argue their design ideas only to be later classified as not being “team players”.  This also occurs in the sugar-sweet agile “we-love-transformation-i-love-my-job” work places.

What gives credence to the “inferior” black South African software developer is that black software developers are apathetic.   I find it difficult to identify with someone who voluntarily accepts being treated as a second class citizen.  I can understand the risks of 20 or more years ago, but life today is different.  We have a voice and we have a deliberate choice of the battles that we can fight.  This is mental slavery all over again.  And to some extent this mental slavery is self imposed.

I can appreciate that someone with a history of poverty that is offered a monthly salary that their parents could only ever earn over several years, is driven by a more basic need and deep empathy.  It goes horribly wrong when that is all there is to it.  You earn your dough, but you don’t earn your stripes.  I know several black developers that have doubled or tripled salaries in a few hops inside of two years.  These developers do nothing but give credence to the ingrained belief that black developers are not skilled.

I am fully aware of black developers working from a weaker maths and science base.  But I am also aware that there are white software developers that also don’t have a strong maths and science base.  I have wondered about this difference: two people with equally weak programming prerequisites yet the black developer struggles and the white developer excels.  I don’t think spoken language is a significant variable here.  The one thing that I have observed is that the white developer receives more opportunity and attention than the black developer by his white leadership.  The black developer in turn believes that they are inferior, and the lack of opportunity and attention furthers erodes self-belief.  Self-doubt is by the far the greatest obstacle to learning.

If you have read this far, there are several among you that have already formulated counter arguments, specific cases that justify your position too.  I am open to hear your side.  The problem with this blog post is that everything until now is just background to create context to put forward my belief.

I believe that there is a wealth of talented software developers in Africa that understands our history and our future. I believe that as a Pan African software community we are capable of designing frugal solutions for our people and for the world.

To achieve this, we must break free of our own mental slavery.  We need to build our own solid foundation that is unshakable at our core.  We must equip ourselves as masters of our craft.  We must let our voices be heard in Africa and the rest of the world as voices that speak substance.  And we must return to be with our people as no more than just people.  Our first step is to rid ourselves of apartheid in software development.  Like Marcus Garvey said and Bob Marley sang “none but ourselves can free our mind”.

I dream that the next thought leaders of software development will be African.

 

Reflections of BDD Stories

There was an interesting discussion on the AgileSA Linked-in group around the use of BDD stories, and whether they should contain technical references or not.  I found myself saying that I don’t mind having, for example, a login story. To help Kevin Trethewey get over the shock and horror of this, I reflected on how my use of BDD stories has changed over time.

I remember Dan North explaining BDD stories to me when it was just a thought in his own mind.  That was around 2005 or 2006 and I remember being so inspired by the simplicity of the BDD grammar. So, seven or eight years later, let me share how my use has changed.  And, Kevin, I hope you enjoy your own journey too. It’s a lot of fun.

On CRUD.  I agree that CRUD is bad, blah, blah, blah.  But there are times when CRUD can a valid, and reasonable design choice. I don’t discount it, but it is not my first choice, and it is very rare for me. Oh, I sometimes just use it because I don’t know anything else about the domain. Once I discover more, I noticed that those CRUD things, quite naturally, fade away.

Who is the best judge of a story?  The customer is unlikely the best person to articulate these stories, nor judge the quality of the story. I have to guide them and extract that. I now ask the following questions.

  • Who are they?
  • What do they need?
  • What do they think they need?
  • What do they really want?

What does the story describe? Of the above questions, the last is the most powerful for me.  It balances my perspective. It stimulates creativity and moves me from the problem space to solution space. The story then exists in the solution space; i.e. it is now reflects a design intention, not a requirement statement in the problem space.

BDD stories are great conversation artifacts. It’s like a book on a coffee table. It stimulates conversation.  It is of same value as using a metaphor. In conversation with the customer, the story is mostly about things in the problem space. In other words, it is an analysis and clarifying tool. I found that direct, literal and very early use of this analysis statement as an executable specification results can result in brittle tests.

On the use of technical references. When I’m working in the design space and writing design stories, then I don’t mind if there is reference to a technical implementation such as a login screen.  At some time, I have to get concrete. I like getting concrete very early and abstract from that. It’s just how my mind works. So, if there is alternative authentication mechanism (say, LDAP or ActiveDirectory), then it is just another concrete implementation. If the authentication choice is an exclusive one, then, the abstraction of proprietary authentication and ActiveDirectory authentication doesn’t offer any benefit. So, I’ll just go for one of those and the story on task board will make reference to the technical aspects directly. It’s a great reminder of a design choice that is explicit to everyone.

Most stories start out as bad stories. My early stories in an unfamiliar domain are awful.  Like code, the story should exhibit single responsibility. That takes a lot of domain insight and discipline. Unfortunately, refactoring stories towards single responsibility is not trivial. It’s not as simple as extract class/method/variable. The result is that my story based test suite is in constant turmoil longer than it is calm with a few small ripples. For this reason, I use the story grammar as a conversation piece, but not as code artifact.

BDD Stories on the backlog. To avoid confusion about when the story is in the problem space or solution space, I don’t use BDD stories on the backlog. I prefer the XP style of a short phrase as a reminder of something to discuss.

On the use of outside-in style testing. I like outside-in to analyse the problem space, but I often find it equally valuable to evolve the design from the assertions inside-out.  I oscillate between the two perspectives rapidly and quite often. I think I’m searching for that harmony in perspective.  I then make it a choice to use the BDD story as an executable test for the outside in perspective. Often, though I find it unnecessary because I already have tests that reflect that perspective; it’s just not using the BDD grammar. Yet, the BDD grammar was a starting point. I just am not fixated on the BDD grammar being the ending point.

On the BDD grammar. From a language perspective, the BDD template is a general grammar that can be used to express any domain.  Just like we can use a general purpose language to solve any problem,  the BDD grammar can be used similarly. Yet, we have learned that domain specific languages can be more expressive of a particular domain.  Equivalently, I keep my mind wide open, looking for a story grammar that is domain specific.  For example, in a time sensitive domain such as investment portfolios, I might extract a grammar that expresses time as a first class citizen. There won’t be a “When” clause. I might have something like “At time t(0), the portfolio capital amount is…, at t(n) the portfolio has …, at t(n+1) the surrender value should be …”.

Remember, these are just reflections and observations about myself. Please don’t treat it as a gospel. You have your own journey that takes different pathways. Just enjoy those excursions.

Split stories as a design activity

“A story should be big enough to fit into a sprint, otherwise chop it up until it does” — this is advice that is more or less given to Scrum teams during planning or backlog grooming. The problem is that this is not easy to do.  My friends at Growing Agile describe a few ways to achieve this (see their blog post Breaking Down User Stories). These techniques are not wrong in any particular way, and they will certainly result in smaller stories.  However, these are what I call “mechanized” techniques. When I’ve been mechanical about splitting stories, I’ve always ended up with weak fracture points in the problem space.  So, I prefer to look in the solution space for boundaries that promote or retain conceptual integrity of the software.

Below are just three techniques that are quite commonly used by Scrum teams.  I steer away from them at all costs.

  • CRUD. 
    I find that thinking in terms of these database operations removes a
    lot of the richness in the domain.  Instead, I think about the life
    cycle of things.  For example, there is no CRUD for an invoice.  Instead
    a customer buys something which means that a sales person issues an invoice. The customer pays the invoice. Perhaps, a debtors clerk requests payment for an overdue
    invoice.  These are all different things that can be done with an
    invoice at different times in its life.  Note also that “creation” is
    a very special case in any life cycle, and to bring something into
    existence that maintains integrity is quite an effort.
  • Dependent Stories.  I try to break all dependencies between stories.  I’ve found that looking to create “stand-alone” stories results in some very deep and powerful analysis of the domain.  Inadvertently, you will crack open a crucial part of the domain.  Often the concept which holds several stories together in sequence turns out to be orthogonal to the original stories.  For example, there is a workflow for invoices (issue, authorise, pay, remind, resend, etc) that can result in several dependent stories.  Alternatively, we can model the invoice state (and operations allowed for each state) independent of the sequence(s) of states.  Now we can build software that deals with specific sequences, independently of the operations for each state.  This separation can lead to such powerful discussions with the domain expert.
  • Job Functions: I’ve
    never found job functions to yield useful modules.  Extending the
    invoice example above, a job function breakdown could be on sales
    (create, authorise), debtors (record payment, payment reminders),
    customer service (credit notes), marketing (cross sales campaigns).  Now
    each of those job functions work with invoices in some way, but the
    conceptual integrity and cohesion is stronger around the invoice and its
    states.  Designing good modules is by far, the hardest part of any software design
    effort.  Get it wrong and it will hurt.  More often than not, it is just too costly to try to create new
    cohesive modules from code that is already laid down along job
    functions (or any other weak boundary criteria).

There are significant consequences to splitting stories in the solution space.

  • The product owner just needs a simple phrase or sentence that describes the problem space, but remains part of the feedback loop for the solution space stories.
  • Backlog grooming becomes an exercise in understanding the problem space. 
  • Sprint planning blitzes (one day or less) is not sufficient.
  • To be effective, sprint planning becomes continuous; i.e. design is continuous
  • Each story can (potentially) be released on completion
  • Sprint boundaries (time boxes) become less important moments in time
  • … and you will tend towards continuous flow.

Live with it for a while

Before I rush off and refactor my code, I like to live with my code for a while. The refactoring I do in the TDD cycle is to get rid of trivial duplication and, perhaps, some better naming. I deliberately delay extracting methods and classes and pushing up or down. For me, those are quite important design choices, and I want to make those decisions only when I have a good understanding of my problem.

What do I mean by “live with it for a while”?  Literally, I leave it alone and move along to something that is in it’s vicinity.  I choose something that is close enough that will need to interact or modify the “living-with” code.  This new use case, scenario or question is to further my understanding of the problem and  I choose it deliberately. If it turns out to be tangential, I don’t sweat it. I just pick something else that will move me closer.  The fact that my first choice was poor is always valuable insight into my lack of understanding of the problem.

Aside: The
simplicity of my solution is directly related to my depth of
understanding.  The deeper I understand the problem, the simpler I can
potentially get.  Shallow understanding leads to more complex solutions. This takes time, and “living with it” gives me freedom to play with the problem from several angles.

I don’t mean “ignore it after a while”.  Ignoring it is like noticing a crack on your lounge wall and after 2 weeks of doing nothing about it, you don’t see it anymore. So, living with my code is not giving myself permission to be sloppy.  It’s deliberate choice to look for a better, simpler solution as I increase my knowledge. Once I have a bit more knowledge, I can start making more adventurous design decisions.  I think it’s almost impossible to find a simple solution if you don’t have deep domain knowledge.

It means that I refactor a little at a time frequently.  Even if I know my code is not clean, I’d rather have a pulse for my code, and let it beat weakly.  All I’m doing is constantly looking for little things that will make that pulse beat a bit stronger.  I now realize that I refactor a little at a time, but almost continuously. I’m not searching for clean code. I’m searching for markers in the domain that confirm or refute my understanding.  The clean code will come with time.

Living with my design for a while has saved me lots of time, especially when I’m not confident of my knowledge of the problem.  Give it a try and let me know whether it works for you too.

Cape Town’s latest agile development course

I got my first break as a software developer about 20 or so years ago.  It was the first time I heard that a table can have keys. That start put me on a career path that I never anticipated, but is thoroughly enjoyable.  And now, finally, I’ve finally got a chance to give back in a way that shows my appreciation for what was offered me all those years ago.  For those of you who know my history (or just peeked at my LinkedIn profile), you know that I’m talking about KRS.

So, I’m quite thrilled to be a contributor and collaborator for their Advanced Agile Developer course.  The inauguration course happens on 19 November. That’s not much time, so book a spot quickly.  I don’t want to give too much away, but I can tell you that it’s going to be fun, intense, and inspiring – and seriously code centric.  There will be times when you will feel like you just don’t know how to code anymore, and then feel like you can conquer the world.

I think this is a course with a difference.  We want to bring together developers that are already competent at writing code and want to become proficient at being agile developers.  We made a big decision to go deep, and not skim the surface of lots of topics.  The result is a course that is very code centric, working at quite an intensity that passionate developers will find inspiring.

Working with Lorraine Steyn and team KRS has the same warmth, openness, and security from 20 years ago. This is something that I know they will bring to this course in a way that I can only hope to, someday, emulate.