Be careful what you wish for

I am irrationally irritated by the expression: ‘what gets measured gets managed.’  Not because it’s untrue, rather because it opens a can of worms which demands a less banal label.  A more apt form of words might be something like ‘what gets measured will become a bizarre new religion trumping everything else, however important’.  Or maybe ‘what gets set as a goal will send you off down the garden path’.

When businesses set objectives, it acts as a signal for what matters most.  So, in my business, if our goal is to win creative awards, we’ll prioritise the best creative opportunities.  It has implications for which projects we spend most time on, how boldly we present ideas, how we judge those ideas and so on.  That’s right, because it gives us the best chance of achieving what we set out to do.  So far so good.

This being the case, we need to be sure that the goals we set are expressed in a way that does actually bring about the outcome we want.  We talk about objectives being SMART (specific, measurable, attainable, relevant and time-bound).  This is important because most aren’t; they’re woolly, vague and a bit meaningless.

I had a colleague in a Government Department who expressed the perfect SMART objectives as : “On Fridays, I always aim to leave the office by 5PM.” Nice.

The bit that’s often underplayed is the ‘relevant’ part.  Because that’s where it can all go horribly wrong.  Let me give a couple of examples.

The NHS wanted to offer better health care and one of the biggest problems they faced was that waiting lists were growing alarmingly for routine operations.  So they set explicit goals for Hospitals to reduce waiting lists.  There was no extra resource – no more beds, no more nurses, Doctors or wards.  So it was a mystery as to how the lists could be reduced when an increasing number of people (ageing population, you see) were in need of more and more operations from the same facilities.  The only way to achieve the goal was to be ‘creative’.  So the hospital administrators prevented sick people from joining the waiting lists.  Genius.  So the well-meaning objective manages to provide a worse health service.  Some people call that ‘The law of unintended consequences’.  I call it not thinking through the objective.  The stated goal wasn’t the real goal.  It failed the ‘relevance’ test.

It happened again recently.  Schools in England are judged on league tables based on exam results.  This is how parents are instructed to choose their children’ school, so it is a major influence on the school’s income.  Schools have to compete for the brightest students who will achieve the highest exam scores.  But they are not allowed to be selective in their intake.  So how can they improve their standing in the league tables?  Here’s how:

olaf

It turns out Sixth forms all over the country – not just St Olaf’s, the school featured in the news story – were habitually weeding out weaker pupils half way through their course to maximise the school’s exam results.  Great plan.  What could possibly go wrong?  Well, for many years, nothing.  except for the poor students, ditched half way through their studies, with no school and no chance of joining a new course.  Their only option was to go back and start again, losing a year of studies.  These weren’t the ‘no-hopers’ you might be imagining.  These were most of the students who didn’t achieve a Grade B in their end of year exams.

The politicians who created the framework of incentives and league tables might perhaps ask the question: Is this a system which promotes the best outcomes for students?  I recently came across a fifteen year old student who was made to do an extra five GCSE exams, on top of the normal ten, because his results (being predicted as better than average) would boost the school’s average results.  Great for the school.  Terrible for the student.  Unintended consequences indeed.  He significantly under-achieved as a result.

These are public sector examples, so the results are clear to see and news-worthy enough to make headlines.  In the private sector this sort of thing is happening all the time; we just don’t see it.  Like businesses that set quarterly revenue targets which everyone strives to achieve – at the expense of delivering a profit.  Or on a bigger scale, any kind of short-term deliverables delivered at the expense of longer term sustainability.

What gets measured does, indeed get managed, whether it’s relevant or not.  So we better make sure it’s the right thing.

……Ooh ooh, stop press, PS, extra extra:

I just discovered there’s a name for this (thanks to Dave Trott and his blog).  It’s called Goodhart’s Law:

As Trott says: Goodhart’s law should be pinned up in our offices:

“When a measurement becomes a target it ceases to be a good measurement.”

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s