.png)
(This blog post is just a small excerpt from our informative newsletter. Sign up here to receive regular newsletters packed with valuable insights, industry updates, and expert tips.)
Introduction
A few weeks ago, I shared a LinkedIn photo of a team experimentation session I conducted, focusing on team behaviours and work patterns. It received a fair bit of love, so I figured why not share a few posts about my approach to experimentation? Today's email is about getting us all on the same page about the term experimentation so we have a common language.
With my current contract, I am partnering with the organisation to drive them forward with internal Innovation and experimentation in EX. This means that from a strategy point of view, I am looking at the business's internal portfolio of EX products, services, experiences functions and teams and analysing these for:
Driving pains
Where the biggest gains and opportunities are
Which should fall into our incubator
Which needs to be sunset
The other part of my role is designing and scaling out an org-wide innovation framework, measurement engine, team practices and experimentation. So I figured why not zoom in on the experimentation given it received some love?
Before I get into this, it is important to note, having the right behaviours and mindset is vital to having an anti-fragile team. if you are new here, then read this
What is experimentation?
Experimentation is something we do every day, if you want to observe experimentation in the wild, spend some time with young children or toddlers. They are constantly in sense-making mode and experimenting, usually by putting things in their mouth
Before we get into team experimentation let's actually break down and clarify what experimentation means, for some the word is welcoming and exciting, for others it can be daunting and non-inclusive, for me experimentation is:
A method to validate or reject a set of hypotheses and assumptions, or to discover something new.
Let's assume we all agree on the above meaning of experimentation, we can quickly look where it adds value
The value of experimentation
Learning: This is ultimately the end goal of every experiment, it's not about if an experiment works or doesn't work, what we are trying to do is get richer insight and understanding into the why and how it worked or didn't work. One of the traps in experimentation is when the experimenter wants the experiment to be successful. This bias can show up in many ways (more to come on that later)
Early solution signals: We can sometimes hit a challenge where we don’t have a potential solution to test. We can use experiments to test different variables of the current challenge to help identify potential opportunities. I do online coaching with a personal trainer, I gave them some feedback about their app being slow and not logging my workout. They couldn't see the challenge on their side, so they ran a few experiments to see what might be causing it. This ranged from reducing the images on my version of an app to moving me on to a different server and also ringing a code analyser to help identify the challenge around the initial challenge.
Discovery & Innovation: It goes without saying but experimentation is vital to new breakthrough discoveries and innovation. To see hundreds of examples of this just simply look into the past. For example the discovery of penicillin in 1928 by Sir Alexander Fleming. As the story goes, he was coming back from his jolly holidays and noticed that a Staphylococcus bacteria in a petri dish had a mould growing on it called Penicillium notatum, this penicillium was effective in stopping the staphylococcus from growing, in fact, it was the penicillium self-defence mechanism that was effective in killing the Staphylococcus. This was massive for medicine and led to the development of the antibiotic
Behaviour Change: We use experimentation here, to create a safe space for managers, teams or indviduals to test out a new behaviour or way of working. We map this to a tension felt in the team, other factors like internal influence, attitudes, and perception. We then co-design an experiment with new behaviour, activity or approach and run it for a set time. This is less about boring change frameworks that often don't work in the real world and take a hands-on approach to testing a new behaviour or way of working
Testing: Often the the main use of experimentation is to test something, this is often a hypothesis about how or what will happen. It is here where they will run the experiment to test if the hypotheses were validated and the outcomes aligned.
Ongoing development: Often seen in big platforms like Facebook, Amazon and so on. These platforms are running a lot of little experiments like A/B testing to constantly develop their platform. In 2011 Amazon set up its internal experimentation platform and has openly talked about experimentation in an old letter to shareholders, imagine how many they run now in 2023
Sounds exciting right and pretty simple…. slow down my friend. We know what the word means, and we know where it can add value, let's look at some of the booby traps that can sneak in and ruin the whole thing.
Booby Traps & Watch Outs
Bias - We touched earlier on how bias can sneak in, but bias can be seen in a number of ways, here are a few to get you going:
Confirmation bias - Where we have a tendency to find, interpret and often favour what confirms our assumption or belief
Sampling bias - Often where we don't bring in a fair spread across the spectrum this could be people, background etc this drives an unfair representation or conclusion
Performance bias - often where an experiment may get more special treatment than the others because it is seen to be more cool or different etc
Variables - After bias, we have variables to think about, there will always be a selection of variables in every experiment (I like to think of these as dials on a machine) these variables are summed up in three key groups:
Independent: This tends to be variables in the experiment that are manipulated, tweaked or changed throughout the run time of the experiment
Dependent: These are the variables that often get measured or observed, depending on the change made in the independent variable
Control: These variables are kept consistent, to make sure that any changes observed are due only to the changing of the independent variables
If your anything like me, this might of just happened:
As a dyslexic when a word is used in various manners too close to each other my brain turns into knotty spaghetti, so here is a high-level example to bring it to life. Let's assume your experiment is around how the length of your working day affects how productive you are:
An independent variable (element we are changing) would be the number of hours in a working day, let's assume you go to 5hr, while another group remain at 8hr.
The dependent variable (the thing we are measuring or observing) would be the quality or quantity of work completed in the 5hr workday or something similar.
The controlled variable (the element that stays consistent) would be the space where they work, the work type, the amount of 121 times with managers etc.
Of course, there are many other factors to the simplified example but you see how X doesn't always equal Y when you start to factor in variables. One more thing to note, that gets overlooked is the:
Controls - When you run an experiment you want to always have a controlled group, this is a group/segmentation that does not know about or go through the experiment. This helps later on down the line when you start to compare and distil the effects of the variables
Here's the rub:
Experimentation is fun, it's always insightful and nearly always leaves you with aha moments. While the word may feel daunting it's nothing to shy away from if children and toddlers can do it am sure you can too. The only real difference is before running any internal experiments you have to be mindful of
Bias
Variables
Controls
(Like what you have seen, want more? Sign up here below to get my bi-weekly email straight to your inbox)