Revive Your Struggling Testing Programme With a Hackathon

There are a lot of posts out there with titles like “10 A/B testing mistakes you are making”. They offer advice on sample sizes and statistical pitfalls. Truthfully however, few conversion programmes struggle because of this type of mistake.

Conversion Rate Optimization (CRO) has been a “thing” for a while now, and there is a lot of expertise out there. In reality, the most common reason for a programme to struggle to make a lasting conversion impact is simply this:

People are not working together as efficiently as they could.

If you see this problem in your company, try holding a CRO hackathon.

What is it?

It is the CRO testing version of the well-known hackathon concept. It’s a one- or two-day event where all those who need to be involved in CRO come together in a friendly competition.

Why should I do it?

Here are five good reasons:

  1. A captive audience. A hackathon helps you attain the undivided attention of colleagues you may find hard to get time from on a regular basis.
  2. Organizational buy-in. You increase knowledge of testing in your wider business. Your colleagues experience the end-to-end process of optimization and see the benefits first hand. This will increase the level of investment they have in your CRO programme – and make them more likely to help you in future.
  3. More ideas. You will build up a pipeline of testing ideas to develop further. Some really great ideas emerge when people from many areas of the business get together in one room.
  4. You can get tests up and running quickly. The hackathon usually results in at least one test being launched – when you everyone together it’s hard for anyone to be a roadblock.
  5. Team building. Practice working together as well-oiled CRO team. Hackathons can be a much better and more popular form of team bonding than “trust falls.”

What would a typical CRO hackathon schedule look like?

Here is an example schedule for a one day CRO hackathon:

9am – Arrivals, coffee and breakfast

9.30am – Introduction

The introductions needs to clearly explain the following:

  • The goal of the CRO hackathon.
  • The hackathon “task.” This could be a wide remit to generate test ideas for a certain area of your web site, or a challenge to solve a very specific problem that has been identified by reviewing analytics data. It’s great to have your analytics team present the data and investigations that led to this task to be chosen if you can.
  • How the winner will be chosen. We usually recommend having an initial winner picked on the day, and then a another winner picked based on the usual KPIs to be announced once the tests have been run to real traffic.
  • Any rules – for example areas of the page that are off limits.

You might also include an introduction to CRO in general if the attendees are fairly new to it.

10am – Divide into teams and generate test ideas

Divide the attendees into teams, which ideally have one person from each functional area.  It’s easier of you have planned the teams in advance. You may also choose team leaders, or leave them to naturally emerge.  Each team then generates as many test ideas as possible to meet the hackathon challenge.

A hackathon team brainstorm ideas

12pm – Lunch

1pm – Choose and refine the best test idea

If you have analysts with you, you can spend a little longer on this section and try to look for real evidence supporting each idea.

2pm – Build a prototype and presentation

The more technical members of each team collaborate to produce a working prototype of the idea using your testing platform. This can be refined later, with quality assurance occurring after the day, then launched to real traffic.

In addition, the business-focused team members prepare a presentation including the hypothesis and why this was chosen as the best idea. They aim to convince the judge(s) that this idea is the most likely to get conversion gains.

4pm – Present finalised idea

Each team presents on their idea, and shows off their prototype.

4.45pm – Winning test idea chosen

A senior person or panel of judges decides which presentation was most convincing and awards the ‘best on the day’ prize.

A few days later – Test launch

The test or tests are launched to live traffic.

A few weeks later – Winners declared

After the pre-agreed test run time, the tests are analysed and the winner declared. Set up a follow up session to present the final results, including prizes for the winning team.

Who should I invite?

Many people discussing test ideas during a CRO hackathon

  1. Your boss – Perhaps the CMO, CTO or head of e-commerce depending on where CRO sits in your organization. Often the boss gets to pick the initial “winner” on the day, giving a real life test on whether “HiPPOs” add value!
  2. Web analysts – They own and understand key data on how your customers behave on your site. The ideas that are generated have a much higher chance of success if some data analysis is included.
  3. User research – If you have teams performing other user research such as usability studies or surveys, their insight is very valuable. Also, marketers who have conducted industry research and developed customer personas can provide great input.
  4. UX – Designs are more likely to succeed with well qualified UX input.
  5. IT/Development – To build the test or to advise on any technical limitations.
  6. Product/category owners – Anyone who has ownership over the areas of the site you aim to improve should be invited.
  7. Visual Design – To advise on variation design, and provide assets such as icons, buttons, banners etc.
  8. Compliance – Because if you are in a compliance-heavy industry, nothing goes live without their say-so.
  9. Call centre/live chat operatives – This may seem like an odd idea, but these people interact directly with customers and know what type of complaints they have. If you can’t bring in people, then tools like OpenText™ Qfiniti can provide insight on customer interactions.
  10. Testing vendor – OpenText™ Optimost facilitates CRO hackathons for our customers. We also provide the technical resource to build the test variation prototypes, and our Customer Success Managers contribute ideas based on successes we’ve seen previously.


OpenText is the leader in Enterprise Information Management (EIM). Our EIM products enable businesses to grow faster, lower operational costs, and reduce information governance and security risks by improving business insight, impact and process speed.

Related Articles