Measuring Success

I've been running an A/B test to test the success of a new process for our customers.  While the current process is poorly designed, it produces a significant amount of revenue for the company, so changing it has been quite the challenge.  The design team came up with a brilliant plan for the change and we implemented on a small portion of our potential customers.  Now that we're running the test, I'm struggling to figure out how to measure the success of it.

Product Managers tend to be fairly analytical, and while we want to make quick decisions, we also want data to back up our decision.  What happens when you know that the change you're implementing is the right one, but your results are proving otherwise?  In the results we're seeing for this test, we're not seeing a significant increase in leads produced or sales completed, but we're also not seeing a decrease - does this mean success?

How are you measuring success?  For this particular example, I'm using traffic to the particular site, number of leads submitted and number of clicks for a particular section of the site.  I'm using a number of sales metrics as well.  Do your numbers typically revolve around sales?  Do your marketing, technology and sales teams care about these numbers?  How can you ensure that the information you're providing is relevant to the business stakeholders?


1 comment:

  1. Hmmm, sounds like your testing is not verifying your assumption. And it sounds like you have some "confirmation bias" seeping into your reasoning ("What happens when you know that the change you're implementing is the right one, but your results are proving otherwise?" screams "I'm right, why doesn't the statistics prove this?")

    Coming at the problem as a scientist (can't help it, I did study physics in university), if what you are measuring tells you that your hypothesis is wrong, look for a new hypothesis.

    Of course, you should also be certain that you are measuring factors that are relevant to the hypothesis under test.

    On a side note: Marketing loves to bombard management and sales with data and metrics on leads, prospects, and web site visits, but in the end, those raw facts don't directly correlate to actions taken. Of course, if I had a magic decoder ring to apply this to, I would be a mega-millionaire marketing/management consultant.

    ReplyDelete