Product Managers tend to be fairly analytical, and while we want to make quick decisions, we also want data to back up our decision. What happens when you know that the change you're implementing is the right one, but your results are proving otherwise? In the results we're seeing for this test, we're not seeing a significant increase in leads produced or sales completed, but we're also not seeing a decrease - does this mean success?
How are you measuring success? For this particular example, I'm using traffic to the particular site, number of leads submitted and number of clicks for a particular section of the site. I'm using a number of sales metrics as well. Do your numbers typically revolve around sales? Do your marketing, technology and sales teams care about these numbers? How can you ensure that the information you're providing is relevant to the business stakeholders?
Hmmm, sounds like your testing is not verifying your assumption. And it sounds like you have some "confirmation bias" seeping into your reasoning ("What happens when you know that the change you're implementing is the right one, but your results are proving otherwise?" screams "I'm right, why doesn't the statistics prove this?")
ReplyDeleteComing at the problem as a scientist (can't help it, I did study physics in university), if what you are measuring tells you that your hypothesis is wrong, look for a new hypothesis.
Of course, you should also be certain that you are measuring factors that are relevant to the hypothesis under test.
On a side note: Marketing loves to bombard management and sales with data and metrics on leads, prospects, and web site visits, but in the end, those raw facts don't directly correlate to actions taken. Of course, if I had a magic decoder ring to apply this to, I would be a mega-millionaire marketing/management consultant.