Hi arunjm: To measure against your competitors, you would need access to their data to make a comparison. You have access to your own data, so what you are measuring is the difference between the performance of something for a given period of time before and after you intervened. A good metric is actionable - so even before you measure anything, you need to know why you’re measuring it, what result is a pass or a fail and what action you’ll take as a result of that data.
But that’s only half the story. JollyZ is right - you need to connect those results to the goals of the business. And your CFO doesn’t care about CTR. She cares how much more $$$ it brings in. So while the raw results of an A/B test on, say, the CTR of a product category given its location in a list might tell you how high in the list it should be, to your money person, it won’t be clear what that means in dollar terms.
To borrow an example made by Carl H from Country Road (who spoke about conversion testing at Melbourne Geek Night a few weeks back), it’s all about 2 things: 1: Just doing it (by getting friendly with some devs) and 2: framing the results in terms of how much the company will [B]lose[/B] over 12 months without the change. If I remember correctly, Carl noticed that people who navigated to the shoes tab often made significant purchases - but the tab itself was buried in the list. He A/B tested the position of the tab and confirmed his hunch that putting shoes closer to the top of the list would make increase the profitability of those sales.
When he presented those findings, he extrapolated his data from the fortnight he ran the test out to a year. Instead of saying how much the the company would make, he showed how much money the company was leaving on the table by burying such a profitable area of the site.
To come back to your original question: instead of evangelising on what you could do with your data, consider the data that you have and what you’re trying to do. Is acquisition a primary concern? What are your conversion funnels for signup? Are people signing up, but not doing anything? How are you measuring activation? How regularly are users logging in? For how long? What do you know about your users?
In my opinion, there isn’t really a playbook for ‘UX metrics.’ It depends what you and your business are trying to achieve. If you haven’t watched it, Dave McClure’s Startup Metrics for Pirates AARRR! is a great macro view of everything(?) that matters to a business that’s growing. Also worth looking at is [URL=“http://leananalyticsbook.com/”]Lean Analytics by Alistair Croll and Benjamin Yoskovitz. They’re main concern is identifying ‘the metrics that matter.’