This month Lean Startup Boston Meetup had David Cancel of Performable to talk about Analytics and A/B testing. Here are my notes. The actual slides from the talk can be found here. I just want to note down my translation of what was presented and the surrounding commentary that you can’t find in the slides.

Talking, Reading & Dreaming are all worthless. Just start DOING! or more closer to “Just F’ing Do It.” Hence, the post is titled “Translating JFDI”. [This is a good wakeup call for me since all my life, I have been conditioned to read and absorb as much information as I can by personality and training. And it’s good to remember it doesn’t matter how much you’ve read, analyzed, discussed unless you have done it and actually helped your startup.]

The single most important thing is “Does anyone give a shit about your dumb idea?” [There were good commentary on how you know if anyone gives a shit. The recommendation is just to start talking to people: anyone on the street, your mother, potential customers, not just startup obsessed hackers. You will know if people care by how engaged they are. How much they let you bother them by going out of their way to discuss, sending you feedback, letting you record their screens or phone calls.  David noted that this is really hard for many engineering type people since we are all more or less introverts.]

The reason to just start doing and stop reading and dreaming is because there is no repeatable pattern for startup success. [It’s kind of funny since right after saying this, David recommended reading two books, Startup Lessons Learned and The Four Steps to Epiphany. Obviously it was a Lean Startup Meetup.]

Analytics and A/B testing are also not the answer. No magic bullet here. They just serve to validate your assumptions.

What about A/B  vs. multivariate?  Multivariate testing is completely useless for a startup mainly because you are too small and you don’t have enough data for it to make sense. [For the giants like Google, they have enough data and traffic to use multivariate testing for good benefit.]

Come up with one assumption, Test and then Iterate. Make that cycle as short as practically possible. [The faster you are, the more likely you will get to what truly matters to the success of your startup. There were questions from the audience on what to test. David noted most of the times, it’s the copy that makes the most difference. Try a bunch of different ways to evoke  emotional connection to the visitors. I disagree to a point David made that it’s all about the emotional connection, not about the facts. It does matter how much information they have seen at the time you are asking them to take the next step along the conversion path. But that’s just a hunch. I’m sure we can test that too!]

Don’t start testing on the home page. Start somewhere else.

Always be testing. Just don’t be stupid about it.

Beware of local maxima. [I have always been concerned about A/B testing optimizing for local maxima and never making the jump to global maxima. When Google obsessed over testing small details over and over (aka the infamous 41 shades of blue) to the point of driving some designer away from working there, I cried ‘local maxima!’  David made a great point of how Facebook is making radical changes,  pissing off many users, as an example of how to make the jump from one local maxima to next. That’s a great point and very valuable insight I took out of this talk.]

Remember to optimize for learning, not to just collect data.

Three Must-haves for any startup that wants to be data-driven.

1. Operational Dashboard – Go make it today.  [ Leave it ugly. Do it manually until someone complains. Everyone in the company can see this. For each item on the dashboard, have someone responsible for that number. Master this before moving to the next step. ]
2. Conversion Funnel Analysis [Monitor it after making any change to see how the change affected your business. Don’t make it real time or get addicted to this. Don’t get too jumpy and react. Make sure it’s long enough and figure out how to fix them.]
3. Cohort Analysis [Read more about it on Fred Wilson’s post. Basically, it’s tracking group(s) of people over a long period of time to see if there are changes in their engagement, activity. This is the part I wished the presentation had more details on. I still don’t know much about it.]

Most importantly, decide what you are optimizing for. [This part is not in the slides but I believe is particularly useful. I filled in the details for better comprehension. ]

If you are optimizing for cash flow/profit, you have a clearer view of what to optimize for. $$$$$

If you are optimizing for raising funds, it’s an art of story telling. Decide your story line and get the numbers you need to support your story line.  It might be traction, it might be proving that you can get >1x returns for each customer you acquire or that you have an  enormous market, or how fast you can grow your customer base. Focus on your story and optimize on that number.

Overall, it was a fantastic talk and great discussion among the audience. I highly recommend attending this meetup if you’re interested in this sort of thing. David is a great speaker to listen live. As awesome as his slides are,  you have to be there to get the full value of the talk.

Some tools mentioned during the talk

Kissmetrics and Mixpanel for funnel analysis.

Usertesting for qualitative testing.

Olark for monitoring your visitor and engaging with them. [One advice that wasn’t on the slide was “keep things high touch. Don’t worry about making anything scalable. That’s a problem you will feel lucky to have later on”. ]