Data Keeps It Real

www.office.com/setup Blogs: Pavan wrote last week about the ways we build Yammer and the core beliefs behind how we develop, iterate, and improve our product. We work really hard to build a product that reflects our users’ needs. And the way we learn about those needs? Data.

We study which features people use and how they use them. And we’re studying this because we care about which features are “working.” We see users getting more value from Yammer as they use the product more and more. So, for us, a feature is working when it increases our users’ engagement and retention.

When we have an idea, something that seems like it would make Yammer a better more valuable experience for our users, we build it. But we don’t have any way to know for sure whether it will actually be valuable to our users. So we do what we always do: we test it.

A/B testing is splitting the user base into multiple groups which get slightly different experiences, and we gather data on how the different features affect their experience. We compare the experience of the group who sees the new feature with the experience of a similar group who doesn’t have the new feature. Sometimes, we find that the new feature improves the product, and provides a statistically significant lift in engagement; so we roll the successful feature out to everyone. If a feature doesn’t enhance the user experience–as indicated by user engagement and retention metrics–we continue to work and iterate on it, or sometimes try an entirely different approach. This data-driven development allows us to constantly and consistently validate our product hypotheses, and subtly adjust our product to the needs of our users.

Take, for example, the Yammer feed, the first thing you see when you log in to Yammer. This basic yet critical part of the Yammer experience is what it is today after having undergone numerous iterations, fine-tuned using data gathered from our users. The Yammer Feed started out very simple, as just a series of unconnected status updates. But, through brainstorming, talking with customers, building and testing, and validating these changes using data, we added features–like threading and replies–which have allowed for a richer user experience.

To use our data effectively, it is critical that we can track patterns from the past which we expect to persist. And for this to happen, we want as little to change between the experiment and the roll out as possible. Typically, this means we want to implement our learnings soon after we learn them. So in order to be data-driven, we need to develop incrementally and rapidly. Unlike most enterprise software companies, Yammer has a rapid release cycle–we release features multiple times per week. And we do this by breaking down all our big ideas into small guesses, and testing them individually. By releasing features in small increments, we can better evaluate their effectiveness, identify incorrect product assumptions early, and course correct as necessary. And through subtle product releases, users gradually adapt to new features as we constantly evolve to keep up with their changing needs.

At Yammer, we’re always coming up with ideas, building them, testing them, iterating, trying again, and releasing. I take pride in getting our users and the companies they work for the maximum benefits of our development process through effectively harnessing our data. It’s exciting to see the subtle changes that compound into a vastly improved user experience.

To learn more about Yammer’s development approach, visit http://blogs.office.com/product-development/

Original Post: https://blogs.office.com/2013/05/28/data-keeps-it-real/

Leave a reply

Your email address will not be published.