Customer-Driven Engineering
Part 2 of 2: The Process
This post is part 2 of 2 in a blog series. If you haven’t already read part 1, you can find here.
Customer-Driven Engineering (CDE) is a culture and product development process that has been cultivated within the Developer Division (DevDiv) at Microsoft; the division responsible for creating tools and services for software makers.
In DevDiv CDE represents so many things, but ultimately it represents two major parts: Culture and Process
In my previous post, I outlined our Customer-Driven Culture.
In this post, I’ll detail the day-to-day process of making products using CDE.
The Customer-Driven Process
Everything outlined in this post has been written about, more extensively, in a book I co-wrote with my colleague Dr. Jessica Rich called The Customer-Driven Playbook: Converting Customer Feedback into Successful Products. This book has become the de-facto handbook for our product teams in DevDiv as it outlines our complete CDE process.
The Hypothesis Progression Framework
As we worked in DevDiv to become more agile and customer-focused, it was clear to us that we needed a way to accelerate our learning. Essentially, rather than having a small team (e.g. our UX Research team) dedicated to learning on behalf of the organization, we needed to empower everyone in DevDiv to learn from our customers.
As product teams began engaging directly with customers, a frequent problem started to emerge. We were excited because customer connections were growing at an exponential rate, but there also was a lot of “noise” in our data. Many of the conversations lacked a defined objective or the scope of investigation was just too broad. Product teams would generate notes from their customer conversations that would often meander through assorted topics, sometimes even hitting on product areas that the team wasn’t even responsible for.
In short, early in the development of Customer-Driven Engineering, our teams were casting too wide of a net, looking for insights they could leverage in their specific feature areas.
Teams that were more successful with their learning, tended to have a more rigorous plan for their learning. They formulated their assumptions into hypotheses and would use those hypotheses to scope their efforts.
Essentially, downstream efforts (e.g. talking with customers, conducting experiments, launching surveys, refining product offerings, etc.) were dependent on the quality of the hypotheses that were being formulated upstream.
It turns out that hypotheses are a fantastic tool to share what you’re trying to learn, as well as, what you’ve already learned. In my previous post, one of the “culture hacks” I discussed was “establishing a common language”. In DevDiv, hypotheses became a vital part of our common language.
To help our product teams refine their hypotheses, we developed the Hypothesis Progression Framework (HPF).
The HPF is a series of stages that represent customer, product, and business development.
Each stage is designed to address a fundamental question during the complete lifecycle of product development:
Customer: Do these customers exist? In other words, who are these customers? What is their unique context? How many of them are there? What are they trying to achieve? What behaviors are they engaging in?
Problem: What problems do these customers have? Are we uniquely positioned to solve those problems?
Concept: What’s the best way to solve our customers’ problems and how can we solve these problems in a way that creates value for them?
Feature: Can customers successfully use this feature (or solution)? Are they satisfied with it?
Business: How will this feature impact our business outcome?
We call this the Hypothesis Progression Framework, because within each stage, we embed a hypothesis template. These templates help teams organize their learning via key hypothesis parameters (e.g. type of customers, motivation, job-to-be-done, problem, concept, feature, etc.). Teams can simply complete any one of these templates to help articulate what they are trying to learn.
The framework is “progressive”, because the learning in one stage can be carried over to the next. It’s important to note here that the HPF isn’t designed to be used from left-to-right for every project. It’s flexible so that teams can start at any one of these stages and begin working to identify the gaps in their understanding. We encourage teams to use the HPF regardless of where they are in their development lifecycle.
Working with a Hypothesis Template
So, for example, let’s imagine that, based on some early conversations and market research, we have a hypothesis that office administrators of small businesses are being asked to take on more responsibilities to support the business. One of those responsibilities we’re interested in exploring is spending time creating marketing brochures that advertise the business’ services. We could use the Customer hypothesis template to help organize these assumptions.
The Customer hypothesis template:
We believe that [type of customers] are motivated to [motivation] when doing [job-to-be-done].
A completed template:
We believe [office administrators working for small businesses] are motivated to [expand the marketing reach of their business] while [creating a brochure that details the services their business provides].
Now, this could be one of many hypotheses that are generated to explore this space. In fact, it’s not uncommon in DevDiv for our teams to track numerous hypotheses while engaging with our customers.
The best part of each of these hypotheses templates is that the structure leads team to the types of questions they should be asking our customers. Our teams take the hypotheses they’ve formulated and create discussion guides, sets of questions they can ask customers to help them validate or invalidate their hypotheses. Additionally, our UX Research team can provide sample questions that our product teams can ask customers, based on the hypothesis parameter they’re looking to explore.
Returning back to our example of office administrators, if we’re looking at the job of creating marketing brochures, we might ask:
“Who in your business is responsible for creating marketing assets?”
“What type of materials does your business use to help customers understand the services your business offers?”
“How effective are your marketing materials in helping you get the word out about your business offerings?”
These types of questions help us assess whether office administrators are responsible for generating marketing assets, to what extent, and what jobs they engage in to deliver on these responsibilities (to learn more about “jobs”, see the jobs-to-be-done framework).
The Customer-Driven Cadence
Within each of the HPF’s stages, we provide our teams a set of tools to help them formulate their assumptions into hypotheses, run experiments to test their understanding, and make sense of the data they’re collecting. We call this motion the Customer-Driven Cadence. This cadence, formulating, experimenting, and sensemaking is continual and happens at every stage of the HPF.
Our experiments can take many forms. They can be quantitative (e.g. product usage analytics or in-product surveys) or qualitative (e.g. customer interviews, quick-pulse usability studies, or focus groups). We advocate for a multi-pronged approach where our teams are engaged in a “healthy heartbeat”; moving back and forth between quantitative and qualitative data to ground their confidence in what they’re learning from our customers.
With the help of tools like the HPF and a co-discovery model between our UX Research and product teams, we’re creating an environment where everyone is empowered to learn from our customers.
It’s the expected norm within DevDiv for our PMs, engineers, content writers, and designers to join in on customer learning sessions, alongside our researchers. In DevDiv, learning from customers is a team sport, where everyone has a vital role to play. In the previous post, I outlined our “culture hack” for “Encouraging Learning vs. Knowing”. Our co-discovery research model is one example that helps create an environment where everyone feels responsible for customer learning.
Teams also follow a “sensemaking loop” where they sift through sources of data, culling the most important information into a “shoebox” (e.g. a Microsoft SharePoint site, Microsoft Teams channel, Microsoft OneNote notebook, etc.). These sites can be a trove of information, so with the help of articulated hypotheses, we can scope what we’re learning and build an “evidence file”, a carefully curated section of our most important learnings.
As their learning evolves, teams begin to schematize and give meaning to what they’re learning. Our PMs, in particular, have a knack for this. They’re constantly finding novel ways to categorize and tag their learnings so that we can better identify meaningful patterns. Tools like GitHub, Azure Boards, Microsoft PowerPoint and again even Microsoft Excel are extremely powerful tools that help our teams organize and share their learning.
Finally, none of this comes together without a powerful set of stories that transfers the empathy that’s being generated through direct customer connection, to the rest of the organization. Our teams are highly communicative of what they’re learning. For example, we have an internal e-mail address where every team member sends out their findings from every customer interview they have. These “customer notes” not only update our division on our teams’ validated or invalidated hypotheses, they also bring the stories of our customers to the surface. It’s these stories that give our work purpose and inspire us to action.
Making Customer Development Approachable
In part 1 of this series, I discussed the customer-driven culture we’re working to build in DevDiv. The customer-driven process detailed in this post simply does not work if you haven’t baked in a learning-first mindset at the core of your organization’s values.
More than anything else, the culture we build or the process and tools we put in place in our organizations should reduce the distance between product teams and their customers.
Tools like the HPF empower teams to take control of their own learning. We leverage the expertise of our UX Research team to help coach our product teams, so they can be more effective with each customer interaction. Giving tools to our product teams to help them generate lists of their hypotheses, create discussions guides for customer interviews, or conduct sensemaking exercises to reason over their collected data, makes our entire engineering process more inclusive and ultimately democratizes organizational learning.
There’s so much to share about this customer-driven journey we’re on in DevDiv. It evolves every single day as we learn new ways to connect with our customers, experiment with and tweak our engineering processes, and create products that empower our customers to achieve more.
For a detailed overview Customer-Driven Engineering, be sure to check out The Customer-Driven Culture: A Microsoft Story and The Customer-Driven Playbook: How to Convert Customer Insights into Products.
Finally, be sure to follow this blog as I have plans to share much more going forward.
