Puppet Test Pilots
- UX research as a service.
- 1600+ members; 100s of user tests a year.
- 2-week turnaround for UX & Product studies.
- Started in 2012 and still running.
Actionable customer insight
Actionable user feedback is a challenge for every software organization. How do you get it? How do you make sense of it? What to do next?
From 2012 until 2014, I lead the creation and expansion of the Puppet Test Pilots program. We turned UX research into a predictable resource for the company. Anyone in product development could, within 2 weeks, have structured UX research sessions with 5-10 DevOps professionals, specifically qualified for the product or feature in question.
I’d run many user tests before, but never something of this scale. Neither had anyone else on the team. We stuck with it, learned a lot, and improved relentlessly. In my last year at Puppet, we averaged over one user test per business day.
The Genesis — PuppetConf 2012
In 2011 I was Puppet’s first design hire, and by 2012 I’d hired two other designers and a UX researcher. A few months before PuppetConf 2012 UX designer Daniel Sauble suggested that we run user tests at the conference.
Daniel was expecting me to accept the idea and lead it as a team project, but I love empowering people, so I gave it to him. Daniel found a partner to help, setup several tests, and created a system to capture and manage feedback. They ran 50 user tests in the 3 days of the conference.
Expansion — PuppetConf 2013 & 2014
Daniel wrote this up nicely, and he’d be the first one to point out potential for improvement. But we did what any good UX team would and iterated, running 100 tests at PuppetConf 2013 and 200 tests at PuppetConf 2014.
The Puppet community responded with enthusiasm. Sessions were booked early, and we had to turn people away. I’ve found this consistently in my career: many people enjoy participating in user tests. People like being asked to help, and users of software enjoy being treated as experts and as stakeholders in the future of the product.
Each year presented a new set of challenges, and each year we had a larger UX team to handle them. In 2014 there were more tests than the UX team could run, so we recruited facilitators from other teams. We trained them in the basics of UX testing (e.g., “Encourage people to talk, but allow them to be frustrated.”). Even though we recorded video of every test, we encouraged facilitators to take notes (these were useful in 2013 when we lost two of the laptops full of recordings).
The tests themselves got better. Much of the work in Puppet is still done on the command line (which has its own challenges), testing of which is less amenable to techniques like paper prototyping and card sorting. Cooperating with the product team, we developed interactive CLI prototypes. This is one of the factors that lead me to create a small proyotyping team inside UX.
We got better at connecting our tests with the product roadmap. This was challenging, particularly given the timing. If product needed feedback the month before PuppetConf, then tests at the conference wouldn’t help. This lead us to get better at testing outside our conferences.
Formalization — Puppet Test Pilots
We already had the brilliant researcher Fei Guo on the team; Jenny Mahmoudi made it a team of two. I thought that was enough to support a more rigorous testing program. Our visual designer Peter Irby made a logo, and we made t-shirts — the true sign of mature technology.
In 2013 we gave a Puppet Test Pilots t-shirt to every participant. This helped with word-of-mouth in the conference, and with spreading the word once participants got back to their offices.
When tracking participants in a spreadsheet stopped scaling well, Jenny and Fei used Salesforce to keep track of participants and prospects. We a put a recruitment form on the Puppet website and started formalizing entry criteria. The research team developed a simple questionairre, getting brief answers to role, experience and expertise with Puppet, type and size of company and team.
Now we had a miniature persona interview for each participant, and we could select them more precisely depending on our needs. So when someone in Product came to us with questions about Puppet installation in a multi-DC environment (much less common in 2012 than today), they could query the Test Pilots database to make a good first list of candidates.
We’d send a recruitment email to about two dozen people, hoping to have good conversations with 6 of them. This ratio usually worked out well. We were careful not to lean too heavily on any individual; we only asked people to participate a couple times a year. This got easier as more people were recruited.
Maturity — A Predictable Service
After PuppetConf 2013, user research was more visible in the Puppet community and the company. There were more demands on the team as design and engineering grew, and managing Puppet Test Pilots was starting to become time-consuming. So we hired Mari Federow for Research Operations.
When Mari joined late in 2013 we had 400 Test Pilots; she grew that by 2017 into 1,600. She sharpened our process and our tools until UX research became a predictable service for the company, in particular the product team. Mari’s good work left Jenny and Fei more free to design studies, facilitate interviews, and help product folks interpret the results.
Credits
Some of the Puppet UX Team, Halloween 2013:
JD Welch, Joe Wagner, Randall Hansen, Daniel Sauble
Jenny Mahmoudi, Peter Irby, Fei Guo, Sze Wa Cheung
Many brilliant people in the UX team, and in the larger company, contributed to making Test Pilots a success. Daniel kicked us off; Fei and Jenny expanded and matured our process and tools; Mari wrapped it up to make it shiny and predictable. Many people in Puppet helped, particularly facilitating tests at conferences. My contributions were leadership and air cover.
Leadership
I wanted to be sure that we kept improving — better tests, better facilitation, better results, more actionable information for our product team and roadmap. I guided the team towards pragmatic neat-term goals, while keeping the long-term vision always in our minds.
We made mistakes, large and small, but took them only as signals of where and how to improve. People on my teams typically find this liberating — I would always rather let someone learn a lesson slowly and thoroughly themselves, rather than dive in and prevent their mistakes.
Air cover
As we spent more time on user testing, and on Puppet Test Pilots, it became more important to inform and align leadership around our efforts. For one, we needed help from many other teams for our efforts to be successful: marketing, customer success, bizops. Getting their help was easy, since they saw how effective our work was at providing product insight and building Puppet’s community.
For another, a user testing program is an awful waste if no one uses it. We had to educate other teams in what user testing was and how to use it effectively. Research had to be good enough to be useful, but Product had to find time and space in their work for it. A two-week turnaround on research is fast, but sometimes Product wants to make decisions faster than that. That meant product had to manage their timeline and decision-making a little differently to take advantage of research.
No vacuums
This illustrates the most important point – UX does not exist in a vacuum, and cannot succeed in one. The best UX team still needs good relations and communications with engineering and product, customer success and marketing. UX done well can be a glue organization that helps the whole company maintain focus on its users.