ecobee
Field testing ecobee Home Monitoring
Scroll ↓
Role
User Researcher
Methods
Surveys, in-home visits, interviews, usability testing.
Industry
Consumer electronics, smart home, IoT
Tools & Frameworks
SurveyMonkey, Excel, Dovetail, User Interviews, Mail Chimp, DocuSign. CSAT, SUS, NPS
Impact
Planned and executed research for a 6 month, 400+ participant beta program
Created a UX benchmarking system to help executives and product managers assess launch readiness
Facilitated a 40 point NPS increase over the course of a 6 month trial and a 4.5 Amazon star rating post-launch
Project
The challenge
In 2019, ecobee was getting ready to launch a suite of products under a new line of business: ecobee Home Monitoring. The product launch included two new hardware products and a software service. We were in uncharted technical and UX territory, so we invested heavily in product testing and validation. Over six months, I worked with Product Managers, Product Marketers, and UX Designers to facilitate a 400+ participant beta program and benchmarking system that served as a weather vein for launch readiness.
The approach
This beta program sought to answer 5 questions related to launch readiness:
Quality & performance — are there any software or firmware bugs? Are the products performing as designed?
Customer satisfaction — are the products living up to customer expectations of a home security system?
User Experience — are the core features of the products comprehensible, dependable, and delightful?
Product Usage — is the product being used in the way that we expected?
Feature prioritization — What are the most impactful features to add post-launch?
Process
Planning & Recruitment
I collaborated with Product Mangers, UX Designers, and marketing to define the research objectives for the beta program. From there, we worked together to define our recruitment criteria, balancing user behaviours and traits with demographics. Over the course of 6 months, our 400+ participants were segmented into six cohorts to test different iterations of the product, as well as different product combinations.
In addition to defining the goals of our beta program, I also conducted a retrospective on past beta programs run by ecobee to inform how we could improve aspects such as timing, tooling, data collection, and reporting.
A photo from a home visit which we conducted to test our set-up and onboarding experiences.
Field Testing
Over the course of 6 months, we conducted multiple rounds of home visits, surveys, and phone interviews. We benchmarked engagement using both traditional metrics (such as Customer Effort Score, CSAT, and NPS), as well as features specific metrics to measure engagement with our core features.
Synthesis & Reporting
Customer satisfaction data and UX metrics were analyzed and reported on a bi-weekly basis. Quantitative data was used to give stakeholders an impression of how products might be perceived by customers post-launch, while qualitative data was used to prioritize what needed to be changed in order to launch our products successfully. As such, performance and usability issues were communicated to Product Managers on a weekly basis.
Impact
The research program we built layered qualitative behavioural data and attitudinal interview data with quantitative usability and satisfaction metrics. The usability and satisfaction metrics helped us tell a story about product progress towards launch, while the qualitative data helped us learn what we needed to change.
Foregrounded customer voice
The approach I designed was a launch-readiness framework that foregrounded customer voice. This framework led us to improve our NPS by 40 points throughout the trial, and achieve an Amazon rating of 4.5 stars post launch
Evidence-based launch framework
Opinions on how to move forward often conflict in times of stress. Our regular reporting sessions created a source of truth and aligned our stakeholders to a common goal: listening and acting on feedback from customers.
Prioritized roadmap
Customer feedback collected during our trials provided evidence to drive the development of new features and products post launch.
Team
Research
Michelle Johnstone
Dennis Lagman
Nav Grewal
Kristen Thompson
Product Management
Steven Aitken
Mirvise Najafe
Emily Santili
Matas Sriubiskis
Dane Woodburn Morgan
Marketing
Harrison Levy