Thoughtful
De-risking a Generative AI Feature
Summary
Product
Thoughtful is an app that helps you combat loneliness and nurture your relationships with a short daily reflective practice. Download it on the Apple App Store.
Context
Users wanted help drafting messages, either to find the right words or save time. We wanted to incorporate generative AI to draft messages, but were concerned that its inclusion might make our product feel inauthentic.
Research Question
Would incorporating a generative AI feature to draft messages to friends and family risk damaging our brand reputation with our target user base?
Outcome & Impact
We ultimately concluded introducing AI into our product posed minimal risk for our brand or user alignment with our mission. In fact, users were generally quite excited about this "meaningful" application of the technology.
Thoughtful's generative AI feature, "pre-drafted messages", was rolled out to all users in October 2023.
Process
Team
1 Researcher (me)
1 Product Manager
1 Designer
My role
Assisted in feature design & Figma prototyping
Led all user testing & analysis
Communicated all results to team
Timeline
From literature review to
share out: 3 weeks
Process Highlights
Step 1
Desk Research
We started with desk research to better understand which segments of our audience (if any) had used AI tools and how they felt about them.
Despite the significant amount of editorial coverage on AI, and the subjects popularity in tech circles, desk research validated our hypothesis that the vast majority of Americans have not tried AI tools themselves. This confirmed the need for more internal research to better understand what the implications might be of incorporating AI into our product.
Step 2
Prototype Design
Next, I worked with a product manager and designer to create a high-fidelity prototype of what AI might look like in product. We used our existing component design to create a few flows, and spliced them into our existing core feature, Daily Actions.
Key demonstrations of AI in Daily Actions included
assistance drafting a message for the anniversary of loss
generating gift ideas for a friend
coming up with hangout ideas to spend time with a friend in real life, based on shared interests
Step 3
User Testing
For agile testing, I decided to do both live interviews and unmoderated user tests.
Live interviews allowed me to collect deeper information about user attitudes towards generative AI, while remote unmoderated tests allowed me to observe organic reactions.
In both test types, no specific questions about AI were asked until the end to avoid priming.
Key positive insights
Key constructive insights
Step 4
Surveys
After the test, participants were asked to complete a survey. It asked questions about mission alignment and overall interest in the product.
We had a significant pool of benchmark data to compare user responses to. I hypothesized that any noticeable shifts in user sentiment could indicate that further research might be needed on this new feature.
However, I ultimately discovered that user alignment with our mission still trended incredibly high, providing additional evidence that positive impressions of the product were unchanged by the introduction of the AI-feature.
Step 5
Analysis & Share Out
I translated my learnings into a brief presentation to the team. Overall, I recommended that we move forward with scoping the feature for build.
the feature had a large potential upside among less confident users, who represented a not insignificant sub-section of our target audience
the minority of users opposed to using AI to generate messages for sensitive dates did not change their opinions of our brand because the feature existed; they would simply not use the feature or turn it off
After hearing my insights, the product team scoped the feature and advanced it to build. In October 2023, the feature — branded "pre drafted messages" — was rolled out to all Thoughtful users.
Other Projects
© Jacqueline Gufford, 2023