Challenge
How fast could we build something to prove or disprove our hypothesis that new LLM technology could be extremely useful for customer service reps?
Solution
Provide an experimental set of tooling that could assist with the task of composing a customer service message.
My Role
AI Product designer
Personal Takeaways
AI Assist was the fastest idea-to-launch project I’d worked on up until that point. It proved both to myself and to my new team that I could ideate and iterate very quickly when working directly with the engineering team.
SITUATION
I had started at Intercom a few months earlier with the mandate to help the machine learning team prototype and release features more quickly. We were in the middle of a strategic workshop reviewing existing research and brainstorming potential upcoming projects, when ChatGPT was released and the AI world went crazy. We realized it held a lot of potential for the two primary jobs of customer service reps: finding the right solution to a customer problem, and composing an appropriate response to that customer.
PROJECT GOALS
Our first goal was to test out the new GPT3.5 model to determine where it held promise and where it performed poorly or was too risky to use. We quickly discovered that the model was really great at text transformation tasks.
We decided to set an ambitious goal to be the first company to prove actual value from this new tech—we wanted to become experts at using these models, as well as be seen as “first movers” in the space.
DESIGN APPROACH
We systematically looked at what GPT3.5 was good at, and what it was less good at. We wanted to understand the potential risks first-hand. To determine which functionality to build, we pulled from my well of CS knowledge from prior roles, and combined it with some experimental approaches from other teammates. I say “we” because I was working in the same way as the engineers: trying different prompts with sample datasets and trying to find the edges of a good experience.
The ability to quickly summarize a conversation up until the current point was one feature for which I advocated strongly, based on prior existing user research and my product intuition and knowledge. The most experimental one was for “expand,” which was supposed to flesh out a brief skeleton of a response into something coherent and complete. We also decided to build a few basic tonal shifts, allowing reps to make their writing more formal or friendly.
Our Inbox product had a primary method whereby users would trigger different actions: the command palette. I learned as much about the Inbox design patterns as quickly as I could so that we could build it into production quickly. For the first iteration, we launched the functionality from the command palette.
PROTOTYPE + TESTING
We were lucky to have an internal customer service (CS) team that “dog foods” most of the functionality that Intercom releases, so we conducted some research sessions where we introduced this functionality to internal CS reps and had them test it out in front of us, with live conversations in their Inboxes.
During these sessions, I watched as teammate repeatedly typed something into the response composer, and then tried to highlight parts of the text to make changes to it. I realized they were approaching this functionality as if they were making a sentence bold or italic, and that we needed to make this functionality available from a toolbar. We actually needed two ways to access this feature—the command palette was useful as a general reference, but it just wasn’t close enough to the context of the written response. So we enabled users to highlight all or a portion of their text response, and then transform the text as they normally would.
LAUNCH + OUTCOMES
We built this product in 6 calendar weeks over the winter holiday period—an impressive feat! The feature launched in beta with quite a lot of customer interest, which did lead to us developing a positive “first mover” identity. The most-used and most-lauded functionality was the summarization, with the “more friendly” being used frequently as well.
You can read more about the features in the launch blog post.
Details
URL: Feature release blog post
Date: Dec 2022 – Jan 2023
Role: Product designer
Tools: Figma, GPT3.5