What It Is

As a Conversation Designer at Microsoft Advertising, I designed and developed the MAST Virtual Agent. This initiative aimed to enhance customer support efficiency and improve user experience by automating responses to common queries related to account management, billing, and campaign operations. Our goal was a complex one: enable customers to self-help for support scenarios, but also enabling customers to quickly reach the right human agent queue for acquisition and sales scenarios.

Here’s the Problem

Microsoft Advertising experienced a surge in support ticket volumes, leading to longer response times and decreased customer satisfaction (CSAT). Many of these inquiries were routine questions that could be addressed through automation. To streamline the support process and alleviate the workload on support staff, my team developed and launched the MAST Virtual Agent.

My Role and Contributions

In my role as Conversation Designer, I was responsible for:

• Research and Planning: I analyzed competitor designs as well as our own historical support tickets and conducted comprehensive research to identify common customer pain points. This analysis informed the foundational structure of the virtual agent’s conversation flows.
• Research and Planning: Recognizing the importance of a realistic testing environment, I advocated for and established a prototype that mirrored the customer-facing interface. This proactive approach allowed for thorough validation of the bot’s performance prior to its official launch.
• Dialogue Design: As a designer with a user-focused mentality, it was critical to me that I see the end-user experience as I design dialogues. Therefore, I taught myself the Power Virtual Agents platform so I could be both the designer and the developer for the Virtual Agent.

• Routing Strategy: I developed a robust routing system to handle complex scenarios, ensuring that only the correct query gets to the correct human agent queue. This strategy involved detailed mapping of dialogue paths to appropriate support channels.

• Introducing Natural Language Understanding (NLU): Our Virtual Agent began as a button-based experience with no user typing. As it grew in complexity, we realized NLU capability was critical to enable customers to quickly navigate. I built a business case to convince leadership of NLU’s value, then worked in Power Virtual Agents to redesign our conversation flow from the ground up to enable a seamless typing-based chatbot experience.

Results and Impact

The MAST Virtual Agent yielded significant improvements in support efficiency:

• Increased Conversations: User engagement with the bot rose by over 50%, as evidenced by the December FY24 data.
• Reduction in Escalations: I optimized escalation protocols, leading to a decrease in unnecessary human interventions.
• Self-Help Success: The self-help success (SHS) rate for solution topics remained robust, contributing to higher customer satisfaction.

Challenges and Lessons

Throughout the project, I encountered and addressed several challenges:

• User Privacy in Testing: Due to privacy constraints, our team couldn’t access actual user conversations. To make up for this, we dove deep into data to understand where users were self-helping, where they were escalating from, and where they were abandoning the chat experience before resolving their query.
• Unauthenticated User Interactions: The chatbot operated outside the authenticated environment, making it challenging to verify the legitimacy of sign-in issues. This limitation required the development of strategies to differentiate between genuine user problems and potential fraudulent activities.

These challenges provided valuable insights and informed continuous improvements to the virtual agent, enhancing its effectiveness and reliability.

In Short

The initiative not only enhanced operational efficiency but also delivered measurable improvements in customer experience. The lessons learned from this project have been instrumental in shaping my future chatbot initiatives, ensuring ongoing innovation and value delivery.