Role: One of two designers on the Flow product while owning design and research on project to increase trust in automation products. Close guidance from a UX researcher, UX lead, content strategist, and working alongside three developers.
Results: Qualitative research around how users feel about automation has helped the team redirect priorities on user experience and product performance. Redesigns to keep users in the loop helped increase user discovery on errors by 5x.
Background and Context
Enterprise businesses on Shopify Plus complete a lot of repetitive tasks. These include tagging orders with a specific property, or capturing payment when an order comes in. Shopify Flow is a new product within Shopify Plus that allows these merchants to build automated tasks based on if/then logic with the goal of driving efficiency in their e-commerce workflows. With this product being relatively new, my projects revolved around maturing interactions between Flow and their users, as well as third-party developers.
While gathering signal from merchants to improve error message content within Shopify Flow, a bigger problem surrounding trust and automation surfaced when I noticed users’ hesitations about using Flow. Whether or not Flow was working as a merchant expected, the lack of situation awareness results in our merchants mistrusting Flow. A large portion of users would speak about Flow with a cautious tone and were reluctant to rely on automation to conduct more high-risk workflows such as capturing payment.
My first instinct was to speak to as many merchants and merchant representatives as possible. I immediately reached out to our UX research team and merchant success team to gather any techniques, tools, and resources.
Through gathering stories about merchants’ first interactions with Flow, to collecting experiences of merchants handling unexpected errors, I plotted them out into an affinity diagram to detect any patterns. Two key issues began to surface.
- Users did not understand when Flow errors were happening or why
- Users could not detect their own logical fallacies when creating their workflow
Additionally, I supported these theses with external research about human-centred automation. Principles about human-centred automation state that automation systems should keep users in the loop in any events and have any information pertaining to how to manually take over a machine upfront and clear. Flow wasn’t doing any of these things, which resulted in the two issues stated above.
Because Flow didn’t surface the errors that were happening within workflows, there were often repetitive errors that wouldn’t be fixed over a period of time. To define what it meant to solve this problem, we decided that a short-term, quantitative success was to decrease the ratio of action runs with errors over time, and in the long-term, have more merchants creating higher risk workflows such as those relating to payments.
A more qualitative measure of success was to increase merchant trust in the young Flow product and have users begin to default to opening the Flow app to solve more repetitive tasks.
Breaking Assumptions and Exploring Solutions
Going forward, there were a few pieces of low-hanging fruit. Implementing easy-to-understand copy instead of HTTP codes so that users could better understand what the errors meant. I worked closely with the engineering teams to better distinguish where the errors were coming form and translate it to actions that users could understand and take.
Though this greatly increased understanding in Flow errors, testing with users showed that this error discovery was still quite low. The messaging was only helpful to those who could find the errored workflows.
As I spoke to more and more merchants, I gathered more put together stories about exactly how people were currently discovering errors. Below, is an example user journey showing the discrepancy between when an error happens, and when the merchant discovers it. I then ideated on solutions that would then bubble up the error discovery to as close to the actual error event as possible within the user journey so that users were as in the loop as possible with how Flow was performing.
I explored solutions both within the Flow application as well as outside of it. Knowing that there was also a separation of truth within a lot of our customers’ organizations, using as many communication channels as possible was a way to provide information to the right people at the right time.
- Keep users informed and in the loop: Before, users had to click into a specific workflow and the specific action run to understand why the errors were happening. This proposal bubble up these errors with a visual indicator to provide a summary of information when hovered over. This solution increased error discovery time by threefold.
Other communication channels that we would implement error discovery was the general Shopify push notification center so users didn’t have to enter the Flow dashboard to gain an understanding for Flow actions, as well as emails for more high-risk workflow errors. By considering not only the Flow product, but also the greater Shopify ecosystem, these solutions brought a more holistic understanding of errors across large enterprise e-commerce merchants.
2. Keep users trained, and make automation flexible and adaptive: An additional solution here was more proactive than reactive. Instead of relying on Flows breaking to provide a solution, we added an intermediary state before making a workflow go live to preview the different options in the logical steps. This allowed a visual way for users to train their understanding for if/then logic, especially as a lot of our user base was non-technical.
It also made it increasingly easier to simulate and show logical or technical fallacies that users would make before they would happen. Though it added a high-friction step between creating a workflow and letting it go live, it allowed 100% of users to better understand the logic behind Flow as opposed to 0% of people who wouldn’t take the time to review a workflow.
By centring iterations around these findings and research principles, I presented each iteration to the Flow team, as well as guerilla tested internally to gather signal on whether or not these solutions were solving the problem at hand.
To measure success, I was able to quantitatively record the number of times and the duration it took for error discovery. Through testing the new workflows against the former, two key quantitative successes and one major qualitative success surfaced based on the measures of success defined earlier:
- System error discoverability increased by 5x when presented with the new visual treatments
- User error discoverability took 20% of the time it originally did when presented with the preview workflow experience
- Merchant success managers gained confidence in their knowledge of Flow, and were more likely to recommend it as a solution for higher risk workflows.
Reflections and Next Steps
Results and Impact
Though this project was implemented after my internship ended, it was a great chance for me to interface closely with the engineering team and gather feedback holistically from both technical and user experience ends. By working with talented UX researchers and writers, it also helped me broaden my understanding of the design process and how to get heads involved as early as possible.
Moving forward, I had helped define product success for what it means for Shopify users to trust Flow as it starts growing a user base. By telling these user stories to the Flow team and emphasizing on the importance of trust in UX, the empathy they gained kicked off a back-end performance-centric project among the engineers to address the runtime issues that often caused user-facing problems. Additionally, we partnered with product marketing to create more educational video tutorials for new automation products — an initiative also driven by the principle of increasing trust.
However, some opportunities that exist within this project as well as to learn and expand my skillset was to continue exploring different micro-interactions to provide the right granularity of information at the right stage, as well as paying more detailed attention to visual design clunkiness. It was helpful to have Shopify’s design system as a guiding light, but I wish I had more time to continue developing visual states for error discovery.
Overall, Being able to extend my research and impact outside of a small error messaging project was I think the most fun and most impactful actions coming out of this project.