AI is here, is the design field ready?
How the traditional methods of measuring the effectiveness of a design might change in this new agentic era.
Disclaimer. This blog has turned out to be a bit longer than I imagined. I would recommend utilising the audio version of this rather than spending about 11 minutes reading this. Thanks to Speechify, Gwyneth Paltrow will read it for you :D
“Jarvis, are you there?”
”For you, sir, always!”
This scene from the first Iron Man movie has always given me goosebumps. Apart from typing his initial start sequence, Tony Stark, has not been shown to be interacting with keyboards or mice for the rest of the franchise.
When the first movie came out in 2008, the word AI was being used only as something mysterious and complex.
Fast forward to a few days ago, when every single AI company is releasing agents. Comet the agentic browser from perplexity, Gemini being always ready in the browser, to Claude MCPs, all information technology is moving towards handing over the simple repetitive tasks to AI agents. These agents are lightweight AI instances, trained to do a specific task really well, that understand the human intent and then perform the tasks across multiple websites/apps to achieve the same result, faster and across multiple windows. For a person who comes to the internet with specific intent, this agent will be more effective than any human being on this planet. Like bruce lee once said,
"I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times.”
The art of user experience design
I started working in the field of UX back when Jony Ive was getting obsessed with gradients and hairline font weights. When we talk about design, the fundamental idea was to create a guided path for the users to achieve their goals. Or if your company is greedy, the design was all about creating a guided path to make users buy more stuff. This meant finding out what the user is trying to achieve, what the company is trying to make the user do, and then combining that into a well-lit path in the maze that users can follow. The Interaction designers are obsessed with crafting a path for the user to follow with proper signboards and warning signs, so that all users can achieve their goals with minimal resistance and/or minimum clicks. But now, with this new agent-based way of performing tasks, how would the design change?
Traditionally, people have been introducing chat interfaces within websites or applications to perform certain tasks or ask questions. However, this takes the user off the guided path and might not be a good idea if it is not well integrated to understand the user. Also, considering the limitations of the AI models, the output they produce is only as good as the context they have. This means the AI model needs to be extremely aware of what the user is trying to do, what the user has done in the past, and if the user needs assistance. For example, a chat interface design to handle the basic queries from the users needs to be of aware of which step of the flow the user is on and be Proactive in anticipating the questions the user may ask. If the user has to start from the beginning, the integration of the chat interface may result in the user losing focus, and the chatbot might end up proving detrimental than helpful.
This is a screenshot from chatbot.ai. A company dedicated to building AI-based assistance within your websites. Have a look at the example they have stated on the website itself. The customer is asking about the expired code while the AI assistant is asking the user to perform a task instead of being contextually aware, checking the logs, determining who has logged in, and sending the email or message directly to the user.
Deterministic vs exploratory users
Before we dive deeper into the impact on the UX as a domain, let’s try to understand the user behaviour types. Broadly, we can classify the users into two categories: Deterministic users and exploratory users.
Deterministic users are the kind that come to an app or a platform with solid intentions to complete the task for the real world. They come to the platform, perform the transaction, and leave. The transaction is mostly straightforward with add-ons or options. Taxi booking, for example, is done by a deterministic user. The users already know they want to go somewhere. They already have a rough idea about how much they are willing to spend. If the pricing is within the pre-thought range, the users sail through the entire flow. No second thought needed.
Exploratory users, on the other hand, are the users who have a vague idea about what they wish to achieve. There could be yes-no-maybe kind of scenarios, comparisons needed to complete the tasks, or simply a user trying to get knowledge about an unknown topic. In these cases, users may or may not complete the tasks. Fashion, e-commerce could be one of the classic examples of exploratory users.
Back to Design
Impact on deterministic user experience
Now that we have a basic understanding of how deterministic and exploratory users interact with systems. Let's dive deeper into how the interfaces might evolve to support these behaviours
For a deterministic user, the focus will not be on completing the task but more on anticipating the user's needs. The paths of execution are straightforward, and instead of just performing them, AI can learn to analyse the circumstances under which the deterministic user behaviour is triggered and be ready to execute. For example, if the AI detects that Isaac orders a taxi to go to his office roughly between 8:30 to 8:45 a.m. AI can get ready to find the taxi proactively at 8:25. The task will be completed every single time without the users having to perform an action.
As a product designer, It is my job to care about the user experience matrix as well. In these cases, the way we measure user experience will not be on a traditional matrix like time to task or number of clicks. But, the better ways to measure the UX metrics would be to measure the following
Proactiveness
Like we discussed in the example above, the measure to evaluate the user experience would be to check if the Agents are performing the tasks proactively, without users having to get involved. We might also check if, in the time of unexpected circumstances, the agents have proactively communicated not just the problem, but also the solutions for the users to finally make a decision.
User control
But while we want AI agents to perform the tasks, the ultimate decision should always remain with the humans. (Why? one word answer… Skynet!). The metric in this case will have to evolve since, traditionally, the users were the decision makers. The way to measure that would be to check if the users can abort the actions anytime and/or reverse the results of the process performed by the AI.
Transparency
The AI models may make mistakes or misinterpret human intentions. That's the reason why all AI models come with a warning sign. The AI model should always show the numeric value of confidence. This means the AI should provide the users with an accurate measure of how confident it is about the information being presented or the appropriateness of the suggested actions.
Consistency
Users traditionally have experienced the consistent layouts across the internet. All hail Jakob! With the new age where the users are just the decision makers while AI uses all the tools, the users’ control will change significantly. The users will shift from doers to governors. This will give rise to a new paradigm of interfaces, not only visual but voice and gesture-based. Alignment with these newly established patterns will be needed to ensure consistency and avoid user confusion. Everybody is growing impatient!
Adherence to the user goals
Users traditionally achieved their goals by navigating through the interfaces. Now, with this new age of agents, AI will be the one navigating through the paths, and the measurements would be based on the alignment of the agent with the actual user needs. The metrics would be to look at how well the agent performed on identifying the cheapest option or the fastest option based on what the user needed at that specific point in time. In case of a user trying to learn, the measure would be if the agent helped the user seek an answer or simply gave it away.
Accessibility
We are getting older. Sooner or later, all of us will need one or more accessibility features to operate the phones. (I am already using font scaling). When it comes to deterministic users, the human has to just provide instructions and monitor the outputs. As long as that medium of communication is accessible, these agents can do wonders for people with disabilities. Let’s assume, for deterministic use cases, users will not open the apps or websites, but ask the assistant built into the operating system or browsers to perform the task. These agents do operate within the chat interfaces built by global giants like Google, OpenAI, Microsoft, or Apple, and it would be safe to assume they will obey the basic laws of accessible design.
Impact on exploratory users
As we have discussed before, the exploratory users approach the tools or apps with a more vague idea of the goal. They could be on the app to find information, browse through products, compare things, or simply consume content. With agents coming into play, these kinds of users will face more drastic changes. The UX metrics too, would drastically change from their current form and shape with the AI taking over the interactive part on behalf of the users.
For the knowledge seekers, the agents will miraculously alter the way users learn. Traditionally, knowledge consumption happened through links embedded inside the text or added at the bottom of scholarly articles. Agents, however, can understand the tech-savviness of the users and cater (maybe rewrite) the article for the users, instead of guiding the users through static texts. The agents will “fill the knowledge gaps” if I were to quote Microsoft’s guidelines.
For users who wish to browse and compare products, AI can help with narrowed-down options. The agents can consider the things beyond traditional filters and search keywords and focus on what the users need the products or services for.
For users who are trying to consume content, the AI agents can strike a balance between structure and Serendipity. Although the AI agents can bring back exactly what the user needs and has preferred in the past, it is essential to keep the “joy of discovery” alive as a human value.
Again, when it comes to measuring the impact of design for the exploratory users, it would be good to ditch the traditional user satisfaction surveys and focus on these new methods of measurement where AI agents are using most of the app while users are just decision makers or ultimate consumers of the information. The traditional conversion rates and revenue metrics still hold true, but here are a few metrics that should be added on top of the existing ones.
Cross and upselling
AI agents can retain context over long periods and analyze users' likes and dislikes to provide recommendations. The measure of success would be how well the agent handles repeat orders of consumables while successfully suggesting novel items that align with users' adventurous sides, and whether users accept these recommendations. These recommendations should be based on the users’ behaviour instead of just being the ‘related products’
Filter bubbles
Another measure is to ensure that the agents do not create filter bubbles. It means that the recommendations by AI are not restricted by the user preferences. This will force users to choose from recommended options, and users might churn due to an artificial lack of options.
Foreground vs background time
When it comes to exploratory users, serendipity is an important factor. The AI agents should always be available but working behind the scenes. The agents should only chime in with the recommendations when the users have asked for help or have broken their search pattern, indicating they are either lost or feeling adventurous. This is true for browsing and product research.
However, when the agents are trying to help users gain domain knowledge, they should be more proactive in suggesting the right articles to follow or fill in the knowledge gaps.
Transparency
While filling up the knowledge gaps, the agents should always quote the sources. The agents should maintain full transparency with the users by quoting the sources and allowing users to make a call on whether to accept the information.
While recommending something, the agents should also quote the reason behind the recommendations so that users are not kept in the dark about AI analysing their usage patterns.
Divergence
Traditionally, the machines are trained to analyse the patterns of multiple users and provide recommendations based on user segmentation. With AI agents running real-time pattern analysis, they should be able to break the ranking and provide recommendations based on what the user might be interested in at that moment. This recommendation can be more personalized based on the analysis of what users don't want.
Accessibility
How can I not talk about accessibility? Unlike the deterministic users, who are on the app to achieve a very specific task, exploratory users are ready to invest time in your app. This means the accessibility of these apps needs to be on point. Considering that the exploratory users will be using the app’s interface to gather information, all the navigation and interactive elements need to be labelled properly and should follow the POUR principles. Visuals or images play a vital role in the exploration, and thus all of them need to be properly labelled within the context. For exploratory users, accessibility could become a major factor in the user funnel as the people with disposable income are aging… rapidly!
In conclusion;
As we move rapidly to this new age beyond the mere information age, it is important to focus on effective design instead of just slapping the AI agents on top of existing processes. We need to understand that we are entering an age where we are not just developing tools, but we are creating entities that can utilise these tools to help us achieve our goals. This means the traditional way of measuring the effectiveness of the design, or in this case, measuring the effectiveness of a system, might not be relevant. Nobody knows what the future holds; however, it is a good practice to start looking at how the fundamental processes are going to change to ensure we can improve upon the systems by measuring them accurately, beginning today!