Earlier today, ReFiBuy announced the availability of the first detailed user behavior study into how consumers shop on ChatGPT from ‘AI Visibility’ (AEO/GEO) through to the transaction. The data in this newsletter and accompanying podcast are the tip of the ‘interesting and actionable data’ iceberg we discovered in this research. You can download the complete 39 page detailed PDF with many more insights, recommendations plus all the data and answers to questions you may have here.
Backstory: Why We Did This and What is It?
We’re big fans of a SEO/GEO/Growth Substack called ‘Growth Memo’ by Kevin Indig. About 9 months ago they had a foundational report that provided details for how people interact with Google’s AIOs and some of the elements in AI Mode. Where that study stopped short was going beyond the super top-level of discovery - today called “AI visibility” and previously we used to call this, in the Google Era, sentiment analysis.
We researched the company behind this wizardry and found Eric Van Buskirk at Clickstream Solutions. They do User Behavior Studies. What’s a User Behavior study? Good question….
Consumer Behavior: Surveys (Stated Preference) vs. Panel Data (Observed Behavior) vs. User Behavior Study
If you’ve been reading Retailgentic for more than a couple days, you’ve seen us publish a TON of data in two buckets:
Consumer Surveys - These are Jason’s favorites. Customer Surveys have an n=X where X is usually 500-10,000 and represents the number of respondents in the survey pool. Statistically you can take these ‘stated preference’ observations and scale them up to the US population. The core problem with surveys is that what people say they would do and what they actually do are often different.
Panel data -Similarweb, Sensor Tower, Profound, Gener8, collect millions to tens of millions of consumer clickstreams that typically come from Chrome extensions and Android apps that when installed collect consumer clickstream data (prompts and all!). There are several issues with this data:
Given the sources (android/chrome extension users) it can be hard to get demographic representation that matches the US consumer ‘target’.
Millions of streams may be from hundreds of thousands of consumers and hard to scale up statistically to match the US demographic/trends.
Some people question the provenance of this data and privacy practices behind it.
Turns out Eric and team have taken the previously out-of-reach ($100-$300k+) third option, a scientific user behavior study, made it virtual and focused to be more approachable. User behavior studies have fewer foundational datapoints than surveys and panels, but give us strong directional behavior because they are an observed behavior experiment with real user interfaces and demographically chosen participants that meet a statistical target. If you want more information on this, we go deep in the topic with Eric on the podcast.
Our ChatGPT Hypothesis (derived from Amazon Buy Box behavior)
At ChannelAdvisor, 2 years after launching the Buy Box concept on Amazon, where third parties, plus first parties could ‘battle’ for the buy box, we had enough closed-loop transactional data to make a startling observation. The seller that ‘owned’ the buy-box grabbed 85-95% of the sales depending on the category.
Owning the buy-box was the solution to the zero-sum game. If you owned it, you got the vast majority of sales, if you didn’t, you lost all but 5-15% of the sales for that SKU.
After realizing that, we were the first to offer Buy-box analytics, buy-box dynamic bidding, detailed FBA analysis and more. To the best of my knowledge, the focus on buy-box ownership still drives the bulk of Amazon success, or failure.
What about ChatGPT?
ChatGPT is similar, but unlike Amazon, you now have product cards (in a variety of display options), the written/textual recommendations and the offer card.
We had several questions that are not answerable today without a user behavior study:
What’s the value of the first product card vs. third vs. fourth?
What % of people click the offer card and are referred to the retailer vs. influenced?
Once the product card is selected, what do consumers click on next? The first/top offer? The second offer?
What influences consumers - the price, the AI recommendation, or the brand?
Both Google AI Mode (powered by Gemini) and ChatGPT have similar user-experiences, is what we learn from ChatGPT similar to Google AI Mode?
What other interesting observations can we tease out of watching real people shop on ChatGPT?
Finally, this is all intellectually interesting, but most importantly: how do we apply it (like the Amazon Buy Box observation) to actionable strategies and tactics that drive sales for retailers and brands?
Working with Eric, in July/August we executed the study and have been working on processing all the data and results. Today we’re excited to reveal the findings. There are three places you can find the information - this Retailgentic newsletter, the detailed report and the podcast
Retailgentic Deep Dive Podcast Available Now…
This newsletter offers my take on the data from the user behavior study. The accompanying podcast is an interview with the behavioral scientist behind the project, Eric Van Buskirk, with Clickstream solutions.
🎧 Listen wherever you get your podcasts: Spotify, Apple, etc.
Or you can listen right here→
Or, we have a video version on YouTube (make sure to subscribe!) →
Brief Anatomy of the ChatGPT Sentiment→Research→Find→Buy Consumer Flow
Before we jump into my takes on the data, here’s a quick reminder of the three core parts of both the ChatGPT and Google AI mode:
Product Card(s) - This is the consumer’s ‘view’ into the underlying Answer Engine’s catalog and the recommendations for the given prompt. Sometimes a single card is shown, sometimes a carousel, sometimes both, sometimes 2-3 cards in a vertical arrangement. To keep the Amazon analogy going, these are like ASINs, though very basic in their development phase compared to what we’re all familiar with on Amazon. Long-time readers will recognize that making sure your product catalog provided via feed or crawler maps cleanly and 100% correctly to the answer engine’s product cards is called canonicalization.
Written Recommendation - LLMs are natural language engines and great at generating narrative. Sometimes this is below the product cards, sometimes above, sometimes lower on the page.
Offer Card - When a consumer clicks on the product card, they are shown a list of offers that match that product. The sorting of this, just like the Amazon Buy Box is built off of an opaque algorithm. We know the inputs, but we don’t know the underlying algorithm (the output). The inputs are enough for us to decode how to optimize the Offer Card position and give the merchant plenty of ability to make changes to show up higher on the list of offers.
Top 5 ChatGPT Findings and Actionable Strategies from the User Behavior Study
With all that background let’s jump right into the findings. Here are my top five findings along with the actionable strategy from each.
Insight One: Product Card Hypothesis Confirmed, Kind Of…
As mentioned, we had a thesis that being the first product card was important. Turns out that product cards in aggregate on the prompt page draw 75.3% of the consumer’s engagement. On top of that when you look at the influence of the product card - 84% of final product choices in ChatGPT came from a product card
The caveat is our hypothesis guessed that the first product card would be the lion’s share of transactions, but the data tells a different story:
When multiple cards were presented, 43.4% of the time, the consumer picked Position 1, then 17.1% position 2 and, surprisingly, there’s a material bump in position 3 and then it declines to sub 10% for positions 4-6.
Obviously when there is one product card, this surges to as close to 100% as statistically possible.
Actionable Strategy One: Product Cards
The User Study highlights how important it is that your product catalog maps to a product card (canonicalization) and the correct card is chosen. If you don’t do this, you will miss out on 75.3% of the engagement of the page and 84% of the influence.
Long-term readers will know that the key to mapping correctly on the product card is optimizing your product data.
Insight Two: The ChatGPT Offer Card is the New Amazon Buy Box
Interestingly, where our hypothesis proved true was one step further in the shopper’s journey - the Offer Card.
In this graphic, we see that 76% of the consumers picked the first offer on the offer card, with position two plummeting to 14%.
Actionable Strategy Two: Offer Cards
After making sure your products are mapped to the product card, the second priority is offer card optimization. The one input into the algorithm that most merchants won’t change is price, but there are up to 10 other factors, all in the merchant’s control.
At ReFiBuy, we’re actively monitoring over 1m SKUs and in that aggregated dataset, we observe that incorrect offer details drive lower offer card placement for the majority of SKUs.
Why? Because crawler-derived offer data is particularly fragile: shipping details, site-wide offers, and return policies are frequently not reflected correctly. Fix these basics and you can move up as many as 2-3 positions on the offer card.
From there a long-term investment in the five levels of enrichment (covered here) is the next step to improving your inputs into the ChatGPT offer card ranking algorithm.
Insight Three: Product Card Influence > Clicks
The first two insights were what we wanted to know more about with the user behavior study and where our hypothesis was focused.
The biggest surprise from the report came from observing what shoppers did after they moved through the ChatGPT results:
The detailed pdf has all the details, but basically, we discovered that 61% of consumers clicked a product card (when multiple were presented). The users, as part of the task, were asked to pick a product. Even after only hovering on the card (not clicking) they would go to the merchant’s site and pick the product seen on the product card. That’s where the 84% number comes from - any engagement with a product card.
Actionable Strategy Three: Product Cards are the new AI Shelf
In Retailgentic, we’ve talked about Dark Agentic Commerce Traffic. This is additive to that - a consumer very likely will see your product on ChatGPT and be influenced to purchase it by going directly to your website. This will appear as ‘direct’ /organic traffic.
Interestingly, we saw this phenomenon when Amazon launched their first ad products. A popular baby product maker would run an Ad on Amazon and sales across physical stores, their website and other retailers would rise due to the influence of the product’s prominence on the Amazon SERP. How does Amazon compare to ChatGPT?
ChatGPT vs. Amazon’s Digital Shelf influence
ChatGPT announced last Tuesday they have crossed 1.2b weekly active users. Best guesstimate that equates to 1.4b monthly active users. ChatGPT has said that 20% of those users are shopping for products, that gives us 280m shopper MAUs.
Amazon does not disclose global shopping MAUs; it reported approximately 193.9 million average monthly active recipients in the EU for the first half of 2026. The EU is about 1/3 of their revenue so let’s guess 600m MAUs? This is a guess on a guess, but illustrates that ChatGPT’s product influence is likely much larger (about one half the size of Amazon’s) than most merchants realize and certainly is not showing up in click data.
Insight Four: Race to Zero Narrative Violation: Consumers Trust Specs and Brands over Price!
The most common concern that both brands and retailers have about Agentic Commerce is that in a world where agents find the best price, brand doesn’t matter and it’s going to be a race to the bottom.
In the study, we asked participants to weight the trust factors in their product decisions:
34.8% said the product’s specifications drove their choice
29% chose a brand they were familiar with
18.3% chose price/value
The study wasn’t built this way, but my guess is this is a waterfall that goes like this:
If the product meets what I’m looking for, then I look for a familiar brand. If I find more than one, I look at price or perhaps take a risk on a new brand at a compelling price difference, all things equal.
Actionable Strategy Four: There is no Agentic Commerce Race to the Bottom.
Focus on making sure Answer Engines know your detailed product specifications and your brand is properly represented. If a consumer is a value-oriented shopper, Agentic Commerce won’t magnify/minimize that, just like the broader e-commerce and even retail, they represent about 20%. Brand and ‘fit’ still matter the majority of times.
Insight Five: ChatGPT Ads Are Working
The real unintended insight from our study came from ChatGPT ads. By design, our participants were on the ChatGPT free tier, which means they saw ads. The study goes into much more detail, but here are four datapoints that caught my attention:
61.7% of our shoppers saw ChatGPT ads
71.2% rated the ads at least ‘as useful’ as traditional Google ads
48.6% said the ads helped them make a choice at least a little
35.1% said they were influenced in their final decision by the ad.
Sometimes insights leave you wanting more. In retrospect, I wish we had measured more about the impact of ads in the design of the study, and, of course, one datapoint is interesting, but not a trend. It would be interesting to see what these datapoints were closer to April/May to see the trajectory. Ad systems are closed-loop learning systems, particularly in the hands of an AI frontier lab. I imagine, the datapoints are increasing from that April/May and by this time next year, the influence will have at least doubled.
Actionable Strategy Five: Don’t Sleep on ChatGPT Ads
It’s early and we need much more data, but directionally this is promising. ChatGPT appears to be executing towards their goal of ads that simultaneously don’t erode trust, add to the overall experience and give the advertiser high ROI for the impact. They may not be there yet, but if ChatGPT can optimize the ad system at even half the logarithmic curve the models are growing on, these ads are on track to give Google, Meta and Amazon a run for their money.
Remember, ChatGPT’s ad system crossed $1b in 200 days, beating Google, Meta and Amazon to the same milestone.
Insight Six: BONUS Insight: Data Quality Problems Drive Consumers Crazy
In the podcast, you’ll hear Eric’s description of how each consumer study was conducted. One important element is the shoppers are asked to verbalize the experience as they go. These three comments from participants show the importance of making sure your product data is picked up 100% correctly by ChatGPT and updated regularly via a datafeed:
Actionable Strategy Six: Details Matter
Invest in making sure ChatGPT and other Agentic Commerce surfaces have the big details right and measure it to make sure you are not eroding consumer confidence in your products and brand.
Interestingly, product data glitches were the biggest negative consumer experience we discovered from the study.
Summary
I hope you enjoyed these six highlights from our agentic commerce User Behavior Study. Here’s a summary of the action items:
Product cards drive 84% of the product choice influence - focus on making sure your catalog is mapped correctly. Correct canonicalization (product card mapping) is your friend, from there make sure the ‘hero’ image is right, the price is dialed in and updated correctly as reflected on the product card. Be sure to measure the % of your catalog that is mapped correctly as a KPI.
The offer card is the new Amazon buy box - driving 76% of the ‘wins’. Once mapped correctly, make sure your offer is reflected and enrich your product data to further enhance your position. Measure your relative offer card position as another KPI.
The product card drives significantly more purchase influence than measurable clicks. Work on tuning your analytics to measure the known DACT and most likely a % of your direct organic traffic is being influenced by ChatGPT product cards. Think of ChatGPT product cards as part of the larger AI Shelf.
There is no evidence that Agentic Commerce shoppers are using the new technology for a ‘race to the bottom’. Instead, Agentic Commerce empowers them to get to the right features and brand they want faster.
ChatGPT ads are surprisingly impactful and viewed positively or neutrally by shoppers. Running a test in the next year may be prudent.
Shoppers hate data quality problems, make sure this holiday if you run promotions they are correctly reflected across all agentic commerce surfaces.
The detailed user study presents 8 different action items based on the broader dataset.
You can download the detailed data here to get the complete study in a ~40 page PDF that also includes the detail of all the shopper demographics, the tasks they were given, the categories they shopped and much much more.