Scott we are the commerce execution backend for many large B2B and B2C companies. One of the questions I have for you is something you said in minute 48. There are many new ways to get you content. The bots and more importantly the impact of the bots on customer experience are getting worse and worse. For instance I have a customer who is working right now with their Infosec and Infrastructure team to deal with what they call malicious bot crawling. In their voracious appetite for data they are entering like a human through the browser and in the search bar entering search terms that intentionally pull back huge amounts of data like .h, h/brand, .blue, .red, or class. This puts extreme performance pressure on their search engines because the loads coming back with data are huge and no human would ever search in such a way. Both the engine and the infrastructure the engine runs on is not optimized for that kind of behavior. One of the ideas we have been throwing around is creating two optimized infrastructures one for humans with subsecond response and another for bots that will chug away until they produce the result and then using front end CDNs to route the traffic to the appropriate use. This is a B2C customer who wants to support Agentic Commerce. In these new ways to get the major agentic commerce companies data will they then turn off the bot crawling so it's easier to separate who the Retailer wants to get the data too vs every Tom, Dick and Harry who can write a malicious bot crawler causing degradation in site performance for no real gain?
Hi Erik - good question! First, a disclaimer - I am not an infrastructure expert. That being said, depending on what system you are using, the first thing you can do is white-list the known 'good' bots - there are 20-25 of them, such as GoogleBot, ChatGPTbot, etc. I don't believe they do anything like the behavior you are describing.
Another best practice is to have a llms.txt file - in there you can give LLMs (not all bots) hints about how to search the site and find what they want.
Of course it sounds like this bot you are dealing with is somewhat malicious/unhinged and should be 'blacklisted'.
yeah sadly, mcp to chatgpt/amazon/gemini/Muse isn't 'a thing' - you're looking at a mix of datafeeds, metadata on your site and UCP/ACP protocols that are more of a 'pull' than an MCP 'push'.
What was rolling around in my head here was whether, if I use the preferred protocols, like maybe MCP, to pump my ACO-optimized content to all the whitelisted engines that I chose to work with, I can block all bot crawling and reserve my website itself for high-performance human interactions. From an infrastructure perspective, pumping the data through a protocol would be so much more efficient than creating a different version of my site that would support bot crawling. Of course, as you say every day, if you don't get your data in order, you will never win in the ACO battle.
Scott we are the commerce execution backend for many large B2B and B2C companies. One of the questions I have for you is something you said in minute 48. There are many new ways to get you content. The bots and more importantly the impact of the bots on customer experience are getting worse and worse. For instance I have a customer who is working right now with their Infosec and Infrastructure team to deal with what they call malicious bot crawling. In their voracious appetite for data they are entering like a human through the browser and in the search bar entering search terms that intentionally pull back huge amounts of data like .h, h/brand, .blue, .red, or class. This puts extreme performance pressure on their search engines because the loads coming back with data are huge and no human would ever search in such a way. Both the engine and the infrastructure the engine runs on is not optimized for that kind of behavior. One of the ideas we have been throwing around is creating two optimized infrastructures one for humans with subsecond response and another for bots that will chug away until they produce the result and then using front end CDNs to route the traffic to the appropriate use. This is a B2C customer who wants to support Agentic Commerce. In these new ways to get the major agentic commerce companies data will they then turn off the bot crawling so it's easier to separate who the Retailer wants to get the data too vs every Tom, Dick and Harry who can write a malicious bot crawler causing degradation in site performance for no real gain?
Hi Erik - good question! First, a disclaimer - I am not an infrastructure expert. That being said, depending on what system you are using, the first thing you can do is white-list the known 'good' bots - there are 20-25 of them, such as GoogleBot, ChatGPTbot, etc. I don't believe they do anything like the behavior you are describing.
Another best practice is to have a llms.txt file - in there you can give LLMs (not all bots) hints about how to search the site and find what they want.
Of course it sounds like this bot you are dealing with is somewhat malicious/unhinged and should be 'blacklisted'.
yeah sadly, mcp to chatgpt/amazon/gemini/Muse isn't 'a thing' - you're looking at a mix of datafeeds, metadata on your site and UCP/ACP protocols that are more of a 'pull' than an MCP 'push'.
What was rolling around in my head here was whether, if I use the preferred protocols, like maybe MCP, to pump my ACO-optimized content to all the whitelisted engines that I chose to work with, I can block all bot crawling and reserve my website itself for high-performance human interactions. From an infrastructure perspective, pumping the data through a protocol would be so much more efficient than creating a different version of my site that would support bot crawling. Of course, as you say every day, if you don't get your data in order, you will never win in the ACO battle.