--- headline: The Internet is Rebuilding Itself for Agents. Are You Ready? author: Wishesh Sterling datePublished: 2025-12-19 dateModified: 2025-12-19 category: Digital Marketing tags: - generative-engine-optimization - ai-agents - seo - llms.txt - agentic-commerce --- # Context This document is a strategic analysis and guide for businesses navigating the profound shift in internet architecture towards AI agents. It outlines the transition from traditional SEO to Generative Engine Optimization (GEO) and introduces new technical standards like `llms.txt` and the Agentic Commerce Protocol (ACP). The post aims to educate and prepare brands for increased AI visibility and participation in agent-driven discovery and transactions. --- The Internet is Rebuilding Itself for Agents. Are You Ready? ============================================================ The era of "Googling it" is ending. As AI agents replace traditional search, businesses must shift from SEO to Generative Engine Optimization (GEO). Discover how to audit your AI visibility, implement \`llms.txt\`, and make your brand accessible to the machines that now control the internet. [Wishesh Sterling](https://surgeo.ai/blog/author/wishesh-sterling) ------------------------------------------------------------------ Dec 19, 2025 · 10 min read Generative Engine Optimization GEO Agentic Search AI Marketing Future of SEO llms.txt Agentic Commerce AI Visibility Search Strategy ChatGPT Optimization ![The Internet is Rebuilding Itself for Agents. Are You Ready?](https://surgeo.ai/m/ygzzfk-l.webp) **Key Takeaways** ----------------- - **Traffic Shift:** Traditional search engine volume is [predicted to decline by 20 % to 50%](https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/new-front-door-to-the-internet-winning-in-the-age-of-ai-search) for unprepared brands as users migrate to AI-driven "answer engines" \[1\] - **New Architecture:** The web is moving from keyword matching to "[Query Fan-out](https://www.kopp-online-marketing.com/from-query-refinement-to-query-fan-out-search-in-times-of-generative-ai-and-ai-agents)," where [agents break complex requests into parallel sub-tasks](https://arxiv.org/abs/2401.06311) \[2, 3\]. - **GEO over SEO:** A [Princeton study](https://collaborate.princeton.edu/en/publications/geo-generative-engine-optimization/) indicates that optimizing for Generative Engines (GEO) using citations and statistics can boost visibility by up to 40%, whereas traditional keyword stuffing may decrease it \[4\] - **Technical Standards:** New protocols like [llms.txt](https://llmstxt.org/) and the [Agentic Commerce Protocol (ACP)](https://stripe.com/blog/developing-an-open-standard-for-agentic-commerce) are establishing the infrastructure for machine-to-machine discovery and transactions \[5, 6\] We have entered a strange, interim period in the history of the internet. If you look at standard web analytics, you might see a worrying trend: referral traffic from search engines is softening for many content-heavy sites. Yet, overall digital activity is higher than ever. The disconnect comes from a fundamental change in user interface. For twenty years, "Google that" was the reflex for curiosity. In the three years since ChatGPT launched, that reflex has been rapidly overwritten by "Ask the AI." We are moving from an era of **Search** (finding a list of links) to an era of **Agentic Discovery** (getting an answer). ![search-and-discovery-evolution.jpeg](https://surgeo.ai/m/e6n7jm-m.webp) *Evolution of Search and Discovery (arXiv:2506.18959) [7]* This isn't just a change in how we type queries; it is a change in the economic infrastructure of the web. The scaffolding for an "Agentic Future" is being built in real-time. With ChatGPT [](https://explodingtopics.com/blog/chatgpt-users)reportedly reaching [900 million weekly active users (WAUs)](https://sensortower.com/report/state-of-ai-apps-2025/download) as of December 2025 \[8\] and generating over [5.6 billion monthly visits](https://explodingtopics.com/blog/chatgpt-users) \[9\], the scale of this shift is undeniable. Meanwhile, McKinsey predicts that unprepared brands could see [a 20% to 50% drop in traditional search engine traffic](https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/new-front-door-to-the-internet-winning-in-the-age-of-ai-search) as search marketing loses significant market share to AI chatbots and virtual agents \[1\]. ![MAUs.jpeg](https://surgeo.ai/m/4a3znm-m.webp) *ChatGPT, Google Gemini, and others (a16z) [13]* ![search-tranffic-forecasts-2.jpeg](https://surgeo.ai/m/852ylm-m.webp) *Traditional Search vs AI Search (SEMRush) [14]* The internet is becoming a place where machines, not humans, are the primary consumers of your content. Most businesses are using AI to write emails or summarize meetings. Very few are realizing that AI agents are becoming the new gatekeepers of commerce. If you are waiting for the dust to settle, you are already too late. Agents aren't the future of the internet; they are the current user. The Death of the Ten Blue Links ------------------------------- To understand how to survive this shift, we have to look under the hood of **Agentic Search**. Traditional search engines were like efficient librarians: you asked a question, and they pointed you to a shelf of books (links) where you might find the answer. You had to do the reading and synthesis yourself. AI Search Agents—like Perplexity, ChatGPT, and Gemini—are research assistants. They read the books for you, synthesize the information, and hand you a one-page summary. Technically, this is a marriage of three components: 1. **An LLM:** The brain that understands language and reasoning. 2. **Search Ranking:** The traditional index (like Google’s) that finds relevant documents. 3. **An Agentic System:** The planner that breaks a user’s complex request into sub-tasks (often using Retrieval Augmented Generation, or RAG). The reason this matters for your brand is a process known as **"Query Fan-out."** When a user asks a complex question like _"Plan a 3-day trip to Austin for a family of four with a $2,000 budget,"_ an agent doesn't just search for that phrase. Instead, it utilizes a "[retrieval–reasoning–refinement cycle](https://arxiv.org/abs/2401.06311)" \[3\]. The agent [decomposes the original query into dozens of sub-queries](https://www.kopp-online-marketing.com/from-query-refinement-to-query-fan-out-search-in-times-of-generative-ai-and-ai-agents)—hotel prices, kid-friendly restaurants, weather forecasts, and event calendars—and executes them simultaneously \[2\]. ![query-fan-out.jpeg](https://surgeo.ai/m/lhd9ks-m.webp) *Traditional keyword driven search vs. Query fan-out [15]* This architecture allows the AI to explore various facets and subtopics in parallel, [significantly expanding the information pool](https://www.kopp-online-marketing.com/from-query-refinement-to-query-fan-out-search-in-times-of-generative-ai-and-ai-agents) available for answer synthesis \[2\]. It mimics a news editor assigning a complex story to [multiple specialized reporters](https://www.youtube.com/watch?v=o8NiE3XMPrM&t=52m27s) at once \[15\]. In this process, your website is no longer a destination for a human user; it is a data source for a machine. If your content cannot be parsed during this millisecond-long "fan-out" phase, you are excluded from the final answer. From SEO to GEO (Generative Engine Optimization) ------------------------------------------------ This shift requires us to move from Search Engine Optimization (SEO) to **Generative Engine Optimization (GEO)**. In the old world, you optimized for keywords. If you put "Best Budget Headphones" in your H1 tag enough times, you ranked. In the agentic world, keywords matter less than **Brand Narratives** and **Corroboration**. The term "[Generative Engine Optimization](https://generative-engines.com/GEO/)" was formalized in a [landmark study](https://collaborate.princeton.edu/en/publications/geo-generative-engine-optimization) by researchers from Princeton University and IIT Delhi \[4, 10\]. Their research highlights a critical divergence from traditional SEO: while keyword density drove rankings in the past, it can actually be detrimental in the agentic era. The study found that traditional keyword stuffing could reduce visibility in generative responses by as much as 10%. Instead, the study identified specific strategies that significantly boost visibility in AI-generated answers: 1. **Cite Sources:** Adding credible citations from authoritative domains improved visibility. 2. **Statistics Addition:** Embedding quantitative data rather than qualitative descriptions. 3. **Quotation Addition:** Incorporating expert quotes to enhance authenticity. Implementing these GEO strategies was shown to [boost visibility by up to 40%](https://collaborate.princeton.edu/en/publications/geo-generative-engine-optimization/) in generative engine responses \[4\]. **This is the core of GEO:** 1. **Be Citable:** You need to be the source of truth that agents reference. The Princeton study confirms that "Statistics Addition" and "Quotation Addition" are top-performing methods. 2. **Be Mentionable:** You need authoritative third parties talking about you. The AI looks for patterns across the "vibes" of the internet—Reddit threads, G2 reviews, and forum posts. 3. **Be Machine-Readable:** Your data must be structured so a robot can parse it without getting a headache. The chart below would illustrate the overlap between SEO and GEO. While technical SEO (speed, crawlability) remains the foundation, the upper tier shifts from "Keyword Density" to "Semantic Authority." ![seo-to-geo.jpeg](https://surgeo.ai/m/rxdxqd-m.webp) *From SEO to GEO (Generative Engine Optimization)* How to Interview Your Brand’s Digital Twin ------------------------------------------ If benchmarks for AI models are hard to pin down—as we see with the constant fluctuation in LLM leaderboards—benchmarking your brand’s visibility in AI is even harder. But you have to try. You need to audit your **AI Visibility**. "AI Visibility" is an emerging metric that quantifies how AI search agents perceive, describe, and recommend a brand compared to its competitors. Unlike "Share of Search (SoS)," which measures user interest, AI Visibility measures AI preference. [Surgeo AI](https://surgeo.ai/) has launched dedicated tools to analyze this, helping brands identify whether they are optimizing their digital presence enough to [prompt recommendations from AI search agents](https://arxiv.org/abs/2506.18959) like ChatGPT, Perplexity, Gemini, and Google AI Model \[7\]. Start by identifying the questions your Ideal Customer Profile (ICP) asks. If you sell accounting software, don't just search for "best accounting software." Ask ChatGPT, Claude, and Perplexity: - _"I am a small business owner with 5 employees. Which accounting software will save me the most time?"_ - _"Compare \[Your Brand\] vs \[Competitor\] for a non-technical user."_ Analyze the output. - **Are you mentioned?** If not, you are invisible to the agentic economy. - **Are you cited?** Does the AI provide a link to your site as proof? - **What is the sentiment?** Is the AI recommending you, or just listing you? You will likely find that brands with strong "pre-ChatGPT" reputations dominate these results. This is because LLMs are trained on historical data. However, newer RAG systems pull in fresh web data. This is your opening. By creating high-value, highly structured content that answers specific questions—utilizing the "Statistics Addition" and "Quotation Addition" techniques proven by the [Princeton study](https://collaborate.princeton.edu/en/publications/geo-generative-engine-optimization) \[4\], [new](https://collaborate.princeton.edu/en/publications/geo-generative-engine-optimization/\)\]—new) brands can "inject" themselves into the AI's answer. The Technical Pivot: llms.txt and The Agentic Web ------------------------------------------------- There is also a boring, technical reality to this future. Agents are lazy browsers. They don't like heavy JavaScript, pop-ups, or complex visual layouts. If your content is rendered entirely on the client-side (meaning the browser has to build the page), an AI bot might just see a blank page. While Google has spent a decade learning to render JavaScript, the "token tax" of processing complex HTML boilerplate (navbars, footers, scripts) wastes context window space and money for AI agents. ![agentic-stack.jpeg](https://surgeo.ai/m/3waas5-m.webp) *The Agentic Stack* We are seeing the emergence of new standards like **llms.txt**. Think of [llms.txt](https://llmstxt.org) as a robots.txt file, but for content. It is a proposed standard—a Markdown file placed at the root of a website—that explicitly tells AI agents: _"Here is who we are, here are our most important documents, and here is a clean, text-only summary of our products"_ \[5\]. Unlike robots.txt, which tells bots what _not_ to index, llms.txt acts as a curated content concierge, guiding LLMs to the most [inference-friendly pages on a site](https://llmstxt.org) \[5\]. Implementing standards like this ensures your site provides a "clean signal" to the agent, making it significantly more likely that your content is consumed and cited during the query fan-out process. The Future: Agentic Commerce ---------------------------- Finally, we need to look at where this is going. We are moving from Agentic Search to **Agentic Commerce**. OpenAI, Stripe, and Shopify aren't just building better chatbots; they are building protocols that allow bots to buy things. The **Agentic Commerce Protocol (ACP)**, co-developed by [Stripe](https://stripe.com/blog/developing-an-open-standard-for-agentic-commerce) and [OpenAI](https://www.agenticcommerce.dev), creates a standardized language for how agents and businesses transact \[6, 11\]. In the near future, a user might say, _"Book me a flight to London under $600,"_ and the agent will go out, search, select, and pay. The ACP facilitates this by allowing agents to securely pass payment tokens between buyers and merchants without the agent ever storing raw credit card numbers. Crucially, the business remains the "[Merchant of Record](https://developers.openai.com/commerce/guides/get-started)," [retaining control over the customer relationship](https://stripe.com/blog/developing-an-open-standard-for-agentic-commerce) and order fulfillment \[6, 12\]. If your checkout process requires a human to solve a captcha or navigate a confused UI, you will lose that sale. The infrastructure is being laid for a world where the buyer is a bot, and the seller must be ready to speak its language. Conclusion ---------- The internet is not dying, but it is rebuilding itself. The "human-first" web of 10 blue links is being layered over by an "agent-first" web of answers and transactions. For the last decade, we optimized for the click. In the agentic future, we must optimize for the answer. This means building a brand that is technically accessible to machines via standards like llms.txt, but reputationally solid enough that those machines _want_ to recommend you based on GEO principles. The agents are already here. The only question is whether they can find you. References ---------- \[1\] Silliman, E., Boudet, J. and Robinson, K. (2025) New front door to the internet: Winning in the age of ai search, McKinsey & Company. Available at: [https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/new-front-door-to-the-internet-winning-in-the-age-of-ai-search](https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/new-front-door-to-the-internet-winning-in-the-age-of-ai-search) (Accessed: 19 December 2025). \[2\] Kopp, Olaf. “From Query Refinement to Query Fan-out: Search in Times of Generative AI and Ai Agents.” | Kopp Consulting, 1 Aug. 2025, [www.kopp-online-marketing.com/from-query-refinement-to-query-fan-out-search-in-times-of-generative-ai-and-ai-agents.](https://www.kopp-online-marketing.com/from-query-refinement-to-query-fan-out-search-in-times-of-generative-ai-and-ai-agents) \[3\] Zhang, Le, et al. “Exploring the Best Practices of Query Expansion with Large Language Models.” [arXiv.Org](http://arXiv.Org), 29 June 2024, [arxiv.org/abs/2401.06311](https://arxiv.org/abs/2401.06311). \[4\] “Geo: Generative Engine Optimization.” Princeton University, The Trustees of Princeton University, [collaborate.princeton.edu/en/publications/geo-generative-engine-optimization](https://collaborate.princeton.edu/en/publications/geo-generative-engine-optimization). Accessed 1 Dec. 2025. \[5\] Howard, Jeremy. “The /LLMS.TXT File – LLMS-Txt.” Llms, 3 Sept. 2024, llmstxt.org/. \[6\] Weinstein, Jeff, and Steve Kaliski. “Developing an Open Standard for Agentic Commerce.” Developing an Open Standard for Agentic Commerce, 29 Sept. 2025, [stripe.com/blog/developing-an-open-standard-for-agentic-commerce](https://stripe.com/blog/developing-an-open-standard-for-agentic-commerce). \[7\] Zhang, Weizhi, et al. “From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents.” [arXiv.Org](http://arXiv.Org), 3 July 2025, [arxiv.org/abs/2506.18959](https://arxiv.org/abs/2506.18959). \[8\] Sensor Tower - Market-Leading Digital Intelligence, sensortower.com/report/state-of-ai-apps-2025/download. Accessed 18 Dec. 2025. \[9\] Duarte, Fabio. “Number of CHATGPT Users (January 2026).” Exploding Topics, Exploding Topics, 14 Dec. 2025, [explodingtopics.com/blog/chatgpt-users](https://explodingtopics.com/blog/chatgpt-users). \[10\] Geo: Generative Engine Optimization, [generative-engines.com/GEO/](https://generative-engines.com/GEO/.). Accessed 1 Dec. 2025. \[11\] “Agentic Commerce Protocol.” Agentic Commerce Protocol, [www.agenticcommerce.dev/](https://www.agenticcommerce.dev/). Accessed 10 Dec. 2025. \[12\] “Agentic Commerce Protocol.” OpenAI Developers, [developers.openai.com/commerce/guides/get-started](https://developers.openai.com/commerce/guides/get-started). Accessed 10 Dec. 2025. \[13\] Olivia Moore, Justine Moore. “State of Consumer AI 2025: Product Hits, Misses, and What’s Next.” Andreessen Horowitz, 18 Dec. 2025, [a16z.com/state-of-consumer-ai-2025-product-hits-misses-and-whats-next](https://a16z.com/state-of-consumer-ai-2025-product-hits-misses-and-whats-next). \[14\] “We Studied the Impact of AI Search on SEO Traffic. Here’s What We Learned.” Semrush Blog, [www.semrush.com/blog/ai-search-seo-traffic-study](https://www.semrush.com/blog/ai-search-seo-traffic-study). Accessed 1 Dec. 2025. \[15\] YouTube, YouTube, [www.youtube.com/watch?v=o8NiE3XMPrM&t=52m27s](https://www.youtube.com/watch?v=o8NiE3XMPrM&t=52m27s). Accessed 12 Dec. 2025. #### About the Author ##### [Wishesh Sterling](https://surgeo.ai/blog/author/wishesh-sterling) Cofounder of Surgeo AI Follow With more than 10 years of experience building B2B SaaS companies, Wishesh sits at the intersection of traditional growth strategies and the future of AI. While he has a proven track record in organic search optimization, his primary focus is now on Agentic Search Engineering. He helps businesses navigate the shift from keyword-based SEO to semantic, agent-ready architectures, ensuring brands remain visible in an AI-first world. [](https://www.linkedin.com/in/automarketerio/)[](https://x.com/automarketerio) ### 0 Comments #### Join the conversation Sign in to share your thoughts and connect with other readers Sign In to Comment #### No comments yet Be the first to share your thoughts on this post!