Meta scrapped our servers but we were not the only ones

This article was initially published on my LinkedIn on May 14th, but got updated with new discoveries I made on the topic in July.

For years, the internet operated on a relatively simple deal: Search engines crawled websites. Websites received traffic in return. It was never perfectly balanced, but at least the incentives mostly aligned. Google indexed your pages, users discovered your platform, and some percentage of that traffic turned into customers. The web remained economically circular.

AI changes that equation entirely.

Over the past few weeks, our lawyer review platform, TheLawyerGuide.com (or Advokatguiden, as its called in Norway) started becoming unstable. Pages slowed down. Requests piled up. Services restarted unexpectedly. Users experienced failed searches and spinning loading screens.

At first, we assumed the problem was internal. Was it bad scaling decisions? Infrastructure bottlenecks? Misconfigured caching? Or just lazy programmers?

That’s the standard founder instinct, right — to assume it is your own fault first. But then we checked the logs.

Between April 16 and April 23, a single crawler identifying itself as Meta’s “meta-externalagent” accounted for 4,819 requests in a sampled 10,000-request window. 48.2% of all sampled traffic. Not half of bot traffic — half of all observed requests.

For comparison:

• Googlebot generated 129 requests

• Bingbot generated 277 requests

• Meta’s crawler generated roughly 37x more requests than Googlebot

The crawler identified itself using Meta’s published crawler signature:

meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

The requests originated from IP ranges allocated to Meta Platforms under AS32934, including dozens of IPs inside the 57.141.20.0/24 range:

• 57.141.20.0

• 57.141.20.14

• 57.141.20.30

• 57.141.20.45

• 57.141.20.60

• 57.141.20.68

• 57.141.20.70

In total, we observed approximately 60 distinct Meta-associated source IPs participating in the crawl activity.

And this was not normal indexing behavior: The crawler aggressively traversed deep API endpoints, generated URL permutations that did not exist in our sitemap, and hammered pagination layers extending hundreds of pages beyond meaningful human navigation patterns.

Some examples from the logs:

/api/profiles?page=312/

api/profiles?page=204

dynamically generated lawyer/location combinations not linked anywhere on the site.

Single IPs fetched multiple API responses between 50 KB and 150 KB within the same second. When we blocked one IP, Meta aggressively automatically shifted to a new IP and continued the scraping.

This lines up with what others are starting to see across the web. Recent data shared by founders tracking AI crawlers shows Meta generating significantly more training traffic than OpenAI or Anthropic combined, in some cases accounting for the majority of all observed AI-related crawl activity.

On one dataset of high-traffic SaaS websites, Meta’s crawler reportedly represented over 70% of total training crawl volume, using the same meta-externalagent signature we observed.

Whether the exact ratios hold everywhere is almost beside the point, the direction is clear. This is not occasional indexing.

This is sustained, high-volume data ingestion happening across hundreds of sites simultaneously.

For reference, The Lawyer Guide is not a static blog. It is a structured legal intelligence platform containing lawyer profiles, legal categories, location mapping, reviews, metadata, reputation signals and localized legal information across multiple geographies

This type of structured data is expensive to build. And increasingly, extremely valuable to AI systems.

Over less than seven days, our frontend services restarted involuntarily 45 times. Cumulative downtime for real users is estimated at 30 to 45 minutes, not including periods of degraded performance. Human visitors ultimately paid the price while Meta’s infrastructure absorbed the data.

That is when the bigger realization hit us: The AI economy may be creating one of the largest value transfers in the history of the internet.

Smaller companies spend years building high-quality structured datasets:

• moderating them

• verifying them

• updating them

• hosting them

• paying bandwidth costs

• maintaining uptime

• fighting spam

• improving search

• building trust systems

Then hyperscalers arrive with distributed crawlers, ingest the informational value at internet scale, and monetize AI systems on top of it.

We effectively pay twice: First to create the data. Then to serve it to the systems extracting value from it.

The public AI discussion mostly focuses on models, GPUs, valuations, and consumer apps. But underneath the AI boom sits an enormous extraction layer crawling the open web at industrial scale. And the incentives are changing fast.

Traditional search engines created traffic loops: Google indexed our lawyer profiles and usually sent users back to our website.

AI systems create extraction loops. If an AI assistant internalizes the information itself, the user may never visit the source platform at all. The infrastructure costs remain decentralized. The monetization becomes centralized.

To be clear, Meta is not the only company competing aggressively for data. Every major AI company is doing some version of this. But Meta occupies a uniquely interesting position.

Mark Zuckerberg is under enormous pressure to catch up in the AI race: OpenAI owns the consumer mindshare. Google owns search distribution. Anthropic owns credibility among technical users. And Perplexity is redefining AI-native discovery.

Meta has world-class infrastructure, world-class researchers, and billions of users across Facebook, Instagram, and WhatsApp. But what Meta does not yet appear to have is a dominant consumer-facing AI product people rely on daily in the same way they use ChatGPT, Claude, or Gemini.

And from where we are standing, the behavior we observed only makes sense in that context.

Because why would a social media company suddenly need large-scale structured legal data across multiple countries and geographies? Why aggressively crawl lawyer directories deep into dynamically generated endpoints and long-tail permutations?

Our guess is simple: Meta is “zergrushing” the open internet. Not indexing it. But absorbing it.

The company appears to be in a massive catch-up phase, “blitzkrieging” structured websites across the web to feed centralized AI training systems powering Llama and whatever consumer AI layer comes next.

Mark our words: Meta is releasing a “ChatGPT-killer” within 6–12 months.

From a strategic perspective, it makes sense. The companies with the best data will likely build the strongest consumer AI products. The problem is who pays the price during that transition. Because right now, the casualties are increasingly the platforms actually producing the underlying informational value. Forums, publisers, directories, review platforms, open-source communities and niche startups. Combined, they make up the internet’s middle layer.

The irony is difficult to ignore: For years, startups building structured data platforms were heavily scrutinized for aggregating public information into better consumer experiences. But when trillion-dollar AI companies ingest the open web at industrial scale, it suddenly becomes innovation.

The technical side of this problem can eventually be solved, such as introducing more aggressive rate limiting, smarter bot detection and distributed infrastructure.

But the larger question is economic: If AI companies can crawl the web at massive scale, internalize the informational value, and monetize AI systems on top of it, who actually profits from the AI economy?

Not always the people creating the data. Not always the people paying the infrastructure bill. Not always the people building the underlying value in the first place.

The old internet bargain was simple: publish openly, receive traffic in return. AI may be quietly breaking that bargain.

Smaller platforms like ours are starting to notice. Our biggest infrastructure challenge this year did not come from users. It came from machines trying to learn from them.

We need to fix the AI economy.

Or just get better at blocking Meta.