Meta removes disputed crawling language from its commercial terms
Tech

Meta removes disputed crawling language from its commercial terms

TechNews Editorial
TechNews EditorialOct 17, 2026 · 1 min read
Share

Why it matters

The update addresses concerns that Meta’s terms could override website owners’ robots.txt preferences, which Meta says remain in place.

The facts

  • Meta removed disputed crawling language from its commercial terms after criticism.
  • Critics said the wording could allow free AI training on scraped content and override robots.txt instructions.
  • Meta says the terms applied only to ads and business products and did not change its web crawling practices.

Advertisement

Meta has removed language from its commercial terms that critics said could let it use content from websites, including news sites, to train its AI. The update follows criticism of wording Meta had recently added to the terms.

The removed passage said Meta could use content from specified domains, web pages or URLs to develop, improve and train products covered by its commercial terms. It no longer appears on the commercial terms page.

Neiman Lab and James Grimmelmann, a professor of digital and information law at Cornell, were among those who read the wording as an attempt to claim the right to train AI on scraped content for free. They also raised concerns that it could override blocking instructions in a site’s robots.txt file.

Meta denied changing how its web crawlers work

Meta disputed that interpretation. Andy Stone, its vice president of communications, and spokesperson Daniel Roberts said, “There is no change to how Meta uses web crawlers; any suggestion to the contrary is false.”

A reporter extends an audio recorder toward a company representative, listening intently as he explains the scope of disputed crawling terms.
Illustration: AI & Tech News

Roberts later told The Verge that the consolidated crawling terms applied only to Meta’s ads and business products. “We’ve heard you and want to clarify: the crawling terms we consolidated only applied to our ads and business products, and we’ve clarified that in the terms themselves. It does not change how Meta uses web crawlers. As always, content owners can use robots.txt to signal their preferences. This is not about scraping news sites, and these updates don’t give us new rights that supersede those preferences.”

The dispute centered on what rights the wording appeared to give Meta, rather than a reported change to its crawling practices. Under the clarified terms, content owners can use robots.txt to signal their preferences, Roberts said.

Newsletter

Get the best AI & tech news daily

A concise daily digest. Unsubscribe anytime.

We use your email only to send this newsletter.

Keep reading