OpenAI says it will honor a shared signal in robots.txt that lets website operators express preferences about using their content for foundational generative AI training. The signal is meant to let operators communicate the same choice to multiple participating AI providers at once.
Website operators previously had to identify individual crawlers and maintain a separate rule for each provider, according to OpenAI. The company says a shared signal should reduce that work.
The signal is voluntary. OpenAI has announced its intent to support it, but has not implemented it yet. The company says it aims to do so in the coming months.
Existing controls will continue to apply
OpenAI says the shared signal will not replace its existing robots.txt crawler controls. Those controls will continue to apply, giving publishers a way to set preferences for their pages.
The announcement draws on work by the Internet Engineering Task Force’s AI Preferences working group. OpenAI joined other participants in announcing the signal and encouraged other AI providers to honor it.
OpenAI says it will share implementation details in another update when it puts the signal in place in the coming months.
