Skip to main content

Ctrl+K

AI-related bots from robots.txt

AI-related bots in robots.txt of media sites

Acquisition

Obtain robots.txt of sites listed on mediabiasfactcheck
Obtain robots.txt of select sites from Wayback

Results

Process robots.txt files
Visualize AI related bots/user agents from collected robots.txt

Search

Ctrl+K

By penguinsfly

© Copyright MIT License, 2023 penguinsfly.