Should You Block AI From Scraping Your Site?

Should You Block AI From Scraping Your Site?
Midjourney AI's blog had a post recently about whether or not you should let the artificial intelligence (AI) spiders crawl your content for large language models.

The author, Tom Capper, makes good points for both sides.

On allowing spiders to index your site side Tom points out three good points:

  1. Present-day traffic acquisition
  2. Present-day brand exposure
  3. Future developments in LLM-powered tools

I tend to agree with him on these. AI, like Claude and ChatGPT, are now citing their sources and I've notes more visitors coming from these AI systems because of these citations. Blocking the AI spiders from crawling and using your content will stop all traffic from coming to you from these systems.

With search going more machine learning and AI driven, this is the new way people are finding information. Why not get a piece of it?

On the flip side, Tom points out the following:

  1. Present-day content moats
  2. Stalling for time
  3. Future developments

I get these arguments, but most of them are negated if you're trying to drive traffic to your site. Yes OpenAI's ChatGPT, along with the others, are making money off your content. But it's not like they aren't in turn sending you traffic and sharing your content with a wider audience.

I'm all in on letting AI scrape my content. Will this opinion change in the future with new developments? Maybe. But as of now, I'll happily let them scrape my content.

What do you think? Let me know!

The Latest From The Entrepreneur's Enigma Podcast

Web Jawn

That's It For This Week! Keep Rocking It!


QR Code takes you to a landing page where you can follow Seth online.
Connect with me online: