Should You Block AI From Scraping Your Site?
Moz.com's blog had a post recently about whether or not you should let the artificial intelligence (AI) spiders crawl your content for large language models.
The author, Tom Capper, makes good points for both sides.
On allowing spiders to index your site side Tom points out three good points:
- Present-day traffic acquisition
- Present-day brand exposure
- Future developments in LLM-powered tools
I tend to agree with him on these. AI, like Claude and ChatGPT, are now citing their sources and I've notes more visitors coming from these AI systems because of these citations. Blocking the AI spiders from crawling and using your content will stop all traffic from coming to you from these systems.
With search going more machine learning and AI driven, this is the new way people are finding information. Why not get a piece of it?
On the flip side, Tom points out the following:
- Present-day content moats
- Stalling for time
- Future developments
I get these arguments, but most of them are negated if you're trying to drive traffic to your site. Yes OpenAI's ChatGPT, along with the others, are making money off your content. But it's not like they aren't in turn sending you traffic and sharing your content with a wider audience.
I'm all in on letting AI scrape my content. Will this opinion change in the future with new developments? Maybe. But as of now, I'll happily let them scrape my content.
What do you think? Let me know!
The Latest From The Entrepreneur's Enigma Podcast
Web Jawn
- How to Write AI Content Optimized for E-E-A-T
There is nothing wrong with using AI for content generation. Just make it so it's in your voice and isn't hallucinating, which it can. - US Senate Approves TikTok Sell-off Proposal
It's happening, TikTok needs to sell it's US-based business or risk being banned. - X Continues To Display Ads Alongside Harmful Content
Is Elon & Co. even trying anymore? - Meta Posts Solid Growth in Q1, Highlights Future Investment in AI
Meta is growing and had a good Q1 of 2024. It's clear that AI is playing a big part in this. - Reddit Rolls Out Mobile UI Updates To Streamline Interaction
Anything that fixes their đź’© app is and improvement. - How the TikTok ban could survive a court challenge
Good analysis from Casey Newton over at Platformer.News - Adobe’s new Firefly model makes it easier to use Photoshop’s AI tools
I've used it and it's really handy and works reasonably well. - Google March Core Update Still Rolling Out & Heated SEO Chatter Continue
End of April and we're still seeing violitity from March's Core Update. - Google Further Postpones Third-Party Cookie Deprecation In Chrome
Not surprised. But it'll probably happen at some point. Sooner rather than later. - Threads hits 150 million active users, adds option to archive posts
Threads is coming into its own. The archiving posts feature could be handy. - Apple's iPhones are so good and last so long people aren't buying new ones as activations slump
I just got the iPhone 15 Pro and I can see why people don't feel the need to upgrade. - How to start a video podcast in 2024
Great primer for beginners and a refresher for more advanced users on the power of video in podcasting in 2024. - F.C.C. Votes To Reinstate Net Neutrality Rules
This is great news and very much needed. - Mastodon Forms A U.S. non-profit
Apparently the German government revoked their non-profit charter, they formed a U.S. non-profit. Some are concerned with the new board of directors, which includes former Twitter executive Biz Stone. I'm not. It's open source and free. People like to worry about stuff like it's a freaking sport.
That's It For This Week! Keep Rocking It!
-Seth
