Protecting your content from Ai crawlers is an option that is available to all content owners. But, some content owners may want to promote their content to Ai crawlers (like Google’s AIO – Ai Overviews).
What are Google Ai Overviews?
Google AIO is a new feature that essentially provides an overview of your content to search users. It is featured more prevalently on mobile browsers than desktop, and until recently was only offered to users who were logged-in to Google.
Not all of the crawlers are looking to place your content in front of users, some of them are hoarding it in large databases to be sold to the highest bidder, using it to train their own Ai products, and even creating new and exciting products based on your ideas.
Why should I promote my content to Ai?
If you are looking to expand your user base and stay relevant in search, then Ai is the way of the future. Google is still the king of search at the time of this writing and they doesn’t look to be changing course anytime soon (regardless of the lawsuits).
Why should I protect my content from Ai?
Just like anything you put your time and effort into creating, your content is worth protecting. Ai scrappers are running 24/7 looking for ideas/data to incorporate/steal for their large scale databases and training models.
From text and ideas to original art and photography, Ai is feeding non-stop.
How do I protect my content from Ai?
There are several steps you can take to block crawlers from accessing your content. Take note, this is largely done on the honor system.
Step 1 : Edit your robots.txt file to disallow user agents like ChatGPT-User, GoogleOther, OmigiliBot, etc. This is flexible as you can block some bots and others though however, you need to know the actual name of the bot. Here is a list of some known Ai bots.
Step 2 : Add a plugin to your website like Block Ai Crawlers.
Step 3 : IP Blocking, this is an ongoing commitment, you need to watch your logs and block new bots as they appear. It is time consuming but worth the extra effort. Set aside an hour a week to do this task.