It takes time and effort for bloggers and online publishers to post content on their websites. And when answer engines like ChatGPT and Perplexity use the content without sending the traffic, it hurts the earning potential immensely.
Therefore, to overcome this situation and help bloggers and publishers rightfully earn for what they create, Cloudflare is developing the Pay Per Crawl technology.
With the use of this new internet phenomenon, you could charge ChatGPT for crawling your website.
Yes, you can do that. Let’s see how.
Cloudflare’s Pay Per Crawl: What Is It and How It Works?
AI bots like GPTBot or ClaudeBot aren’t people that you can stop by appointing bouncers outside your home or private location. They are internet entities, so to block them, you need something online.
Our internet is functional due to the phenomenon of HTTP (Hypertext Transfer Protocol).
And HTTP is all about client-server communication that works on HTTP Request and HTTP Response.
So, in order to block access to AI bots, we need to work hand in hand with those two: Request and Response. And that’s what Cloudflare’s Pay Per Crawl is working on. It is utilizing the HTTP response codes 200, 402, and 403 to charge the AI crawlers.
How?
Let’s see.
Working of Cloudflare’s Pay Per Crawl
Most of the time, when someone accesses our platform, or we access theirs, the party that accesses becomes the client.
Hence, when AI bots access your platform, they become the client, and you are the server.
The first thing that Pay Per Crawl does is layer your website with the HTTP 402 Payment Required response code. This means that whenever an AI crawler sends a request to access data on your website, it will see the 402 Payment Required Response.
Now, if the crawler agrees to pay, it will receive an HTTP response code 200 once the payment is complete, indicating success and allowing it to access the content.
If the AI crawler arrives with payment already, it will receive an HTTP 200 OK response. It’s like showing your pass without being asked by the bouncers at the bar, and they nod for you to go inside.
However, if the AI crawler refuses to pay, it will receive an HTTP 403 response, which denies current access but remains open to future access if payment is made. You can say it’s one of those situations when bouncers tell you to go and get your pass, and then they will let you in.
This is the whole concept of Cloudflare’s Pay Per Crawl.
It’s the routing of a crawler request so that no single party gains all the benefits. It’s simply stopping the AI crawlers, asking for the payment, and once it’s done, they can access it. You can say Cloudflare’s Pay Per Crawl is like Stop, Pay, and Access.
Or, according to Cloudflare’s AI Audit dashboard, it’s Allow, Charge, and Block.
Will It Stop Google Bot?
No, the program is only to block AI crawlers, so that they can pay and access your data because they don’t send traffic. Whereas Google Bot and other search engines’ bots list your website in the search results and send traffic.
Therefore, once you enable Cloudflare’s Pay Per Crawl on your website, it won’t impact the regular traffic coming from various search engines.
It will continue to function as it was. So, worry not.
How Will the Payment Work?
There are multiple levels of technicalities involved here.
But if you want to understand it at a surface level, know that both the client (AI crawlers) and the server (you) need to configure payment details in their Cloudflare accounts.
Every successful HTTP 200 OK response code will be charged through a crawler-charged header (the standard method using Web Bot Auth proposals). Then Cloudflare will collect all the events, use that to charge the crawler, and distribute the payments to the publisher.
Let’s understand this through the following infographic:
This simply explains that for a crawler to access the content on a website layered with Pay Per Crawl, it must go through multiple stages, from generating an Ed25519 key pair to identify the server-side (you) crawler and access the paid resources.
However, you don’t need to worry about these, as they will be handled by the crawler and Cloudflare. You simply need to save your payment details.
That’s it.
How Can It Help Bloggers and Online Publishers?
Since the launch of Perplexity and search functionality in ChatGPT, the traffic for many online platforms has decreased.
The reason? These AI tools scrape the content from the publisher’s website and never send traffic.
As a result, there is a drop in publishers’ earnings only.
However, with the Pay Per Crawl initiative, publishers can earn money from AI tools while accessing their websites.
No longer do they have to persuade AI bots to send traffic back to their platforms or pages, where the AI bots retrieve the information. Instead, simply pay for what you access. If they don’t, no access granted.
Hence, there’s a possibility of an increase in traffic as the information will be available on the publisher’s website.
Second, earnings from the AI crawlers who agree to pay.
Therefore, if you’re a blogger or online publisher, with Cloudflare’s Pay Per Crawl, you can expect:
- Increased traffic
- Increased earning potential
No longer do you have to curse the AI crawlers for secretly accessing your website and never compensating you for what they take.
Should You Use Cloudflare’s Pay Per Crawl on Your Website?
This should not create a complicated situation. However, for many, it can become one.
It feels good that you can earn from AI crawlers while they access data on your website. However, how far it is good to use it depends totally on the size and authority of your website.
If you’re just getting started, there’s no point in blocking AI crawlers, as you can lose the opportunity to be mentioned in the AI results, for example, AIOs.
But if you’re someone like TechCrunch, it’s a clever move if you enable the Pay Per Crawl function on your website.
Look at the size and traffic of your platform and act according to that. If you’re new and have low traffic, don’t use it. If you’re receiving huge traffic, you should enable it. The earning potential could be high.
How to Enable Cloudflare’s Pay Per Crawl Option
To enable Cloudflare’s Pay Per Crawl option, you need to first avail yourself of its cloud network services. Then, visit the Cloudflare Pay Per Crawl’s early access private beta page.
There, click the “Request access” button.
It will direct you to the sign-up form. Simply enter your details and sign up for the early access.
As you submit your details, a confirmation email will be sent to your email address confirming your inclusion in the list. Shortly after, one of Cloudflare’s executives will reach out to provide more details regarding the pay-per-crawl feature.
What’s Ahead for Websites, SearchBots, and AIBots?
Cloudflare’s Pay Per Crawl is just the first light in the rise of the AI era to support creators.
With time, many platforms will launch similar programs to fairly compensate bloggers and online publishers for their work.
This reflects that more human content will be in demand. The more human-written content you publish on your website, the better you’ll earn because AI strives for raw. Something that’s not available anywhere.
Since it cannot go into the outside world, it will rely on what we provide until then.
Hence, what you publish will be the ultimate resource for it, and by applying technologies like Pay Per Crawl, you can turn your passion into a living.
This will force AI platforms like ChatGPT and Perplexity to make bold moves, such as showing ads in their free plans, because they have to generate revenue to balance this. Setting a budget based on subscription earnings for data access isn’t favourable.
Thus, they will start to show ads. It’s inevitable, look what Netflix is supporting these days, once an ad-free streaming platform, now vouching for them.
Another possibility is that AI platforms, too, offer their own payment procedures for allowing them access to your website. In this way, they don’t have to negotiate with a middleman like Cloudflare and save resources by directly paying you.
By looking at all these scenarios, one thing is clear: blogging and publishing online content will be the next high-paying job you could do full-time without being an employee in an organization.
So, if you’re thinking about writing online, this could be the right time to start.