I Built CrawlGuard — A Real-Time AI Crawler Monitor for Your Website
Jimoh Sherifdeen / August 29, 2026
6 min read
After building the AI Crawler Checker extension, I had a follow-up question.
The extension tells you what a site's robots.txt says, which bots are allowed or blocked in theory. But it can't answer the more interesting question: did they actually come?
robots.txt is the honour system. Bots can ignore it. And even when they follow it, you'd never know they visited unless you were watching your server logs in real time.
That gap is what CrawlGuard fills.
What CrawlGuard Does
You sign up, add your website, and get a unique script tag:
<script src="https://crawlguard.com/tracker.js?siteId=abc123" async></script>
Paste that into your site's <head>. Every time an AI crawler visits - GPTBot, ClaudeBot, CCBot, Bytespider, PerplexityBot - it gets logged. You see which bot, which page they crawled, and exactly when.
No server configuration. No complex setup. One line of code.
How the Tracking Actually Works
When a bot visits your site, the script tag sends a POST request to CrawlGuard's API:
javascriptfetch('https://crawlguard.com/api/track', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ scriptId: siteId, pageUrl: window.location.href }),
})
The API reads the User-Agent header from the request and checks it against a list of known AI crawlers:
typescriptconst AI_CRAWLERS = [
{ pattern: /GPTBot/i, name: 'GPTBot', company: 'OpenAI' },
{ pattern: /ClaudeBot/i, name: 'ClaudeBot', company: 'Anthropic' },
{ pattern: /CCBot/i, name: 'CCBot', company: 'Common Crawl' },
{ pattern: /PerplexityBot/i, name: 'PerplexityBot', company: 'Perplexity' },
{ pattern: /Bytespider/i, name: 'Bytespider', company: 'ByteDance' },
// ... and more
]
If the user-agent matches a known bot, the visit gets logged to the database. If it's a regular browser, it's silently ignored, no data stored, no overhead.
The Stack
Next.js + TypeScript for the frontend and API routes. The tracker endpoint lives at /api/track as a Next.js route handler.
Supabase for the database and authentication. Supabase runs on PostgreSQL, and it's where I learned something that genuinely changed how I think about security.
Vercel for deployment. The Next.js app lives here.
Resend for transactional email. Magic link authentication, no passwords.
The Thing I Learned About Database Security
Most backend developers, myself included, enforce data access in application code:
php// In your controller
$sites = Site::where('user_id', auth()->id())->get();
This works. But the security only exists in your application layer. Miss that filter in one endpoint and users start seeing each other's data. No warning. No error. A silent leak.
Supabase uses PostgreSQL's Row Level Security, a feature that puts access rules directly in the database itself:
CREATE POLICY "Users can view own sites"
ON sites FOR SELECT
USING (auth.uid() = user_id);
Now it doesn't matter what your application code does. The database enforces the rule on every single query. A junior dev can't accidentally bypass it. A rushed PR can't bypass it. The database refuses to return rows that don't belong to you.
The auth.uid() function reads the user's ID from their JWT token, which Supabase attaches to every database request automatically. It's security at the lowest possible layer.
The Public Tracking Endpoint Problem
One issue I ran into: the tracking API needs to be publicly accessible; it's called by a script on someone else's website. But Supabase's Row Level Security blocked it because no authenticated user made the request.
The fix was using Supabase's service role key for the tracking endpoint specifically. The service role bypasses RLS entirely, which is fine here because I'm doing my own validation, checking that the scriptId exists in the database before logging anything.
typescript// Use service role key for public endpoints only
const supabase = createClient(
process.env.NEXT_PUBLIC_SUPABASE_URL!,
process.env.SUPABASE_SERVICE_ROLE_KEY!
)
For every other endpoint, dashboard, and site management, the regular authenticated client with RLS stays in place.
Authentication Without Passwords
CrawlGuard uses magic link authentication. You enter your email, receive a link, click it, and you're in. No password to remember, no password to leak.
Supabase handles this out of the box. The tricky part was deployment; Supabase needs to know which URLs are valid redirect targets for the magic link. In development, it's localhost:3000. In production, it's your Vercel URL. You have to whitelist both in the Supabase dashboard, or the links redirect to the wrong place.
For email delivery, I used Resend, which required setting up DNS records to verify domain ownership, TXT records for DKIM signing, and MX records for bounce handling. Not something I'd done before. Worth knowing for any project that sends transactional email.
What I'd Improve
Persistence on the tracker. Right now, if CrawlGuard's server is down, visits are lost. A queue, even a simple one, would help.
Alerts. If GPTBot crawls your site for the first time, you probably want to know immediately. Email or webhook notifications are the obvious next step.
Historical charts. Right now the dashboard shows a raw table. A simple chart showing crawler activity over time would make patterns much clearer.
Bot verification. User-agent strings can be spoofed. A determined actor could send fake GPTBot visits to inflate your numbers. Verifying the request's IP against known bot IP ranges would make the data more trustworthy.
The Result
CrawlGuard is live and free to use during beta.
Add your site, get your script tag, and within days you'll know exactly which AI companies are crawling your content, how often, and which pages they care about most.
Try it here: crawlguard-gamma.vercel.app
The Bigger Picture
Both tools came out of the same learning challenge: one concept a day, one thing shipped.
The Chrome extension taught me how browsers work at the extension layer. CrawlGuard taught me database security, DNS, transactional email, and the difference between theory and reality when it comes to AI bot behaviour.
The AI scraping conversation is one of the most important ones happening in tech right now. Knowing which bots are visiting your site, and what your robots.txt actually lets them do, feels like basic information every site owner should have access to.
Both tools are free. Build something with what you learn.