I Built My First Chrome Extension, Here's Everything I Learned

A few weeks ago, I had never built a Chrome extension. I didn't know what a manifest file was; I didn't know the difference between a background script and a content script, and I definitely didn't know that most modern browsers share the same engine under the hood.

Today I have a published extension on the Chrome Web Store.

This is the story of how I built it, what I learned, and why the concept behind it matters more than the code itself.

The Idea

I was learning about how AI companies crawl the web to train their models. GPTBot for OpenAI, ClaudeBot for Anthropic, and CCBot for Common Crawl all identify themselves in their HTTP request headers using something called a User-Agent string.

Every website has a file called robots.txt that tells these bots what they're allowed to crawl. But most people have never looked at their own robots.txt, let alone any other site's.

I wanted to build something that makes this information visible in one click. You visit any website, click the extension, and instantly see which AI bots are allowed or blocked.

That became AI Crawler Checker.

How Chrome Extensions Actually Work

Before writing any code, I had to understand the architecture. A Chrome extension is made of three separate parts that can't directly call each other; they communicate through Chrome's message passing system.

manifest.json is the config file. It tells Chrome what your extension is, what permissions it needs, and which files do what. Think of it as the business license.

popup.html and popup.js are what users see when they click the extension icon. It renders the UI.

background.js runs silently behind the scenes as a service worker. It handles data fetching and Chrome APIs.

The key thing I learned: these three parts talk through messages, not direct function calls.

// popup.js asks background.js for data chrome.runtime.sendMessage( { type: 'GET_ANALYSIS', hostname }, (response) => { renderResults(response) } ) // background.js listens and responds chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { if (message.type === 'GET_ANALYSIS') { analyseHostname(message.hostname).then(sendResponse) return true // keeps the channel open for async response } })

Parsing robots.txt

The core of the extension reads and parses robots.txt files. This file lives at the root of every website, medium.com, nytimes. and it's publicly accessible to anyone.

The format looks like this:

User-agent: GPTBot Disallow: / User-agent: Googlebot Allow: /

I wrote a parser that reads through the file line by line, groups rules by user-agent, and determines whether each known AI crawler is allowed, blocked, or partially restricted.

The interesting thing about robots.txt is that it's based on the honour system. There's no technical enforcement; it's just a convention. Bots can choose to ignore it. But most legitimate AI companies do follow it.

The Permissions Problem

When I first submitted the extension to the Chrome Web Store, Google flagged it for having broad host permissions. My original manifest included:

"host_permissions": ["<all_urls>"]

Google's concern was valid. This gives the extension permanent access to every URL on the internet, way more than it needs. The fix was switching to activeTab:

"permissions": ["activeTab", "scripting", "storage"]

This only grants access to the current tab, and only when the user clicks the extension icon. Same functionality, far less permission surface. The review went through much faster after this change.

The Chromium Discovery

Here's the thing that genuinely surprised me while building this.

I built it for Chrome. But it works on Edge, Brave, Arc, and Opera without changing a single line of code.

The reason is Chromium: Google's open-source browser engine that Chrome is built on. Most modern browsers are built on top of it. They all share the same extension API, which means building a Chrome extension means building for most of the web's browsers.

Firefox is the main exception. It uses its own engine, Gecko, and has slight API differences, but even then, porting a Chrome extension to Firefox is usually a small effort.

What I'd Do Differently

A few things I'd change if I built it again:

Start with the simplest permissions. I wasted time dealing with the review process because I grabbed more permissions than I needed upfront.

Write a test for the robots.txt parser. The parser is the most important piece of the extension, and I have no automated tests for it. Some edge cases in real robots.txt files caught me off guard.

Cache smarter. Right now, results are cached for one hour. Some robots.txt files rarely change. Others change frequently. A smarter cache would check the file's Last-Modified header.

The Result

AI Crawler Checker is live on the Chrome Web Store, free to install.

Visit any website, a blog, a news site, a product page, click the icon, and you'll see which AI bots are allowed in and which are locked out. Medium blocks all 13 crawlers I track. OpenAI's own website allows most of them, which is ironic.

Install it here: AI Crawler Checker on the Chrome Web Store

What's Next

The extension tells you what a site's robots.txt says.

But it can't tell you if those bots actually showed up.

That's a different problem, and it's what I built next.