SurfSense/.cursor/skills/technical-seo-checker/references/robots-txt-reference.md

# Robots.txt Reference Guide

Complete reference for creating, testing, and troubleshooting robots.txt files.

## Syntax Guide

### Basic Structure

```
User-agent: [bot name]
Disallow: [path to block]
Allow: [path to allow]
Sitemap: [sitemap URL]
Crawl-delay: [seconds]
```

---

## Core Directives

### User-agent

Specifies which bot the rules apply to.

**Syntax**: `User-agent: [bot-name]`

**Common user-agents**:
```
User-agent: *                    # All bots
User-agent: Googlebot            # Google's crawler
User-agent: Bingbot              # Bing's crawler
User-agent: GPTBot               # OpenAI's crawler
User-agent: CCBot                # Common Crawl bot
User-agent: anthropic-ai         # Anthropic's crawler
User-agent: PerplexityBot        # Perplexity AI crawler
User-agent: ClaudeBot            # Claude's web crawler
```

**Multiple user-agents**: Group rules by leaving no blank lines between user-agent declarations.

```
User-agent: Googlebot
User-agent: Bingbot
Disallow: /admin/
```

---

### Disallow

Blocks bots from crawling specified paths.

**Syntax**: `Disallow: [path]`

**Examples**:
```
Disallow: /                      # Block entire site
Disallow: /admin/                # Block admin directory
Disallow: /private               # Block private directory (and subdirectories)
Disallow: /*.pdf$                # Block all PDF files
Disallow: /*?                    # Block all URLs with parameters
Disallow:                        # Allow everything (empty disallow)
```

**Path matching**:
- `/` at end = block directory and all subdirectories
- Without `/` at end = block all paths starting with string
- `*` = wildcard, matches any sequence
- `$` = end of URL

---

### Allow

Explicitly allows crawling (overrides Disallow).

**Syntax**: `Allow: [path]`

**Common use**: Allow specific subdirectories within blocked parent.

```
User-agent: *
Disallow: /admin/
Allow: /admin/public/
```

**Note**: Allow is not standard but supported by Google, Bing, and most major crawlers.

---

### Sitemap

Specifies location of XML sitemap.

**Syntax**: `Sitemap: [absolute URL]`

**Examples**:
```
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap_index.xml
Sitemap: https://example.com/blog/sitemap.xml
```

**Best practices**:
- Use absolute URLs (not relative)
- Can include multiple Sitemap directives
- Place at end of file
- Submit same sitemap(s) to Google Search Console

---

### Crawl-delay

Adds delay between requests (seconds).

**Syntax**: `Crawl-delay: [seconds]`

**Example**:
```
User-agent: *
Crawl-delay: 10
```

**Warning**: Not supported by Googlebot (use Search Console rate limiting instead). Supported by Bing, Yandex, and others.

---

## Common Configurations

### 1. Allow All Bots (Default)

```
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml
```

Use when you want all bots to crawl entire site.

---

### 2. Block All Bots

```
User-agent: *
Disallow: /
```

Use for development/staging sites or private content.

---

### 3. Block Specific Directories

```
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /temp/
Disallow: /cgi-bin/

Sitemap: https://example.com/sitemap.xml
```

Standard configuration blocking admin and utility directories.

---

### 4. Block All AI Crawlers

```
# Block OpenAI
User-agent: GPTBot
Disallow: /

# Block Anthropic
User-agent: anthropic-ai
User-agent: ClaudeBot
Disallow: /

# Block Common Crawl
User-agent: CCBot
Disallow: /

# Block Perplexity
User-agent: PerplexityBot
Disallow: /

# Block Google-Extended (Bard training)
User-agent: Google-Extended
Disallow: /

# Allow search engines
User-agent: Googlebot
Disallow:

User-agent: Bingbot
Disallow:

Sitemap: https://example.com/sitemap.xml
```

Use when you want search indexing but not AI training.

---

### 5. Allow Search Engines, Block Everything Else

```
# Block all by default
User-agent: *
Disallow: /

# Allow Google
User-agent: Googlebot
Disallow:

# Allow Bing
User-agent: Bingbot
Disallow:

# Allow DuckDuckGo
User-agent: DuckDuckBot
Disallow:

Sitemap: https://example.com/sitemap.xml
```

---

### 6. Block URL Parameters

```
User-agent: *
Disallow: /*?                    # Block all URLs with parameters
Allow: /?                        # Allow homepage with parameters

Sitemap: https://example.com/sitemap.xml
```

Prevents duplicate content from parameter variations.

---

### 7. Block File Types

```
User-agent: *
Disallow: /*.pdf$
Disallow: /*.doc$
Disallow: /*.xls$
Disallow: /*.zip$

Sitemap: https://example.com/sitemap.xml
```

---

### 8. E-commerce Configuration

```
User-agent: *
# Block search/filter pages
Disallow: /*?q=
Disallow: /*?sort=
Disallow: /*?filter=

# Block account pages
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

# Block admin
Disallow: /admin/

# Allow product pages
Allow: /products/

Sitemap: https://example.com/sitemap.xml
```

---

### 9. WordPress Configuration

```
User-agent: *
# WordPress core
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# WordPress directories
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/

# Allow uploads
Allow: /wp-content/uploads/

# Block parameter pages
Disallow: /?s=
Disallow: /feed/
Disallow: /trackback/

Sitemap: https://example.com/sitemap_index.xml
```

---

### 10. Shopify Configuration

```
User-agent: *
# Block admin and account
Disallow: /admin
Disallow: /account
Disallow: /cart
Disallow: /checkout

# Block search
Disallow: /search

# Block collections with filters
Disallow: /collections/*+*
Disallow: /collections/*?*

Sitemap: https://example.com/sitemap.xml
```

---

## Platform-Specific Templates

### Wix

```
User-agent: *
Disallow: /_api/
Disallow: /_partials/

Sitemap: https://example.com/sitemap.xml
```

### Squarespace

```
User-agent: *
Disallow: /config/
Disallow: /search

Sitemap: https://example.com/sitemap.xml
```

### Webflow

```
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
```

### Drupal

```
User-agent: *
Disallow: /admin/
Disallow: /user/
Disallow: /node/add/
Disallow: /?q=

Sitemap: https://example.com/sitemap.xml
```

---

## Testing and Validation

### Google Search Console Robots.txt Tester

1. Go to: Search Console → Settings → robots.txt
2. View current robots.txt
3. Test specific URLs
4. See which user-agents are affected

### Manual Testing

Test URL pattern: `https://example.com/robots.txt`

Check file is:
- Accessible (returns 200 status)
- Plain text format
- UTF-8 encoded
- Located at root domain
- No more than 500KB (Google limit)

### Common Testing Scenarios

Test these URLs in tester:
- Homepage: `/`
- Product page: `/products/example`
- Admin page: `/admin/`
- Parameter page: `/search?q=test`
- File: `/document.pdf`

---

## Common Mistakes and Fixes

### Mistake 1: Blocking CSS/JS Files

**Wrong**:
```
User-agent: *
Disallow: /css/
Disallow: /js/
```

**Why it's wrong**: Google needs CSS/JS to render pages properly.

**Fix**:
```
User-agent: *
Allow: /css/
Allow: /js/
```

---

### Mistake 2: Using Relative URLs for Sitemap

**Wrong**:
```
Sitemap: /sitemap.xml
```

**Fix**:
```
Sitemap: https://example.com/sitemap.xml
```

---

### Mistake 3: Spaces in Directives

**Wrong**:
```
User-agent : Googlebot
Disallow : /admin/
```

**Fix** (no spaces before colons):
```
User-agent: Googlebot
Disallow: /admin/
```

---

### Mistake 4: Forgetting Trailing Slash

**Intention**: Block /admin directory

**Wrong**:
```
Disallow: /admin
```

**Result**: Also blocks /admin-panel, /administrator, etc.

**Fix**:
```
Disallow: /admin/
```

---

### Mistake 5: Blocking Entire Site Accidentally

**Wrong**:
```
User-agent: *
Disallow: /
Allow: /blog/
```

**Why it's wrong**: Many bots don't support Allow directive.

**Fix**: Use noindex meta tags for pages you don't want indexed, not robots.txt.

---

### Mistake 6: Not Blocking Development Environments

**Wrong**: No robots.txt on staging.example.com

**Result**: Staging site gets indexed.

**Fix**:
```
User-agent: *
Disallow: /
```

On all non-production environments.

---

### Mistake 7: Case Sensitivity Errors

**Note**: Directives are case-insensitive, but paths are case-sensitive.

**Example**:
```
Disallow: /Admin/        # Blocks /Admin/ but not /admin/
```

**Fix**: Block both if needed:
```
Disallow: /admin/
Disallow: /Admin/
```

---

## Advanced Patterns

### Wildcard Examples

```
# Block all PDFs
Disallow: /*.pdf$

# Block all URLs with parameters
Disallow: /*?

# Block all URLs ending in .php
Disallow: /*.php$

# Block all admin paths regardless of location
Disallow: /*/admin/
```

### Multiple Sitemaps

```
Sitemap: https://example.com/sitemap-pages.xml
Sitemap: https://example.com/sitemap-posts.xml
Sitemap: https://example.com/sitemap-products.xml
```

### Bot-Specific Rules

```
# Aggressive bot - slow it down
User-agent: BadBot
Crawl-delay: 60
Disallow: /

# Good bots - full access
User-agent: Googlebot
User-agent: Bingbot
Disallow:

# Default for others
User-agent: *
Crawl-delay: 10
Disallow: /admin/
```

---

## Robots.txt vs Meta Robots vs X-Robots-Tag

### When to use each:

**Robots.txt**:
- Block crawling of entire directories
- Reduce crawl budget waste
- Block parameter variations
- Does NOT prevent indexing if page is linked from elsewhere

**Meta robots tag**:
- Prevent specific pages from being indexed
- Control snippet display
- Control following links
- Example: `<meta name="robots" content="noindex,follow">`

**X-Robots-Tag HTTP header**:
- Control non-HTML files (PDFs, images)
- Server-level control
- Example: `X-Robots-Tag: noindex`

**Important**: If you don't want a page indexed, use noindex (meta tag or header), NOT robots.txt.

---

## Monitoring and Maintenance

### Regular Checks

**Monthly**:
- [ ] Verify robots.txt is accessible
- [ ] Check Search Console for blocked URLs
- [ ] Review crawl stats for blocked resources

**Quarterly**:
- [ ] Audit blocked paths - still relevant?
- [ ] Check for new admin/private sections to block
- [ ] Review AI crawler landscape (new bots?)

**After site changes**:
- [ ] Update robots.txt if URL structure changed
- [ ] Test new sections (should they be blocked?)
- [ ] Verify sitemaps still referenced

### Search Console Monitoring

Check these reports:
- **Coverage** → Excluded by robots.txt
- **Settings** → Crawl stats
- **URL Inspection** → Test specific URLs

---

## Robots.txt Checklist

Before deploying:

- [ ] File is named exactly `robots.txt` (lowercase)
- [ ] Located at root domain (`example.com/robots.txt`)
- [ ] Plain text format (not HTML or PDF)
- [ ] UTF-8 encoding
- [ ] No HTML tags in file
- [ ] All paths start with `/`
- [ ] Sitemap URLs are absolute
- [ ] No spaces before colons
- [ ] Tested in Search Console robots.txt tester
- [ ] Not blocking important CSS/JS/images
- [ ] Not blocking content you want indexed
- [ ] Trailing slashes used correctly for directories
- [ ] Wildcard patterns tested
- [ ] File size under 500KB

---

## Emergency Fixes

### Accidentally Blocked Entire Site

**Symptom**: All pages blocked in Search Console

**Fix**:
1. Edit robots.txt to:
```
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml
```
2. Test in Search Console
3. Request urgent recrawl for key pages
4. Monitor Coverage report for recovery

**Recovery time**: 1-7 days

---

### Blocked CSS/JS Files

**Symptom**: "Blocked by robots.txt" in Mobile-Friendly Test

**Fix**:
1. Add Allow directives:
```
User-agent: *
Allow: /css/
Allow: /js/
Allow: /wp-content/uploads/
```
2. Test in robots.txt tester
3. Request re-render in URL Inspection tool

---

### Staging Site Indexed

**Symptom**: staging.example.com appears in search results

**Fix**:
1. Add to staging robots.txt:
```
User-agent: *
Disallow: /
```
2. Add noindex meta tag to all staging pages
3. Remove staging URLs in Search Console (Removals tool)

---

## Resources and Tools

**Testing**:
- Google Search Console robots.txt tester
- Bing Webmaster Tools robots.txt analyzer
- Technical SEO browser extensions

**Validation**:
- https://www.google.com/webmasters/tools/robots-testing-tool
- https://en.ryte.com/free-tools/robots-txt/
- https://technicalseo.com/tools/robots-txt/

**Documentation**:
- Google: https://developers.google.com/search/docs/crawling-indexing/robots/intro
- Bing: https://www.bing.com/webmasters/help/robots-txt-validation
- Robots.txt spec: https://www.robotstxt.org/