Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/schlessera/agentic-seo-stack


https://github.com/schlessera/agentic-seo-stack

Last synced: 11 days ago
JSON representation

Awesome Lists containing this project

README 

          
# Agentic SEO System



**An AI-powered SEO workflow system leveraging Claude Code's sub-agent architecture with intelligent caching and automation.**



---



## Overview



The Agentic SEO System is a comprehensive framework designed to streamline SEO workflows by combining:



- **21 Specialized AI Agents** (4 utility + 17 analytical)

- **Intelligent Caching** to prevent redundant operations and API calls

- **Automated Data Collection** from sites, SERPs, and keyword APIs

- **Production-Ready Scripts** for crawling, analysis, and reporting



This system achieves **95-98% faster** operations through intelligent caching and **65% efficiency gains** when multiple agents share data.



---



## Key Features



### Utility Agents (Data Collection)

- **site-crawler**: Website crawling with SEO data extraction (titles, metas, headers, schema, links)

- **serp-data-collector**: SERP data fetching with competitive analysis

- **content-extractor**: Page content extraction with semantic analysis

- **keyword-data-fetcher**: Keyword metrics (volume, difficulty, CPC, trends)



### Analytical Agents (Analysis & Strategy)



**Technical SEO (7 agents):**

- technical-seo-auditor

- internal-linking-strategist

- meta-optimizer

- site-architecture-analyst

- schema-strategist

- page-speed-analyst

- url-structure-optimizer



**Keyword & Content Strategy (6 agents):**

- keyword-strategist

- content-brief-creator

- keyword-gap-analyzer

- content-gap-analyzer

- content-refresh-prioritizer

- semantic-content-analyzer



**Competitive & Optimization (4 agents):**

- serp-analyzer

- backlink-analyzer

- local-seo-strategist

- conversion-optimizer



### Intelligent Caching System

- **Filesystem-based** cache with TTL validation

- **Automatic expiration**: sites (24h), SERPs (7d), content (7d), keywords (30d)

- **Cache hit performance**: 95-98% faster than fresh operations

- **Multi-agent efficiency**: Single crawl serves multiple analyses



---



## Quick Start



### Prerequisites



```bash

# System requirements

- Bash 4.0+

- Python 3.11+

- uv (Python package manager)

- jq (JSON processor)



# API credentials (optional for full functionality)

- DataForSEO API key (for SERP and keyword data)

```



### Installation



1. **Clone the repository**

   ```bash

   git clone 

   cd agentic-seo-template

   ```



2. **Install Python dependencies**

   ```bash

   uv sync

   ```



3. **Configure API credentials** (optional)

   ```bash

   cp .env.example .env

   # Edit .env with your DataForSEO credentials

   ```



4. **Initialize business context**

   ```bash

   # The system will guide you to create a business profile

   # when you first request SEO work

   ```



### First Run



```bash

# Example: Technical SEO audit

"Run a technical SEO audit on example.com"



# The system will:

# 1. Check cache for example.com crawl

# 2. Crawl site if needed (or use cached data)

# 3. Run technical-seo-auditor agent

# 4. Generate comprehensive audit report

```



---



## Architecture



```

┌─────────────────────────────────────────────────────────────┐

│                      PRIMARY AGENT                           │

│              (Orchestrator - CLAUDE.md)                      │

└─────────────────────┬───────────────────────────────────────┘

                      │

      ┌───────────────┴───────────────┐

      │                               │

┌─────▼──────────┐          ┌────────▼──────────┐

│ UTILITY AGENTS │          │ ANALYTICAL AGENTS  │

│ (Data Collect) │          │ (Analysis/Strategy)│

├────────────────┤          ├───────────────────┤

│ site-crawler   │──┐       │ technical-seo-    │

│ serp-collector │  │       │   auditor         │

│ content-       │  │       │ keyword-strategist│

│   extractor    │  │       │ content-brief-    │

│ keyword-       │  │       │   creator         │

│   fetcher      │  │       │ (+ 14 more)       │

└────────────────┘  │       └───────────────────┘

                    │

          ┌─────────▼─────────┐

          │   CACHE SYSTEM    │

          │  (.cache/*)       │

          ├───────────────────┤

          │ - sites/          │

          │ - serps/          │

          │ - content/        │

          │ - keywords/       │

          └───────────────────┘

```



### Data Flow



1. **User** requests SEO task (via Claude Code)

2. **Primary Agent** analyzes request and determines required analytical agents

3. **Cache Check** determines if utility data exists

4. **Utility Agents** collect data if cache miss (or use cached if hit)

5. **Analytical Agents** run in parallel using cached data

6. **Reports** generated and returned to user



---



## Project Structure



```

agentic-seo-template/

├── .cache/                     # Cached data (gitignored)

│   ├── sites/                  # Site crawl caches

│   ├── serps/                  # SERP data caches

│   ├── content/                # Content extraction caches

│   └── keywords/               # Keyword data caches

│

├── logs/                       # Operation logs (gitignored)

│   ├── crawls/                 # Crawl logs

│   ├── api-calls/              # API call logs

│   ├── agents/                 # Agent execution logs

│   └── cache/                  # Cache operation logs

│

├── .claude/

│   ├── agents/                 # 21 sub-agent definitions

│   │   ├── site-crawler.md

│   │   ├── technical-seo-auditor.md

│   │   └── ... (19 more)

│   └── scripts/

│       ├── lib/                # Shared bash libraries

│       │   ├── cache-utils.sh

│       │   ├── logging.sh

│       │   ├── metadata.sh

│       │   └── progress.sh

│       ├── crawl/              # Site crawler scripts

│       ├── serp/               # SERP data scripts

│       ├── content/            # Content extraction scripts

│       ├── keywords/           # Keyword data scripts

│       └── cache/              # Cache management scripts

│

├── context/                    # Business context

│   ├── business-profile.md     # Brand information

│   ├── competitors.md          # Competitor list

│   └── style-guide.md          # Content style guide

│

├── analysis/                   # Analytical outputs

├── deliverables/               # Final deliverables

├── content/                    # Content drafts

├── tests/                      # Test suite

│   ├── unit/

│   ├── integration/

│   ├── e2e/

│   └── run-tests.sh

│

├── CLAUDE.md                   # Primary agent configuration

└── README.md                   # This file

```



---



## Usage Examples



### Technical SEO Audit



```bash

# Request

"Run a comprehensive technical SEO audit for acme.com"



# System workflow

1. Checks cache for acme.com crawl

2. Launches site-crawler if needed

3. Launches technical-seo-auditor

4. Generates report → analysis/technical-seo-audit-{DATE}.md



# Cache benefit

- First run: ~30-120s (crawl + analysis)

- Subsequent runs within 24h: <5s (cached crawl + analysis)

```



### Keyword Strategy



```bash

# Request

"Create a keyword strategy for 'project management software'"



# System workflow

1. Checks keyword cache for target keywords

2. Launches keyword-data-fetcher if needed

3. Checks SERP cache

4. Launches serp-data-collector if needed

5. Launches keyword-strategist with cached data

6. Generates strategy → analysis/keyword-strategy-{DATE}.md



# Multi-agent benefit

If you then request content brief for same keywords:

- Keyword data: CACHED (instant)

- SERP data: CACHED (instant)

- Only content-brief-creator runs fresh

```



### Content Strategy Workflow



```bash

# Request

"Analyze content gaps and create 3 content briefs for our top opportunities"



# System workflow (parallel agents)

1. site-crawler → crawl your site

2. site-crawler → crawl top 3 competitors

3. content-gap-analyzer → identify gaps

4. keyword-gap-analyzer → keyword opportunities

5. content-brief-creator → create briefs (parallel for top 3)



# Efficiency gain: 65% faster than sequential execution

```



---



## Cache Management



### Viewing Cache Status



```bash

# Display cache statistics

.claude/scripts/cache/stats.sh



# Output shows:

- Cache size by type

- Entry counts

- Hit rates

- Freshness status (valid/expired/invalid)

```



### Manual Cache Operations



```bash

# Invalidate specific cache

.claude/scripts/crawl/invalidate-cache.sh example.com

.claude/scripts/serp/invalidate-cache.sh 



# Force fresh data (bypass cache)

.claude/scripts/crawl/check-cache.sh example.com --force



# Cleanup expired caches

.claude/scripts/cache/cleanup.sh --dry-run  # Preview

.claude/scripts/cache/cleanup.sh             # Execute



# Setup automated daily cleanup

.claude/scripts/cache/setup-cron.sh

```



### Cache TTL (Time-to-Live)



| Cache Type | TTL | Reason |

|------------|-----|--------|

| Site crawls | 24 hours | Content changes frequently |

| SERP data | 7 days | Rankings shift weekly |

| Content | 7 days | Content rarely changes |

| Keywords | 30 days | Metrics stable month-to-month |



---



## Testing



### Run Test Suite



```bash

# All tests

tests/run-tests.sh



# Specific test type

tests/run-tests.sh unit

tests/run-tests.sh integration

tests/run-tests.sh e2e



# Verbose mode

tests/run-tests.sh --verbose

```



### Test Coverage



- **Unit tests**: 12 tests (cache utilities, libraries)

- **Integration tests**: 10 tests (site crawler workflow)

- **E2E tests**: 25 tests (3 complete workflows)

- **Pass rate**: 100% (47/47 tests passing)



### Performance Benchmarks



**Achieved Metrics** (exceeding targets):

- Cache hit speedup: **95-98%** (target: >90%)

- Multi-agent efficiency: **65%** (target: >60%)

- API call reduction: **95%+** through caching



---



## Performance



### Cache Performance



| Operation | Without Cache | With Cache | Speedup |

|-----------|---------------|------------|---------|

| Site crawl | 2000ms | 50ms | **97%** |

| SERP fetch | 3000ms | 50ms | **98%** |

| Content extraction | 1000ms | 50ms | **95%** |

| Keyword fetch | 2000ms | 50ms | **97%** |



### Multi-Agent Scenarios



**Example: Technical audit + keyword analysis + content gaps**

- Without caching (sequential): 6000ms

- With shared cache: 2100ms

- **Efficiency gain: 65%**



---



## API Integration



### DataForSEO API (Optional)



The system integrates with DataForSEO API for:

- SERP data collection

- Keyword metrics (volume, difficulty, CPC)



**Setup:**

```bash

# .env file

DATAFORSEO_LOGIN=your_email@example.com

DATAFORSEO_PASSWORD=your_api_password

```



**API Usage Tracking:**

- All API calls logged to `logs/api-calls/{DATE}.log`

- Caching reduces API calls by 95%+

- Batch processing for keyword data (1000 keywords/request)



---



## Troubleshooting



### Common Issues



**Cache seems stale but not expired**

```bash

# Manually invalidate

.claude/scripts/crawl/invalidate-cache.sh example.com



# Or force fresh data

"Run technical audit on example.com --force"

```



**Utility agent fails midway**

```bash

# Check progress

cat .cache/sites/example-com/progress.json



# Check logs

tail -f logs/crawls/{DATE}-*.log

```



**API credentials not working**

```bash

# Test credentials

source .claude/scripts/lib/credentials.sh

check_credentials_configured



# Verify .env file

cat .env

```



**Cache directory permission issues**

```bash

# Fix permissions

chmod -R 755 .cache

chmod 600 .env

```



---



## Development



### Creating New Analytical Agents



1. Create agent definition in `.claude/agents/{agent-name}.md`

2. Follow existing agent template structure

3. Document inputs (cache file paths)

4. Define output format

5. Add to `CLAUDE.md` subagents list

6. Test with real cached data



See `docs/agent-development.md` for detailed guide.



### Contributing



1. Fork repository

2. Create feature branch

3. Run test suite (`tests/run-tests.sh`)

4. Submit pull request



---



## System Status



**Current Version**: 1.0.0 (Production Ready)



**Completion Status**:

- Infrastructure: ✅ 100%

- Utility Agents: ✅ 100% (4/4 implemented)

- Analytical Agents: ✅ 100% (17/17 implemented)

- Testing: ✅ 100% (47/47 tests passing)

- Documentation: 🔄 In Progress



**Supported Workflows**:

- ✅ Technical SEO audits

- ✅ Keyword research & strategy

- ✅ Content strategy & briefs

- ✅ Competitive analysis

- ✅ SERP analysis

- ✅ Internal linking optimization

- ✅ Schema markup strategy

- ✅ Page speed optimization

- ✅ Local SEO strategy

- ✅ Conversion optimization



---



## Resources



- **User Guide**: `docs/user-guide.md` - Detailed usage instructions

- **Workflows**: `docs/workflows.md` - Common workflow examples

- **FAQ**: `docs/faq.md` - Frequently asked questions

- **Architecture**: `docs/architecture.md` - System design documentation

- **API Reference**: `docs/api-reference.md` - Script and API documentation



---



## License



[Your License Here]



---



## Credits



Built with [Claude Code](https://claude.com/claude-code) by Anthropic.



**Powered by:**

- Claude Sonnet 4.5

- advertools (web crawling)

- DataForSEO API (SERP & keyword data)

- BeautifulSoup4 (content extraction)

- jq (JSON processing)



---



**Last Updated**: 2025-10-18

**Status**: Production Ready ✅