{"id":46183467,"url":"https://github.com/greysquirr3l/stygian","last_synced_at":"2026-04-27T04:01:48.005Z","repository":{"id":341565077,"uuid":"1169777451","full_name":"greysquirr3l/stygian","owner":"greysquirr3l","description":"High-performance graph-based web scraping engine + anti-detection browser automation for Rust","archived":false,"fork":false,"pushed_at":"2026-04-20T23:08:20.000Z","size":16049,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-20T23:35:23.382Z","etag":null,"topics":["anti-bot","async-rust","browser-automation","chrome-devtools-protocol","dag","headless-browser","llm","rust","tokio","web-scraping"],"latest_commit_sha":null,"homepage":"https://greysquirr3l.github.io/stygian","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/greysquirr3l.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-03-01T07:34:28.000Z","updated_at":"2026-04-20T23:08:24.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/greysquirr3l/stygian","commit_stats":null,"previous_names":["greysquirr3l/stygian"],"tags_count":35,"template":false,"template_full_name":null,"purl":"pkg:github/greysquirr3l/stygian","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/greysquirr3l%2Fstygian","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/greysquirr3l%2Fstygian/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/greysquirr3l%2Fstygian/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/greysquirr3l%2Fstygian/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/greysquirr3l","download_url":"https://codeload.github.com/greysquirr3l/stygian/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/greysquirr3l%2Fstygian/sbom","scorecard":{"id":1244224,"data":{"date":"2026-03-02T16:37:12Z","repo":{"name":"github.com/greysquirr3l/stygian","commit":"82756519bbb1cd05b723406a36762db7a46b6880"},"scorecard":{"version":"v5.3.0","commit":"c22063e786c11f9dd714d777a687ff7c4599b600"},"score":4.4,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/20 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"project was created within the last 90 days. Please review its contents carefully","details":["Warn: Repository was created within the last 90 days."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":10,"reason":"GitHub workflow tokens follow principle of least privilege","details":["Warn: jobLevel 'contents' permission set to 'write': .github/workflows/release.yml:93","Info: jobLevel 'actions' permission set to 'read': .github/workflows/scorecard.yml:21","Info: jobLevel 'contents' permission set to 'read': .github/workflows/scorecard.yml:20","Info: topLevel 'contents' permission set to 'read': .github/workflows/ci.yml:14","Info: topLevel 'contents' permission set to 'read': .github/workflows/docs.yml:14","Info: topLevel 'contents' permission set to 'read': .github/workflows/release.yml:12","Info: topLevel permissions set to 'read-all': .github/workflows/scorecard.yml:10","Info: topLevel 'contents' permission set to 'read': .github/workflows/security.yml:22"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:91: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:94: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:105: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:108: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:111: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:129: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:132: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:137: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:156: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:159: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:162: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:21: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:27: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:43: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:46: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:49: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:67: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:70: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:75: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/docs.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/docs.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/docs.yml:32: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/docs.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/docs.yml:35: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/docs.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/docs.yml:77: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/docs.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/docs.yml:91: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/docs.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/release.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/release.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:44: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/release.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:49: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/release.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:52: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/release.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:95: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/release.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:113: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/release.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/scorecard.yml:25: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/scorecard.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/scorecard.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/scorecard.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/scorecard.yml:37: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/scorecard.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/scorecard.yml:44: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/scorecard.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/security.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/security.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/security.yml:32: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/security.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/security.yml:44: update your workflow using https://app.stepsecurity.io/secureworkflow/greysquirr3l/stygian/security.yml/main?enable=pin","Info:   0 out of  26 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of  13 third-party GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":6,"reason":"4 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: RUSTSEC-2025-0052","Warn: Project is vulnerable to: RUSTSEC-2025-0057","Warn: Project is vulnerable to: RUSTSEC-2025-0119","Warn: Project is vulnerable to: RUSTSEC-2023-0071"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 13 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#sast"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: GNU Affero General Public License v3.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#fuzzing"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/release.yml:39"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#packaging"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#signed-releases"}},{"name":"Contributors","score":0,"reason":"project has 0 contributing companies or organizations -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project has a set of contributors from multiple organizations (e.g., companies).","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#contributors"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#branch-protection"}},{"name":"Dependency-Update-Tool","score":0,"reason":"no update tool detected","details":["Warn: no dependency update tool configurations found"],"documentation":{"short":"Determines if the project uses a dependency update tool.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#dependency-update-tool"}},{"name":"CI-Tests","score":10,"reason":"3 out of 3 merged PRs checked by a CI test -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project runs tests before pull requests are merged.","url":"https://github.com/ossf/scorecard/blob/c22063e786c11f9dd714d777a687ff7c4599b600/docs/checks.md#ci-tests"}}]},"last_synced_at":"2026-03-02T16:40:51.215Z","repository_id":341565077,"created_at":"2026-03-02T16:40:51.215Z","updated_at":"2026-03-02T16:40:51.215Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32321940,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T23:26:28.701Z","status":"online","status_checked_at":"2026-04-27T02:00:06.769Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anti-bot","async-rust","browser-automation","chrome-devtools-protocol","dag","headless-browser","llm","rust","tokio","web-scraping"],"created_at":"2026-03-02T21:19:39.844Z","updated_at":"2026-04-27T04:01:47.997Z","avatar_url":"https://github.com/greysquirr3l.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# stygian\n\n![stygian](assets/img/stygian-logo.png)\n\n**High-performance web scraping toolkit for Rust** — graph-based execution engine + anti-detection browser automation.\n\n[![CI](https://github.com/greysquirr3l/stygian/actions/workflows/ci.yml/badge.svg)](https://github.com/greysquirr3l/stygian/actions/workflows/ci.yml)\n[![Security Audit](https://github.com/greysquirr3l/stygian/actions/workflows/security.yml/badge.svg)](https://github.com/greysquirr3l/stygian/actions/workflows/security.yml)\n[![Documentation](https://github.com/greysquirr3l/stygian/actions/workflows/docs.yml/badge.svg)](https://github.com/greysquirr3l/stygian/actions/workflows/docs.yml)\n[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/greysquirr3l/stygian/badge)](https://securityscorecards.dev/viewer/?uri=github.com/greysquirr3l/stygian)\n[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](LICENSE)\n[![License: Commercial](https://img.shields.io/badge/License-Commercial-green.svg)](LICENSE-COMMERCIAL.md)\n\n---\n\n## What is stygian?\n\nStygian is a **monorepo** containing five complementary Rust crates for building robust, scalable web scraping systems:\n\n### [stygian-graph](crates/stygian-graph)\n\nGraph-based scraping engine treating pipelines as DAGs with pluggable service modules:\n\n- **Hexagonal architecture** — domain core isolated from infrastructure\n- **Extreme concurrency** — Tokio for I/O, Rayon for CPU-bound tasks\n- **AI extraction** — Claude, GPT, Gemini, GitHub Copilot, Ollama support\n- **Multi-modal** — images, PDFs, videos via LLM vision APIs\n- **Distributed execution** — Redis/Valkey-backed work queues\n- **Circuit breaker** — graceful degradation when services fail\n- **Idempotency** — safe retries with deduplication keys\n- **Graph introspection** — runtime inspection, impact analysis, execution waves\n\n### [stygian-browser](crates/stygian-browser)\n\nAnti-detection browser automation library for bypassing modern bot protection:\n\n- **Browser pooling** — warm pool, sub-100ms acquisition\n- **CDP-based** — Chrome DevTools Protocol via chromiumoxide\n- **Stealth features** — navigator spoofing, canvas noise, WebGL randomization\n- **Human behavior** — Bézier mouse paths, realistic typing\n- **TLS fingerprinting** — profile-matched JA3/JA4 signatures\n- **Cloudflare/DataDome/PerimeterX** — bypass detection layers\n\n### [stygian-proxy](crates/stygian-proxy)\n\nProxy pool management with intelligent rotation:\n\n- **Multi-protocol** — HTTP, HTTPS, SOCKS5 support\n- **Health checking** — automatic dead proxy removal\n- **Sticky sessions** — domain-bound proxy affinity\n- **Weighted selection** — prioritize faster/more reliable proxies\n\n### [stygian-mcp](crates/stygian-mcp)\n\nMCP (Model Context Protocol) aggregator for LLM tool integration:\n\n- **Unified interface** — single JSON-RPC 2.0 server over stdin/stdout\n- **Tool namespacing** — `graph_*`, `browser_*`, `proxy_*` prefixes\n- **Cross-crate tools** — `scrape_proxied`, `browser_proxied`\n- **VS Code/Claude** — direct integration with MCP-compatible clients\n\nMCP tool matrix (aggregator surface):\n\n| Namespace | Representative tools | Purpose |\n| --------- | -------------------- | ------- |\n| `graph_*` | `graph_scrape`, `graph_scrape_rest`, `graph_scrape_graphql`, `graph_pipeline_validate`, `graph_pipeline_run` | HTTP/API/feed scraping and DAG execution |\n| `browser_*` | `browser_acquire`, `browser_acquire_and_extract`, `browser_navigate`, `browser_query`, `browser_extract`, `browser_extract_with_fallback`, `browser_extract_resilient`, `browser_release` | Headless browser automation and structured extraction |\n| `proxy_*` | `proxy_add`, `proxy_remove`, `proxy_pool_stats`, `proxy_acquire`, `proxy_acquire_for_domain`, `proxy_acquire_with_capabilities`, `proxy_fetch_freelist`, `proxy_fetch_freeapiproxies`, `proxy_release` | Proxy pool management, capability-aware leasing, and feed bootstrap |\n| cross-crate | `scrape_proxied`, `browser_proxied` | End-to-end orchestration across graph/browser/proxy |\n\n### [stygian-extract-derive](crates/stygian-extract-derive)\n\nProc-macro backend that powers `#[derive(Extract)]` in `stygian-browser`:\n\n- **Declarative extraction** — annotate structs with CSS selectors and attribute targets\n- **Internal crate** — do not add directly; enable via `stygian-browser`'s `extract` feature\n- **Zero boilerplate** — generates typed DOM-to-struct deserialization at compile time\n\n```toml\nstygian-browser = { version = \"*\", features = [\"extract\"] }\n```\n\n---\n\n## Quick Start\n\n### Graph Scraping Pipeline\n\n```rust,ignore\nuse serde_json::json;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n    let pipeline = PipelineBuilder::new()\n        .node(\"fetch\", HttpAdapter::new())\n        .node(\"parse\", MyParserAdapter)\n        .edge(\"fetch\", \"parse\")\n        .build()?;\n\n    let results = pipeline\n        .execute(json!({\"url\": \"https://example.com\"}))\n        .await?;\n    \n    println!(\"Results: {:?}\", results);\n    Ok(())\n}\n```\n\n### Browser Automation\n\n```rust,ignore\nuse std::time::Duration;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n    let pool = BrowserPool::new(BrowserConfig::default()).await?;\n    let handle = pool.acquire().await?;\n\n    let browser = handle\n        .browser()\n        .ok_or_else(|| std::io::Error::other(\"browser handle already released\"))?;\n    let mut page = browser.new_page().await?;\n    page.navigate(\n        \"https://example.com\",\n        WaitUntil::Selector(\"body\".to_string()),\n        Duration::from_secs(30),\n    ).await?;\n\n    let html = page.content().await?;\n    println!(\"Page loaded: {} bytes\", html.len());\n\n    handle.release().await;\n    Ok(())\n}\n```\n\n---\n\n## Installation\n\nAdd to your `Cargo.toml`:\n\n```toml\n[dependencies]\nstygian-graph = { version = \"*\", features = [\"browser\"] }\nstygian-browser = \"*\"     # optional, for JavaScript rendering\nstygian-proxy = \"*\"       # optional, for proxy pool management\ntokio = { version = \"1\", features = [\"full\"] }\n```\n\nFor MCP integration, install the `stygian-mcp` binary with the `extract` feature for full tool coverage:\n\n```bash\n# From crates.io\ncargo install stygian-mcp --features extract\n\n# Or from source\ncargo install --path crates/stygian-mcp --features extract --locked\n```\n\nThen wire it into your MCP client. **VS Code** (`.vscode/mcp.json` or `settings.json`):\n\n```json\n{\n  \"mcp\": {\n    \"servers\": {\n      \"stygian\": {\n        \"command\": \"stygian-mcp\",\n        \"args\": [],\n        \"type\": \"stdio\"\n      }\n    }\n  }\n}\n```\n\n**Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`):\n\n```json\n{\n  \"mcpServers\": {\n    \"stygian\": {\n      \"command\": \"stygian-mcp\",\n      \"args\": []\n    }\n  }\n}\n```\n\n\u003e **Note:** Browser tools require Chrome/Chromium. On macOS: `brew install --cask google-chrome`\n\n### Common Feature Combinations\n\n```toml\n# Minimal: HTTP scraping only\nstygian-graph = \"*\"\n\n# Full-featured: browser, AI extraction, distributed queue\nstygian-graph = { version = \"*\", features = [\"full\"] }\n\n# Browser + Proxy integration\nstygian-browser = { version = \"*\", features = [\"stealth\", \"tls-config\"] }\nstygian-proxy = { version = \"*\", features = [\"browser\", \"socks\"] }\n```\n\n### Runner-First Acquisition (Recommended)\n\nFor hostile or variable targets, prefer a single `browser_acquire_and_extract` call over manually chaining low-level browser tools.\n\nOptional Browserbase integration:\n\n- Build `stygian-browser` with feature `browserbase` to enable the Browserbase-managed stage.\n- Per request, set `browserbase_enabled` (or alias `use_browserbase`) to `true`.\n- Provide runtime credentials via `BROWSERBASE_API_KEY` and `BROWSERBASE_PROJECT_ID`.\n\nMode guide:\n\n| Mode | When to use |\n| ---- | ----------- |\n| `fast` | Low-friction pages where speed matters most |\n| `resilient` | Default for general production scraping with moderate anti-bot pressure |\n| `hostile` | High-friction targets needing heavier escalation and retries |\n| `investigate` | Diagnostics-first runs to understand which strategy tier succeeds |\n\nEnd-to-end example:\n\n```json\n{\n  \"jsonrpc\": \"2.0\",\n  \"method\": \"tools/call\",\n  \"params\": {\n    \"name\": \"browser_acquire_and_extract\",\n    \"arguments\": {\n      \"url\": \"https://example.com/products\",\n      \"mode\": \"resilient\",\n      \"wait_for_selector\": \"article.product\",\n      \"extraction_js\": \"Array.from(document.querySelectorAll('article.product h2')).map(n =\u003e n.textContent?.trim()).filter(Boolean)\",\n      \"total_timeout_secs\": 45,\n      \"browserbase_enabled\": true\n    }\n  }\n}\n```\n\nMigration note (old path vs runner path):\n\n- Old low-level path: `browser_acquire` -\u003e `browser_navigate` -\u003e `browser_eval`/`browser_extract` -\u003e `browser_release`.\n- New runner path: one `browser_acquire_and_extract` call with `mode` and optional `wait_for_selector`/`extraction_js`/`browserbase_enabled`.\n- Keep low-level tools when you need custom multi-step interaction. Use runner-first for deterministic escalation with fewer moving parts.\n\n---\n\n## Architecture\n\n### stygian-graph: Hexagonal (Ports \u0026 Adapters)\n\n```\nDomain Layer (business logic)\n    ↑\nPorts (trait definitions)\n    ↑\nAdapters (HTTP, browser, AI providers, storage)\n```\n\n- **Zero I/O dependencies** in domain layer\n- **Dependency inversion** — adapters depend on ports, not vice versa\n- **Extreme testability** — mock any external system\n\n### stygian-browser: Modular\n\n- Self-contained modules with clear interfaces\n- Pool management with resource limits\n- Graceful degradation on browser unavailability\n\n---\n\n## Project Structure\n\n```\nstygian/\n├── crates/\n│   ├── stygian-graph/          # Scraping engine\n│   ├── stygian-browser/        # Browser automation\n│   ├── stygian-proxy/          # Proxy pool management\n│   ├── stygian-mcp/            # MCP aggregator server\n│   └── stygian-extract-derive/ # Proc-macro for #[derive(Extract)]\n├── examples/                # Example pipelines\n├── book/                    # mdBook documentation\n├── docs/                    # Architecture docs\n└── assets/                  # Diagrams, images\n```\n\n---\n\n## Development\n\n### Setup\n\n```bash\n# Install Rust 1.94.0+\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n\n# Build workspace\ncargo build --workspace\n\n# Run tests\ncargo test --workspace\n\n# Run clippy\ncargo clippy --workspace -- -D warnings\n```\n\n### Testing\n\n```bash\n# Unit tests\ncargo test --lib\n\n# Integration tests\ncargo test --test '*'\n\n# All tests (browser integration tests require Chrome)\ncargo test --all-features\n\n# Measure coverage (requires cargo-tarpaulin)\ncargo tarpaulin --workspace --all-features --ignore-tests --out Lcov\n```\n\n`stygian-graph` achieves strong unit coverage across domain, ports, and adapter layers.\n`stygian-browser` coverage is structurally bounded by the Chrome CDP requirement — all tests\nthat spin up a real browser are marked `#[ignore = \"requires Chrome\"]`; pure-logic tests are\nfully covered.\n\n---\n\n## Contributing\n\nContributions welcome! Please:\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'feat: add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n### Commit Convention\n\nUse [Conventional Commits](https://www.conventionalcommits.org/):\n\n- `feat:` — new feature\n- `fix:` — bug fix\n- `refactor:` — code restructuring\n- `test:` — test additions/changes\n- `docs:` — documentation updates\n\n---\n\n## License\n\nDual-licensed under:\n\n- **[GNU Affero General Public License v3.0](LICENSE)** (`AGPL-3.0-only`) — free for open-source use\n- **[Commercial License](LICENSE-COMMERCIAL.md)** — available for proprietary/closed-source use\n\nUnder the AGPL, any modifications or derivative works must also be released under the AGPL-3.0, including when the software is used to provide a network service. For commercial licensing options that permit proprietary use, see [LICENSE-COMMERCIAL.md](LICENSE-COMMERCIAL.md).\n\n### Contribution\n\nUnless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you shall be dual-licensed as above, without any additional terms or conditions.\n\n---\n\n## Acknowledgments\n\nBuilt with:\n\n- [chromiumoxide](https://github.com/mattsse/chromiumoxide) — CDP client\n- [petgraph](https://github.com/petgraph/petgraph) — graph algorithms\n- [tokio](https://tokio.rs/) — async runtime\n- [reqwest](https://github.com/seanmonstar/reqwest) — HTTP client\n\n---\n\n**Status**: Active development | Rust 2024 edition | Linux + macOS\n\nFor detailed documentation, see the [project docs site](https://greysquirr3l.github.io/stygian).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgreysquirr3l%2Fstygian","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgreysquirr3l%2Fstygian","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgreysquirr3l%2Fstygian/lists"}