{"id":20286447,"url":"https://github.com/paganini2008/greenfinger","last_synced_at":"2026-02-27T00:04:11.532Z","repository":{"id":57721978,"uuid":"335221428","full_name":"paganini2008/greenfinger","owner":"paganini2008","description":"GreenFinger is a cutting-edge distributed web crawling framework built on Spring Cloud, PostgreSQL, and Elasticsearch, powered by the high-performance Netty NIO engine. It features an intuitive Web UI for managing and monitoring tasks, dynamic node scaling, and real-time data processing. ","archived":false,"fork":false,"pushed_at":"2025-02-04T00:37:47.000Z","size":285966,"stargazers_count":1,"open_issues_count":7,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-22T16:43:16.765Z","etag":null,"topics":["distributed-systems","high-performance","mircoservice","netty4","playwright","realtime-messaging","selenium","springboot2","webcrawler","webscraping"],"latest_commit_sha":null,"homepage":"https://paganini2008.github.io/2021/06/Introduction-to-Greenfinger-High-Performance-Distributed-Web-Crawling-Framework/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paganini2008.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-02-02T08:37:54.000Z","updated_at":"2025-02-04T00:37:50.000Z","dependencies_parsed_at":"2025-02-01T09:18:45.744Z","dependency_job_id":"ddf14a3c-8372-4153-a976-be6c680f7920","html_url":"https://github.com/paganini2008/greenfinger","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/paganini2008/greenfinger","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paganini2008%2Fgreenfinger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paganini2008%2Fgreenfinger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paganini2008%2Fgreenfinger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paganini2008%2Fgreenfinger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paganini2008","download_url":"https://codeload.github.com/paganini2008/greenfinger/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paganini2008%2Fgreenfinger/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29878275,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-26T23:51:21.483Z","status":"ssl_error","status_checked_at":"2026-02-26T23:50:46.793Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","high-performance","mircoservice","netty4","playwright","realtime-messaging","selenium","springboot2","webcrawler","webscraping"],"created_at":"2024-11-14T14:34:19.242Z","updated_at":"2026-02-27T00:04:11.508Z","avatar_url":"https://github.com/paganini2008.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Greenfinger\n\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n[![Spring Cloud](https://img.shields.io/badge/Spring%20Cloud-Compatible-brightgreen.svg)](https://spring.io/projects/spring-cloud)\n[![PostgreSQL](https://img.shields.io/badge/PostgreSQL-Compatible-blue.svg)](https://www.postgresql.org/)\n[![Elasticsearch](https://img.shields.io/badge/Elasticsearch-Compatible-green.svg)](https://www.elastic.co/)\n[![Netty](https://img.shields.io/badge/Netty-Based-brightgreen.svg)](https://netty.io/)\n\n[**GreenFinger**](https://github.com/paganini2008/greenfinger) is a high-performance, highly scalable distributed web crawler built in Java. Designed for both enterprise and individual users, it offers an intuitive user interface and minimal configuration, enabling seamless and efficient web resource extraction. As an open-source solution, [**GreenFinger**](https://github.com/paganini2008/greenfinger)  provides a powerful yet user-friendly approach to large-scale web crawling and data acquisition.\n\n\n\n\n## 🌟Features: \n------------------------------\n\n1. Seamless Spring Boot Integration\n   Natively integrates with Spring Boot, ensuring effortless configuration, deployment, and maintenance. \n2. Scalable, High-Throughput Distributed Crawling \n   Architected for distributed environments, enabling seamless horizontal scaling to handle massive workloads efficiently. \n3. Optimized Network Communication with Netty \n   Leverages Netty for ultra-low-latency networking, with additional support for Mina and Grizzly for flexible communication strategies. \n4. Enterprise-Grade URL Deduplication \n   Implements billion-scale deduplication using Bloom Filter and RocksDB, ensuring optimal storage efficiency and crawl accuracy. \n5. Granular URL Customization \n   Supports fine-grained control over URL selection, allowing users to define initial URLs, retain only relevant URLs, and exclude undesired links dynamically. \n6. Advanced Fault Tolerance \u0026 Crawler Constraints \n   Incorporates intelligent retry mechanisms, configurable timeouts, target URL limits, and maximum crawl depth enforcement for robust error handling. \n7. Multi-Engine Web Content Extraction \n   Integrates Playwright, Selenium, and HtmlUnit to capture and process dynamic web content efficiently.  \n8. Strict Adherence to Robots.txt \n   Fully complies with the Robots Exclusion Protocol, ensuring ethical and responsible web crawling.  \n9. Comprehensive Developer API \n   Exposes a rich set of APIs, enabling seamless customization, extension, and integration into diverse ecosystems. \n10. Automated Authentication Handling \n    Supports intelligent login and logout workflows, facilitating seamless authentication across secured web portals. \n11. Version-Controlled Web Document Management \n    Assigns unique versioning to crawled documents, enabling multi-version indexing for enhanced content tracking and retrieval. \n12. Intuitive Angular-Based Web Interface \n    Provides a modern, interactive dashboard built with Angular, empowering users with real-time monitoring, configuration, and management capabilities. \n\n\n\n## 🚀  Technology Stack\n-----------------------------------------\n\n| Technology      | Version Requirement | Description |\n|---------------|----------------|-------------|\n| ☕ **JDK**      | 17 or later    | Core Java runtime environment |\n| 🌱 **Spring Boot** | 2.7.18        | Backend framework for microservices and rapid development |\n| ⚡ **Netty**     | 4.x           | High-performance asynchronous networking framework |\n| 🔥 **Redis**     | 7.x or later  | In-memory data store for caching and message queuing |\n| 🐘 **PostgreSQL** | 9.x or later  | High-performance, open-source relational database |\n| 🔍 **ElasticSearch** | 7.16.2 or later | Distributed search and analytics engine |\n| 🕷 **Selenium** | 4.x           | Web automation framework for headless and UI-based scraping |\n| 🎭 **Playwright** | 1.48          | Modern browser automation tool for scraping and testing |\n| 📄 **HtmlUnit**  | 2.6           | Lightweight headless browser for quick HTML processing |\n| 🌐 **Angular**    | 19.x                  | Frontend framework for building interactive web applications |\n| 🎨 **Angular Material** | Latest        | UI component library for modern, responsive designs |\n\n\n\n## Install:\n-----------------------------\n\n* Git Repository：\n  https://github.com/paganini2008/greenfinger.git\n* Wiki:\n  https://paganini2008.github.io/greenfinger/\n* Directory Structure：\n``` shell\n📂 greenfinger\n├── 📂 greenfinger-ui  \n│   ├── 📜 pom.xml  \n│   ├── 📂 src  \n│   │   ├── 📂 config  # Configuration files  \n│   │   ├── 📂 db      # Database-related scripts and configurations  \n│   │   └── ...  \n├── 📂 greenfinger-spring-boot-starter  \n│   ├── 📜 pom.xml  \n│   ├── 📂 src  \n│   └── ...  \n├── 📜 LICENSE  \n├── 📜 pom.xml  \n└── 📜 README.md  \n\n```\n### Steps: \n1.  Modify configuration:\n\n``` yaml\nspring:\n  redis:\n    database: 0\n    host: 127.0.0.1\n    port: 6379\n    password: 123456\n  elasticsearch:\n    rest:\n      uris: http://127.0.0.1:9200\n      connection-timeout: 10000\n      read-timeout: 60000\n  datasource:\n    driver-class-name: org.postgresql.Driver\n    url: jdbc:postgresql://localhost:5432/test?characterEncoding=utf8\u0026allowMultiQueries=true\u0026useSSL=false\u0026stringtype=unspecified\n    username: admin\n    password: 123456\n# Binding host name is preferred\ndoodler:\n  transmitter:\n    nio:\n      server:\n        bindHostName: 127.0.0.1\n# Internal Work ThreadPool Threads      \ngreenfinger:\n  workThreads: 1000\n\n```\n\n2. Create database and import table scripts\n   **execute db/crawler.sql**\n3. \u003ccode\u003emvn clean install\u003c/code\u003e\n4. run jar with \u003ccode\u003ejava  --add-opens=java.base/java.lang=ALL-UNNAMED -jar greenfinger-ui-service-1.0.0-SNAPSHOT.jar\u003c/code\u003e\n\n2. Open the Web UI\n    http://localhost:6120/ui/index.html\n\n ![image.png](https://paganini2008.github.io/assets/images/greenfinger/p0.png)\n\n\n\n##  Greenfinger UI Guide：\n-------------------------\n####  Catalog Management\n ![image.png](https://paganini2008.github.io/assets/images/greenfinger/p1.png)\n\n#### Create a catalog\n ![image.png](https://paganini2008.github.io/assets/images/greenfinger/p2.png)\n\n#### Edit a catalog\n ![image.png](https://paganini2008.github.io/assets/images/greenfinger/p3.png)\n\n#### Run web crawler\n ![image.png](https://paganini2008.github.io/assets/images/greenfinger/p4.png)\n\n#### Monitor\n ![image.png](https://paganini2008.github.io/assets/images/greenfinger/p5.png)\n\n####  Query\n ![image.png](https://paganini2008.github.io/assets/images/greenfinger/p6.png)\n\n\n\n## Customize your application\n\n### Application Integration\n-------------------------\n**Step1**:  add dependency in your pom.xml:\n\n``` xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.paganini2008\u003c/groupId\u003e\n  \u003cartifactId\u003egreenfinger-spring-boot-starter\u003c/artifactId\u003e\n  \u003cversion\u003e1.0.0-SNAPSHOT\u003c/version\u003e\n\u003c/dependency\u003e\n```\n**Step2**: add \u003ccode\u003e@EnableGreenFingerServer\u003c/code\u003e on the main:\n\n``` java\n@EnableAsync(proxyTargetClass = true)\n@EnableScheduling\n@EnableGreenfingerServer\n@SpringBootApplication\npublic class GreenFingerServerConsoleMain {\n\n\tpublic static void main(String[] args) {\n\t\tSpringApplication.run(GreenFingerServerConsoleMain.class, args);\n\t}\n}\n```\n**Step3**:  Run it\n\n\n## Documentation\nFor detailed setup instructions, API references, and advanced configuration, visit the [Official Documentation](https://github.com/paganini2008/greenfinger/wiki/QuickStart).\n\n---\n\n## Contributing\nContributions are welcome! Refer to the [Contributing Guide](CONTRIBUTING.md) for more information.\n\n---\n\n## License\nGreenfinger is licensed under the Apache License. See the [LICENSE](LICENSE) file for more details.\n\n---\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaganini2008%2Fgreenfinger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaganini2008%2Fgreenfinger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaganini2008%2Fgreenfinger/lists"}