Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/rmncldyo/firecrawl-toolkit
The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities.
https://github.com/rmncldyo/firecrawl-toolkit
ai-batch-scrape ai-crawler ai-scraper ai-toolkit batch-scrape crawl fire-crawl firecrawl firecrawl-ai map scrape sitemap sitemap-crawler sitemap-scraper web-crawler web-scraper
Last synced: 3 days ago
JSON representation
The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities.
- Host: GitHub
- URL: https://github.com/rmncldyo/firecrawl-toolkit
- Owner: RMNCLDYO
- License: mit
- Created: 2024-12-18T18:30:45.000Z (19 days ago)
- Default Branch: main
- Last Pushed: 2024-12-21T02:44:19.000Z (17 days ago)
- Last Synced: 2024-12-21T03:23:46.515Z (17 days ago)
- Topics: ai-batch-scrape, ai-crawler, ai-scraper, ai-toolkit, batch-scrape, crawl, fire-crawl, firecrawl, firecrawl-ai, map, scrape, sitemap, sitemap-crawler, sitemap-scraper, web-crawler, web-scraper
- Language: Python
- Homepage:
- Size: 85 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
- Security: .github/SECURITY.md
Awesome Lists containing this project
README
The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities. It offers seamless integration for web crawling, content extraction, and site mapping, allowing you to process websites with advanced features like custom actions, multiple output formats, and batch processing—all in one comprehensive package with minimal dependencies.
## 🚀 Features
- 🕷️ **Web Crawling**: Traverse websites with customizable depth and path controls, supporting both internal and external link processing
- 📄 **Content Extraction**: Extract content in multiple formats (Markdown, HTML, raw HTML) with smart content filtering
- 🗺️ **Site Mapping**: Generate comprehensive site maps with advanced search and subdomain capabilities
- 🔄 **Batch Processing**: Process multiple URLs simultaneously with unified configurations
- 🤖 **Custom Actions**: Automate complex interactions (clicking, scrolling, form filling) during scraping
- 📱 **Device Emulation**: Switch between mobile and desktop views with customizable headers
- 🌎 **Geolocation**: Simulate different locations with country and language preferences
- ⚡ **Smart Retry Logic**: Built-in retry mechanism with real-time status monitoring and webhooks
- 🪶 **Lightweight Design**: Minimal dependencies powered by requests for easy setup and deployment
- 🔒 **Robust Error Handling**: Comprehensive error catching and validation system
- 🎯 **Parameter Validation**: Extensive validation for all API inputs and configurations
- 📊 **Multiple Output Formats**: Support for various output types (Markdown, HTML, screenshots, etc.)
## 📋 Table of Contents
- [Installation](#-installation)
- [API Key Configuration](#-configuration)
- [Usage](#-usage)
- [Advanced Configuration](#%EF%B8%8F-advanced-configuration)
- [Available Formats](#-available-formats)
- [Supported Actions](#-supported-actions)
- [Error Handling and Safety](#-error-handling-and-safety)
- [Contributing](#-contributing)
- [Issues and Support](#-issues-and-support)
- [Feature Requests](#-feature-requests)
- [Versioning and Changelog](#-versioning-and-changelog)
- [Security](#-security)
- [License](#-license)
## 🛠 Installation
1. Clone the repository:
```bash
git clone https://github.com/RMNCLDYO/firecrawl-toolkit.git
```
2. Navigate to the repository folder:
```bash
cd firecrawl-toolkit
```
3. Install the required dependencies:
```bash
pip install -r requirements.txt
```
## 🔑 Configuration
1. Obtain an API key from [Firecrawl](https://www.firecrawl.dev/app/api-keys).
2. You have three options for managing your API key:
Click here to view the API key configuration options
- **Setting it as an environment variable on your device (recommended for everyday use)**
- Navigate to your terminal.
- Add your API key like so:
```shell
export FIRECRAWL_API_KEY=your_api_key
```
This method allows the API key to be loaded automatically when using the wrapper.
- **Using an .env file (recommended for development):**
- Install python-dotenv if you haven't already: `pip install python-dotenv`.
- Create a .env file in the project's root directory.
- Add your API key to the .env file like so:
```makefile
FIRECRAWL_API_KEY=your_api_key
```
This method allows the API key to be loaded automatically when using the wrapper.
- **Direct Input:**
- If you prefer not to use a `.env` file, you can directly pass your API key as an argument to the wrapper function.
***Wrapper***
```shell
api_key="your_api_key"
```
This method requires manually inputting your API key each time you initiate an API call.
## 💻 Usage
### Web Crawling
*For traversing websites and extracting content with customizable depth and path controls.*
```python
import firecrawl
# Basic crawling
firecrawl.crawl(
url="https://example.com",
formats=["markdown", "html"]
)
```
### Content Scraping
*For extracting content from specific URLs with custom actions and formatting.*
```python
import firecrawl
# Single URL scraping
firecrawl.scrape(
url="https://example.com",
formats=["markdown", "html"],
onlyMainContent=True
)
```
### Batch Scraping
*For processing multiple URLs simultaneously with shared configurations.*
```python
import firecrawl
# Batch scraping
firecrawl.batch_scrape(
urls=["https://example.com", "https://sitemaps.org"],
formats=["markdown", "html"]
)
```
### Site Mapping
*For generating comprehensive site maps with search capabilities.*
```python
import firecrawl
firecrawl.map(
url="https://example.com",
includeSubdomains=True,
limit=1000
)
```
## ⚙️ Advanced Configuration
| Description | Parameter | Type | Example |
|-------------|-----------|------|---------|
| Output Formats | `formats` | List | `["markdown", "html", "rawHtml"]` |
| Main Content Only | `onlyMainContent` | Boolean | `True` |
| Include Tags | `includeTags` | List | `["article", "main"]` |
| Exclude Tags | `excludeTags` | List | `["nav", "footer"]` |
| Custom Headers | `headers` | Dict | `{"User-Agent": "Custom"}` |
| Wait Time | `waitFor` | Integer | `1000` |
| Mobile View | `mobile` | Boolean | `False` |
| Custom Actions | `actions` | List | `[{"type": "click", "selector": "#btn"}]` |
| Location | `location` | Dict | `{"country": "US", "languages": ["en-US"]}` |
## 📊 Available Formats
- `markdown`: Formatted Markdown content
- `html`: Clean HTML content
- `rawHtml`: Original HTML content
- `links`: Extracted links
- `extract`: Custom content extraction
- `screenshot`: Page screenshot
- `screenshot@fullPage`: Full page screenshot
## 📁 Supported Actions
The toolkit supports various page interactions:
| Action Type | Description | Parameters |
|-------------|-------------|------------|
| `wait` | Wait for element/time | `milliseconds`, `selector` |
| `click` | Click elements | `selector` |
| `write` | Input text | `selector`, `text` |
| `press` | Press keyboard keys | `key` |
| `scroll` | Scroll page | `direction`, `amount` |
| `screenshot` | Take screenshots | `fullPage` |
| `scrape` | Get current page state | None |
## 🔒 Error Handling and Safety
| Error Type | Description | Solution |
|------------|-------------|----------|
| ConfigurationError | Missing or invalid configuration | Check config.yaml and API key |
| ValidationError | Invalid request parameters | Verify parameter values |
| APIError | API-related issues | Check error message for details |
| NetworkError | Connection problems | Verify internet connection |
| ResponseError | Invalid API response | Check response format expectations |
## 🤝 Contributing
Contributions are welcome!
Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for detailed guidelines on how to contribute to this project.
## 🐛 Issues and Support
Encountered a bug? We'd love to hear about it. Please follow these steps to report any issues:
1. Check if the issue has already been reported.
2. Use the [Bug Report](.github/ISSUE_TEMPLATE/bug_report.md) template to create a detailed report.
3. Submit the report [here](https://github.com/RMNCLDYO/firecrawl-toolkit/issues).
Your report will help us make the project better for everyone.
## 💡 Feature Requests
Got an idea for a new feature? Feel free to suggest it. Here's how:
1. Check if the feature has already been suggested or implemented.
2. Use the [Feature Request](.github/ISSUE_TEMPLATE/feature_request.md) template to create a detailed request.
3. Submit the request [here](https://github.com/RMNCLDYO/firecrawl-toolkit/issues).
Your suggestions for improvements are always welcome.
## 🔁 Versioning and Changelog
Stay up-to-date with the latest changes and improvements in each version:
- [CHANGELOG.md](.github/CHANGELOG.md) provides detailed descriptions of each release.
## 🔐 Security
Your security is important to us. If you discover a security vulnerability, please follow our responsible disclosure guidelines found in [SECURITY.md](.github/SECURITY.md). Please refrain from disclosing any vulnerabilities publicly until said vulnerability has been reported and addressed.
## 📄 License
Licensed under the MIT License. See [LICENSE](LICENSE) for details.