https://github.com/jasonlovesdoggo/skyfence
https://github.com/jasonlovesdoggo/skyfence
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/jasonlovesdoggo/skyfence
- Owner: JasonLovesDoggo
- Created: 2025-04-23T01:22:16.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2025-04-23T01:28:36.000Z (about 2 months ago)
- Last Synced: 2025-05-08T07:40:31.090Z (about 1 month ago)
- Language: Go
- Size: 125 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Fetchers Module
The **Fetchers Module** is a Go package designed to fetch IP ranges for various cloud providers and services. It is part of the **Caddy Defender** project, which provides middleware for blocking or manipulating traffic based on IP ranges. This module supports fetching IP ranges from AWS, Google Cloud Platform (GCP), OpenAI, GitHub Copilot, and more.
---
## Features
- **Multiple Cloud Providers**: Fetch IP ranges for AWS, GCP, OpenAI, GitHub Copilot, and other services.
- **Region-Specific IP Ranges**: Fetch IP ranges for specific AWS regions (e.g., `us-east-1`, `eu-west-1`).
- **Dynamic Fetching**: IP ranges are fetched dynamically from official sources (e.g., AWS, GCP).
- **Pregenerated Results**: Use pregenerated IP ranges from the `ranges/data` directory for faster setup.
- **Extensible**: Easily add new fetchers for additional services or providers.
- **Concurrency**: Fetch IP ranges concurrently for improved performance.
---
## Supported Fetchers
Please see [the readme](https://github.com/JasonLovesDoggo/caddy-defender#embedded-ip-ranges)
---
## Pregenerated Results
For convenience, pregenerated IP ranges are available in the `ranges/data` directory. These ranges are generated by running the fetchers and can be used directly in your project without needing to fetch them dynamically.
### Using Pregenerated Results
To use the pregenerated IP ranges, import the `data` package and access the `IPRanges` map:
```go
package main
import (
"fmt"
"github.com/jasonlovesdoggo/caddy-defender/ranges/data"
)
func main() {
// Access pregenerated IP ranges for AWS
awsRanges := data.IPRanges["aws"]
fmt.Println("AWS IP ranges:", awsRanges)
// Access pregenerated IP ranges for GCP
gcloudRanges := data.IPRanges["gcloud"]
fmt.Println("GCP IP ranges:", gcloudRanges)
// Access pregenerated IP ranges for OpenAI
openaiRanges := data.IPRanges["openai"]
fmt.Println("OpenAI IP ranges:", openaiRanges)
// Access pregenerated IP ranges for Mistral
mistralRanges := data.IPRanges["mistral"]
fmt.Println("Mistral IP ranges:", mistralRanges)
// ...
}
```
### Available Pregenerated Ranges
The `IPRanges` map in the `data` package contains the following keys:
| Key | Description |
|-----------------|----------------------------------------------------------|
| `vpn` | Known VPN services |
| `aws` | Global IP ranges for AWS services. |
| `aws-us-east-1` | IP ranges for the AWS `us-east-1` region. |
| `aws-us-west-1` | IP ranges for the AWS `us-west-1` region. |
| `gcloud` | IP ranges for Google Cloud Platform (GCP) services. |
| `openai` | IP ranges for OpenAI services (e.g., ChatGPT, GPTBot). |
| `oci` | IP ranges for Oracle Cloud Infrastructure (OCI) services |
| `githubcopilot` | IP ranges for GitHub Copilot services. |
| `private` | IP ranges for private networks (used for testing). |
| `mistral` | IP ranges for Mistral services. |
| `vultr` | IP ranges for Vultr Cloud services. |
| `cloudflare` | IP ranges for Cloudflare services. |
| `digitalocean` | IP ranges for Digital Ocean services. |
| `linode` | IP ranges for Linode services. |
| `tor` | IP addresses of Tor exit nodes (disabled by default). |
| `asn` | IP ranges for specific ASNs (disabled by default). |
### Regenerating Pregenerated Results
To regenerate the pregenerated results, run the `main.go` file in the `ranges` directory:
```bash
cd ranges
go run main.go
```
This will fetch the latest IP ranges from all supported services and update the `generated.go` file in the `data` directory.
---
## Installation
To use the Fetchers Module in your Go project, install it using `go get`:
```bash
go get github.com/jasonlovesdoggo/caddy-defender/ranges/fetchers
```
---
## Usage
### Fetching IP Ranges
To fetch IP ranges for a specific service, create an instance of the corresponding fetcher and call the `FetchIPRanges` method:
```go
package main
import (
"fmt"
"github.com/jasonlovesdoggo/caddy-defender/ranges/fetchers"
)
func main() {
// Fetch global AWS IP ranges
awsFetcher := fetchers.AWSFetcher{}
ranges, err := awsFetcher.FetchIPRanges()
if err != nil {
fmt.Println("Error fetching AWS IP ranges:", err)
} else {
fmt.Println("AWS IP ranges:", ranges)
}
// Fetch GCP IP ranges
gcloudFetcher := fetchers.GCloudFetcher{}
ranges, err = gcloudFetcher.FetchIPRanges()
if err != nil {
fmt.Println("Error fetching GCP IP ranges:", err)
} else {
fmt.Println("GCP IP ranges:", ranges)
}
}
```
### Using in Caddy Defender
The Fetchers Module is integrated into the **Caddy Defender** middleware. To use it, configure your `Caddyfile` with the `defender` directive:
```caddyfile
localhost:8080 {
defender block {
ranges aws gcloud openai mistral
}
respond "Hello, world!"
}
```
This configuration blocks requests from IP ranges associated with AWS, GCP, OpenAI, and Mistral.
---
## Adding New Fetchers
To add a new fetcher for a service or provider:
1. **Create a New Fetcher**:
- Create a new file in the `fetchers` directory (e.g., `my_service.go`).
- Implement the `IPRangeFetcher` interface:
```go
package fetchers
import (
"fmt"
)
// MyServiceFetcher implements the IPRangeFetcher interface for MyService.
type MyServiceFetcher struct{}
func (f MyServiceFetcher) Name() string {
return "MyService"
}
func (f MyServiceFetcher) Description() string {
return "Fetches IP ranges for MyService."
}
func (f MyServiceFetcher) FetchIPRanges() ([]string, error) {
// Fetch IP ranges for MyService
return []string{"203.0.113.0/24", "198.51.100.0/24"}, nil
}
```
2. **Add the Fetcher to `main.go`**:
- Update the `fetchersList` in `main.go` to include your new fetcher:
```go
fetchersList := []fetchers.IPRangeFetcher{
fetchers.AWSFetcher{},
fetchers.GCloudFetcher{},
fetchers.MyServiceFetcher{}, // Add your new fetcher here
}
```
3. **Rebuild and Test**:
- Rebuild the project and test the new fetcher to ensure it works as expected.
---
## Example Fetchers
### AWS Fetcher
Fetches global IP ranges for AWS services:
```go
awsFetcher := fetchers.AWSFetcher{}
ranges, err := awsFetcher.FetchIPRanges()
```
### AWS Region Fetcher
Fetches IP ranges for a specific AWS region (e.g., `us-east-1`):
```go
awsRegionFetcher := fetchers.AWSRegionFetcher{Region: "us-east-1"}
ranges, err := awsRegionFetcher.FetchIPRanges()
```
### GCloud Fetcher
Fetches IP ranges for Google Cloud Platform (GCP):
```go
gcloudFetcher := fetchers.GCloudFetcher{}
ranges, err := gcloudFetcher.FetchIPRanges()
```
---
## Contributing
We welcome contributions! If you want to add new fetchers or improve existing ones, follow these steps:
1. **Fork the Repository**:
- Fork the [Caddy Defender repository](https://github.com/jasonlovesdoggo/caddy-defender).
2. **Create a New Branch**:
- Create a branch for your changes:
```bash
git checkout -b my-new-fetcher
```
3. **Make Your Changes**:
- Add your new fetcher or make improvements to existing code.
4. **Test Your Changes**:
- Run the tests and ensure everything works as expected.
5. **Submit a Pull Request**:
- Submit a pull request with a description of your changes.
---
## License
This project is licensed under the **MIT License**. See the [LICENSE](LICENSE) file for details.
---
## Acknowledgments
- **Caddy Server**: Built with ❤️ using [Caddy](https://caddyserver.com).
- **AWS and GCP**: For providing publicly accessible IP range data.
- **OpenAI and GitHub**: For their IP range documentation.
---
## Star History
[](https://star-history.com/#JasonLovesDoggo/caddy-defender&Date)