Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/benpate/digital-dome
Silicon Dome is a shield against AI and bot scanners
https://github.com/benpate/digital-dome
Last synced: about 1 month ago
JSON representation
Silicon Dome is a shield against AI and bot scanners
- Host: GitHub
- URL: https://github.com/benpate/digital-dome
- Owner: benpate
- License: apache-2.0
- Created: 2024-12-02T00:16:21.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2024-12-03T00:10:54.000Z (about 2 months ago)
- Last Synced: 2024-12-03T00:29:43.344Z (about 2 months ago)
- Language: Go
- Size: 26.4 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Digital Dome ✨
[](http://pkg.go.dev/github.com/benpate/digital-dome)
[](https://github.com/benpate/digital-dome/releases)
[](https://github.com/benpate/digital-dome/actions/workflows/go.yml)
[](https://goreportcard.com/report/github.com/benpate/digital-dome)
[](https://codecov.io/gh/benpate/digital-dome)
## Opinionated Shield from AIs and Malicious Scanners.
Digital dome is a fast, minimal web application firewall that uses request information to protect a site against AI scanners. It hosts several configurable rules, along with sensible (if aggressive) defaults.
## Quickstart
```golang
import github.com/benpate/digital-dome/dome
import github.com/benpate/digital-dome/dome4echo
domeConfig := dome.New() // Create a new digital dome (using sensible defaults)
middleware := dome4echo.New(domeConfig) // Create echo middleware
e.Pre(middleware) // Use the middleware
// easy peasy.
```
## Block AI Scrapers
I've manually collected this list of AI scrapers that Digital Dome uses to protect your website. Any requests that contain any of the values below will be blocked before they reach your application.
In addition, since every legitimate browser includes a `User-Agent` value, requests with an empty value are also automatically blocked.
```
Amazonbot
anthropic-ai
AdsBot-Google
Applebot
Applebot-Extended
AwarioRssBot
AwarioSmartBot
Bytespider
CCBot
ChatGPT
ChatGPT-User
Claude
ClaudeBot
Claude-Web
cohere-ai
DataForSeoBot
Diffbot
FacebookBot
FacebookExternalHit
FriendlyCrawler
Google-CloudVertexBot
Google-Extended
GPTBot
ImagesiftBot
magpie-crawler
Meta-ExternalAgent
meta-externalagent
NewsNow
news-please
OAI-SearchBot
omgili
omgilibot
peer39_crawler
PerplexityBot
PetalBot
Quora-Bot
Scrapy
TurnitinBot
Twitterbot
YaK
Yandex
YouBot
```
## Block Commonly Scanned Paths
Like the list of User-Agent strings above, Digital Dome also maintains a list of URL paths that are commonly scanned for vulnerabilities. For instance, default WordPress directories are commonly scanned as a precursor to hacking a server. Since Digital Dome is build specifically for Go applications, there's no problem blocking every directory related to WordPress.
## Block Malicious IP Addresses
Digital Dome tracks errors generated by your application and returned through the middleware.
If a specific IP address generates too many errors in a designated timespan, future requests from that IP will be blocked before they reach your application.
The length of time an IP is blocked grows exponentially with the number of bad requests they make. So, honest mistakes will heal quickly and automatically, and script-kiddie scanners will ban themselves into oblivion.
By default, Digital Dome counts all `StatusForbidden` responses towards the quota for any IP address, and begins blocking all traffic after 5 forbidden requests within one minute. You can calibrate the kinds of responses that trigger this behavior using the `BlockStatusCodes()` option.
```golang
domeConfig := dome.New( // Create the digital dome shield
dome.BlockStatusCodes(404) // Choose status codes to trigger blocking behavior
```
## Configuration Options
Digital Dome uses optional functional parameters to configure its behavior. You can apply these at startup when you create the dome, or afterward when your app is running.
```golang
dd := dome.New( // Apply options at creation time
BlockKnownAIBots(), // Block AI bots only
BlockKnownPaths(), // Block known paths (DEFAULT)
BlockCache(2048), // Expand the size of the blocked IP cache
)
dd.With(BlockStatusCodes(404)) // Or apply other options later
```
| Option | Description |
|--------|-------------|
| **Block User Agents** | |
| `BlockUserAgents(strings...)` | Digital Dome can block requests based on any number of provided `User-Agent` strings. It uses an efficient [Aho-Corasick](https://github.com/cloudflare/ahocorasick) string matching algorithm from CloudFlare to perform this operation quickly. |
| `BlockKnownAIBots()` | Digital Dome maintains a [list of known AI bots](https://github.com/benpate/digital-dome/blob/main/dome/constant_userAgents.go#L59) that it can compare against each request's `User-Agent` |
| `BlockKnownBadBots()` | (DEFAULT) Digital Dome maintains a [list of known bad actors](https://github.com/benpate/digital-dome/blob/main/dome/constant_userAgents.go#L11) that it can compare against each request's `User-Agent`. This includes all of the AI bots listed above, plus several hundred more non-search-engine user agents that are used for scraping your website. |
| **Block Paths** | |
| `BlockPaths(strings...)` | Digital Dome can block requests based on any number of provided path names. As with `User-Agent` blocking, it uses an efficient Aho-Corasick string matching algorithm from CloudFlare to perform this operation quickly. |
| `BlockKnownPaths()` | (DEFAULT) Digital Dome maintains a [list of known paths]() that are frequently scanned by scammers and are blocked by default. |
| **Log Errors** | |
| `LogDatabase(data.Collection)` | Digital Dome can log failed requests to a database if a data.Collection is provided to this Option (see Storage below) |
| `LogStatusCodes(ints...)` | Customize the status codes are logged using this Option. Default values are: `StatusBadRequest`, `StatusNotFound`, and `StatusForbidden` |
| **Block Malicious Requests** | |
| `BlockStatusCodes(ints...)` | Digital Dome can track when requests trigger specific errors (for example, `StatusForbidden`) and block all requests from that IP address. See `Blocking` below for details. |
| `BlockCache(capacity)` | This option sets the capacity of the blocked IP address cache. Default is 1024 IP addresses. |
## Custom Router Middleware
Digital Dome is build to work with any Go HTTP Router library or framework. There is currently one adapter, made for [labstack echo](https://echo.labstack.com). Middleware adapters are very easy to make, so if your router is not listed below, please [file an issue](https://github.com/benpate/digital-dome/issues) to get one made.
## Custom Database Storage
Digital Dome uses a [storage adapter](github.com/benpate/data) to write log files to a database. Currently, I have only written an adapter for MongoDB, but adapters can be written for any database, so please [file an issue](https://github.com/benpate/digital-dome/issues) to make one for your chosen database.
To begin writing log files, simply pass a `data.Collection` into the `LogDatabase` option, and Digital Dome will use it whenever it needs to log a request.
```golang
db, err := mongodb.Connect(...) // Connect to mongodb
collection := mongodb. // Wrap mongo with a data.Collection adapter
NewSession(db).
Collection("SiliconDome_LogFiles")
domeConfig := dome.New( // Create the digital dome shield
dome.LogDatabase(collection), // Use this database collection to log errors
dome.LogStatusCodes(404) // Choose status codes to log
)
```
## Banner Image Used Without Permission
**Image Credits. None.** The banner image was an AI-generated using https://www.freepik.com. AI images cannot be copyrighted or owned by anyone. Savor the delicious irony.
## No Warranty
Digital Dome is a solid defense against many AI scanners and malicious bots. But it is not a complete Web Application Firewall, and it comes with NO WARRANTIES or GUARANTEES against any particular online threats. Be mindful.
## Pull Requests Welcome
The battle agains rampant AIs is ongoing, and this module will require constant upkeep to stay current. I will happily accept any and all experience reports, use cases, and contributions that will make Digital Dome better. If you have an idea for making Digital Dome better, [file an issue](https://github.com/benpate/digital-dome/issues) or [send in a pull request](https://github.com/benpate/digital-dome/pulls). We're all in this together! ✨