https://github.com/dense-analysis/dank
Dense Analysis Network Knowledge
https://github.com/dense-analysis/dank
ai clickhouse knowledge-graph python redis scraping scraping-websites
Last synced: 3 months ago
JSON representation
Dense Analysis Network Knowledge
- Host: GitHub
- URL: https://github.com/dense-analysis/dank
- Owner: dense-analysis
- Created: 2025-04-01T17:13:50.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-26T17:50:06.000Z (about 1 year ago)
- Last Synced: 2025-08-19T17:09:59.189Z (11 months ago)
- Topics: ai, clickhouse, knowledge-graph, python, redis, scraping, scraping-websites
- Language: Python
- Homepage:
- Size: 14.6 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# DANK - Dense Analysis Network Knowledge
DANK is a Dense Analysis project focused on collecting and analyzing live data
from the public Internet. It uses API access, web scraping, RSS feeds, and
semantic indexing tools to ingest external content in real time. It applies
sentiment analysis, semantic clustering, and AI models to build structured
insights about the world, including trends, public perception, and evolving
narratives. The goal is to automate contextual understanding and surface
relevant knowledge as it emerges.
## Requirements
- Python 3.13
- uv
- ClickHouse (local server)
## ClickHouse setup
1. Install ClickHouse: https://clickhouse.com/docs/en/install
2. Start the ClickHouse server (systemd or `clickhouse server`).
3. Create the schema:
```
~/clickhouse/clickhouse client --multiquery < schema.sql
```
The schema uses the `dank` database by default. Adjust `config.toml` if you
need a different database name.
## Configuration
Configuration lives in `config.toml` and should not be committed. Example:
```toml
sources = [
{ domain = "x.com", accounts = ["example"] },
"blog.codinghorror.com",
]
[clickhouse]
host = "localhost"
port = 8123
database = "dank"
username = "default"
password = ""
secure = false
use_http = true
[x]
username = "your-x-username"
password = "your-x-password"
max_posts = 200
max_scrolls = 20
scroll_pause_seconds = 1.5
[storage]
data_dir = "data"
max_asset_bytes = 10485760
[browser]
# Optional: full path or command name for a Chromium-based browser.
executable_path = "thorium-browser"
# Optional: extra time to wait for the browser to start.
connection_timeout = 1.0
# Optional: connection retry count for slow browser startups.
connection_max_tries = 30
[email]
# Optional: IMAP settings for OTP codes.
host = "imap.example.com"
username = "you@example.com"
password = "your-imap-password"
port = 993
[logging]
# Optional: file path for scrape/process logs.
file = "dank.log"
# Optional: logging level (DEBUG, INFO, WARNING, ERROR).
level = "INFO"
```
`sources` controls which domains to scrape and process. Each entry can provide
accounts for account-based sources like `x.com`.
If any particular domain lacks a specific configuration, the root of the
domain will be scraped to discover RSS feeds to read from.
`browser.executable_path` sets the browser binary to launch. If unset, DANK
will try common Chromium locations.
`storage.max_asset_bytes` caps asset downloads (bytes). Larger assets are
skipped but still recorded.
When X prompts for a one-time code, DANK will poll the IMAP inbox for messages
from `x.com` that arrived after the login attempt and extract the confirmation
code.
If the browser takes longer to start, increase
`browser.connection_timeout` or `browser.connection_max_tries`.
`logging.file` controls where scrape/process logs are written. Relative paths
are resolved from the current working directory.
## Usage
Dank offers the following commands.
* `uv run scrape` -- Scrape the web for data
* Pass `--domains` to scrape only matching domains from `sources`,
for example `--domains '^x\\.com$'`.
* `uv run process` -- Process previously scraped data
* The `--age` argument can be given a duration to process, for example
`6hours` or `2days`.
* `uv run clickhouse-query` -- Run queries on the database
* You can only run `SELECT`, `SHOW`, or `EXPLAIN` queries through this tool
* Query results are well formatted and easy to read
* Query results are truncated unless you pass `--full`
* `uv run embed-text "your text"` -- Print an embedding vector
* Output is a JSON `list[float]` for easy copy/paste into other tools.
* `uv run download-embedding-model` -- Download and cache embeddings model
* Pass `--model` to choose another Hugging Face model id.
* `uv run web` -- Start a simple web server to view content.
* Pass `--no-reload` to disable hot code reloading.
* Supports search filters for domain/account and a days-back slider.
## Testing
* `uv run pytest` -- Run default test suite.
* `uv run pytest -m embeddings -s` -- Run real-model embedding checks.
* These tests are skipped by default and require the model cache.
* Includes per-case similarity and margin output for each model.