{"id":41060863,"url":"https://github.com/kstonekuan/digital-twin-proxy","last_synced_at":"2026-01-22T11:43:52.731Z","repository":{"id":305924203,"uuid":"1024377474","full_name":"kstonekuan/digital-twin-proxy","owner":"kstonekuan","description":"A forward proxy to turn network traffic into personal memory for AI agents","archived":false,"fork":false,"pushed_at":"2025-12-22T04:43:49.000Z","size":57,"stargazers_count":26,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-12-23T14:58:19.502Z","etag":null,"topics":["context","memory","proxy","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kstonekuan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-22T15:49:23.000Z","updated_at":"2025-12-22T04:43:45.000Z","dependencies_parsed_at":"2025-08-25T10:21:53.844Z","dependency_job_id":"1e93823e-421e-4058-884e-516f745cbf67","html_url":"https://github.com/kstonekuan/digital-twin-proxy","commit_stats":null,"previous_names":["kstonekuan/ai-proxy","kstonekuan/digital-twin-proxy"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kstonekuan/digital-twin-proxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kstonekuan%2Fdigital-twin-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kstonekuan%2Fdigital-twin-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kstonekuan%2Fdigital-twin-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kstonekuan%2Fdigital-twin-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kstonekuan","download_url":"https://codeload.github.com/kstonekuan/digital-twin-proxy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kstonekuan%2Fdigital-twin-proxy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28662321,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T01:17:37.254Z","status":"online","status_checked_at":"2026-01-22T02:00:07.137Z","response_time":144,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["context","memory","proxy","rust"],"created_at":"2026-01-22T11:43:51.692Z","updated_at":"2026-01-22T11:43:52.725Z","avatar_url":"https://github.com/kstonekuan.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Digital Twin Proxy\n\nTurn web browsing into personal memory for AI agents.\n\nDigital Twin Proxy logs web traffic and uses a local or remote large language model (LLM) to generate an analysis of your browsing patterns. It's designed for developers, researchers, and anyone interested in understanding their online activity through the lens of AI.\n\n## Features\n\n- **HTTP/S Traffic Logging**: Captures all web requests made through the proxy.\n- **Agentic, Content-Aware Analysis**: Uses an LLM to not only analyze traffic patterns but also to decide which pages to fetch and analyze in more depth.\n- **Flexible Operation Modes**: Run in the background, log traffic continuously, or perform one-off analysis.\n- **Customizable**: Easily change the AI model, analysis interval, and other settings.\n- **OpenAI-Compatible**: Works with any OpenAI-compatible API, including local providers like Ollama and LM Studio, as well as remote services like OpenAI, Groq, etc.\n- **Privacy-Focused**: By using a local LLM, you can ensure that your browsing history remains private and is not sent to any third-party service.\n\n## A Note on Privacy\n\nThis application is designed to work with your personal browsing history. As such, we strongly recommend using a local large language model (LLM) provider like [Ollama](https://ollama.com/), [LM Studio](https://lmstudio.ai/), [vLLM](https://github.com/vllm-project/vllm), or [TGI](https://github.com/huggingface/text-generation-inference). By running the LLM on your own machine, you can ensure that your browsing data remains private and is never sent to a third-party service.\n\nWhile you can use any OpenAI-compatible API, please be aware of the privacy implications of sending your browsing data to a remote service.\n\n## Context for Agentic Applications\n\nThe primary output of Digital Twin Proxy is a structured log of your web traffic, along with AI-generated analysis. This data can serve as a powerful source of real-time context for other agentic applications.\n\nBy providing an analysis of recent browsing history, you can engineer a more informed context window for other AI agents, enabling them to:\n-   **Personalize responses**: An agent can tailor its behavior based on your current tasks and interests.\n-   **Anticipate needs**: An agent can proactively offer assistance based on the websites you are visiting.\n-   **Improve tool usage**: An agent can better understand the context of your work and select the right tools for the job.\n\nThis process of \"context engineering\" allows you to create a more powerful and personalized AI experience.\n\n## Planned Features\n\n### MCP Server\n\nWe will soon expose the context from your digital twin as an MCP server to support AI agents.\n\n### In-Browser Context Injection\n\nTo create a more interactive and personalized web experience, we are developing a feature to inject real-time context directly into your browser for any agentic AI app (ChatGPT, Perplexity, etc.) to access your digital twin.\n\n## How It Works\n\nThe proxy operates by routing your browser's traffic through a local Squid instance. The application then uses an AI agent to analyze the traffic and decide which pages to fetch and analyze further.\n\n```\nBrowser → Squid Proxy (port 8888) → Internet\n              ↓\n         Access Logs\n              ↓\nDigital Twin Proxy App → OpenAI-compatible API → Decides to fetch content → Fetches Page Content → OpenAI-compatible API → Analysis\n```\n\n1.  **Traffic Interception**: Your browser is configured to send all HTTP and HTTPS requests to the Digital Twin Proxy listener on port 8888.\n2.  **Logging**: The proxy, powered by Squid, logs every request's URL and host.\n3.  **Agentic Analysis**: The `digital-twin-proxy` application sends the list of visited URLs to an LLM via an OpenAI-compatible API. The LLM then acts as an agent, deciding which URLs are interesting enough to warrant a deeper look.\n4.  **Content Fetching**: If the agent decides to investigate a URL, it uses a tool to fetch the content of that page.\n5.  **In-Depth Analysis**: The agent then analyzes the content of the fetched page to generate a more in-depth and meaningful summary of your browsing patterns.\n\n## Getting Started\n\n### Prerequisites\n\n- [**Rust**](https://www.rust-lang.org/tools/install) toolchain\n- [**Squid**](https://www.squid-cache.org/) proxy\n- LLM access via **OpenAI-compatible API**. We recommend local service like [Ollama](https://ollama.com/), [LM Studio](https://lmstudio.ai/), [vLLM](https://github.com/vllm-project/vllm), or [TGI](https://github.com/huggingface/text-generation-inference).\n\n### Installation\n\nClone the repository and build the project:\n\n```bash\ngit clone https://github.com/kstonekuan/digital-twin-proxy.git\ncd digital-twin-proxy\ncargo build --release\n```\n\nThe binary will be located at `target/release/digital-twin-proxy`.\n\n### Configuration\n\n#### 1. Configure Your Browser\n\nSet your browser's HTTP and HTTPS proxy to `127.0.0.1:8888`.\n\n#### 2. Configure the API Endpoint\n\nYou can configure the application using three methods (in order of priority):\n\n1. **Command-line flags** (highest priority)\n2. **Environment variables**\n3. **`.env` file** (lowest priority)\n\n##### Available Configuration Options\n\n| Option             | Environment Variable | CLI Flag      | Default       | Description                             |\n| ------------------ | -------------------- | ------------- | ------------- | --------------------------------------- |\n| API Base URL       | `API_BASE`           | `--api-base`  | (required)    | OpenAI-compatible API endpoint          |\n| API Key            | `API_KEY`            | `--api-key`   | (optional)    | API key for the service                 |\n| Model              | `MODEL`              | `--model`     | `gpt-oss:20b` | LLM model to use                        |\n| Ambient Interval   | `AMBIENT_INTERVAL`   | `--interval`  | `30`          | Seconds between analyses (ambient mode) |\n| Max Analysis Items | `MAX_ANALYSIS_ITEMS` | `--max-items` | `500`         | Maximum URLs to analyze per batch       |\n\n##### Configuration Methods\n\n**Method 1: Using a `.env` file:**\n\nCopy the example configuration and edit it:\n\n```bash\ncp .env.example .env\n# Edit .env with your values\n```\n\nExample `.env` file:\n```\nAPI_BASE=http://localhost:11434/v1\nAPI_KEY=your-api-key-if-needed\nMODEL=gpt-oss:20b\nAMBIENT_INTERVAL=60\nMAX_ANALYSIS_ITEMS=1000\n```\n\n**Method 2: Using environment variables:**\n\n```bash\nexport API_BASE=http://localhost:11434/v1\nexport MODEL=gpt-oss:20b\n./digital-twin-proxy analyze --since 1h\n```\n\n**Method 3: Using command-line flags:**\n\n```bash\n./digital-twin-proxy analyze \\\n  --since 1h \\\n  --api-base http://localhost:11434/v1 \\\n  --model gpt-oss:20b \\\n  --max-items 1000\n```\n\n#### 3. Verify\n\nStart the proxy in logging mode and visit a website.\n\n```bash\n# Terminal 1: Start the proxy\n./target/release/digital-twin-proxy log\n\n# Terminal 2: Tail the logs\ntail -f ~/.local/share/ai-proxy/log.ndjson\n```\n\nYou should see JSON objects representing your web traffic.\n\n## Usage\n\nDigital Twin Proxy has three main commands:\n\n- `log`: Start the proxy and only log traffic.\n- `analyze`: Perform a one-shot, content-aware analysis of traffic logged since a given duration.\n- `ambient`: Run the proxy and periodically perform content-aware analysis of traffic in the background.\n\n**Examples:**\n\n```bash\n# Log traffic without analysis\n./digital-twin-proxy log\n\n# Analyze traffic from the last hour with a local Ollama model\n./digital-twin-proxy analyze --since 1h --model gpt-oss:20b --api-base http://localhost:11434/v1\n\n# Run in ambient mode, analyzing every 5 minutes with the OpenAI API\n./digital-twin-proxy ambient --interval 300 --model gpt-5 --api-base https://api.openai.com/v1 --api-key $OPENAI_API_KEY\n```\n\n## WSL (Windows Subsystem for Linux) Setup\n\nIf you're using WSL, there are additional networking considerations:\n\n1. **Install Squid in WSL:**\n   ```bash\n   sudo apt update\n   sudo apt install squid\n   ```\n\n2. **Configure WSL networking:**\n   - The proxy will run on `127.0.0.1:8888` within WSL\n   - From Windows, you'll need to access it via the WSL IP address\n   - Find your WSL IP: `ip addr show eth0 | grep inet`\n\n3. **Configure Windows browser:**\n   - Set proxy to `\u003cWSL_IP\u003e:8888` (e.g., `172.20.240.2:8888`)\n   - Or use `127.0.0.1:8888` if you set up port forwarding\n\n4. **Optional - Set up port forwarding (Windows PowerShell as Administrator):**\n   ```powershell\n   netsh interface portproxy add v4tov4 listenport=8888 listenaddress=127.0.0.1 connectport=8888 connectaddress=\u003cWSL_IP\u003e\n   ```\n\n5. **Configure WSL firewall (if needed):**\n   ```bash\n   sudo ufw allow 8888\n   ```\n\n## Development\n\nThis project uses `rustfmt` for formatting and `clippy` for linting.\n\n```bash\n# Format code\ncargo fmt\n\n# Run linter\ncargo clippy --all-targets --all-features\n\n# Build and run tests\ncargo build\ncargo test\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a pull request or open an issue.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkstonekuan%2Fdigital-twin-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkstonekuan%2Fdigital-twin-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkstonekuan%2Fdigital-twin-proxy/lists"}