https://github.com/techmedaddy/torrentedge

TorrentEdge is a distributed BitTorrent client using Node.js, Nginx, and Kafka. It enables peer-to-peer file sharing, real-time updates, and scalable communication. Docker is used for deployment, and the frontend is built with React.
https://github.com/techmedaddy/torrentedge

docker docker-compose expressjs javascript kafka mongodb nginx nodejs

Last synced: 3 months ago
JSON representation

TorrentEdge is a distributed BitTorrent client using Node.js, Nginx, and Kafka. It enables peer-to-peer file sharing, real-time updates, and scalable communication. Docker is used for deployment, and the frontend is built with React.

• Host: GitHub
• URL: https://github.com/techmedaddy/torrentedge
• Owner: techmedaddy
• Created: 2024-09-18T06:09:16.000Z (almost 2 years ago)
• Default Branch: master
• Last Pushed: 2026-03-14T17:56:22.000Z (3 months ago)
• Last Synced: 2026-03-15T04:05:39.637Z (3 months ago)
• Topics: docker, docker-compose, expressjs, javascript, kafka, mongodb, nginx, nodejs
• Language: JavaScript
• Homepage: https://medium.com/@techmedaddy/torrentedge-a-distributed-bittorrent-client-acb4c06b9c27
• Size: 901 KB
• Stars: 4
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# TorrentEdge 🚀

A modern, production-ready BitTorrent client built from scratch in Node.js with real-time monitoring and event streaming capabilities.

[![Node.js](https://img.shields.io/badge/Node.js-18+-green.svg)](https://nodejs.org/)

[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

[![Docker](https://img.shields.io/badge/docker-ready-brightgreen.svg)](docker-compose.yml)



## ✨ Features

### Core BitTorrent Protocol

- ✅ **Full BitTorrent Protocol** implementation (BEP 3)

- ✅ **Magnet Link Support** with metadata exchange (BEP 9)

- ✅ **Extension Protocol** for advanced features (BEP 10)

- ✅ **Multi-file Torrents** with proper piece alignment

- ✅ **Piece Verification** using SHA1 hashing

- ✅ **Resume Downloads** with state persistence

- ✅ **Tracker Protocol** (HTTP/HTTPS/UDP)

### Advanced Features

- 🔥 **Multi-Torrent Queue Management** with priority-based scheduling

- 🔥 **Bandwidth Throttling** using token bucket algorithm

- 🔥 **Upload Management** with tit-for-tat choking algorithm

- 🔥 **Super-Seeding Mode** for efficient distribution

- 🔥 **Piece Selection Strategies**: Rarest-first, endgame mode

- 🔥 **Comprehensive Error Handling** with retry logic and peer banning

- 🔥 **Per-File Progress Tracking** for multi-file torrents

  

### Real-time & Scalable Architecture

- 📡 **Socket.IO** for live progress updates to web clients

- 📊 **Kafka Event Streaming** for analytics and monitoring

- 💾 **MongoDB** for torrent metadata and user management

- 🚀 **Redis Caching** for improved performance (optional)

- 🐳 **Docker & Docker Compose** for easy deployment

- 📈 **Health Monitoring** with tracker and peer health tracking



### Production Ready

- ⚡ **Concurrent Download Management** (default: 3 active, unlimited queued)

- 🔒 **Peer Ban System** with strike tracking

- 🔄 **Automatic Reconnection** with exponential backoff

- 📦 **State Backup & Recovery** with rotation

- 🎯 **Smart Piece Selection** with availability tracking

- 📝 **Comprehensive Logging** with categorized errors

## 🏗️ Architecture

```

┌─────────────────────────────────────────────────────────────┐

│                       TorrentEdge API                        │

│                    (Express + Socket.IO)                     │

└────────────┬───────────────────────────────────┬────────────┘

             │                                   │

    ┌────────▼─────────┐              ┌─────────▼──────────┐

    │  Torrent Engine  │              │   Kafka Producer   │

    │   (Core Logic)   │──────────────▶│   (Analytics)      │

    └────────┬─────────┘              └────────────────────┘

             │

    ┌────────▼──────────────────────────────────────────────┐

    │              Component Layer                          │

    ├───────────────┬───────────────┬──────────────────────┤

    │ Queue Manager │ State Manager │  Peer Manager        │

    │ Download Mgr  │ Upload Mgr    │  Throttler           │

    │ File Writer   │ Piece Manager │  Retry Manager       │

    └───────────────┴───────────────┴──────────────────────┘

                     │

            ┌────────▼─────────┐

            │  Peer Network    │

            │  (TCP + Tracker) │

            └──────────────────┘

```

### Key Components

- **Torrent Engine**: Core orchestration for all torrent operations

- **Queue Manager**: Priority-based multi-torrent scheduling

- **Download Manager**: Piece selection, endgame mode, retry logic

- **Upload Manager**: Tit-for-tat choking, super-seeding

- **Peer Manager**: Connection pooling, health tracking, reconnection

- **State Manager**: Persistence with backup rotation

- **Tracker Manager**: Multi-tracker failover with health states

  

## 🚀 Quick Start

### Prerequisites

- **Node.js** 18+ and npm

- **MongoDB** 4.4+ (for persistence)

- **Redis** 6+ (optional, for caching)

- **Apache Kafka** 2.8+ (optional, for analytics)

### Installation

```bash

# Clone the repository

git clone https://github.com/yourusername/torrentedge.git

cd torrentedge

# Install dependencies

npm install

# Configure environment

cp .env.example .env

# Edit .env with your configuration

# Start the server

npm run dev

```

### Docker Setup

```bash

# Start all services (MongoDB, Redis, Kafka, TorrentEdge)

docker-compose up -d

# View logs

docker-compose logs -f torrentedge

# Stop services

docker-compose down

```

The application will be available at:

- **API**: http://localhost:3000

- **WebSocket**: ws://localhost:3000

## 📡 API Reference

### Add Torrent from File

```bash

curl -X POST http://localhost:3000/api/torrents \

  -H "Authorization: Bearer YOUR_TOKEN" \

  -F "file=@ubuntu.torrent"

```

### Add Magnet Link

```bash

curl -X POST http://localhost:3000/api/torrents \

  -H "Authorization: Bearer YOUR_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{

    "magnetURI": "magnet:?xt=urn:btih:HASH&dn=Name&tr=tracker_url"

  }'

```

### Get All Torrents

```bash

curl http://localhost:3000/api/torrents \

  -H "Authorization: Bearer YOUR_TOKEN"

```

### Get Torrent Status

```bash

curl http://localhost:3000/api/torrents/:infoHash \

  -H "Authorization: Bearer YOUR_TOKEN"

```

### Pause/Resume Torrent

```bash

# Pause

curl -X POST http://localhost:3000/api/torrents/:infoHash/pause \

  -H "Authorization: Bearer YOUR_TOKEN"

# Resume

curl -X POST http://localhost:3000/api/torrents/:infoHash/resume \

  -H "Authorization: Bearer YOUR_TOKEN"

```

### Remove Torrent

```bash

curl -X DELETE http://localhost:3000/api/torrents/:infoHash \

  -H "Authorization: Bearer YOUR_TOKEN"

```

### Set Priority

```bash

curl -X PUT http://localhost:3000/api/torrents/:infoHash/priority \

  -H "Authorization: Bearer YOUR_TOKEN" \

  -H "Content-Type: application/json" \

  -d '{"priority": "high"}'

```

### Get Statistics

```bash

curl http://localhost:3000/api/statistics \

  -H "Authorization: Bearer YOUR_TOKEN"

```

## 🔌 WebSocket Events

### Client → Server

```javascript

const socket = io('http://localhost:3000');

// Authenticate

socket.emit('authenticate', { token: 'YOUR_TOKEN' });

// Subscribe to torrent updates

socket.emit('subscribe:torrent', { infoHash: 'HASH' });

```

### Server → Client

```javascript

// Torrent added

socket.on('torrent:added', (data) => {

  console.log('New torrent:', data.infoHash);

});

// Progress update

socket.on('torrent:progress', (data) => {

  console.log(`${data.name}: ${data.percentage.toFixed(2)}%`);

  console.log(`Speed: ${formatBytes(data.downloadSpeed)}/s`);

  console.log(`Peers: ${data.peers.connected}/${data.peers.total}`);

});

// Piece completed

socket.on('torrent:piece', (data) => {

  console.log(`Piece ${data.index} completed`);

});

// Download complete

socket.on('torrent:complete', (data) => {

  console.log(`Download complete: ${data.name}`);

});

// Seeding started

socket.on('torrent:seeding', (data) => {

  console.log(`Seeding: ${data.name}`);

  console.log(`Ratio: ${data.ratio.toFixed(2)}`);

});

// Error occurred

socket.on('torrent:error', (data) => {

  console.error(`Error: ${data.message}`);

});

// Queue updated

socket.on('queue:updated', (data) => {

  console.log(`Active: ${data.stats.activeCount}`);

  console.log(`Queued: ${data.stats.queuedCount}`);

});

```



## ⚙️ Configuration

### Environment Variables

| Variable | Description | Default |

|----------|-------------|---------|

| `PORT` | Server port | `3000` |

| `MONGODB_URI` | MongoDB connection string | `mongodb://localhost:27017/torrentedge` |

| `REDIS_HOST` | Redis host | `localhost` |

| `REDIS_PORT` | Redis port | `6379` |

| `KAFKA_BROKERS` | Kafka broker list | `localhost:9092` |

| `JWT_SECRET` | JWT signing secret | `your-secret-key` |

| `DOWNLOAD_PATH` | Default download directory | `./downloads` |

| `MAX_CONCURRENT_TORRENTS` | Max active downloads | `3` |

| `MAX_PEER_CONNECTIONS` | Max peers per torrent | `50` |

| `UPLOAD_LIMIT` | Global upload limit (bytes/s) | `0` (unlimited) |

| `DOWNLOAD_LIMIT` | Global download limit (bytes/s) | `0` (unlimited) |

### Torrent Engine Options

```javascript

const engine = new TorrentEngine({

  downloadPath: './downloads',

  port: 6881,

  maxConcurrent: 3,

  maxConnections: 50,

  uploadLimit: 1024 * 1024,      // 1 MB/s

  downloadLimit: 5 * 1024 * 1024, // 5 MB/s

  seedRatioLimit: 2.0,            // Stop seeding at 2.0 ratio

  seedTimeLimit: 0,               // Seed forever (minutes)

  autoResume: true,               // Resume on restart

  verifyOnResume: false           // Skip verification on resume

});

```

## 🧪 Testing

```bash

# Run all tests

npm test

# Run unit tests

npm run test:unit

# Run integration tests

npm run test:integration

# Run with coverage

npm run test:coverage

```

## 📊 Performance

- ✅ Handles **100+ concurrent torrents**

- ✅ Supports **1000+ peer connections**

- ✅ Tested with torrents up to **50GB+**

- ✅ Memory efficient: **~50MB base + ~10MB per active torrent**

- ✅ CPU efficient: **~5% on average, ~20% during verification**

### Benchmarks

| Operation | Performance |

|-----------|-------------|

| Piece verification | ~150 MB/s |

| File writes | ~200 MB/s |

| Peer handshakes | ~100/s |

| Socket.IO events | ~10k/s |

| Kafka messages | ~50k/s |

## 🛠️ Tech Stack

### Core

- **Runtime**: Node.js 18+

- **Protocol**: BitTorrent (BEP 3, 9, 10)

- **Network**: TCP/IP, UDP

- **Cryptography**: SHA1, crypto-js

### Backend

- **Framework**: Express.js

- **Real-time**: Socket.IO

- **Database**: MongoDB + Mongoose

- **Cache**: Redis

- **Message Queue**: Apache Kafka + KafkaJS

### DevOps

- **Containerization**: Docker

- **Orchestration**: Docker Compose

- **Reverse Proxy**: Nginx

## 📁 Project Structure

```

TorrentEdge/

├── src/

│   ├── server/

│   │   ├── torrentEngine/          # Core BitTorrent implementation

│   │   │   ├── torrent.js          # Torrent state machine

│   │   │   ├── engine.js           # Main engine orchestrator

│   │   │   ├── queueManager.js     # Multi-torrent queue

│   │   │   ├── stateManager.js     # Persistence layer

│   │   │   ├── peerManager.js      # Peer connection pool

│   │   │   ├── peerConnection.js   # Individual peer protocol

│   │   │   ├── downloadManager.js  # Download coordination

│   │   │   ├── uploadManager.js    # Seeding and uploads

│   │   │   ├── fileWriter.js       # Disk I/O

│   │   │   ├── pieceManager.js     # Piece verification

│   │   │   ├── throttler.js        # Bandwidth limiting

│   │   │   ├── retryManager.js     # Error handling

│   │   │   └── tracker.js          # Tracker protocol

│   │   ├── controllers/            # API controllers

│   │   ├── models/                 # MongoDB models

│   │   ├── routes/                 # Express routes

│   │   ├── middleware/             # Auth, validation

│   │   ├── kafka/                  # Kafka producers/consumers

│   │   └── server.js               # Main entry point

│   └── api/                        # API controllers

├── tests/                          # Test suites

├── nginx/                          # Nginx configuration

├── docker-compose.yml              # Docker setup

├── package.json

└── README.md

```

## 🔍 Key Algorithms

### Piece Selection

1. **Rarest First**: Download rarest pieces first to improve swarm health

2. **Random First**: First 4 pieces selected randomly for quick startup

3. **Endgame Mode**: Request last pieces from multiple peers (at 95% completion)

### Choking Algorithm (Tit-for-Tat)

- Unchoke top 4 uploaders every 10 seconds

- Optimistic unchoke rotates every 30 seconds

- Rewards peers who upload to us

- Prevents freeloading

### Token Bucket Throttling

- Bucket fills at configured rate (bytes/second)

- Each transfer consumes tokens

- Burst support up to 1 second worth of tokens

- Separate buckets for upload/download

### Error Handling

- **Retry Manager**: Exponential backoff with jitter

- **Peer Banning**: Strike system (3 strikes → temporary ban)

- **Tracker Failover**: Multi-tracker with health states

- **Automatic Recovery**: Reconnection with backoff

## 🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository

2. Create a feature branch (`git checkout -b feature/amazing-feature`)

3. Commit your changes (`git commit -m 'Add amazing feature'`)

4. Push to the branch (`git push origin feature/amazing-feature`)

5. Open a Pull Request

---

For detailed architecture documentation, see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).