https://github.com/pkeffect/numbers

Math things and stuff
https://github.com/pkeffect/numbers

constants docker docker-compose eluer fastpi math phi pi python sqlite

Last synced: about 1 month ago
JSON representation

Math things and stuff

Host: GitHub
URL: https://github.com/pkeffect/numbers
Owner: pkeffect
License: mit
Created: 2025-09-28T23:15:20.000Z (9 months ago)
Default Branch: main
Last Pushed: 2025-09-30T21:31:18.000Z (9 months ago)
Last Synced: 2025-09-30T23:25:21.315Z (9 months ago)
Topics: constants, docker, docker-compose, eluer, fastpi, math, phi, pi, python, sqlite
Language: Python
Homepage: https://internode.info/
Size: 97.7 KB
Stars: 0
Watchers: 0
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.md

Awesome Lists containing this project

README

# 🔢 Math Constants Storage System

> **⚠️ IMPORTANT: DATA FILES NOT INCLUDED**
>
> This repository contains the API and storage infrastructure **only**. See [Data Sources](#-data-sources) section for obtaining data files.

High-performance, triple-redundancy storage system for accessing billions of digits of mathematical constants (π, φ, e, √2, and more) with FastAPI backend, Redis caching, WebSocket streaming, dedicated endpoints per constant, and accuracy-first design.

## ✨ Features

- **12 Mathematical Constants**: Pi, Euler's number, Golden ratio, Square roots, Logarithms, and more
- **Dedicated Endpoints**: Each constant has its own clean API namespace (`/pi/*`, `/e/*`, etc.)
- **Triple Redundancy**: Original file + SQLite chunks + Binary cache
- **Redis Caching**: Lightning-fast access with intelligent cache invalidation
- **WebSocket Streaming**: Real-time digit streaming with chunked delivery
- **Smart Cache Management**: Automatically skips already-built caches
- **Accuracy First**: Automatic verification and corruption detection
- **High Performance**: Optimized for random access to billions of digits
- **Modular Architecture**: Clean separation with 16 focused router files
- **Hot Reloading**: Development-friendly Docker setup
- **Backward Compatible**: Legacy parameterized endpoints still work

## 🚀 Quick Start

```bash
# Clone the repository
git clone https://github.com/pkeffect/numbers
cd numbers

# Copy environment configuration
cp .env.example .env

# Place your math constant files in data/ directory
# Required filenames: pi_digits.txt, e_digits.txt, phi_digits.txt, etc.

# Start the development environment (includes Redis)
docker-compose up --build

# Build caches (after first startup)
curl -X POST http://localhost:8000/admin/build-all-caches
```

## 🆕 New Features: Redis & WebSocket

### Redis Caching

**Benefits:**
- ⚡ 10-100x faster response times for cached queries
- 📊 Automatic cache hit/miss tracking
- 🔄 Smart cache invalidation on data updates
- 💾 Configurable TTL and memory limits

**Cached Operations:**
- Digit retrieval
- Search results
- Statistical analysis

**Cache Control:**
```bash
# Use cache (default)
GET /pi/digits?start=0&length=100&use_cache=true

# Bypass cache
GET /pi/digits?start=0&length=100&use_cache=false

# Clear cache for a constant
DELETE /pi/cache

# Get cache statistics
GET /health # includes cache stats
```

### WebSocket Streaming

**Real-time Features:**
- 📡 Live digit streaming
- 🔔 Cache build notifications
- 📊 Live status updates
- 🎯 Per-constant subscriptions

**WebSocket Endpoints:**
```javascript
// General connection (system-wide events)
ws://localhost:8000/ws/connect

// Constant-specific connection
ws://localhost:8000/ws/constant/pi

// Simplified streaming
ws://localhost:8000/ws/stream/pi?start=0&length=1000
```

**Example Usage:**
```javascript
const ws = new WebSocket('ws://localhost:8000/ws/constant/pi');

ws.onopen = () => {
// Stream 1000 digits in chunks of 100
ws.send(JSON.stringify({
command: 'stream_digits',
start: 0,
length: 1000,
chunk_size: 100
}));
};

ws.onmessage = (event) => {
const data = JSON.parse(event.data);

if (data.type === 'stream_chunk') {
console.log(`Chunk ${data.chunk_number}: ${data.digits}`);
console.log(`Remaining: ${data.remaining} digits`);
}
};
```

**Test WebSocket:**
Open `http://localhost:3000/websocket_test.html` in your browser for an interactive test interface.

## 🔧 API Endpoints

### **General Endpoints**

```bash
GET / # API root with system info
GET /health # System health check (includes Redis stats)
GET /constants # List all constants with status
```

### **Per-Constant Endpoints** (Pattern for all 12 constants)

Each constant has 8 dedicated endpoints (7 original + 1 new):

```bash
GET /{constant}/status # Status and cache information
GET /{constant}/digits # Retrieve digits (Redis cached)
GET /{constant}/search # Search sequences (Redis cached)
GET /{constant}/stats # Statistical analysis (Redis cached)
GET /{constant}/random # Get random digits
POST /{constant}/build-cache # Build SQLite and binary cache
POST /{constant}/verify # Verify data integrity
DELETE /{constant}/cache # Clear Redis cache (NEW!)
```

### **WebSocket Endpoints**

```bash
WS /ws/connect # General system connection
WS /ws/constant/{constant_id} # Constant-specific connection
WS /ws/stream/{constant_id} # Simplified streaming
```

### **Admin Endpoints**

```bash
POST /admin/build-all-caches # Build caches for all constants
GET /admin/status # Administrative status
```

## 🐳 Docker Services

The system now includes Redis:

```bash
# Start all services (API + Redis)
docker-compose up

# Start with debug tools (API + Redis + Redis Commander + SQLite Browser)
docker-compose --profile debug up

# Start with frontend (API + Redis + Nginx frontend)
docker-compose --profile frontend up
```

**Services:**
- **math-constants-api** (port 8000) - Main API
- **redis** (port 6379) - Caching layer
- **redis-commander** (port 8081, debug profile) - Redis web UI
- **sqlite-browser** (port 8080, debug profile) - SQLite web UI
- **frontend** (port 3000, frontend profile) - Test interface

## 📊 Performance Improvements

### Before Redis (Direct SQLite/File):
- Average response time: 50-100ms
- Search queries: 200-500ms
- Stats calculations: 300-600ms

### After Redis Caching:
- Cached digit retrieval: **5-10ms** (90% faster)
- Cached search results: **2-5ms** (95% faster)
- Cached statistics: **3-8ms** (95% faster)

### Cache Hit Rates:
- Digit queries: ~60-80%
- Search queries: ~40-60%
- Stats queries: ~70-90%

## 🔄 Cache Invalidation

Redis cache is automatically invalidated when:
- Cache is rebuilt via `/admin/build-all-caches`
- Cache is rebuilt for specific constant via `/{constant}/build-cache`
- Manual cache clear via `DELETE /{constant}/cache`

## 📡 WebSocket Commands

### General Connection Commands:
```javascript
// Ping-pong for keepalive
{ "type": "ping", "timestamp": 1234567890 }

// Get connection statistics
{ "type": "get_stats" }
```

### Constant-Specific Commands:
```javascript
// Stream digits
{
"command": "stream_digits",
"start": 0,
"length": 1000,
"chunk_size": 100
}

// Get constant status
{ "command": "get_status" }

// Ping
{ "command": "ping" }
```

### Server Messages:
```javascript
// Stream chunk
{
"type": "stream_chunk",
"chunk_number": 5,
"position": 500,
"digits": "1234567890...",
"remaining": 500
}

// Broadcast notifications
{
"type": "digits_accessed",
"constant_id": "pi",
"start": 100,
"length": 50
}

{
"type": "cache_build_complete",
"constant_id": "pi"
}
```

## ⚙️ Configuration

Key environment variables for Redis & WebSocket (see `.env.example`):

```bash
# Redis Configuration
REDIS_URL=redis://redis:6379/0
REDIS_MAX_CONNECTIONS=50
CACHE_TTL=3600
ENABLE_CACHE=true

# WebSocket Configuration
WS_HEARTBEAT_INTERVAL=30
WS_MAX_CONNECTIONS=1000
WS_MESSAGE_QUEUE_SIZE=100
```

## 🧪 Testing

### Test Redis Connection:
```bash
# Connect to Redis CLI
docker-compose exec redis redis-cli

# Check keys
KEYS mathconst:*

# Get cache stats
INFO stats
```

### Test WebSocket:
```bash
# Using wscat (install: npm install -g wscat)
wscat -c ws://localhost:8000/ws/connect

# Send ping
{"type":"ping","timestamp":1234567890}
```

### Test Caching:
```bash
# First request (cache miss)
time curl "http://localhost:8000/pi/digits?start=0&length=100"

# Second request (cache hit - much faster!)
time curl "http://localhost:8000/pi/digits?start=0&length=100"
```

## 🤝 Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.

## 📄 License

MIT License - see [LICENSE.md](LICENSE.md)

## 🔮 Roadmap

- [x] Redis caching layer
- [x] WebSocket streaming support
- [ ] Redis pub/sub for distributed deployments
- [ ] GraphQL API support
- [ ] Advanced pattern recognition with ML
- [ ] Real-time collaborative features
- [ ] Kubernetes deployment manifests

## 📞 Support

- 📚 [Full Documentation](docs/)
- 🐛 [Issue Tracker](https://github.com/your-repo/issues)
- 💬 [Discussions](https://github.com/your-repo/discussions)

---

**New in v2.1:** Redis caching and WebSocket streaming for real-time applications!

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/pkeffect/numbers

Awesome Lists containing this project

README