Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/siri1404/llm-infrastructure

End-to-end platform for serving Large Language Models with streaming capabilities, privacy-preserving audit trails, drift monitoring, and compliance support for SEC, MiFID II, FINRA, and GDPR standards.
https://github.com/siri1404/llm-infrastructure

ai audit compliance inference kafka llm mlops python serving

Last synced: 7 months ago
JSON representation

End-to-end platform for serving Large Language Models with streaming capabilities, privacy-preserving audit trails, drift monitoring, and compliance support for SEC, MiFID II, FINRA, and GDPR standards.

Awesome Lists containing this project

README 

          
# Production-Ready, Regulatory-Compliant Real-Time LLM Infrastructure



End-to-end platform for serving Large Language Models with streaming capabilities, privacy-preserving audit trails, drift monitoring, and compliance support for SEC, MiFID II, FINRA, and GDPR standards.



[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)

[![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-green.svg)](https://opensource.org/licenses/Apache-2.0)

[![Compliance: SEC/FINRA/GDPR](https://img.shields.io/badge/Compliance-SEC%2FFINRA%2FGDPR-orange.svg)]()



Built for Apple Silicon. Optimized for Finance. Production-ready.



[Features](#key-features) • [Quick Start](#quick-start) • [Documentation](#documentation) • [Benchmarks](#benchmarks) • [Contributing](#contributing)



---



## Overview



This infrastructure prioritizes enterprise requirements: regulatory compliance, privacy-by-design, explainability, and production-grade observability. High-performance inference with built-in audit trails, drift detection, and compliance APIs designed for financial institutions and regulated industries.



### Design Principles



Financial services, healthcare, and regulated industries require LLM infrastructure that:

- Audits all requests and responses (SEC/FINRA/GDPR compliant)

- Explains model decisions (LIME/SHAP for regulatory transparency)

- Detects performance drift (KS-tests, Z-scores, alert APIs)

- Preserves privacy (field masking, encryption, tenant isolation)

- Scales horizontally (Kafka streaming, Kubernetes-ready)

- Deploys across environments (Apple Silicon optimized, multi-cloud)



**Performance Metrics:**

- Sub-50ms p99 latency (with proper hardware)

- 10k+ req/sec throughput (horizontally scaled)

- Full audit coverage (zero data loss)

- Real-time drift alerts (<100ms detection)



---



## Key Features



### Real-Time Event-Driven Processing

- Kafka-based streaming architecture for high-throughput document processing

- Sub-50ms p99 latency with proper hardware configuration

- 10,000+ requests/second when horizontally scaled

- Graceful degradation and retry logic for production resilience



### Comprehensive Audit Trails

- Complete request/response logging with SHA256 hashing for deduplication

- Compliance query API with SEC/FINRA/GDPR filters

- CSV export for regulatory reporting

- Per-tenant isolation with role-based access control

- Privacy-by-design: field masking, anonymization, optional encryption



### Drift Detection & Monitoring

- Statistical drift detection using Kolmogorov-Smirnov tests and Z-scores

- Multi-metric tracking: output quality, user feedback, synthetic drift tests

- REST API for alerts and dashboard integration

- Grafana/Streamlit-ready metrics export

- Configurable thresholds and baseline management



### Explainability & Model Validation

- LIME integration for feature importance explanations

- SHAP support (stub implementation, extensible)

- Finance-specific validation using regex and heuristics

- Validation results stored in audit logs

- Batch explanation support for bulk processing



### Privacy-by-Design

- Field masking/anonymization in audit logs by default

- Per-tenant encryption option for audit database

- GDPR/CCPA compliance documentation and tooling

- Secure multi-tenancy with tenant isolation



### Apple Silicon Optimized

- Benchmarked on M1/M2/M3 with documented latency/throughput

- Core ML API compatibility notes and deployment guides

- Native ARM64 support for optimal performance

- Metal GPU acceleration where applicable



### Flexible Deployment

- Docker Compose for local development

- Kubernetes manifests included (`k8s/` directory)

- Multi-cloud ready (AWS, GCP, Azure compatible)

- Environment-based configuration for easy deployment



---



## Quick Start



### Prerequisites



- Python 3.8+

- Docker & Docker Compose (for Kafka/Ollama)

- 8GB+ RAM (16GB recommended for production workloads)



### Installation



```bash

# Clone the repository

git clone https://github.com/siri1404/llm-infrastructure.git

cd llm-infrastructure



# Install Python dependencies

pip install -r requirements.txt



# Start infrastructure services (Kafka, Ollama)

docker compose up -d



# Wait for services to be ready (~30 seconds)

python scripts/test_everything.py

```



### Run Your First Pipeline



```bash

# Terminal 1: Start the LLM processor

python src/kafka_llm_processor.py



# Terminal 2: Send test documents

python src/test_producer.py --count 5



# Terminal 3: View results

python src/test_consumer.py --max-messages 5

```



You now have a working real-time LLM pipeline with automatic audit logging.



### Start Compliance API



```bash

# Terminal 4: Start compliance API

python src/compliance_api.py



# Query audit logs

curl http://localhost:5000/api/compliance/statistics

```



See [QUICKSTART.md](QUICKSTART.md) for detailed setup instructions.



---



## Documentation



| Document | Description |

|----------|-------------|

| [QUICKSTART.md](QUICKSTART.md) | Get up and running in 5 minutes |

| [KAFKA_SETUP.md](KAFKA_SETUP.md) | Kafka integration and architecture |

| [AUDIT_TRAIL_SETUP.md](AUDIT_TRAIL_SETUP.md) | Audit logging, privacy, multi-tenancy |

| [DRIFT_DETECTION_SETUP.md](DRIFT_DETECTION_SETUP.md) | Drift detection and monitoring |

| [docs/APPLE_SILICON.md](docs/APPLE_SILICON.md) | Apple Silicon deployment guide |



---



## Architecture



```

┌─────────────┐     ┌──────────────┐     ┌─────────────┐     ┌──────────────┐

│   Producer  │────▶│ Kafka Topic  │────▶│   LLM       │────▶│   Results    │

│  (Documents)│     │ (financial-  │     │  Processor  │     │    Topic     │

│             │     │  documents)  │     │              │     │              │

└─────────────┘     └──────────────┘     └──────┬───────┘     └──────────────┘

                                                  │

                                                  ▼

                                         ┌──────────────┐

                                         │ Audit Logger │

                                         │  + Drift     │

                                         │  + Explain   │

                                         └──────────────┘

```



**Key Components:**

- **Kafka**: Event streaming backbone

- **LLM Processor**: Handles inference (vLLM/Ollama compatible)

- **Audit Logger**: Comprehensive request tracking

- **Drift Detector**: Statistical monitoring

- **Compliance API**: REST endpoints for queries

- **Explainability**: LIME/SHAP integration



---



## Benchmarks



### Performance Metrics



| Metric | Target | Achieved (M2 Pro) | Notes |

|--------|--------|-------------------|-------|

| **p99 Latency** | <50ms | ~45ms | With optimized batch size |

| **Throughput** | 10k req/s | 8.5k req/s | Single node, scales linearly |

| **Audit Overhead** | <5% | ~3% | Non-blocking async logging |

| **Drift Detection** | <100ms | ~80ms | Per-batch processing |



### Apple Silicon Benchmarks



See [docs/APPLE_SILICON.md](docs/APPLE_SILICON.md) for detailed benchmarks on:

- **M1 MacBook Air**: CPU-only performance

- **M2 Pro**: GPU acceleration with Metal

- **M3 Max**: Multi-GPU scaling



---



## Compliance Support



### Supported Standards



- **SEC**: Audit trails, request deduplication, time-range queries

- **FINRA**: Complete request/response logging, export capabilities

- **MiFID II**: Transaction reporting, explainability requirements

- **GDPR**: Privacy-by-design, data anonymization, right to deletion



### Prebuilt Compliance Queries



```python

# SEC Compliance: Find all requests for a specific document

POST /api/compliance/query

{

    "input_hash": "sha256_hash_of_document",

    "start_time": "2025-01-01T00:00:00Z",

    "end_time": "2025-01-31T23:59:59Z"

}



# FINRA Compliance: Export all audit logs for a date range

POST /api/compliance/export

{

    "start_time": "2025-01-01T00:00:00Z",

    "end_time": "2025-01-31T23:59:59Z",

    "format": "csv"

}



# GDPR Compliance: Find all requests for a user

POST /api/compliance/query

{

    "user_id": "user-123",

    "tenant_id": "tenant-abc"

}

```



See [AUDIT_TRAIL_SETUP.md](AUDIT_TRAIL_SETUP.md) for more examples.



---



## Deploying on Apple Silicon



This infrastructure is optimized for Apple Silicon (M1/M2/M3) with native ARM64 support.



### Quick Setup



```bash

# Install dependencies (ARM64 native)

pip install -r requirements.txt



# Use Ollama (optimized for Apple Silicon)

docker compose up -d ollama



# Pull a model

python scripts/setup_ollama.py --model llama2



# Run pipeline

python src/kafka_llm_processor.py

```



### Core ML Integration (Planned)



Core ML API integration is planned for enhanced performance. See [docs/APPLE_SILICON.md](docs/APPLE_SILICON.md) for:

- Benchmark results

- GPU acceleration notes

- Core ML compatibility guide



---



## Demo & Testing



### Jupyter Notebook Demo



```bash

# Start Jupyter

jupyter notebook notebooks/demo.ipynb

```



The demo notebook includes:

- End-to-end pipeline execution

- Audit log queries

- Drift detection visualization

- Explainability examples



### Streamlit Dashboard



```bash

# Install Streamlit

pip install streamlit



# Run dashboard

streamlit run dashboard/app.py

```



---



## Configuration



### Environment Variables



```bash

# LLM Configuration

LLM_URL=http://localhost:11434  # Ollama default

MODEL_NAME=llama2



# Kafka Configuration

KAFKA_BROKERS=localhost:9092

INPUT_TOPIC=financial-documents

OUTPUT_TOPIC=llm-results



# Audit Logging

ENABLE_AUDIT_LOGGING=true

AUDIT_DB_PATH=audit_logs.db

ENCRYPT_AUDIT_DB=false  # Set to true for encryption



# Privacy Settings

MASK_SENSITIVE_FIELDS=true

ANONYMIZE_USER_IDS=false



# Compliance API

COMPLIANCE_API_PORT=5000

COMPLIANCE_API_HOST=0.0.0.0



# Drift Detection

DRIFT_DB_PATH=drift_alerts.db

DRIFT_THRESHOLD=0.05

```



See `.env.example` for full configuration options.



---



## Use Cases



### Financial Services

- **Earnings Report Analysis**: Extract key metrics from quarterly reports

- **SEC Filing Processing**: Automated 10-K/10-Q document analysis

- **Regulatory Compliance**: Audit trails for FINRA reporting



### Healthcare

- **Medical Record Processing**: HIPAA-compliant document analysis

- **Clinical Trial Documentation**: Extract structured data from unstructured text



### Legal

- **Contract Analysis**: Extract key terms and clauses

- **Discovery Document Processing**: Large-scale document review



---



## Contributing



We welcome contributions. This project is designed to be extended.



### Areas for Contribution



- **Compliance**: Add support for new regulatory standards (SOX, HIPAA, etc.)

- **Privacy**: Advanced encryption, differential privacy, federated learning

- **Model Types**: Support for more LLM backends (Anthropic, Cohere, etc.)

- **Dashboards**: Grafana panels, Streamlit improvements

- **Benchmarks**: More hardware configurations, latency optimizations



### How to Contribute



1. Fork the repository

2. Create a feature branch (`git checkout -b feature/amazing-feature`)

3. Commit your changes (`git commit -m 'Add amazing feature'`)

4. Push to the branch (`git push origin feature/amazing-feature`)

5. Open a Pull Request



See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.



---



## Roadmap



### Short Term (Next Release)

- [ ] SHAP explainability full implementation

- [ ] Grafana dashboard templates

- [ ] Enhanced output validation rules

- [ ] More compliance prebuilt queries



### Medium Term

- [ ] Core ML API integration

- [ ] Federated learning support

- [ ] Advanced privacy (differential privacy)

- [ ] Multi-model ensemble support



### Long Term

- [ ] Real-time model updating

- [ ] Advanced drift detection (concept drift)

- [ ] Auto-scaling based on drift

- [ ] Cross-cloud deployment tools



---



## License



This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.



---



## Acknowledgments



- **vLLM** for inspiration on high-performance LLM serving

- **Apache Kafka** for robust event streaming

- **LIME/SHAP** communities for explainability tools

- **Apple** for making ARM64 development accessible



---



## Contact & Support



- **GitHub Issues**: [Report bugs or request features](https://github.com/siri1404/llm-infrastructure/issues)

- **Discussions**: [Ask questions](https://github.com/siri1404/llm-infrastructure/discussions)

- **Security**: [Report security issues](https://github.com/siri1404/llm-infrastructure/security/advisories)



---






**Production ML infrastructure for regulated industries**



[Star us on GitHub](https://github.com/siri1404/llm-infrastructure) • [Read the Docs](https://github.com/siri1404/llm-infrastructure/wiki) • [Report Issues](https://github.com/siri1404/llm-infrastructure/issues)