{"id":35134685,"url":"https://github.com/iamkyr0/atlas","last_synced_at":"2026-04-24T15:33:05.379Z","repository":{"id":329326433,"uuid":"1118938128","full_name":"iamkyr0/atlas","owner":"iamkyr0","description":"Decentralized AI platform for fine-tuning, RAG, and federated learning 🤖🌐","archived":false,"fork":false,"pushed_at":"2025-12-18T20:38:13.000Z","size":176,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-21T21:46:20.580Z","etag":null,"topics":["ai","ai-infrastructure","blockchain","cosmos-sdk","decentralized","decentralized-ai","distributed-compute","federated-learning","fine-tunning","ipfs","llm","lora","web3"],"latest_commit_sha":null,"homepage":"https://iamkyr0.github.io/atlas/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iamkyr0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-18T13:56:49.000Z","updated_at":"2025-12-20T09:01:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/iamkyr0/atlas","commit_stats":null,"previous_names":["iamkyr0/atlas"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/iamkyr0/atlas","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamkyr0%2Fatlas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamkyr0%2Fatlas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamkyr0%2Fatlas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamkyr0%2Fatlas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iamkyr0","download_url":"https://codeload.github.com/iamkyr0/atlas/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamkyr0%2Fatlas/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32228988,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T13:21:15.438Z","status":"ssl_error","status_checked_at":"2026-04-24T13:21:15.005Z","response_time":64,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-infrastructure","blockchain","cosmos-sdk","decentralized","decentralized-ai","distributed-compute","federated-learning","fine-tunning","ipfs","llm","lora","web3"],"created_at":"2025-12-28T08:08:05.672Z","updated_at":"2026-04-24T15:33:05.372Z","avatar_url":"https://github.com/iamkyr0.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Atlas - Decentralized AI Platform\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Go Version](https://img.shields.io/badge/Go-1.21+-blue.svg)](https://golang.org/)\n[![Python Version](https://img.shields.io/badge/Python-3.10+-blue.svg)](https://www.python.org/)\n\n**Atlas** is a decentralized Infrastructure-as-a-Service (IaaS) platform for fine-tuning and serving AI models. The platform uses blockchain for coordination and reward distribution, with nodes contributing compute and storage resources similar to cryptocurrency mining systems.\n\n## 🌟 Features\n\n- **Decentralized Training**: Fine-tune AI models using distributed compute nodes\n- **Federated Learning**: Privacy-preserving distributed training with gradient aggregation\n- **LoRA Fine-Tuning**: Efficient fine-tuning with Low-Rank Adaptation\n- **Model Serving**: Serve models for inference via HTTP/gRPC API\n- **Inference Network**: Distributed network for LLM, Vision, Speech-to-text, and Embedding services\n- **Blockchain Coordination**: Cosmos SDK blockchain for job coordination and reward distribution\n- **IPFS Storage**: Decentralized storage for datasets, models, and checkpoints\n- **P2P Communication**: Direct peer-to-peer communication without API Gateway\n- **Auto Resource Detection**: Automatic detection of CPU, GPU, RAM, storage, and network speed\n- **Fault Tolerance**: Automatic task reassignment, checkpoint recovery, and graceful degradation\n\n## 🏗️ Architecture\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    ATLAS PLATFORM                            │\n├─────────────────────────────────────────────────────────────┤\n│                                                              │\n│  ┌──────────┐      ┌──────────┐      ┌──────────┐        │\n│  │  Client  │      │  Node 1  │      │  Node 2  │        │\n│  │   SDK    │      │          │      │          │        │\n│  └────┬─────┘      └────┬─────┘      └────┬─────┘        │\n│       │                 │                 │               │\n│       ├─────────────────┴─────────────────┤               │\n│       │                                     │               │\n│       │         IPFS Pub/Sub               │               │\n│       │    (Real-time messaging)           │               │\n│       │                                     │               │\n│       ├─────────────────────────────────────┤               │\n│       │                                     │               │\n│       │         IPFS Storage                │               │\n│       │    (Files: datasets, models)       │               │\n│       │                                     │               │\n│       └─────────────────┬───────────────────┘               │\n│                        │                                    │\n│                        │ gRPC/RPC                           │\n│                        │                                    │\n│                 ┌──────▼──────┐                            │\n│                 │   Cosmos    │                            │\n│                 │   Chain     │                            │\n│                 │  (Blockchain)│                           │\n│                 └─────────────┘                            │\n│                                                             │\n└─────────────────────────────────────────────────────────────┘\n```\n\n### Core Components\n\n- **Blockchain Layer** (`chain/`): Cosmos SDK with 10 custom modules\n- **Compute Node** (`node/`): Distributed compute nodes for training \u0026 inference\n- **Federated Learning** (`federated-learning/`): Privacy-preserving distributed training\n- **LoRA** (`lora/`): Efficient fine-tuning with Low-Rank Adaptation\n- **Storage Layer** (`storage/`): IPFS-based decentralized storage\n- **Client SDK** (`sdk/`): Python SDK with CLI and daemon mode\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- **Go** 1.21+\n- **Python** 3.10+\n- **Docker** (optional, for containerized deployment)\n- **IPFS node** (or use Infura/IPFS gateway)\n- **Cosmos SDK** (for blockchain)\n\n### Installation\n\n```bash\n# Clone repository\ngit clone https://github.com/iamkyr0/atlas.git\ncd atlas\n\n# Setup Go dependencies\ncd chain\ngo mod download\n\n# Setup Python SDK\ncd ../sdk/python\npip install -r requirements.txt\n\n# Start IPFS node (or use Infura)\nipfs daemon\n```\n\n### Docker Compose (Recommended)\n\n```bash\n# Start all services (chain, IPFS, node)\ndocker-compose up -d\n\n# Check status\ndocker-compose ps\n```\n\n## 📖 Usage Guide\n\n### 🎯 As a Client (Using Atlas for Fine-Tuning)\n\n#### 1. Install Python SDK\n\n```bash\ncd sdk/python\npip install -r requirements.txt\n```\n\n#### 2. Initialize Client\n\n```python\nfrom atlas import AtlasClient\n\nclient = AtlasClient(\n    ipfs_api_url=\"/ip4/127.0.0.1/tcp/5001\",  # or \"https://ipfs.infura.io:5001\"\n    chain_grpc_url=\"localhost:9090\",  # Format: host:port\n    creator=\"your_wallet_address\"\n)\n```\n\n#### 3. Upload Dataset \u0026 Submit Training Job\n\n```python\nimport asyncio\n\nasync def train_model():\n    async with AtlasClient() as client:\n        # 1. Upload dataset\n        dataset_cid = await client.upload_dataset(\"dataset.zip\")\n        print(f\"Dataset CID: {dataset_cid}\")\n        \n        # 2. Submit training job\n        job_id = await client.submit_job(\n            model_id=\"model-123\",\n            dataset_cid=dataset_cid,\n            config={\n                \"epochs\": 10,\n                \"batch_size\": 32,\n                \"learning_rate\": 0.001,\n                \"lora_rank\": 8\n            }\n        )\n        print(f\"Job ID: {job_id}\")\n        \n        # 3. Monitor progress (real-time via IPFS Pub/Sub)\n        async for update in client.subscribe_to_job_updates(job_id):\n            status = update.get('status')\n            progress = update.get('progress', 0.0) * 100\n            print(f\"Status: {status}, Progress: {progress:.1f}%\")\n            \n            if status == \"completed\":\n                break\n        \n        # 4. Download trained model\n        job = await client.get_job(job_id)\n        await client.download_model(\n            model_cid=job.get('model_cid'),\n            output_path=\"./trained_model.pt\"\n        )\n        print(\"Model downloaded!\")\n\nasyncio.run(train_model())\n```\n\n#### 4. Using Model for Inference\n\n```python\nfrom atlas.serving.predictor import Predictor\nfrom atlas.serving.loader import ModelLoader\n\n# Load model from IPFS (with caching)\nloader = ModelLoader(ipfs_api_url=\"/ip4/127.0.0.1/tcp/5001\")\nmodel_path = loader.load(model_cid=\"QmXXX...\")\n\n# Create predictor\npredictor = Predictor(model_path=model_path)\n\n# Run inference\nresult = predictor.predict(\n    input_data=\"Hello, how are you?\",\n    model_type=\"llm\",  # or \"vision\", \"speech\", \"embedding\", \"auto\"\n    options={\n        \"max_length\": 100,\n        \"temperature\": 0.7\n    }\n)\n\nprint(f\"Result: {result}\")\n```\n\n#### 5. Serve Model as HTTP Server\n\n```bash\n# Via CLI\natlas serve-model model-123 --port 8000\n\n# Or via Python\nfrom atlas.serving.server import ModelServer\n\nserver = ModelServer(chain_client=chain_client)\nawait server.serve(model_id=\"model-123\", host=\"0.0.0.0\", port=8000)\n```\n\n#### 6. CLI Commands\n\n```bash\n# Upload dataset\natlas upload-dataset dataset.zip --encrypt\n\n# Submit training job\natlas submit-job model-123 QmXXX... --config config.json\n\n# Monitor job\natlas get-job job-123\natlas list-jobs\n\n# Download model\natlas download-model model-123 --output ./models/\n\n# Serve model\natlas serve-model model-123 --port 8000\n\n# Start daemon (REST API)\natlas daemon --port 8080\n```\n\n### 🖥️ As a Node (Providing Compute \u0026 Storage)\n\n#### 1. Prerequisites\n\n- Go 1.21+\n- Docker\n- IPFS node\n- GPU (optional, for faster training)\n- Minimum 8GB RAM\n- Minimum 100GB storage\n\n#### 2. Setup IPFS Node\n\n```bash\n# Install IPFS\nwget https://dist.ipfs.io/go-ipfs/v0.20.0/go-ipfs_v0.20.0_linux-amd64.tar.gz\ntar -xvzf go-ipfs_v0.20.0_linux-amd64.tar.gz\ncd go-ipfs\nsudo ./install.sh\n\n# Initialize \u0026 start\nipfs init\nipfs daemon\n```\n\n#### 3. Build \u0026 Run Node\n\n```bash\n# Build node binary\ncd node\ngo mod download\ngo build -o atlas-node cmd/node/main.go\n\n# Run node\n./atlas-node start \\\n  --chain-rpc localhost:9090 \\\n  --ipfs-api /ip4/127.0.0.1/tcp/5001 \\\n  --node-id node-001 \\\n  --address your_wallet_address\n```\n\n#### 4. Register Node to Blockchain\n\n```bash\n# Register via blockchain CLI\natlasd tx compute register-node \\\n  --node-id \"node-001\" \\\n  --address \"your_wallet_address\" \\\n  --from mykey \\\n  --chain-id atlas-chain\n```\n\n#### 5. Monitor Node Status\n\n```bash\n# Check node status\natlas-node status\n\n# Check rewards\natlasd query reward node-rewards node-001\n\n# Check reputation\natlasd query compute node-reputation node-001\n```\n\n#### 6. Docker Deployment\n\n```bash\n# Build image\ndocker build -t atlas-node:latest -f docker/Dockerfile.node .\n\n# Run container\ndocker run -d \\\n  --name atlas-node \\\n  --gpus all \\\n  -v /tmp/atlas-work:/work \\\n  -e CHAIN_RPC_URL=localhost:9090 \\\n  -e IPFS_API_URL=/ip4/127.0.0.1/tcp/5001 \\\n  -e NODE_ID=node-001 \\\n  atlas-node:latest\n```\n\n## 🔧 Advanced Features\n\n### Federated Learning\n\n```python\njob = await client.submit_job(\n    model_id=\"model-123\",\n    dataset_cid=dataset_cid,\n    config={\n        \"training_type\": \"federated\",\n        \"min_clients\": 5,\n        \"rounds\": 10,\n        \"lora_enabled\": True\n    }\n)\n```\n\n### Custom Training Script\n\n```python\n# Upload custom training script\nscript_cid = await client.upload_dataset(\"custom_train.py\")\n\n# Submit job with custom script\njob = await client.submit_job(\n    model_id=\"model-123\",\n    dataset_cid=dataset_cid,\n    config={\n        \"training_script_cid\": script_cid,\n        \"epochs\": 10\n    }\n)\n```\n\n### Private Datasets\n\n```python\n# Upload with encryption\ndataset_cid = await client.upload_dataset(\n    \"private_dataset.zip\",\n    encrypt=True,\n    private_network=True\n)\n```\n\n### Inference Network\n\n```python\n# LLM Inference\nresult = predictor.predict(\n    input_data=\"Explain quantum computing\",\n    model_type=\"llm\",\n    options={\"max_length\": 200, \"temperature\": 0.8}\n)\n\n# Vision Model\nresult = predictor.predict(\n    input_data=\"./image.jpg\",\n    model_type=\"vision\"\n)\n\n# Speech-to-Text\nresult = predictor.predict(\n    input_data=\"./audio.wav\",\n    model_type=\"speech\"\n)\n\n# Embedding Service\nresult = predictor.predict(\n    input_data=\"text to embed\",\n    model_type=\"embedding\"\n)\n```\n\n## 📊 Monitoring \u0026 Progress Tracking\n\n### Real-time Progress Monitoring\n\n```python\n# Subscribe to job updates via IPFS Pub/Sub\nasync for update in client.subscribe_to_job_updates(job_id):\n    print(f\"Status: {update.get('status')}\")\n    print(f\"Progress: {update.get('progress', 0.0) * 100:.1f}%\")\n    print(f\"Tasks: {update.get('tasks', [])}\")\n    \n    if update.get(\"status\") == \"completed\":\n        break\n```\n\n### Manual Query\n\n```python\n# Get job status\njob = await client.get_job(job_id)\nprint(f\"Status: {job.get('status')}\")\nprint(f\"Progress: {job.get('progress', 0.0) * 100:.1f}%\")\n\n# List all jobs\njobs = await client.list_jobs()\nfor job in jobs:\n    print(f\"{job.get('id')}: {job.get('status')} - {job.get('progress', 0.0) * 100:.1f}%\")\n```\n\n## 🚢 Deployment\n\n### Local Development\n\n```bash\n# Start IPFS\nipfs daemon\n\n# Start chain (in separate terminal)\ncd chain\natlasd start\n\n# Start node (in separate terminal)\ncd node\n./atlas-node start\n```\n\n### Docker Compose\n\n```bash\n# Start all services\ndocker-compose up -d\n\n# Check logs\ndocker-compose logs -f\n\n# Stop services\ndocker-compose down\n```\n\n### Kubernetes\n\n```bash\n# Deploy chain\nkubectl apply -f k8s/chain-deployment.yaml\n\n# Deploy IPFS\nkubectl apply -f k8s/ipfs-deployment.yaml\n\n# Deploy node\nkubectl apply -f k8s/node-deployment.yaml\n```\n\n## 🔐 Security\n\n- **Encryption**: Private datasets are encrypted before upload to IPFS\n- **Private Networks**: Support for private IPFS networks\n- **Proof of Computation**: Cryptographic proofs for task completion\n- **Validation**: Content hash validation for shards\n- **Reputation System**: Node reputation for trust management\n- **Authentication**: API key and JWT support\n- **Authorization**: RBAC (Role-Based Access Control)\n\n## 🛠️ Configuration\n\n### Environment Variables\n\n```bash\n# Client\nexport ATLAS_IPFS_API=\"/ip4/127.0.0.1/tcp/5001\"\nexport ATLAS_CHAIN_GRPC=\"localhost:9090\"\nexport ATLAS_CREATOR=\"your_wallet_address\"\n\n# Node\nexport CHAIN_RPC_URL=\"localhost:9090\"\nexport IPFS_API_URL=\"/ip4/127.0.0.1/tcp/5001\"\nexport NODE_ID=\"node-001\"\nexport WORK_DIR=\"/tmp/atlas-work\"\n```\n\n### Config File\n\n```yaml\n# config.yaml\nnode:\n  id: \"node-001\"\n  address: \"your_wallet_address\"\n  chain_rpc_url: \"localhost:9090\"\n  ipfs_api_url: \"/ip4/127.0.0.1/tcp/5001\"\n\nresources:\n  cpu_cores: 4\n  gpu_enabled: true\n  memory_gb: 16\n  storage_gb: 500\n\nheartbeat:\n  interval_seconds: 30\n  timeout_seconds: 90\n```\n\n## 📚 Documentation\n\n### Main Documentation\n\n- **[Client Usage Guide](client-usage.md)** - Complete guide for using Atlas as a client\n- **[Node Usage Guide](node-usage.md)** - Guide for running compute nodes\n- **[Architecture](architecture.md)** - System architecture overview\n- **[Developer Guide](developer-guide.md)** - Guide for developers\n\n### Component Documentation\n\n- **[Chain](chain/README.md)** - Blockchain modules and implementation\n- **[Node](node/README.md)** - Compute node implementation\n- **[Federated Learning](federated-learning/README.md)** - FL implementation\n- **[LoRA](lora/README.md)** - LoRA fine-tuning\n- **[Storage](storage/README.md)** - Storage layer\n- **[SDK](sdk/README.md)** - Client SDK documentation\n\n## 🧪 Testing\n\n```bash\n# Run unit tests\ncd chain\ngo test ./x/... -v\n\ncd node\ngo test ./... -v\n\n# Run integration tests\ncd tests/integration\npytest -v\n```\n\n## 🐛 Troubleshooting\n\n### Connection Issues\n\n- **IPFS not connecting**: Check `ipfs id` and ensure IPFS daemon is running\n- **Chain gRPC error**: Verify chain gRPC URL and port (default: 9090)\n- **Network timeout**: Check firewall rules and network connectivity\n\n### Job Failures\n\n- **Job stuck**: Check node status and resource availability\n- **Model download failed**: Verify model CID in IPFS network\n- **Training error**: Check task logs in work directory\n\n### Node Issues\n\n- **Node not receiving tasks**: Verify node registration and status \"online\"\n- **Resource exhaustion**: Check CPU/GPU/Memory utilization\n- **Heartbeat timeout**: Verify network connectivity to blockchain\n\n## 🤝 Contributing\n\nContributions are welcome! Please read our contributing guidelines and submit pull requests.\n\n## 📄 License\n\nMIT License - see LICENSE file for details\n\n## 🔗 Links\n\n- **Repository**: [GitHub](https://github.com/iamkyr0/atlas)\n- **Documentation**: See [Documentation](#-documentation) section above\n- **Issues**: [GitHub Issues](https://github.com/iamkyr0/atlas/issues)\n\n## 🙏 Acknowledgments\n\n- Cosmos SDK for blockchain framework\n- IPFS for decentralized storage\n- PyTorch and TensorFlow for ML frameworks\n\n---\n\n**Made with ❤️ by the Atlas Team**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamkyr0%2Fatlas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiamkyr0%2Fatlas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamkyr0%2Fatlas/lists"}