{"id":29981323,"url":"https://github.com/nolancacheux/nestjs-multi-database-backend","last_synced_at":"2026-04-10T12:31:46.671Z","repository":{"id":307504437,"uuid":"1029747912","full_name":"nolancacheux/nestjs-multi-database-backend","owner":"nolancacheux","description":"Enterprise-grade NestJS backend implementing polyglot persistence with distributed storage architecture. Integrates PostgreSQL, Cassandra, Neo4j, Redis, and MinIO for scalable multi-database operations. Containerized deployment with Docker Compose, comprehensive REST API, and performance analysis tools.","archived":false,"fork":false,"pushed_at":"2026-02-11T09:42:00.000Z","size":304,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-11T14:54:37.001Z","etag":null,"topics":["academic-project","big-data","caching","cassandra","database-integration","distributed-systems","docker-compose","graph-database","microservices-architecture","minio","neo4j","nestjs","nosql","object-storage","polyglot-persistence","postgresql","redis","rest-api","scalability","typescript"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nolancacheux.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-31T14:07:05.000Z","updated_at":"2026-02-11T09:42:04.000Z","dependencies_parsed_at":"2025-07-31T18:06:44.508Z","dependency_job_id":"40aad0af-1df2-4e7e-bea9-d5596c1641b9","html_url":"https://github.com/nolancacheux/nestjs-multi-database-backend","commit_stats":null,"previous_names":["nolancacheux/nestjs-multi-database-backend"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nolancacheux/nestjs-multi-database-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nolancacheux%2Fnestjs-multi-database-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nolancacheux%2Fnestjs-multi-database-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nolancacheux%2Fnestjs-multi-database-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nolancacheux%2Fnestjs-multi-database-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nolancacheux","download_url":"https://codeload.github.com/nolancacheux/nestjs-multi-database-backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nolancacheux%2Fnestjs-multi-database-backend/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31642714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T07:40:12.752Z","status":"ssl_error","status_checked_at":"2026-04-10T07:40:11.664Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["academic-project","big-data","caching","cassandra","database-integration","distributed-systems","docker-compose","graph-database","microservices-architecture","minio","neo4j","nestjs","nosql","object-storage","polyglot-persistence","postgresql","redis","rest-api","scalability","typescript"],"created_at":"2025-08-04T16:02:48.477Z","updated_at":"2026-04-10T12:31:45.999Z","avatar_url":"https://github.com/nolancacheux.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Polyglot Distributed API - NestJS Multi-Database Backend\n\n![NestJS](https://img.shields.io/badge/nestjs-%23E0234E.svg?style=for-the-badge\u0026logo=nestjs\u0026logoColor=white)\n![PostgreSQL](https://img.shields.io/badge/postgresql-%23316192.svg?style=for-the-badge\u0026logo=postgresql\u0026logoColor=white)\n![Cassandra](https://img.shields.io/badge/cassandra-%231287B1.svg?style=for-the-badge\u0026logo=apache-cassandra\u0026logoColor=white)\n![Neo4j](https://img.shields.io/badge/neo4j-%23008CC1.svg?style=for-the-badge\u0026logo=neo4j\u0026logoColor=white)\n![Redis](https://img.shields.io/badge/redis-%23DD0031.svg?style=for-the-badge\u0026logo=redis\u0026logoColor=white)\n![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge\u0026logo=docker\u0026logoColor=white)\n![TypeScript](https://img.shields.io/badge/typescript-%23007ACC.svg?style=for-the-badge\u0026logo=typescript\u0026logoColor=white)\n\n## Table of Contents\n\n1. [Introduction](#introduction)\n2. [Project Context](#project-context)\n3. [Project Objectives](#project-objectives)\n4. [Technology Stack](#technology-stack)\n5. [Architecture Overview](#architecture-overview)\n6. [Prerequisites](#prerequisites)\n7. [Installation \u0026 Deployment](#installation--deployment)\n   - [Quick Start](#quick-start)\n   - [Environment Configuration](#environment-configuration)\n   - [Docker Deployment](#docker-deployment)\n   - [Data Generation](#data-generation)\n   - [Development Commands](#development-commands)\n8. [API Documentation](#api-documentation)\n   - [PostgreSQL Endpoints](#postgresql-via-typeorm)\n   - [Cassandra Endpoints](#cassandra-via-cassandra-driver)\n   - [Neo4j Interface](#neo4j-graph-database)\n   - [Redis Access](#redis)\n   - [Storage Endpoints](#minio-s3-compatible-storage)\n9. [Project Structure](#project-structure)\n   - [Directory Overview](#directory-overview)\n   - [Folder Descriptions](#folder-descriptions)\n10. [Database Schemas](#database-schemas)\n11. [Performance Analysis](#performance-analysis)\n12. [Testing](#testing)\n13. [Troubleshooting](#troubleshooting)\n14. [Contributing](#contributing)\n15. [License](#license)\n16. [Author](#author)\n\n## Introduction\n\nThis project is an advanced RESTful backend solution developed with NestJS to manage a distributed and scalable data architecture, tailored for the Summer-Trip web application. Faced with rapid growth in data volume and complexity, this application requires a robust, scalable, and resilient solution integrating multiple specialized databases (PostgreSQL, Cassandra, Neo4j, Redis, and S3-compatible Bucket).\n\n## Project Context\n\nDeveloped as part of the Master 1 Big Data program at JUNIA ISEN (2024), the project addresses the following challenge:\n\n**\"How to develop the data architecture of a storage system to ensure its scalability and resilience in the face of increasing load and data volume?\"**\n\nThe backend (Nolan Cacheux) focuses on the management and advanced testing of each storage system using automatically generated mock data, ensuring a comprehensive technical validation of the distributed architecture.\n\n## Project Objectives\n\n- Provide a distributed REST API to manage users, messages, notifications, groups, and files\n- Ensure data persistence with PostgreSQL, Cassandra, Neo4j, Redis, and Bucket\n- Optimize performance through Redis caching\n- Automate multi-container deployment with Docker and Docker Compose\n- Analyze performance and test integration using advanced Jupyter Notebooks\n- Effectively manage horizontal scalability\n\n## Technology Stack\n\n### Backend Framework\n- **NestJS** - Progressive Node.js framework for building efficient server-side applications\n- **TypeScript** - Type-safe development environment\n\n### Databases\n- **PostgreSQL** - Relational database for structured data (users, groups)\n- **Cassandra** - NoSQL database for high-volume data (messages, notifications)\n- **Neo4j** - Graph database for relationship management\n- **Redis** - In-memory cache for performance optimization\n- **MinIO** - S3-compatible object storage for file management\n\n### Infrastructure\n- **Docker \u0026 Docker Compose** - Containerization and orchestration\n- **TypeORM** - Object-relational mapping for PostgreSQL\n- **Faker.js** - Mock data generation for testing\n\n## Architecture Overview\n\nThis distributed backend follows a microservices-oriented architecture with specialized database systems:\n\n- **User Management**: PostgreSQL for relational user and group data\n- **Messaging System**: Cassandra for scalable message storage\n- **Social Connections**: Neo4j for complex relationship graphs\n- **Caching Layer**: Redis for performance optimization\n- **File Storage**: MinIO for scalable object storage\n\nEach database is accessed through dedicated NestJS modules with their own controllers, services, and models, ensuring separation of concerns and optimal performance for each data type.\n\n## Prerequisites\n\nBefore running this project, ensure you have:\n\n- **Docker** \u003e= 20.10.0\n- **Docker Compose** \u003e= 2.0.0\n- **Node.js** \u003e= 16.x (for local development)\n- **Git** for version control\n\n### System Requirements\n- **RAM**: Minimum 8GB (recommended 16GB)\n- **Storage**: At least 10GB free space\n- **OS**: Windows 10/11, macOS, or Linux\n\n## Installation \u0026 Deployment\n\n### Quick Start\n\n1. **Clone the project**\n\n```bash\ngit clone https://github.com/nolancacheux/backend-distributed-api.git\ncd backend-distributed-api\n```\n\n### Environment Configuration\n\n2. **Configure the .env file**\n\nCreate a .env file at the root of the project with:\n\n```env\n# Server Configuration\nPORT=3000\nNODE_ENV=development\n\n# PostgreSQL Database Configuration\nDATABASE_HOST=postgres\nDATABASE_PORT=5432\nDATABASE_USER=postgres\nDATABASE_PASSWORD=postgres\nDATABASE_NAME=mydb\n\n# Cassandra Configuration\nCASSANDRA_HOST=cassandra\nCASSANDRA_PORT=9042\nCASSANDRA_KEYSPACE=summertrip \n\n# Neo4j Configuration\nNEO4J_HOST=host.docker.internal\nNEO4J_PORT=7687\nNEO4J_USER=neo4j\nNEO4J_PASSWORD=neo4jazerty\n\n# Redis Configuration\nREDIS_HOST=host.docker.internal\nREDIS_PORT=6379\n\n# Storage Configuration (MinIO)\nSTORAGE_ENDPOINT=storage\nSTORAGE_PORT=9000\nSTORAGE_ACCESS_KEY=minioadmin\nSTORAGE_SECRET_KEY=minioadmin\nSTORAGE_BUCKET=uploads\n```\n\n### Docker Deployment\n\n3. **Build and launch Docker containers**\n\n```bash\ndocker-compose up -d --build\n```\n\nThe API will be accessible at: http://localhost:3000\n\n### Data Generation\n\n4. **Generate mock data for testing**\n\nUse the dedicated scripts to populate each database:\n\n```bash\ndocker exec -it backend-api node dist/scripts/postgres_fake_data.js\ndocker exec -it backend-api node dist/scripts/cassandra_fake_data.js\ndocker exec -it backend-api node dist/scripts/neo4j_fake_data.js\ndocker exec -it backend-api node dist/scripts/storage_upload_test.js\n```\n\n### Development Commands\n\nIf you modify the scripts, you can run the following commands to recompile the TypeScript files:\n\n1. **Remove the `dist` folder inside the container:**\n\n```bash\ndocker exec -it backend-api rm -rf dist\n```\n\n2. **Recompile the TypeScript files:**\n\n```bash\ndocker exec -it backend-api npx tsc \n```\n\n**Note**: If the above command does not produce any output, it is often due to the `tsconfig.json` file being misplaced or missing in the container (expected at `/app/tsconfig.json`). To resolve this, copy the file from the host machine:\n\n```bash\ndocker cp tsconfig.json backend-api:/app/tsconfig.json\n```\n\n3. **View the content of the file (for example: `postgres_fake_data.ts`):**\n\n```bash\ndocker exec -it backend-api cat dist/scripts/postgres_fake_data.js\n```\n\n4. **Run the scripts:**\n\n```bash\ndocker exec -it backend-api node dist/scripts/postgres_fake_data.js\ndocker exec -it backend-api node dist/scripts/cassandra_fake_data.js\ndocker exec -it backend-api node dist/scripts/neo4j_fake_data.js\ndocker exec -it backend-api node dist/scripts/storage_upload_test.js\n```\n\n5. **Restart Docker containers if necessary**\n\n```bash\ndocker-compose down\ndocker-compose up -d --build\n```\n\n## API Documentation\n\n### PostgreSQL (via TypeORM)\n- **GET** `http://localhost:3000/users` - Retrieve the list of users\n- **GET** `http://localhost:3000/users/:id` - Retrieve a user by ID\n- **POST** `http://localhost:3000/users` - Create a new user\n- **GET** `http://localhost:3000/groups` - Retrieve the list of groups\n- **GET** `http://localhost:3000/groups/:id` - Retrieve a group by ID\n- **POST** `http://localhost:3000/groups` - Create a new group\n\n### Cassandra (via cassandra-driver)\n- **GET** `http://localhost:3000/messages/:conversationId` - Retrieve messages from a conversation\n- **POST** `http://localhost:3000/messages` - Insert a new message\n- **GET** `http://localhost:3000/notifications/:userId` - Retrieve notifications for a user\n- **POST** `http://localhost:3000/notifications` - Insert a new notification\n\n### Neo4j (Graph Database)\nNo API endpoints are currently defined in the controllers. Data insertion is handled via scripts.  \nTo view relationships, access the Neo4j Browser interface at:  \n`http://localhost:7474`  \n\n**Login credentials:**  \n- **Username**: `neo4j`  \n- **Password**: `neo4jazerty`  \n\n**Example Cypher query to view relationships:**\n```cypher\nMATCH (a:User)-[r:FRIEND]-\u003e(b:User) RETURN a, r, b LIMIT 25;\n```\n\n### Redis\nNo API endpoints are defined for Redis. Data can be accessed using a client such as RedisInsight or via the Redis CLI:\n```bash\ndocker exec -it redis redis-cli\n```\n\n### MinIO (S3-Compatible Storage)\n\n- **POST** `http://localhost:3000/storage/upload`  \n    Upload a file using the `file` field in form-data\n\n- **GET** `http://localhost:3000/storage/download/:filename`  \n    Download a file by specifying the filename in the URL  \n    Example: `http://localhost:3000/storage/download/test-image.png`\n\n**MinIO dashboard is accessible at:**  \n`http://localhost:9001`  \n\n**Login credentials:**  \n- **Access Key**: `minioadmin`  \n- **Secret Key**: `minioadmin`\n\n## Project Structure\n\n### Directory Overview\n\n```\nProjet-M1/\n├── backend-distributed-api/\n│   ├── dist/                        # Compiled files after build\n│   ├── node_modules/                # Project dependencies\n│   ├── notebooks/                   # API testing and performance analysis\n│   │   ├── api_tests.ipynb          # Comprehensive database tests\n│   │   └── data_analysis.ipynb      # Advanced analyses (later)\n│   ├── src/\n│   │   ├── config/                  # General \u0026 DB configurations\n│   │   │   ├── config.module.ts\n│   │   │   ├── postgres.config.ts\n│   │   │   ├── redis.config.ts\n│   │   │   ├── neo4j.config.ts\n│   │   │   ├── cassandra.config.ts\n│   │   │   └── storage.config.ts\n│   │   ├── controllers/             # REST API controllers\n│   │   │   ├── user.controller.ts\n│   │   │   ├── message.controller.ts\n│   │   │   ├── notification.controller.ts\n│   │   │   ├── group.controller.ts\n│   │   │   └── storage.controller.ts\n│   │   ├── databases/               # Database-specific modules\n│   │   │   ├── postgres/\n│   │   │   │   ├── postgres.module.ts\n│   │   │   │   └── postgres.provider.ts\n│   │   │   ├── redis/\n│   │   │   │   ├── redis.module.ts\n│   │   │   │   └── redis.provider.ts\n│   │   │   ├── neo4j/\n│   │   │   │   ├── neo4j.module.ts\n│   │   │   │   └── neo4j.provider.ts\n│   │   │   ├── cassandra/\n│   │   │   │   ├── cassandra.module.ts\n│   │   │   │   └── cassandra.provider.ts\n│   │   │   └── storage/\n│   │   │       ├── storage.module.ts\n│   │   │       └── storage.provider.ts\n│   │   ├── models/                  # Database schemas\n│   │   │   ├── postgres/\n│   │   │   │   ├── user.entity.ts\n│   │   │   │   └── group.entity.ts\n│   │   │   ├── cassandra/\n│   │   │   │   ├── message.model.ts\n│   │   │   │   └── notification.model.ts\n│   │   │   └── neo4j/\n│   │   │       └── relationship.model.ts\n│   │   ├── services/                # Business logic\n│   │   │   ├── postgres/\n│   │   │   │   ├── user.service.ts\n│   │   │   │   └── group.service.ts\n│   │   │   ├── cassandra/\n│   │   │   │   ├── message.service.ts\n│   │   │   │   └── notification.service.ts\n│   │   │   ├── neo4j/\n│   │   │   │   └── relationship.service.ts\n│   │   │   ├── redis/\n│   │   │   │   └── cache.service.ts\n│   │   │   └── storage/\n│   │   │       └── file-storage.service.ts\n│   │   ├── scripts/                 # Advanced utility scripts\n│   │   │   ├── postgres_fake_data.ts\n│   │   │   ├── cassandra_fake_data.ts\n│   │   │   ├── neo4j_fake_data.ts\n│   │   │   └── storage_upload_test.ts\n│   │   ├── shared/                  # Common interfaces and DTOs\n│   │   │   ├── dto/\n│   │   │   │   ├── user.dto.ts\n│   │   │   │   ├── message.dto.ts\n│   │   │   │   └── notification.dto.ts\n│   │   │   └── interfaces/\n│   │   │       └── generic.interface.ts\n│   │   ├── app.module.ts            # Root NestJS module\n│   │   └── main.ts                  # Application entry point\n│   ├── uploads/                     # Temporary file storage before Bucket upload\n│   ├── .dockerignore\n│   ├── .env                         # Global environment variables\n│   ├── .gitignore\n│   ├── .prettierrc\n│   ├── Dockerfile\n│   ├── docker-compose.yml\n│   ├── package.json\n│   ├── tsconfig.json\n│   └── README.md\n```\n\n### Folder Descriptions\n\n- **config/**: Configuration modules for each database\n- **controllers/**: REST route management for each entity\n- **databases/**: Providers and modules specific to database connections\n- **models/**: Schemas and entities for each database\n- **services/**: Business logic for database interactions\n- **scripts/**: Mock data generation scripts for testing\n- **shared/**: Shared DTOs and interfaces between services\n- **uploads/**: Temporary file storage before Bucket upload\n\n## Database Schemas\n\n### PostgreSQL Schema\n- **Users**: id, name, email, created_at\n- **Groups**: id, name, description, created_at\n\n### Cassandra Schema\n- **Messages**: conversation_id, message_id, user_id, content, timestamp\n- **Notifications**: user_id, notification_id, type, content, timestamp\n\n### Neo4j Relationships\n- **User-User**: FRIEND relationships\n- **User-Group**: MEMBER_OF relationships\n\n### Redis Cache Structure\n- User sessions and temporary data storage\n\n## Performance Analysis\n\n### Jupyter Notebooks\nThe project includes comprehensive analysis tools in the `notebooks/` directory:\n\n- **api_tests.ipynb**: Database performance testing and API validation\n- **data_analysis.ipynb**: Advanced performance metrics and optimization analysis\n\n### Running Performance Tests\n```bash\n# Access Jupyter environment (if configured)\ndocker exec -it backend-api jupyter notebook --allow-root --ip=0.0.0.0\n```\n\n### Metrics Monitored\n- Database query response times\n- API endpoint performance\n- Cache hit rates\n- Storage I/O performance\n\n## Testing\n\n### Mock Data Generation\nGenerate test data for all databases:\n\n```bash\n# PostgreSQL test data\ndocker exec -it backend-api node dist/scripts/postgres_fake_data.js\n\n# Cassandra test data\ndocker exec -it backend-api node dist/scripts/cassandra_fake_data.js\n\n# Neo4j test data\ndocker exec -it backend-api node dist/scripts/neo4j_fake_data.js\n\n# Storage test files\ndocker exec -it backend-api node dist/scripts/storage_upload_test.js\n```\n\n### API Testing\nUse the provided Jupyter notebooks for comprehensive API testing and performance analysis.\n\n## Troubleshooting\n\n### Common Issues\n\n#### TypeScript Compilation Issues\nIf script compilation fails:\n```bash\n# Remove dist folder\ndocker exec -it backend-api rm -rf dist\n\n# Copy tsconfig.json if missing\ndocker cp tsconfig.json backend-api:/app/tsconfig.json\n\n# Recompile\ndocker exec -it backend-api npx tsc\n```\n\n#### Database Connection Errors\n- Ensure all containers are running: `docker-compose ps`\n- Check logs: `docker-compose logs [service-name]`\n- Verify environment variables in `.env`\n\n#### Container Issues\n```bash\n# Restart all services\ndocker-compose down\ndocker-compose up -d --build\n\n# View container logs\ndocker-compose logs -f\n```\n\n## Contributing\n\nThis project is part of a Master's thesis. For academic collaboration:\n\n1. Fork the repository\n2. Create a feature branch\n3. Follow the existing code structure\n4. Test with all databases\n5. Submit a pull request\n\n### Development Guidelines\n- Follow NestJS best practices\n- Maintain separation of concerns per database\n- Include appropriate error handling\n- Document any new endpoints\n\n## License\n\nThis project is developed for academic purposes as part of the Master 1 Big Data program at JUNIA ISEN (2024).\n\n## Author\n\n**Nolan Cacheux**  \nMaster 1 Big Data - JUNIA ISEN (2024)  \nFocus: Distributed data architecture and scalability optimization\n\n### Academic Context\n*\"How to develop the data architecture of a storage system to ensure its scalability and resilience in the face of increasing load and data volume?\"*","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnolancacheux%2Fnestjs-multi-database-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnolancacheux%2Fnestjs-multi-database-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnolancacheux%2Fnestjs-multi-database-backend/lists"}