{"id":25162720,"url":"https://github.com/denpalrius/data-processing-app","last_synced_at":"2026-04-04T08:36:44.569Z","repository":{"id":276102479,"uuid":"926415161","full_name":"denpalrius/data-processing-app","owner":"denpalrius","description":" Data processing microservice application","archived":false,"fork":false,"pushed_at":"2025-02-17T05:41:30.000Z","size":1102,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-03T14:27:07.554Z","etag":null,"topics":["aws","docker","fastapi","file-upload","kuberne","kubernetes","microservices-architecture","minio","nats-jetstream","nestjs","nestjs-backend","pandas","python","reactjs","server-side-events","sse-client","terraform","websocket"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/denpalrius.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-03T08:01:35.000Z","updated_at":"2025-02-27T18:09:01.000Z","dependencies_parsed_at":"2025-02-13T19:28:38.881Z","dependency_job_id":null,"html_url":"https://github.com/denpalrius/data-processing-app","commit_stats":null,"previous_names":["denpalrius/data-processing-app"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/denpalrius/data-processing-app","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denpalrius%2Fdata-processing-app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denpalrius%2Fdata-processing-app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denpalrius%2Fdata-processing-app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denpalrius%2Fdata-processing-app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/denpalrius","download_url":"https://codeload.github.com/denpalrius/data-processing-app/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denpalrius%2Fdata-processing-app/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260477921,"owners_count":23015064,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","docker","fastapi","file-upload","kuberne","kubernetes","microservices-architecture","minio","nats-jetstream","nestjs","nestjs-backend","pandas","python","reactjs","server-side-events","sse-client","terraform","websocket"],"created_at":"2025-02-09T03:17:43.727Z","updated_at":"2025-12-30T19:05:15.237Z","avatar_url":"https://github.com/denpalrius.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Processing App\n\nThis repository contains the code and configuration for a data processing microservice application. The application includes the following services and applications:\n\n- **NATS Server**: A high-performance messaging system.\n- **MinIO Server**: An object storage server compatible with Amazon S3.\n- **PostgreSQL Database**: A relational database for storing application data.\n- **Frontend Application**: A web-based user interface.\n- **Python File Processor**: A service for processing files using Python.\n- **NestJS Storage Service**: A storage service built with NestJS.\n- **FastAPI WebSocket**: A WebSocket server built with FastAPI.\n\nThis guide provides instructions on how to run and test the application manually, using Docker Compose, and deploying it to a Kubernetes cluster.\n\n## Prerequisites\n\n- Docker\n- Docker Compose\n- Kubernetes (kubectl)\n- Node.js 22\n- Python 3.11.11\n\n## Configuring Environment Variables\n\nCreate a `.env` file in the root directory of the project (same directory as the `docker-compose.yml` file) and add the following configuration:\n\n```dotenv\n# PostgreSQL\nPOSTGRES_USER=admin\nPOSTGRES_PASSWORD=dev123_admin\n\nDATABASE_HOST=postgres\nDATABASE_PORT=5432\nDATABASE_USER=admin\nDATABASE_PASSWORD=dev123_admin\nDATABASE_NAME=file_processor\n\n# MinIO\nMINIO_ENDPOINT=minio\nMINIO_REPLACEMENT_URL=http://localhost:9002 # The one exposed by the host\nMINIO_PORT=9000 # The one exposed by the container\nMINIO_USE_SSL=false\nMINIO_ROOT_USER=admin\nMINIO_ROOT_PASSWORD=dev123_admin\nMINIO_ACCESS_KEY=UpcdoQgnk5Z8liDwyaXp\nMINIO_SECRET_KEY=N9WYPZlYUMroTdIsnZwlZQVFTI6pQ5pFi55kC9UD\nMINIO_STAGING_BUCKET=staging\nMINIO_PROCESSED_BUCKET=processed\n\n# NATS\nNATS_SERVERS=nats://nats:4222\nNATS_STAGED_SUBJECT=file.upload.completed\nNATS_PROCESSED_SUBJECT=file.processing.completed\n\n# SSE\nSSE_EVENTS_URL=http://sse-server:8081/events\nSSE_BROADCAST_URL=http://sse-server:8081/broadcast\n\n# File Upload Service\nLOGGER_LEVEL=log\n```\n\n## Running the Application with Docker Compose\n\n1. Navigate to the root directory of the project.\n2. Run the following command to start the services:\n\n```bash\ndocker-compose up\n```\n\nThis command will start all the services as defined in the `docker-compose.yml` file.\n\n## Running the Application with Kubernetes [*not ready for use]\n\nTo deploy the application to a Kubernetes cluster, follow these steps:\n\n1. Ensure your Kubernetes cluster is up and running.\n2. Apply the Kubernetes manifest:\n\n```bash\nkubectl apply -f ./infrastructure/k8s/deployment.yaml\n```\n\nAlternatively, you can use the provided `deploy.sh` script to apply the manifest:\n\n```bash\n./infrastructure/k8s/deploy.sh\n```\n\n## Testing the Application\n\nOpen your browser and navigate to `http://localhost:81`. You should see the frontend application and be able to interact with it.\n\nDifferent layers of restrictions have been placed for file/mime types from the frontend, the upload service, and the processing service.\nFor preview, use either a `CSV` or `Excel` file.\n\n### Testing with a Tabular CSV File\n\nTo test the application with a tabular CSV file, follow these steps:\n\n1. Prepare a tabular CSV file. See sample structure below:\n\n```csv\nid,name,age,email\n1,John Doe,30,john.doe@example.com\n2,Jane Smith,25,jane.smith@example.com\n3,Bob Johnson,40,bob.johnson@example.com\n```\n\nYou can find sample CSV files from the Florida State University [CSV archive](https://people.sc.fsu.edu/~jburkardt/data/csv/csv.html).\n\n2. Open the frontend application in your browser at `http://localhost:81`.\n3. Use the file upload feature to upload the prepared CSV file.\n4. Monitor the logs of the `file-upload-service` and `file-processing-service` to ensure the file is processed correctly.\n\n\u003e For quick prototyping, most of the secrets are in plain text and included with the config files. An ideal situation would be to use a secret store to store and retrieve them.\n\n## Cleaning Up\n\nTo stop and remove the Docker Compose services, run the following command:\n\n```bash\ndocker-compose down\n```\n\nTo delete the Kubernetes resources, run the following command:\n\n```bash\nkubectl delete -f infrastructure/k8s-deployment.yaml\n```\n\n## Troubleshooting\n\nIf you encounter any issues, check the logs of the respective services for more information. You can view the logs using the following commands:\n\n- Docker Compose:\n\n```bash\ndocker-compose logs\n```\n\n- Kubernetes:\n\n```bash\nkubectl logs \u003cpod-name\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenpalrius%2Fdata-processing-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdenpalrius%2Fdata-processing-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenpalrius%2Fdata-processing-app/lists"}