{"id":15066718,"url":"https://github.com/bessouat40/python-data-loader","last_synced_at":"2026-03-06T18:32:36.575Z","repository":{"id":231864085,"uuid":"779203254","full_name":"Bessouat40/python-data-loader","owner":"Bessouat40","description":"Python service : insert data from a directory inside postgres db or inside elastic.","archived":false,"fork":false,"pushed_at":"2024-07-15T12:51:03.000Z","size":760,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-15T23:44:22.152Z","etag":null,"topics":["api-rest","data-insertion","data-loader","database","databases","elasticsearch","fastapi","kibana","postgresql","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bessouat40.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-29T09:22:15.000Z","updated_at":"2024-07-15T12:49:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"81e45220-7f91-4856-adac-64b27eabbbc5","html_url":"https://github.com/Bessouat40/python-data-loader","commit_stats":null,"previous_names":["bessouat40/python-data-loader"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bessouat40%2Fpython-data-loader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bessouat40%2Fpython-data-loader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bessouat40%2Fpython-data-loader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bessouat40%2Fpython-data-loader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bessouat40","download_url":"https://codeload.github.com/Bessouat40/python-data-loader/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242008058,"owners_count":20056838,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-rest","data-insertion","data-loader","database","databases","elasticsearch","fastapi","kibana","postgresql","python"],"created_at":"2024-09-25T01:11:12.887Z","updated_at":"2026-03-06T18:32:36.529Z","avatar_url":"https://github.com/Bessouat40.png","language":"Python","readme":"# Data Ingestor Service\n\nThis project is a python service for data ingestion inside a `Postgres` database and inside an `Elastic`.\n\n- **Postgres :** It takes all `csv files` from `data/postgres` and insert data inside database,\n- **Elastic :** It takes all documents from `data/elastic` and insert them inside elastic.\n\n![schema](./media/schema.jpeg)\n\n## Usage\n\n- Setup environment :\n\n  ```bash\n  mv .env.example .env\n  ```\n\n- Fill `.env` with your values\n\n- Launch databases and Python service :\n\n  ```bash\n  make start\n  ```\n\n## Postgres Service\n\n- Put your csv file(s) inside `data/postgres`folder and then run :\n\n  ```bash\n  python utils/ingestPg.py\n  ```\n\n### Results Postgres\n\n```bash\nINFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)\n---------- Start Data Ingestion ----------\n\n\nWe found 3 csv files to process\n\n\n-------------------------\n\n\n2 lines from .../data-ingestor-service/data/test.csv inserted into test\n.../data-ingestor-service/fastapi_service/postgresIngestor.py:57: UserWarning: We can't add data from .../data-ingestor-service/data/test2.csv, columns doesn't match tables from public schema...\n  warnings.warn(f\"We can't add data from {path}, columns doesn't match tables from {SCHEMA} schema...\")\n2 lines from .../data-ingestor-service/data/test_copy.csv inserted into test\n\n\n-------------------------\n\n\nTotal : 4 inserted lines\n\n\n---------- End Data Ingestion ----------\nINFO:     127.0.0.1:64419 - \"POST /ingest HTTP/1.1\" 200 OK\n```\n\n## Elastic Service\n\n- Put your documents inside `data/elastic`folder and then run :\n\n  ```bash\n  python utils/ingestElastic.py\n  ```\n\n### Results Elastic\n\n```bash\ndata-ingestor-service-backend-1  | ---------- Start Data Ingestion ----------\ndata-ingestor-service-backend-1  |\ndata-ingestor-service-backend-1  | We found 2 documents to process\ndata-ingestor-service-backend-1  |\ndata-ingestor-service-backend-1  | -------------------------\ndata-ingestor-service-backend-1  |\ndata-ingestor-service-backend-1  |\ndata-ingestor-service-backend-1  |\ndata-ingestor-service-backend-1  |\ndata-ingestor-service-backend-1  | -------------------------\ndata-ingestor-service-backend-1  |\ndata-ingestor-service-backend-1  |\ndata-ingestor-service-backend-1  | Total : 2 documents inserted\ndata-ingestor-service-backend-1  |\ndata-ingestor-service-backend-1  | ---------- End Data Ingestion ----------\ndata-ingestor-service-backend-1  | INFO:     192.168.0.1:61372 - \"GET /ingestElastic HTTP/1.1\" 200 OK\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbessouat40%2Fpython-data-loader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbessouat40%2Fpython-data-loader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbessouat40%2Fpython-data-loader/lists"}