{"id":25442300,"url":"https://github.com/arvind-4/python-logs","last_synced_at":"2026-05-07T02:36:36.052Z","repository":{"id":212642402,"uuid":"731977428","full_name":"Arvind-4/python-logs","owner":"Arvind-4","description":"This project implements a log ingestor that can efficiently handle large volumes of log data, and a simple interface for querying the ingested logs.","archived":false,"fork":false,"pushed_at":"2023-12-15T10:43:18.000Z","size":922,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-05-30T16:56:53.182Z","etag":null,"topics":["docker","elasticsearch","fastapi","kibana","python"],"latest_commit_sha":null,"homepage":"https://github.com/Arvind-4/python-logs","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Arvind-4.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-12-15T10:33:02.000Z","updated_at":"2023-12-15T10:42:22.000Z","dependencies_parsed_at":"2023-12-15T11:47:21.862Z","dependency_job_id":null,"html_url":"https://github.com/Arvind-4/python-logs","commit_stats":null,"previous_names":["arvind-4/python-logs"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Arvind-4%2Fpython-logs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Arvind-4%2Fpython-logs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Arvind-4%2Fpython-logs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Arvind-4%2Fpython-logs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Arvind-4","download_url":"https://codeload.github.com/Arvind-4/python-logs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239293921,"owners_count":19615043,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","elasticsearch","fastapi","kibana","python"],"created_at":"2025-02-17T13:18:03.211Z","updated_at":"2025-11-01T14:30:26.809Z","avatar_url":"https://github.com/Arvind-4.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Log Ingestor and Query System\r\n\r\n\r\n## About The Project\r\n\r\nThis project implements a log ingestor that can efficiently handle large volumes of log data, and a simple interface for querying the ingested logs.\r\n\r\n## Features\r\n\r\n-   Log ingestor that accepts log data over HTTP on port 3000\r\n-   Storage engine that can scale to handle vast log volumes\r\n-   Query interface with full-text search and field filters\r\n\r\n## Screenshots\r\n\r\n- **Elastic Search**\r\n  \u003cimg src=\".github/static/elasticsearch.png\" alt=\"Elastic Search\" /\u003e\r\n- **Kibana**\r\n  \u003cimg src=\".github/static/kibana.png\" alt=\"Kibana\" /\u003e\r\n\r\n\r\n## Built With\r\n\r\n- FastAPI\r\n- Docker\r\n- Elasticsearch\r\n- Kibana\r\n\r\n## Getting Started\r\n\r\nThis is an example of how you may give instructions on setting up your project locally.\r\n\r\n### Prerequisite\r\n-   Python 3.8+\r\n-   Docker\r\n\r\n\r\n### Installation\r\n\r\n1. Clone the repo\r\n```bash\r\ngit clone https://github.com/Arvind-4/python-logs.git .\r\n```\r\n2. Copy contents of `.sample.env` to `.env`.\r\n```bash\r\ncp .sample.env .env \r\n```\r\n3. Run the docker file\r\n```bash\r\ndocker compose up --build \r\n```\r\n\r\n### Populate the Data\r\n\r\nTo populate sample log data:\r\n\r\n-   Start Elasticsearch and Kibana\r\n-   Run  `data/dump_data.py`  to load sample logs\r\n-   The script sends logs to the ingestor API endpoint\r\n\r\n## Usage\r\n\r\nThe log ingestor stack consists of:\r\n\r\n-   **Kibana**  - Query interface and log visualization, runs on port 5601\r\n-   **Elasticsearch**  - Storage engine for logs, runs on port 9200\r\n-   **Log ingestor**  - Backend service to ingest logs, runs on port 3000\r\n\r\nLogs are ingested by sending a **POST** request to the ingestor API endpoint:\r\n\r\n```bash\r\ncurl  --header  \"Content-Type: application/json\"  \\\r\n--request  POST  \\\r\n--data  '{ \"level\": \"error\", \"message\": \"Failed to connect to DB\", \"resourceId\": \"server-1234\", \"timestamp\": \"2023-09-15T08:00:00Z\",\"traceId\": \"abc-xyz-124\", \"spanId\": \"span-456\", \"commit\": \"5e5342f\", \"metadata\": {\"parentResourceId\": \"server-0987\"}}'  \\\r\nhttp://0.0.0.0:3000/api/v1/insert\r\n```\r\n\r\nThe ingestor stores the logs in Elasticsearch for querying.\r\n\r\nThe Kibana interface allows searching the logged data via:\r\n\r\n-   Full text search across all fields\r\n-   Filtering by specific log fields\r\n\r\nVisualizations and dashboards can also be built in Kibana to analyze the log data.\r\n\r\n## License\r\n\r\nDistributed under the MIT License. See `LICENSE.md` for more information.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farvind-4%2Fpython-logs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farvind-4%2Fpython-logs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farvind-4%2Fpython-logs/lists"}