{"id":23734237,"url":"https://github.com/achanandhi-m/logflow","last_synced_at":"2026-05-02T10:39:33.999Z","repository":{"id":269164020,"uuid":"906261421","full_name":"Achanandhi-M/Logflow","owner":"Achanandhi-M","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-21T12:48:00.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-31T05:38:04.285Z","etag":null,"topics":["athena","aws","aws-lambda","cloudformation","glue","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Achanandhi-M.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-20T14:01:37.000Z","updated_at":"2024-12-21T13:57:55.000Z","dependencies_parsed_at":"2024-12-21T12:31:21.897Z","dependency_job_id":"4849292c-e0c2-415b-8c94-a83d67c3fbe1","html_url":"https://github.com/Achanandhi-M/Logflow","commit_stats":null,"previous_names":["achanandhi-m/logflow"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Achanandhi-M%2FLogflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Achanandhi-M%2FLogflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Achanandhi-M%2FLogflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Achanandhi-M%2FLogflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Achanandhi-M","download_url":"https://codeload.github.com/Achanandhi-M/Logflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239859567,"owners_count":19708865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["athena","aws","aws-lambda","cloudformation","glue","python3"],"created_at":"2024-12-31T05:38:22.989Z","updated_at":"2026-02-24T19:30:16.649Z","avatar_url":"https://github.com/Achanandhi-M.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LogFlow\n\n**LogFlow** is an automated log management pipeline that transforms logs in Excel format into Parquet, uploads them to an S3 bucket, triggers AWS Lambda to run a Glue Crawler for metadata extraction, and makes the data searchable using Athena. The entire process is automated using AWS CloudFormation.\n\n---\n\n## Features\n\n- **Excel to Parquet Conversion:** Converts Excel logs into Parquet format for efficient storage and querying.\n- **Automated AWS Workflow:** Seamlessly integrates S3, Lambda, Glue Crawler, and Athena for end-to-end log management.\n- **Serverless Architecture:** Utilizes AWS services to provide a highly scalable and cost-effective solution.\n- **Searchable Logs:** Enables advanced log analysis using Amazon Athena.\n\n---\n\n## Architecture\n\n1. **Excel Logs**: Input logs in `.xlsx` format.\n2. **Parquet Conversion**: Transforms logs into `.parquet` format.\n3. **S3 Bucket**: Stores the Parquet files.\n4. **AWS Lambda**: Triggers Glue Crawler when new data is uploaded.\n5. **AWS Glue Crawler**: Extracts metadata and updates the Glue Data Catalog.\n6. **Amazon Athena**: Queries and searches the processed log data.\n\n---\n\n## Prerequisites\n\n- AWS Account\n- AWS CLI\n- Python 3.x\n- CloudFormation templates (included in the repository)\n- IAM roles with required permissions\n\n---\n\n## Setup and Usage\n\n### Step 1: Clone the Repository\nClone the repository and navigate to the project directory:\n  ```bash\n   git clone https://github.com/Achanandhi-M/Logflow.git\n   cd Logflow\n  ```\n\n### Step 2: Deploy CloudFormation Stacks\nDeploy the CloudFormation stacks to set up all required AWS resources:\n\n1. Deploy the S3 bucket stack:\n    ```bash\n    aws cloudformation deploy --template-file CloudFormation/Create_S3_Bucket.yaml --stack-name LogFlowS3Stack\n    ```\n\n2. Deploy the Glue and Lambda stack:\n    ```bash\n    aws cloudformation deploy --template-file CloudFormation/Create_Glue_Lambda.yaml --stack-name LogFlowGlueLambdaStack\n    ```\n\n### Step 3: Convert and Upload Logs\nConvert your Excel log file to Parquet and upload it to the S3 bucket:\n```bash\npython scripts/convert_excel_to_parquet.py \n```\n\n### Step 4: Query Logs with Athena\n1. Open the Amazon Athena console.\n2. Select the database created by the Glue Crawler.\n3. Start querying your log data using SQL.\n\n---\n\n## Directory Structure\n```plaintext\nlogManagement/\n│\n├── CloudFormation/                        # Directory for CloudFormation templates\n│   ├── Create_Glue_Lambda.yaml            # Template for creating Glue and Lambda resources\n│   ├── Create_S3_Bucket.yaml              # Template for creating S3 bucket\n│\n├── lambda/                                # Directory for Lambda function code\n│   └── trigger_glue_crawler.py            # Lambda function to trigger Glue Crawler\n│\n├── scripts/                               # Directory for helper scripts\n│   └── convert_excel_to_parquet.py        # Script to convert Excel to Parquet and upload to S3\n│\n├── README.md                              # Project documentation\n```\n\n## Author\n\nDeveloped by Your Achanandhi M","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fachanandhi-m%2Flogflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fachanandhi-m%2Flogflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fachanandhi-m%2Flogflow/lists"}