{"id":24064525,"url":"https://github.com/sourasishbasu/ledgerly","last_synced_at":"2025-02-26T18:14:30.708Z","repository":{"id":270157951,"uuid":"908703907","full_name":"SourasishBasu/ledgerly","owner":"SourasishBasu","description":"cloud project wing devops project","archived":false,"fork":false,"pushed_at":"2025-02-25T12:17:56.000Z","size":933,"stargazers_count":0,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-25T13:23:55.877Z","etag":null,"topics":["api","aws","devops","docker","fastapi","gemini","goaccess","grafana","monitoring","postgresql","prometheus","python","traefik"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SourasishBasu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-26T18:31:41.000Z","updated_at":"2025-02-25T12:17:59.000Z","dependencies_parsed_at":"2025-02-25T13:21:41.305Z","dependency_job_id":"19815c81-9752-493e-8057-3b30602a9152","html_url":"https://github.com/SourasishBasu/ledgerly","commit_stats":null,"previous_names":["sourasishbasu/expense-tracker","sourasishbasu/ledgerly"],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SourasishBasu%2Fledgerly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SourasishBasu%2Fledgerly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SourasishBasu%2Fledgerly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SourasishBasu%2Fledgerly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SourasishBasu","download_url":"https://codeload.github.com/SourasishBasu/ledgerly/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240907853,"owners_count":19876691,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","aws","devops","docker","fastapi","gemini","goaccess","grafana","monitoring","postgresql","prometheus","python","traefik"],"created_at":"2025-01-09T10:37:30.537Z","updated_at":"2025-02-26T18:14:30.700Z","avatar_url":"https://github.com/SourasishBasu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![banner](./assets/banner.png)\n\u003ch1 align=\"center\"\u003eLedgerly\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  An open source, self hostable expense tracking platform built with NextJS, Python\u003cbr\u003e\n  and AWS for the Lambda functions and S3 object storage.\u003cbr\u003eManage and gain insights from your expenses.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#features\"\u003e\u003cstrong\u003eFeatures\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"#overview\"\u003e\u003cstrong\u003eOverview\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"#aws-setup\"\u003e\u003cstrong\u003eAWS Setup\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"#remote-backend-setup\"\u003e\u003cstrong\u003eBackend Setup\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"#usage\"\u003e\u003cstrong\u003eUsage\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"#monitoring\"\u003e\u003cstrong\u003eMonitoring\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"#authors\"\u003e\u003cstrong\u003eAuthors\u003c/strong\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Features\n\n- **Website**\n  - [NextJS](https://nextjs.org) App Router\n  - [Amazon Web Services](https://docs.aws.amazon.com/) for backend functionality with `EC2`\n  - Support for `S3` File Storage, and `Lambda` Container image based Functions\n  - Edge runtime-ready\n  \n- **AWS Infrastructure**\n  - [Amazon S3](https://aws.amazon.com/s3) Utilized for image storage.\n  - [AWS Lambda](https://aws.amazon.com/lambda) for processing JSON and filtering required data\n  - [Amazon EC2](https://aws.amazon.com/sns) for provisioning VM instances \n  - [Amazon ECR](https://aws.amazon.com/ecr) for privately hosting container images \n\n- **External**\n  - [Prometheus](https://prometheus.io/docs/introduction/overview/), [Grafana](https://grafana.com/docs/grafana/latest/) and [GoAccess](https://goaccess.io/) for extensive observability and monitoring of resources. \n  - [Gemini API](https://ai.google.dev/gemini-api/docs) for image to text extraction using Vision Model within free tier limits.\n  - [Github Actions](https://github.com/features/actions) CI pipelines to build, test and push application images from Github to various registries.\n  - [Traefik](https://doc.traefik.io/) acts as a dynamic reverse proxy and automatically manages SSL/TLS certificates\n\n### Tech Stack\n![NextJs](https://img.shields.io/badge/Nextjs-black?style=for-the-badge\u0026logo=nextdotjs\u0026logoColor=white)\n![Python](https://img.shields.io/badge/Python-blue?style=for-the-badge\u0026logo=python\u0026logoColor=white)\n![Uvicorn](https://img.shields.io/badge/uvicorn-E6526F.svg?style=for-the-badge\u0026logo=gunicorn\u0026logoColor=white)\n![EC2](https://img.shields.io/badge/ec2-orange?style=for-the-badge\u0026logo=amazon-ec2\u0026logoColor=white)\n![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge\u0026logo=docker\u0026logoColor=white)\n![ECR](https://img.shields.io/badge/ecr-f06611.svg?style=for-the-badge\u0026logo=square\u0026logoColor=white)\n![S3](https://img.shields.io/badge/S3-darkgreen?style=for-the-badge\u0026logo=amazon-s3\u0026logoColor=white)\n![Lambda](https://img.shields.io/badge/Lambda-FF9900?style=for-the-badge\u0026logo=aws-lambda\u0026logoColor=white)\n![Gemini](https://img.shields.io/badge/gemini-8E75B2?style=for-the-badge\u0026logo=google%20gemini\u0026logoColor=white)\n![GitHub Actions](https://img.shields.io/badge/github%20actions-%232671E5.svg?style=for-the-badge\u0026logo=githubactions\u0026logoColor=white)\n![Grafana](https://img.shields.io/badge/grafana-%23F46800.svg?style=for-the-badge\u0026logo=grafana\u0026logoColor=white)\n![Traefik](https://img.shields.io/badge/Traefik-%2300ADD8.svg?style=for-the-badge\u0026logo=go\u0026logoColor=white)\n![Ansible](https://img.shields.io/badge/ansible-%231A1918.svg?style=for-the-badge\u0026logo=ansible\u0026logoColor=white)\n![Prometheus](https://img.shields.io/badge/Prometheus-E6522C?style=for-the-badge\u0026logo=Prometheus\u0026logoColor=white)\n\n## Overview\n\u003cimg alt=\"AWS Architecture\" src=\"./assets/arch.png\"\u003e\n\n- The **backend** consists of 3 main services being the **Python based REST API** developed using **FastAPI** for serving requests, performing CRUD operations, a **RDS Postgres** database for data storage and retrieval and a **S3 Bucket** for image storage and hosting.\n- The other 5 services consist of **Traefik** as reverse proxy, utilized for automatic SSL provision, log creation and **Prometheus**, **Grafana** for resource usage collection and visualization using custom **log exporters**.\n- All of these services are run using **Docker** containers to ensure availability and performance.\n\n## Project Structure\n\n```\n.\n├── README.md\n├── assets\n│   ├── arch.png\n│   ├── banner.png\n│   ├── er.png\n│   ├── goaccess.png\n│   ├── grafana.png\n│   ├── inboundrules.png\n│   └── lambda-uri.png\n├── deployment\n│   ├── ansible\n│   │   ├── inventory\n│   │   ├── playbook.yml\n│   │   └── roles\n│   │       ├── common\n│   │       │   └── tasks\n│   │       │       └── main.yml\n│   │       ├── docker\n│   │       │   └── tasks\n│   │       │       └── main.yml\n│   │       ├── files\n│   │       │   └── tasks\n│   │       │       └── main.yml\n│   │       └── tools\n│   │           └── tasks\n│   │               └── main.yml\n│   ├── docker-compose.yml\n│   ├── grafana\n│   │   └── datasources.yml\n│   └── prometheus\n│       └── prometheus.yml\n└── services\n    ├── backend\n    │   ├── Dockerfile\n    │   ├── app\n    │   │   ├── __init__.py\n    │   │   ├── app.py\n    │   │   ├── core\n    │   │   │   ├── __init__.py\n    │   │   │   ├── config.py\n    │   │   │   ├── db.py\n    │   │   │   └── utils.py\n    │   │   ├── db\n    │   │   │   ├── __init__.py\n    │   │   │   └── models.py\n    │   │   ├── main.py\n    │   │   └── routers\n    │   │       ├── __init__.py\n    │   │       ├── auth.py\n    │   │       ├── deps.py\n    │   │       └── receipts.py\n    │   ├── pyproject.toml\n    │   └── uv.lock\n    ├── ledgerly.sql\n    └── receipt-ocr\n        ├── Dockerfile\n        └── app.py\n\n21 directories, 36 files\n```\n\nPrimary Services:\n- `services/backend:` This subdirectory consists of the primary backend API for the entire application facilitating user registration, login, image upload and CRUD operations with the database.\n- `services/receipt-ocr:` This subdirectory consists of the helper application which extracts relevant data using an LLM from user submitted receipt images and adds these records into the database.\n- `monitoring/:` This folder consists of all configurations required for the monitoring systems in place.\n\n# AWS Setup\n\n\u003e [!NOTE]  \n\u003e The default architecture is based on AWS services, however all of the services and tooling can be setup within any other cloud platform of choice or self hosted locally as well if required.\n\nAll the services except S3 must be setup within the same region in AWS. \n\nAll the services and configurations below adhere to the `AWS Free Tier` for a brand new account upto 12 months.\n\n## ECR\n\nAWS Elastic Container Registry will be used to host the container images to be run on AWS Lambda.\n\n\u003e [!NOTE]  \n\u003e AWS Lambda can only run images hosted on private registries in ECR.\n\n- Go to Private registry \u003e Repositories \u003e Create Repository.\n- Provide a repo namespace and name and keep the rest as default\n- Once the repository is created, select it from the Repositories and go to Actions \u003e Permissions.\n- Under Permissions click on Edit Policy JSON and paste the below policy.\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"LambdaECRImageRetrievalPolicy\",\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"Service\": \"lambda.amazonaws.com\"\n      },\n      \"Action\": [\n        \"ecr:BatchGetImage\",\n        \"ecr:GetDownloadUrlForLayer\"\n      ]\n    }\n  ]\n}\n\n```\n- On the Repositories tab and note View push commands for the respective platform to use with Docker during local development.\n- Build the image for `services/receipt-ocr` using Docker locally with the tag `latest` and push it to the ECR repository just created.\n\n## RDS\n\nAWS RDS with PostgreSQL will serve as the primary database for storing user and receipt data for our application.\n\n- Go to Databases \u003e Create Database \u003e Standard Create.\n- Select PostgreSQL. Under Engine Version, choose PostgreSQL 17.2-R1.\n- Under Templates, choose Free Tier.\n- Under Settings, provide a name for the database under DB Instance identifier. \n- Create the Master Username and Master Password.\n- Under Connectivity, choose Yes under Public Access\n- Choose Create new under VPC Security Group (firewall) and provide a name.\n\nDatabase Connection Endpoint: `[database-name].abcdef123456.[region].rds.amazonaws.com`\n\n\n## Lambda\n\nAWS Lambda will be used for running the container image for the `services/receipt-ocr` application in a serverless environment.\n\n- Create a Function \u003e Container Image.\n- Provide a function name. Under Container Image URI \u003e Browse Images choose the ECR repository created earlier and select the image tagged as latest.\n\n  \u003cimg alt=\"Lambda URI\" src=\"./assets/lambda-uri.png\"\u003e\n\n- Under architecture choose x86_64 and leave the rest as default. Choose Create function.\n\n### Configuration\n\nUnder General configuration change the defaults as below:\n  - `Memory` - 512MB\n  - `Timeout` - 30 sec\n\n### Permissions\n\n- Under Permissions \u003e Execution Role Name, click the role to redirect to IAM. \n- In the IAM console, under Permission Policies \u003e Add Permissions \u003e Create Inline Policies.\nSwitch to JSON mode and paste the below policy. Then click Create.\n\n  ```json\n  {\n      \"Version\": \"2012-10-17\",\n      \"Statement\": [\n          {\n              \"Sid\": \"VisualEditor0\",\n              \"Effect\": \"Allow\",\n              \"Action\": [\n                  \"s3:GetObjectAcl\",\n                  \"s3:GetObject\",\n                  \"s3:GetObjectAttributes\"\n              ],\n              \"Resource\": [\n                  \"arn:aws:s3:*:replace-account-id:accesspoint/*\",\n                  \"arn:aws:s3:::replace-bucket-name/*\"\n              ]\n          }\n      ]\n  }\n  ```\n- Again, go to Add Permissions \u003e Attach policies. \n- Choose `AWSLambdaVPCAccessExecutionPermissions` from the list \u003e Add permissions. \n\n### Environment\n\nGo to Environment Variables \u003e Edit. Add the following variables required by the application.\n\n- `API_KEY` - [Gemini API KEY]\n- `DB_HOST` - [Connection String for the Database (RDS/Docker/etc)]\n- `DB_PASS` - [Database Password]\n- `DB_USER` - [Database Username]\n- `S3_ENDPOINT` - [S3 Bucket URL of the form https://bucket-name.s3.region.amazonaws.com/]\n\n## S3\n\nAWS S3 will be utilized for encrypted storage and hosting of user's receipt images.\n\n- Create a new bucket. Choose a name and select ACLs enabled under Object Ownership.\n- Under Block all public settings for this bucket, disable Block All public Access.\n- After bucket creation, go to Properties \u003e Event Notifications \u003e Create Event Notification.\n- Provide a name and under Event Types, choose All object create events.\n- Under Destination select Lambda function \u003e Choose from your Lambda functions \u003e Select the function created in \u003ca href=\"#lambda\"\u003eLambda\u003c/a\u003e section.\n\n## EC2\n\nAWS EC2 will be used for provisioning a virtual machine for running Docker containers of the application and various services.\n\nGo to Instances \u003e Launch Instances. Create an EC2 instance with the following specs:\n\n- `OS Image:` Ubuntu Server 24.04 LTS (64-bit x86)\n- `Instance Type:` t2.micro\n- `Key Pair:` Create a new .pem key to use with OpenSSH locally\n- `VPC:` default\n- `Security Group:` Allow SSH from Anywhere\n- `Storage:` 15GB\n\nOnce the Instance is Running and passed all system reachability checks, select Instance \u003e Security \u003e Security Details.\n\n### Inbound Rules\n\n- Go to Security Groups \u003e Inbound Rules \u003e Edit Inbound Rules.\n- Add the below rules and click Save Rules.\n\n  \u003cimg alt=\"Inbound Rules\" src=\"./assets/inboundrules.png\"\u003e\n\n### IAM Role\n\n- Go to Instances \u003e Actions \u003e Security \u003e Modify IAM Role \u003e Create New IAM Role.\n- Under Create Role choose AWS Service as Trusted Entity Type and EC2 under Use case.\n- Under Permissions select `AmazonS3FullAccess`. Provide a role name and choose Create Role.\n- Go back to Modify IAM Role \u003e Choose the Role just created above from IAM role dropdown \u003e Click Update IAM role.\n---\n\n# Remote Backend Setup\n\n1. Clone the repository locally. Add the `.env` file into the `deployment` directory.\n\n```bash\ngit clone https://github.com/sourasishbasu/ledgerly.git\ncd ledgerly/deployment\ntouch .env\n```\n2. Install Ansible.\n\n```bash\nsudo apt install ansible\n```\n\n3. Run the playbook at `deployment/ansible/playbook.yml` to setup the VM for deployment.\n\n```bash\ncd ansible\nansible-playbook -i inventory playbook.yml\n```\n\n## Database\n\nThe default architecture utilizes the free `AWS RDS t4g.micro` instance for hosting the Postgres database. As an alternative a `postgres` container can also be run within `Docker` on an `EC2` machine for the same utility. \n\n### Default Credentials\n`username` - user\n\n`pass` - pass\n\n`DB Name` - testdb\n\n`port` - 5432\n\n`host` - EC2 public IP or RDS Connection URL\n\n### Creating tables\n\n\u003cimg alt=\"AWS Architecture\" src=\"./assets/er.png\"\u003e\n\nCopy the contents of `services/ledgerly.sql` into the SQL Query Editor within any database tool used for connecting to the postgres container.\n\n## Traefik\n\nTraefik serves as a **reverse proxy** to route HTTP requests to the appropriate backend services. \n\n\n- Traefik automatically **detects and routes traffic** to Docker services. Services must explicitly enable routing via labels, ensuring only intended services are exposed within the `Docker Compose` specification.\n\n- **Web traffic** is routed through an HTTP entrypoint on port 80. The API service listens internally on port 5000, which Traefik uses to forward incoming requests.\n\n- **Access logs** are enabled and stored in the ./logs/traefik-access.log file, used by `GoAccess` to create the dashboard\n\n- Handle **HTTPS** for the application by integrating with `Let's Encrypt` for automatic certificate management, simplifying secure communication for all exposed services.\n\n## Monitoring\n\nThe Ledgerly backend includes a comprehensive monitoring setup using metrics' exporters, Prometheus and Grafana. This allows user to collect metrics, visualize data, and set up alerts for your application.\n\n\u003e [!NOTE]  \n\u003e Due to infrastructure limitations, all monitoring tools are being setup on the same machine as the application services however this deviates from best practices and is not recommended in production.\n\n### Exporters\n\nServices which collect and expose system and application metrics from specific resources or environments, making them available in a format that monitoring and observability systems such as Prometheus can scrape and process.\n\n- **Node Exporter**: Used for collectin hardware and OS-level metrics from a Linux server such as Disk I/O, CPU, Memory usage etc.\n\n- **cAdvisor**: Used for monitoring resource usage and performance metrics for containers.\n\n### Prometheus\n\nPrometheus is used to scrape and store metrics from various endpoints. The configuration for Prometheus is located in `monitoring/prometheus/prometheus.yaml`.\n\n```bash\nhostname -I # Host Private IP\n```\n\nReplace EC2 instance's private IP into `backend/prometheus/prometheus.yml` under `node_exporter` and `cAdvisor` job's targets.\n\n```yaml\nscrape_configs:\n  - job_name: node_exporter\n    static_configs:\n      - targets: [\"\u003cinstance-private-ip\u003e:9100\"]\n```\n\n### Grafana\n\nGrafana is used to visualize the metrics collected by Prometheus. The configuration for Grafana datasources is located in `monitoring/grafana/datasources.yml`.\n\n- Login to Grafana website. Go to Dashboards \u003e New \u003e Import\n- Go to Import Dashboard with ID. Enter `16310` \u003e Load\n- Select Main (Prometheus) as the Data Source if prompted.\n\n### Default Credentials\n`username` - admin\n\n`pass` - projectwing\n\nGrafana Dashboard URL: `http://\u003cinstance-public-ip\u003e:3000`\n\n\u003cimg alt=\"AWS Architecture\" src=\"./assets/grafana.png\"\u003e\n\n### GoAccess\n\nGenerates an extremely detailed real time HTML report of web traffic of an API such as visitors, hits, geolocation etc.\n\n#### Credentials\n\n`username` - admin\n\n`pass` - projectwing\n\nGoAccess Dashboard URL: `http://\u003cinstance-public-ip\u003e:7880`\n\n\u003cimg alt=\"AWS Architecture\" src=\"./assets/goaccess.png\"\u003e\n\n# Usage\n\n**Set up AWS credentials**:\n   Ensure you have your AWS credentials configured. You can do this by setting environment variables or using the AWS CLI.\n\nRun the containers with `Docker Compose`.\n\n```\ndocker compose up --build --pull missing -d\n```\n\nOnce the Docker container is up, API will be accessible at `http://\u003cinstance-public-ipv4-dns\u003e`.\n\n\u003e [!NOTE]\n\u003e Detailed API docs along with examples and template data can be found at [Scalar Docs](https://expense-tracker.apidocumentation.com/).\n\n\n## Expected Result\n\n```bash\n$ docker compose ps\nNAME            IMAGE                                              COMMAND                  SERVICE         CREATED        STATUS                    PORTS\napi             ghcr.io/sourasishbasu/expense-tracker-api:latest   \"fastapi run app/mai…\"   api             24 hours ago   Up 10 minutes (healthy)   5000/tcp\ncadvisor        gcr.io/cadvisor/cadvisor:latest                    \"/usr/bin/cadvisor -…\"   cadvisor        24 hours ago   Up 10 minutes (healthy)   0.0.0.0:8090-\u003e8080/tcp, [::]:8090-\u003e8080/tcp\ngoaccess        xavierh/goaccess-for-nginxproxymanager:latest      \"tini -- /goan/start…\"   goaccess        24 hours ago   Up 10 minutes             0.0.0.0:7880-\u003e7880/tcp, :::7880-\u003e7880/tcp\ngrafana         grafana/grafana-oss:11.4.0                         \"/run.sh\"                grafana         24 hours ago   Up 10 minutes             0.0.0.0:3000-\u003e3000/tcp, :::3000-\u003e3000/tcp\nnode_exporter   quay.io/prometheus/node-exporter:latest            \"/bin/node_exporter …\"   node_exporter   24 hours ago   Up 10 minutes\nprometheus      prom/prometheus:latest                             \"/bin/prometheus --c…\"   prometheus      24 hours ago   Up 10 minutes             0.0.0.0:9090-\u003e9090/tcp, :::9090-\u003e9090/tcp\ntraefik         traefik:latest                                     \"/entrypoint.sh --ap…\"   traefik         24 hours ago   Up 10 minutes             0.0.0.0:80-\u003e80/tcp, :::80-\u003e80/tcp, 0.0.0.0:8080-\u003e8080/tcp, :::8080-\u003e8080/tcp\n```\n\n## Authors\n\nThis project was made for Project Wing 2025 by [MLSAKIIT](https://mlsakiit.com/).\n\n- Sourasish Basu ([@SourasishBasu](https://github.com/SourasishBasu)) - [MLSA KIIT](https://mlsakiit.com)\n\n## Version\n| Version | Date          \t\t| Comments        |\n| ------- | ------------------- | --------------- |\n| 1.0     | Dec 29th, 2024   | Initial release |","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsourasishbasu%2Fledgerly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsourasishbasu%2Fledgerly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsourasishbasu%2Fledgerly/lists"}