{"id":20196261,"url":"https://github.com/hangrybear666/fiscalismia-backend","last_synced_at":"2025-03-03T08:14:03.968Z","repository":{"id":242557087,"uuid":"487078971","full_name":"hangrybear666/fiscalismia-backend","owner":"hangrybear666","description":"Fiscalismia is a Web Service for visualizing, analyzing, aggregating, importing and exporting personal finance data, such as variable and fixed costs, income, sales and investments.","archived":false,"fork":false,"pushed_at":"2024-09-26T11:44:11.000Z","size":4428,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-13T19:39:20.327Z","etag":null,"topics":["ci-cd-pipeline","docker","eslint","express","github-actions","jsonwebtoken","node-postgres","nodejs","postgresql-database","rest-api","snyk","supertest","typescript","winston-logger"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hangrybear666.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-29T18:49:56.000Z","updated_at":"2024-09-26T11:44:15.000Z","dependencies_parsed_at":"2024-06-03T19:21:58.255Z","dependency_job_id":"6714b98c-d316-4104-a1fb-98965c50f363","html_url":"https://github.com/hangrybear666/fiscalismia-backend","commit_stats":null,"previous_names":["hangrybear666/fiscalismia-backend"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hangrybear666%2Ffiscalismia-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hangrybear666%2Ffiscalismia-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hangrybear666%2Ffiscalismia-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hangrybear666%2Ffiscalismia-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hangrybear666","download_url":"https://codeload.github.com/hangrybear666/fiscalismia-backend/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241629769,"owners_count":19993710,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci-cd-pipeline","docker","eslint","express","github-actions","jsonwebtoken","node-postgres","nodejs","postgresql-database","rest-api","snyk","supertest","typescript","winston-logger"],"created_at":"2024-11-14T04:22:57.056Z","updated_at":"2025-03-03T08:14:03.948Z","avatar_url":"https://github.com/hangrybear666.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fiscalismia Backend\nBackground Service handling the Fiscalismia infrastructure.\nFiscalismia is a Web Service for visualizing, analyzing, aggregating, importing and exporting personal finance data. Data can consist of variable and fixed costs, income, sales and investments. Advanced capabilities are available for dynamically updating supermarket grocery deals via web automation scraping data from supermarket websites.\n\n## Technical Overview\n\nfiscalismia-backend consists of an express server running a REST API. Requests from the frontend are handled by the backend's REST API querying data from a postgres db. The database runs within a docker container for development. JWT tokens are used for authentication. Local Browser Storage for Session Cookies. The REST API is designed with full CRUD operations in mind, allowing for dynamic user input, sanitized before being commited to the db. In production, we use a cloud-hosted scale-to-zero PostgreSQL database on Neon.tech with a generous free tier. The backend is built in a continuous integration pipeline, tested, scanned for vulnerabilities and published as a docker image to a public docker registry for later deployment in your environment of choice.\n\n\n## Table of Contents\n\n- [Technologies](#technologies)\n- [DevOps Pipeline](#pipeline)\n- [Setup](#setup)\n- [Running](#running)\n- [Testing](#testing)\n- [Usage](#usage)\n- [Production \u0026 Deployment](#production)\n- [License](#license)\n\n## Technologies\n\n- **Github Actions:** CI/CD Pipeline for automating type checking, eslint analysis, REST API testing, vulnerability scanning, building, publishing and deploying.\n- **Docker:** The final build artifact from compiled src code is a docker image published to a registry and deployed in the cloud.\n- **TypeScript:** Statically typed JS with high strictness level and compile target ESNext. Mid-project Migration from plain JavaScript (ECMAScript 2016).\n- **Node.js:** A JavaScript runtime built on Chrome's V8 JavaScript engine, used for server-side development.\n- **Express Server:** A fast, unopinionated, minimalist web framework for Node.js, used to build the backend server.\n- **Supertest:** A testing library for HTTP assertions, employed for REST API testing to ensure the reliability of the server.\n- **JWT Auth:** JSON Web Token authentication is used for securing and verifying the authenticity of API requests.\n- **Snyk:** Static Code security analysis, dependency security analysis, monitoring and notifications on detected security issues.\n- **ESLint and Prettier:** Linter and Formatter for ensuring code quality and enforcing coding standards.\n- **Winston Logger:** A versatile logging library for Node.js, utilized for logging events and errors in the server.\n- **Nodemon/ts-node:** Hot Reload upon file changes of the server during development, enhancing the development workflow.\n\n## Pipeline\n\n1. **Triggers:**\n   - Runs on every push and pull request to the `main` branch.\n\n2. **Job: `test`**:\n   - **Steps:**\n     - Set up Node.js (v20.12.2), install dependencies and Snyk.\n     - Run type checks and ESLint analysis.\n     - Perform Snyk static code and dependency security analysis.\n     - Publish type check, ESLint, and Snyk reports as artifacts.\n     - Initialize a fresh Postgres database and seed with DDL/DML scripts.\n     - Run API tests against the pipeline database using `supertest`.\n\n3. **Job: `build`**:\n   - **Steps:**\n     - Build Backend Docker image.\n     - Publish Docker image to GHCR (TODO: Switch to AWS ECR)\n\n4. **Job: `deploy`**:\n   - **Steps:**\n     - TODO: Deploy on Hetzner Self Managed Kubernetes via ArgoCD (K3s)\n     - TODO: Setup AWS S3 bucket for file upload persistence\n\n## Setup\n\n**Dependencies**\n\n1. **Node.js:** Ensure that Node.js is installed on your local machine, with a minimum version of 20.9. You can download Node.js via Node Version Switcher [here](https://github.com/jasongin/nvs) or directly from the source [here](https://nodejs.org/).\n\n2. **Docker \u0026 Docker Compose** Ensure that Docker is installed in your local development environment. Get Docker [here](https://docs.docker.com/get-docker/) and Docker Compose [here](https://docs.docker.com/compose/install/).\n\n3. **Clone the Repository:**\n   git clone https://github.com/your-username/fiscalismia-backend.git\n\n**Installation**\n\n1. **Navigate to the Project Folder:**\n\n   ```bash\n   cd fiscalismia-backend\n   ```\n\n2. **Install Dependencies:**\n\n   ```bash\n   npm install\n   ```\n\n3. **Environment Variables:**\n\n   Store the `.env` in the root folder of `fiscalismia-backend`. Ensure that you never upload this file to Git, as it contains sensitive information!\n   `DB_CONNECTION_URL` is only required in production, as this points to the cloud hosted db.\n   ```bash\n   # App \u0026 Server\n   JWT_SECRET=xxx\n   DB_CONNECTION_URL=\n   FRONTEND_PORT=3001\n   BACKEND_PORT=3002\n   HOST_ADDRESS=localhost\n\n   # Local Docker DB Setup\n   POSTGRES_USER=fiscalismia_api\n   POSTGRES_HOST=fiscalismia-postgres\n   POSTGRES_DB=fiscalismia\n   POSTGRES_PASSWORD=xxx\n   POSTGRES_PORT=5432\n\n   # PIPELINE INTEGRATIONS\n   SNYK_TOKEN=\n   ```\n\n4. **Github Secrets:**\n\n   Set up Github Secrets in your Repository Settings, for the pipeline to run successfully. These can and should be the same as in your `.env` file.\n   ```bash\n   JWT_SECRET\n   POSTGRES_PASSWORD\n   SNYK_TOKEN\n   ```\n\n## Running\n\n1. **Option 1: Docker Compose**\n\n   To run the entire stack in development mode\n\n   ```bash\n   docker compose build\n   docker compose up\n   ```\n\n2. **Option 2: Locally**\n\n   Development Database\n   ```bash\n   docker compose up fiscalismia-postgres\n   ```\n\n   Run only the backend locally pointing to local db defined in `.env` file keys.\n   ```bash\n   npm run server\n   ```\n\n3. **Option 3: Docker**\n\n   Development Database\n   ```bash\n   docker compose up fiscalismia-postgres\n   ```\n\n   \u003cdetails open\u003e\n   \u003csummary\u003e\u003cb\u003eDocker Run (Linux Syntax)\u003c/b\u003e\u003c/summary\u003e\n\n   Run only the backend dev container pointing to local db defined in `.env` file keys.\n   ```bash\n   docker build --pull --no-cache --rm -f \"Dockerfile.dev\" -t fiscalismia-backend-dev:latest \".\"\n   docker run -v $PWD/src:/fiscalismia-backend/src -v $PWD/public:/fiscalismia-backend/public --env-file .env --net fiscalismia-network --rm -it -p 3002:3002 --name fiscalismia-backend-dev fiscalismia-backend-dev:latest\n   ```\n   \u003c/details\u003e\n   \u003cdetails closed\u003e\n   \u003csummary\u003e\u003cb\u003eDocker Run (Windows Syntax)\u003c/b\u003e\u003c/summary\u003e\n\n   Run only the backend dev container pointing to local db defined in `.env` file keys.\n   ```bash\n   docker build --pull --no-cache --rm -f \"Dockerfile.dev\" -t fiscalismia-backend-dev:latest \".\"\n   docker run -v %cd%\\src:/fiscalismia-backend/src -v %cd%\\public:/fiscalismia-backend/public --env-file .env --net fiscalismia-network --rm -it -p 3002:3002 --name fiscalismia-backend-dev fiscalismia-backend-dev:latest\n   ```\n   \u003c/details\u003e\n\n   ------\n\n   \u003cdetails closed\u003e\n   \u003csummary\u003e\u003cb\u003eProduction build with cloud database (Linux Syntax)\u003c/b\u003e\u003c/summary\u003e\n\n   NOTE: `DB_CONNECTION_URL` to remote postgres must be set in `.env` file.\n\n   ```bash\n   docker build --pull --no-cache --rm -f \"Dockerfile\" -t fiscalismia-backend:latest \".\"\n   docker run --network fiscalismia-network -v $PWD/public:/fiscalismia-backend/public --env-file .env --net fiscalismia-network --rm -it -p 3002:3002 --name fiscalismia-backend fiscalismia-backend:latest\n   ```\n   \u003c/details\u003e\n\n   ------\n\n   \u003cdetails closed\u003e\n   \u003csummary\u003e\u003cb\u003eProduction build with cloud database (Windows Syntax)\u003c/b\u003e\u003c/summary\u003e\n\n   NOTE: `DB_CONNECTION_URL` to remote postgres must be set in `.env` file.\n\n   ```bash\n   docker build --pull --no-cache --rm -f \"Dockerfile\" -t fiscalismia-backend:latest \".\"\n   docker run --network fiscalismia-network -v %cd%\\public:/fiscalismia-backend/public --env-file .env --net fiscalismia-network --rm -it -p 3002:3002 --name fiscalismia-backend fiscalismia-backend:latest\n   ```\n   \u003c/details\u003e\n\n## Testing\n\n**All tests generate report files in the `reports/` subdirectory.**\n\n1. **REST API tests /w supertest:**\n\n   We use supertest for testing the REST API which can be executed via running the test script from a second console while the dev database is up and running.\n\n   ```bash\n   npm run test\n   ```\n\n2. **Static Code Analysis**\n\n   Eslint is used to ensure a consistent codebase adhering to certain coding standards configured in `.eslintrc.js`.\n   Typecheck runs the Typescript Compiler which is configured with high strictness in `tsconfig.json`.\n\n   ```bash\n   npm run typeCheck\n   npm run eslintAnalysis\n   ```\n\n3. **Snyk Security Analysis**\n\n   `SNYK_TOKEN` has to be set in `.env` file.\n   Get one for free by creating a snyk account [here](https://app.snyk.io/login)\n\n   Once per workspace\n   ```bash\n   # with snyk cli installed\n   snyk auth SNYK_TOKEN\n   # without snyk cli installed\n   npx snyk auth SNYK_TOKEN\n   ```\n\n   Vulnerability scanning of both the codebase, especially relevant for issues such as SQL Injection and XSS(Cross Site Scripting).\n   ```bash\n   npm run snykCodeAnalysis\n   npm run snykDependencyAnalysis\n   ```\n\n## Usage\n\nOnce the server is up and running, it will be ready to handle API requests from your frontend application or REST API Client at http://localhost:3002/api/fiscalismia\n\n**Accessing the Database via CLI**\n\n   ```bash\n   docker exec -it YOUR_DOCKER_HASH sh\n   psql -U $POSTGRES_USER -d $POSTGRES_DB -h $POSTGRES_HOST -p $POSTGRES_PORT\n   ```\n\n**Accessing the Database via DB Client**\n\n   You can run DBeaver or a database client of your choice and use the .env file connection strings\n\n   `POSTGRES_USER | POSTGRES_HOST | POSTGRES_DB | POSTGRES_PASSWORD | POSTGRES_PORT`\n\n**Accessing Protected Routes**\n\n   A valid user is required.\n   Default credentials are\n   ```admin changeit```\n\n   Alternatively use a POST request to http://localhost:3002/api/fiscalismia/um/credentials carrying a user object.\n\n   ```json\n   // the user has to be whitelisted in the db table username_whitelist\n   { \"username\": \"yourUser\", \"email\": \"user@mailserver.domain\", \"password\": \"yourPassword\" }\n   ```\n\n   User can also be created easily via the **frontend login mask**.\n\n   All important routes are protected and require an Authorization header reading 'Bearer token' where token is a jwt-token received after posting valid user credentials to http://localhost:3002/api/fiscalismia/um/login\n\n   result:\n   `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7InVzZXJJZCI6MSwidXNlck5hbWUiOiJhZG1pbiIsInVzZXJFbWFpbCI6ImhlcnBfZGVycEBnbWFpbC5pbyJ9LCJpYXQiOjE3MDczMDk4MTgsImV4cCI6MTcwNzM5NjIxOH0.RkxSnXZZAwHIi-QPR57KtLiVdeRn3FybfPtCosM4rqY`\n\n   ```bash\n   GET http://localhost:3002/api/fiscalismia/ HTTP/1.1\n   Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7InVzZXJJZCI6MSwidXNlck5hbWUiOiJhZG1pbiIsInVzZXJFbWFpbCI6ImhlcnBfZGVycEBnbWFpbC5pbyJ9LCJpYXQiOjE3MDczMDk4MTgsImV4cCI6MTcwNzM5NjIxOH0.RkxSnXZZAwHIi-QPR57KtLiVdeRn3FybfPtCosM4rqY\n   ```\n\n## Production\n\n1. **Set up Cloud DB**\n\n   I recommend using scale-to-zero cloud-native postgresql with a generous free tier at [neon.tech](https://console.neon.tech/)\n\n2. **Build docker image**\n\n   *Hint: %cd% backslash file path command is unique to windows, in unix based systems, replace it with $PWD forward slash*\n   ```\n   docker build --pull --no-cache --rm -f \"Dockerfile\" -t fiscalismia-backend:latest \".\"\n   docker run -v %cd%\\public:/fiscalismia-backend/public --env-file .env --rm -it -p 3002:3002 --name fiscalismia-backend fiscalismia-backend:latest\n   ```\n\n3. **Provisioning of AWS S3 Bucket for Image Upload**\n\n   Todo\n\n4. **Automatic Docker Build and Push to private AWS ECR in Github Actions Pipeline**\n\n   Todo\n\n5. **Deployment to public AWS EC2 Instance via Github Actions Pipeline**\n\n   Todo\n\n6. **Configure and Initialize Database**\n\n   The development database initialization script can be used for this purpose, specifically `database/pgsql-ddl.sql`.\n\n   Bulk data insertion is handled via TSV files in a specific format in the admin area.\n   The essentials can be gathered from the first lines of `database/pgsql-dml.sql` especially the username whitelist and username, email, password in the `um_` tables.\n\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhangrybear666%2Ffiscalismia-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhangrybear666%2Ffiscalismia-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhangrybear666%2Ffiscalismia-backend/lists"}