{"id":26232723,"url":"https://github.com/fahimfba/postgresql-v2-bills-maxint-app","last_synced_at":"2026-05-15T12:35:18.897Z","repository":{"id":279975179,"uuid":"939798524","full_name":"FahimFBA/postgreSQL-v2-bills-maxint-app","owner":"FahimFBA","description":"Solution V2 to given project from maxint-app","archived":false,"fork":false,"pushed_at":"2025-03-02T07:54:52.000Z","size":7305,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-12T21:40:01.786Z","etag":null,"topics":["csv","eda","finance","notebook-jupyter","python"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FahimFBA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-27T05:58:59.000Z","updated_at":"2025-03-02T07:54:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"4ed0318c-e95d-4775-9fe9-55760b1d38ea","html_url":"https://github.com/FahimFBA/postgreSQL-v2-bills-maxint-app","commit_stats":null,"previous_names":["fahimfba/postgresql-v2-bills-maxint-app"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FahimFBA/postgreSQL-v2-bills-maxint-app","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FahimFBA%2FpostgreSQL-v2-bills-maxint-app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FahimFBA%2FpostgreSQL-v2-bills-maxint-app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FahimFBA%2FpostgreSQL-v2-bills-maxint-app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FahimFBA%2FpostgreSQL-v2-bills-maxint-app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FahimFBA","download_url":"https://codeload.github.com/FahimFBA/postgreSQL-v2-bills-maxint-app/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FahimFBA%2FpostgreSQL-v2-bills-maxint-app/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33067471,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T11:35:32.926Z","status":"ssl_error","status_checked_at":"2026-05-15T11:35:31.362Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","eda","finance","notebook-jupyter","python"],"created_at":"2025-03-13T00:38:13.663Z","updated_at":"2026-05-15T12:35:18.892Z","avatar_url":"https://github.com/FahimFBA.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PostgreSQL V2 Bills Maxint App\n\n## Tasks/Goals\nThe entire goals of this project is located in the [tasks](./tasks/) directory.\n\n## Step by Step Approach to the Solution\n\n### Analysis of the Database\nThe database is a CSV file containing bank transaction data with columns: id, createdAt, externalId, type, amount, date, description, category, counterParty, recurring, tag, accountExternalId, and location.\n\n| Parameter         | Description                                                    |\n| ----------------- | -------------------------------------------------------------- |\n| id                | The unique identifier for the transaction.                     |\n| createdAt         | The timestamp when the transaction was created.                |\n| externalId        | The external identifier for the transaction.                   |\n| type              | The type of transaction (e.g., card_payment, digital_payment). |\n| amount            | The amount of the transaction.                                 |\n| date              | The date of the transaction.                                   |\n| description       | The description of the transaction.                            |\n| category          | The category of the transaction.                               |\n| counterParty      | The counterparty involved in the transaction.                  |\n| recurring         | A flag indicating whether the transaction is recurring.        |\n| tag               | A tag associated with the transaction.                         |\n| accountExternalId | The external identifier for the account.                       |\n| location          | The location of the transaction.                               |\n\n### EDA (Exploratory Data Analysis)\nThe EDA is done in the [notebooks](./notebooks/) directory.\n\n### Parameter Analysis\n\n#### id\n\nThe id parameter is a unique identifier for the transaction. It is a string data type. There is 100 unique values in the id column. It is the representation of the transaction.\n\n![id](./notebooks/id%20Types%20Count.png)\n\n#### createdAt\n\nThe createdAt parameter is the timestamp when the transaction was created. There is only 1 unique value in the createdAt column. Therefore, all of the transactions were created at the same time.\n\n![createdAt](./notebooks/createdAt%20Types%20Count.png)\n\n#### externalId\n\nThe externalId parameter is the external identifier for the transaction. It is a string data type. There is 100 unique values in the externalId column. It is the representation of the transaction.\n\n![externalId](./notebooks/externalId%20Types%20Count.png)\n\n#### type\n\nThe type parameter is the type of transaction (e.g., card_payment, digital_payment). It is a string data type. There are 16 unique values in the type column. \n\n![type](./notebooks/type%20Types%20Count.png)\n\nThe maximum number of transactions is card_payment. The minimum number of transactions is charge.\n\n### amount\n\nThe amount parameter is the amount of the transaction. It is a float data type. There are 88 positive values and 12 negative amounts in the amount column.\n\n![amount](./notebooks/amount_distribution.png)\n\n### date\n\nThe date parameter is the date of the transaction. It is a string data type. There are 76 unique values in the date column.\n\n![date](./notebooks/date%20Types%20Count.png)\n\n\n### description\n\nThe description parameter is the description of the transaction. It is a string data type. There are 57 unique values in the description column.\n\n![description](./notebooks/description%20Types%20Count.png)\n\n### category\n\nThe category parameter is the category of the transaction. It is a string data type. There are 13 unique values in the category column.\n\n![category](./notebooks/category.png)\n\n### counterParty\n\nThe counterParty parameter is the counterparty involved in the transaction. It is a string data type. There are 57 unique values in the counterParty column.\n\n![counterParty](./notebooks/counterParty%20Types%20Count.png)\n\n### recurring\n\nThe recurring parameter is a flag indicating whether the transaction is recurring. It is a boolean data type. There are 0 unique values in the recurring column.\n\n### tag\n\nThe tag parameter is a tag associated with the transaction. It is a string data type. There are 3 unique values in the tag column.\n\n![tag](./notebooks/tag%20Types%20Count.png)\n\n### accountExternalId\n\nThe accountExternalId parameter is the external identifier for the account. It is a string data type. There are 3 unique values in the accountExternalId column.\n\n![accountExternalId](./notebooks/accountExternalId%20Types%20Count.png)\n\n### location\n\nThe location parameter is the location of the transaction. It is a string data type. There are 3 unique values in the location column.\n\n![location](./notebooks/location%20Types%20Count.png)\n\n\n## Recurring Identification Methodology\n\nThe methodology for identifying recurring transactions is located in the [docs](./docs/) directory.\n\n## Solution\n\nThis project processes bank transaction data to identify recurring bill payments and uploads the results to Supabase.\n\n## Setup\n\n1. Install the required dependencies:\n   ```\n   pip install -r requirements.txt\n   ```\n\n2. Set up your Supabase credentials in the `.env` file:\n   ```\n   SUPABASE_URL=your_supabase_project_url\n   SUPABASE_KEY=your_supabase_api_key\n   ```\n\n## Usage\n\n1. Set up the database structure in Supabase:\n   - Log in to your Supabase project.\n   - Navigate to the SQL Editor.\n   - Copy the contents of the `scripts/create_recurring_transactions_view.sql` file.\n   - Paste the SQL into the SQL Editor and execute it.\n   \n   This will create the necessary table and view in your Supabase project.\n\n2. Process the raw CSV file and generate recurring transactions:\n   ```\n   python scripts/process_transactions.py\n   ```\n   This will create a new file `tasks/recurring_transactions.csv` with the identified recurring transactions.\n\n3. Upload the processed data to Supabase:\n   ```\n   python scripts/upload_to_supabase.py\n   ```\n   This will upload the data from `tasks/recurring_transactions.csv` to your Supabase project.\n\n## SQL Execution Sequence\n\nThere is only one SQL file that needs to be executed:\n\n1. `scripts/create_recurring_transactions_view.sql`: This script creates the `recurring_transactions` table and the `recurring_transactions_view` view. It also grants the necessary permissions to the authenticated role.\n\nExecute this script in your Supabase SQL Editor before running the Python scripts to process and upload data.\n\n## Viewing Results\n\nAfter setting up the table and view, and uploading data, you can query the view in your Supabase project using:\n\n```sql\nSELECT * FROM recurring_transactions_view;\n```\n\nThis will display the recurring transactions with the following columns:\n- amount: The amount due for the bill\n- description: The description of the transaction\n- next_date: The predicted date for the next occurrence\n- last_date: The date of the last occurrence\n- first_date: The date of the first occurrence\n\nThe view is ordered by the next_date in ascending order.\n\n## Testing\n\nTo verify that your setup is working correctly, you can run the following tests in your Supabase SQL Editor:\n\n1. Insert test data:\n\n```sql\nINSERT INTO recurring_transactions (amount, description, next_date, last_date, first_date)\nVALUES \n(100.00, 'Test Subscription', '2023-04-01', '2023-03-01', '2023-01-01'),\n(50.50, 'Test Utility Bill', '2023-04-15', '2023-03-15', '2023-02-15');\n```\n\n2. Query the view to check the inserted data:\n\n```sql\nSELECT * FROM recurring_transactions_view;\n```\n\nYou should see two rows in the result, one for each inserted transaction, with the columns as described in the \"Viewing Results\" section.\n\n3. To clean up after testing, you can run:\n\n```sql\nDELETE FROM recurring_transactions WHERE description LIKE 'Test%';\n```\n\nThese tests will help you confirm that:\n- You can insert data into the recurring_transactions table.\n- The recurring_transactions_view is correctly set up and displaying data.\n- The view is ordering results by next_date as expected.\n\nIf you encounter any issues during testing, double-check that you've correctly executed the SQL script to create the table and view, and that you have the necessary permissions.\n\n## Project Structure\n\n- `docs/`: Contains project documentation\n  - `recurring_identification_methodology.md`: Describes the methodology for identifying recurring transactions\n  - `implementation_details.md`: Provides details on the implementation of the recurring transaction identification\n- `notebooks/`: Contains Jupyter notebooks for exploratory data analysis\n- `scripts/`: Contains Python scripts for processing and uploading data\n  - `process_transactions.py`: Identifies recurring transactions from raw data\n  - `upload_to_supabase.py`: Uploads processed data to Supabase\n  - `create_recurring_transactions_view.sql`: SQL script to create the view in Supabase\n- `tasks/`: Contains the raw transaction data and project goals\n\n## Implementation Details\n\n- The `process_transactions.py` script uses a scoring system to identify recurring transactions based on:\n  - Time interval regularity (weight: 0.3)\n  - Amount consistency (weight: 0.25)\n  - Description match (weight: 0.25, always 1 in the current implementation)\n  - Transaction type match (weight: 0.1)\n  - Category match (weight: 0.1)\n- The system uses a threshold of 0.4 to identify recurring transactions, balancing between sensitivity and specificity.\n- The process flow includes:\n  1. Loading and preprocessing the transaction data\n  2. Grouping transactions by description\n  3. Calculating scores for each group, including detailed debugging information\n  4. Filtering recurring transactions based on score (\u003e 0.4) and frequency (\u003e= 2 occurrences)\n  5. Predicting the next occurrence date for each recurring transaction\n  6. Applying a maximum future date limit (10 years from current date) for predicted next dates\n  7. Sorting the recurring transactions by next predicted date\n  8. Generating a new CSV with identified recurring transactions\n- The `upload_to_supabase.py` script uses the Supabase Python client to upload the processed data to your Supabase project.\n- The PostgreSQL view `recurring_transactions_view` provides an efficient way to query the recurring transactions data.\n\nFor more detailed information about the implementation, including future improvements and validation processes, please refer to the `docs/implementation_details.md` file.\n\n## Note\n\nMake sure to handle your Supabase credentials securely and not share them publicly.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffahimfba%2Fpostgresql-v2-bills-maxint-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffahimfba%2Fpostgresql-v2-bills-maxint-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffahimfba%2Fpostgresql-v2-bills-maxint-app/lists"}