{"id":18597604,"url":"https://github.com/lealre/excel-schema-validator","last_synced_at":"2026-04-02T18:53:34.264Z","repository":{"id":228252335,"uuid":"752839648","full_name":"lealre/excel-schema-validator","owner":"lealre","description":"Web application that validates a specific Excel schema and stores the data in a PostgresSQL Database.","archived":false,"fork":false,"pushed_at":"2025-05-10T13:55:32.000Z","size":10879,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-10T14:41:35.792Z","etag":null,"topics":["ci","docker","pandas","postgresql","pydantic","streamlit"],"latest_commit_sha":null,"homepage":"https://excel-schema-validator.streamlit.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lealre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-04T23:35:06.000Z","updated_at":"2025-05-10T13:55:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"21d659aa-8883-4d40-8b77-225755905d14","html_url":"https://github.com/lealre/excel-schema-validator","commit_stats":null,"previous_names":["lealre/excel-schema-validator"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fexcel-schema-validator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fexcel-schema-validator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fexcel-schema-validator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lealre%2Fexcel-schema-validator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lealre","download_url":"https://codeload.github.com/lealre/excel-schema-validator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254538633,"owners_count":22087896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","docker","pandas","postgresql","pydantic","streamlit"],"created_at":"2024-11-07T01:28:45.686Z","updated_at":"2025-12-30T23:04:58.355Z","avatar_url":"https://github.com/lealre.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Excel Schema Validator\n\nThis project consists of a web application that validates a specific Excel schema and stores the data in a PostgreSQL database. The predefined schema is the contract schema, and the application validates it from Excel files using Pydantic, a Python validation library.\n\nAdditionally, this project includes Continuous Integration (CI) for every Pull Request made using GitHub Actions. The schema contract is tested, along with functional and integration tests in each PR. All tests are performed using the pytest library.\n\n\u003cimg src=\"media/demo.gif\" width = 1000 /\u003e\n\n## Context\n\nThe context of this project reflects a common scenario within organizations, where various departments routinely exchange data using Excel files that adhere to a predefined table schema. However, errors or inconsistencies, such as missing data or unexpected data formats, can disrupt workflows and cause delays in projects. \n\nBy defining a clear data contract that outlines the expected structure, the application can identify any deviations from this contract.\n\nThe contract schema that this specific project aims to validate is as follows:\n\n| Column   | Data Type                                 |\n|----------|-------------------------------------------|\n| email    | Email structure (text@text.com)           |\n| date     | Date (yyyy-mm-dd)                         |\n| value    | Numeric Positive                          |\n| product  | Text                                      |\n| amount   | Numeric Positive Integer                  |\n| category | Only 3 options (*category1*, *category2* or *category3*) |\n\nThe app functionality can be tested with the 3 files inside the [data](https://github.com/lealre/excel-schema-validator/tree/master/data) folder.\n\n## How it works\n\nThe user can upload Excel files in the app, and if the schema differs from the contract, the application will display a message indicating where the data schema differs from the contract schema, highlighting the specific rows and columns where the data does not conform to the agreed-upon schema.\n\nIf the schema from the uploaded file passes validation, the app gives the option to store the data in the database\n\n\u003cimg src=\"media/app-diagram.png\" width = 1000 /\u003e\n\n### Project Folder Structure\n\n```\n├── Dockerfile\n├── README.md\n├── data\n│   ├── aditional_column.xlsx\n│   ├── correct.xlsx\n│   └── wrong.xlsx\n├── media\n│   ├── app-diagram.png\n│   └── demo.gif\n├── pyproject.toml\n├── requirements.txt\n├── source\n│   ├── app.py\n│   ├── backend.py\n│   ├── contract.py\n│   └── frontend.py\n├── sql\n│   └── create.sql\n└── tests\n    ├── __init__.py\n    ├── test_app.py\n    ├── test_contract.py\n    └── test_integration.py\n```\n\n## How to run this project\n\nAll the steps from here were intended to a `bash` terminal.\n\nTo properly run this project locally, you need to connect the app to your own PostgresSQL database. You can do this by following the steps below:\n\n1.1 - Clone the repository locally:\n```bash\ngit clone https://github.com/lealre/excel-schema-validator.git\n```\n\n1.2 - Access the project folder:\n```bash\ncd excel-schema-validator\n```\n\n1.3 - Create the `.env` file in the root folder, passing the respective keys from your own PostgresSQL Database:\n```bash\necho \"POSTGRES_USER=\u003cyour-database-keys\u003e\" \u003e\u003e .env\necho \"POSTGRES_PASSWORD=\u003cyour-database-keys\u003e\" \u003e\u003e .env\necho \"POSTGRES_HOST=\u003cyour-database-keys\u003e\" \u003e\u003e .env\necho \"POSTGRES_PORT=\u003cyour-database-keys\u003e\" \u003e\u003e .env\necho \"POSTGRES_DB=\u003cyour-database-keys\u003e\" \u003e\u003e .env\n```\n\n1.4 - Make sure `.env` file is included in `.gitignore`.\n\nFrom here, we have two options to run the project:\n\n- Configuring the local setup\n- Using Docker\n\n### Local Setup\n\nTo configure the local setup, we will use `pyenv` (version 2.3.35).\n\nOnce we have cloned the repository and accessed the root folder (steps 1.1 and 1.2), we follow the steps below:\n\n2.1 - Install Python version 3.11.5:\n```bash\npyenv install 3.11.5\n```\n\n2.2 - Set the local version of Python to 3.11.5:\n```bash\npyenv local 3.11.5\n```\n\n2.3 - Create a virtual enviroment and activate it:\n```bash\npython -m venv .venv\nsource .venv/bin/activate\n```\n\n2.4 - Install all dependencies from the project:\n```bash\npip install -r requirements.txt\n```\n\n2.5 - Run the project:\n```bash\ntask run\n```\n\n2.6 -  (Optional) Run the tests:\n```bash\ntask test\n```\n\n### Using Docker\n\nOnce we have cloned the repository and accessed the root folder (steps 1.1 and 1.2), we follow the steps below:\n\n3.1 -  Build the image:\n```bash\ndocker build -t excel-schema .\n```\n\n3.2-  Create the container:\n```bash\ndocker run -d -p 8051:8051 --name excel-schema-container excel-schema\n```\n\n3.3 - Access link in your browser\n\n[http://localhost:8051/](http://localhost:8051/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flealre%2Fexcel-schema-validator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flealre%2Fexcel-schema-validator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flealre%2Fexcel-schema-validator/lists"}