{"id":21534862,"url":"https://github.com/dimzachar/datatalksclub-projects","last_synced_at":"2025-07-18T02:38:47.780Z","repository":{"id":192015742,"uuid":"685082369","full_name":"dimzachar/DataTalksClub-Projects","owner":"dimzachar","description":"Streamlit-Powered DataTalksClub Project Analyzer: Interactive Insights at Your Fingertips","archived":false,"fork":false,"pushed_at":"2025-01-24T20:02:19.000Z","size":600,"stargazers_count":20,"open_issues_count":3,"forks_count":6,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-10T01:51:09.309Z","etag":null,"topics":["data-science","gpt","machine-learning","openai","python","streamlit","vizualisation"],"latest_commit_sha":null,"homepage":"https://datatalksclub-projects.streamlit.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dimzachar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-30T13:32:34.000Z","updated_at":"2025-01-24T20:02:23.000Z","dependencies_parsed_at":"2023-09-02T03:40:48.731Z","dependency_job_id":"994db87f-2403-4fc8-b41a-ba103933b694","html_url":"https://github.com/dimzachar/DataTalksClub-Projects","commit_stats":null,"previous_names":["dimzachar/datatalksclub-projects"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dimzachar/DataTalksClub-Projects","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimzachar%2FDataTalksClub-Projects","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimzachar%2FDataTalksClub-Projects/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimzachar%2FDataTalksClub-Projects/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimzachar%2FDataTalksClub-Projects/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dimzachar","download_url":"https://codeload.github.com/dimzachar/DataTalksClub-Projects/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimzachar%2FDataTalksClub-Projects/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265692646,"owners_count":23812204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","gpt","machine-learning","openai","python","streamlit","vizualisation"],"created_at":"2024-11-24T03:12:54.015Z","updated_at":"2025-07-18T02:38:47.721Z","avatar_url":"https://github.com/dimzachar.png","language":"Python","funding_links":["https://www.paypal.com/donate/?hosted_button_id=LR3PQYHZY4CJ4"],"categories":[],"sub_categories":[],"readme":"# DataTalksClub-Projects\n\n[![Streamlit App](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://datatalksclub-projects.streamlit.app/)\n\nhttps://github.com/dimzachar/DataTalksClub-Projects/assets/113017737/c3c3235c-951c-47e8-aa6a-a6dffa159e46\n\n## Table of Contents\n\n- [Introduction](#introduction)\n- [Folder Structure](#folder-structure)\n- [Getting Started](#getting-started)\n    - [Clone the Repository](#clone-the-repository)\n    - [Environment Setup](#environment-setup)\n    - [Environment Variables](#environment-variables)\n- [Makefile Usage](#makefile-usage)\n- [CI/CD Pipeline](#cicd-pipeline)\n- [Configuration Files](#configuration-files)\n- [Contributing](#contributing)\n- [License](#license)\n- [Contact](#contact)\n\n## Introduction\n\nDataTalksClub-Projects is a Python repository aimed at automating the analysis of projects from [DataTalksClub](https://github.com/DataTalksClub) courses. It focuses on data from [ML Zoomcamp](https://github.com/DataTalksClub/machine-learning-zoomcamp), [MLOps Zoomcamp](https://github.com/DataTalksClub/mlops-zoomcamp) and [DE Zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp) for the years of 2021-2023. The repository includes Python scripts for tasks like web scraping, data handling, and API interactions. The `Data/` folder contains all the datasets I have generated for the courses. It also aims to implement comprehensive tests and data visualizations.\n\n\n\u003e **Note**: Titles for projects are generated using OpenAI and may require refinement. Future course iterations should include project titles for easier processing.\n\n\n## Folder Structure\n\n```\n.\n├── Data/               # Data files\n├── src/                # Python source files\n├── tests/              # Test files (TBD)\n├── utils/              # Utility files\n├── .env                # Environment variables\n├── .gitignore          # Git ignore rules\n├── LICENSE             # License file\n├── Makefile            # Makefile for automation\n├── README.md           # This file\n├── app.py              # Streamlit app\n├── help.log            # Unknown titles\n├── pyproject.toml      # Build settings\n└── requirements.txt    # Dependency list\n```\n\n## Getting Started\n\n### Clone the Repository\n\n```bash\ngit clone https://github.com/yourusername/DataTalksClub-Projects.git\ncd DataTalksClub-Projects\n```\n\n### Environment Setup\n\n```bash\npip install --upgrade pip\npip install -r requirements.txt\n```\n\n### Environment Variables\n\n\u003c!-- To run this project, you'll need to add a `.env` file in your project root. --\u003e\n\n## Makefile Usage\n\nThe Makefile included in this repository provides a convenient way to run various tasks. Below are the commands you can use:\n\n### Run tests (TBD)\n\nThis command will run all the unit tests and integration tests for the project.\n\n```bash\nmake test\n```\n\n### Quality Checks\n\nRun this command to perform code quality checks. It includes isort, black and pylint.\n\n```bash\nmake quality_checks\n```\n\n### Scrape Data\n\nUse this command to scrape data from specified sources. The data will be saved in the appropriate format and location.\n\n```bash\nmake scrape\n```\n\n### Generate Titles\n\nThis command will generate titles for the projects using OpenAI's API.\n\n```bash\nmake titles\n```\n\n### Check Deployments\n\nRun this command to check the deployment status of project services such as web, batch or streaming.\n\n```bash\nmake deploy\n```\n\n### Run All Tasks\n\nThis command is a shortcut to run all of the above tasks in sequence. It's a quick way to ensure that everything is set up correctly.\n\n```bash\nmake all\n```\n\n### Streamlit app\n\nRun the Streamlit app using the Makefile\n\n```bash\nmake streamlit\n```\n\n## CI/CD Pipeline (TBD)\n\nThis repository includes a Continuous Integration (CI) workflow that automatically builds and tests the Python project upon each push or pull request. This ensures that the codebase remains stable and free of errors as new changes are integrated.\n\nThe CI workflow is configured to perform the following tasks:\n\n- Code quality checks\n- Unit tests\n- Integration tests\n\n## Configuration Files\n\n- `.gitignore`: Specifies files and folders to ignore in Git.\n- `LICENSE`: Contains the license information.\n- `Pipfile` \u0026 `Pipfile.lock`: Manage project dependencies.\n- `pyproject.toml`: Contains build-related settings.\n\n## Contributing (Coming soon)\n\n1. Fork the repository.\n2. Create a new feature branch.\n3. Make changes.\n4. Run tests (TBD).\n5. Submit a pull request.\n\n## License\n\nThis project is licensed under the MIT License. See the `LICENSE` file for details.\n\n## Contact\n\nFor inquiries, connect with me on [Linkedin](https://www.linkedin.com/in/zacharenakis/)\n\n## Support this project\n\n[![Donate with PayPal](https://www.paypalobjects.com/digitalassets/c/website/marketing/apac/C2/logos-buttons/optimize/26_Yellow_PayPal_Pill_Button.png)](https://www.paypal.com/donate/?hosted_button_id=LR3PQYHZY4CJ4)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimzachar%2Fdatatalksclub-projects","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdimzachar%2Fdatatalksclub-projects","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimzachar%2Fdatatalksclub-projects/lists"}