{"id":21133463,"url":"https://github.com/sheitak/datalake-jljq","last_synced_at":"2026-06-23T15:33:08.353Z","repository":{"id":210903669,"uuid":"722900242","full_name":"Sheitak/datalake-jljq","owner":"Sheitak","description":"Data Lake project for ingest and transform financial data and dashboard BI proposal ","archived":false,"fork":false,"pushed_at":"2024-01-30T09:27:56.000Z","size":42,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-11T17:57:30.561Z","etag":null,"topics":["airflow-docker","big-data","datalake","pyspark","python3","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Sheitak.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-24T08:10:33.000Z","updated_at":"2023-12-12T08:34:03.000Z","dependencies_parsed_at":"2023-12-29T11:30:47.369Z","dependency_job_id":"a2f8b812-cd13-4cdf-b779-0db011da48cf","html_url":"https://github.com/Sheitak/datalake-jljq","commit_stats":null,"previous_names":["sheitak/datalake-jljq"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Sheitak/datalake-jljq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sheitak%2Fdatalake-jljq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sheitak%2Fdatalake-jljq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sheitak%2Fdatalake-jljq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sheitak%2Fdatalake-jljq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Sheitak","download_url":"https://codeload.github.com/Sheitak/datalake-jljq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sheitak%2Fdatalake-jljq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34696897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow-docker","big-data","datalake","pyspark","python3","spark"],"created_at":"2024-11-20T06:10:14.870Z","updated_at":"2026-06-23T15:33:08.334Z","avatar_url":"https://github.com/Sheitak.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Lake Project - JLJQ\n\nThis project is part of a study into the creation of a Data Lake, and focuses on the field of finance. It gathers different data sources to be formatted to obtain the final visualization of a trend in this field.\n\n**Maintainers**\n\n- [Johan Vertut](https://github.com/Nanificateur)\n- [Louis Bayle](https://github.com/LouisBDev19)\n- [Jules Mekontso](https://github.com/julesauffred)\n- [Quentin Moreau](https://github.com/Sheitak)\n\n### Starting Project\n\nClone this repository and place yourself inside it.\n\nYou can copy ``.env.example`` file to ``.env`` and replace the essential information.\n\n### Apache-Airflow Docker\n\nRefer to the [Airflow With Docker](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html) on the official airflow website for a clean installation with Docker.\n\nRefer to the [Airflow Package In Docker](https://airflow.apache.org/docs/docker-stack/build.html#example-of-adding-airflow-provider-package-and-apt-package) For creating Docker File with packages.\n\n**1. Initializing Environment**\n\nGenerate Folders And Define Airflow User If Necessary\n```\nmkdir -p ./dags ./logs ./plugins ./config\necho -e \"AIRFLOW_UID=$(id -u)\" \u003e .env\n```\n\nInitialize Database\n```\ndocker compose up airflow-init\n```\n\nRun Airflow Set-Up\n```\ndocker compose up -d\n```\n\n**2. Accessing the environment**\n\nRunning the CLI commands\n```\ndocker compose run airflow-worker airflow info\n```\n\n***OR***\n\nOpen in browser\n```\nlocalhost:8080 or IP:8080\n```\n\n**Defaults Information**\n\nAirflow Credentials\n```\nusername: airflow\npassword: airflow\n```\n\n### API Used\n\n- [CoinMarketCap]()\n- [Coinranking]()\n- [CoinGecko]()\n- [Alpha Vantage](https://www.alphavantage.co/documentation/)\n\n### Resources\n\n- [Airflow Documentation](https://airflow.apache.org/docs/apache-airflow/stable/index.html)\n- [Apache Spark Documentation](https://spark.apache.org/docs/latest/)\n\nCopyright (C) 2007 Free Software Foundation, Inc. \u003chttps://fsf.org/\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsheitak%2Fdatalake-jljq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsheitak%2Fdatalake-jljq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsheitak%2Fdatalake-jljq/lists"}