{"id":17124368,"url":"https://github.com/wittline/data-engineer-challenge","last_synced_at":"2025-09-05T10:34:21.720Z","repository":{"id":38240902,"uuid":"499378795","full_name":"Wittline/data-engineer-challenge","owner":"Wittline","description":"Challenge Data Engineer","archived":false,"fork":false,"pushed_at":"2022-06-13T15:21:30.000Z","size":32,"stargazers_count":25,"open_issues_count":1,"forks_count":8,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-21T18:03:45.510Z","etag":null,"topics":["data-engineering","data-pipeline","dataengineering","docker","docker-compose","fastapi","postgresql"],"latest_commit_sha":null,"homepage":"https://wittline.github.io/data-engineer-challenge/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Wittline.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-03T04:29:49.000Z","updated_at":"2025-07-10T11:14:18.000Z","dependencies_parsed_at":"2022-09-04T22:12:30.667Z","dependency_job_id":null,"html_url":"https://github.com/Wittline/data-engineer-challenge","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Wittline/data-engineer-challenge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wittline%2Fdata-engineer-challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wittline%2Fdata-engineer-challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wittline%2Fdata-engineer-challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wittline%2Fdata-engineer-challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Wittline","download_url":"https://codeload.github.com/Wittline/data-engineer-challenge/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wittline%2Fdata-engineer-challenge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273746809,"owners_count":25160647,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-pipeline","dataengineering","docker","docker-compose","fastapi","postgresql"],"created_at":"2024-10-14T18:42:28.756Z","updated_at":"2025-09-05T10:34:21.680Z","avatar_url":"https://github.com/Wittline.png","language":"Python","readme":"# Data Engineer Challenge\n\nThis goal of this repository is based on solving a technical challenge for the data engineering position.\n\nCheck the article here:  \u003ca href=\"https://coraspe-ramses.medium.com/design-development-and-deployment-of-a-simple-data-pipeline-6f1d59d0fd6a/\"\u003eDesign, Development and Deployment of a simple Data Pipeline\u003c/a\u003e\n\n\n![image](https://user-images.githubusercontent.com/8701464/172062180-c90e7f11-ae64-4fd2-9772-8cfd2fb6abf6.png)\n\n\n- Install Docker Desktop on Windows, it will install Docker Compose as well, Docker Compose will allow you to run multiple container applications.\n- Install git-bash for windows, once installed, open git bash and download this repository, this will download the docker-compose.yaml file, and other files needed.\n\n\n```linux \nramse@DESKTOP-K6K6E5A MINGW64 /c\n$ git clone https://github.com/Wittline/data-engineer-challenge.git\n```\n\n- Once all the files needed were downloaded from the repository, let's run everything. We will use the git bash tool again, go to the folder data-engineer-challenge and we will run the Docker Compose command:\n\n```linux \nramse@DESKTOP-K6K6E5A MINGW64 /c\n$ cd data-engineer-challenge\n```\n\n```linux \nramse@DESKTOP-K6K6E5A MINGW64 /c/data-engineer-challenge\n$ cd code\n```\n```linux \nramse@DESKTOP-K6K6E5A MINGW64 /c/data-engineer-challenge/code\n$ cd apps\n```\n\n```linux \n@DESKTOP-K6K6E5A MINGW64 /c/data-engineer-challenge/code/apps\n$ docker-compose up\n```\n\n- After wait for a couple of minutes the final result of executing the above command should look like this:\n\n![image](https://user-images.githubusercontent.com/8701464/172062212-05193fba-d980-4917-9fe1-f1134d72afb8.png)\n\n\n- docker-compose executionlet's check the API REST, go to your browser and search for: http://localhost:8080/docs#/default\n\n![image](https://user-images.githubusercontent.com/8701464/172062217-9fbd6026-6a49-42fc-bbbb-9efb58743cc9.png)\n\n\n- let's try the request to the API REST\n\n![image](https://user-images.githubusercontent.com/8701464/172062224-b9ec90bd-8454-4e5a-a4f5-2adc932a41a2.png)\n\n\n- let's check the response from the API REST\n\n![image](https://user-images.githubusercontent.com/8701464/172062241-8fc66570-9ca8-4ff9-b56e-239fb6eee118.png)\n\n# Contributing and Feedback\nAny ideas or feedback about this repository?. Help me to improve it.\n\n# Authors\n- Created by \u003ca href=\"https://twitter.com/RamsesCoraspe\"\u003e\u003cstrong\u003eRamses Alexander Coraspe Valdez\u003c/strong\u003e\u003c/a\u003e\n- Created on 2022\n\n# License\nThis project is licensed under the terms of the Apache License.\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwittline%2Fdata-engineer-challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwittline%2Fdata-engineer-challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwittline%2Fdata-engineer-challenge/lists"}