{"id":14970617,"url":"https://github.com/davidzajac1/zillacode","last_synced_at":"2025-04-04T18:03:15.050Z","repository":{"id":253319434,"uuid":"842750311","full_name":"davidzajac1/zillacode","owner":"davidzajac1","description":"Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake","archived":false,"fork":false,"pushed_at":"2025-02-10T19:46:47.000Z","size":2815,"stargazers_count":160,"open_issues_count":4,"forks_count":36,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-28T17:02:01.020Z","etag":null,"topics":["aws","coding-interview","dbt","docker","github-actions","leetcode","pandas","pyspark","python","react","snowflake","spark","terraform"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidzajac1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-15T02:15:48.000Z","updated_at":"2025-03-27T23:29:01.000Z","dependencies_parsed_at":"2025-02-09T20:11:00.727Z","dependency_job_id":"8f01bc30-5d6b-4ba9-9316-a7ff54ee3801","html_url":"https://github.com/davidzajac1/zillacode","commit_stats":{"total_commits":7,"total_committers":3,"mean_commits":"2.3333333333333335","dds":0.2857142857142857,"last_synced_commit":"5b24cc07131c9b6acbfbf279b83e91b8c7f219c0"},"previous_names":["davidzajac1/zillacode"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidzajac1%2Fzillacode","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidzajac1%2Fzillacode/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidzajac1%2Fzillacode/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidzajac1%2Fzillacode/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidzajac1","download_url":"https://codeload.github.com/davidzajac1/zillacode/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247226211,"owners_count":20904465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","coding-interview","dbt","docker","github-actions","leetcode","pandas","pyspark","python","react","snowflake","spark","terraform"],"created_at":"2024-09-24T13:43:51.890Z","updated_at":"2025-04-04T18:03:15.002Z","avatar_url":"https://github.com/davidzajac1.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg width=\"auto\" src=\"frontend/public/img/github_header.jpg\" alt=\"Header\"\u003e\n\u003c/p\u003e\n\n# Zillacode - _Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake_\n\n![Language](https://img.shields.io/badge/Language-PySpark-informational?style=flat)\n![Language](https://img.shields.io/badge/Language-Spark-informational?style=flat)\n![Language](https://img.shields.io/badge/Language-DBT-informational?style=flat)\n![Language](https://img.shields.io/badge/Language-SnowSQL-informational?style=flat)\n\nZillacode is the only platform on the internet where you can practice 50+ LeetCode like questions using PySpark, Spark, DBT and Snowflake. The entire platform spins up on any machine with one command using Docker Compose.\n\nIn addition to solving Zillacode problems you can also learn from the repo itself. Zillacode was once a live micro-serviced B2C SAAS platform deployed using the AWS serverless stack. All of the Infrastructure as Code files have been left in place for anyone who wants to learn about Devops or try to deploy it themselves.\n\n### Run Locally\n\n```bash\ngit clone https://github.com/davidzajac1/zillacode.git\ncd zillacode\ndocker compose up\n```\n\nThen navigate to [http://localhost:5173](http://localhost:5173)\n\n#### Endless Loading/Troubleshooting\n\nMake sure nothing is running on ports 5001, 5173, 8001, 8002, 8003 before running locally. See the commands below to kill anything running on these ports:\n\n```bash\n# Mac/Linux\nsudo kill -9 $(sudo lsof -ti:5001,5173,8001,8002,8003)\n# Windows\nnetstat -ano | findstr :5001 :5173 :8001 :8002 :8003\n```\n\nZillacode previously used port 5000 which caused issues for many users and thus now uses port 5001 instead. If you are experiencing issues like problems endlessly loading please make sure you are using newest version of the `master` branch and the newest images from the [zillacode-backend](https://hub.docker.com/r/davidzajac1/zillacode-backend) and [zillacode-frontend](https://hub.docker.com/r/davidzajac1/zillacode-frontend) Docker Hub repos and that you have inspected the IDE page to make sure there are no requests going to port 5000.\n\n### Optional - Configuring Snowflake\n\nNote Snowflake is not Open Source thus to solve problems with DBT/Snowflake you need to add a `.env` file with Snowflake credentials in the `backend` directory. All problems except for DBT/Snowflake problem will work normally without the `.env` file.\n\n```\nSNOWFLAKE_USERNAME=\u003cSNOWFLAKE_USERNAME\u003e\nSNOWFLAKE_PASSWORD=\u003cSNOWFLAKE_PASSWORD\u003e\nSNOWFLAKE_ACCOUNT=\u003cSNOWFLAKE_ACCOUNT\u003e\nSNOWFLAKE_WAREHOUSE=\u003cSNOWFLAKE_WAREHOUSE\u003e\n```\n\nYou can set up your own Snowflake account, all queries run by Zillacode will not result in any spend as they all just compile as Common Table Expressions without querying any tables. Your Snowflake user must have permissions to use your Snowflake warehouse.\n\n### Questions/Issues\n\nFor any bugs/issues please make an Issue in GitHub. See the Contributing section on how to make a Pull Request if you want to fix the problem yourself.\n\nFeel free to ask any questions about the project or anything Data Engineering, Spark, etc. related in a GitHub Discussion. All announcements about the project will be made in GitHub Discussions as well.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"auto\" src=\"frontend/public/img/ide.jpg\" alt=\"Header\"\u003e\n\u003c/p\u003e\n\n## Development/Contributing\n\nAll contributions from Issues to Pull Requests are greatly appreciated!\n\nZillacode runs locally using Docker Compose and is divided into 5 microservices. The app can be run as is just by running `make up` but if you want to make changes see the sections below for how development works with each service.\n\nNote Zillacode at was originally a SAAS product but now the project is just designed to run locally with Docker Compose, but all of the Terraform/Zappa/GitHub Actions config files are left in the project for anyone who wants to try to deploy it to the cloud.\n\n### Pull Requests\n\nTo make a Pull Request fork the project then make a Pull Request to the `master` branch. Please be specific about the changes you are making.\n\n### Installations\n\n- [Make](https://gnuwin32.sourceforge.net/packages/make.htm)\n- [Docker](https://docs.docker.com/get-docker/)\n- [Node](https://nodejs.org/en/download/)\n- [Yarn](https://classic.yarnpkg.com/lang/en/docs/install/#windows-stable)\n- [React Dev Tools](https://reactjs.org/blog/2015/09/02/new-react-developer-tools.html#installation)\n- [Python](https://www.python.org/downloads/)\n- [Pipenv](https://github.com/pypa/pipenv)\n- [Pre-Commit](https://pre-commit.com/#installation)\n\n### Frontend\n\nFor making frontend changes, spin up the backend in the background using Docker by running `make up-backend`, this will spin up the backend on http://localhost:5000. Then run `make serve-frontend` to serve the frontend on http://localhost:5173. The frontend will send requests to the backend hosted locally.\n\nFor adding new dependencies run `make add package=\u003cPACKAGE TO INSTALL\u003e`.\n\nTo clean up run `make down` to shut down the backend containers Docker is running.\n\n### Backend Microservices\n\nThe backend is comprised of 4 separate services hosted on [AWS Lambda](https://aws.amazon.com/lambda/). To rebuild changes made to any of these services run `make rebuild container=\u003cbackend|spark|scala-spark|db-lambda\u003e`.\n\n#### Flask Lambda\n\nThe Zillacode backend API is a [Flask](https://flask.palletsprojects.com/) application that receives HTTP requests from the frontend. This service is contained in the `backend` directory. All endpoints thus to create a new endpoint you only need to change the Python code in the `./backend/app.py` file.\n\n#### Spark Lambda\n\nThe Spark Lambda is used to run PySpark code and is built from a Docker Image. It is contained in the `spark-lambda` directory.\n\n#### Scala Spark Lambda\n\nThe Scala Spark Lambda is essentially the same as the Spark Lambda but is used to run Scala code. It is contained in the `scala-spark-lambda` directory. The Scala Spark Lambda image is built using `sbt`, a common Scala build framework. It is slow and may take about 15 minutes to build the Docker image.\n\n#### DB Lambda\n\nThis Lambda is used to run Python Pandas code. It is separated from the backend Flask App because it uses the Python `exec()` function to remotely execute code thus it needs to be in its own service that has no permissions and can't make outbound connections.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidzajac1%2Fzillacode","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidzajac1%2Fzillacode","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidzajac1%2Fzillacode/lists"}