{"id":21917154,"url":"https://github.com/zsvoboda/ngods","last_synced_at":"2025-04-19T07:35:36.929Z","repository":{"id":54773784,"uuid":"494421226","full_name":"zsvoboda/ngods","owner":"zsvoboda","description":"New generation opensource data stack","archived":false,"fork":false,"pushed_at":"2022-05-20T15:50:41.000Z","size":1695,"stargazers_count":65,"open_issues_count":1,"forks_count":9,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-29T06:51:34.376Z","etag":null,"topics":["analytics","data","data-pipeline","iceberg","jdbc","presto","prestodb","prestosql","python","scala","spark","spark-sql","sparksql","sql","trino","trinodb"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zsvoboda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-20T10:30:48.000Z","updated_at":"2025-01-06T08:46:53.000Z","dependencies_parsed_at":"2022-08-14T02:20:48.860Z","dependency_job_id":null,"html_url":"https://github.com/zsvoboda/ngods","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zsvoboda%2Fngods","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zsvoboda%2Fngods/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zsvoboda%2Fngods/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zsvoboda%2Fngods/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zsvoboda","download_url":"https://codeload.github.com/zsvoboda/ngods/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249638251,"owners_count":21304305,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","data","data-pipeline","iceberg","jdbc","presto","prestodb","prestosql","python","scala","spark","spark-sql","sparksql","sql","trino","trinodb"],"created_at":"2024-11-28T19:24:03.319Z","updated_at":"2025-04-19T07:35:36.908Z","avatar_url":"https://github.com/zsvoboda.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# ngods: opensource data stack\nThis repository contains Docker compose script that creates opensource data analytics stack on your local machine.  \n\n![ngods architecture](https://raw.githubusercontent.com/zsvoboda/ngods/main/img/ngods.png)\n\nCurrently, the stack consists of multiple components: \n\n- [**Trino**](https://trino.io/) for low-latency distributed SQL queries \n- [**Apache Spark**](https://spark.apache.org/) for data pipeline\n- [**Apache Iceberg**](https://iceberg.apache.org/) for atomic data pipeline with schema evolution, and partitioning for performance \n- [**Hive Metastore**](https://cwiki.apache.org/confluence/display/hive/design#Design-Metastore) for metadata management (metadata are stored in MariaDB)\n- [**Minio**](https://min.io/) for mimicking S3 storage on a local computer\n\nI plan to add more components soon: \n\n- [**Postgres**](https://www.postgresql.org/) for low-latency queries (if Trino on Iceberg doesn't provide satisfactory low-latency queries). I'll also add Postgre Foreign Data Wrapper technology for more convenient ELT between Trino and Postgres\n- [**DBT**](https://www.getdbt.com/) for ELT on top of Spark SQL or Trino\n- [**GoodData.CN**](https://www.gooddata.com/developers/cloud-native/) or [**Cube.dev**](https://cube.dev/) for analytics model and metrics \n- [**Metabase**](https://www.metabase.com/) or [**Apache Superset**](https://superset.apache.org/) for dashboards and data visualization\n\n## How ngods works: Simple example\nYou can simply start the ngds by executing \n\n```bash\ndocker-compose up\n```\n\nfrom the top-level directory where you've cloned this repo. \n\nOnce all images are pulled and all containers start, open Minio in your browser [http://localhost:9000](http://localhost:9000) log in with ```minio``` username and  ```minio123``` password and create a top level bucket called ```bronze``` .\n\nThen use your favorite SQL console tool (I use [DBeaver](https://dbeaver.io/)) and connect to the Trino instance running on your local machine (jdbc url: ```jdbc:trino://localhost:8080```, username: ```trino```, empty database) and execute the following script:\n\n```sql\ncreate schema if not exists warehouse.bronze with (location = 's3a://bronze/');\n\ndrop table if exists warehouse.bronze.employee;\n\ncreate table warehouse.bronze.employee (\n   employee_id integer not null,\n   employee_name varchar not null\n)\nwith (\n   format = 'parquet'\n);\n\ninsert into warehouse.bronze.employee (employee_id, employee_name) values (1, 'john doe');\ninsert into warehouse.bronze.employee (employee_id, employee_name) values (2, 'jane doe');\ninsert into warehouse.bronze.employee (employee_id, employee_name) values (3, 'joe doe');\ninsert into warehouse.bronze.employee (employee_id, employee_name) values (4, 'james doe');\n\nselect * from warehouse.bronze.employee;\n```\n\n## How ngods works: Loading parquet file as table\nNow we'll load the February 2022 NYC taxi trip data parquet file as a new table to ngods.\n\nFirst, create a new ```nyc``` directory in the ```./data/stage``` directory and download the [February 2022 NYC taxi trips parquet file](https://s3.amazonaws.com/nyc-tlc/trip%20data/fhvhv_tripdata_2022-02.parquet) to it.\n\nThen open and execute the ngods [Spark notebook script](http://localhost:8888/notebooks/notebooks/spark.nyc.taxti.example.ipynb) that loads the data as a new table to the ```bronze``` schema of a ```warehouse``` database.\n\n```python\ndf=spark.read.parquet(\"/home/data/stage/nyc/fhvhv_tripdata_2022-02.parquet\")\ndf.writeTo(\"warehouse.bronze.ny_taxis_feb\").create()\n```\n\nNow open your SQL console again, connect to the the Trino instance running on your local machine (jdbc url: ```jdbc:trino://localhost:8080```, username: ```trino```, empty database) and execute this SQL queries:\n\n```sql\nselect count(*) from warehouse.bronze.ny_taxis_feb;\n\nselect \n\thour(pickup_datetime),\n\tsum(trip_miles),\n\tsum(trip_time),\n\tsum(base_passenger_fare)\nfrom warehouse.bronze.ny_taxis_feb group by 1 order by 1;\n```\n\nYou should see your query results in no time!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzsvoboda%2Fngods","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzsvoboda%2Fngods","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzsvoboda%2Fngods/lists"}