{"id":16479320,"url":"https://github.com/timebertt/big-data-refuel","last_synced_at":"2026-06-11T17:31:53.819Z","repository":{"id":114161181,"uuid":"429551466","full_name":"timebertt/big-data-refuel","owner":"timebertt","description":null,"archived":false,"fork":false,"pushed_at":"2021-12-13T18:22:31.000Z","size":9479,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-28T19:18:42.419Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timebertt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-18T19:18:05.000Z","updated_at":"2023-03-05T09:12:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"f78cea4b-3f0c-45ee-afd3-984bba4506d0","html_url":"https://github.com/timebertt/big-data-refuel","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/timebertt/big-data-refuel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timebertt%2Fbig-data-refuel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timebertt%2Fbig-data-refuel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timebertt%2Fbig-data-refuel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timebertt%2Fbig-data-refuel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timebertt","download_url":"https://codeload.github.com/timebertt/big-data-refuel/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timebertt%2Fbig-data-refuel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34211061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T12:51:38.014Z","updated_at":"2026-06-11T17:31:53.791Z","avatar_url":"https://github.com/timebertt.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Use Case: Analyzing fuel prices\n\n## Prerequisites\n\nNamespace `refuel`:\n```bash\nk create ns refuel\n```\n\nA running Hadoop cluster with YARN:\n\n```bash\nhelm repo add stable https://charts.helm.sh/stable\nstorage_class=default # replace with faster StorageClass if desired \nhelm install --namespace=refuel --set hdfs.dataNode.replicas=1 \\\n  --set hdfs.webhdfs.enabled=true \\\n  --set yarn.nodeManager.replicas=3 \\\n  --set yarn.nodeManager.resources.requests.cpu=2,yarn.nodeManager.resources.requests.memory=4Gi \\\n  --set yarn.nodeManager.resources.limits.cpu=3,yarn.nodeManager.resources.limits.memory=6Gi \\\n  --set persistence.nameNode.enabled=true \\\n  --set persistence.nameNode.storageClass=$storage_class \\\n  --set persistence.nameNode.size=15Gi \\\n  --set persistence.dataNode.enabled=true \\\n  --set persistence.dataNode.storageClass=$storage_class \\\n  --set persistence.dataNode.size=50Gi \\\n  hadoop stable/hadoop\n```\n\nNote: `yarn.nodeManager.replicas` is the number of workers that can run spark jobs submitted to yarn.\nWhen setting `yarn.nodeManager.resources.requests.{cpu,memory}`, consider the amount of compute resources available in your cluster,\ne.g. a minikube cluster is limited by the size of your machine.\n\nIf deployed with this helm chart, the HDFS namenode will format on each start (see https://github.com/helm/charts/issues/19499).\nThis results in a new cluster ID and thus datanodes (that have the old cluster ID persisted on disk) won't be able to\njoin the cluster with errors like this:\n```text\n2021-12-12 09:58:14,108 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/root/hdfs/datanode/\njava.io.IOException: Incompatible clusterIDs in /root/hdfs/datanode: namenode clusterID = CID-19a00965-8914-47d2-8d61-0b4fc3a8d2cd; datanode clusterID = CID-566449ab-0985-4b54-9aa2-7382ca1f3a93\n\tat org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:760)\n\tat org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:293)\n\tat org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:409)\n\tat org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:388)\n\tat org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:556)\n\tat org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1649)\n\tat org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1610)\n\tat org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:374)\n\tat org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:280)\n\tat org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)\n\tat java.lang.Thread.run(Thread.java:745)\n2021-12-12 09:58:14,111 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool \u003cregistering\u003e (Datanode Uuid 46b21fcc-74de-4ec6-9484-8cb2e5a3bbf9) service to hadoop-hadoop-hdfs-nn/100.96.0.22:9000. Exiting.\n```\n\nTo fix this behaviour, apply the following patch to the `hadoop` ConfigMap:\n```bash\nk patch cm hadoop-hadoop --patch-file k8s/hadoop-bootstrap-fix.yaml\n```\n(Remember to patch the ConfigMap again after upgrading the helm chart.)\n\n## Deploy\n\nTo develop using [Skaffold](https://skaffold.dev/), use `skaffold dev` (development mode with automatic rebuilds on file changes) or `skaffold run` (one-time deploy).\n\n### Download CronJob\n\nWe use a [Kubernetes CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) to fetch raw data from https://dev.azure.com/tankerkoenig/tankerkoenig-data/_git/tankerkoenig-data and store it in HDFS.\nEach execution downloads data for one day starting with a configured start date (env var `DOWNLOAD_START_DATE`). It can be executed with a high frequency (e.g. every 2 minutes) to simulate new data arriving every day.\n\n```\n$ k exec hadoop-hadoop-hdfs-dn-0 -- hdfs dfs -ls \"/input/*/*\"\n-rw-r--r--   3 root supergroup   26716562 2021-12-09 19:46 /input/prices/2021-11-01-prices.csv\n-rw-r--r--   3 root supergroup   27371251 2021-12-09 19:46 /input/prices/2021-11-02-prices.csv\n-rw-r--r--   3 root supergroup   27861891 2021-12-09 19:47 /input/prices/2021-11-03-prices.csv\n-rw-r--r--   3 root supergroup   27217196 2021-12-09 19:47 /input/prices/2021-11-04-prices.csv\n-rw-r--r--   3 root supergroup   27511444 2021-12-09 19:48 /input/prices/2021-11-05-prices.csv\n-rw-r--r--   3 root supergroup   24953811 2021-12-09 19:48 /input/prices/2021-11-06-prices.csv\n-rw-r--r--   3 root supergroup    4442792 2021-12-09 19:46 /input/stations/2021-11-01-stations.csv\n-rw-r--r--   3 root supergroup    4439113 2021-12-09 19:46 /input/stations/2021-11-02-stations.csv\n-rw-r--r--   3 root supergroup    4440561 2021-12-09 19:47 /input/stations/2021-11-03-stations.csv\n-rw-r--r--   3 root supergroup    4439959 2021-12-09 19:48 /input/stations/2021-11-04-stations.csv\n-rw-r--r--   3 root supergroup    4443412 2021-12-09 19:48 /input/stations/2021-11-05-stations.csv\n-rw-r--r--   3 root supergroup    4442876 2021-12-09 19:49 /input/stations/2021-11-06-stations.csv\n\n$ k exec hadoop-hadoop-hdfs-dn-0 -- hdfs dfs -du -h \"/input\"\n11       /input/last_downloaded_date\n176.4 M  /input/prices\n25.4 M   /input/stations\n```\n\n### Spark App\n\nOur Spark app will read the downloaded data from HDFS.\nConnection details are configured in `core-site.xml` and `hdfs-site.xml` (file location is configured via `HADOOP_CONF_DIR` environment variable).\n\nTo run the spark app locally in a Docker container:\n```bash\n$ docker-compose up -d --build mysql spark-app\n$ docker exec -it big-data-refuel_spark-app_1 bash\nspark@ef55f43b991d:/spark-app$ /entrypoint.sh spark-app.py\n...\n```\n\nOur Spark app checkpoints to HDFS under `/checkpoint`:\n```\n$ k exec hadoop-hadoop-hdfs-dn-0 -- hdfs dfs -du -h \"/checkpoint/*\"\n41  /checkpoint/commits/0\n41  /checkpoint/commits/1\n45  /checkpoint/metadata\n488  /checkpoint/offsets/0\n500  /checkpoint/offsets/1\n500  /checkpoint/offsets/2\n712  /checkpoint/sources/0\n736  /checkpoint/sources/1\n3.8 M    /checkpoint/state/0\n137.2 M  /checkpoint/state/1\n623.2 K  /checkpoint/state/2\n```\n\nIf the App is interrupted (e.g. because of a Pod deletion/failure), execution is continued from where it left off.\nHowever, it might take a few minutes to initialize the Spark Jobs from the checkpoints.\n\n### Frontend\n\nTo run the web app locally in a Docker container:\n```bash\n$ docker-compose up -d mysql # start mysql prerequisite\n$ docker-compose up --build web-app\n$ # open http://localhost:5555 in browser\n$ # after code changes: stop (^C) and run again\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimebertt%2Fbig-data-refuel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimebertt%2Fbig-data-refuel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimebertt%2Fbig-data-refuel/lists"}