{"id":18391741,"url":"https://github.com/moscicky/data-warehouse","last_synced_at":"2025-04-12T09:42:44.416Z","repository":{"id":83328307,"uuid":"234609996","full_name":"moscicky/data-warehouse","owner":"moscicky","description":null,"archived":false,"fork":false,"pushed_at":"2020-01-22T16:30:27.000Z","size":1549,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T14:49:46.926Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moscicky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-17T18:21:05.000Z","updated_at":"2020-08-06T11:35:35.000Z","dependencies_parsed_at":"2023-03-01T10:32:21.172Z","dependency_job_id":null,"html_url":"https://github.com/moscicky/data-warehouse","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moscicky%2Fdata-warehouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moscicky%2Fdata-warehouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moscicky%2Fdata-warehouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moscicky%2Fdata-warehouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moscicky","download_url":"https://codeload.github.com/moscicky/data-warehouse/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248549741,"owners_count":21122926,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T01:53:07.874Z","updated_at":"2025-04-12T09:42:44.383Z","avatar_url":"https://github.com/moscicky.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Data warehouse\n\n![schema](readme-assets/updated_schema.png)\n\n### Tworzenie jara\n1. Stwórz obiekt ETLa z metodą main w pakiecie `executables`. Wykorzystaj klasę ETL. \nWażne żeby dodać `enableHiveSupport()` i parametryzować ścieżki do plików jak w przykładzie.\n![1](readme-assets/tut7.png)\n1. Wybierz `File \u003e Project Structure \u003e Artifacts`\nKliknij `+` w lewym górnym rogu. Wybierz `JAR` a następnie \n`empty`\n![2](readme-assets/tut5.png)\n1. Kliknij `+` wybierz `Module Output` a następnie `data-warehouse`\n![3](readme-assets/tut6.png)\n1. Wybierz `Build \u003e Build Artifacts...`. Z listy wybierz swojego jara \ni kliknij `Build`. \n![4](readme-assets/tut4.png)\n1. Jar zostanie umieszczony w folderze `out/artifcats`\n\n### Odpalanie na klastrze\n1. wgraj pliki wejściowe na swój bucket do folderu `warehouse-input` na google storage:\n\n        AirQuality.csv\n        CityofLondonPoliceOutcomes.txt\n        CityofLondonPoliceRecords.csv\n        MetropolitanPoliceServiceOutcomes.txt\n        MetropolitanPoliceServiceRecords.csv\n        Norm.csv\n        Source.csv\n        \nPliki `Norm.csv` oraz `Source.csv` znajduja się w katalogu `src/main/scala/warehouse/data`\n    \n1. z shella VMki na klastrze przekopiuj pliki z bucketa na klaster \n        \n        gsutil cp -r gs://\u003cbucket-name\u003e/warehouse-input .\n1. stwórz folder na pliki wejściowe na hdfs \n\n        hadoop fs -mkdir -p input\n1. przekopiuj pliki wejściowe z klastra na hdfs \n        \n        hadoop fs -copyFromLocal warehouse-input/* input/\n        \n1. (opcjonalnie) przekopiuj jara na klaster \n        \n        gsutil cp -r gs://\u003cbucket-name\u003e/etl.jar .\n \n1. odpal spark-shella i włącz tryp wklejania\n    \n        spark-shell \n        :paste\n        \n1. wklej zawartość pliku `SparkShellLoad.scala`\n        \n1. klinij enter i `crtl + d`, obiekty powinny się utworzyć\n1. stwórz schemat tabel \n\n        val schema = new SchemaCreator(spark)\n        schema.createAll()\n1. wyjdź ze spark shella `crl + d`\n1. odpal etla wskazując ścieżkę do maina. PRZYKŁAD:\n        \n        spark-submit --class warehouse.executables.TimeTableEtl \\\n        --master yarn --num-executors 5 --driver-memory 512m \\\n        --executor-memory 512m --executor-cores 1 etl.jar input true\n\nzmieniająć parametr `--class` na jedną z klas w `executables` uruchamiamy wybranego ETLA \nOstatnie dwa argumenty oznaczają \n1) folder z plikami wejściowymi \n2) czy używać małych wersji plików \n\n### Odpalanie wszystkiego \n```\nspark-submit --class warehouse.executables.TimeTableEtl \\\n--master yarn --num-executors 5 --driver-memory 512m \\\n--executor-memory 512m --executor-cores 1 etl.jar input false\n\nspark-submit --class warehouse.executables.SourceTableEtl \\\n--master yarn --num-executors 5 --driver-memory 512m \\\n--executor-memory 512m --executor-cores 1 etl.jar input false\n\nspark-submit --class warehouse.executables.AirPollutionTypeEtl \\\n--master yarn --num-executors 5 --driver-memory 512m \\\n--executor-memory 512m --executor-cores 1 etl.jar input false\n\nspark-submit --class warehouse.executables.CrimeTypeEtl \\\n--master yarn --num-executors 5 --driver-memory 512m \\\n--executor-memory 512m --executor-cores 1 etl.jar input false\n\nspark-submit --class warehouse.executables.LocationEtl \\\n--master yarn --num-executors 5 --driver-memory 512m \\\n--executor-memory 512m --executor-cores 1 etl.jar input false\n\nspark-submit --class warehouse.executables.OutcomeTypeEtl \\\n--master yarn --num-executors 5 --driver-memory 512m \\\n--executor-memory 512m --executor-cores 1 etl.jar input false\n\nspark-submit --class warehouse.executables.CrimeTableEtl \\\n--master yarn --num-executors 5 --driver-memory 512m \\\n--executor-memory 512m --executor-cores 1 etl.jar input false\n\nspark-submit --class warehouse.executables.AirQualityTableEtl \\\n--master yarn --num-executors 5 --driver-memory 512m \\\n--executor-memory 512m --executor-cores 1 etl.jar input false\n```\n### Zeppelin\nZaimportuj plik `Zeppelin_Analysis.json` do Zeppelina i uruchom.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoscicky%2Fdata-warehouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoscicky%2Fdata-warehouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoscicky%2Fdata-warehouse/lists"}