{"id":15060670,"url":"https://github.com/ka-zo/booking-data-analysis","last_synced_at":"2026-02-09T08:05:19.001Z","repository":{"id":245942442,"uuid":"805911949","full_name":"ka-zo/booking-data-analysis","owner":"ka-zo","description":"Booking data analysis","archived":false,"fork":false,"pushed_at":"2024-06-10T10:38:30.000Z","size":2320,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-18T00:50:31.797Z","etag":null,"topics":["airline-booking","apache-beam","bigquery","google-cloud","looker-studio","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ka-zo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-25T20:30:05.000Z","updated_at":"2024-06-24T21:32:50.000Z","dependencies_parsed_at":"2024-06-24T23:46:23.125Z","dependency_job_id":"bb853d01-883d-4c22-a36f-98cf2ae9ddb4","html_url":"https://github.com/ka-zo/booking-data-analysis","commit_stats":null,"previous_names":["ka-zo/booking-data-analysis"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ka-zo%2Fbooking-data-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ka-zo%2Fbooking-data-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ka-zo%2Fbooking-data-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ka-zo%2Fbooking-data-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ka-zo","download_url":"https://codeload.github.com/ka-zo/booking-data-analysis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238065159,"owners_count":19410588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airline-booking","apache-beam","bigquery","google-cloud","looker-studio","python3"],"created_at":"2024-09-24T23:02:32.434Z","updated_at":"2026-02-09T08:05:13.981Z","avatar_url":"https://github.com/ka-zo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Commercial Booking Analysis\n\n## Preparing local environment\n\nOpen your shell, and make sure you follow the present guide properly.\n\nMake sure you clone the present git repository to your system and then\nyou need to enter the directory of the cloned repository in your shell.\n\nYou will also need to install the `gcloud` CLI, you need to perform\nauthentication and you need to set a specific project, you created before,\nin order to allow running the python scripts below.\n\n\u003e [!WARNING]\n\u003e None of the steps related to setting up a working environment for Google\n\u003e Cloud are explained in this document, but you can refer to the\n\u003e [gcloud CLI How-To](https://cloud.google.com/sdk/docs/how-to) pages.\n\nAfter this, make sure you have at least python 3.9 installed on your system.\nIf you don't have python, please install it on your system.\n\n\u003e [!WARNING]\n\u003e Please check yourself how to install python on your system, as it depends\n\u003e on your system, and if you have the right privileges.\n\n```bash\n$ python --version\nPython 3.10.11\n```\n\nUpgrade your pip version, if necessary.\n``` bash\n$ python -m pip install --upgrade pip\n```\n\nCreate a python virtual environment and then activate it.\nMake sure you use the right command for activating the virtual\nenvironment, as it depends on your underlying operating system.\nThe code below shows, how to do it on Windows. Follow this link to\n['venv — Creation of virtual environments'](https://docs.python.org/3/library/venv.html)\nif you want to learn more about python virtual environments.\n\n```bash\n$ python -m venv venv\n$ source venv/Scripts/activate\n$ pip install -r requirements.txt\n```\n\n## General remarks regarding implementation\n\nThe implemented data processing system has the following architecture:\n\n![Source -\u003e Apache Beam (Google Dataflow) -\u003e Data Warehouse (BigQuery)\n-\u003e Reporting (Looker Studio).](assets/data_processing_architecture.svg)\n\n- **Source**: The source in this specific case are `data/bookings/booking.json`\nand `data/airports/airports.dat` input files, that are batched processed by\nApache Beam. The location of these files in the architecture can be\nanything that Apache Beam can handle. Currently they are expected to be\navailable locally on the computer, where Apache Beam shall run.\n- **Apache Beam**: Apache Beam can perform both batch and stream\ndata processing, can run on many different platforms, such as Google Dataflow,\nis scalable, and can integrate with many different sources and sinks. It can also\nperform many different data processing functions. Please note, that as there is\nno free tier of Dataflow to the best of my knowledge, therefore I decided to not\nuse it, however the pipeline in the script can directly or with minor\nmodifications (e.g. source) run on Google Dataflow.\n- **Data Warehouse**: Google BigQuery is a fast Data Warehouse solution. The\nApache Beam pipeline feeds data into BigQuery tables. Apache Beam shall feed\nbookings and airports data into BigQuery. For each of them, Apache Beam shall\ndynamically create a table for incorrect data, that did not pass the data\ncleansing process. All other, proper, clean data shall be fed into already\nexisting tables.\n- **Reporting**: Reporting is done using Looker Studio. The report shows the most\nvisited countries in the user specified data range, which can also be a single\nday. Please note, that Apache Beam can handle not just batch, but also stream\nprocessing, therefore it is possible to create dynamically updating reports.\n\nThere are 3 python scripts implemented:\n\n- `code/create_empty_tables.py`: Creates empty tables for airports and bookings\nin BigQuery in your Google Cloud project. Table schemas and clustering\ninformation are of course part of the script. Make sure you have already\ncreated your Google Cloud project, and you have already created a dataset in\nBigQuery. The reason for creating empty tables before running the pipeline, is that\nit is not possible to provide clustering information in the table schema when\nwriting to BigQuery.\n- `code/dataflow/bookings_pipeline.py`: Creates an Apache Beam pipeline and\nperforms ETL batch processing on the bookings input file provided as command\nline parameter. The script can perform the batch processing either locally with\noutput to local text files, or locally with output to BigQuery tables. The\npipeline in the script can however be run on Google Dataflow and the source can\nbe changed. The pipeline can also perform stream processing of bookings, making\nit possible to create dynamically updating reports.\n- `code/dataflow/airports_pipeline.py`: Creates an Apache Beam pipeline and\nperforms ETL batch processing on the airports input file provided as command\nline parameter. The script can perform the batch processing either locally with\noutput to local text files, or locally with output to BigQuery tables. The\npipeline in the script can however be run on Google Dataflow and the source can\nbe changed.\n\n## Run Apache Beam pipeline locally with local output\n\nThe following command shall create two output files, one for the proper,\ncleaned flight bookings, and one for the incorrect bookings. The name of\nboth files shall start with the word 'output'.\n\n```bash\n$ python code/dataflow/bookings_pipeline.py -f data/bookings/booking.json\n```\n\nThe following code creates similar output files as before, but this time for\nthe airports file. Make sure those output files are either deleted, renamed\nor moved to another directory, otherwise the following code shall throw an\nexception, as the script would like to create output files with the same\nname.\n\n```bash\n$ python code/dataflow/airports_pipeline.py -f data/airports/airports.dat\n```\n\nExceptions might occur for the following reasons:\n\n- Logging the reason for an incorrect booking.\n- Other system or code related exceptions, such as input file not found,\noutput file already exists, etc.\n\nThe script might also log warnings, to report recoverable issues with data,\nsuch as incorrect age information, which in this case shall be nullified, as\nthe field in the corresponding table is NULLABLE. The logging level of the\nscript is set to ERROR, which allows the logging interface to log only\nexceptions but not warnings.\n\n## Run Apache Beam pipeline locally with output to BigQuery\n\nBefore running the scripts, you need to create a temporary Google cloud\nstorage bucket to upload the data to a BigQuery table. This storage shall\nbe used by Dataflow and BigQuery automatically. The should be globally\nunique, therefore you may need to try a couple of times, before you succeed\ncreating it.\n\nAn example for creating such a storage is shown below:\n```bash\n$ gcloud storage buckets create --location europe-west3 gs://bookings-temp\n```\n\nAfter this, you can run the scripts below.\n\n\u003e [!WARNING]\n\u003e Make sure, you replace the BigQuery table ID provided below with your own\n\u003e table ID. The default value for the bookings table ID is\n\u003e `booking-data-analysis.booking_data_analysis.bookings`. The default value\n\u003e for the airports table ID is\n\u003e `booking-data-analysis.booking_data_analysis.airports`.\n\n\nThis command below uploads the bookings data:\n```bash\n$ python code/dataflow/bookings_pipeline.py --big_query \\\n-t \u003cproject_id\u003e:\u003cdataset_id\u003e.\u003cbookings_table_name\u003e \\\n-f data/bookings/booking.json --temp_location gs://bookings-temp\n```\n\nThe command below uploads the airports data:\n```bash\n$ python code/dataflow/airports_pipeline.py --big_query \\\n-t \u003cproject_id\u003e:\u003cdataset_id\u003e.\u003cairports_table_name\u003e \\\n-f data/airports/airports.dat --temp_location gs://bookings-temp\n```\n\n## BigQuery Data Analysis\n\nOnce both tables are created in BigQuery, you can run the data analysis SQL\nscript saved in\n`code/bigquery/top_destination_countries_per_season_weekday_date_range.sql`.\n\n\u003e [!WARNING]\n\u003e It is important, that it is not possible to parameterize the table ID in\n\u003e the SQL script, as a consequence you need to manually replace the hardcoded\n\u003e table IDs. Please search\n\u003e `booking-data-analysis.booking_data_analysis.bookings` and\n\u003e `booking-data-analysis.booking_data_analysis.airports` and replace them with\n\u003e your table names, the ones you used above.\n\nAn example execution from command line using the `bq` command can be seen\nbelow. Please note, that the `bq` command is available as soon as you install\nthe `gcloud` CLI, as it was suggested above.\n\n```bash\n$ bq query --use_legacy_sql=false --format=csv -n 1000 \\\n--parameter=DS_START_DATE::20190401 \\\n--parameter=DS_END_DATE::20190430 \u003c \\\ncode/bigquery/top_destination_countries_per_season_weekday_date_range.sql\n```\n\nThe start and end date for the analysis can be provided as command line\nparameters. The format of the dates should be YYYYMMDD, where YYYY\ncorresponds to the 4 digit year, MM to the 2 digit month and DD to the\n2 digit day.\n\nThe SQL script mentioned above can be directly used, when creating a\nLooker Studio report, assuming you replaced the table IDs with the ones\nyou used.\n\n## Looker Studio Report\n\nLooker Studio can be used to visualize the results as shown in the screenshot\nbelow.\n\n![Looker Studio Report](assets/looker_studio_report.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fka-zo%2Fbooking-data-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fka-zo%2Fbooking-data-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fka-zo%2Fbooking-data-analysis/lists"}