{"id":28567028,"url":"https://github.com/altinity/clickhouse-demo-dataset-migration","last_synced_at":"2026-02-02T15:03:08.790Z","repository":{"id":103608487,"uuid":"98982545","full_name":"Altinity/clickhouse-demo-dataset-migration","owner":"Altinity","description":null,"archived":false,"fork":false,"pushed_at":"2017-08-07T07:10:54.000Z","size":16,"stargazers_count":11,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-15T11:50:10.262Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Altinity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-08-01T09:12:47.000Z","updated_at":"2025-04-04T02:07:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"c47d1170-8604-49ac-a3a0-4012618892e3","html_url":"https://github.com/Altinity/clickhouse-demo-dataset-migration","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Altinity/clickhouse-demo-dataset-migration","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-demo-dataset-migration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-demo-dataset-migration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-demo-dataset-migration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-demo-dataset-migration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Altinity","download_url":"https://codeload.github.com/Altinity/clickhouse-demo-dataset-migration/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclickhouse-demo-dataset-migration/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29013710,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-02T14:58:54.169Z","status":"ssl_error","status_checked_at":"2026-02-02T14:58:51.285Z","response_time":58,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-10T15:40:26.564Z","updated_at":"2026-02-02T15:03:08.783Z","avatar_url":"https://github.com/Altinity.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# DEMO DATASET setup\n\n------\n\n## Table of Contents\n\n  * [Introduction](#introduction)\n  * [Preparation](#preparation)\n    * [Prepare ClickHouse Repo](#prepare-clickhouse-repo)\n    * [Prepare Etalon Dataset Server Access](#prepare-etalon-dataset-server-access)\n  * [Install and Configure ClickHouse](#install-and-configure-clickhouse)\n    * [Install ClickHouse](#install-clickhouse)\n    * [Configure ClickHouse](#configure-clickhouse)\n      * [Setup Users](#setup-users)\n      * [Setup Dictionaries](#setup-dictionaries)\n  * [SSH-tunnel setup](#ssh-tunnel-setup)\n  * [Datasets setup](#datasets-setup)\n    * [Dataset NYC Taxi Rides](#dataset-nyc-taxi-rides)\n      * [Setup NYC Taxi Rides Database](#setup-nyc-taxi-rides-database)\n      * [Setup NYC Taxi Rides Tables](#setup-nyc-taxi-rides-tables)\n      * [Copy NYC Taxi Rides Dataset](#copy-nyc-taxi-rides-dataset)\n      * [Check NYC Taxi Rides Dataset](#check-nyc-taxi-rides-dataset)\n    * [Dataset STAR](#dataset-star)\n      * [Setup STAR Database](#setup-star-database)\n      * [Setup STAR Tables](#setup-star-tables)\n      * [Copy STAR Dataset](#copy-star-dataset)\n      * [Check STAR Dataset](#check-star-dataset)\n    * [Dataset AIRLINE](#dataset-airline)\n      * [Setup AIRLINE Database](#setup-airline-database)\n      * [Setup AIRLINE Tables](#setup-airline-tables)\n      * [Copy AIRLINE Dataset](#copy-airline-dataset)\n      * [Check AIRLINE Dataset](#check-airline-dataset)\n  * [Close SSH-tunnel](#close-ssh-tunnel)\n  * [Conclusion](#conclusion)\n\n------\n\n\n## Introduction\n\nAll instructions in this manual were tested on Ubuntu 16.04.\nThere is no need to setup all datasets from this manual - feel free to skip any of them, if you don't need it.\nSSH-tunnel section is provided because 'etalon dataset server' is located behind the firewall, but you may not need this step.\n\n## Preparation\n\n### Prepare ClickHouse Repo\n\nEnsure we have all `apt` - related tools installed\n```bash\nsudo apt install software-properties-common\n```\n\nInclude keyserver for repo\n```bash\nsudo apt-key adv --keyserver keyserver.ubuntu.com --recv E0C56BD4\n```\n\nFigure out distro codename\n```bash\ncodename=`lsb_release -c|awk '{print $2}'`\necho $codename\n```\n\nBuild URL to ClickHouse repo based on distro’s codename\n```bash\nREPOURL=\"http://repo.yandex.ru/clickhouse/$codename\"\necho $REPOURL\n```\n\nAdd ClickHouse repo located in `$REPOURL`\n```bash\nsudo apt-add-repository \"deb $REPOURL stable main\"\n```\n\nUpdate list of available packages\n```bash\nsudo apt update\n```\n\n### Prepare Etalon Dataset Server Access\n\nYou'll need to prepare: \n  * hostname or IP address of the 'etalon dataset server' \n  * access key in order to get SSH-access to 'etalon dataset server'\n\nReplace 127.0.0.1 with your 'etalon dataset server' address/hostname\n\n```bash\n# replace 127.0.0.1 with your 'etalon dataset server' address/hostname\nDATASET_SERVER=\"127.0.0.1\" \n```\n\nEnsure you either already have access key in your `~/.ssh/` folder\n\n```bash\nls -l ~/.ssh/\n...\n-rw-------  1 user user 1675 Aug  2 11:55 chdemo\n...\n\n```\n\nor, if you don't have access key, create access key file and store ssh-access key locally\n\n\n```bash\nmkdir -p ~/.ssh\ntouch ~/.ssh/chdemo\n```\n\nEdit `~/.ssh/chdemo` and save key in it\\\nAlso it has to have limited access rights\n```bash\nchmod 600 ~/.ssh/chdemo\n```\n\nSpecify **FULL PATH** to acess key file as ENV variable.\\\n**IMPORTANT** - please, do not use shortcuts like `~/.ssh/chdemo` - specify real **FULL PATH**. Shortcuts will lead to mess when expanding it in sub-shell's.\\\n**PLEASE** - full path only.\n\n```bash\n# replace \"chdemo\" with your FULL PATH to your 'etalon dataset server' access key file\nDATASET_SERVER_KEY_FILENAME=\"/home/user/.ssh/chdemo\" \n```\n\n## Install and configure ClickHouse\n\n### Install ClickHouse\n\nInstall all ClickHouse-related packages: server, client \u0026 tools\n```bash\nsudo apt install clickhouse-client 'clickhouse-server*'\n```\n\nNow let’s setup installed ClickHouse\n\nEnsure service is down\n```bash\nsudo service clickhouse-server stop\n```\n\nCreate folder where ClickHouse data would be kept. Default is `/var/lib/clickhouse`, if it is ok, just skip this section\n```bash\nsudo mkdir -p /data1/clickhouse\nsudo mkdir -p /data1/clickhouse/tmp\nsudo chown -R clickhouse.clickhouse /data1/clickhouse\n```\n\nCreate folder where dictionaries specs would be kept\n```bash\nsudo mkdir -p /etc/clickhouse-server/dicts\nsudo chown -R clickhouse.clickhouse /etc/clickhouse-server/dicts\n```\n\n### Configure ClickHouse\n\nSetup ClickHouse to listen on all network interfaces for both IPv4 and IPv6 \\\nEdit file `/etc/clickhouse-server/config.xml` \\\nEnsure `\u003clisten_host\u003e` tags have the following content:\n\n```xml\n        \u003clisten_host\u003e::\u003c/listen_host\u003e\n        \u003clisten_host\u003e0.0.0.0\u003c/listen_host\u003e\n```\n\nSetup ClickHouse to keep data in specified dirs - in case default `/var/lib/clickhouse` is not OK\\\nEdit file `/etc/clickhouse-server/config.xml`\\\nEnsure `\u003cpath\u003e` and `\u003ctmp_path\u003e` tags have the following content:\n\n```xml\n\t\u003c!-- Path to data directory, with trailing slash. --\u003e\n\t\u003cpath\u003e/data1/clickhouse/\u003c/path\u003e\n\t\u003c!-- Path to temporary data for processing hard queries. --\u003e\n\t\u003ctmp_path\u003e/data1/clickhouse/tmp/\u003c/tmp_path\u003e\n```\nSetup ClickHouse to look for dictionaries in specified dir\\\nEdit file `/etc/clickhouse-server/config.xml`\\\nEnsure `\u003cdictionaries_config\u003e` tag has the following content:\n\n```xml\n\u003cdictionaries_config\u003e/etc/clickhouse-server/dicts/*.xml\u003c/dictionaries_config\u003e\n```\n\n#### Setup Users\n\nSetup access for default user from localhost only.\\\nEdit file `/etc/clickhouse-server/users.xml`\\\nEnsure `default` user (located inside `\u003cyandex\u003e\u003cusers\u003e\u003cdefault\u003e`) has `\u003cip\u003e` tags specified with localhost values only\n\n```xml\n\u003cdefault\u003e\n\t\u003cnetworks incl=\"networks\" replace=\"replace\"\u003e\n\t\t\u003cip\u003e::1\u003c/ip\u003e\n\t\t\u003cip\u003e127.0.0.1\u003c/ip\u003e\n\t\u003c/networks\u003e\n\u003c/default\u003e\n```\n\nSetup read-only user for ClickHouse with access from all over the world.\\\nUsername would be testuser and it would not have any password.\\\nEdit file `/etc/clickhouse-server/users.xml`\\\nAdd new profile called `readonly_set_settings` in `\u003cyandex\u003e\u003cprofiles\u003e` section right after `\u003cdefault\u003e` profile tag\n\n```xml\n\u003c!-- Profile that allows only read queries and SET settings. --\u003e\n\u003creadonly_set_settings\u003e\n\t\u003c!-- Maximum memory usage for processing single query, in bytes. --\u003e\n\t\u003cmax_memory_usage\u003e10000000000\u003c/max_memory_usage\u003e\n\n\t\u003c!-- Use cache of uncompressed blocks of data. Meaningful only for processing many of very short queries. --\u003e\n\t\u003cuse_uncompressed_cache\u003e0\u003c/use_uncompressed_cache\u003e\n\n\t\u003c!-- How to choose between replicas during distributed query processing.\n\trandom - choose random replica from set of replicas with minimum number of errors\n\tnearest_hostname - from set of replicas with minimum number of errors, choose replica\n\twith minumum number of different symbols between replica's hostname and local hostname\n\t(Hamming distance).\n\tin_order - first live replica is choosen in specified order.\n\t--\u003e\n\t\u003cload_balancing\u003erandom\u003c/load_balancing\u003e\n\t\u003creadonly\u003e2\u003c/readonly\u003e\n\u003c/readonly_set_settings\u003e\n```\n\nAdd new `\u003ctestuser\u003e` tag with profile referring to just inserted `readonly_set_settings` profile in `\u003cyandex\u003e\u003cusers\u003e` section right after `\u003cdefault\u003e` user tag:\n\n```xml\n\u003ctestuser\u003e\n\t\u003cpassword\u003e\u003c/password\u003e\n\t\u003cnetworks incl=\"networks\" replace=\"replace\"\u003e\n\t\t\u003c!-- access from everywhere --\u003e\n\t\t\u003cip\u003e::/0\u003c/ip\u003e\n\t\t\u003cip\u003e0.0.0.0\u003c/ip\u003e\n\t\u003c/networks\u003e\n\t\u003cprofile\u003ereadonly_set_settings\u003c/profile\u003e\n\t\u003cquota\u003edefault\u003c/quota\u003e\n\u003c/testuser\u003e\n```\n\n#### Setup Dictionaries\n\nPrepare dictionaries specifications.\\\nWe’ll need **SSH** access to ‘etalon dataset server’.\\\nCopy dictionaries specifications from ‘etalon dataset server’ to `/etc/clickhouse-server/dicts`\n\n```bash\ncd /etc/clickhouse-server/dicts\nsudo scp -i $DATASET_SERVER_KEY_FILENAME -P 2222 \"root@$DATASET_SERVER:/etc/clickhouse-server/dicts/*\" .\n```\n\nEnsure we have the following files in `/etc/clickhouse-server/dicts`\n\n```bash\nls -l /etc/clickhouse-server/dicts/*\n-rw-r--r-- 1 clickhouse clickhouse  831 Jul  6 06:25 taxi_zones.xml\n-rw-r--r-- 1 clickhouse clickhouse 2392 Jul  6 06:26 weather.xml\n```\n\nEnsure ClickHouse server is running\n```bash\nsudo service clickhouse-server restart\n```\n\n## SSH-tunnel setup\n\nAlso we need to have ClickHouse to have access to ‘etalon dataset server’. Since it is behind the firewall, we need to setup SSH-tunnel for this. \\\nMake local socket `127.0.0.1:9999` to be forwarded on server `$DATASET_SERVER` to local socket `127.0.0.1:9000` on that server. \\\nThus, connecting to `127.0.0.1:9999` we’ll have connect via **SSH** to `127.0.0.1:9000` on server `$DATASET_SERVER`\n\n```bash\nssh -f -N -i $DATASET_SERVER_KEY_FILENAME -p 2222 root@$DATASET_SERVER -L 127.0.0.1:9999:127.0.0.1:9000\n```\n\n## Datasets setup\n\nNow let’s setup demo datasets\n\n### Dataset NYC Taxi Rides\n\nNow let’s setup New-York City Taxi Rides dataset\n\n#### Setup NYC Taxi Rides Database\nCreate database we’ll use\n\n```bash\nclickhouse-client -q \"CREATE DATABASE IF NOT EXISTS nyc_taxi_rides;\"\n```\n\n#### Setup NYC Taxi Rides Tables\n\nDrop existing tables if they already exist\n\n```bash\nclickhouse-client -q \"DROP TABLE IF EXISTS nyc_taxi_rides.central_park_weather_observations;\"\nclickhouse-client -q \"DROP TABLE IF EXISTS nyc_taxi_rides.taxi_zones;\"\nclickhouse-client -q \"DROP TABLE IF EXISTS nyc_taxi_rides.tripdata;\"\n```\n\nCreate tables we’ll use\n\n```bash\nclickhouse-client -q \"CREATE TABLE nyc_taxi_rides.central_park_weather_observations (\n  station_id String,  \n  station_name String,  \n  weather_date Date,  \n  precipitation Float32,  \n  snow_depth Float32,  \n  snowfall Int32,  \n  max_temperature Float32,  \n  min_temperature Float32,  \n  average_wind_speed Float32\n) ENGINE = MergeTree(weather_date, station_id, 8192);\"\n\nclickhouse-client -q \"CREATE TABLE nyc_taxi_rides.taxi_zones (\n  location_id UInt32,  \n  zone String,  \n  create_date Date DEFAULT toDate(0)\n) ENGINE = MergeTree(create_date, location_id, 8192);\"\n\nclickhouse-client -q \"CREATE TABLE nyc_taxi_rides.tripdata (\n  pickup_date Date DEFAULT toDate(tpep_pickup_datetime),  \n  id UInt64,  \n  vendor_id String,  \n  tpep_pickup_datetime DateTime,  \n  tpep_dropoff_datetime DateTime,  \n  passenger_count Int32,  \n  trip_distance Float32,  \n  pickup_longitude Float32,  \n  pickup_latitude Float32,  \n  rate_code_id String,  \n  store_and_fwd_flag String,  \n  dropoff_longitude Float32,  \n  dropoff_latitude Float32,  \n  payment_type String,  \n  fare_amount String,  \n  extra String,  \n  mta_tax String,  \n  tip_amount String,  \n  tolls_amount String,  \n  improvement_surcharge String,  \n  total_amount Float32,  \n  pickup_location_id UInt32,  \n  dropoff_location_id UInt32,  \n  junk1 String,  \n  junk2 String\n) ENGINE = MergeTree(pickup_date, (id, pickup_location_id, dropoff_location_id, vendor_id), 8192);\"\n```\n\n#### Copy NYC Taxi Rides Dataset\n\nFill newly created tables with data from remote ‘etalon dataset server’\n\n**IMPORTANT:** This operation requires big amount of data to be copied and takes quite long time\n\n```bash\nclickhouse-client -q \"INSERT INTO nyc_taxi_rides.central_park_weather_observations SELECT * FROM remote('127.0.0.1:9999', 'nyc_taxi_rides.central_park_weather_observations');\"\nclickhouse-client -q \"INSERT INTO nyc_taxi_rides.taxi_zones SELECT * FROM remote('127.0.0.1:9999', 'nyc_taxi_rides.taxi_zones');\"\nclickhouse-client -q \"INSERT INTO nyc_taxi_rides.tripdata SELECT * FROM remote('127.0.0.1:9999', 'nyc_taxi_rides.tripdata');\"\n```\n\n#### Check NYC Taxi Rides Dataset\n\nAfter all data copied ensure we have main tables filled with data:\n\n```bash\nclickhouse-client -q \"SELECT count() FROM nyc_taxi_rides.central_park_weather_observations;\"\nclickhouse-client -q \"SELECT count() FROM nyc_taxi_rides.taxi_zones;\"\nclickhouse-client -q \"SELECT count() FROM nyc_taxi_rides.tripdata;\"\n```\n\nEnsure all dictionaries are healthy via\n\n```bash\nclickhouse-client -q \"SELECT * FROM system.dictionaries;\"\n```\n\nThere should be two dictionaries and no errors reported on their statuses \n\n### Dataset STAR\n\nNow let’s setup Star Observations dataset\n\n#### Setup STAR Database\n\nCreate database we’ll use\n\n```bash\nclickhouse-client -q \"CREATE DATABASE IF NOT EXISTS star;\"\n```\n\n#### Setup STAR Tables\n\nDrop existing tables if they already exist\n\n```bash\nclickhouse-client -q \"DROP TABLE IF EXISTS star.starexp;\"\n```\n\nCreate tables we’ll use\n\n```bash\nclickhouse-client -q \"CREATE TABLE star.starexp (\n  antiNucleus UInt32,\n  eventFile UInt32,  \n  eventNumber UInt32,\n  eventTime Float64,\n  histFile UInt32,\n  multiplicity UInt32,\n  NaboveLb UInt32,\n  NbelowLb UInt32,\n  NLb UInt32,\n  primaryTracks UInt32,\n  prodTime Float64,\n  Pt Float32,\n  runNumber UInt32,\n  vertexX Float32,\n  vertexY Float32,\n  vertexZ Float32,\n  eventDate Date DEFAULT  CAST(concat(substring(toString(floor(eventTime)), 1, 4), '-', substring(toString(floor(eventTime)), 5, 2), '-', substring(toString(floor(eventTime)), 7, 2)) AS Date)\n) ENGINE = MergeTree(eventDate, (eventNumber, eventTime, runNumber, eventFile, multiplicity), 8192);\"\n```\n\n#### Copy STAR Dataset\n\nFill newly created tables with data from remote ‘etalon dataset server’\n\n**IMPORTANT:** This operation requires big amount of data to be copied and takes quite long time\n\n```bash\nclickhouse-client -q \"INSERT INTO star.starexp SELECT * FROM remote('127.0.0.1:9999', 'star.starexp');\"\n```\n\n#### Check STAR Dataset\n\nAfter all data copied ensure we have main tables filled with data:\n\n```bash\nclickhouse-client -q \"SELECT count() FROM star.starexp;\"\n```\n\n### Dataset AIRLINE\n\nNow let’s setup AIRLINE dataset\n\n#### Setup AIRLINE Database\n\nCreate database we’ll use\n\n```bash\nclickhouse-client -q \"CREATE DATABASE IF NOT EXISTS airline;\"\n```\n\n#### Setup AIRLINE Tables\n\nDrop existing tables if they already exist\n\n```bash\nclickhouse-client -q \"DROP TABLE IF EXISTS airline.ontime;\"\n```\n\nCreate tables we’ll use\n\n```bash\nclickhouse-client -q \"CREATE TABLE IF NOT EXISTS airline.ontime (\n  Year UInt16,\n  Quarter UInt8,\n  Month UInt8,\n  DayofMonth UInt8,\n  DayOfWeek UInt8,\n  FlightDate Date,\n  UniqueCarrier String,\n  AirlineID UInt32,\n  Carrier String,\n  TailNum String,\n  FlightNum String,\n  OriginAirportID UInt32,\n  OriginAirportSeqID UInt32,\n  OriginCityMarketID UInt32,\n  Origin String,\n  OriginCityName String,\n  OriginState String,\n  OriginStateFips String,\n  OriginStateName String,\n  OriginWac UInt32,\n  DestAirportID UInt32,\n  DestAirportSeqID UInt32,\n  DestCityMarketID UInt32,\n  Dest String,\n  DestCityName String,\n  DestState String,\n  DestStateFips String,\n  DestStateName String,\n  DestWac UInt32,\n  CRSDepTime UInt32,\n  DepTime UInt32,\n  DepDelay Float32,\n  DepDelayMinutes Float32,\n  DepDel15 Float32,\n  DepartureDelayGroups Int32,\n  DepTimeBlk String,\n  TaxiOut Float32,\n  WheelsOff UInt32,\n  WheelsOn UInt32,\n  TaxiIn Float32,\n  CRSArrTime UInt32,\n  ArrTime UInt32,\n  ArrDelay Float32,\n  ArrDelayMinutes Float32,\n  ArrDel15 Float32,\n  ArrivalDelayGroups Int32,\n  ArrTimeBlk String,\n  Cancelled Float32,\n  CancellationCode String,\n  Diverted Float32,\n  CRSElapsedTime Float32,\n  ActualElapsedTime Float32,\n  AirTime Float32,\n  Flights Float32,\n  Distance Float32,\n  DistanceGroup Float32,\n  CarrierDelay Float32,\n  WeatherDelay Float32,\n  NASDelay Float32,\n  SecurityDelay Float32,\n  LateAircraftDelay Float32,\n  FirstDepTime String,\n  TotalAddGTime String,\n  LongestAddGTime String,\n  DivAirportLandings String,\n  DivReachedDest String,\n  DivActualElapsedTime String,\n  DivArrDelay String,\n  DivDistance String,\n  Div1Airport String,\n  Div1AirportID UInt32,\n  Div1AirportSeqID UInt32,\n  Div1WheelsOn String,\n  Div1TotalGTime String,\n  Div1LongestGTime String,\n  Div1WheelsOff String,\n  Div1TailNum String,\n  Div2Airport String,\n  Div2AirportID UInt32,\n  Div2AirportSeqID UInt32,\n  Div2WheelsOn String,\n  Div2TotalGTime String,\n  Div2LongestGTime String,\n  Div2WheelsOff String,\n  Div2TailNum String,\n  Div3Airport String,\n  Div3AirportID UInt32,\n  Div3AirportSeqID UInt32,\n  Div3WheelsOn String,\n  Div3TotalGTime String,\n  Div3LongestGTime String,\n  Div3WheelsOff String,\n  Div3TailNum String,\n  Div4Airport String,\n  Div4AirportID UInt32,\n  Div4AirportSeqID UInt32,\n  Div4WheelsOn String,\n  Div4TotalGTime String,\n  Div4LongestGTime String,\n  Div4WheelsOff String,\n  Div4TailNum String,\n  Div5Airport String,\n  Div5AirportID UInt32,\n  Div5AirportSeqID UInt32,\n  Div5WheelsOn String,\n  Div5TotalGTime String,\n  Div5LongestGTime String,\n  Div5WheelsOff String,\n  Div5TailNum String\n)\nENGINE = MergeTree(FlightDate, (FlightDate, Year, Month, DepDel15), 8192);\"\n```\n\n#### Copy AIRLINE Dataset\n\nFill newly created tables with data from remote ‘etalon dataset server’\n\n**IMPORTANT**: This operation requires big amount of data to be copied and takes quite long time\n\n```bash\nclickhouse-client -q \"INSERT INTO airline.ontime SELECT * FROM remote('127.0.0.1:9999', 'airline.ontime');\"\n```\n\n#### Check AIRLINE Dataset\n\nAfter all data copied ensure we have main tables filled with data:\n\n```bash\nclickhouse-client -q \"SELECT count() FROM airline.ontime;\"\n```\n\n## Close SSH-tunnel\n\nNow let’s terminate SSH-tunnel to ‘etalon dataset server’\n\nFind `SSH`-tunnel process `PID`\n```bash\nSSHPID=`sudo netstat -antp|grep LIST|grep 9999|grep ssh|awk '{print $7}'| sed -e 's/\\/.*//g'`\necho $SSHPID\n```\n\nEnsure found `SSH` `PID` is reasonable - it is our `SSH`-tunnel\n```bash\nps ax | grep $SSHPID | grep -v grep\n```\n\nand kill it with kill command\n```bash\nkill $SSHPID\n```\n\n## Conclusion\n\nIn case all steps were completed successfully, we'll have local copy of one (or more) datasets migrated from 'etalon dataset server'\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltinity%2Fclickhouse-demo-dataset-migration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faltinity%2Fclickhouse-demo-dataset-migration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltinity%2Fclickhouse-demo-dataset-migration/lists"}