{"id":22703622,"url":"https://github.com/alifa98/point2hex","last_synced_at":"2025-07-03T14:37:05.780Z","repository":{"id":176430004,"uuid":"610995464","full_name":"alifa98/point2hex","owner":"alifa98","description":"A CLI tool for generating higher-order trajectory datasets.","archived":false,"fork":false,"pushed_at":"2024-03-06T20:36:46.000Z","size":49099,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-06-02T18:10:58.270Z","etag":null,"topics":["cli","dataset","higher-order-trajectory","python","trajectory","trajectory-prediction"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alifa98.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-03-07T22:28:45.000Z","updated_at":"2025-03-14T09:28:59.000Z","dependencies_parsed_at":"2025-04-13T09:44:25.130Z","dependency_job_id":"a36dfbe6-e3b9-4460-be58-b18b75595a25","html_url":"https://github.com/alifa98/point2hex","commit_stats":{"total_commits":127,"total_committers":5,"mean_commits":25.4,"dds":0.5354330708661417,"last_synced_commit":"4a67786bbfeb0571095d1ee5067027f7f813470e"},"previous_names":["alifa98/point2hex"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/alifa98/point2hex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alifa98%2Fpoint2hex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alifa98%2Fpoint2hex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alifa98%2Fpoint2hex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alifa98%2Fpoint2hex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alifa98","download_url":"https://codeload.github.com/alifa98/point2hex/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alifa98%2Fpoint2hex/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261101617,"owners_count":23109860,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","dataset","higher-order-trajectory","python","trajectory","trajectory-prediction"],"created_at":"2024-12-10T08:12:01.276Z","updated_at":"2025-06-21T09:39:10.036Z","avatar_url":"https://github.com/alifa98.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Point to Hexagon\nThis is an implementation of how to convert trajectory datasets to higher-order trajectory datasets.\nWe provide the code and datasets used in our paper: [Point2Hex: Higher-order Mobility Flow Data and Resources](https://dl.acm.org/doi/10.1145/3589132.3625619).\n\n\n![Overview of Tessellation](img/heatmap1.png)\n\n\n## Table of Contents\n- [Point to Hexagon](#point-to-hexagon)\n  - [Table of Contents](#table-of-contents)\n  - [Description](#description)\n  - [Tutorial on How to Use the Code](#tutorial-on-how-to-use-the-code)\n  - [Dependencies](#dependencies)\n    - [Routing Engine](#routing-engine)\n    - [Tessellation Engine](#tessellation-engine)\n    - [Python Dependencies](#python-dependencies)\n  - [Convert Check-ins to Route Points](#convert-check-ins-to-route-points)\n    - [Data Format](#data-format)\n    - [Command Arguments for Converting Check-in Data to Trajectory Points](#command-arguments-for-converting-check-in-data-to-trajectory-points)\n    - [Map Matching](#map-matching)\n    - [Using Other Routing Engines](#using-other-routing-engines)\n  - [Convert Route Points to Hexagon Sequences](#convert-route-points-to-hexagon-sequences)\n    - [Data Format](#data-format-1)\n    - [Map Matching](#map-matching-1)\n    - [Command Arguments for Converting Route Points to Hexagon Sequences](#command-arguments-for-converting-route-points-to-hexagon-sequences)\n  - [Visualization](#visualization)\n    - [Command-line Arguments](#command-line-arguments)\n    - [Running the Script](#running-the-script)\n  - [Output](#output)\n  - [Published Datasets](#published-datasets)\n  - [License](#license)\n  - [Contact](#contact)\n  - [Citation](#citation)\n\n\n## Description\nThis repository contains a pipeline to convert datasets from check-ins to hexagon sequences.\n\nWe usually have two types of trajectory datasets: check-ins and GPS traces.\n\nA check-in dataset is a sequence of check-ins, each of which is a pair of locations as the source and destination of a trip (Taxi), or a single location as a check-in location (FourSquare).\nThis type of dataset can be gathered from social media platforms or any platforms that allow users to input check-ins, or from organizations that collect check-ins from users.\nFor example, check-ins in the taxi dataset are collected from taxi companies (we only have the source location and the destination location),\nor in the FourSquare dataset, the check-ins are collected from users (we have the check-in location, date, and the user ID).\n\nWe could also have GPS traces, which are a sequence of GPS points. These GPS traces can be collected from GPS devices or mobile phones.\nFor example, the GPS traces in the Porto taxi dataset are collected from GPS devices installed in taxis,\nor we can collect GPS traces from mobile phones, which is more common nowadays.\n\nGPS traces are more accurate than check-ins; however, they are more difficult to collect and process. Also, due to errors in locating the GPS points, the GPS traces are usually noisy and need to be cleaned and map-matched.\n\nThe following diagram shows the general pipeline for converting datasets to higher-order trajectory datasets.\n\n![Overview of Data Conversion](img/pipeline-new.png)\n\n\n## Tutorial on How to Use the Code\n- [Convert a GPS-based Trajectory Dataset to Hexagon Sequences](tutorial/Points2Hex.md)\n\n## Dependencies\n\n### Routing Engine\nWe are using OSRM as the routing engine and map-matching engine.\n\nHere is a link to learn how to install and run OSRM: [How to setup OSRM using Docker](https://github.com/Project-OSRM/osrm-backend#using-docker)\n\n### Tessellation Engine\nWe are also using H3 as the hexagon tessellation engine.\n\n\n### Python Dependencies\nFor the Python code, you can install the dependencies by running the following command:\n```sh\nconda env create --name point2hex --file=environment.yml\n```\n\nThe command above will create a conda environment named `point2hex` and install the dependencies in that environment.\n\nTo run the command, you need to have conda installed on your system.\n\n\n## Convert Check-ins to Route Points\n\nTo convert check-ins to route points, we need to use a routing engine to generate the route points between the source and destination of each check-in.\nMake sure you have the information of the routing engine you are using, such as the host and port of the routing engine.\n\n### Data Format\nThe input data should be a CSV file and should have the following columns:\n\n```\nstart_point_lon, start_point_lat, end_point_lon, end_point_lat\n```\n\n\n### Command Arguments for Converting Check-in Data to Trajectory Points\n\nThe `loc2point_run.py` script accepts command-line arguments for converting check-in data to trajectory points.\nBelow is a description of each argument:\n\n- `input_file` (positional argument): The path to the input file containing the check-in data.\n\nOptional arguments:\n\n- `-o, --output`: The path to the output file. If not specified, the default value is `output.csv`.\n\n- `-slon, --start-column-longitude`: The column name for the longitude of the start point. Default value: `start_point_lon`.\n\n- `-slat, --start-column-latitude`: The column name for the latitude of the start point. Default value: `start_point_lat`.\n\n- `-elon, --end-column-longitude`: The column name for the longitude of the end point. Default value: `end_point_lon`.\n\n- `-elat, --end-column-latitude`: The column name for the latitude of the end point. Default value: `end_point_lat`.\n\n- `-or, --output-route`: The column name for the route in the output file. Default value: `route_points`.\n\n- `-u, --base-url`: The base URL of the routing service (specifically OSRM). Default value: `http://127.0.0.1:5000`.\n\n- `-t, --threads`: The number of threads to use for processing. Default value: `70`.\n\n- `-s, --split`: Flag to indicate whether to save the output of each thread separately. If the split option is enabled, the output file will contain multiple CSV files (one for each thread). Default value: `False`.\n\nTo run the script with the desired arguments, use the following command:\n\n```\npython loc2point_run.py [input_file] [-o OUTPUT] [-slon START_COLUMN_LONGITUDE] [-slat START_COLUMN_LATITUDE]\n               [-elon END_COLUMN_LONGITUDE] [-elat END_COLUMN_LATITUDE] [-or OUTPUT_ROUTE] [-u BASE_URL]\n               [-t THREADS] [-s]\n```\n\nMake sure to replace `[input_file]` with the actual path to the input file.\n\nNote: The default values mentioned above are used when the corresponding argument is not provided.\n\n\n### Map Matching\nAs we use OSRM to generate the route points, we do not need map matching for check-ins because we already have map-matched points for our trajectories.\n\n### Using Other Routing Engines\nOur implementation is compatible with other routing engines. You just need to implement the routing engine API like the `lib/api/OpenStreetMap.py` file and change the API class in `loc2point_run.py` to your routing engine API class.\n\n## Convert Route Points to Hexagon Sequences\n\nIf the dataset is a GPS-based dataset originally, we need to map-match the GPS points to the road network and then convert the map-matched GPS points to hexagon sequences.\nIf the dataset is a check-in dataset, and we have used a routing engine to generate the route points, we can directly convert the route points to hexagon sequences.\n\n### Data Format\nEach row in the input CSV files should have a column that contains a list of tuples or a list of lists, each containing two items (longitude and latitude).\nFor example: `[(long, lat), (long, lat),...(long, lat)]`.\n\n\n### Map Matching\n\nFor map matching of the route points, we use OSRM to map-match the route points to the road network.\nYou can run the `matching_run.py` script to map-match the route points.\n\nThe `matching_run.py` script is designed to match route points on a map using a Map Matching Service.\nIt utilizes threading to improve performance by processing multiple routes simultaneously.\nThe script takes command-line arguments to specify input and output files, column names, base URL of the Map Matching Service, and the number of threads to use for parallel processing.\n\n**Usage:**\n```\npython matching_run.py [input_file] [-o OUTPUT] [-c COLUMN] [-u BASE_URL] [-t THREADS]\n```\n\n**Arguments:**\n- `input_file`: Path to the input file containing route data.\n- `-o, --output OUTPUT`: Path to the output file where the matched route data will be saved. (Default: 'output.csv')\n- `-c, --column COLUMN`: The name of the column in the input file that contains the route points. (Default: 'route_points')\n- `-u, --base-url BASE_URL`: The base URL of the Map Matching Service. (Default: 'http://127.0.0.1:5000')\n- `-t, --threads THREADS`: The number of threads to use for parallel processing. (Default: 70)\n\n**Script Overview:**\n1. Parse the command-line arguments.\n2. Define the `MatchRoutePointsThread` class, which represents a thread responsible for matching route points.\n3. Initialize the Map Matching Service API and the logger.\n4. Load the input data file specified by `input_file`.\n5. Split the data points into multiple segments based on the number of threads.\n6. Create threads for each segment of data.\n7. Start the threads to initiate the map matching process.\n8. Wait for all threads to finish their execution.\n9. Save the output data to the specified output file.\n10. Display a completion message.\n\n**Thread Execution:**\nThe script uses threading to process route points in parallel.\nEach thread represents a segment of the data and is responsible for matching the route points within that segment.\nThe threads operate as follows:\n1. Each thread retrieves route points from the input data for a trajectory.\n2. If the number of route points is less than 2, a warning is logged, and the thread skips to the next trajectory.\n3. The thread sends a request to the Map Matching Service API with the original route points.\n4. If the response is successful, the matched route points are obtained from the response. If multiple segments are returned, the thread will send requests for routing between the matched segments (the routing is done by the OSRM routing engine, between the last point of the previous segment and the first point of the next segment of the trajectory).\n5. If the response is not successful, an error is logged, and the thread skips to the next segment.\n6. Once all threads have finished executing, the output data is saved to the specified output file.\n\n**Example of Map Matching on the City Map**\n\nHere is an example of running the map matching on the Porto dataset:\n\n\nNoisy GPS Trace             |  Map-Matched GPS Trace\n:-------------------------:|:-------------------------:\n\u003cimg src=img/unmatched-gps-trace.png width=50% height=50%\u003e  |  \u003cimg src=img/matched-gps-trace.png width=50% height=50%\u003e\n\n\n### Command Arguments for Converting Route Points to Hexagon Sequences\n\nThe `point2hex_run.py` script accepts command-line arguments for converting sequences of geographical coordinates into a sequence of hexagons.\nBelow is a description of each argument:\n\n- `input_dir` (positional argument): The directory path where the input CSV files containing the route data are stored.\n\nOptional arguments:\n\n- `-o, --output`: The directory path where the output file will be stored. The output file will be named as `output_resX.csv`, where X denotes the resolution of hexagons. If not specified, the default value is `./` (current directory).\n\n- `-r, --resolution`: A list of resolutions for generating hexagons, provided as comma-separated values. The default resolution is `6` if this argument is not provided.\n\n- `-c, --column`: The name of the column in the input .csv files that contains route points. The default column name is `route_points` if this argument is not provided.\n\nTo run the script with the desired arguments, use the following command:\n\n```\npython point2hex_run.py [input_dir] [-o OUTPUT] [-r RESOLUTION] [-c COLUMN]\n```\n\nMake sure to replace `[input_dir]` with the actual path to the directory containing the input CSV files.\n\nNote: The default values mentioned above are used when the corresponding argument is not provided.\n\n\n## Visualization\nThe `plot_run.py` script reads an input CSV file containing higher order trajectory sequences and generates a hexagon heatmap visualization.\nThe script allows setting the zoom level of the map visualization via command-line arguments.\n\n### Command-line Arguments\n\nThe script accepts the following command-line arguments:\n\n- `input_file` (positional argument): The path to the input CSV file containing higher order trajectory sequences.\n\n- `-z, --zoom` (optional argument): The zoom level of the map visualization. If not specified, the default zoom level is set to `12`.\n\n### Running the Script\n\nTo execute the script, run the following command:\n\n```bash\npython plot_run.py \u003cinput_file\u003e [-z ZOOM]\n```\n\nHere `\u003cinput_file\u003e` should be replaced with the actual path to the input file, and `-z ZOOM` is an optional argument specifying the zoom level of the map.\nIf the zoom level is not provided, it defaults to `12`.\n\nFor example:\n\n```bash\npython plot_run.py ./input/data.csv -z 11.5\n```\n\nIn this example, the script will read the CSV file from the `./input/data.csv` path, and the map zoom level will be `11.5`.\n\n\nNote: The hexagons should be in a column named `higher_order_trajectory` separated by a space in the input file.\nThe column name is the same as the output column name of the `point2hex_run.py` script for hexagon sequences.\n\n\n## Output\n\nThe script generates a heatmap visualization using the provided higher-order trajectory sequences.\nThe heatmap will be saved in the `out/plots` folder with the name `heatmap-zoomX-timestamp.jpeg`.\n\n\n## Published Datasets\nDue to storage limitations, you can find more datasets on [Zenodo](https://zenodo.org/records/8076553). \n\n\n## License\nDistributed under the MIT License.\n\n\n## Contact\n\nAli Faraji - faraji@yorku.ca\n\nJing Li - jliellen@yorku.ca\n\nProject Link: [https://github.com/alifa98/point2hex](https://github.com/alifa98/point2hex)\n\n\n## Citation\n\nIf you like our work and/or use the code/datasets we provide, please consider citing our work as:\n\nAli Faraji, Jing Li, Gian Alix, Mahmoud Alsaeed, Nina Yanin, Amirhossein Nadiri, and Manos Papagelis. 2023. Point2Hex: Higher-order Mobility Flow Data and Resources. In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems (SIGSPATIAL '23). Association for Computing Machinery, New York, NY, USA, Article 69, 1–4. https://doi.org/10.1145/3589132.3625619\n\n\nAlternatively, you can use the following BibTeX formatting:\n\n```tex\n@inproceedings{Faraji2023point2hex,\nauthor = {Faraji, Ali and Li, Jing and Alix, Gian and Alsaeed, Mahmoud and Yanin, Nina and Nadiri, Amirhossein and Papagelis, Manos},\ntitle = {Point2Hex: Higher-order Mobility Flow Data and Resources},\nyear = {2023},\nisbn = {9798400701689},\npublisher = {Association for Computing Machinery},\naddress = {New York, NY, USA},\nurl = {https://doi.org/10.1145/3589132.3625619},\ndoi = {10.1145/3589132.3625619},\nbooktitle = {Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems},\narticleno = {69},\nnumpages = {4},\nkeywords = {trajectory datasets, higher-order mobility flow datasets, generator},\nlocation = {Hamburg, Germany},\nseries = {SIGSPATIAL '23}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falifa98%2Fpoint2hex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falifa98%2Fpoint2hex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falifa98%2Fpoint2hex/lists"}