{"id":41202629,"url":"https://github.com/databricks-industry-solutions/ship2ship-transfers","last_synced_at":"2026-01-22T21:30:42.882Z","repository":{"id":159960652,"uuid":"521836200","full_name":"databricks-industry-solutions/ship2ship-transfers","owner":"databricks-industry-solutions","description":"Ship-to-Ship Transfer Identification using Geospatial Analytics","archived":false,"fork":false,"pushed_at":"2023-05-01T17:58:34.000Z","size":65,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-01-29T09:36:35.606Z","etag":null,"topics":["databricks-industry-solutions","geospatial","mfg","pubsec"],"latest_commit_sha":null,"homepage":"https://medium.com/@timo.roest/ship-to-ship-transfer-detection-b370dd9d43e8","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks-industry-solutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null}},"created_at":"2022-08-06T03:28:32.000Z","updated_at":"2023-11-25T15:14:48.000Z","dependencies_parsed_at":"2023-09-25T01:46:33.288Z","dependency_job_id":null,"html_url":"https://github.com/databricks-industry-solutions/ship2ship-transfers","commit_stats":{"total_commits":22,"total_committers":4,"mean_commits":5.5,"dds":0.5454545454545454,"last_synced_commit":"8942ebcb2a4cb00888112c1726b8db4759af9abd"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":"databricks-industry-solutions/industry-solutions-blueprints","purl":"pkg:github/databricks-industry-solutions/ship2ship-transfers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fship2ship-transfers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fship2ship-transfers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fship2ship-transfers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fship2ship-transfers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks-industry-solutions","download_url":"https://codeload.github.com/databricks-industry-solutions/ship2ship-transfers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fship2ship-transfers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28671719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T20:48:19.482Z","status":"ssl_error","status_checked_at":"2026-01-22T20:48:14.968Z","response_time":144,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["databricks-industry-solutions","geospatial","mfg","pubsec"],"created_at":"2026-01-22T21:30:41.892Z","updated_at":"2026-01-22T21:30:42.876Z","avatar_url":"https://github.com/databricks-industry-solutions.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ship2Ship Transfer Detection\nShip Overlap\n\n\u003cimg src=\"https://github.com/databrickslabs/mosaic/raw/main/notebooks/examples/python/Ship2ShipTransfers/images/kepler_output.png\" width=70%\u003e\n\n## Introduction\nThis is an algorithmic implementation to detect Ship to Ship transfers at scale using Databricks. It was presented at the [Data and AI Summit 2022](https://www.youtube.com/watch?v=XQNflqbgP7Q). \n\nThis Mosaic example explores a novel, algorithmic approach to detecting Ship to Ship transfers at scale using AIS data. In particular it aims to surpass existing, naive implementations that are just based on a particular distance radius like the one shown below:\n\n\u003cimg src=\"https://github.com/databrickslabs/mosaic/raw/main/notebooks/examples/python/Ship2ShipTransfers/images/naive_approach.png\" width=50%\u003e\n\nAlthough the naive approach can be optimised with indices to be quite performant, additional improvements can be made. This is quite apparent if we look at the following data points below:\n\nNaive Approach with buffers\n\n\u003cimg src=\"https://github.com/databrickslabs/mosaic/raw/main/notebooks/examples/python/Ship2ShipTransfers/images/buffer_approach.png\" \u003e\n\nAccording to our naive approach, where we buffer around our LAT/LONG points, the two vessels would not intersect. However, if we construct the actual path the vessels took, our algorithmic implementation would detect an overlap between the two paths, as shown below:\n\nPath Line Strings approach\n\n\u003cimg src=\"https://github.com/databrickslabs/mosaic/raw/main/notebooks/examples/python/Ship2ShipTransfers/images/linestring_approach.png\" \u003e\n\nThis model is expanded upon in the course of the attached notebooks. It shows how to ingest AIS data, how to process it at scale leveraging Mosaic in Databricks, and provides examples of how to extend the analysis to incorporate additional sources. \n\n*created by [@tiems90](https://github.com/tiems90)* \n\n| library                                | description             | license             | source                                              |\n|----------------------------------------|-------------------------|---------------------|-----------------------------------------------------|\n| mosaic                                 | library                 | Databricks License  | https://github.com/databrickslabs/mosaic            |\n\nTo run this accelerator, clone this repo into a Databricks workspace. Attach the RUNME notebook to any cluster running a DBR 11.0 or later runtime, and execute the notebook via Run-All. A multi-step-job describing the accelerator pipeline will be created, and the link will be provided. Execute the multi-step-job to see how the pipeline runs.\n\nThe job configuration is written in the RUNME notebook in json format. The cost associated with running the accelerator is the user's responsibility.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-industry-solutions%2Fship2ship-transfers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks-industry-solutions%2Fship2ship-transfers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-industry-solutions%2Fship2ship-transfers/lists"}