{"id":19795599,"url":"https://github.com/dkogan/ntsb-wrecks","last_synced_at":"2026-05-10T22:44:41.146Z","repository":{"id":136714566,"uuid":"164380376","full_name":"dkogan/ntsb-wrecks","owner":"dkogan","description":"Plane-crash locations from NTSB reports","archived":false,"fork":false,"pushed_at":"2019-02-10T09:09:41.000Z","size":9548,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-02-11T02:25:37.190Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dkogan.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-07T04:59:50.000Z","updated_at":"2019-06-11T06:08:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"277c0e6d-2f0e-4d9a-bc2f-ba54c4469c8b","html_url":"https://github.com/dkogan/ntsb-wrecks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dkogan/ntsb-wrecks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkogan%2Fntsb-wrecks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkogan%2Fntsb-wrecks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkogan%2Fntsb-wrecks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkogan%2Fntsb-wrecks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dkogan","download_url":"https://codeload.github.com/dkogan/ntsb-wrecks/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkogan%2Fntsb-wrecks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32874700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"ssl_error","status_checked_at":"2026-05-10T13:40:02.145Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T07:16:48.338Z","updated_at":"2026-05-10T22:44:41.117Z","avatar_url":"https://github.com/dkogan.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"* Synopsis\n\nCheck this out!\n\n[[file:wrecks.png]]\n\nInteractive map at\n\nhttp://ntsb.secretsauce.net/map/\n\n* Overview\n\nThis is a set of scripts to massage NTSB incident reports into a visualization\nof crash sites. After an aviation incident occurs, the NTSB investigates each\nevent, and publishes a report. These often contain information about the crash\nsite location. Sometimes this exists explicitly: as a latitude, longitude pair.\nAt other times, the report lists the bearing and range to an \"observing\nfacility\". This set of tools extracts both of these into a visualize-able form.\n\n** Caveats\n\nThe data isn't particularly accurate: most of the locations I know of are in the\nright ballpark (within a few miles), but some are completely off. It is,\nhowever, useful as a jumping-off point for further research.\n\n*If you hike to the exact coordinates given, you're unlikely to find anything*\n\nThe data I'm visualizing here comes from the post-1982 incident database. A\ndatabase of older incidents exists also, but it doesn't appear to contain any\nlocation information. Finally, while civilian incidents are well-covered by this\ndataset, I don't know if military ones are covered as well.\n\n* Details\n\nWe download some data, massage it, and produce a single data-store that contains\neverything we care about. Below, I describe how I made this file, but a\npre-cooked copy is available in this repo: [[file:joint.vnl]] (plain text) or\n[[file:joint.sqlite]] (sqlite database). This can be used to visualize specific\nareas, as described [[Visualize-able data output][below]].\n\nThis all works /today/ (January 2019) on my up-to-date Debian/sid box. The URLs\nand tools and such can change at any time, so interpret all this as a set of\nloose guidelines. Make adjustments as required.\n\n** NTSB incident database\n\nThe main website that describes the reports is this:\nhttps://www.ntsb.gov/_layouts/ntsb.aviation/index.aspx\n\nIt points to a repository of databases:\nhttps://app.ntsb.gov/avdata/\n\nWe grab the database of events we care about. Today the post-1982 data can be\nobtained and extracted thusly:\n\n#+BEGIN_EXAMPLE\n$ wget https://app.ntsb.gov/avdata/Access/avall.zip\n\n$ unzip avall.zip\n#+END_EXAMPLE\n\nWe now have a Microsoft Access database file: =avall.mdb= with all the data. The\nthings I care about live in the =events= and =aircraft= tables. I\n\n- extract those into an sqlite database\n- query the columns I want as csv\n- make the latitude and longitude fields numeric (i.e.\"360148N 1190347W\" becomes\n  \"36.03 -119.063055555556\")\n- convert to a form I can work with: [[http://github.com/dkogan/vnlog][vnlog]]\n\n#+BEGIN_EXAMPLE\n$ ( mdb-schema avall.mdb sqlite -T aircraft;\n    echo \"BEGIN;\";\n    mdb-export -I sqlite avall.mdb aircraft;\n    echo \"COMMIT;\";\n  ) | sqlite3 data.sqlite\n\n$ ( mdb-schema avall.mdb sqlite -T events;\n    echo \"BEGIN;\";\n    mdb-export -I sqlite avall.mdb events;\n    echo \"COMMIT;\";\n  ) | sqlite3 data.sqlite\n\n$ sqlite3 -header -csv data.sqlite \\\n    'select events.ev_id,ev_year,acft_make,acft_model,acft_series,acft_serial_no,latitude,longitude,wx_obs_fac_id,wx_obs_dist,wx_obs_dir from events,aircraft where events.ev_id==aircraft.ev_id;' |\n    ./csv-to-vnl.pl |\n    vnl-filter --perl \\\n      --sub 'conv{return undef unless defined $_[0]; return $1 + ($2 + $3/60.)/60. if $_[0] =~ /^([0-9]+)([0-9][0-9])([0-9][0-9])[NE]$/i; return -($1 + ($2 + $3/60.)/60.) if $_[0] =~ /^([0-9]+)([0-9][0-9])([0-9][0-9])[SW]$/i; return undef;}' \\\n       -p '!(latitude|longitude),latitude=conv(latitude),longitude=conv(longitude)' \\\n    \u003e data.vnl\n#+END_EXAMPLE\n\nYes, =vnlog= isn't a real database, so lookups will be much slower than if I\ncontinued with sqlite. But many things become simple, and I'm going to be doing\nvery few lookups, so this is ok.\n\n** Airport codes\n\nAlright. If we're getting the location in reference to some observing airport,\nwe need to know where each airport is. There're a number of lists on the\ninternet of airport codes. For this project I'd like to know the location of a\nbunch of tiny, obscure airports, so most lists were insufficient. This list [[https://datahub.io/core/airport-codes][here]]\nwas sufficiently complete. The data is licensed under the Public Domain\nDedication and License (PDDL). I download the dataset, and convert it to a nice\nvnlog containing just the data I need\n\n#+BEGIN_EXAMPLE\n$ wget https://datahub.io/core/airport-codes/r/airport-codes.csv\n\n$ ./parse-airport-data.pl \u003e airports.vnl\n#+END_EXAMPLE\n\n** Putting it all together\n\nAnd then I add these location fields to the main dataset\n\n#+BEGIN_EXAMPLE\n$ vnl-join -a1 --vnl-sort - -j wx_obs_fac_id                                               \\\n    data.vnl                                                                               \\\n    \u003c(\u003c airports.vnl vnl-filter -p wx_obs_fac_id=code,lat_observing=lat,lon_observing=lon) \\\n  \u003e joint.vnl\n#+END_EXAMPLE\n\nI later decided to make a website that can select subsets of the data, so I\nreally did need fast lookups. So I can convert this vnlog to an sqlite db:\n\n#+BEGIN_EXAMPLE\n./vnl-to-sql.pl --real  latitude,longitude,lat_observing,lon_observing \\\n                --index latitude,longitude,lat_observing,lon_observing \\\n    \u003c joint.vnl | sqlite3 joint.sqlite\n#+END_EXAMPLE\n\n** Visualize-able data output\n\nNow we have all the information, and we run another script to generate a GeoJSON\nfile that mapping tools can ingest. I cut it down to a small region because web\nbrowsers are glacially slow. To see all the incidents in the San Gabriel\nMountains:\n\n#+BEGIN_EXAMPLE\n$ ./make-map.pl 34.08 -118.52 34.54 -117.38 joint.sqlite \u003e wrecks.json\n#+END_EXAMPLE\n\nThis file is visualized in that interactive map linked above.\n\n* Acknowledgement\n\nHuge thanks to Erik Price and Chris McKenzie for help in getting the interactive\nmap working. I'm not a web dev.\n\n* License\n\nAll code Copyright 2019 Dima Kogan, released under the terms of the Lesser GNU\nPublic License (any version)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkogan%2Fntsb-wrecks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdkogan%2Fntsb-wrecks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkogan%2Fntsb-wrecks/lists"}