{"id":19948875,"url":"https://github.com/brannondorsey/spectrum-wrangler-docker","last_synced_at":"2025-05-03T18:32:06.693Z","repository":{"id":56858760,"uuid":"144502825","full_name":"brannondorsey/spectrum-wrangler-docker","owner":"brannondorsey","description":"A Dockerized version of Spectrum Wrangler that downloads and geo indexes public FCC license data ","archived":false,"fork":false,"pushed_at":"2018-08-22T14:28:27.000Z","size":14,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-04-14T18:50:18.116Z","etag":null,"topics":["amateur-radio","database","docker","docker-compose","fcc","fcc-data","fcc-license","postgis","postgresql","public-data","radio"],"latest_commit_sha":null,"homepage":null,"language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brannondorsey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-12T21:42:30.000Z","updated_at":"2023-08-21T14:01:16.000Z","dependencies_parsed_at":"2022-09-02T17:09:02.375Z","dependency_job_id":null,"html_url":"https://github.com/brannondorsey/spectrum-wrangler-docker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brannondorsey%2Fspectrum-wrangler-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brannondorsey%2Fspectrum-wrangler-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brannondorsey%2Fspectrum-wrangler-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brannondorsey%2Fspectrum-wrangler-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brannondorsey","download_url":"https://codeload.github.com/brannondorsey/spectrum-wrangler-docker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224370116,"owners_count":17299968,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amateur-radio","database","docker","docker-compose","fcc","fcc-data","fcc-license","postgis","postgresql","public-data","radio"],"created_at":"2024-11-13T00:43:07.027Z","updated_at":"2024-11-13T00:43:07.681Z","avatar_url":"https://github.com/brannondorsey.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spectrum Wrangler Docker\n\nA Dockerized version of Marc DaCosta's awesome [Spectrum Wrangler](https://github.com/marcdacosta/spectrum-wrangler) repository, a utility script to import the FCC license database into a PostgreSQL database with GIS and geo indexing support.\n\nThis repo creates docker images, one for the PostgreSQL + PostGIS database and another other for the original python import script for Spectrum Wrangler. This allows for a contained environment, a custom location for the 40GB+ Postgres database, and easy automated install on remote servers.\n\n## Installing\n\nBefore you install this software, first make sure [docker](https://www.docker.com/get-started) and [docker-compose](https://docs.docker.com/compose/install/) are installed on the host machine.\n\n```bash\n# clone the repo\ngit clone https://github.com/brannondorsey/spectrum-wrangler-docker\ncd spectrum-wrangler-docker\n\n# download the dockerized brannondorsey/spectrum-wrangler fork\ngit submodule init\ngit submodule update\n\n# build and launch the postgis/Dockerfile image. This creates a postgis container\n# with it's database stored in postgis/data_directory. Note: you can't be\n# running a PostgreSQL server on the default port of the host machine as ports\n# will conflict.\ndocker-compose up postgis\n\n# open a new terminal tab, and manually run the spectrum-wrangler docker service\n# to better monitor it's logs. This service runs spectrum-wrangler/load.py\n# which downloads a ~11GB CSV from the FCC's website, dumps it into the postgres\n# database and creates postgis indexes on lat/long for quick spatial queries.\ndocker-compose up spectrum-wrangler\n\n# optionally launch pgpadmin4 (PostgreSQL Admin tool) on http://localhost:5050.\n# This can be useful if you want to make queries against the database. Simply\n# create a new database connection to host: \"postgis\" using user and pass \"fcc2\"\ndocker-compose up pgadmin4\n```\n\n## Waiting...\n\nProvided you don't see any errors in your console, the download should be working. This can take ~1-2hrs+ depending on your network connection and your computer's resources. Provided you don't see an error output from the `docker-compose up spectrum-wrangler` process, you should be good to go.\n\n\n## Querying\n\nThe `postgis` docker service exposes the postgres service on port of the host machine. You can interact with it using whatever PostgreSQL admin tool you like, or using the `psql` shell in the docker container itself:\n\n```bash\ndocker-compose exec postgis psql -U fcc2\n...\npsql (10.5 (Debian 10.5-1.pgdg90+1))\nType \"help\" for help.\n\nfcc2=#\n```\n\nThe `/tmp/fcclicense` folder in the `postgis` container is a shared volume with your host machine, so anything you save in there (like .csv files exported from postgres) will appear in the `spectrum-wrangler/fcclicense/` folder.\n\nHere are some example queries now that you have everything setup:\n\n```sql\n-- test that the PostGIS extension has been installed and configured properly\nSELECT round(dms2dd('43° 0''50.60\"S'),9) as latitude, round(dms2dd('147°12''18.20\"E'),9) as longitude;\n\n-- select licenses within 1km of Trump Tower, NYC\nSELECT * FROM fcclicenses WHERE ST_DWithin(\n    geom::geography, \n    ST_GeogFromText('POINT(-74.0076834 40.7253319)'), \n    1000, \n    false) \n) \n```\n\nSee `spectrum-wrangler/example-queries.sql` for several additional query examples.\n\n## Going Further\n\nBe default, the `fcclicense` table has a geo index on lat \u0026 long. I've found it useful to add a few additional indexes to dramatically speed up query times. You can add several additional indexes to the database like so:\n\n```bash\n# copy additional-indexes.sql to spectrum-wrangler/fcclicense/ so that it will\n# appear in /tmp/fcclicense in the container's filesystem.\n# Note: this may require sudo depending on how docker has created the volume\ncp spectrum-wrangler/additional-indexes.sql spectrum-wrangler/fcclicense/\n\n# add some additional indexes, this may take a while...\ndocker-compose exec postgis psql -U fcc2 -f /tmp/fcclicense/additional-indexes.sql fcc2 \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrannondorsey%2Fspectrum-wrangler-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrannondorsey%2Fspectrum-wrangler-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrannondorsey%2Fspectrum-wrangler-docker/lists"}