Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/MatzFan/gis_scraper
Utility to recursively scrape ArcGIS MapServer data using the ArcGIS REST API.
https://github.com/MatzFan/gis_scraper
Last synced: 3 months ago
JSON representation
Utility to recursively scrape ArcGIS MapServer data using the ArcGIS REST API.
- Host: GitHub
- URL: https://github.com/MatzFan/gis_scraper
- Owner: MatzFan
- License: mit
- Created: 2015-09-07T19:29:15.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2016-06-10T20:49:32.000Z (over 8 years ago)
- Last Synced: 2024-04-16T18:05:32.094Z (7 months ago)
- Language: Ruby
- Homepage:
- Size: 271 KB
- Stars: 9
- Watchers: 4
- Forks: 5
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# gis_scraper Ruby Gem
[](http://travis-ci.org/MatzFan/gis_scraper)
[](http://badge.fury.io/rb/gis_scraper)
Utility to recursively scrape ArcGIS MapServer data using the ArcGIS REST API.
ArcGIS MapServer REST queries are limited to 1,000 objects in some cases. This tool makes repeated calls until all data for a given layer (and all sub-layers) is extracted. Output can be GeoJSON file format or data may be written directly to Postgres database tables in PostGIS format. GIS clients - e.g. QGIS - can be configured to use vector layer data from PostGIS sources.
## Requirements
Ruby 2.0 or above - see Travis badge for tested Ruby versions.
A Postgres database with the PostGIS extension enabled for database export.
For data import to a database [GDAL](http://gdal.org) must be installed and specifically the [ogr2ogr](http://www.gdal.org/ogr2ogr.html) executable must be available in your path.
## Known Limitations
*NIX systems only - Linux/Mac OS X. ArcGIS MapServer data is readable directly by ArcGIS Windows clients 😉
The following esri geometry types are so far supported:
- esriGeometryPoint, esriGeometryMultipoint, esriGeometryLine, esriGeometryPolyline, esriGeometryPolygon
Annotation layers are ignored, as are layers with no esri geometryType.
Currently the JSON data for a whole layer is held in memory before being output. For large layers - e.g. >100,000 objects - this can be multiple GB of memory. Is this causes a problem for you please add a comment to [issue #4](https://github.com/MatzFan/gis_scraper/issues/4).
## Installation
Add this line to your application's Gemfile:
```ruby
gem 'gis_scraper'
```
And then execute:
$ bundle
Or install it yourself as:
$ gem install gis_scraper
## Configuration
Configuration options may be set via a hash or specified in a Yaml file. The following options are available:
- ```:threads``` Scraping is multi-threaded. The number of threads to use may be set with this option (default: 8)
- ```:output_path``` For JSON output, the path used to write files to (default: '~/Desktop')
The following options are used to connect to a database:
- ```:host``` (default: 'localhost')
- ```:port``` (default: 5432)
- ```:dbname``` (default: 'postgres')
- ```:user``` (default: 'postgres')
- ```:password``` (default: nil)
These additional options are available when using output to a database and are applied to the ```ogr2ogr``` command:
- ```:srs``` Used to overide the source spacial reference system. Currently only EPSG string format is valid - e.g. 'EPSG:3109' (default: no overide)
**To set via a hash**
```Ruby
GisScraper.configure(:threads => 16)
```
**Using a Yaml configuration file**
```Ruby
GisScraper.configure_with 'path-to-Yaml-file'
```
```Ruby
GisScraper.config # returns the hash of configuration values
```
## Usage
A LayerWriter object must be instantiated with one required arg - a Service/Layer URL (ending in an integer representing the layer number). Example:
```
writer = LayerWriter.new('http://gps.digimap.gg/arcgis/rest/services/StatesOfJersey/JerseyMappingOL/MapServer/0')
```
An optional second argument for the output path for JSON files may be specified. If so this overides the configuration option. Example:
```
writer = LayerWriter.new('http://gps.digimap.gg/arcgis/rest/services/StatesOfJersey/JerseyMappingOL/MapServer/0', '~/Desktop')
```
**JSON output**
```
writer.output_json
```
If the layer is type 'Feature Layer', a single file of JSON data will be saved (named the same as the layer). If the layer is type 'Group Layer', the sub-group structure is traversed recursively thus: Directories for each sub-group layer are created and JSON data files for each constituent feature layer written to them.
**Output to a database**
Valid database config options must be set. The following command will convert JSON files, create tables for each layer (& sub-layers, if any) and import the data. Table names are lowercased, prefixed '_' and have spaces replaced with undescores. If a table with the same name exists the name is appended with '_'.
```
writer.output_to_db
```
## Specification and Tests
For the full specification clone this repo and run:
`rake spec`
## Contributing
Bug reports, pull requests (and feature requests) are welcome on GitHub at https://github.com/MatzFan/gis_scraper.
## License
The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses)