{"id":42640828,"url":"https://github.com/spatialcurrent/railgun","last_synced_at":"2026-01-29T06:14:42.585Z","repository":{"id":57554907,"uuid":"145319369","full_name":"spatialcurrent/railgun","owner":"spatialcurrent","description":"A simple and fast data processing tool","archived":false,"fork":false,"pushed_at":"2019-10-08T02:22:00.000Z","size":451,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-06-20T11:05:58.087Z","etag":null,"topics":["big-data","data","geospatial","gis"],"latest_commit_sha":null,"homepage":"https://railgun.spatialcurrent.io/swagger.yml","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spatialcurrent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-19T16:26:00.000Z","updated_at":"2022-12-16T07:55:00.000Z","dependencies_parsed_at":"2022-09-26T18:51:28.228Z","dependency_job_id":null,"html_url":"https://github.com/spatialcurrent/railgun","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/spatialcurrent/railgun","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spatialcurrent%2Frailgun","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spatialcurrent%2Frailgun/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spatialcurrent%2Frailgun/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spatialcurrent%2Frailgun/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spatialcurrent","download_url":"https://codeload.github.com/spatialcurrent/railgun/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spatialcurrent%2Frailgun/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28865712,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T05:56:06.453Z","status":"ssl_error","status_checked_at":"2026-01-29T05:55:57.668Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","data","geospatial","gis"],"created_at":"2026-01-29T06:14:41.587Z","updated_at":"2026-01-29T06:14:42.566Z","avatar_url":"https://github.com/spatialcurrent.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CircleCI](https://circleci.com/gh/spatialcurrent/railgun/tree/master.svg?style=svg)](https://circleci.com/gh/spatialcurrent/railgun/tree/master) [![Go Report Card](https://goreportcard.com/badge/spatialcurrent/railgun)](https://goreportcard.com/report/spatialcurrent/railgun)  [![GoDoc](https://godoc.org/github.com/spatialcurrent/railgun?status.svg)](https://godoc.org/github.com/spatialcurrent/railgun) [![license](http://img.shields.io/badge/license-MIT-red.svg?style=flat)](https://github.com/spatialcurrent/railgun/blob/master/LICENSE)\n\n# Railgun\n\n# Description\n\n**Railgun** is a simple and fast data processing tool.  **Railgun** uses:\n- [go-reader](https://github.com/spatialcurrent/go-reader) for opening and reading from URIs,\n- [go-simple-serializer](https://github.com/spatialcurrent/go-simple-serializer) (GSS) for reading/writing objects to standard formats, and\n- [go-dfl](https://github.com/spatialcurrent/go-dfl) for filtering and transforming data.\n\n**Railgun** uses the **Dynamic Filter Language** through **go-dfl**.  See the `*_test` files in the [dfl](https://github.com/spatialcurrent/go-dfl/tree/master/dfl) source folder on GitHub for comprehensive examples of the syntax.\n\ngo-reader can read from `stdin`, `http/https`, the local filesystem, [AWS S3](https://aws.amazon.com/s3/), and [HDFS](https://hortonworks.com/apache/hdfs/).\n\ngo-simple-serializer (GSS) supports `bson`, `csv`, `tsv`, `hcl`, `hcl2`, `json`, `jsonl`, `properties`, `toml`, `yaml`.  `hcl` and `hcl2` implementation is fragile and very much in `alpha`.\n\nFor an interactive demo, see the [railgun notebook](https://beta.observablehq.com/@pjdufour/railgun) on [ObservableHQ](http://observablehq.com). It is very heavy, so only use WiFi.\n\n# Usage\n\n**CLI**\n\nYou can use the command line tool to process data.\n\n```\nUsage: railgun -input_format INPUT_FORMAT -o OUTPUT_FORMAT [-input_uri INPUT_URI] [-input_compression [bzip2|gzip|snappy]] [-h HEADER] [-c COMMENT] [-object_path PATH] [-dfl_exp DFL_EXPRESSION] [-dfl_file DFL_FILE] [-output_path OUTPUT_PATH] [-max MAX_COUNT]\nOptions:\n  -aws_access_key_id string\n    \tDefaults to value of environment variable AWS_ACCESS_KEY_ID\n  -aws_default_region string\n    \tDefaults to value of environment variable AWS_DEFAULT_REGION.\n  -aws_secret_access_key string\n    \tDefaults to value of environment variable AWS_SECRET_ACCESS_KEY.\n  -aws_session_token string\n    \tDefaults to value of environment variable AWS_SESSION_TOKEN.\n  -c string\n    \tThe input comment character, e.g., #.  Commented lines are not sent to output.\n  -dfl_exp string\n    \tProcess using dfl expression\n  -dfl_file string\n    \tProcess using dfl file.\n  -h string\n    \tThe input header if the stdin input has no header.\n  -hdfs_name_node string\n    \tDefaults to value of environment variable HDFS_DEFAULT_NAME_NODE.\n  -help\n    \tPrint help.\n  -input_compression string\n    \tThe input compression: none, bzip2, gzip, snappy (default \"none\")\n  -input_format string\n    \tThe input format: bson, csv, tsv, hcl, hcl2, json, jsonl, properties, toml, yaml\n  -input_reader_buffer_size int\n    \tThe input reader buffer size (default 4096)\n  -input_uri string\n    \tThe input uri (default \"stdin\")\n  -max int\n    \tThe maximum number of objects to output (default -1)\n  -output_format string\n    \tThe output format: bson, csv, tsv, hcl, hcl2, json, jsonl, properties, toml, yaml\n  -output_uri string\n    \tThe output uri (default \"stdout\")\n  -version\n    \tPrints version to stdout.\n```\n\n# Releases\n\n**Railgun** is currently in **alpha**.  See releases at https://github.com/spatialcurrent/railgun/releases.\n\n# Examples\n\n**Search for Cuisine**\n\n```\n~/go/src/github.com/spatialcurrent/go-osm/bin/osm_linux_amd64 -input_uri 'http://download.geofabrik.de/north-america/us/district-of-columbia-latest.osm.bz2' -ways_to_nodes -output_format geojsonl -filter_keys_keep amenity -output_uri stdout | railgun -input_format jsonl  -output_format json -dfl_file ~/go/src/github.com/spatialcurrent/railgun/examples/mexican.dfl -output_uri mexican.json\n```\n\n**Tsunami Feed**\n\n```\nconst pipeline = [\"filter(@features, '(@properties?.tsunami != null) and (@properties.tsunami == 1)')\", \"sort(@, '@properties?.mag', true)\", \"map(@, '@properties?.place ?: \\\"\\\"')\", \"limit(@, 10)\"];\n(await fetch(\"https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_month.geojson\")).json().then(earthquakes =\u003e {\n  result = railgun.process(earthquakes, {\"dfl\": pipeline, \"output_format\": \"yaml\"});\n  console.log(result);\n})\n```\n\n**Encrypt as Yaml / Decrypt as JSON**\n\n```\n# Encrypt secrets.yml and output to secrets.yml.enc\nread -s -p 'Password: ' password \u0026\u0026 echo \u0026\u0026 railgun_linux_amd64 -input_uri secrets.yml -output_uri secrets.yml.enc -output_passphrase $password\n...\n# Decrypt secrets.yml.enc and output to stdout\nread -s -p 'Password: ' password \u0026\u0026 echo \u0026\u0026 railgun_linux_amd64 -input_uri secrets.yml.enc -input_passphrase $password -output_format json\n```\n\n# Building\n\n**CLI**\n\nThe `build_cli.sh` script is used to build executables for Linux and Windows.\n\n**JavaScript**\n\nYou can compile GSS to pure JavaScript with the `scripts/build_javascript.sh` script.\n\n**Changing Destination**\n\nThe default destination for build artifacts is `railgun/bin`, but you can change the destination with a CLI argument.  For building on a Chromebook consider saving the artifacts in `/usr/local/go/bin`, e.g., `bash scripts/build_cli.sh /usr/local/go/bin`\n\n# Deploying\n\n```\nmkdir -p /usr/local/terraform\naws-vault exec default -- terraform init # to download aws provider\ncp -R .terraform/plugins/linux_amd64/terraform-provider-aws_v1.43.2_x4 /usr/local/terraform\naws-vault exec default -- terraform init -plugin-dir=/usr/local/terraform\naws-vault exec default -- terraform plan\n```\n\n# Contributing\n\n[Spatial Current, Inc.](https://spatialcurrent.io) is currently accepting pull requests for this repository.  We'd love to have your contributions!  Please see [Contributing.md](https://github.com/spatialcurrent/railgun/blob/master/CONTRIBUTING.md) for how to get started.\n\n# License\n\nThis work is distributed under the **MIT License**.  See **LICENSE** file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspatialcurrent%2Frailgun","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspatialcurrent%2Frailgun","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspatialcurrent%2Frailgun/lists"}