{"id":18977310,"url":"https://github.com/mikhailms/logstash-input-google_bigquery","last_synced_at":"2026-04-09T17:30:18.645Z","repository":{"id":173919448,"uuid":"651486401","full_name":"MikhailMS/logstash-input-google_bigquery","owner":"MikhailMS","description":"Logstash input plugin to fetch data from Google Cloud BigQuery","archived":false,"fork":false,"pushed_at":"2024-03-06T15:35:16.000Z","size":12,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-01T09:42:08.503Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MikhailMS.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-09T10:44:11.000Z","updated_at":"2023-07-03T07:50:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"c1600a77-8d37-4559-a861-eaed834acae3","html_url":"https://github.com/MikhailMS/logstash-input-google_bigquery","commit_stats":null,"previous_names":["mikhailms/logstash-input-google_bigquery"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MikhailMS%2Flogstash-input-google_bigquery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MikhailMS%2Flogstash-input-google_bigquery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MikhailMS%2Flogstash-input-google_bigquery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MikhailMS%2Flogstash-input-google_bigquery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MikhailMS","download_url":"https://codeload.github.com/MikhailMS/logstash-input-google_bigquery/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239978046,"owners_count":19728271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T15:28:32.059Z","updated_at":"2026-04-09T17:30:18.615Z","avatar_url":"https://github.com/MikhailMS.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Logstash Google Cloud BigQuery input plugin\n\n\nThis is an input plugin for [Logstash](https://github.com/elastic/logstash) to enable consumption of data from Google Cloud BigQuery\n\n\n## Building\n0. To build the plugin, you need to have:\n    1. JRuby    installed\n    2. Java     installed\n    3. Logstash binaries\n1. It is recommended to have `rbenv` installed, so you can install any required JRuby (as logstash gets upgraded, you may need newer JRuby each time you build plugin for that version of logstash)\n2. It is recommended to have `jenv`  installed, so you can install any required Java  (as logstash gets upgraded, you may need newer Java  each time you build plugin for that version of logstash)\n3. You need to set 2 env varialbes\n```bash\nexport LOGSTASH_PATH=/path/to/logstash/binaries\nexport LOGSTASH_SOURCE=1\n```\n4. Install dependencies*\n```sh\nbundle install\nbundle exec rake vendor\n```\n5. Build plugin\n```sh\ngem build logstash-input-google_bigquery.gemspec\n```\n\n\n## Test\n- Run tests\n\n```sh\nbundle exec rspec\n```\n\n\n## Use plugin in Logstash\n1. Build plugin locally\n2. Install plugin to the local version of Logstash that was used to build the plugin\n3. Once plugin is installed to local Logstash, create offline-package with the plugin\n```sh\n./bin/logstash-plugin prepare-offline-pack logstash-input-google_bigquery\n```\n4. Deploy artifact generated in Step 3 onto server where Logstash runs\n5. To install offline-package run following command from Logstash directory\n```sh\n./bin/logstash-plugin install --local file:///path/to/custom/plugin.zip\n```\n\n\n## Notes:\n1. If plugin is used on Logstash that is behind proxies, ensure to have ENV proxy variables set before starting Logstash:\n```\nexport HTTPS_PROXY=\u003cproxy_url\u003e:\u003cproxy_port\u003e\n```\n2. Interesting, but `google-cloud-bigquery` gem is not being packed together with this plugin when it gets build (probably missing something), so `google-cloud-bigquery` gem (and its dependencies) must be manually copied into `LOGSTASH_PATH/vendor/cache`, otherwise you won't be able to install it (only locally, when you prepare-offline-pack it gets sorted nicely)\n3. Known limitations:\n    1. Do not handle fetching large amounts of data (pagination)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikhailms%2Flogstash-input-google_bigquery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmikhailms%2Flogstash-input-google_bigquery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikhailms%2Flogstash-input-google_bigquery/lists"}