{"id":15060785,"url":"https://github.com/triglav-dataflow/triglav-agent-bigquery","last_synced_at":"2026-02-14T13:03:14.602Z","repository":{"id":59158068,"uuid":"83004843","full_name":"triglav-dataflow/triglav-agent-bigquery","owner":"triglav-dataflow","description":"BigQuery agent for Triglav, data-driven workflow tool","archived":false,"fork":false,"pushed_at":"2017-04-22T02:15:38.000Z","size":60,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-01-20T17:07:00.753Z","etag":null,"topics":["bigquery","ruby","triglav-agent"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/triglav-dataflow.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-24T05:28:43.000Z","updated_at":"2017-03-14T15:10:58.000Z","dependencies_parsed_at":"2022-09-13T20:10:59.136Z","dependency_job_id":null,"html_url":"https://github.com/triglav-dataflow/triglav-agent-bigquery","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/triglav-dataflow/triglav-agent-bigquery","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triglav-dataflow%2Ftriglav-agent-bigquery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triglav-dataflow%2Ftriglav-agent-bigquery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triglav-dataflow%2Ftriglav-agent-bigquery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triglav-dataflow%2Ftriglav-agent-bigquery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/triglav-dataflow","download_url":"https://codeload.github.com/triglav-dataflow/triglav-agent-bigquery/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triglav-dataflow%2Ftriglav-agent-bigquery/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29444061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T12:43:28.304Z","status":"ssl_error","status_checked_at":"2026-02-14T12:43:14.160Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","ruby","triglav-agent"],"created_at":"2024-09-24T23:04:26.324Z","updated_at":"2026-02-14T13:03:14.584Z","avatar_url":"https://github.com/triglav-dataflow.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Triglav::Agent::Bigquery\n\nTriglav Agent for BigQuery\n\n## Requirements\n\n* Ruby \u003e= 2.3.0\n\n## Prerequisites\n\n* BigQuery view is not supported\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'triglav-agent-bigquery'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install triglav-agent-bigquery\n\n## CLI\n\n```\nUsage: triglav-agent-bigquery [options]\n    -c, --config VALUE               Config file (default: config.yml)\n    -s, --status VALUE               Status stroage file (default: status.yml)\n    -t, --token VALUE                Triglav access token storage file (default: token.yml)\n        --dotenv                     Load environment variables from .env file (default: false)\n    -h, --help                       help\n        --log VALUE                  Log path (default: STDOUT)\n        --log-level VALUE            Log level (default: info)\n```\n\nRun as:\n\n```\nTRIGLAV_ENV=development bundle exec triglav-agent-bigquery --dotenv -c config.yml\n```\n\n## Configuration\n\nPrepare config.yml as [example/config.yml](./example/config.yml).\n\nYou can use erb template. You may load environment variables from .env file with `--dotenv` option as an [example/example.env](./example/example.env) file shows.\n\n### serverengine section\n\nYou can specify any [serverengine](https://github.com/fluent/serverengine) options at this section\n\n### triglav section\n\nSpecify triglav api url, and a credential to authenticate.\n\nThe access token obtained is stored into a token storage file (--token option).\n\n### bigquery section\n\nThis section is the special section for triglav-agent-bigquery.\n\n* **monitor_interval**: The interval to watch tables (number, default: 60)\n* **connection_info**: key-value pairs of bigquery connection info where keys are resource URI pattern in regular expression, and values are connection infomation\n  * **auth_method**: Authentication method. Must be one of `service_account`, `authorized_user` (for oauth2), `compute_engine`, and `application_default`. Default obtains from credentials.\n  * **credentials_file**: Credentials file path such as service account json.\n  * **credentials**: Instead of `credentials_file`, you may pass json contents as a string\n\n### Specification of Resource URI\n\nResource URI must be a form of:\n\n```\nhttps://bigquery.cloud.google.com/table/#{project}:#{dataset}.#{table}\n```\n\n`#{table}` also accepts strftime formatted suffix such as\n\n```\n#{table}_%Y%m%d\n```\n\nand strftime formatted partition decorator for a partitioned table such as\n\n```\n#{table}$%Y%m%d\n```\n\n## How it behaves\n\n1. Authenticate with triglav\n  * Store the access token into the token storage file\n  * Read the token from the token storage file next time\n  * Refresh the access token if it is expired\n2. Repeat followings in `monitor_interval` seconds:\n3. Obtain resource (table) lists of the specified prefix (keys of connection_info) from triglav.\n4. Connect to bigquery with an appropriate connection info for a resource uri, and find tables which are newer than last check.\n5. Store checking information into the status storage file for the next time check.\n\n## Development\n\n### Prepare\n\n```\n./prepare.sh\n```\n\nEdit `.env` or `config.yml` file directly.\n\n### Start\n\nStart up triglav api on localhost.\n\nRun triglav-anget-bigquery as:\n\n```\nTRIGLAV_ENV=development bundle exec triglav-agent-bigquery --dotenv --debug -c example/config.yml\n```\n\nThe debug mode with --debug option ignores the `last_modified_time` value in status file.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/triglav-dataflow/triglav-agent-bigquery. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.\n\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftriglav-dataflow%2Ftriglav-agent-bigquery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftriglav-dataflow%2Ftriglav-agent-bigquery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftriglav-dataflow%2Ftriglav-agent-bigquery/lists"}