{"id":26678342,"url":"https://github.com/telexintegrations/bigquery-monitor","last_synced_at":"2025-08-19T08:39:25.331Z","repository":{"id":278833366,"uuid":"936775872","full_name":"telexintegrations/Bigquery-monitor","owner":"telexintegrations","description":"A Telex integration that monitors bigquery by reporting daily resource utilization and failed job runs.   ","archived":false,"fork":false,"pushed_at":"2025-06-16T22:19:11.000Z","size":59,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-26T22:17:58.542Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/telexintegrations.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-21T17:03:21.000Z","updated_at":"2025-02-24T21:24:56.000Z","dependencies_parsed_at":"2025-02-22T00:28:29.878Z","dependency_job_id":"b5ab651c-7c32-43b8-bccb-30b269168107","html_url":"https://github.com/telexintegrations/Bigquery-monitor","commit_stats":null,"previous_names":["telexintegrations/bigquery-monitor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/telexintegrations/Bigquery-monitor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telexintegrations%2FBigquery-monitor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telexintegrations%2FBigquery-monitor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telexintegrations%2FBigquery-monitor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telexintegrations%2FBigquery-monitor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/telexintegrations","download_url":"https://codeload.github.com/telexintegrations/Bigquery-monitor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/telexintegrations%2FBigquery-monitor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271123474,"owners_count":24703235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-19T02:00:09.176Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-26T05:15:30.256Z","updated_at":"2025-08-19T08:39:25.305Z","avatar_url":"https://github.com/telexintegrations.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BigQuery Monitor (Telex Integration)\n\n## Overview\n\nThis project is a Telex \"interval integration\" for Google's BigQuery (a distributed data querying and processing engine). It allows team to monitor their Big Query performance and resource utilization on a day-to-day basis. Everyday at 11:59pm (this can be changed to suit any other use case), it fetches data on job runs (like a database query) that failed as well as the details of the error. It also collects information on the amount of slot time (resource usage) consumed for the day and calculates the day-on-day percentage change. This helps team detect usage spikes (a potential indicator of a problematic job) and increase provisioning where necessary. \n\n## Features\n\n- 🔍 Daily report on job run errors\n- 📝 Daily updates on BigQuery resource utilization\n- 🔒 CORS middleware enabled\n\n## Project Structures\n\n```\nBigquery-monitor/\n├── bq_functions/\n│   ├── bigquery_funcs.py/ # Wrapper functions for BigQuery DB queries\n│   │ \n├── integration_json/\n│   └── telex_json.py           # JSON file configuration that Telex needs to configure the app\n├── tests/\n│   ├── __init__.py\n│   └── test_app.py       # API endpoint tests\n├── models/\n│   │\n│   └── models.py           # Pydantic data models\n├── main.py                 # Application entry point\n├── requirements.txt        # Project dependencies\n└── README.md\n```\n\n## Technologies Used\n\n- Python 3.12\n- FastAPI\n- Pydantic\n- Google Cloud SDK\n- uvicorn\n\n## Installation\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/telexintegrations/Bigquery-monitor\ncd Bigquery-monitor\n```\n\n2. Create a virtual environment:\n\n```bash\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\n```\n\n3. Install dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\n## Running the Application (Testing)\n\n### Authentication with Google Cloud Services\n\nThis integration leverages the Google Cloud API to collect data from Google Cloud Services (GCS). Therefore, it needs to authenticate with GCS, using a service account key, via the `gcloud API`. This key can be generated via the Google Cloud Console within the `Service Accounts` interface. Note, however, that the service account must have the `BigQuery/Admin` role permissions enabled. \n\nPaste this key within the `telex_json.py` file as the value for the `default` key under the `Service Account Key` label. In addition, the integration requires a GCS `project ID` and `region`. Default values can be set within the `telex_json.py` file.\n\n1. Start the server:\n\n```bash\nuvicorn main:app \n```\n\n2. Access the API documentation:\n\n- Swagger UI: http://localhost:8000/docs\n- ReDoc: http://localhost:8000/redoc\n\n## API Endpoints\n\n### Books\n\n- `GET /integration.json` - Returns the JSON configuration file that the integration provides to Telex\n- `POST /tick` - Triggers the app to fetch data from BigQuery and send to the designated Telex channel. \n\n## Running Tests\n\n```bash\npytest\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/AmazingFeature`)\n3. Commit changes (`git commit -m 'Add AmazingFeature'`)\n4. Push to branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\nFor support, please open an issue in the GitHub repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftelexintegrations%2Fbigquery-monitor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftelexintegrations%2Fbigquery-monitor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftelexintegrations%2Fbigquery-monitor/lists"}