https://github.com/ivdatahub/pypi-package-stats

Project for ingest pypi packages data from BigQuery and send to DataDog for analysis and insights with dashboards, monitors and more
https://github.com/ivdatahub/pypi-package-stats

bigquery cloud data-engineering data-warehouse gcp software-engineering

Last synced: about 1 month ago
JSON representation

Project for ingest pypi packages data from BigQuery and send to DataDog for analysis and insights with dashboards, monitors and more

• Host: GitHub
• URL: https://github.com/ivdatahub/pypi-package-stats
• Owner: ivdatahub
• License: mit
• Created: 2024-07-25T18:06:45.000Z (5 months ago)
• Default Branch: main
• Last Pushed: 2024-10-24T14:54:45.000Z (2 months ago)
• Last Synced: 2024-10-25T19:22:38.368Z (2 months ago)
• Topics: bigquery, cloud, data-engineering, data-warehouse, gcp, software-engineering
• Language: Python
• Homepage:
• Size: 206 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • Funding: .github/funding.yml
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
## Google Cloud BigQuery: BigQuery SQL Repository

![Project Status](https://img.shields.io/badge/status-done-green?style=for-the-badge&logo=github)

![License](https://img.shields.io/badge/license-MIT-blue?style=for-the-badge&logo=mit)

[![BigQuery Executor](https://img.shields.io/github/actions/workflow/status/IvanildoBarauna/bigquery-execute/pypi-analysis.yaml?&style=for-the-badge&logo=google&cacheSeconds=60&label=BigQuery-Executor)](https://github.com/IvanildoBarauna/bigquery-execute/actions/workflows/pypi-analysis.yaml)

## Project Summary

This project contains SQL scripts for creating and managing BigQuery tables, views, and other objects. It is intended to be used as a repository for storing and sharing SQL code across multiple projects.

## Development Stack

[![My Skills](https://skillicons.dev/icons?i=pycharm,github,gcp&perline=7)](https://skillicons.dev)

## Cloud Stack (GCP)



- Cloud IAM: Identity and Access Management (IAM) service provided by GCP for managing user access to cloud resources.

- BigQuery: Fully managed, serverless data warehouse provided by GCP for storing and analyzing large datasets.

## Continuous Integration and Continuous Deployment (CI/CD, DevOps)

![My Skills](https://skillicons.dev/icons?i=githubactions)

## Contributing

See the following docs:

- [Contributing Guide](https://github.com/IvanildoBarauna/GCP-streaming-pipeline/blob/main/CONTRIBUTING.md)

- [Code Of Conduct](https://github.com/IvanildoBarauna/GCP-streaming-pipeline/blob/main/CODE_OF_CONDUCT.md)

## Project Highlights:

- Modular Architecture: Separation of SQL files and execution scripts for easier maintenance and updates.

- Automated SQL Execution: GitHub Actions workflows to automatically execute SQL scripts on BigQuery based on defined triggers.

- Configuration Management: Use of environment variables and configuration files for managing execution parameters and credentials.

- CI/CD Pipelines: Automated build, test, and deployment processes using GitHub Actions to keep SQL scripts up-to-date.

- Comprehensive Documentation: Detailed documentation including installation instructions, usage examples, and troubleshooting guides.