{"id":26020884,"url":"https://github.com/thaddeusjiang/sensor_logs","last_synced_at":"2026-04-11T17:44:27.378Z","repository":{"id":280730657,"uuid":"942442844","full_name":"ThaddeusJiang/sensor_logs","owner":"ThaddeusJiang","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-06T08:33:28.000Z","size":115,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-01T07:43:36.637Z","etag":null,"topics":["big-query","bun","cloud-run","docker","gcp","google-cloud-platform","google-pubsub","terraform"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ThaddeusJiang.png","metadata":{"files":{"readme":"README-ja.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-04T05:42:34.000Z","updated_at":"2025-03-06T08:37:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"ad5d638e-267f-4526-b3af-96f3dac7e0f1","html_url":"https://github.com/ThaddeusJiang/sensor_logs","commit_stats":null,"previous_names":["thaddeusjiang/sensor_logs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ThaddeusJiang/sensor_logs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThaddeusJiang%2Fsensor_logs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThaddeusJiang%2Fsensor_logs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThaddeusJiang%2Fsensor_logs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThaddeusJiang%2Fsensor_logs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ThaddeusJiang","download_url":"https://codeload.github.com/ThaddeusJiang/sensor_logs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThaddeusJiang%2Fsensor_logs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274152761,"owners_count":25231293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-08T02:00:09.813Z","response_time":121,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-query","bun","cloud-run","docker","gcp","google-cloud-platform","google-pubsub","terraform"],"created_at":"2025-03-06T08:21:51.949Z","updated_at":"2026-04-11T17:44:27.333Z","avatar_url":"https://github.com/ThaddeusJiang.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IoT センサーデータ処理システム\n\nGoogle Cloud Platform 上に構築された30万個のIoTセンサーデータを処理・監視するシステム。\n\n## ドキュメント\n- English: [README.md](README.md)\n- 中文: [README-zh.md](README-zh.md)\n\n## 機能\n\n- 30万個のIoTセンサーからのデータをほぼリアルタイムで処理\n- センサーの監視とオフライン通知\n- 設定可能なバッチデータ処理\n- デッドレターキューによるエラー処理\n- Terraformによる自動化されたインフラ管理\n- GitHub Actionsによる CI/CD\n\n## アーキテクチャ\n\n```mermaid\ngraph LR\n    subgraph \"IoT デバイス\"\n        subgraph \"デバイス 1\"\n            S1[温度センサー]\n            S2[湿度センサー]\n            S3[電圧センサー]\n        end\n        D2[デバイス 2]\n        D3[デバイス 3]\n        D4[デバイス N]\n    end\n\n    subgraph \"Google Cloud Platform\"\n        PS[Cloud Pub/Sub]\n        DLQ[デッドレターキュー]\n        W[BigQuery Worker]\n        BQ[(BigQuery)]\n    end\n\n    S1 \u0026 S2 \u0026 S3 --\u003e|センサーデータ| PS\n    D2 \u0026 D3 \u0026 D4 --\u003e|センサーデータ| PS\n    PS --\u003e|メッセージ| W\n    W --\u003e|失敗したメッセージ| DLQ\n    W --\u003e|バッチ挿入| BQ\n```\n\n### アーキテクチャ概要\n\nプロジェクトで使用する GCP リソース：\n\n1. **BigQuery データセットとテーブル**\n   - データセット: `sensor_data`\n   - テーブル: `sensor_logs` (日付でパーティション化、device_id でクラスタリング)\n   - センサーデータの保存\n\n2. **Cloud Pub/Sub**\n   - トピック: `sensor-logs-topic`\n   - サブスクリプション: `sensor-logs-sub-01`\n   - デッドレターキュー: `sensor-logs-dlq`\n   - リアルタイムセンサーデータの受信\n\n3. **サービスアカウント**\n   - クライアントアプリケーション用\n   - Pub/Sub と BigQuery に必要な IAM 権限を含む\n\n## コンポーネント\n\n1. **インフラストラクチャ** (terraform/)\n   - Terraform で管理される GCP リソース\n   - BigQuery、Pub/Sub、IAM の設定\n\n2. **IoT クライアント** (apps/iot-client/)\n   - 複数の IoT デバイスをシミュレート\n   - デバイス数と送信頻度を設定可能\n\n3. **BigQuery ワーカー** (apps/bigquery-worker/)\n   - Pub/Sub からのメッセージを処理\n   - BigQuery へのバッチ挿入\n   - デッドレターキューでエラー処理\n\n## 前提条件\n\n- Terraform \u003e= 1.0\n- Google Cloud SDK\n- Bun \u003e= 1.2.2\n- TypeScript \u003e= 5.0.0\n\n## クイックスタート\n\n1. 必要なツールをインストール：\n   - [Terraform](https://developer.hashicorp.com/terraform/downloads)\n   - [Google Cloud SDK](https://cloud.google.com/sdk/docs/install)\n\n2. プロジェクトをクローンして移動：\n   ```bash\n   git clone https://github.com/ThaddeusJiang/sensor_logs.git\n   cd sensor_logs\n   ```\n\n3. GCP 認証を設定：\n   ```bash\n   gcloud auth application-default login\n   ```\n\n4. 初期化とデプロイ：\n   ```bash\n   cd terraform\n   terraform init\n   terraform plan\n   terraform apply\n   ```\n\n5. センサーの初期化：\n   ```bash\n   bun run apps/bigquery-worker/src/scripts/init-sensors.ts\n   ```\n\n6. IoT クライアントを実行、詳細は [apps/iot-client](apps/iot-client) を参照\n7. BigQuery ワーカーを実行、詳細は [apps/bigquery-worker](apps/bigquery-worker) を参照\n\n## データモデル\n\n```sql\nCREATE TABLE `sensor_data.sensor_logs` (\n    `device_id` STRING,\n    `sensor_id` STRING,\n    `timestamp` TIMESTAMP,\n    `temperature` FLOAT64,\n    `humidity` FLOAT64,\n    `voltage` FLOAT64,\n    `error_code` STRING,\n    `status` STRING\n)\nPARTITION BY DATE(`timestamp`)\nCLUSTER BY `device_id`, `sensor_id`\n```\n\n```sql\nCREATE TABLE `sensor_data.sensors`\n(\n    `device_id` STRING,\n    `sensor_id` STRING,\n    `created_at` TIMESTAMP,\n    `updated_at` TIMESTAMP,\n    `status` STRING\n)\n```\n\n## リソースのクリーンアップ\n\n作成したリソースを削除：\n```bash\ncd terraform\nterraform destroy\n```\n\n## コントリビューション\n\nPull Requests 歓迎です！以下を確認してください：\n1. プロジェクト規約に従ったコード\n2. ドキュメントの更新\n3. 必要なテストの追加\n\n## ライセンス\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthaddeusjiang%2Fsensor_logs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthaddeusjiang%2Fsensor_logs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthaddeusjiang%2Fsensor_logs/lists"}