{"id":22897752,"url":"https://github.com/khoidm2004/iot-pipeline","last_synced_at":"2026-04-14T19:32:57.177Z","repository":{"id":267895937,"uuid":"902687463","full_name":"khoidm2004/IoT-Pipeline","owner":"khoidm2004","description":"Humidity Data Collection IoT System","archived":false,"fork":false,"pushed_at":"2025-04-11T14:26:12.000Z","size":723,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-11T16:08:03.307Z","etag":null,"topics":["api-rest","azure","hosting-az","iot-application","nodejs","reactjs","server"],"latest_commit_sha":null,"homepage":"https://youtu.be/FhHddqXZUsI?si=17N1RcTIJZhAWMEP","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/khoidm2004.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-13T04:15:53.000Z","updated_at":"2024-12-13T12:28:34.000Z","dependencies_parsed_at":"2025-10-11T16:07:17.939Z","dependency_job_id":"d75cb68d-e9a5-4f16-93e1-98af32d8684e","html_url":"https://github.com/khoidm2004/IoT-Pipeline","commit_stats":null,"previous_names":["khoidm2004/iot-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/khoidm2004/IoT-Pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khoidm2004%2FIoT-Pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khoidm2004%2FIoT-Pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khoidm2004%2FIoT-Pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khoidm2004%2FIoT-Pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/khoidm2004","download_url":"https://codeload.github.com/khoidm2004/IoT-Pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khoidm2004%2FIoT-Pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31812968,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"ssl_error","status_checked_at":"2026-04-14T18:05:01.765Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-rest","azure","hosting-az","iot-application","nodejs","reactjs","server"],"created_at":"2024-12-14T00:18:59.792Z","updated_at":"2026-04-14T19:32:57.170Z","avatar_url":"https://github.com/khoidm2004.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IoT Pipeline Project Documentation\n\n## Overview\n\nThis project establishes an IoT data pipeline connecting a Raspberry Pi Pico W running MicroPython to a cloud-based infrastructure using a VPS with Ubuntu 24.04 LTS, Node.js 23, InfluxDB v2.7.10, and Nginx v1.24.0.\n\n---\n\n## System Architecture\n\n### Components:\n\n- **IoT Device:** Raspberry Pi Pico W\n- **Backend:** Node.js\n- **Frontend:** Vite + React\n- **Database:** InfluxDB\n- **Web Server:** Nginx\n- **Operating System:** Ubuntu 24.04 LTS (VPS)\n\n---\n\n## Implementation Details\n\n### 1. IoT Device Setup\n\n- **Language:** MicroPython\n- **Data Collection:** Temperature and humidity sensors\n- **Data Transmission:** HTTP POST requests to the backend API.\n\n#### Embedded Code Snippet\n\n```python\nfrom machine import Pin\nfrom dht import DHT11\nfrom utime import sleep\nimport time\nfrom network import WLAN\nimport network\nimport urequests as requests\n\n# Initial values\n# Initialize DHT11 Sensor\ndhtPin = 0\ndht = DHT11(Pin(dhtPin, Pin.IN))\nLED = Pin(\"LED\",Pin.OUT)\n\nINTERVAL = 20 # 20 seconds for cloud\n\n# Wifi Details\nWF_SSID = \"DN.Matthias\"\nWF_PASS = \"idontknow\"\n\n# REST API Details\nBASE_URL = \"http://68.219.251.214/db_api/api/v1/\"\nDATA_ENDPOINT = \"/embed\"\nDEBUG = True\n\ndef log(data)-\u003eNone:\n    if DEBUG:\n        print(repr(data))\n    return None\n\ndef sendData(endpoint: str, humidity: int)-\u003eNone:\n    url = f\"{BASE_URL}/{endpoint}?value={humidity}\"\n    try:\n        log(\"Sending request to {url}\")\n        res = requests.get(url)\n        log(f\"Response status: {res.status_code}\")\n        log(f\"Response message: {res.text}\")\n    except Exception as error:\n        print(\"Error:\",error)\n\n\ndef connectWifi()-\u003eWLAN:\n    wlan = network.WLAN(network.STA_IF)\n    print(\"Connecting\", end=\"\")\n    wlan.active(True)\n    wlan.connect(WF_SSID,WF_PASS)\n    while not wlan.isconnected():\n        print(\"\\n...\",end=\"\")\n        time.sleep(0.1)\n    print(\"\\nConnected!\")\n    log(wlan.ifconfig())\n    return wlan\n\ndef main()-\u003eNone:\n    print(\"Program starting.\")\n    wlan = connectWifi()\n\n    while True:\n        dht.measure()\n        humid = dht.humidity()\n        print('Humidity:', humid, '%')\n        LED.on()\n        sendData(DATA_ENDPOINT,humid)\n        time.sleep(0.2)\n        LED.off()\n        time.sleep(INTERVAL)\n\n\nmain()\n```\n\n---\n\n### 2. Backend API\n\n- **Framework:** Express.js\n- **Functionality:** Receive, sanitize, and store incoming data in InfluxDB.\n\n#### Backend Code Snippet\n\n```javascript\nimport express from \"express\";\nimport { InfluxDB, Point } from \"@influxdata/influxdb-client\";\nimport { getEnvs } from \"./envs.mjs\";\nconst ENV = getEnvs();\nconst app = express();\nconsole.log(ENV.INFLUX.HOST);\n// 1.2 Initialize DB connection\nconst DB_CLIENT = new InfluxDB({\n  url: ENV.INFLUX.HOST,\n  token: ENV.INFLUX.TOKEN,\n});\nconst DB_WRITE_POINT = DB_CLIENT.getWriteApi(ENV.INFLUX.ORG, ENV.INFLUX.BUCKET);\nDB_WRITE_POINT.useDefaultTags({ app: \"db_api\" });\n// Endpoint - embed\napp.get(\"/api/v1/\", (_, res) =\u003e res.sendStatus(200));\napp.get(\"/api/v1/embed\", async (req, res) =\u003e {\n  try {\n    const value = req.query.value;\n    const numeric_value = parseFloat(value);\n    const point = new Point(\"qparams\");\n    point.floatField(\"value\", numeric_value);\n    DB_WRITE_POINT.writePoint(point); // starts transaction\n    await DB_WRITE_POINT.flush(); // end the transaction =\u003e save\n    res.send(`Value: '${value}' written.`);\n  } catch (err) {\n    console.error(err);\n    // console.log({ db: ENV.INFLUX.HOST });\n    res.sendStatus(500);\n  }\n});\n\n// Enpoints - base\napp.get(\"\", (_, res) =\u003e res.send(\"OK\"));\n\n// Enpoints - test query params\napp.get(\"/test\", (req, res) =\u003e {\n  console.log(req.query);\n  res.send(\"received queryparams!\");\n});\n\n// Enpoints - Fetch data from InfluxDB\napp.get(\"/api/v1/getData\", async (req, res) =\u003e {\n  const query = `\n        from(bucket: \"${ENV.INFLUX.BUCKET}\")\n        |\u003e range(start: -30d)\n        |\u003e filter(fn: (r) =\u003e r._measurement == \"qparams\")\n        |\u003e filter(fn: (r) =\u003e r._field == \"value\")\n    `;\n\n  try {\n    const data = [];\n    const DB_READ_API = DB_CLIENT.getQueryApi(ENV.INFLUX.ORG);\n\n    await DB_READ_API.queryRows(query, {\n      next(row, tableMeta) {\n        const res = tableMeta.toObject(row);\n        data.push(res);\n      },\n      error(error) {\n        console.error(\"Error during query:\", error);\n        res.status(500).send(\"Error fetching data from InfluxDB\");\n      },\n      complete() {\n        if (data.length === 0) {\n          res.status(404).send(\"No data found\");\n        } else {\n          res.json(data);\n        }\n      },\n    });\n  } catch (err) {\n    console.error(\"Error in /get-data route:\", err); // Log lỗi nếu có\n    res.status(500).send(\"Error fetching data from InfluxDB\");\n  }\n});\n\napp.listen(ENV.PORT, ENV.HOST, () =\u003e {\n  console.log(`Listening http://${ENV.HOST}:${ENV.PORT}`);\n});\n```\n\n#### Loading Env Variables Code Snippet\n\n```javascript\n/**\n * @typedef {object} INFLUX_CONF\n * @property {string} HOST Address to influxDB\n * @property {string} ORG Organization\n * @property {string} BUCKET Bucket name\n * @property {string} TOKEN Token name\n */\n\n/**\n * @typedef {object} ENV\n * @property {number} PORT 1024-65535\n * @property {string} HOST IP or FQDN(Fully Qualified Domain Name)\n * @property {INFLUX_CONF} INFLUX\n */\n\n/** @type {ENV} */\nconst ENV = {\n  PORT: -1,\n  HOST: \"\",\n  INFLUX: {\n    HOST: \"\",\n    ORG: \"\",\n    BUCKET: \"\",\n    TOKEN: \"\",\n  },\n};\n\n/**\n * Gets the environment variables.\n * @returns {ENV}\n * @throws {Error}\n */\nexport const getEnvs = () =\u003e {\n  if (ENV.PORT == -1) {\n    try {\n      // Load host address\n      ENV.HOST =\n        process.env.HOST !== undefined\n          ? process.env.HOST\n          : () =\u003e {\n              throw new Error(\"HOST is not defined in the .env\");\n            };\n      const port = parseInt(process.env.PORT, 10);\n      if (isNaN(port) || port \u003c 1024 || port \u003e 65535) {\n        throw new Error(\"PORT must be 1024-65535.\");\n      }\n      ENV.PORT = port;\n      // Influx\n      ENV.INFLUX.HOST = process.env.DB_INFLUX_HOST || \"http://localhost:8086\";\n      ENV.INFLUX.ORG = process.env.DB_INFLUX_ORG\n        ? process.env.DB_INFLUX_ORG\n        : () =\u003e {\n            throw new Error(\"DB_INFLUX_ORG undefined.\");\n          };\n      ENV.INFLUX.BUCKET = process.env.DB_INFLUX_BUCKET\n        ? process.env.DB_INFLUX_BUCKET\n        : () =\u003e {\n            throw new Error(\"DB_INFLUX_BUCKET undefined.\");\n          };\n      ENV.INFLUX.TOKEN = process.env.DB_INFLUX_TOKEN\n        ? process.env.DB_INFLUX_TOKEN\n        : () =\u003e {\n            throw new Error(\"DB_INFLUX_TOKEN undefined.\");\n          };\n      return ENV;\n    } catch (err) {\n      console.error(err);\n      process.exit(1);\n    }\n  } else {\n    return ENV;\n  }\n};\n```\n\n---\n\n### 3. Data Visualization\n\n- **Dashboard:** Custom React+ Chart.js for live data representation.\n\n#### Frontend Code Snippet\n\n```javascript\nimport { useState, useEffect } from \"react\";\nimport { Line } from \"react-chartjs-2\";\nimport {\n  Chart as ChartJS,\n  CategoryScale,\n  LinearScale,\n  PointElement,\n  LineElement,\n  Title,\n  Tooltip,\n  Legend,\n} from \"chart.js\";\nimport \"./App.css\";\n\nChartJS.register(\n  CategoryScale,\n  LinearScale,\n  PointElement,\n  LineElement,\n  Title,\n  Tooltip,\n  Legend\n);\n\nconst App = () =\u003e {\n  const [data, setData] = useState([]);\n  const [loading, setLoading] = useState(true);\n  const [error, setError] = useState(null);\n  const [maxHumidity, setMaxHumidity] = useState(null);\n  const [minHumidity, setMinHumidity] = useState(null);\n  const [meanHumidity, setMeanHumidity] = useState(null);\n\n  useEffect(() =\u003e {\n    fetch(\"/api/db_api/api/v1/getData/\")\n      .then((response) =\u003e {\n        if (!response.ok) {\n          throw new Error(\"Network response was not ok\");\n        }\n        return response.json();\n      })\n      .then((data) =\u003e {\n        setData(data);\n        setLoading(false);\n\n        // Find the day with the highest and lowest humidity\n        const maxData = data.reduce(\n          (max, item) =\u003e (item._value \u003e max._value ? item : max),\n          data[0]\n        );\n        const minData = data.reduce(\n          (min, item) =\u003e (item._value \u003c min._value ? item : min),\n          data[0]\n        );\n\n        // Calculate the mean humidity\n        const sumHumidity = data.reduce((sum, item) =\u003e sum + item._value, 0);\n        const meanHumidityValue = sumHumidity / data.length;\n\n        setMaxHumidity(maxData);\n        setMinHumidity(minData);\n        setMeanHumidity(meanHumidityValue.toFixed(2));\n\n        console.log(data);\n      })\n      .catch((error) =\u003e {\n        setError(error.message);\n        setLoading(false);\n      });\n  }, []);\n\n  if (loading) return \u003cp\u003eLoading...\u003c/p\u003e;\n  if (error) return \u003cp\u003eError: {error}\u003c/p\u003e;\n\n  const chartData = {\n    labels: data.map((item) =\u003e new Date(item._time).toLocaleString()),\n    datasets: [\n      {\n        label: \"Humidity\",\n        data: data.map((item) =\u003e item._value),\n        borderColor: \"#42A5F5\",\n        fill: false,\n      },\n    ],\n  };\n\n  return (\n    \u003cdiv className=\"dashboard\"\u003e\n      \u003ch1\u003eHumidity Data Dashboard\u003c/h1\u003e\n      {maxHumidity \u0026\u0026 (\n        \u003cp\u003e\n          Highest Humidity: {maxHumidity._value}% on{\" \"}\n          {new Date(maxHumidity._time).toLocaleString()}\n        \u003c/p\u003e\n      )}\n      {minHumidity \u0026\u0026 (\n        \u003cp\u003e\n          Lowest Humidity: {minHumidity._value}% on{\" \"}\n          {new Date(minHumidity._time).toLocaleString()}\n        \u003c/p\u003e\n      )}\n      {meanHumidity \u0026\u0026 \u003cp\u003eMean Humidity: {meanHumidity}%\u003c/p\u003e}\n      \u003cdiv className=\"chart-container\"\u003e\n        \u003cLine data={chartData} options={{ responsive: true }} /\u003e\n      \u003c/div\u003e\n    \u003c/div\u003e\n  );\n};\n\nexport default App;\n```\n\n---\n\n## Benefits of the Selected Services\n\n- **Raspberry Pi Pico W:** Affordable and capable of running MicroPython.\n- **VPS with Ubuntu:** Reliable, scalable, and supports 24/7 operation.\n- **Node.js:** High-performance backend for real-time data handling.\n- **InfluxDB:** Optimized for time-series data storage.\n- **Nginx:** Efficient web server and reverse proxy.\n- **React:** Enhancing code reusability and maintainability.\n\n---\n\n## Conclusion\n\nThis IoT pipeline successfully collects, processes, and visualizes sensor data in real time. The system operates continuously and ensures high availability.\n\n### Additional Deliverables:\n\n- **Video Presentation:** [[YouTube Video Link](https://youtu.be/FhHddqXZUsI)]\n- **Source Code Repositories:** [[GitHub Repository Link](https://github.com/khoidm2004/IoT-Pipeline)]\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhoidm2004%2Fiot-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkhoidm2004%2Fiot-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhoidm2004%2Fiot-pipeline/lists"}