{"id":26410463,"url":"https://github.com/wayfi-wireless/dune-analytics-uploader","last_synced_at":"2026-05-07T00:38:12.757Z","repository":{"id":282673504,"uuid":"949317890","full_name":"WayFi-Wireless/Dune-Analytics-Uploader","owner":"WayFi-Wireless","description":"Fetches metrics from our metabase instance and uploads them to Dune Analytics","archived":false,"fork":false,"pushed_at":"2025-03-16T07:22:28.000Z","size":40,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-16T08:19:49.500Z","etag":null,"topics":["analytics","csv","duneanalytics","javascript","js","json","metabase","metabase-data","node","node-js","nodejs"],"latest_commit_sha":null,"homepage":"https://dune.com/wayfiwireless/stats","language":"JavaScript","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WayFi-Wireless.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-16T07:05:52.000Z","updated_at":"2025-03-16T07:22:32.000Z","dependencies_parsed_at":"2025-03-16T10:15:23.396Z","dependency_job_id":null,"html_url":"https://github.com/WayFi-Wireless/Dune-Analytics-Uploader","commit_stats":null,"previous_names":["wayfi-wireless/dune-analytics-uploader"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WayFi-Wireless%2FDune-Analytics-Uploader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WayFi-Wireless%2FDune-Analytics-Uploader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WayFi-Wireless%2FDune-Analytics-Uploader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WayFi-Wireless%2FDune-Analytics-Uploader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WayFi-Wireless","download_url":"https://codeload.github.com/WayFi-Wireless/Dune-Analytics-Uploader/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244102849,"owners_count":20398386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","csv","duneanalytics","javascript","js","json","metabase","metabase-data","node","node-js","nodejs"],"created_at":"2025-03-17T20:17:58.001Z","updated_at":"2026-05-07T00:38:12.716Z","avatar_url":"https://github.com/WayFi-Wireless.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WayFi Dune Analytics Integration\n\nThis project automatically fetches WayFi data from Metabase JSON endpoints, converts it to CSV format, and uploads it to Dune Analytics via their CSV Upload API on a daily basis.\n\n[![Website](https://img.shields.io/badge/Website-WayFi%20Wireless-blue)](https://wayfiwireless.com)\n[![Twitter](https://img.shields.io/badge/Twitter-@wayfiwireless-1DA1F2?logo=twitter)](https://twitter.com/wayfiwireless)\n[![LinkedIn](https://img.shields.io/badge/LinkedIn-WayFi%20Wireless-0077B5?logo=linkedin)](https://www.linkedin.com/company/wayfi)\n\n## Features\n\n- Fetches data from multiple Metabase JSON endpoints\n- Converts JSON data to CSV format\n- Uploads CSV data to Dune Analytics datasets via the CSV Upload API\n- Runs automatically every 24 hours\n- Logs all operations for monitoring and debugging\n- Stores local copies of fetched data (both JSON and CSV) for backup\n\n## Data Sources\n\nThe following data is collected from Metabase:\n\n1. Total GB Offloaded\n2. Total Users Served\n3. Total Unique Sessions\n4. Identity Providers Offloading\n5. Active Locations in the Last 30 Days\n6. Radius Serving\n\n## Prerequisites\n\n- Node.js (v14 or higher)\n- npm or yarn\n- Dune Analytics API key (from [dune.com/settings/api](https://dune.com/settings/api))\n\n## Installation\n\n1. Clone this repository:\n   ```\n   git clone https://github.com/yourusername/wayfi-dune-analytics.git\n   cd wayfi-dune-analytics\n   ```\n\n2. Install dependencies:\n   ```\n   npm install\n   ```\n\n3. Run the setup script:\n   ```\n   npm run setup\n   ```\n   \n   This script will:\n   - Create necessary directories\n   - Guide you through creating a `.env` file\n   - Prompt for your Dune Analytics API key\n\nAlternatively, you can manually:\n   - Copy `.env.example` to `.env`\n   - Edit the `.env` file to add your Dune Analytics API key\n\n## Usage\n\n### Run the process once\n\nTo run the data fetch and upload process once without scheduling:\n\n```\nnpm start -- --run-now --no-schedule\n```\n\n### Run as a scheduled service\n\nTo start the service with the scheduled job (runs at midnight every day):\n\n```\nnpm start\n```\n\nThe process will run in the background. Press Ctrl+C to exit.\n\n### GitHub Actions Automation\n\nThis project includes a GitHub Actions workflow that:\n\n1. Runs every 24 hours at midnight UTC\n2. Runs whenever changes are pushed to the main branch\n3. Can be triggered manually from the GitHub Actions tab\n\nTo set up the GitHub Actions workflow:\n\n1. Push this repository to GitHub\n2. Go to your repository's Settings \u003e Secrets and variables \u003e Actions\n3. Add a new repository secret named `DUNE_API_KEY` with your Dune Analytics API key\n4. The workflow will now run automatically according to the schedule\n\nThe workflow logs are archived as artifacts and can be downloaded from the GitHub Actions tab.\n\n### Configuration\n\nYou can modify the `config.js` file to:\n\n- Change the data endpoints\n- Adjust the schedule frequency\n- Update Dune dataset names\n\n## Project Structure\n\n```\n├── config.js                    # Configuration settings\n├── index.js                     # Main application file\n├── package.json                 # Project dependencies\n├── setup.js                     # Setup script\n├── dune_queries.sql             # Example SQL queries for Dune Analytics\n├── dune_visualization_guide.md  # Guide for creating visualizations in Dune\n├── .env.example                 # Example environment variables\n├── .env                         # Environment variables (create from .env.example)\n├── .gitignore                   # Git ignore configuration\n├── LICENSE                      # MIT license\n├── README.md                    # Project documentation\n├── data/                        # Directory for storing fetched data (JSON and CSV)\n│   └── .gitkeep\n├── logs/                        # Directory for storing operation logs\n│   └── .gitkeep\n├── .github/\n│   └── workflows/\n│       └── data-sync.yml        # GitHub Actions workflow\n└── utils/\n    ├── dataFetcher.js           # Utility for fetching data from endpoints\n    ├── jsonToCsv.js             # Utility for converting JSON to CSV\n    └── duneUploader.js          # Utility for uploading data to Dune\n```\n\n## Logs\n\nAll operations are logged in the `logs` directory:\n\n- `summary_*.json`: Summary of each complete process run\n- `upload_*.json`: Details of each upload operation\n- `error_*.json`: Details of any errors that occur\n\n## Dune Analytics Queries and Visualization\n\nExample SQL queries for visualizing the data in Dune Analytics are provided in the `dune_queries.sql` file. These queries include:\n\n- Basic queries to retrieve the latest data for each dataset\n- Trend analysis queries to show data changes over time\n- Percentage breakdown queries for categorical data\n- Combined queries that join multiple datasets for comprehensive analysis\n- Dashboard queries for key metrics\n\nA detailed guide on how to use these queries to create visualizations and dashboards in Dune Analytics is provided in the `dune_visualization_guide.md` file. This guide includes:\n\n- Step-by-step instructions for creating visualizations\n- Recommended visualization types for each query\n- Guidelines for creating a comprehensive dashboard\n- Tips for effective data visualization\n- Troubleshooting advice\n\n## Troubleshooting\n\nIf you encounter issues:\n\n1. Check the error logs in the `logs` directory\n2. Verify your Dune API key is correct\n3. Ensure all endpoints are accessible\n4. Check your internet connection\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwayfi-wireless%2Fdune-analytics-uploader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwayfi-wireless%2Fdune-analytics-uploader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwayfi-wireless%2Fdune-analytics-uploader/lists"}