{"id":48550317,"url":"https://github.com/njfdev/smathhacks-2026-vocalpathing","last_synced_at":"2026-04-08T08:04:24.732Z","repository":{"id":344522660,"uuid":"1181733609","full_name":"njfdev/smathhacks-2026-vocalpathing","owner":"njfdev","description":null,"archived":false,"fork":false,"pushed_at":"2026-03-15T03:52:53.000Z","size":14430,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-15T14:55:02.027Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/njfdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-14T14:53:44.000Z","updated_at":"2026-03-15T03:52:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/njfdev/smathhacks-2026-vocalpathing","commit_stats":null,"previous_names":["njfdev/smathhacks-2026-vocalpathing"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/njfdev/smathhacks-2026-vocalpathing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njfdev%2Fsmathhacks-2026-vocalpathing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njfdev%2Fsmathhacks-2026-vocalpathing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njfdev%2Fsmathhacks-2026-vocalpathing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njfdev%2Fsmathhacks-2026-vocalpathing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/njfdev","download_url":"https://codeload.github.com/njfdev/smathhacks-2026-vocalpathing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njfdev%2Fsmathhacks-2026-vocalpathing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31545909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T16:28:08.000Z","status":"online","status_checked_at":"2026-04-08T02:00:06.127Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-08T08:04:06.177Z","updated_at":"2026-04-08T08:04:24.719Z","avatar_url":"https://github.com/njfdev.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Vocal Pathing\n\nA real-time marine audio monitoring system built for SmathHacks 2026. It streams live microphone audio from multiple devices to a central dashboard, classifies the sounds using a whale species AI model, and triangulates the location of the sound source.\n\n---\n\n## What It Does\n\n1. **Stream audio** from multiple microphone devices to a dashboard over the internet in real time\n2. **Classify sounds** using a Google whale species AI model (identifies humpback whales, orcas, blue whales, and more)\n3. **Triangulate position** using the time difference between when each microphone hears the sound (TDOA)\n\n---\n\n## Project Structure\n\n```\nsmathhacks-2026-vocalpathing/\n├── vocalpathing-website/       # The website (dashboard + recorder)\n│   ├── app/\n│   │   ├── page.tsx            # Dashboard: shows connected mics, classifications, position\n│   │   └── recorder/page.tsx   # Recorder: captures mic audio and streams it\n│   ├── server.ts               # Backend server that routes audio between devices\n│   └── public/\n│       └── audio-processor.js  # Handles audio capture timing in the browser\n└── ml_stuff/                   # Python AI scripts\n    ├── classify_stream.py      # Classifies incoming audio as whale species\n    ├── triangulate_stream.py   # Calculates XY position of the sound source\n    ├── audio_triangle.py       # Core TDOA triangulation math\n    ├── audio_segmentation.py   # Separates mixed audio into individual sources\n    └── cool_ahh_model.py       # Standalone whale classification script\n```\n\n---\n\n## How It Works\n\n### Audio Streaming\n\nEach device that opens `/recorder` in the browser captures microphone audio and sends it to the server over a WebSocket connection. The server labels each chunk with the device ID and forwards it to the dashboard in real time.\n\n### Species Classification\n\nWhen a recorder connects, the server automatically starts a Python process (`classify_stream.py`) for that device. Audio is piped into it continuously. Every 1 second it runs the Google Multispecies Whale model on the last 5 seconds of audio and sends the result back to the dashboard.\n\nThe model can detect 12 types:\n- Humpback whale (song and call)\n- Orca (call and echolocation)\n- Blue whale\n- Fin whale\n- Minke whale\n- Bryde's whale (two call types)\n- Narwhal (upcall and gunshot)\n- North Pacific right whale\n\n### Triangulation\n\nWhen 3 or more recorders are connected, the server starts `triangulate_stream.py`. We are able to use the difference in time it takes for a sound to reach each microphone. Since sound travels at a fixed speed, if a sound is closer to one mic it will arrive there slightly earlier. By measuring those tiny time gaps across all 3 microphones, we can figure out where the only point in space is that explains all of them at once. That point is where the sound came from.\n\n---\n\n## Setup\n\n### Requirements\n\n- Node.js\n- Python 3 with a virtual environment at `venv/`\n- `mkcert` for local HTTPS (required because the browser needs HTTPS to access the microphone)\n\n### Python Dependencies\n\nFrom the repo root:\n\n```bash\npython3 -m venv venv\nsource venv/bin/activate\npip install numpy scipy librosa tensorflow tensorflow-hub torch asteroid-filterbanks pyroomacoustics soundfile\n```\n\n### SSL Certificates\n\nThe server requires HTTPS. Generate local certificates inside `vocalpathing-website/`:\n\n```bash\nbrew install mkcert\nmkcert -install\ncd vocalpathing-website\nmkcert localhost\n```\n\nThis creates `localhost.pem` and `localhost-key.pem` in that folder.\n\n### Install Node Dependencies\n\n```bash\ncd vocalpathing-website\nnpm install\n```\n\n---\n\n## Running the App\n\n```bash\ncd vocalpathing-website\nnpm run dev\n```\n\nThen open **https://localhost:3000** in your browser.\n\n- The server restarts automatically when you save `server.ts`\n- The browser updates automatically when you save frontend files\n\n---\n\n## Using the Dashboard\n\n1. Open **https://localhost:3000** on the device you want to use as the dashboard\n2. Open **https://localhost:3000/recorder** on each device you want to use as a microphone\n3. On the recorder page, press **Start Recording** to begin streaming\n4. Back on the dashboard, press **Start Listening** to hear the audio\n5. Species classifications will appear under each connected device\n6. Once 3 recorders are connected, the triangulated XY position will appear at the top\n\n### Setting Microphone Positions\n\nFor triangulation to work correctly, you need to enter the physical position of each microphone in meters. Use the **Microphone Positions** section on the dashboard and press **Apply** when done.\n\nExample: if mic 0 is at the origin, mic 1 is 1 meter to the right, and mic 2 is 0.5 meters right and 1 meter forward:\n- Mic 0: X=0, Y=0, Z=0\n- Mic 1: X=1, Y=0, Z=0\n- Mic 2: X=0.5, Y=1, Z=0\n\n---\n\n## WebSocket Message Types\n\n| Message | Direction | Description |\n|---------|-----------|-------------|\n| `register` | Client to Server | Registers a device as a recorder or dashboard |\n| `connect` | Server to Dashboard | A new recorder connected |\n| `disconnect` | Server to Dashboard | A recorder disconnected |\n| `classification` | Server to Dashboard | Whale species detection result |\n| `status` | Server to Dashboard | AI model loading status |\n| `triangulation` | Server to Dashboard | XY position of sound source |\n| `set_mic_positions` | Dashboard to Server | Update microphone positions |\n| Binary PCM | Recorder to Server to Dashboard | Raw audio data |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjfdev%2Fsmathhacks-2026-vocalpathing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnjfdev%2Fsmathhacks-2026-vocalpathing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjfdev%2Fsmathhacks-2026-vocalpathing/lists"}