{"id":47801538,"url":"https://github.com/jsalsman/neshsec-poc","last_synced_at":"2026-04-03T17:04:06.144Z","repository":{"id":341305083,"uuid":"1169608947","full_name":"jsalsman/neshsec-poc","owner":"jsalsman","description":"Native English Speaker Homograph Stress Exemplar Crowdsourcer proof of concept","archived":false,"fork":false,"pushed_at":"2026-03-01T05:32:48.000Z","size":1406,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-01T05:38:06.419Z","etag":null,"topics":["a2a-protocol","crowdsourcing","prolific","pronunciation","proof-of-concept"],"latest_commit_sha":null,"homepage":"https://neshsec-poc.talknicer.com/status","language":"TypeScript","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jsalsman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-28T23:50:07.000Z","updated_at":"2026-03-01T05:32:52.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jsalsman/neshsec-poc","commit_stats":null,"previous_names":["jsalsman/neshsec-poc"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jsalsman/neshsec-poc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsalsman%2Fneshsec-poc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsalsman%2Fneshsec-poc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsalsman%2Fneshsec-poc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsalsman%2Fneshsec-poc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jsalsman","download_url":"https://codeload.github.com/jsalsman/neshsec-poc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsalsman%2Fneshsec-poc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31364614,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-03T15:19:21.178Z","status":"ssl_error","status_checked_at":"2026-04-03T15:19:20.670Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["a2a-protocol","crowdsourcing","prolific","pronunciation","proof-of-concept"],"created_at":"2026-04-03T17:04:05.508Z","updated_at":"2026-04-03T17:04:06.128Z","avatar_url":"https://github.com/jsalsman.png","language":"TypeScript","funding_links":["https://paypal.me/jsalsman"],"categories":[],"sub_categories":[],"readme":"# Native English Speaker Homograph Stress Exemplar Crowdsourcer (NESHSEC) proof of concept agent\n\n[![Try on Cloud Run](https://img.shields.io/badge/Try_on_Cloud_Run-darkgreen)](https://neshsec-poc.talknicer.com/status)\n[![Agent health](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fneshsec-poc.talknicer.com%2Fapi%2Fhealthz\u0026query=%24.status\u0026label=Agent%20health\u0026color=brightgreen\u0026labelColor=indigo)](https://neshsec-poc.talknicer.com/api/healthz)\n[![TypeScript version](https://img.shields.io/badge/TypeScript-5.9.3-3178C6?logo=typescript\u0026logoColor=white)](https://www.typescriptlang.org/)\n[![A2A Compatible](https://img.shields.io/badge/A2A-compatible-purple)](https://a2aprotocol.ai/)\n[![Apache 2.0 License](https://img.shields.io/badge/License-Apache%202.0-brightgreen)](https://opensource.org/licenses/apache-2-0)\n[![Donate](https://img.shields.io/badge/Donate-gold?logo=paypal)](https://paypal.me/jsalsman)\n\nThe NESHSEC PoC Agent is a TypeScript A2A service that orchestrates crowdsourced native English speech collection for homograph stress modeling. It exists to bootstrap the [Syllable Stress Assessment Agent](https://guildaidemo.talknicer.com)’s learned threshold calibration by launching a [Prolific.com](https://www.prolific.com) study, collecting native English speaker participants' paragraph recordings, and forwarding those recordings as native exemplars so the backend can improve stress-decision quality across target word pairs.\n\n## How it works\n\nThe service exposes A2A skills that let an operator launch, inspect, and close a Prolific study targeted at native English speakers. During launch, the agent publishes an existing dashboard-configured study and registers a webhook so completed submissions can be observed by the service. This keeps the study lifecycle operationally centralized in one A2A-callable component.\n\nParticipants enter through the Prolific external study URL and land on `GET /record`, where they are assigned two paragraphs with compact `h4` headings, shown both texts to read, and given sequential controls to record, stop, preview playback, and submit both recordings. Each paragraph keeps the recording button, playback control, and recording status message on one line for quicker review while recording. The browser converts captured microphone input into 16kHz mono WAV before upload so submitted audio matches backend requirements.\n\nOn `POST /submit`, the service forwards both recordings to the Syllable Stress Assessment Agent backend via `pronunciation.evaluate` with `native_exemplar: true`. As exemplars accumulate, the backend’s adaptive pipeline improves threshold calibration, and the PoC agent can query best-effort progress using `convergence_status` while tracking study activity and submission counters toward the full set of 69 target homograph pairs.\n\n## A2A skills\n\n### study_control\n\nThis skill manages Prolific study lifecycle actions (launch, status, close).\n\nInput:\n\n```json\n{ \"skill_id\": \"study_control\", \"params\": { \"action\": \"launch\" | \"status\" | \"close\" } }\n```\n\nOutput:\n- `launch`: `{ success, studyId, studyStatus, publishResponse, study: { name, reward, totalPlaces, totalCost } }`\n- `status`: `{ success, study, submissions: { count }, agentState }`\n- `close`: `{ success, studyId, studyStatus, closeResponse }`\n\n`launch` publishes an existing Prolific dashboard-configured study, registers the webhook, and returns cost metadata from the Prolific API.\n\nExample invocation:\n\n```json\n{\n  \"skill_id\": \"study_control\",\n  \"params\": {\n    \"action\": \"launch\",\n    \"study_id\": \"69a4571f9c16323be5f0c2ec\"\n  }\n}\n```\n\n### convergence_status\n\nThis skill requests best-effort backend convergence telemetry.\n\nInput:\n\n```json\n{ \"skill_id\": \"convergence_status\", \"params\": {} }\n```\n\nOutput: the raw `agent.about` result returned by the Syllable Stress Assessment Agent backend.\n\nA dedicated convergence endpoint does not yet exist on the backend, so this is currently a best-effort status check. A future backend A2A method (`convergence_status`) is expected to expose per-word `decision_method` counts directly.\n\nExample invocation:\n\n```json\n{\n  \"skill_id\": \"convergence_status\",\n  \"params\": {}\n}\n```\n\n### submit_exemplar\n\nThis skill forwards one native exemplar recording for backend evaluation and calibration.\n\nInput:\n\n```json\n{ \"skill_id\": \"submit_exemplar\", \"params\": { \"paragraph_id\": 3, \"audio_wav_base64\": \"\u003cbase64-encoded 16kHz mono WAV\u003e\" } }\n```\n\nOutput: the full `pronunciation.evaluate` result from the backend, including per-target stress evaluations and `score_summary`.\n\nAudio must be 16kHz mono WAV. The service recording page now converts browser-captured audio to that format before submission.\n\nExample invocation:\n\n```json\n{\n  \"skill_id\": \"submit_exemplar\",\n  \"params\": {\n    \"paragraph_id\": 3,\n    \"audio_wav_base64\": \"UklGRiQAAABXQVZFZm10IBAAAAABAAEA...\"\n  }\n}\n```\n\n## Express routes\n\n### GET /status\n\nBrowser-friendly status dashboard showing study state and backend convergence data side by side. Auto-refreshes every 10 seconds. Links to /launch, /close, and /record.\n\n### GET /launch\n\nBrowser-accessible form page with a study ID text input that POSTs to\n/api/study/launch on button click and displays the JSON result inline.\nThe study must already be configured in the Prolific dashboard before launching.\n\n### GET /close\n\nBrowser-accessible confirmation page that POSTs to /api/study/close on button click and displays the JSON result inline. Includes a link to /status.\n\n### GET /api/study/status\n\nReturns current `agentState` and live Prolific study data (submission count and study object) if a study is active. Returns `prolific: null` if no study has been launched. Useful for monitoring without an A2A client.\n\n### POST /api/study/launch\n\nPublishes an existing Prolific study (already configured in the Prolific dashboard)\nand begins tracking it. Requires a JSON body with a `study_id` field. Returns `409`\nif a study is already active, `400` if `study_id` is missing. Cost information is\nfetched from the Prolific API rather than computed locally.\n\nExample:\n\n```bash\ncurl -X POST https://neshsec-poc.talknicer.com/api/study/launch \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"study_id\": \"69a4571f9c16323be5f0c2ec\"}'\n```\n\n### POST /api/study/close\n\nStops the active Prolific study. Returns `404` if no study is active.\n\nExample:\n\n```bash\ncurl -X POST https://neshsec-poc.talknicer.com/api/study/close\n```\n\n### GET /api/convergence\n\nCalls the backend `agent.about` endpoint and returns convergence status.\n\nExample:\n\n```bash\ncurl https://neshsec-poc.talknicer.com/api/convergence\n```\n\n### GET /record\n\nThis route assigns two paragraphs (round-robin fallback), fetches both paragraph texts from the backend, and returns a minimal HTML page with two recording controls (one per paragraph), playback for each recording, and a shared submit button that enables only after both recordings are ready. The submit button remains in the browser's default disabled gray state until activation, then turns blue when both recordings are ready. Each record toggle button is styled inline to appear green while ready to start and red while actively recording, and when one paragraph is actively recording, the other paragraph's start button is temporarily disabled. Prolific query parameters `pid`, `study_id`, and `submission_id` are captured from the URL, while paragraph assignment is handled server-side when a custom field is unavailable. In production, paragraph distribution should move to Prolific Taskflow variants rather than relying on server-side rotation.\n\n### POST /submit\n\nThis route accepts `multipart/form-data` with fields `audio_1` (file), `audio_2` (file), `pid`, `study_id`, `submission_id`, `paragraph_id_1`, and `paragraph_id_2`. It forwards both recordings to backend `pronunciation.evaluate` as native exemplars and returns:\n\n```json\n{ \"success\": true, \"completion_code\": \"STRESS_DONE\", \"result1\": { \"...\": \"backend response\" }, \"result2\": { \"...\": \"backend response\" } }\n```\n\nThe backend requires 16kHz mono WAV, and the recording page performs conversion before upload.\n\n### POST /webhook/prolific\n\nThis route receives Prolific `submissions.completed` events and increments the in-memory `submissionsReceived` counter. A production enhancement would retrieve persisted audio from GCS and forward automatically when webhook events arrive.\n\n### GET /api/healthz\n\nReturns `200` with JSON:\n\n```json\n{ \"status\": \"ok\" }\n```\n\n### A2A routes\n\nThe A2A SDK middleware handles `/.well-known/agent.json` and `/a2a`. `GET /` now issues a temporary redirect (`302`) to `https://github.com/jsalsman/neshsec-poc`; JSON-RPC remains available on `/` for non-GET methods via middleware fallback. The correct agent card route is `/.well-known/agent.json` (some documentation references `agent-card.json`, which is a typo).\n\n## Environment variables\n\n| Variable | Default | Description |\n|---|---|---|\n| PROLIFIC_API_TOKEN | PLACEHOLDER | Prolific Bearer token. Get from prolific.com. |\n| BACKEND_URL | https://guildaidemo.talknicer.com | Base URL of Syllable Stress Assessment Agent. |\n| SERVICE_URL | https://neshsec-poc.talknicer.com | Public URL of this service, used in agent card and Prolific study URL. |\n| GCS_BUCKET_NAME | neshsec-poc | GCS bucket name for persistent state and recording sidecars. |\n| GOOGLE_CREDENTIALS | (none) | Service account JSON string for GCS access. If unset, falls back to Application Default Credentials (works automatically on Cloud Run with a configured service account). |\n| PORT | 8080 | HTTP listen port. |\n\n## Local development quickstart\n\n```bash\nexport PROLIFIC_API_TOKEN=your_token\nexport BACKEND_URL=https://guildaidemo.talknicer.com\nexport SERVICE_URL=http://localhost:8080\nnpm install \u0026\u0026 npm run build \u0026\u0026 npm run start\n```\n\nIf `npm run build` reports missing modules or type declarations (for example `@google-cloud/storage` or `undici`), install dependencies from the lockfile first:\n\n```bash\nnpm ci\n```\n\nWithout a valid `PROLIFIC_API_TOKEN`, `study_control` calls return a clear `PROLIFIC_API_TOKEN_MISSING_OR_INVALID` error. The `/record` and `/submit` routes and `convergence_status` skill still operate independently of Prolific credentials.\n\n## Cloud Run deploy\n\n```bash\ngcloud run deploy neshsec-poc \\\n  --source . \\\n  --region us-west1 \\\n  --allow-unauthenticated \\\n  --port 8080 \\\n  --set-env-vars PROLIFIC_API_TOKEN=...,\\\nBACKEND_URL=https://guildaidemo.talknicer.com,\\\nSERVICE_URL=https://neshsec-poc.talknicer.com,\\\nGCS_BUCKET_NAME=neshsec-poc,\\\nGOOGLE_CREDENTIALS='{\"type\":\"service_account\",\"project_id\":\"...\"}'\n```\n\n## Production notes\n\nGCS persistence is now live in the PoC when `GOOGLE_CREDENTIALS` is configured. `agentState` (including `studyId`, `studyStatus`, submission counters, and `lastAssignedParagraphId`) is loaded from `agent-state.json` in the configured GCS bucket on startup and saved after every mutation. On Cloud Run, set `GOOGLE_CREDENTIALS` to the service account JSON string for the service account that has Storage Object Admin on the bucket. Recording and analysis sidecar persistence to GCS is stubbed as a TODO in the `/submit` route for the production upgrade.\n\nAccurate backend alignment depends on audio format: convert browser-captured webm/opus into 16kHz mono WAV before evaluation. In production this can be done using ffmpeg server-side or a robust WebAssembly converter client-side; this step is essential for PocketSphinx alignment quality.\n\nUse Prolific Taskflow API as the primary distribution strategy for paragraph balancing. Instead of a shared round-robin counter, create 10 paragraph-specific study variants with 15 slots each (10 × 15) to avoid concurrency edge cases and ensure deterministic sampling.\n\nCost is determined by the study configuration in the Prolific dashboard and returned\nfrom the Prolific API at launch time.\n\n## References\n\n- Syllable Stress Assessment Agent backend: https://guildaidemo.talknicer.com\n- A2A JavaScript SDK: https://github.com/a2aproject/a2a-js\n- Prolific API docs: https://docs.prolific.com/api-reference/introduction\n\n## Draft Guild.ai agent project proposal\n\n**The Oath:**\nThe Native English Speaker Homograph Stress Exemplar Crowdsourcer (NESHSEC) is an agent that recruits native English speakers via Prolific to record themselves reading paragraphs containing noun/verb homograph pairs, then submits those recordings to the Syllable Stress Assessment Agent as native exemplars. Its purpose is to bootstrap the data-driven stress-inference calibration of that backend, bringing threshold accuracy from a naive duration heuristic to approximately 95% correct — the practical ceiling given natural within-speaker variability. Without sufficient native exemplar data the Syllable Stress Assessment Agent falls back to a simple \"longer syllable wins\" heuristic; this agent exists to replace that fallback with statistically grounded, learned thresholds for all 69 target homograph pairs.\n\n**The Reagents:**\nThe agent is implemented in TypeScript and deployed on the Guild platform. It depends on the Prolific API to create and monitor a study, recruit participants, and retrieve completed submission metadata. Each participant is presented with two of ten paragraphs covering all 69 target noun/verb homograph pairs as both parts of speech, and records themselves reading each aloud via a browser-based interface with per-paragraph toggle record, playback, and submit controls. Completed audio submissions are forwarded in parallel to the existing Syllable Stress Assessment Agent — a live A2A-compatible Python backend at guildaidemo.talknicer.com — via its `pronunciation.evaluate` JSON-RPC endpoint with `native_exemplar: true`, which persists each WAV and analysis sidecar to Google Cloud Storage and folds the new data into the backend's adaptive threshold computation. Agent state (study ID, submission counters, paragraph assignment cursor) is persisted to a separate GCS bucket via the `GOOGLE_CREDENTIALS` service account, surviving restarts. Prolific participant fees are fetched from the Prolific API at launch time rather\nthan estimated locally; the study is configured directly in the Prolific dashboard\nbefore being published via the agent's /launch interface.\n\n**The Ritual:**\nThe agent exposes three A2A skills (`study_control`, `convergence_status`, `submit_exemplar`) and companion human-navigable browser routes (`/launch`, `/close`, `/status`, `/record`). Launching via `/launch` or the `study_control` skill publishes and tracks an existing Prolific dashboard-configured study and registers a Prolific webhook in a single call. Participants land on `/record`, are assigned two paragraphs via a server-side round-robin counter, record both, preview playback, and submit; the `/submit` route forwards both WAVs in parallel to the backend as native exemplars. Prolific completion webhook events increment the submission counter in persisted agent state. The `/status` dashboard displays live study state, submission counters, and backend capability metadata, auto-refreshing every ten seconds. The `convergence_status` skill and `/api/convergence` route query the backend's `agent.about` endpoint as a best-effort convergence check until a dedicated `convergence_status` backend method is added.\n\n**The Proof:**\nThe agent has succeeded when all 69 target homograph pairs report `decision_method: learned_threshold` in the Syllable Stress Assessment Agent's evaluation responses, indicating that naive duration fallback has been fully replaced by native-exemplar-derived inference. The headline before/after metric is `percent_correct` on a held-out validation set of test fixture WAVs replayed against the backend before the Prolific study begins and again after convergence, quantifying the accuracy improvement the crowdsourced exemplar data delivered. Study completion and per-word convergence progress are themselves exposed as observable agent state via the `/status` dashboard and `convergence_status` skill, making the crowdsourcing pipeline inspectable and steerable throughout its run.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsalsman%2Fneshsec-poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjsalsman%2Fneshsec-poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsalsman%2Fneshsec-poc/lists"}