{"id":27916666,"url":"https://github.com/strawberry-computer/speakbuild","last_synced_at":"2025-06-17T01:38:51.832Z","repository":{"id":273452221,"uuid":"919766177","full_name":"Strawberry-Computer/speakbuild","owner":"Strawberry-Computer","description":"Mobile voice assistant that can generate UI on the go","archived":false,"fork":false,"pushed_at":"2025-05-03T00:48:01.000Z","size":2484,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-06T18:09:12.524Z","etag":null,"topics":["ai","vibe-coding","web4"],"latest_commit_sha":null,"homepage":"https://strawberry.computer","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Strawberry-Computer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-21T01:09:21.000Z","updated_at":"2025-05-03T00:48:05.000Z","dependencies_parsed_at":"2025-02-07T13:26:16.664Z","dependency_job_id":"f5b1e825-78ea-42df-81e9-70e7c8ae7ef9","html_url":"https://github.com/Strawberry-Computer/speakbuild","commit_stats":null,"previous_names":["vgrichina/everything-app","vgrichina/speakbuild","strawberry-computer/speakbuild"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/Strawberry-Computer/speakbuild","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Strawberry-Computer%2Fspeakbuild","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Strawberry-Computer%2Fspeakbuild/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Strawberry-Computer%2Fspeakbuild/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Strawberry-Computer%2Fspeakbuild/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Strawberry-Computer","download_url":"https://codeload.github.com/Strawberry-Computer/speakbuild/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Strawberry-Computer%2Fspeakbuild/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260273453,"owners_count":22984495,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","vibe-coding","web4"],"created_at":"2025-05-06T16:05:32.753Z","updated_at":"2025-06-17T01:38:51.806Z","avatar_url":"https://github.com/Strawberry-Computer.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Voice-Driven React Native Component Generator\n\nAn AI-powered voice assistant that generates and modifies React Native components through natural language commands. Built with Expo and OpenRouter API (Claude).\n\n## Features\n\n- 🎙️ Voice commands to generate UI components\n- ⚡ Real-time component generation and preview\n- 🔄 Component modification through natural language\n- 📱 Cross-platform (iOS/Android) support\n- 💾 Persistent component storage\n- 🧪 Debug generation interface for testing\n\n## Installation\n\n### Prerequisites\n\n- Node.js\n- Yarn or npm\n- Expo CLI\n- Bun (for evaluation scripts)\n- iOS/Android development environment\n\n### Setup\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/yourusername/voice-assistant.git\ncd voice-assistant\n```\n\n2. Install dependencies:\n```bash\nyarn install\n```\n\n3. Configure environment:\n- Set up OpenRouter API key in app settings\n- Configure required permissions:\n  - Microphone access\n  - Speech recognition\n  - Internet access\n\n### Running the App\n\nThis app requires a native build due to dependencies on native modules (speech recognition, etc). It cannot run in Expo Go.\n\n```bash\n# Build and run on iOS simulator/device\nyarn ios\n\n# Build and run on Android emulator/device\nyarn android\n\n# Clean build cache if needed\nyarn clean\n```\n\nNote: Web platform support is limited due to native module dependencies.\n\n## How It Works\n\n### 1. User Interaction Flow\n\nThe system follows this interaction flow:\n\n1. **Voice/Text Input**: User provides input via voice button or keyboard\n2. **Transcription**: Audio is converted to text (for voice input)\n3. **Analysis**: Input is analyzed to determine intent and widget specification\n4. **Component Generation**: A React Native component is generated based on the specification\n5. **Rendering**: The component is rendered and displayed to the user\n6. **History Management**: The interaction is saved in conversation history\n\n### 2. Widget Specification System\n\nThe system uses a structured URL-based widget specification system:\n\n```\ncategory/type/style/theme?with_feature=yes\u0026params=name:type\n```\n\nCategories:\n- `display`: Information display (clock, weather, calendar)\n- `input`: User data entry (lists, notes, forms)\n- `interactive`: User actions (timer, player, calculator)\n- `feedback`: System responses (progress, loading, alerts)\n- `media`: Rich content (images, video, audio)\n- `content`: for educational/informational content (articles, explanations, facts)\n\nFeatures flags (with_*):\n- `with_controls`: Play/pause/reset controls\n- `with_dates`: Date/time handling\n- `with_progress`: Progress tracking\n- `with_checkboxes`: Checkbox toggles\n- `with_hourly`: Hourly breakdown\n- `with_daily`: Daily breakdown\n- `with_alarm`: Alarm functionality\n- `with_sections`: Content sections\n\nExample:\n```\ndisplay/weather/forecast/light?with_daily=yes\u0026params=location:caption,unit:caption,date:string,days:integer\n```\n\n### 2. Parameter Types\n\nThe system supports strongly-typed parameters:\n\nText Types:\n- `caption`: Short labels (1-3 words)\n- `title`: Headings with context (3-7 words)\n- `sentence`: Single complete thought\n- `paragraph`: Multiple sentences\n- `story`: Long-form content\n- `url`: Web URLs\n\nNumber Types:\n- Basic: `integer`, `decimal`\n- Semantic: `size`, `duration`, `count`, `percentage`, `interval`, `goal`, `currency`\n\nArrays:\n- `caption[]`: Lists of short items\n- `sentence[]`: Lists of tasks/notes\n- `{text:string,done:boolean}[]`: Basic todo items\n- `{text:string,done:boolean,time:string}[]`: Scheduled todo items\n- `{text:string,selected:boolean,value:string}[]`: Selection list items\n\n## Technical Architecture\n\n### Core Services\n\n`assistantService.js`:\n- Central service that manages the voice assistant state\n- Handles audio recording, transcription, and component generation\n- Maintains status (IDLE, LISTENING, THINKING, PROCESSING, ERROR)\n- Supports different interaction modes (PTT, CALL)\n- Emits events for UI updates\n\n`audioSession.js`:\n- Manages WebSocket connections for audio streaming\n- Handles microphone access and audio processing\n- Provides volume level monitoring\n- Supports push-to-talk and call modes\n\n`analysis.js`:\n- Analyzes user requests using Claude\n- Determines intent (new/modify)\n- Generates widget URLs and parameters\n- Maintains request history context\n\n`api.js`:\n- Handles OpenRouter API communication\n- Supports both streaming and non-streaming completions\n- Includes detailed request/response logging\n- Handles SSE for real-time responses\n\n`componentGeneration.js`:\n- Creates React Native components from widget specifications\n- Supports streaming generation with progress callbacks\n- Handles component validation and error handling\n- Provides abort capability for in-progress generations\n\n`componentUtils.js`:\n- Provides utilities for creating and rendering components\n- Handles component sandboxing and error boundaries\n- Manages React and React Native dependencies injection\n- Supports dynamic component rendering with props\n\n`widgetStorage.js`:\n- Manages persistent storage of generated components\n- Stores components by widget URL\n- Maintains version history with timestamps\n- Provides retrieval and update capabilities\n\n`componentHistoryService.js`:\n- Manages conversation and component history\n- Supports navigation through previous components\n- Maintains current component state\n- Provides event-based state updates\n\n## Platform-Specific Configuration\n\n### iOS\n- Bundle Identifier: `com.voiceassistant.app`\n- Required Permissions:\n  - Microphone Usage\n  - Speech Recognition\n  - Background Audio Mode\n\n### Android\n- Package: `com.voiceassistant.app`\n- Required Permissions:\n  - RECORD_AUDIO\n  - INTERNET\n- Build Configuration:\n  - Kotlin Version: 1.8.10\n  - Compile SDK: 35\n  - Target SDK: 34\n  - Build Tools: 34.0.0\n\n## Evaluation System\n\nThe app includes two evaluation scripts for testing the AI components:\n\n### Analysis Evaluation\n\nTests the system's ability to understand user requests and convert them to structured widget specifications:\n\n```bash\nbun scripts/evaluate-analysis.js [model]\n# Default: anthropic/claude-3.5-sonnet\n```\n\nExample test case:\n```json\n{\n    \"request\": \"What time is it?\",\n    \"expected\": {\n        \"intent\": \"new\",\n        \"widgetUrl\": \"display/clock/digital/light?params=format:caption,size:integer\",\n        \"params\": {\n            \"format\": \"HH:mm\",\n            \"size\": 48\n        }\n    }\n}\n```\n\n### Component Generation Evaluation\n\nTests the system's ability to generate functional React Native components from widget specifications:\n\n```bash\nbun scripts/evaluate-generation.js [model]\n# Default: anthropic/claude-3.5-sonnet\n```\n\nEach evaluation generates a detailed report with:\n- Success rate percentage\n- Average response time\n- Detailed per-test results\n- Error analysis\n\nReports are saved in the `evaluations/` directory with filenames:\n- `analysis-[date]-[model].md`\n- `generation-[date]-[model].md`\n\n## Dependencies\n\nKey packages:\n- `expo` ~52.0.0\n- `@expo/vector-icons` ^14.0.0\n- `expo-av` ~15.0.2\n- `expo-clipboard` ~7.0.1\n- `expo-file-system` ~18.0.7\n- `expo-haptics` ~14.0.1\n- `expo-image-picker` ~16.0.4\n- `expo-linking` ~7.0.0\n- `expo-location` ~18.0.5\n- `expo-media-library` ~17.0.5\n- `expo-notifications` ~0.29.12\n- `expo-sensors` ~14.0.2\n- `expo-sharing` ~13.0.1\n- `react` 18.3.1\n- `react-native` 0.76.6\n- `@react-native-async-storage/async-storage` 1.23.1\n- `react-native-gesture-handler` ~2.20.2\n- `react-native-reanimated` ~3.16.1\n- `react-native-audio-record` ^0.2.2\n- `@expo/vector-icons` ^14.0.4\n- `partial-json` ^0.1.7\n\nFor full list of dependencies, see `package.json`.\n\n## Development Notes\n\n- Uses Expo Router for navigation\n- Supports TypeScript\n- Includes custom Expo plugins for speech recognition\n- Configured for both light and dark mode support\n- Uses EventEmitter pattern for state management\n- Implements custom hooks for component state (useAssistantState)\n- Uses MMKV for high-performance storage\n- Supports both voice and keyboard input methods\n- Implements WebSocket-based audio streaming\n\n## License\n\nMIT License\n\n## Test API Keys for Builds\n\n### Local Development with Test Keys\n\nFor local development, you can include test API keys so you don't need to enter them in the app:\n\n1. Create a `.env` file with your test API keys:\n\n```bash\n# Copy the example file\ncp .env.example .env\n\n# Edit the .env file with your test keys\n# EXPO_PUBLIC_TEST_ULTRAVOX_KEY=your-ultravox-test-key\n# EXPO_PUBLIC_TEST_OPENROUTER_KEY=your-openrouter-test-key\n```\n\n2. Run the app with the environment variables loaded:\n\n```bash\n# For iOS\nyarn ios\n\n# For Android\nyarn android\n```\n\n### CI/CD Builds with Test Keys\n\nFor automated builds via GitHub Actions, test API keys are injected at build time:\n\n1. Store your API keys as GitHub repository secrets:\n   - `EXPO_PUBLIC_TEST_ULTRAVOX_KEY`\n   - `EXPO_PUBLIC_TEST_OPENROUTER_KEY`\n\n2. The CI workflow automatically passes these secrets to EAS Build:\n\n```yaml\n# From .github/workflows/release.yml\n- name: Build and submit iOS app\n  run: eas build --platform ios --profile production --non-interactive --auto-submit\n  env:\n    EXPO_PUBLIC_TEST_ULTRAVOX_KEY: ${{ secrets.EXPO_PUBLIC_TEST_ULTRAVOX_KEY }}\n    EXPO_PUBLIC_TEST_OPENROUTER_KEY: ${{ secrets.EXPO_PUBLIC_TEST_OPENROUTER_KEY }}\n```\n\nThis allows TestFlight and Play Store testers to use the app without needing to enter API keys.\n\n## Contributing\n\n1. Fork the repository\n2. Create your feature branch\n3. Commit your changes\n4. Push to the branch\n5. Create a new Pull Request\n\n### Development Workflow\n\n1. **Setup Environment**:\n   - Install dependencies: `yarn install`\n   - Configure API keys in `.env` file\n\n2. **Run Tests**:\n   - Evaluate analysis: `yarn evaluate-analysis`\n   - Evaluate generation: `yarn evaluate-generation`\n\n3. **Build and Test**:\n   - Development build: `yarn ios` or `yarn android`\n   - Production build: `yarn build:ios` or `yarn build:android`\n\n4. **Submit Changes**:\n   - Ensure all tests pass\n   - Follow the existing code style\n   - Include documentation updates\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrawberry-computer%2Fspeakbuild","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstrawberry-computer%2Fspeakbuild","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrawberry-computer%2Fspeakbuild/lists"}