{"id":31884327,"url":"https://github.com/copilotkit/canvas-with-llamaindex","last_synced_at":"2025-10-13T03:58:04.406Z","repository":{"id":316474551,"uuid":"1059652862","full_name":"CopilotKit/canvas-with-llamaindex","owner":"CopilotKit","description":null,"archived":false,"fork":false,"pushed_at":"2025-09-24T19:40:21.000Z","size":256,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-24T21:26:15.678Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CopilotKit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-18T18:35:50.000Z","updated_at":"2025-09-24T19:40:24.000Z","dependencies_parsed_at":"2025-09-24T21:36:27.640Z","dependency_job_id":null,"html_url":"https://github.com/CopilotKit/canvas-with-llamaindex","commit_stats":null,"previous_names":["copilotkit/canvas-with-llamaindex"],"tags_count":null,"template":true,"template_full_name":null,"purl":"pkg:github/CopilotKit/canvas-with-llamaindex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CopilotKit%2Fcanvas-with-llamaindex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CopilotKit%2Fcanvas-with-llamaindex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CopilotKit%2Fcanvas-with-llamaindex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CopilotKit%2Fcanvas-with-llamaindex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CopilotKit","download_url":"https://codeload.github.com/CopilotKit/canvas-with-llamaindex/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CopilotKit%2Fcanvas-with-llamaindex/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279013508,"owners_count":26085369,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-13T03:58:00.958Z","updated_at":"2025-10-13T03:58:04.398Z","avatar_url":"https://github.com/CopilotKit.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CopilotKit \u003c\u003e LlamaIndex AG-UI Canvas Starter\n\nThis is a starter template for building AI-powered canvas applications using [LlamaIndex](https://llamaindex.com) and [CopilotKit](https://copilotkit.ai). It provides a modern Next.js application with an integrated LlamaIndex agent that manages a visual canvas of interactive cards with real-time AI synchronization.\n\nhttps://github.com/user-attachments/assets/2a4ec718-b83b-4968-9cbe-7c1fe082e958\n\n## 🚀 Key Features\n\n- **Visual Canvas Interface**: Drag-free canvas displaying cards in a responsive grid layout\n- **Four Card Types**: \n  - **Project**: Includes text fields, dropdown, date picker, and checklist\n  - **Entity**: Features text fields, dropdown, and multi-select tags\n  - **Note**: Simple rich text content area\n  - **Chart**: Visual metrics with percentage-based bar charts\n- **Real-time AI Sync**: Bidirectional synchronization between the AI agent and UI canvas\n- **Multi-step Planning**: AI can create and execute plans with visual progress tracking\n- **Human-in-the-Loop (HITL)**: Intelligent interrupts for clarification when needed\n- **JSON View**: Toggle between visual canvas and raw JSON state\n- **Responsive Design**: Optimized for both desktop (sidebar chat) and mobile (popup chat)\n\n## Prerequisites\n\n- Node.js 18+ \n- Python 3.8+\n- OpenAI API Key (for the LlamaIndex agent)\n- [uv](https://docs.astral.sh/uv/getting-started/installation/)\n- Any of the following package managers:\n  - [pnpm](https://pnpm.io/installation) (recommended)\n  - npm\n  - [yarn](https://classic.yarnpkg.com/lang/en/docs/install/#mac-stable)\n  - [bun](https://bun.sh/)\n\n\u003e **Note:** This repository ignores lock files (package-lock.json, yarn.lock, pnpm-lock.yaml, bun.lockb) to avoid conflicts between different package managers. Each developer should generate their own lock file using their preferred package manager. After that, make sure to delete it from the .gitignore.\n\n## Getting Started\n\n1. Install dependencies using your preferred package manager:\n```bash\n# Using pnpm (recommended)\npnpm install\n\n# Using npm\nnpm install\n\n# Using yarn\nyarn install\n\n# Using bun\nbun install\n```\n\n2. Install Python dependencies for the LlamaIndex agent (requires uv).\n   If you don't have uv installed, install it first using one of the following:\n   - macOS (Homebrew): `brew install uv`\n   - macOS/Linux (official installer): `curl -LsSf https://astral.sh/uv/install.sh | sh`\n   - Or with pipx: `pipx install uv`\n```bash\n# Using pnpm\npnpm install:agent\n\n# Using npm\nnpm run install:agent\n\n# Using yarn\nyarn install:agent\n\n# Using bun\nbun run install:agent\n```\n\n\u003e **Note:** This will automatically setup a `.venv` (virtual environment) inside the `agent` directory.  \n\u003e\n\u003e To activate the virtual environment manually, you can run:\n\u003e ```bash\n\u003e source agent/.venv/bin/activate\n\u003e ```\n\n3. Set up your OpenAI API key:\n```bash\nexport OPENAI_API_KEY=\"your-openai-api-key-here\"\n```\n\n4. Start the development server:\n```bash\n# Using pnpm\npnpm dev\n\n# Using npm\nnpm run dev\n\n# Using yarn\nyarn dev\n\n# Using bun\nbun run dev\n```\n\nThis will start both the UI and agent servers concurrently.\n\n## Getting Started with the Canvas\n\nOnce the application is running, you can:\n\n1. **Create Cards**: Use the \"New Item\" button or ask the AI to create cards\n   - \"Create a new project\"\n   - \"Add an entity and a note\"\n   - \"Create a chart with sample metrics\"\n\n2. **Edit Cards**: Click on any field to edit directly, or ask the AI\n   - \"Set the project field1 to 'Q1 Planning'\"\n   - \"Add a checklist item 'Review budget'\"\n   - \"Update the chart metrics\"\n\n3. **Execute Plans**: Give the AI multi-step instructions\n   - \"Create 3 projects with different priorities and add 2 checklist items to each\"\n   - The AI will create a plan and execute it step by step with visual progress\n\n4. **View JSON**: Toggle between the visual canvas and JSON view using the button at the bottom\n\n## Available Scripts\nThe following scripts can also be run using your preferred package manager:\n- `dev` - Starts both UI and agent servers in development mode\n- `dev:debug` - Starts development servers with debug logging enabled\n- `dev:ui` - Starts only the Next.js UI server\n- `dev:agent` - Starts only the LlamaIndex agent server\n- `install:agent` - Installs Python dependencies for the agent\n- `build` - Builds the Next.js application for production\n- `start` - Starts the production server\n- `lint` - Runs ESLint for code linting\n\n## Architecture Overview\n\n```mermaid\ngraph TB\n    subgraph \"Frontend (Next.js)\"\n        UI[Canvas UI\u003cbr/\u003epage.tsx]\n        Actions[Frontend Actions\u003cbr/\u003euseCopilotAction]\n        State[State Management\u003cbr/\u003euseCoAgent]\n        Chat[CopilotChat]\n    end\n    \n    subgraph \"Backend (Python)\"\n        Agent[LlamaIndex Agent\u003cbr/\u003eagent.py]\n        Tools[Backend Tools\u003cbr/\u003e- set_plan\u003cbr/\u003e- update_plan_progress\u003cbr/\u003e- complete_plan]\n        AgentState[Workflow Context\u003cbr/\u003eState Management]\n        Model[LLM\u003cbr/\u003eGPT-4o]\n    end\n    \n    subgraph \"Communication\"\n        Runtime[CopilotKit Runtime\u003cbr/\u003e:9000]\n    end\n    \n    UI \u003c--\u003e State\n    State \u003c--\u003e Runtime\n    Chat \u003c--\u003e Runtime\n    Actions --\u003e Runtime\n    Runtime \u003c--\u003e Agent\n    Agent --\u003e Tools\n    Agent --\u003e AgentState\n    Agent --\u003e Model\n    \n    style UI fill:#e1f5fe\n    style Agent fill:#fff3e0\n    style Runtime fill:#f3e5f5\n    \n    click UI \"https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/src/app/page.tsx\"\n    click Agent \"https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/agent/agent/agent.py\"\n```\n\n### Frontend (Next.js + CopilotKit)\nThe main UI component is in [`src/app/page.tsx`](https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/src/app/page.tsx). It includes:\n- **Canvas Management**: Visual grid of cards with create, read, update, and delete operations\n- **State Synchronization**: Uses `useCoAgent` hook for real-time state sync with the agent\n- **Frontend Actions**: Exposed as tools to the AI agent via `useCopilotAction`\n- **Plan Visualization**: Shows multi-step plan execution with progress indicators\n- **HITL (Tool-based)**: Uses `useCopilotAction` with `renderAndWaitForResponse` for disambiguation prompts (e.g., choosing an item or card type)\n\n### Backend (LlamaIndex Agent)\nThe agent logic is in [`agent/agent/agent.py`](https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/agent/agent/agent.py). It features:\n- **Workflow Context**: Uses LlamaIndex's Context for state management and event streaming\n- **Tool Integration**: Backend tools for planning, frontend tools integration via CopilotKit\n- **Strict Grounding**: Enforces data consistency by always using shared state as truth\n- **Loop Control**: Prevents infinite loops and redundant operations\n- **Planning System**: Can create and execute multi-step plans with status tracking\n- **FastAPI Router**: Uses `get_ag_ui_workflow_router` for seamless integration\n\n### Card Field Schema\nEach card type has specific fields defined in the agent:\n- **Project**: field1 (text), field2 (select), field3 (date), field4 (checklist)\n- **Entity**: field1 (text), field2 (select), field3 (tags), field3_options (available tags)\n- **Note**: field1 (textarea content)\n- **Chart**: field1 (array of metrics with label and value 0-100)\n\n### Data Flow\n\n```mermaid\nsequenceDiagram\n    participant User\n    participant UI as Canvas UI\n    participant CK as CopilotKit\n    participant Agent as LlamaIndex Agent\n    participant Tools\n    \n    User-\u003e\u003eUI: Interact with canvas\n    UI-\u003e\u003eCK: Update state via useCoAgent\n    CK-\u003e\u003eAgent: Send state + message\n    Agent-\u003e\u003eAgent: Process with GPT-4o\n    Agent-\u003e\u003eTools: Execute tools\n    Tools--\u003e\u003eAgent: Return results\n    Agent-\u003e\u003eCK: Return updated state\n    CK-\u003e\u003eUI: Sync state changes\n    UI-\u003e\u003eUser: Display updates\n    \n    Note over Agent: Maintains ground truth\n    Note over UI,CK: Real-time bidirectional sync\n```\n\n## Customization Guide\n\n### Adding New Card Types\n1. Define the data schema in [`src/lib/canvas/types.ts`](https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/src/lib/canvas/types.ts)\n2. Add the card type to the `CardType` union\n3. Create rendering logic in [`src/components/canvas/CardRenderer.tsx`](https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/src/components/canvas/CardRenderer.tsx)\n4. Update the agent's field schema in [`agent/agent/agent.py`](https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/agent/agent/agent.py)\n5. Add corresponding frontend actions in [`src/app/page.tsx`](https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/src/app/page.tsx)\n\n### Modifying Existing Cards\n- Field definitions are in the agent's FIELD_SCHEMA constant\n- UI components are in [`CardRenderer.tsx`](https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/src/components/canvas/CardRenderer.tsx)\n- Frontend actions follow the pattern: `set[Type]Field[Number]`\n\n### Styling\n- Global styles: [`src/app/globals.css`](https://github.com/CopilotKit/canvas-with-llamaindex/blob/main/src/app/globals.css)\n- Component styles use Tailwind CSS with shadcn/ui components\n- Theme colors can be modified via CSS custom properties\n\n## 📚 Documentation\n\n- [LlamaIndex Documentation](https://docs.llamaindex.com/introduction) - Learn more about LlamaIndex and its features\n- [CopilotKit Documentation](https://docs.copilotkit.ai) - Explore CopilotKit's capabilities\n- [Next.js Documentation](https://nextjs.org/docs) - Learn about Next.js features and API\n\n## Contributing\n\nFeel free to submit issues and enhancement requests! This starter is designed to be easily extensible.\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## Troubleshooting\n\n### Agent Connection Issues\nIf you see \"I'm having trouble connecting to my tools\", make sure:\n1. The LlamaIndex agent is running on port 9000 (check terminal output)\n2. Your OpenAI API key is set correctly as an environment variable\n3. Both servers started successfully (UI and agent)\n\n### Port Already in Use\nIf you see \"[Errno 48] Address already in use\":\n1. The agent might still be running from a previous session\n2. Kill the process using the port: `lsof -ti:9000 | xargs kill -9`\n3. For the UI port: `lsof -ti:3000 | xargs kill -9`\n\n### State Synchronization Issues\nIf the canvas and AI seem out of sync:\n1. Check the browser console for errors\n2. Ensure all frontend actions are properly registered\n3. Verify the agent is using the latest shared state (not cached values)\n\n### Python Dependencies\nIf you encounter Python import errors:\n```bash\ncd agent\nuv sync\n```\n\n### Dependency Conflicts\nIf issues persist, recreate the virtual environment:\n```bash\ncd agent\nrm -rf .venv\nuv venv\nuv sync\n```\n---\n\n\u003e [!IMPORTANT]\n\u003e Some features are still under active development and may not yet work as expected. If you encounter a problem using this template, please [report an issue](https://github.com/CopilotKit/canvas-with-llamaindex/issues/new/choose) to this repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcopilotkit%2Fcanvas-with-llamaindex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcopilotkit%2Fcanvas-with-llamaindex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcopilotkit%2Fcanvas-with-llamaindex/lists"}