{"id":49866237,"url":"https://github.com/lenml/canvas-chat","last_synced_at":"2026-05-15T03:01:41.089Z","repository":{"id":354956976,"uuid":"1225256153","full_name":"lenML/canvas-chat","owner":"lenML","description":"infinite canvs with chat","archived":false,"fork":false,"pushed_at":"2026-05-01T05:38:36.000Z","size":260,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-01T07:17:27.698Z","etag":null,"topics":["agent","canvas","chatgpt","gpt-image-2","image-generation","openai"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lenML.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-30T05:21:44.000Z","updated_at":"2026-05-01T05:38:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lenML/canvas-chat","commit_stats":null,"previous_names":["lenml/canvas-chat"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/lenML/canvas-chat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenML%2Fcanvas-chat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenML%2Fcanvas-chat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenML%2Fcanvas-chat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenML%2Fcanvas-chat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lenML","download_url":"https://codeload.github.com/lenML/canvas-chat/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenML%2Fcanvas-chat/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33051875,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-15T02:00:06.351Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","canvas","chatgpt","gpt-image-2","image-generation","openai"],"created_at":"2026-05-15T03:01:10.296Z","updated_at":"2026-05-15T03:01:41.076Z","avatar_url":"https://github.com/lenML.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tldraw agent\n\nThis starter kit demonstrates how to build an agent that can manipulate the [tldraw](https://github.com/tldraw/tldraw) canvas.\n\nA chat panel on the right side of the screen lets users communicate with the agent, add context, and see chat history.\n\n## Environment setup\n\nCreate a `.dev.vars` file in the root directory and add API keys for any model providers you want to use.\n\n```\nANTHROPIC_API_KEY=your_anthropic_api_key_here\nGOOGLE_API_KEY=your_google_api_key_here\nOPENAI_API_KEY=your_openai_api_key_here\n```\n\nWe recommend using Anthropic for best results. Get your API key from the [Anthropic dashboard](https://console.anthropic.com/settings/keys).\n\n## Local development\n\nInstall dependencies with `yarn` or `npm install`.\n\nRun the development server with `yarn dev` or `npm run dev`.\n\nOpen `http://localhost:5173/` in your browser to see the app.\n\n## Agent overview\n\nWith its default configuration, the agent can perform the following actions:\n\n- Create, update and delete shapes.\n- Draw freehand pen strokes.\n- Use higher-level operations on multiple shapes at once: Rotate, resize, align, distribute, stack and reorder shapes.\n- Write out its thinking and send messages to the user.\n- Keep track of its task by writing and updating a todo list.\n- Move its viewport to look at different parts of the canvas.\n- Count shapes matching a given expression.\n- Schedule further work and reviews to be carried out in follow-up requests.\n- Call example external APIs: Looking up country information.\n\nTo make decisions on what to do, we send the agent information from various sources:\n\n- The user's message.\n- The user's current selection of shapes.\n- What the user can currently see on their screen.\n- Any additional context that the user has provided, such as specific shapes or a particular position or area on the canvas.\n- Actions the user has recently taken.\n- A screenshot of the agent's current view of the canvas.\n- A simplified format of all shapes within the agent's viewport.\n- Information on clusters of shapes outside the agent's viewport.\n- The history of the current session, including the user's messages and all the agent's actions.\n- Lints identifying potential issues with shapes on the canvas.\n\n## Use the agent programmatically\n\nAside from using the chat panel UI, you can also prompt the agent programmatically.\n\nThe simplest way is to call the `prompt()` method to start an agentic loop. The agent continues until it finishes the task.\n\n```ts\n// Inside a component wrapped by TldrawAgentAppProvider\nconst agent = useAgent()\nagent.prompt('Draw a cat')\n```\n\nYou can specify further details about the request as an `AgentInput` object:\n\n```ts\nagent.prompt({\n\tmessage: 'Draw a cat in this area',\n\tbounds: { x: 0, y: 0, w: 300, h: 400 },\n})\n```\n\nThe `TldrawAgent` class has additional methods:\n\n- `agent.cancel()` - Cancel the agent's current task.\n- `agent.reset()` - Reset the agent's chat and memory.\n- `agent.request(input)` - Send a single request to the agent and handle its response _without_ entering into an agentic loop.\n\n## Architecture overview\n\nThe agent starter is organized into three main areas:\n\n- **`client/`** - React components, agent logic, and utils that run in the browser\n- **`worker/`** - Cloudflare Worker that handles model requests and prompt building\n- **`shared/`** - Types, schemas, and utilities shared between client and worker\n\n## Customize the agent\n\nThe agent's behavior is defined in `client/modes/AgentModeDefinitions.ts`. The `AGENT_MODE_DEFINITIONS` array contains mode definitions. Each mode has two arrays:\n\n- `parts` determine what the agent can **see**.\n- `actions` determine what the agent can **do**.\n\nAdd, edit or remove an entry in either array to change what the agent can see or do in a given mode.\n\n### Mode system\n\nThe agent uses a **mode system** to control what parts and actions it has access to at any given time. Modes are defined in `client/modes/AgentModeDefinitions.ts`.\n\nThe default `working` mode includes all standard capabilities. You can create additional modes with different subsets of parts and actions.\n\nModes can be transitioned between over the course of a prompt depending on the behavior you desire. Call `agent.mode.setMode(modeType)` to change modes. To control the lifecycles of different modes, you can optionally implement any desired mode lifecycle hooks in `client/modes/AgentModeChart.ts`. You have access to:\n\n- `onEnter(agent, fromMode)` - runs when you enter a mode\n- `onExit(agent, toMode)` - runs when you exit a mode\n- `onPromptStart(agent, request)` - runs when a prompt commences, either because a user has prompted it or because it has entered another step in its agentic loop\n- `onPromptEnd(agent, request)` - runs when a prompt ends\n- `onPromptCancel(agent, request)` - runs when a prompt is canceled\n\n## Change what the agent can see\n\n**Change what the agent can see by adding, editing or removing a prompt part.**\n\nPrompt parts assemble and build the prompt that we give to the model, with each util adding a different piece of information. This includes the user's message, the model name, the system prompt, chat history and more.\n\nThis example shows how to let the model see what the current time is.\n\nFirst, define a prompt part type in `shared/schema/PromptPartDefinitions.ts`:\n\n```ts\nexport interface TimePart extends BasePromptPart\u003c'time'\u003e {\n\ttime: string\n}\n```\n\nNext, create a prompt part util in `client/parts/`:\n\n```ts\nexport const TimePartUtil = registerPromptPartUtil(\n\tclass TimePartUtil extends PromptPartUtil\u003cTimePart\u003e {\n\t\tstatic override type = 'time' as const\n\n\t\toverride getPart(): TimePart {\n\t\t\treturn {\n\t\t\t\ttype: 'time',\n\t\t\t\ttime: new Date().toLocaleTimeString(),\n\t\t\t}\n\t\t}\n\t}\n)\n```\n\nThe `getPart` method gather any data needed to construct the prompt. It can take `(request: AgentRequest, helpers: AgentHelpers)` parameters for access to the current request and helper methods.\n\nThen, back in `shared/schema/PromptPartDefinition.ts`, create the definition for that prompt part.\n\n```ts\nexport const TimePartDefinition: PromptPartDefinition\u003cTimePart\u003e = {\n\ttype: 'time',\n\tpriority: -100,\n\tbuildContent({ time }: TimePart) {\n\t\treturn [`The user's current time is: ${time}`]\n\t},\n}\n```\n\nThe prompt part definition is used by the worker to turn prompt parts into messages sent to the model. Override `priority` to control what order the part should be added in the messages. Override `buildContent` to control how the data is turned into a message for the model.\n\nThere are other methods available on the `PromptPartDefinition` interface that you can override for more granular control.\n\n- `getModelName` - Determine which AI model to use.\n- `buildMessages` - Manually override how prompt messages are constructed from the prompt part.\n\n**Enable the prompt part**\n\nTo enable the prompt part, import its util in `client/modes/AgentModeDefinitions.ts` and add its type to a mode's `parts` array. It's important to make sure you import it here and use its `type` field, instead of using the type string literal. This is to ensure the util properly self-registers.\n\n```ts\nimport { TimePartUtil } from '../parts/TimePartUtil'\n\n// Then in the mode definition:\nparts: [\n\t// ... other parts\n\tTimePartUtil.type,\n]\n```\n\n## Change what the agent can do\n\n**Change what the agent can do by adding, editing or removing an agent action.**\n\nAgent action utils define the actions the agent can perform. Each `AgentActionUtil` adds a different capability.\n\nThis example shows how to allow the agent to clear the screen.\n\nFirst, define an agent action schema in `shared/schema/AgentActionSchemas.ts`:\n\n```ts\nexport const ClearAction = z\n\t// All agent actions must have a _type field\n\t// The underscore encourages the model to put this field first\n\t.object({\n\t\t_type: z.literal('clear'),\n\t})\n\t// A title and description tell the model what the action does\n\t.meta({\n\t\ttitle: 'Clear',\n\t\tdescription: 'The agent deletes all shapes on the canvas.',\n\t})\n\n// Infer the action's type\nexport type ClearAction = z.infer\u003ctypeof ClearAction\u003e\n```\n\nThen, create an agent action util in `client/actions/`:\n\n```ts\nexport const ClearActionUtil = registerActionUtil(\n\tclass ClearActionUtil extends AgentActionUtil\u003cClearAction\u003e {\n\t\tstatic override type = 'clear' as const\n\n\t\toverride applyAction(action: Streaming\u003cClearAction\u003e) {\n\t\t\t// Don't do anything until the action has finished streaming\n\t\t\tif (!action.complete) return\n\n\t\t\t// Delete all shapes on the page\n\t\t\tconst { editor } = this\n\t\t\tconst shapes = editor.getCurrentPageShapes()\n\t\t\teditor.deleteShapes(shapes)\n\t\t}\n\t}\n)\n```\n\nThe `applyAction` method executes the action. It can take a second `helpers: AgentHelpers` parameter for access to helper methods.\n\nOverride these methods on `AgentActionUtil` for more control:\n\n- `getInfo` - Determine how the action gets displayed in the chat panel UI.\n- `savesToHistory` - Control whether actions get saved to chat history or not.\n- `sanitizeAction` - Sanitize the action before saving it to history and applying it. More details on [sanitization](#sanitize-data-received-from-the-model) below.\n\n**Enable the agent action part**\n\nTo enable the agent action, import its util in `client/modes/AgentModeDefinitions.ts` and add its type to a mode's `actions` array.\n\n```ts\nimport { ClearActionUtil } from '../actions/ClearActionUtil'\n\n// Then in the mode definition:\nactions: [\n\t// ... other actions\n\tClearActionUtil.type,\n]\n```\n\n## Change how actions appear in chat history\n\nConfigure the icon and description of an action in the chat panel using the `getInfo()` method.\n\n```ts\noverride getInfo() {\n\treturn {\n\t\ticon: 'trash' as const,\n\t\tdescription: 'Cleared the canvas',\n\t}\n}\n```\n\nYou can make an action collapsible by adding a `summary` property.\n\n```ts\noverride getInfo() {\n\treturn {\n\t\tsummary: 'Cleared the canvas',\n\t\tdescription: 'After much consideration, the agent decided to clear the canvas',\n\t}\n}\n```\n\nTo customize an action's appearance via CSS, you can define style for the `agent-action-type-{TYPE}` class where `{TYPE}` is the type of the action.\n\n```css\n.agent-action-type-clear {\n\tcolor: red;\n}\n```\n\n## Managers\n\nManagers are classes that encapsulate specific concerns and extend the functionality of `TldrawAgent` or `TldrawAgentApp`. Each manager handles a single responsibility—like chat history, model selection, or context management—and exposes methods to interact with that state.\n\nManagers are available as properties on the agent instance (e.g., `agent.chat`, `agent.modelName`, `agent.context`). To create a custom manager, extend `BaseAgentManager` or `BaseAgentAppManager` and add it to the agent in `client/agent/TldrawAgent.ts`.\n\n## Registering `PromptPartUtil`s and `AgentActionUtil`s\n\nUtils use a **self-registration pattern**. When you create a new `PromptPartUtil` or `AgentActionUtil`, wrap it with a registration function:\n\n```ts\nexport const MyPartUtil = registerPromptPartUtil(\n\tclass MyPartUtil extends PromptPartUtil\u003cMyPart\u003e {\n\t\t// ...\n\t}\n)\n```\n\nThis pattern ensures utils are discovered automatically when their modules are imported in `AgentModeDefinitions.ts`.\n\n### Mode-scoped actions\n\nDifferent modes can implement actions with the same `_type`. This allows modes to have different behavior for the same action type without requiring globally unique action names.\n\nFor example, a \"team-member\" mode and a \"solo\" mode might both have a `mark-task-done` action, but with different implementations. The system automatically resolves the correct `AgentActionUtil` and schema based on the current mode.\n\n**Registering a mode-specific action util:**\n\nUse the `forModes` option when registering a util:\n\n```ts\n// client/actions/MarkSoloTaskDoneActionUtil.ts\n// Default implementation (used when no mode-specific binding exists)\nexport const MarkSoloTaskDoneActionUtil = registerActionUtil(\n\tclass MarkSoloTaskDoneActionUtil extends AgentActionUtil\u003cMarkSoloTaskDoneAction\u003e {\n\t\tstatic override type = 'mark-task-done' as const\n\t\toverride applyAction(action: Streaming\u003cMarkSoloTaskDoneAction\u003e) {\n\t\t\t// Default implementation\n\t\t}\n\t}\n)\n\n// client/actions/MarkTeamMemberTaskDoneActionUtil.ts\n// Mode-specific implementation for \"drone\" mode\nexport const MarkTeamMemberTaskDoneActionUtil = registerActionUtil(\n\tclass MarkTeamMemberTaskDoneActionUtil extends AgentActionUtil\u003cMarkTeamMemberTaskDoneAction\u003e {\n\t\tstatic override type = 'mark-task-done' as const // Same type as default\n\t\toverride applyAction(action: Streaming\u003cMarkTeamMemberTaskDoneAction\u003e) {\n\t\t\t// Team member-specific implementation\n\t\t}\n\t},\n\t{ forModes: ['team-member'] }\n)\n```\n\n**Registering a mode-specific schema:**\n\nIf a mode needs a different schema for an action, register the schema with `forModes`:\n\n```ts\n// shared/schema/AgentActionSchemas.ts\n\n// Default schema\nexport const MarkSoloTaskDoneAction = z\n\t.object({\n\t\t_type: z.literal('mark-task-done'),\n\t\ttaskId: z.string(),\n\t})\n\t.meta({ title: 'Mark Task Done', description: 'Mark a task as complete.' })\n\n// Mode-specific schema with additional fields\nexport const MarkTeamMemberTaskDoneAction = z\n\t.object({\n\t\t_type: z.literal('mark-task-done'),\n\t\ttaskId: z.string(),\n\t\tteamId: z.string(), // Extra field for this mode\n\t})\n\t.meta({ title: 'Mark Task Done', description: 'Mark a task as complete with notes.' })\n\n// Register the mode-specific schema\nregisterActionSchema('mark-task-done', MarkTeamMemberTaskDoneAction, { forModes: ['team-member'] })\n```\n\nDefault schemas are auto-registered when exported from `AgentActionSchemas.ts`. Call `registerActionSchema` explicitly only for mode-specific schemas.\n\nThe system maintains two registries (default and mode-specific) and resolves the correct util/schema based on the current mode, falling back to the default when no mode-specific binding exists.\n\n## Schedule further work\n\nLet the agent work over multiple turns by scheduling further work using the `schedule` method.\n\nThis example shows how to schedule an extra step for adding detail to the canvas.\n\n```ts\noverride applyAction(action: Streaming\u003cAddDetailAction\u003e) {\n\tif (!action.complete) return\n\tthis.agent.schedule('Add more detail to the canvas.')\n}\n```\n\nAs with the `prompt` method, you can specify further details about the request.\n\n```ts\nagent.schedule({\n\tmessage: 'Add more detail in this area.',\n\tbounds: { x: 0, y: 0, w: 100, h: 100 },\n})\n```\n\nSchedule multiple items by calling the `schedule` method more than once.\n\n```ts\nagent.schedule('Add more detail to the canvas.')\nagent.schedule('Check for spelling mistakes.')\n```\n\nIf you want to interrupt the agent with a new prompt, instead of waiting until the current prompt ends, you can use the agent's `interrupt` method. `interrupt` also lets you specify a mode to transition into.\n\nThis example shows how one might use the `interrupt` method to allow the agent to decide to enter a new mode called `'reviewing'` in order to review some work.\n\n```ts\noverride applyAction(action: Streaming\u003cEnterReviewingModeAction\u003e){\n\tif (!action.complete) return\n\tthis.agent.interrupt({\n\t\tmode: 'reviewing',\n\t\tinput: {\n\t\t\tmessage: 'Review the new area thoroughly for any mistakes',\n\t\t\tbounds: action.bounds\n\t\t}\n\t})\n}\n```\n\nUse this for things like switching modes, or for programatically telling it to correct a mistake it's made.\n\n## Retrieve data from an external API\n\nTo retrieve information from an external API, fetch the data within `applyAction` and schedule a follow-up request with the data.\n\n```ts\noverride async applyAction(action: Streaming\u003cCountryInfoAction\u003e) {\n\tif (!action.complete) return\n\n\t// Fetch from the external API\n\tconst data = await fetchCountryInfo(action.code)\n\n\t// Schedule a follow-up request with the data\n\tthis.agent.schedule({ data: [data] })\n}\n```\n\n## Sanitize data received from the model\n\nThe model can make mistakes. Sometimes this is due to hallucinations, sometimes because the canvas changed since the model last saw it. Either way, an incoming action might contain invalid data.\n\nTo correct mistakes, apply fixes in the `sanitizeAction` method. The system runs these before applying the action to the editor or saving it to chat history.\n\nFor example, use `ensureShapeIdExists` to verify that a shape ID from the model refers to an existing shape.\n\n```ts\noverride sanitizeAction(action: Streaming\u003cDeleteAction\u003e, helpers: AgentHelpers) {\n\tif (!action.complete) return action\n\n\t// Ensure the shape ID refers to an existing shape\n\taction.shapeId = helpers.ensureShapeIdExists(action.shapeId)\n\n\t// If the shape ID doesn't refer to an existing shape, cancel the action\n\tif (!action.shapeId) return null\n\n\treturn action\n}\n```\n\n`AgentHelpers` provides these sanitization helpers:\n\n- `ensureShapeIdExists` - Ensure that a shape ID refers to a real shape. Useful for interacting with existing shapes.\n- `ensureShapeIdsExist` - Ensure that multiple shape IDs refer to real shapes. Useful for bulk operations.\n- `ensureShapeIdIsUnique` - Ensure that a shape ID is unique. Useful for creating new shapes.\n- `ensureValueIsVec`, `ensureValueIsNumber`, etc - Useful for more complex actions where the model is more likely to make mistakes.\n\n## Send positions to and from the model\n\nBy default, every position sent to the model is offset by the starting position of the current chat.\n\nTo apply this offset to a position sent to the model, use the `applyOffsetToVec` method.\n\n```ts\noverride getPart(request: AgentRequest, helpers: AgentHelpers): ViewportCenterPart {\n\tif (!this.editor) return { part: 'user-viewport-center', center: null, }\n\n\t// Get the center of the user's viewport\n\tconst viewportCenter = this.editor.getViewportBounds().center\n\n\t// Apply the chat's offset to the vector\n\tconst offsetViewportCenter = helpers.applyOffsetToVec(viewportCenter)\n\n\t// Return the prompt part\n\treturn {\n\t\tpart: 'user-viewport-center',\n\t\tcenter: offsetViewportCenter,\n\t}\n}\n```\n\nTo remove the offset from a position received from the model, use the `removeOffsetFromVec` method.\n\n```ts\noverride applyAction(action: Streaming\u003cMoveAction\u003e, helpers: AgentHelpers) {\n\tif (!action.complete) return\n\n\t// Remove the offset from the position\n\tconst position = helpers.removeOffsetFromVec({ x: action.x, y: action.y })\n\n\t// Do something with the position...\n}\n```\n\nBox-level helpers for working with bounds:\n\n- `applyOffsetToBox` / `removeOffsetFromBox` - Apply or remove offset from a `{ x, y, w, h }` box.\n- `applyOffsetToShapePartial` / `removeOffsetFromShapePartial` - Apply or remove offset from a partial shape.\n\nRound numbers before sending them to the model. To restore the original number later, use `roundAndSaveNumber` and `unroundAndRestoreNumber`.\n\n```ts\n// In `getPart`...\nconst roundedX = helpers.roundAndSaveNumber(x, 'my_key_x')\nconst roundedY = helpers.roundAndSaveNumber(y, 'my_key_y')\n\n// In `applyAction`...\nconst unroundedX = helpers.unroundAndRestoreNumber(x, 'my_key_x')\nconst unroundedY = helpers.unroundAndRestoreNumber(y, 'my_key_y')\n```\n\nTo round all the numbers on a shape, use the `roundShape` and `unroundShape` methods. See the [shapes](#send-shapes-to-the-model) section below for more details on sending shapes to the model.\n\n```ts\n// In `getPart`...\nconst roundedShape = helpers.roundShape(shape)\n\n// In `applyAction`...\nconst unroundedShape = helpers.unroundShape(roundedShape)\n```\n\nAdditional rounding helpers:\n\n- `roundBox` - Round the coordinates and dimensions of a box.\n\n## Send shapes to the model\n\nThe agent converts tldraw shapes to simplified formats to improve model understanding and performance.\n\nThree main formats:\n\n- `BlurryShape` - Format for shapes within the agent's viewport. Contains bounds, id, type, and text. The \"blurry\" name indicates the agent can't make out shape details—it provides an overview of what the agent sees.\n- `FocusedShape` - Format for shapes the agent is focusing on, such as those you've manually added to its context. Contains most shape properties: color, fill, alignment, and shape-specific information. The \"focused\" name indicates these are shapes the agent is directly examining.\n  - This is also the format that the model outputs when creating shapes.\n- `PeripheralShapeCluster` - Format for shapes outside the agent's viewport. Groups nearby shapes into clusters with bounds and shape count. The least detailed format—gives the model awareness of shapes elsewhere on the page.\n\nUse conversion functions in `shared/format/` to send shapes in these formats, such as `convertTldrawShapeToFocusedShape`.\n\nThis example picks one random shape on the canvas and sends it to the model in the Focused format.\n\n```ts\noverride getPart(request: AgentRequest, helpers: AgentHelpers): RandomShapePart {\n\tif (!this.editor) return { type: 'random-shape', shape: null}\n\tconst { editor } = this\n\n\t// Get a random shape\n\tconst shapes = editor.getCurrentPageShapes()\n\tconst randomShape = shapes[Math.floor(Math.random() * shapes.length)]\n\n\t// Convert the shape to the Focused format\n\tconst focusedShape = convertTldrawShapeToFocusedShape(randomShape, editor)\n\n\t// Normalize the shape's position\n\tconst offsetShape = helpers.applyOffsetToShape(focusedShape)\n\tconst roundedShape = helpers.roundShape(offsetShape)\n\n\treturn {\n\t\ttype: 'random-shape',\n\t\tshape: roundedShape,\n\t}\n}\n```\n\n## Change the system prompt\n\nThe system prompt lives in `worker/prompt/buildSystemPrompt.ts`. Edit the sections in `worker/prompt/sections/` to change the system prompt.\n\nThe system prompt is rebuilt for each step in the agentic loop depending on which actions and parts are available in the agent's current mode. If you add new actions or parts, you can give the model more detailed instructions for how to use them in `worker/prompt/sections/rules-section.ts`.\n\nThe schema showing the actions the agent can output is also automatically added to the system prompt.\n\n## Change to a different model\n\nSet an agent's model using the `setModelName` method on the `modelName` manager.\n\n```ts\nagent.modelName.setModelName('gemini-3-flash-preview')\n```\n\nTo change the logic for deciding which model to use for a request, you can edit `ModelNamePartUtil`.\n\n## Support a different model\n\nAdd the model's definition to `AGENT_MODEL_DEFINITIONS` in `shared/models.ts`.\n\n```ts\n'claude-sonnet-4-5': {\n\tname: 'claude-sonnet-4-5',\n\tid: 'claude-sonnet-4-5',\n\tprovider: 'anthropic',\n}\n```\n\nAdd extra setup or configuration for your provider in `worker/do/AgentService.ts`.\n\n## Support custom shapes\n\nIf your app includes [custom shapes](https://tldraw.dev/docs/shapes#Custom-shapes-1), the agent can see, move, delete, resize, rotate, and arrange them with no extra setup. However, you might also want to let the agent create and edit them, and read their custom properties.\n\nTo support custom shapes, you have two main options:\n\n1. Add an action that lets the agent create your custom shape.\n   See the [Let the agent create custom shapes with an action](#let-the-agent-create-custom-shapes-with-an-action) section below.\n2. Add your custom shape to the schema so that the agent read, edit and create it like any other shape.\n   See the [Add your custom shape to the schema](#add-your-custom-shape-to-the-schema) section below.\n\n### Let the agent create a custom shape with an action\n\nFor partial support, let the agent create a custom shape with an [agent action](#change-what-the-agent-can-do). This example creates a custom \"sticker\" shape:\n\n```ts\n// In shared/schema/AgentActionSchemas.ts\nexport const StickerAction = z\n\t.object({\n\t\t_type: z.literal('sticker'),\n\t\tstickerType: z.enum(['heart', 'star']),\n\t\tx: z.number(),\n\t\ty: z.number(),\n\t})\n\t.meta({\n\t\ttitle: 'Sticker',\n\t\tdescription: 'Add a sticker to the canvas.',\n\t})\n\nexport type StickerAction = z.infer\u003ctypeof StickerAction\u003e\n```\n\nCreate an action util to define how the action applies to the canvas:\n\n```ts\n// client/actions/StickerActionUtil.ts\nexport const StickerActionUtil = registerActionUtil(\n\tclass StickerActionUtil extends AgentActionUtil\u003cStickerAction\u003e {\n\t\tstatic override type = 'sticker' as const\n\n\t\t// How to display the action in chat history\n\t\toverride getInfo(action: Streaming\u003cStickerAction\u003e) {\n\t\t\treturn {\n\t\t\t\ticon: 'pencil' as const,\n\t\t\t\tdescription: 'Added a sticker',\n\t\t\t}\n\t\t}\n\n\t\t// Execute the action\n\t\toverride applyAction(action: Streaming\u003cStickerAction\u003e, helpers: AgentHelpers) {\n\t\t\tif (!action.complete) return\n\n\t\t\t// Normalize the position\n\t\t\tconst position = helpers.removeOffsetFromVec({ x: action.x, y: action.y })\n\n\t\t\t// Create the custom shape\n\t\t\tthis.editor.createShape({\n\t\t\t\ttype: 'sticker',\n\t\t\t\tid: createShapeId(),\n\t\t\t\tx: position.x,\n\t\t\t\ty: position.y,\n\t\t\t\tprops: { stickerType: action.stickerType },\n\t\t\t})\n\t\t}\n\t}\n)\n```\n\n### Add a custom shape to the schema\n\nTo let the agent see the custom properties of your custom shape, add it to the schema in `shared/format/FocusedShape.ts`\n\nFor example, here's a schema for a custom sticker shape.\n\n```ts\nconst FocusedStickerShape = z\n\t.object({\n\t\t// Required properties\n\t\t_type: z.literal('sticker'),\n\t\tnote: z.string(),\n\t\tshapeId: z.string(),\n\n\t\t// Custom properties\n\t\tstickerType: z.enum(['heart', 'star']),\n\t\tx: z.number(),\n\t\ty: z.number(),\n\t})\n\t.meta({\n\t\t// Information about the shape to give to the agent\n\t\ttitle: 'Sticker Shape',\n\t\tdescription:\n\t\t\t'A sticker shape is a small symbol stamped onto the canvas. There are two types of stickers: heart and star.',\n\t})\n```\n\nThe `_type` and `shapeId` properties are required so that the app can identify your shape. The `note` property is also required. The agent uses it to leave notes for itself.\n\nFor optional properties, it's worth considering how the agent should see your custom shape. You might want to leave out some properties and focus on showing the most important ones. It's also best to keep them in alphabetical order for better performance with Gemini models.\n\nEnable your custom shape schema by adding it to the list of `FOCUSED_SHAPES` in the same file to enable it.\n\n```ts\nconst FOCUSED_SHAPES = [\n\tFocusedDrawShape,\n\tFocusedGeoShape,\n\tFocusedLineShape,\n\tFocusedTextShape,\n\tFocusedArrowShape,\n\tFocusedNoteShape,\n\tFocusedUnknownShape,\n\n\t// Our custom shape\n\tFocusedStickerShape,\n] as const\n```\n\nTell the app how to convert your custom shape into the `FocusedShape` format by adding it as a case in `shared/format/convertTldrawShapeToFocusedShape.ts`.\n\n```ts\nexport function convertTldrawShapeToFocusedShape(editor: Editor, shape: TLShape): FocusedShape {\n\tswitch (shape.type) {\n\t\t// ...\n\t\tcase 'sticker':\n\t\t\tconst bounds = getShapeBounds(shape)\n\t\t\treturn {\n\t\t\t\t_type: 'sticker',\n\t\t\t\tnote: (shape.meta.note as string) ?? '',\n\t\t\t\tshapeId: convertTldrawIdToSimpleId(shape.id),\n\t\t\t\tstickerType: shape.props.stickerType,\n\t\t\t\tx: bounds.x,\n\t\t\t\ty: bounds.y,\n\t\t\t}\n\t\t// ...\n\t}\n}\n```\n\nTo allow the agent to edit your custom shape's properties, tell the app how to convert your shape from the `FocusedShape` format that the model outputs to the actual format of your shape.\n\n```ts\nexport function convertFocusedShapeToTldrawShape(\n\teditor: Editor,\n\tfocusedShape: TLShape\n\t{ defaultShape }: { defaultShape: Partial\u003cTLShape\u003e }\n): {\n\tswitch (focusedShape.type) {\n\t\t// ...\n\t\tcase 'sticker':\n\t\t\tconst shapeId = convertSimpleIdToTldrawId(focusedShape.shapeId)\n\t\t\treturn {\n\t\t\t\tshape: {\n\t\t\t\t\tid: shapeId\n\t\t\t\t\tx: focusedShape.x,\n\t\t\t\t\ty: focusedShape.y\n\t\t\t\t\t// ...\n\t\t\t\t\tprops: {\n\t\t\t\t\t\t// ...\n\t\t\t\t\t\tstickerType: focusedShape.stickerType\n\t\t\t\t\t},\n\t\t\t\t\tmeta: {\n\t\t\t\t\t\tnote: focusedShape.note ?? ''\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t// ...\n\t}\n}\n```\n\n## License\n\nThis project is part of the tldraw SDK. It is provided under the [tldraw SDK license](https://github.com/tldraw/tldraw/blob/main/LICENSE.md).\n\nYou can use the tldraw SDK in commercial or non-commercial projects so long as you preserve the \"Made with tldraw\" watermark on the canvas. To remove the watermark, you can purchase a [business license](https://tldraw.dev/pricing). Visit [tldraw.dev](https://tldraw.dev) to learn more.\n\n## Trademarks\n\nCopyright (c) 2025-present tldraw Inc. The tldraw name and logo are trademarks of tldraw. Please see our [trademark guidelines](https://github.com/tldraw/tldraw/blob/main/TRADEMARKS.md) for info on acceptable usage.\n\n## Distributions\n\nYou can find tldraw on npm [here](https://www.npmjs.com/package/@tldraw/tldraw?activeTab=versions).\n\n## Contribution\n\nFound a bug? Please [submit an issue](https://github.com/tldraw/tldraw/issues/new).\n\n## Community\n\nHave questions, comments or feedback? [Join our discord](https://discord.gg/rhsyWMUJxd). For the latest news and release notes, visit [tldraw.dev](https://tldraw.dev).\n\n## Contact\n\nFind us on Twitter/X at [@tldraw](https://twitter.com/tldraw) or email us at [mailto:hello@tldraw.com](hello@tldraw.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flenml%2Fcanvas-chat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flenml%2Fcanvas-chat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flenml%2Fcanvas-chat/lists"}