{"id":39405990,"url":"https://github.com/lukekim/demo","last_synced_at":"2026-01-18T03:30:38.532Z","repository":{"id":173200653,"uuid":"636614985","full_name":"lukekim/demo","owner":"lukekim","description":"Luke's Spice.ai demo app","archived":false,"fork":false,"pushed_at":"2024-12-16T18:49:19.000Z","size":111,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-16T19:51:57.161Z","etag":null,"topics":["ai","data","web3"],"latest_commit_sha":null,"homepage":"https://spice.ai","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lukekim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-05-05T08:50:46.000Z","updated_at":"2024-12-16T18:49:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"e1085106-d473-40c1-bfd5-48cd4ac60b53","html_url":"https://github.com/lukekim/demo","commit_stats":null,"previous_names":["lukekim/demo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lukekim/demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukekim%2Fdemo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukekim%2Fdemo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukekim%2Fdemo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukekim%2Fdemo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lukekim","download_url":"https://codeload.github.com/lukekim/demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukekim%2Fdemo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28528114,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","data","web3"],"created_at":"2026-01-18T03:30:37.784Z","updated_at":"2026-01-18T03:30:38.456Z","avatar_url":"https://github.com/lukekim.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spice.ai Demo App\n\nThis is a [Spice.ai](https://spice.ai/) data and AI app.\n\n## Prerequisites\n\n- [Spice.ai CLI](https://docs.spice.ai/get-started/installation) installed\n- OpenAI API key\n- Hugging Face API token (optional, for LLaMA model)\n- `curl` and `jq` for API calls\n\n## Learn More\n\nTo learn more about Spice.ai, take a look at the following resources:\n\n- [Spice.ai](https://docs.spice.ai) - learn about Spice.ai features, data, and API.\n- [Get started with Spice.ai](https://docs.spice.ai/get-started) - try out the API and make basic queries.\n\nConnect with us on [Discord](https://discord.gg/PUCapX22En) - your feedback is appreciated!\n\n---\n\n## Demo Steps\n\n### Publishing a Spice App in the Cloud\n\n#### Step 1: Forking and Using the Dataset\n\n1. Fork the repository `https://github.com/jeadie/evals` into your GitHub org.\n\n#### Step 2: Creating a New App in the Cloud\n\n1. Log into the [Spice.ai Cloud Platform](https://spice.ai/login) and create a new app called `evals`. The app will start empty.\n2. Connect the app to your repository:\n   - Go to the **App Settings** tab and select **Connect Repository**.\n   - If the repository is not yet linked, follow the prompts to authenticate and link it.\n\n#### Step 3: Deploying the App\n\n1. Set the app to **Public**:\n   - Navigate to the app's settings and toggle the visibility to public.\n2. Redeploy the app:\n   - Click **Redeploy** to load the datasets and configurations from the repository.\n\n#### Step 4: Verifying and Testing\n\n1. Check the datasets in the Spice.ai Cloud:\n   - Verify that the datasets are correctly loaded and accessible.\n2. Test public access:\n   - Log in with a different account to confirm the app is accessible to external users.\n\n---\n\n### Initializing a Local Spice App\n\n1. **Initialize a new local Spice app**\n\n   ```bash\n   mkdir demo\n   cd demo\n   spice init\n   ```\n\n2. **Login to Spice.ai Cloud**\n\n   ```bash\n   spice login\n   ```\n\n3. **Get spicepod from Spicerack**\n   Navigate to [spicerack.org](https://spicerack.org), search for `evals`.\n\n   \u003cimg width=\"1679\" alt=\"image\" src=\"https://github.com/user-attachments/assets/248bc281-fbcd-4312-a724-fc295ee0dc13\" /\u003e\n\nClick on \u003cusername\u003e/evals, click on **Use this app**, and copy the `spice connect` command.\n\n   \u003cimg width=\"1679\" alt=\"image\" src=\"https://github.com/user-attachments/assets/ee044042-4a21-4eae-a571-ca9b5c82b976\" /\u003e\n \n Paste the command into the terminal.\n   Navigate to [spicerack.org](https://spicerack.org), search for `evals`, click on \u003cusername\u003e/evals, click on **Use this app**, and copy the `spice connect` command. Paste the command into the terminal.\n\n```bash\nspice connect \u003cusername\u003e/evals\n```\n\nThe `spicepod.yml` should be updated to:\n\n```yaml\nversion: v1beta1\nkind: Spicepod\nname: demo\n\ndependencies:\n  - Jeadie/evals\n```\n\n5. **Add a model to the spicepod**\n\n   ```yaml\n   models:\n     - name: gpt-4o\n       from: openai:gpt-4o\n       params:\n         openai_api_key: ${ secrets:SPICE_OPENAI_API_KEY }\n   ```\n\n6. **Start spice**\n\n   ```bash\n   spice run\n   ```\n\n7. **Run an eval**\n\n   ```bash\n   curl -XPOST \"http://localhost:8090/v1/evals/taxes\"      -H \"Content-Type: application/json\"      -d '{\n       \"model\": \"gpt-4o\"\n     }' | jq\n   ```\n\n8. **Explore incorrect results**\n\n   ```bash\n   spice sql\n   ```\n\n   ```sql\n   SELECT\n     input,\n     output,\n     actual\n   FROM eval.results\n   WHERE value=0.0 LIMIT 5;\n   ```\n\n---\n\n### Optional: Create an Eval to Use a Smaller Model\n\n1. Track the outputs of all AI model calls:\n\n   ```yaml\n   runtime:\n     task_history:\n       captured_output: truncated\n   ```\n\n2. Define a new view and evaluation:\n\n   ```yaml\n   views:\n     - name: user_queries\n       sql: |\n         SELECT\n           json_get_json(input, 'messages') AS input,\n           json_get_str((captured_output -\u003e 0), 'content') as ideal\n         FROM runtime.task_history\n         WHERE task='ai_completion'\n     - name: latest_eval_runs\n       sql: |\n         SELECT model, MAX(created_at) as latest_run\n            FROM eval.runs\n            GROUP BY model\n     - name: model_stats\n       sql: |\n         SELECT\n           r.model,\n           COUNT(*) as total_queries,\n           SUM(CASE WHEN res.value = 1.0 THEN 1 ELSE 0 END) as correct_answers,\n           AVG(res.value) as accuracy\n         FROM eval.runs r\n         JOIN latest_eval_runs lr ON r.model = lr.model AND r.created_at = lr.latest_run\n         JOIN eval.results res ON res.run_id = r.id\n         GROUP BY r.model\n\n   evals:\n     - name: mimic-user-queries\n       description: |\n         Evaluates how well a model can copy the exact answers already returned to a user. Useful for testing if a smaller/cheaper model is sufficient.\n       dataset: user_queries\n       scorers:\n         - match\n   ```\n\n3. Add a smaller model to the spicepod:\n\n   ```yaml\n   models:\n     - name: llama3\n       from: huggingface:huggingface.co/meta-llama/Llama-3.2-3B-Instruct\n       params:\n         hf_token: ${ secrets:SPICE_HUGGINGFACE_API_KEY }\n\n     - name: gpt-4o # Keep previous model.\n   ```\n\n4. Verify models are loaded:\n\n   ```bash\n   spice models\n   ```\n\n   You should see both models listed:\n\n   ```shell\n   NAME    FROM                                                         STATUS\n   gpt-4o  openai:gpt-4o                                                ready\n   llama3  huggingface:huggingface.co/meta-llama/Llama-3.3-70B-Instruct ready\n   ```\n\n5. Restart the Spice app:\n\n   ```bash\n   spice run\n   ```\n\n6. Test the larger model or run another eval:\n\n   ```bash\n   spice chat\n   ```\n\n7. Run evaluations on both models:\n\n   ```bash\n   # Run eval with GPT-4\n   curl -XPOST \"http://localhost:8090/v1/evals/mimic-user-queries\" \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\"model\": \"gpt-4o\"}' | jq\n\n   # Run eval with LLaMA\n   curl -XPOST \"http://localhost:8090/v1/evals/mimic-user-queries\" \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\"model\": \"llama3\"}' | jq\n   ```\n\n8. Compare model performance:\n\n   ```bash\n   spice sql\n   ```\n\n   ```sql\n   SELECT\n     model,\n     total_queries,\n     correct_answers,\n     ROUND(accuracy * 100, 2) as accuracy_percentage\n   FROM model_stats\n   ORDER BY accuracy_percentage DESC;\n   ```\n\n   This query will show:\n\n   - Total number of queries processed\n   - Number of correct answers\n   - Accuracy percentage as a percentage\n\n   You can use these metrics to decide if the smaller model provides acceptable performance for your use case.\n\n---\n\n### Full Spicepod Configuration\n\nInclude the following `spicepod.yml` for reference:\n\n```yaml\nversion: v1beta1\nkind: Spicepod\nname: demo\n\ndependencies:\n  - Jeadie/evals\n\nruntime:\n  task_history:\n    captured_output: truncated\n\nviews:\n  - name: user_queries\n    sql: |\n      SELECT\n        json_get_json(input, 'messages') AS input,\n        json_get_str((captured_output -\u003e 0), 'content') as ideal\n      FROM runtime.task_history\n      WHERE task='ai_completion'\n\nevals:\n  - name: mimic-user-queries\n    description: |\n      Evaluates how well a model can copy the exact answers already returned to a user. Useful for testing if a smaller/cheaper model is sufficient.\n    dataset: user_queries\n    scorers:\n      - match\n\nmodels:\n  - name: gpt-4o\n    from: openai:gpt-4o\n    params:\n      openai_api_key: ${ secrets:SPICE_OPENAI_API_KEY }\n\n  - name: llama3\n    from: huggingface:huggingface.co/meta-llama/Llama-3.2-3B-Instruct\n    params:\n      hf_token: ${ secrets:SPICE_HUGGINGFACE_API_KEY }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukekim%2Fdemo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukekim%2Fdemo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukekim%2Fdemo/lists"}