{"id":24144498,"url":"https://github.com/metaskills/fast-llama-inference","last_synced_at":"2025-03-01T14:44:21.087Z","repository":{"id":269895289,"uuid":"908741899","full_name":"metaskills/fast-llama-inference","owner":"metaskills","description":"Exploring Accelerated Compound AI Systems with SambaNova \u0026 Llama 3.3-70B","archived":false,"fork":false,"pushed_at":"2025-02-16T00:15:57.000Z","size":490,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-24T03:45:35.765Z","etag":null,"topics":["artificial-intelligence","evaluation","llama3","sambanova","sambanova-cloud","typescript","vercel","vercel-ai-sdk"],"latest_commit_sha":null,"homepage":"https://www.unremarkable.ai/exploring-accelerated-compound-ai-systems-with-sambanova-llama-3-3/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metaskills.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-26T21:04:33.000Z","updated_at":"2025-02-16T00:16:01.000Z","dependencies_parsed_at":"2024-12-27T02:17:19.543Z","dependency_job_id":null,"html_url":"https://github.com/metaskills/fast-llama-inference","commit_stats":null,"previous_names":["metaskills/fast-llama-inference"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metaskills%2Ffast-llama-inference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metaskills%2Ffast-llama-inference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metaskills%2Ffast-llama-inference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metaskills%2Ffast-llama-inference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metaskills","download_url":"https://codeload.github.com/metaskills/fast-llama-inference/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241381521,"owners_count":19953749,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","evaluation","llama3","sambanova","sambanova-cloud","typescript","vercel","vercel-ai-sdk"],"created_at":"2025-01-12T06:11:35.219Z","updated_at":"2025-03-01T14:44:21.080Z","avatar_url":"https://github.com/metaskills.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fast Llama Inference with SambaNova\n\nhttps://www.unremarkable.ai/exploring-accelerated-compound-ai-systems-with-sambanova-llama-3-3/\n\n![Xyz](images/samba-nova.png)\n\n## Setup\n\nMust use 23.6.0 or higher of Node.js with TypeScript support.\n\nRun npm install. \n\n```shell\nnpm install\n```\n\nMake sure you have the following environment variables needed:\n\n- `SAMBANOVA_API_KEY` - Your [SambaNova](https://cloud.sambanova.ai/apis?ref=unremarkable.ai) API Key.\n\nAll demos \u0026 experiments leverage the following tools:\n\n1. Use of [Inquirer.js](https://www.npmjs.com/package/inquirer?ref=unremarkable.ai) with the CLI to prompt for user questions.\n2. The [Vercel AI SDK](https://sdk.vercel.ai?ref=unremarkable.ai) is used to invoke and stream model output to the CLI.\n3. Using the `Meta-Llama-3.3-70B-Instruct` model via hyper fast inference thanks to [SambaNova](https://sambanova.ai?ref=unremarkable.ai).\n\n```shell\nnpm run demo\nnpm run evalVerifyClaims\n```\n\nIf you are testing other providers such as OpenAI or Bedrock you will need to make sure you have supporting API keys in your environment. For example `OPENAI_API_KEY` and standard AWS environment variables.\n\n```shell\nMODEL=bedrock npm run evalVerifyClaims\n```\n\n## Notes\n\nThe bedrock-cross-region.md file contains the following claims which the demo verifies.\n\n```\n✅ Amazon Bedrock launched cross-region inference on August 27, 2024\n   Summary: The claim is correct as Amazon Bedrock did launch cross-region inference on August 27, 2024, as stated in the provided sources.\n✅ Cross-region inference provides up to 2x the allocated in-region quotas\n   Summary: The claim is correct as cross-region inference provides up to 2x the allocated in-region quotas according to the sources.\n✅ Cross-region inference support was extended to Knowledge Bases on September 13, 2024\n   Summary: The claim is correct as cross-region inference support was indeed extended to Knowledge Bases on September 13, 2024, as stated in the provided sources.\n```\n\nBut if you change the first claim in that file to August 25, 2024, you will see the following output.\n\n```\n❌ Amazon Bedrock launched cross-region inference on August 25, 2024\n   Summary: The claim is incorrect because Amazon Bedrock launched cross-region inference on August 27, 2024, not August 25, 2024.\n   Fixed Claim: Amazon Bedrock launched cross-region inference on August 27, 2024\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetaskills%2Ffast-llama-inference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetaskills%2Ffast-llama-inference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetaskills%2Ffast-llama-inference/lists"}