{"id":24989175,"url":"https://github.com/paradite/deepseek-r1-speed-benchmark","last_synced_at":"2025-10-14T21:33:46.255Z","repository":{"id":274748029,"uuid":"923941133","full_name":"paradite/deepseek-r1-speed-benchmark","owner":"paradite","description":"Code for benchmarking the speed of DeepSeek R1 from different providers' APIs.","archived":false,"fork":false,"pushed_at":"2025-03-21T09:22:16.000Z","size":280,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T11:43:28.839Z","etag":null,"topics":["benchmark","deepinfra","deepseek","deepseek-r1","fireworks","hyperbolic","together"],"latest_commit_sha":null,"homepage":"https://prompt.16x.engineer/blog/deepseek-r1-cost-pricing-speed","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paradite.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-29T05:30:29.000Z","updated_at":"2025-03-21T10:18:38.000Z","dependencies_parsed_at":"2025-03-21T10:32:20.568Z","dependency_job_id":"d35ed519-8deb-47f4-882e-74214edd2ae8","html_url":"https://github.com/paradite/deepseek-r1-speed-benchmark","commit_stats":null,"previous_names":["paradite/deepseek-r1-speed-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradite%2Fdeepseek-r1-speed-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradite%2Fdeepseek-r1-speed-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradite%2Fdeepseek-r1-speed-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paradite%2Fdeepseek-r1-speed-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paradite","download_url":"https://codeload.github.com/paradite/deepseek-r1-speed-benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249766541,"owners_count":21322616,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","deepinfra","deepseek","deepseek-r1","fireworks","hyperbolic","together"],"created_at":"2025-02-04T12:30:19.794Z","updated_at":"2025-10-14T21:33:46.171Z","avatar_url":"https://github.com/paradite.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DeepSeek R1 speed benchmark\n\nCode for benchmarking the speed of DeepSeek R1 from different providers' APIs.\n\nRead the full report: [DeepSeek R1: Comparing Pricing and Speed Across Providers](https://prompt.16x.engineer/blog/deepseek-r1-cost-pricing-speed)\n\n## Providers\n\nCurrently supported:\n\n- [DeepSeek](https://www.deepseek.com/)\n- [DeepInfra](https://deepinfra.com/)\n- [Fireworks](https://fireworks.ai/)\n- [Together](https://www.together.ai/)\n- [Chutes](https://chutes.ai/)\n- [Hyperbolic](https://hyperbolic.xyz/)\n- [Azure AI Foundry](https://azure.microsoft.com/en-us/products/ai-foundry)\n- [Nebius](https://nebius.com/)\n- [Nvidia NIM](https://build.nvidia.com/deepseek-ai/deepseek-r1)\n- [Kluster](https://www.kluster.ai/)\n- [Novita](https://novita.ai/)\n\n\u003e Nvidia NIM is stuck at streaming the response without a timeout, so it is skipped for now.\n\nTODO:\n\n- [Sambanova Cloud](https://cloud.sambanova.ai/plans/pricing) (waiting list)\n- [replicate](https://replicate.com/deepseek-ai/deepseek-r1)\n- [SiliconFlow](https://siliconflow.cn/)\n\nProviders that I am not able to test due to high costs or lack of open access:\n\n- [Awesome Cloud](https://awesomecloud.ai/secure-deepseek-r1/) (Contact sales)\n- [AWS Bedrock](https://aws.amazon.com/blogs/aws/deepseek-r1-models-now-available-on-aws/) (Requires dedicated ec2 instance)\n- [featherless](https://featherless.ai/#pricing) (Requires subscription)\n- [Avian](https://avian.io/) (Requires dedicated deployment with 4 GPUs)\n\nWatch list for DeepSeek R1 support:\n\n- [Groq](https://www.groq.com/)\n- [Cerebras](https://cerebras.ai/)\n\n## Speed statistics\n\nStatistics of the speed of the API automatically generated by running `analyze-speed.js`.\n\n```\n=== Overall Speed Statistics (tokens/second) ===\nUsing latest 50 benchmark runs\n\nFireworks : Median/Mean: 23.80/29.93, Range:  6.77-78.17 ±19.34, Error rate:  0.00%, Success/Error: 32/0\nKluster   : Median/Mean: 20.25/20.66, Range: 14.44-33.75 ± 6.11, Error rate: 12.50%, Success/Error: 7/1\nDeepSeek  : Median/Mean: 18.02/23.96, Range: 11.28-67.60 ±15.56, Error rate: 15.63%, Success/Error: 27/5\nTogether  : Median/Mean: 16.98/23.74, Range:  7.57-92.77 ±22.28, Error rate:  3.13%, Success/Error: 31/1\nNovita    : Median/Mean: 15.03/16.29, Range:  9.75-23.85 ± 4.86, Error rate:  0.00%, Success/Error: 9/0\nHyperbolic: Median/Mean: 14.45/15.11, Range:  4.49-31.26 ± 7.77, Error rate: 17.24%, Success/Error: 24/5\nNvidia    : Median/Mean: 11.87/14.13, Range:  4.50-34.26 ± 9.63, Error rate: 33.33%, Success/Error: 12/6\nNebius    : Median/Mean:  9.17/10.68, Range:  3.21-26.72 ± 7.42, Error rate:  0.00%, Success/Error: 22/0\nDeepInfra : Median/Mean:  7.80/ 7.97, Range:  3.09-12.01 ± 1.75, Error rate:  0.00%, Success/Error: 32/0\nAzure     : Median/Mean:  6.61/11.01, Range:  1.89-41.85 ±12.36, Error rate:  3.85%, Success/Error: 25/1\n\n=== Daily Statistics ===\n\nDate: 21/03/2025\nTogether  : Median/Mean: 86.10/88.27, Range: 85.93-92.77 ± 3.19, Error rate: 25.00%, Success/Error: 3/1\nFireworks : Median/Mean: 76.19/74.04, Range: 65.62-78.17 ± 4.95, Error rate:  0.00%, Success/Error: 4/0\nAzure     : Median/Mean: 39.81/39.03, Range: 34.65-41.85 ± 2.67, Error rate:  0.00%, Success/Error: 4/0\nNebius    : Median/Mean: 24.02/24.58, Range: 23.53-26.72 ± 1.29, Error rate:  0.00%, Success/Error: 4/0\nHyperbolic: Median/Mean: 23.68/23.26, Range: 20.09-25.59 ± 2.15, Error rate:  0.00%, Success/Error: 4/0\nNovita    : Median/Mean: 22.00/20.55, Range: 14.34-23.85 ± 3.73, Error rate:  0.00%, Success/Error: 4/0\nKluster   : Median/Mean: 20.25/18.70, Range: 14.44-21.40 ± 3.05, Error rate: 25.00%, Success/Error: 3/1\nDeepSeek  : Median/Mean: 15.57/15.69, Range: 14.06-17.57 ± 1.61, Error rate:  0.00%, Success/Error: 4/0\nDeepInfra : Median/Mean: 10.40/10.14, Range:  7.73-12.01 ± 1.73, Error rate:  0.00%, Success/Error: 4/0\n\nDate: 11/03/2025\nNovita    : Median/Mean: 13.57/12.88, Range:  9.75-15.39 ± 2.28, Error rate:  0.00%, Success/Error: 5/0\n\nDate: 12/02/2025\nTogether  : Median/Mean: 38.42/38.42, Range: 35.93-40.91 ± 2.49, Error rate:  0.00%, Success/Error: 2/0\nNvidia    : Median/Mean: 33.92/33.92, Range: 33.59-34.26 ± 0.33, Error rate:  0.00%, Success/Error: 2/0\nFireworks : Median/Mean: 27.05/27.05, Range: 21.84-32.25 ± 5.21, Error rate:  0.00%, Success/Error: 2/0\nKluster   : Median/Mean: 19.59/22.14, Range: 15.62-33.75 ± 7.31, Error rate:  0.00%, Success/Error: 4/0\nDeepSeek  : Median/Mean: 16.07/16.07, Range: 14.13-18.02 ± 1.94, Error rate:  0.00%, Success/Error: 2/0\nNebius    : Median/Mean:  9.43/ 9.43, Range:  8.43-10.42 ± 1.00, Error rate:  0.00%, Success/Error: 2/0\nDeepInfra : Median/Mean:  8.80/ 8.80, Range:  8.52- 9.09 ± 0.29, Error rate:  0.00%, Success/Error: 2/0\nAzure     : Median/Mean:  6.04/ 6.04, Range:  5.50- 6.59 ± 0.54, Error rate:  0.00%, Success/Error: 2/0\nHyperbolic: Median/Mean:  5.40/ 5.40, Range:  4.49- 6.31 ± 0.91, Error rate:  0.00%, Success/Error: 2/0\n\nDate: 05/02/2025\nHyperbolic: Median/Mean: 21.13/21.13, Range: 16.03-26.24 ± 5.10, Error rate: 33.33%, Success/Error: 2/1\nDeepSeek  : Median/Mean: 20.00/20.00, Range: 16.35-23.64 ± 3.64, Error rate: 33.33%, Success/Error: 2/1\nFireworks : Median/Mean: 13.44/13.67, Range:  8.68-18.90 ± 4.18, Error rate:  0.00%, Success/Error: 3/0\nTogether  : Median/Mean: 12.67/14.44, Range: 10.88-19.78 ± 3.84, Error rate:  0.00%, Success/Error: 3/0\nDeepInfra : Median/Mean:  7.45/ 7.89, Range:  7.18- 9.03 ± 0.82, Error rate:  0.00%, Success/Error: 3/0\nNebius    : Median/Mean:  6.75/ 6.76, Range:  3.21-10.31 ± 2.90, Error rate:  0.00%, Success/Error: 3/0\nAzure     : Median/Mean:  3.68/ 5.33, Range:  3.67- 8.64 ± 2.34, Error rate:  0.00%, Success/Error: 3/0\nNvidia    : Error rate: 100.00%, Success/Error: 0/3\n\nDate: 03/02/2025\nFireworks : Median/Mean: 31.70/31.34, Range: 29.53-32.79 ± 1.36, Error rate:  0.00%, Success/Error: 3/0\nTogether  : Median/Mean: 16.87/16.51, Range: 15.67-16.98 ± 0.59, Error rate:  0.00%, Success/Error: 3/0\nDeepSeek  : Median/Mean: 16.34/16.34, Range: 16.34-16.34 ± 0.00, Error rate: 66.67%, Success/Error: 1/2\nNebius    : Median/Mean:  9.91/ 8.80, Range:  3.25-13.23 ± 4.15, Error rate:  0.00%, Success/Error: 3/0\nDeepInfra : Median/Mean:  7.83/ 7.90, Range:  7.30- 8.56 ± 0.52, Error rate:  0.00%, Success/Error: 3/0\nAzure     : Median/Mean:  6.71/ 6.69, Range:  6.61- 6.75 ± 0.06, Error rate:  0.00%, Success/Error: 3/0\nHyperbolic: Median/Mean:  6.66/ 6.85, Range:  6.44- 7.46 ± 0.44, Error rate:  0.00%, Success/Error: 3/0\nNvidia    : Error rate: 100.00%, Success/Error: 0/3\n\nDate: 02/02/2025\nFireworks : Median/Mean: 26.83/27.46, Range: 25.77-29.77 ± 1.69, Error rate:  0.00%, Success/Error: 3/0\nTogether  : Median/Mean: 14.14/14.16, Range: 12.79-15.55 ± 1.13, Error rate:  0.00%, Success/Error: 3/0\nHyperbolic: Median/Mean: 13.64/13.77, Range: 12.41-15.26 ± 1.17, Error rate:  0.00%, Success/Error: 3/0\nDeepSeek  : Median/Mean: 13.44/13.37, Range: 13.21-13.45 ± 0.11, Error rate:  0.00%, Success/Error: 3/0\nDeepInfra : Median/Mean:  8.89/ 8.80, Range:  8.23- 9.28 ± 0.43, Error rate:  0.00%, Success/Error: 3/0\nNvidia    : Median/Mean:  6.76/ 9.39, Range:  6.60-14.82 ± 3.84, Error rate:  0.00%, Success/Error: 3/0\nAzure     : Median/Mean:  6.67/ 6.69, Range:  6.62- 6.78 ± 0.07, Error rate:  0.00%, Success/Error: 3/0\nNebius    : Median/Mean:  4.80/ 6.23, Range:  3.73-10.15 ± 2.81, Error rate:  0.00%, Success/Error: 3/0\n\nDate: 01/02/2025\nFireworks : Median/Mean: 27.15/27.15, Range: 26.94-27.35 ± 0.21, Error rate:  0.00%, Success/Error: 2/0\nTogether  : Median/Mean: 20.88/20.88, Range: 20.79-20.97 ± 0.09, Error rate:  0.00%, Success/Error: 2/0\nNvidia    : Median/Mean: 15.43/15.43, Range: 14.76-16.09 ± 0.67, Error rate:  0.00%, Success/Error: 2/0\nAzure     : Median/Mean:  6.92/ 6.92, Range:  6.92- 6.92 ± 0.00, Error rate: 50.00%, Success/Error: 1/1\nDeepInfra : Median/Mean:  6.81/ 6.81, Range:  6.46- 7.17 ± 0.35, Error rate:  0.00%, Success/Error: 2/0\nNebius    : Median/Mean:  4.83/ 4.83, Range:  4.16- 5.49 ± 0.67, Error rate:  0.00%, Success/Error: 2/0\nDeepSeek  : Error rate: 100.00%, Success/Error: 0/2\nHyperbolic: Error rate: 100.00%, Success/Error: 0/2\n\nDate: 31/01/2025\nFireworks : Median/Mean: 28.99/32.51, Range: 13.24-54.62 ±16.21, Error rate:  0.00%, Success/Error: 5/0\nDeepSeek  : Median/Mean: 26.14/37.87, Range: 17.55-67.60 ±19.86, Error rate:  0.00%, Success/Error: 5/0\nTogether  : Median/Mean: 18.22/18.58, Range: 15.44-20.66 ± 1.91, Error rate:  0.00%, Success/Error: 5/0\nNvidia    : Median/Mean:  9.41/ 8.53, Range:  4.50-13.45 ± 3.37, Error rate:  0.00%, Success/Error: 5/0\nDeepInfra : Median/Mean:  7.56/ 6.07, Range:  3.09- 8.45 ± 2.29, Error rate:  0.00%, Success/Error: 5/0\nHyperbolic: Median/Mean:  6.56/ 9.16, Range:  5.65-15.28 ± 4.34, Error rate: 40.00%, Success/Error: 3/2\nNebius    : Median/Mean:  5.80/ 8.56, Range:  4.10-17.24 ± 4.87, Error rate:  0.00%, Success/Error: 5/0\nAzure     : Median/Mean:  5.64/ 4.64, Range:  1.89- 6.90 ± 2.09, Error rate:  0.00%, Success/Error: 5/0\n\nDate: 30/01/2025\nFireworks : Median/Mean: 20.39/17.56, Range:  6.77-21.41 ± 5.18, Error rate:  0.00%, Success/Error: 6/0\nDeepSeek  : Median/Mean: 20.30/23.64, Range: 11.28-46.72 ±11.04, Error rate:  0.00%, Success/Error: 6/0\nTogether  : Median/Mean: 15.54/15.03, Range:  8.69-21.72 ± 4.87, Error rate:  0.00%, Success/Error: 6/0\nHyperbolic: Median/Mean: 13.19/16.31, Range:  7.21-31.26 ± 8.14, Error rate:  0.00%, Success/Error: 5/0\nDeepInfra : Median/Mean:  7.22/ 7.33, Range:  6.87- 7.89 ± 0.36, Error rate:  0.00%, Success/Error: 6/0\nAzure     : Median/Mean:  5.49/ 5.22, Range:  3.97- 5.95 ± 0.75, Error rate:  0.00%, Success/Error: 4/0\n\nDate: 29/01/2025\nHyperbolic: Median/Mean: 22.84/22.84, Range: 20.22-25.47 ± 2.63, Error rate:  0.00%, Success/Error: 2/0\nDeepSeek  : Median/Mean: 20.49/31.09, Range: 15.80-67.60 ±21.17, Error rate:  0.00%, Success/Error: 4/0\nFireworks : Median/Mean: 16.73/16.98, Range: 14.15-20.32 ± 2.27, Error rate:  0.00%, Success/Error: 4/0\nDeepInfra : Median/Mean:  9.46/ 8.82, Range:  6.61- 9.76 ± 1.28, Error rate:  0.00%, Success/Error: 4/0\nTogether  : Median/Mean:  8.76/ 8.57, Range:  7.57- 9.18 ± 0.63, Error rate:  0.00%, Success/Error: 4/0\n```\n\n## Sample output for a single run\n\n```\n=== Final Benchmark Results ===\nCurrent time: 2025-01-30T07:23:37.790Z\nTest prompt: What is the capital of France?\n\nDeepSeek  : Speed: 24.34 tokens/s, Total: 424 tokens, Prompt: 12 tokens, Completion: 412 tokens, Time: 16.93s, Latency: 1.11s, Length: 1904 chars\nFireworks : Speed: 21.41 tokens/s, Total: 373 tokens, Prompt: 10 tokens, Completion: 363 tokens, Time: 16.96s, Latency: 2.16s, Length: 1785 chars\nTogether  : Speed: 18.65 tokens/s, Total: 393 tokens, Prompt: 10 tokens, Completion: 383 tokens, Time: 20.54s, Latency: 0.56s, Length: 1782 chars\nDeepInfra : Speed: 6.87 tokens/s, Total: 73 tokens, Prompt: 10 tokens, Completion: 63 tokens, Time: 9.16s, Latency: 1.04s, Length: 297 chars\nAzure     : Speed: 5.54 tokens/s, Total: 372 tokens, Prompt: 10 tokens, Completion: 362 tokens, Time: 65.34s, Latency: 5.35s, Length: 1783 chars\n```\n\nFull outputs:\n\n- Check [outputs](outputs) directory for full outputs\n\n## Missing data for some providers\n\nA timeout of 10 seconds is used for all providers.\n\nIf the API does not respond within 10 seconds, the provider is skipped for that run.\n\nThis is why some providers are missing data.\n\n## Setup\n\n1. Install dependencies:\n\n```bash\nnpm install\n```\n\n2. Create a `.env` file in the root directory.\n\nFollow the sample in the `.env.example` file to set up your API keys.\n\n3. Make sure you have Node.js version 20 or higher installed.\n\n## Usage\n\nRun the benchmark:\n\n```bash\nnpm run benchmark        # Regular benchmark\nnpm run benchmark-show-output # Show the API response while benchmarking\nnpm run analyze-speed     # Analyze the speed of the API\n```\n\nThe script will measure:\n\n- Total tokens generated\n- Response time\n- First response latency\n- Tokens per second\n- Prompt and completion token counts\n\n## How it works\n\nThe benchmark script sends a standardized prompt to the DeepSeek API and measures:\n\n- The time taken to receive the complete response\n- The number of tokens in both the prompt and response\n- Calculates the overall tokens per second processing speed\n\nThis helps in understanding the real-world performance of the DeepSeek API in your specific environment and use case.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparadite%2Fdeepseek-r1-speed-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparadite%2Fdeepseek-r1-speed-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparadite%2Fdeepseek-r1-speed-benchmark/lists"}