{"id":27957962,"url":"https://github.com/pfcclab/paddleapitest","last_synced_at":"2026-02-12T14:13:02.768Z","repository":{"id":271702028,"uuid":"914306043","full_name":"PFCCLab/PaddleAPITest","owner":"PFCCLab","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-07T08:16:18.000Z","size":230019,"stargazers_count":4,"open_issues_count":5,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-07T18:15:03.279Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PFCCLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-09T10:41:31.000Z","updated_at":"2025-05-07T08:16:26.000Z","dependencies_parsed_at":"2025-01-24T05:23:19.114Z","dependency_job_id":"b3ccef5e-6a9f-450a-80f0-443b8446d919","html_url":"https://github.com/PFCCLab/PaddleAPITest","commit_stats":null,"previous_names":["wanghuancoder/paddleapitest","pfcclab/paddleapitest"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PFCCLab%2FPaddleAPITest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PFCCLab%2FPaddleAPITest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PFCCLab%2FPaddleAPITest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PFCCLab%2FPaddleAPITest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PFCCLab","download_url":"https://codeload.github.com/PFCCLab/PaddleAPITest/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931507,"owners_count":21827112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T18:15:34.796Z","updated_at":"2026-02-12T14:13:02.703Z","avatar_url":"https://github.com/PFCCLab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PaddleAPITest\n******\n## 1. 项目背景\nAPI正确性是Paddle质量的基石，影响业务训练、推理，影响用户对Paddle的信赖。至关重要。为保障Paddle API正确性，我们开发了PaddleAPITest。\n\n目前主要考虑的正确性问题有**3**类：\n\u003e1）API**精度**正确性；\n\u003e\n\u003e2）一些**特大**Tensor，尤其是numel超过int32上限的Tensor计算异常；\n\u003e\n\u003e3）**0-Size**（numel为0的Tensor） Tensor不支持。\n\nPaddleAPITest主要工作思路如下：\n1. 在Paddle开发Trace API机制，具体见 https://github.com/PaddlePaddle/Paddle/pull/70752 ，用于抓取API调用配置，下面是一个配置例子：\n```\npaddle.concat(tuple(Tensor([31376, 768],\"float32\"),Tensor([1, 768],\"float32\"),), axis=0, )\n```\n2. 在所有Paddle单元测试（CI）、集成测试（CE）流水线中，抓取所有Paddle API的调用配置，形成了PaddleAPITest/tester/api_config下以“api_config_CI”，“api_config_CE”开头的配置集。对以上配置集进行去重、排序、梳理得到了以“api_config_merged”开头的配置集。\n\n3. 在 PaddleAPITest 中开发一套**引擎**，加载配置集，初始化相应Tensor，调用相应API执行前/反向测试。\n\n4. 在 PaddleAPITest 中开发一套**转换工具**，在调用Paddle API测试的同时，等同的调用Torch API，做精度对比测试。\n\n5. 对采集到的配置集进行shape篡改，得到了“bigtensor”、“0sizetensor”开头的配置集。\n\n6. 通过与Torch对比，如果出现以下情况则认为Paddle API有必要确认是否正确并修复：\n\u003ea. 精度diff\n\u003e\n\u003eb. Torch正常，Paddle报错\n\u003e\n\u003ec. Torch正常，Paddle **CoreDump**或**CUDA Error**\n\n\n## 2. 项目结构\n\n目前项目结构如下所示，主要分为report和tester文件夹，report用于储存内核报错的api信息，tester用于测试配置的正确性。\n\n在引擎补齐这一任务中，出现的内核报错均放置于report/fresh_report中。\n\ntester/api_config中存放测试通过（merged*）/暂未通过（merged_not_support*）的配置。\n\ntester/api_config/config_analyzer.py是引擎补齐任务的核心代码。\n\n```\n├── engine.py\n├── __init__.py\n├── README.md\n├── report\n│   ├── 0size_tensor\n│   ├── big_tensor\n│   ├── ci_ce\n│   └── fresh_report\n└── tester\n    ├── accuracy.py\n    ├── api_config\n    ├── base.py\n    ├── __init__.py\n    ├── paddle_cinn_vs_dygraph.py\n    ├── paddle_only.py\n    ├── paddle_to_torch\n    └── __pycache__\n```\n\n## 3. 使用介绍\n\n### 环境配置\n运行环境分为**cpu**环境与**gpu**环境，cpu和gpu上运行的结果**可能存在差异**，即存在cpu上能够正确运行，但gpu上报错的情况。因此需要根据需求正确安装环境。\n\n下载链接：https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/develop/install/pip/windows-pip.html\n\n若需要本地编译paddle，可参考链接：https://www.paddlepaddle.org.cn/documentation/docs/zh/install/compile/linux-compile-by-make.html\n\n测试CPU除了通过上述链接安装CPU的最新develop包之外，还可使用如下指令设置Paddle工作在CPU模式：\n```\npaddle.device.set_device(\"cpu\")\n```\n\n### 使用说明\n\n所有测试前，**必须创建**一个目录：PaddleAPITest/tester/api_config/test_log/，用于存放测试所产生的测试结果和checkpoint。\n\nPaddleAPITest目前支持paddle_only、accuracy、paddle_cinn三种测试：\n\u003epaddle_only，用于单纯把配置在Paddle动态图跑一遍，验证PaddleAPITest 引擎**是否支持**该配置。\n\u003e\n\u003eaccuracy，用于将Paddle API的前反向与**Torch**的前反向做精度对比测试。\n\u003e\n\u003epaddle_cinn，用于Paddle动态图与Paddle静态图编译器做精度对比测试。\n\n当测试**单个配置**时，可使用下面的代码，--api_config中输入待测试的配置内容：\n\n仅测试paddle**是否支持**：\n```\npython engine.py --paddle_only=True --api_config='paddle.abs(Tensor([1, 100],\"float64\"), )'\n```\n测试输出**是否准确**：\n```\npython engine.py --accuracy=True --api_config='paddle.abs(Tensor([1, 100],\"float64\"), )'\n```\n动态图和静态图测试：\n```\npython engine.py --paddle_cinn=True --api_config='paddle.abs(Tensor([1, 100],\"float64\"), )'\n```\n\n**值得注意**的是配置txt中统一使用双引号\"，因此建议--api_config=''使用单引号，或在配置中手动添加转义斜杠\\\n\n当需要测试的配置数目较多时，手动单次输入将**非常低效**，这种情况下可以使用如下所示的**批量测试**指令，将配置保存在一个txt中，并将指令中的路径设置为txt的路径即可：\n```\npython engine.py --api_config_file=/host_home/wanghuan29/PaddleAPITest/tester/api_config/api_config.txt --accuracy=True \u003e tester/api_config/test_log/log.log 2\u003e\u00261\n\npython engine.py --api_config_file=/host_home/wanghuan29/PaddleAPITest/tester/api_config/api_config.txt --paddle_only=True \u003e tester/api_config/test_log/log.log 2\u003e\u00261\n\npython engine.py --api_config_file=/host_home/wanghuan29/PaddleAPITest/tester/api_config/api_config.txt --paddle_cinn=True \u003e tester/api_config/test_log/log.log 2\u003e\u00261\n```\n\n运行配置时，可能会存在bug导致**测试中断**。PaddleAPITest开发了checkpoint机制，记录“已测试”配置。可以通过如下命令启动批量测试：\n```\nfor i in {1..10000}; do python engine.py --api_config_file=/host_home/wanghuan29/PaddleAPITest/tester/api_config/api_config_merge.txt --accuracy=True \u003e\u003e tester/api_config/test_log/log.log 2\u003e\u00261; done\n\nfor i in {1..10000}; do python engine.py --api_config_file=/host_home/wanghuan29/PaddleAPITest/tester/api_config/api_config_merge.txt --paddle_only=True \u003e\u003e tester/api_config/test_log/log.log 2\u003e\u00261; done\n\nfor i in {1..10000}; do python engine.py --api_config_file=/host_home/wanghuan29/PaddleAPITest/tester/api_config/api_config_merge.txt --paddle_cinn=True \u003e\u003e tester/api_config/test_log/log.log 2\u003e\u00261; done\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpfcclab%2Fpaddleapitest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpfcclab%2Fpaddleapitest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpfcclab%2Fpaddleapitest/lists"}