https://github.com/QiYao-Wang/AutoPatent
Repository of AutoPatent.
https://github.com/QiYao-Wang/AutoPatent
llm multiagent-systems patents
Last synced: about 2 months ago
JSON representation
Repository of AutoPatent.
- Host: GitHub
- URL: https://github.com/QiYao-Wang/AutoPatent
- Owner: QiYao-Wang
- Created: 2024-10-23T00:54:58.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2025-01-20T05:09:17.000Z (3 months ago)
- Last Synced: 2025-03-04T05:02:33.394Z (about 2 months ago)
- Topics: llm, multiagent-systems, patents
- Language: JavaScript
- Homepage: https://qiyao-wang.github.io/AutoPatent/
- Size: 47.9 MB
- Stars: 95
- Watchers: 4
- Forks: 12
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- StarryDivineSky - QiYao-Wang/AutoPatent
README
```
Note: We will make the data and code available upon the paper's acceptance.
```
AutoPatent: A Multi-Agent Framework for Automatic Patent Generation
Qiyao Wang1,2*,
Shiwen Ni1*,
Huaren Liu2,
Shule Lu2,
Guhong Chen1,3,
Xi Feng1,
Chi Wei1,
Qiang Qu1,
Hamid Alinejad-Rokny5,
Yuan Lin2†,
Min Yang1,4†
*Equal Contribution, † Corresponding Authors.
1Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
2Dalian University of Technology
3Southern University of Science and Technology
4Shenzhen University of Advanced Technology
5The University of New South Wales
## 📢 News
- [2024-08-30] Research Beginning.
- [2024-12-13] We have submitted our paper to arXiv, and it will be publicly available soon.
- [2024-12-13] We have released the initial version of AutoPatent on GitHub. The complete code and data will be made publicly available following the paper’s acceptance.
- [2024-12-16] The paper has been public at Arxiv.
- [2024-12-20] This work has been fortunate to receive attention and coverage from Xin Zhi Yuan, and it will continue to be expanded and improved in the future. [[News](https://mp.weixin.qq.com/s/H6zsbruPl4wtYOFCOIphBA)]
- [2024-12-20] We have released 10 generated patent samples in this GitHub repository. Please see the “example” folder.
## Table of Contents
- [Overview](#-overview)
- [Quick Start](#-overview)
- [Experiment Detail](#experiment-detail)
- [Metric](#metric)
- [Objective Metric](#objective-metric)
- [Human Evaluation](#human-evaluation)
- [Compared Method](#compared-method)
- [Zero-Shot Prompting](#zero-shot-prompting)
- [Supervised Fine-Tuning](#supervised-fine-tuning)
- [Results](#results)
- [Objective Metric Results](#objective-metric-results)
- [Human Evaluation Results](#human-evaluation-results)
- [Demo](#demo)
- [Citation](#citation)
- [Contact](#contact)
## 📖 Overview
We introduce a novel and practical task known as **Draft2Patent**, along with its corresponding **D2P benchmark**, which challenges LLMs to generate full-length patents averaging 17K tokens based on initial drafts. Patents present a significant challenge to LLMs due to their specialized nature, standardized terminology, and extensive length.
We propose a multi-agent framework called **AutoPatent** which leverages the LLM-based planner agent, writer agents, and examiner agent with PGTree and RRAG to generate to craft lengthy, intricate, and high-quality complete patent documents.
## 🧐 Quick Start
We will make the data and code available upon the paper's acceptance.
## Experiment Detail
### Metric
#### Objective Metric
We use the n-gram-based metric, BLEU, the F1 scores of ROUGE-1, ROUGE-2, and ROUGE-L as the objective metrics.
We propose a new metric, termed IRR (Inverse Repetition Rate), to measure the degree of sentence repetition within the patent $\mathcal{P}=\\{s_i|1\le i\le n\\}$, which consists of $n$ sentences.
The IRR is defined as:
$$
IRR (\mathcal{P}, t) = \frac{C_n^2}{\sum_{i=1}^{n-1} \sum_{j=i+1}^{n} f(s_i, s_j) + \varepsilon}
$$
Where the time complexity of the IRR metric is $O(n^2)$, $\varepsilon$ is a small value added for smoothing to prevent division by zero, and $t$ is threshold for determining whether two sentences, $s_i$ and $s_j$, are considered repetitions based on their Jaccard similarity $J$, calculated after removing stop words.
The function $f(s_i, s_j)$ is defined as:
$$
f(s_i, s_j) =
\begin{cases}
1, & \text{if } J(s_i, s_j) \geq t, \\
0, & \text{if } J(s_i, s_j) < t.
\end{cases}
$$
#### Human Evaluation
We invite three experts who are **familiar with the patent law** and **patent drafting** to evaluate the quality of generated patent using a single-bind review.
### Compared Method
#### Zero-Shot Prompting
The prompt is provided in Appendix C.1 of the paper.
**Models:**
- Commercial Model
- GPT-4o
- GPT-4o-mini
- Open source model
- LLAMA3.1 (8B and 70B)
- Qwen2.5 (7B, 14B, 32B and 72B)
- Mistral-7B
#### Supervised Fine-Tuning
We utilize 1,500 draft-patent pairs from D2P’s training set to perform fully supervised fine-tuning on LLAMA3.1-8B, Qwen2.5-7B, and Mistral-7B models (each with fewer than 14 billion parameters).
The fine-tuning process leverages [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) as the tool for efficiently fine-tuning models.
### Results
#### Objective Metric Results
#### Human Evaluation Results
## Demo
We have released the first demo video of AutoPatent on our [website](https://qiyao-wang.github.io/AutoPatent/).
A publicly accessible and customizable demo will be available upon the paper's acceptance.
## Citation
If you find this repository helpful, please consider citing the following paper:
```bib
@article{wang2024autopatent,
title={AutoPatent: A Multi-Agent Framework for Automatic Patent Generation},
author={Wang, Qiyao and Ni, Shiwen and Liu, Huaren and Lu, Shule and Chen, Guhong and Feng, Xi and Wei, Chi and Qu, Qiang and Alinejad-Rokny, Hamid and Lin, Yuan and others},
journal={arXiv preprint arXiv:2412.09796},
year={2024}
}
```
## Star History
[](https://star-history.com/#QiYao-Wang/AutoPatent&Date)
## Contact
If you have any questions, feel free to contact us at `[email protected]` or `[email protected]`.