https://github.com/EZ-hwh/AutoCrawler

Official implement of paper "AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation"
https://github.com/EZ-hwh/AutoCrawler

webscraping

Last synced: about 2 months ago
JSON representation

Official implement of paper "AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation"

• Host: GitHub
• URL: https://github.com/EZ-hwh/AutoCrawler
• Owner: EZ-hwh
• License: apache-2.0
• Created: 2024-04-19T09:19:31.000Z (5 months ago)
• Default Branch: main
• Last Pushed: 2024-06-18T12:16:39.000Z (3 months ago)
• Last Synced: 2024-06-19T13:52:31.629Z (3 months ago)
• Topics: webscraping
• Language: Python
• Homepage:
• Size: 1.25 MB
• Stars: 355
• Watchers: 11
• Forks: 30
• Open Issues: 7
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# AutoCrawler



    

    

    



This is the official code for paper *"AutoCrawler: A Progressive Understanding Web Agent for Web

Crawler Generation"*.

![](assets/Framework.png)

## Setup 

```bash

# Clone the AutoCrawler repository

git clone https://github.com/EZ-hwh/AutoCrawler

# Change directory into the cloned repository

cd AutoCrawler

# Optional: Create a Conda environment for AutoCrawler

# conda create -n autocrawler python=3.9

# conda activate autocrawler

# Install required dependencies

pip install -r requirements.txt

```

## TODOs

- [x] Public the experimental code.

- [ ] Adapt AutoCrawler for real-world websites.

- [ ] Website for showing our demo and testing.

## Experiments

If you want to reproduce the result we report in paper.

```bash

# Generate crawler with AutoCrawler

python crawler_generation.py \

    --pattern reflexion \

    --dataset swde \

    --model ChatGPT \

    --seed_website 3 \

    --save_name ChatGPT \

    --overwrite False

# Extract information with crawler

python crawler_extraction.py \

    --pattern autocrawler \

    --dataset swde \

    --model GPT4

# Evaluate the extraction on SWDE dataset

python run_swde/evaluate.py \

    --model GPT4 \

    --pattern autocrawler

```

## 📝 Citation

If you find this work useful, please consider citing our work:

```

@misc{huang2024autocrawler,

      title={AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation}, 

      author={Wenhao Huang and Chenghao Peng and Zhixu Li and Jiaqing Liang and Yanghua Xiao and Liqian Wen and Zulong Chen},

      year={2024},

      eprint={2404.12753},

      archivePrefix={arXiv},

      primaryClass={cs.CL}

}

```