https://github.com/EZ-hwh/AutoScraper

Official implement of paper "AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation"
https://github.com/EZ-hwh/AutoScraper

webscraping

Last synced: 29 days ago
JSON representation

Official implement of paper "AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation"

• Host: GitHub
• URL: https://github.com/EZ-hwh/AutoScraper
• Owner: EZ-hwh
• License: apache-2.0
• Created: 2024-04-19T09:19:31.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2025-01-03T03:10:45.000Z (about 2 months ago)
• Last Synced: 2025-01-03T04:20:42.884Z (about 2 months ago)
• Topics: webscraping
• Language: Python
• Homepage:
• Size: 1.25 MB
• Stars: 441
• Watchers: 12
• Forks: 36
• Open Issues: 6
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# AutoScraper



    

    

    



This is the official code for paper *"AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation"*.

![](assets/Framework.png)

## Setup 

```bash

# Clone the AutoScraper repository

git clone https://github.com/EZ-hwh/AutoCrawler

# Change directory into the cloned repository

cd AutoCrawler

# Optional: Create a Conda environment for AutoScraper

# conda create -n autocrawler python=3.9

# conda activate autocrawler

# Install required dependencies

pip install -r requirements.txt

```

## TODOs

- [x] Public the experimental code.

- [ ] Adapt AutoScraper for real-world websites.

- [ ] Website for showing our demo and testing.

## Experiments

If you want to reproduce the result we report in paper.

```bash

# Generate scraper with AutoScraper

python crawler_generation.py \

    --pattern reflexion \

    --dataset swde \

    --model ChatGPT \

    --seed_website 3 \

    --save_name ChatGPT \

    --overwrite False

# Extract information with scraper

python crawler_extraction.py \

    --pattern autocrawler \

    --dataset swde \

    --model GPT4

# Evaluate the extraction on SWDE dataset

python run_swde/evaluate.py \

    --model GPT4 \

    --pattern autocrawler

```

## 📝 Citation

If you find this work useful, please consider citing our work:

```

@misc{huang2024autoscraperprogressiveunderstandingweb,

      title={AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation}, 

      author={Wenhao Huang and Zhouhong Gu and Chenghao Peng and Zhixu Li and Jiaqing Liang and Yanghua Xiao and Liqian Wen and Zulong Chen},

      year={2024},

      eprint={2404.12753},

      archivePrefix={arXiv},

      primaryClass={cs.CL},

      url={https://arxiv.org/abs/2404.12753}, 

}

```