https://github.com/coko7/vegapull
👒 One Piece TCG data scraper written in Rust
https://github.com/coko7/vegapull
cli command-line command-line-interface command-line-tool one-piece one-piece-card-game one-piece-tcg punk-records rust scraper scraping tcg vega vegapull vegapunk
Last synced: 13 days ago
JSON representation
👒 One Piece TCG data scraper written in Rust
- Host: GitHub
- URL: https://github.com/coko7/vegapull
- Owner: coko7
- License: gpl-3.0
- Created: 2024-08-25T19:48:33.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2026-05-06T23:37:02.000Z (26 days ago)
- Last Synced: 2026-05-07T01:28:15.399Z (25 days ago)
- Topics: cli, command-line, command-line-interface, command-line-tool, one-piece, one-piece-card-game, one-piece-tcg, punk-records, rust, scraper, scraping, tcg, vega, vegapull, vegapunk
- Language: Rust
- Homepage: https://crates.io/crates/vegapull
- Size: 291 KB
- Stars: 28
- Watchers: 1
- Forks: 9
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# 👒 vegapull
A CLI utility to retrieve data for the One Piece Trading Cards Game (TCG).
The CLI directly goes against the [onepiece-cardgame.com](https://onepiece-cardgame.com) website and attempts to scrap information such as packs, cards and images.
Since [v0.5.0](https://crates.io/crates/vegapull/0.5.0), vegapull now supports multi-threaded image downloads natively for a ***Blazingly faaaast experience 🚀***
> [!WARNING]
> 🚧 I am currently working on a massive rework of `vegapull` on branch [v1-rework](https://github.com/coko7/vegapull/tree/v1-rework). This rework will introduce breaking changes to the CLI and the support for multi-threaded downloads on every single pull modes. v1 will also come with a big improvement to the existing **interactive** mode as I want to make it the default mode for most users.
## Where can I find prefetched datasets?
There are currently two Git repositories with JSON data:
- [buhbbl/punk-records](https://github.com/buhbbl/punk-records) (all languages)
- [Coko7/vegapull-records](https://github.com/Coko7/vegapull-records) (english/japanese only)
## Installation
The easiest way to install is through [crates.io](https://crates.io/crates/vegapull):
```console
cargo install vegapull
```
The other option is to build from source:
```console
git clone https://github.com/Coko7/vegapull.git
cd vegapull
cargo build --release
```
## How to use?
```console
coko7@example:~$ vegapull -h
Dynamically fetch data for the One Piece TCG from official sites.
Usage: vegapull [OPTIONS]
Commands:
pack Get the list of all existing packs
diff Compare datasets
card Get all cards within the given pack
image Download all card images for a given pack
inter Launch into interactive mode
test-config Test what configuration files are found
help Print this message or the help of the given subcommand(s)
Options:
-l, --language
Language to use for the data [default: english] [possible values: chinese-hongkong, chinese-simplified, chinese-taiwan, english, english-asia, japanese, thai, french]
-c, --config-dir
Specify path to the config directory (where locales are stored)
-v, --verbose...
Increase logging verbosity
-q, --quiet...
Decrease logging verbosity
-h, --help
Print help
-V, --version
Print version
```
## Helper Scripts
You can use the example scripts that directly use the `vegapull` CLI to download data for all existing packs:
### Bash
```console
coko7@example:~$ bash scripts/pull-all.sh
// the `gum` one is more complete but requires some additional tooling to install in your shell:
coko7@example:~$ bash scripts/pull-all-gum.sh
```
### Go
> [!NOTE]
> Requires [Go](https://go.dev/) to be installed.
```console
➜ go run scripts/pull.go
```
### Python
You can find a Python helper script on this repository: https://github.com/buhbbl/punk-records
## 🃏 Supported card fields
```rust
#[derive(Debug, Deserialize, Serialize)]
pub struct Card {
pub id: String,
pub name: String,
pub rarity: CardRarity,
pub category: CardCategory,
// pub number: i32,
// #[serde(skip_serializing)]
// pub set_id: String,
// pub copyright: String,
// Images
pub img_url: String,
// pub illustration: CardIllustration,
// pub illustrator_name: String,
// Gameplay
pub colors: Vec,
pub cost: Option, // Only Character, Event and Stage (called life for Leader)
pub attributes: Vec, // Only Leader and Character
pub power: Option, // Only Leader and Character
pub counter: Option, // Only Character
pub types: Vec,
pub effect: String,
pub trigger: Option,
// pub notes: String,
}
```
Fields have been named following the terms used in the official [rule book](https://en.onepiece-cardgame.com/pdf/rule_comprehensive.pdf)
## 🐛 Issues
When using `jp` locale to fetch data, the scraper will likely fail when handling `counter` or `colors` values for some cards.
## 🗺️ Road Map
- [x] Fetch card sets data
- [x] Better error handling
- [x] Fetch cards data for each card set (wip)
- [x] Get card data for all card sets
- [x] Organize and save cards data as JSON to files
- [x] Add logs
- [x] Support more card fields
- [x] Download card images as well
- [x] Make it locale-agnostic to be able to download data from Japanese and other versions
- [ ] Handle problems with the `jp` version (inconsistent cards data on official site)
- [ ] Better configuration
- [x] User friendly CLI
- [ ] Add tests