https://github.com/cornayy/dofus-scraper
An open-source Dofus encyclopedia scraper.
https://github.com/cornayy/dofus-scraper
cheerio dofus dofus-scraper encyclopedia node nodejs scraping ts typescript
Last synced: 9 months ago
JSON representation
An open-source Dofus encyclopedia scraper.
- Host: GitHub
- URL: https://github.com/cornayy/dofus-scraper
- Owner: Cornayy
- License: mit
- Created: 2020-07-27T19:24:00.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2023-04-21T05:04:35.000Z (about 3 years ago)
- Last Synced: 2025-04-06T15:41:37.162Z (about 1 year ago)
- Topics: cheerio, dofus, dofus-scraper, encyclopedia, node, nodejs, scraping, ts, typescript
- Language: TypeScript
- Homepage:
- Size: 2.75 MB
- Stars: 7
- Watchers: 4
- Forks: 3
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# dofus-scraper
An open-source [Dofus](https://www.dofus.com/en) encyclopedia scraper. Scrape your desired encyclopedia categories and save it to your configured database.
## Contributing
The most important categories of the encyclopedia have been scraped, if you feel like some information about items are missing, feel free to open an issue or contribute.
## Modes
The 'MODE' environment variable is configurable in the following modes.
```
existing - the scraper will look for existing .json files.
new - the scraper will fetch the newest data.
```
## Usage
Create a .env file in the root of your project, a typical .env should look like this. You can also copy the contents of the .env.example file in the root directory.
```
DB_TYPE=mongodb
HOST=localhost
PORT=27017
DB_NAME=dofus-scraper
MODE=existing
```
However, this project is using [TypeORM](https://typeorm.io/#/), so the use of database is to your preference. When your .env is configured, run one of the following commands to start the scraping.
```
npm run existing
npm run new
```
## Example
## TODO
- [x] Equipment
- [x] Sets
- [x] Weapons
- [x] Resources
- [x] Pets
- [x] Ceremonial items
- [x] Mounts
- [x] Consumables
- [x] Bestiary
- [x] Idols
## Less important
- [ ] Professions
- [ ] Classes
- [ ] Sidekicks
- [ ] Haven bags
- [ ] Harnesses
## License
[MIT](LICENSE)