https://github.com/hurricaneinteractive/epic-games-scraper
A simple Hyper API to extract game details from the Epic Games website
https://github.com/hurricaneinteractive/epic-games-scraper
api epic-games hyper rust scraper
Last synced: 10 months ago
JSON representation
A simple Hyper API to extract game details from the Epic Games website
- Host: GitHub
- URL: https://github.com/hurricaneinteractive/epic-games-scraper
- Owner: HurricaneInteractive
- Created: 2020-07-15T01:57:47.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2022-12-06T14:10:36.000Z (over 3 years ago)
- Last Synced: 2023-03-03T10:07:44.832Z (over 3 years ago)
- Topics: api, epic-games, hyper, rust, scraper
- Language: Rust
- Homepage:
- Size: 285 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Epic games store scraper
This scraper is a small part of a larger project and is meant to scrap all the games from the epic games store.
One issue with the store is that it uses React to render the frontend, meaning we aren't able to make a requests to the different URL and process the DOM as no games will be available.
The work around is a bit meh but I am new to Rust and I am trying my best.
I created a small API with [hyper](https://hyper.rs/) that will process the DOM into a array of games using this format;
```ts
Array<{
url: String,
name: String,
id: String,
type: "product" | "bundles"
}>
```
## What is required?
- [Postman](https://www.postman.com/)
- [Rust](https://www.rust-lang.org/)
## How to scrape?
1. Clone this repo and run `cargo run`, that will start a local server at `http://127.0.0.1:3000/`.
2. Open Postman and create a **POST** request with a endpoint of `http://127.0.0.1:3000/scrape`.
3. Open [Epic Games](https://www.epicgames.com/store/en-US/browse?sortBy=title&sortDir=ASC&pageSize=1000), you are able to do this for individual categories or just all of them. Remember to set the `pageSize` to 1000 (as high as it can go).
4. Run the JS below and copy the output.
5. Back in Postman, paste the output in `Body > Raw` and make sure it is set to `JSON`.
6. Send the request.
JS Code
```js
(function() {
console.log(
JSON.stringify({
dom: document.getElementById("dieselReactWrapper").outerHTML
})
)
})()
```
That should create a file on your system called `scrapped-text.json` with the scrapped data.
At this point you are feel to do what you want with this data.
There are some weird edge cases that are missed. For example [Ark](https://www.epicgames.com/store/en-US/product/ark/home), there are about 400 different editions/addons that each have their own page, but are only accessible through the Ark homepage and aren't in the listing page. I can imagine that it would be applicable to many games who have different editions/DLC.
## What am I doing with it?
> Coming soon...