https://github.com/zachstence/project-euler-scrape
Several spreadsheets and JSON files for various data (text, HTML, files/images) that was extracted from each ProjectEuler problem using a web scraper
https://github.com/zachstence/project-euler-scrape
coding-challenge project-euler python web-scraping
Last synced: 11 months ago
JSON representation
Several spreadsheets and JSON files for various data (text, HTML, files/images) that was extracted from each ProjectEuler problem using a web scraper
- Host: GitHub
- URL: https://github.com/zachstence/project-euler-scrape
- Owner: zachstence
- Created: 2018-05-13T17:07:41.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2022-12-08T02:19:21.000Z (over 3 years ago)
- Last Synced: 2025-08-31T12:44:20.927Z (11 months ago)
- Topics: coding-challenge, project-euler, python, web-scraping
- Language: Python
- Homepage:
- Size: 4.89 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Project-Euler-Scrape
A complete web-scrape of every Project Euler programming challenge problem including information about the problem, the problems themselves, and all files/images.
### What information was scraped?
I extracted as much information as I could find that was useful, including:
* Problem number (purple)
* Problem title (blue)
* Problem information (green)
* Publish date/time
* Number of solvers
* Difficulty rating
* Problem description (orange)
* Raw HTML from the page
* Plain text
* Any images in the problem description (red)
* Any files in the problem description (yellow)
Most of the data I scraped is in the file [`1_631.json`](https://github.com/zachstence/Project-Euler-Scrape/raw/master/data/1_631.json). The structure of the data is:
```
{
"": {
"number": 1,
"url": "",
"title": "",
"info": {
"difficulty": ,
"published": "",
"solved":
},
"content": {
"images": ,
"html": ,
"files":
}
},
...
}
```
The images and files from the problems are found in the [`images/`](https://github.com/zachstence/Project-Euler-Scrape/blob/master/data/images/) and [`files/`](https://github.com/zachstence/Project-Euler-Scrape/blob/master/data/files/) directories respectively.
### How was information scraped?
In previous commits I used a program called [ParseHub]() to do the scraping as I was fairly new to the concept and didn't think about doing it in a programming language. However, recently I redid everything in Python using [`requests`](http://docs.python-requests.org/en/master/) to get the webpages and [`BeautifulSoup`](https://www.crummy.com/software/BeautifulSoup/) to parse the HTML and scrape the information I wanted with regular expressions. All of the code is in [`pe_scrape.py`](https://github.com/zachstence/Project-Euler-Scrape/blob/master/pe_scrape.py)
### Why?
I am in the process of making a [portfolio](https://cs.txstate.edu/~zms22/portfolio/) of all of the programming projects I have done. Naturally, I have solved a couple of the Project Euler problems and wanted to include their descriptions, title, etc in my website without manually entering it all. So I decided to have the webpages dynamically filled with PHP using a json file containing all the necessary information, hence this project!
Feel free to use the data I scraped, or modify my code to suit your needs!