https://github.com/mpschrader/mpi-webcrawlling-tutorium
Material for a single day web crawling workshop in Python
https://github.com/mpschrader/mpi-webcrawlling-tutorium
python tutorial webcrawling
Last synced: 2 months ago
JSON representation
Material for a single day web crawling workshop in Python
- Host: GitHub
- URL: https://github.com/mpschrader/mpi-webcrawlling-tutorium
- Owner: mpSchrader
- License: gpl-3.0
- Created: 2018-01-26T08:58:14.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2019-10-20T18:06:22.000Z (almost 7 years ago)
- Last Synced: 2025-07-18T04:15:48.844Z (about 1 year ago)
- Topics: python, tutorial, webcrawling
- Language: Jupyter Notebook
- Homepage:
- Size: 173 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# MPI-Webcrawlling-Tutorium
**(Work in Progress)**
This Git repository contains all materials for the Webcrawling Tutorium held at the Max-Planck-Institute for Innovation and Competition.
### Prerequisites
To be able to participate in the workshop please ensure your machine meets all prerequisites.
- **Anaconda:** You should have Anaconda for Python 3.6 installed. \
Installation Instructions: https://conda.io/docs/user-guide/install/windows.html \
Download Page: https://www.anaconda.com/download/#windows
- **PyCharm:** Install PyCharm Community Edition or as a PhD Student the Professional Edition, which get for free as a student.\
PyCharm Download: https://www.jetbrains.com/pycharm/ \
Student Registration: https://www.jetbrains.com/student/
- **Python Packages:** Install the following packages using anaconda ([Instructions](https://conda.io/docs/user-guide/tasks/manage-pkgs.html)): pandas, requests, beautifulsoup4, scrapy
### Workshop Content
1. **Software Setup**\
-Why do we use Python and Anaconda?\
-Basics in Python?\
-What is a Juypter Notebook?
2. **Introduction to web crawling** \
-HTML Basics and how websites linked \
-Types of Http Requests
3. **Low Level Web Crawling with requests and beautifulsoup**\
Using a real world example we will explore:\
-How to get a website with Python\
-How to extract information from a website\
-How to automatically walk through a website
4. **High Level Web Crawling with Scrapy** \
Now we will transfer the low level code from the previous section
into a simpler structure using the Package Scrapy.
5. **Discussion and Questions**
### Further Readings
In this section you will find a list of helpful resources to write more sophisticated web crawler.
- General Python Guide [Ref](http://docs.python-guide.org/en/latest/)
- Web Crawling Best Practices [Ref1](https://www.promptcloud.com/blog/web-scraping-best-practices), [Ref2](http://www.botscraper.com/blog/Follow-these-best-practices-while-web-crawling-and-scraping)
- Multithreading in Python [Ref](https://www.toptal.com/python/beginners-guide-to-concurrency-and-parallelism-in-python)
- Natural Language Processing [Ref](http://www.nltk.org/book/ch01.html)
- Python Data Science Handbook: Essential Tools for working with Data [Ref](https://smile.amazon.de/Python-Data-Science-Handbook-Essential/dp/1491912057/)
**Contact**\
In case you have an further topics to discus feel free to get in touch with me via [LinkedIn](https://www.linkedin.com/in/max-philipp-schrader/)