Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/paulseperformance/cryptopanic_scraper
cryptopanic news feed scraper
https://github.com/paulseperformance/cryptopanic_scraper
chromedriver crypto jupyter notebook python python3 selenium selenium-python selenium-webdriver webscraper webscraping
Last synced: 3 months ago
JSON representation
cryptopanic news feed scraper
- Host: GitHub
- URL: https://github.com/paulseperformance/cryptopanic_scraper
- Owner: pAulseperformance
- License: mit
- Created: 2019-02-05T06:52:12.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2022-12-08T06:39:58.000Z (about 2 years ago)
- Last Synced: 2023-03-08T03:24:55.567Z (almost 2 years ago)
- Topics: chromedriver, crypto, jupyter, notebook, python, python3, selenium, selenium-python, selenium-webdriver, webscraper, webscraping
- Language: Jupyter Notebook
- Homepage: https://www.cryptopanic.com
- Size: 7.48 MB
- Stars: 13
- Watchers: 0
- Forks: 2
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
[![Contributors][contributors-shield]][contributors-url]
[![Forks][forks-shield]][forks-url]
[![Stargazers][stars-shield]][stars-url]
[![Issues][issues-shield]][issues-url]
[![MIT License][license-shield]][license-url]
[![LinkedIn][linkedin-shield]][linkedin-url]
Cryptopanic Scraper
Headless chromedriver for automatic scraping of cryptopanics asynchronous newsfeed.
Explore the docs »
Report Bug
·
Request Feature
## Table of Contents
* [About the Project](#about-the-project)
* [Built With](#built-with)
* [Getting Started](#getting-started)
* [Prerequisites](#prerequisites)
* [Installation](#installation)
* [Usage](#usage)
* [Roadmap](#roadmap)
* [Contributing](#contributing)
* [License](#license)
* [Contact](#contact)
## About The Project
[![Product Name Screen Shot][product-screenshot]](https://cryptopanic.com/)
Cryptopanic is a crypto news aggregator that offers realtime news feeds of all things crypto as well
as user input for ratings.
This project was designed to scrape the data from their website so it could be later analyzed using NLP.
### Built With
* [Python](https://github.com/topics/python)
* [Selenium](https://github.com/topics/selenium)
## Getting Started
To get a local copy up and running follow these simple steps.
### Prerequisites
* python 3
* pip
### Installation
1. Clone the cryptopanic_scraper
```sh
git clone https:://github.com/grilledchickenthighs/cryptopanic_scraper.git
```
2. Change directory
```sh
cd cryptopanic_scraper
```
3. Install packages
```sh
pip install -r requirements.txt
```
## Usage
Simply run:
```sh
python cryptopanic_scraper.py --headless
```
If you want to see it in action, run the script without any flags.
```sh
python cryptopanic_scraper.py
```
If you want to filter the type of news to scrape add the --filter flag and choose
a type. {all,hot,rising,bullish,bearish,lol,commented,important,saved}
```sh
python cryptopanic_scraper.py --filter hot
```
You can always use the --help flag if you forget these commands:
```sh
python cryptopanic_scraper.py --help
usage: cryptopanic_webdriver.py [-h] [-v]
[-f {all,hot,rising,bullish,bearish,lol,commented,important,saved}]
[-s]
optional arguments:
-h, --help show this help message and exit
-v, --verbose increase output verbosity
-f {all,hot,rising,bullish,bearish,lol,commented,important,saved}, --filter {all,hot,rising,bullish,bearish,lol,commented,important,saved}
Type of News filter
-s, --headless Run Chrome driver headless
```
If your interested in analyzing the data:
Please feel free to check out the [jupyter](https://github.com/GrilledChickenThighs/cryptopanic_scraper/tree/master/jupyter) directory for getting started.
## Roadmap
See the [open issues](https://github.com/grilledchickenthighs/cryptopanic_scraper/issues) for a list of proposed features (and known issues).
## Contributing
Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**.
1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the Branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
## License
Distributed under the MIT License. See `LICENSE` for more information.
## Contact
[Paul Mendes](https://grilledchickenthighs.github.io/) - [@BTCTradeNation](https://twitter.com/BTCTradeNation) - [[email protected]](mailto:[email protected])
Project Link: [https://github.com/grilledchickenthighs/cryptopanic_scraper](https://github.com/grilledchickenthighs/cryptopanic_scraper)
[contributors-shield]: https://img.shields.io/github/contributors/grilledchickenthighs/cryptopanic_scraper?style=flat-square
[contributors-url]: https://github.com/GrilledChickenThighs/cryptopanic_scraper/graphs/contributors
[forks-shield]: https://img.shields.io/github/forks/grilledchickenthighs/cryptopanic_scraper?style=flat-sqaure
[forks-url]: https://github.com/GrilledChickenThighs/cryptopanic_scraper/network/members
[stars-shield]: https://img.shields.io/github/stars/grilledchickenthighs/cryptopanic_scraper?style=flat-square
[stars-url]: https://github.com/grilledchickenthighs/cryptopanic_scraper/stargazers
[issues-shield]: https://img.shields.io/github/issues/grilledchickenthighs/cryptopanic_scraper.svg?style=flat-square
[issues-url]: https://github.com/grilledchickenthighs/cryptopanic_scraper/issues
[license-shield]: https://img.shields.io/github/license/grilledchickenthighs/cryptopanic_scraper.svg?style=flat-square
[license-url]: https://github.com/grilledchickenthighs/cryptopanic_scraper/blob/master/LICENSE.txt
[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=flat-square&logo=linkedin&colorB=555
[linkedin-url]: https://linkedin.com/in/paul-mendes
[product-screenshot]: images/screenshot.png