https://github.com/gr1d99-ke/simple_links_scraper
Extract links from any web application
https://github.com/gr1d99-ke/simple_links_scraper
learning-by-doing ruby ruby-on-rails scraper
Last synced: 2 months ago
JSON representation
Extract links from any web application
- Host: GitHub
- URL: https://github.com/gr1d99-ke/simple_links_scraper
- Owner: gr1d99-ke
- License: mit
- Created: 2019-11-11T09:12:09.000Z (over 6 years ago)
- Default Branch: develop
- Last Pushed: 2023-01-19T15:48:02.000Z (over 3 years ago)
- Last Synced: 2025-04-09T20:19:39.437Z (about 1 year ago)
- Topics: learning-by-doing, ruby, ruby-on-rails, scraper
- Language: Ruby
- Homepage:
- Size: 178 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](http://hits.dwyl.io/gr1d99/my-scraper)
# simple_links_scraper
Is a simple Ruby on Rail web application that attempts to extract all links in any given web application __(except web apps that require javascript)__.
The application only extracts links of the given url, this means it will not follow identified links that it finds.
## Getting Started
These instructions will get you a copy of the project up and running on your local machine
### Prerequisites
What things you need to install the software and how to install them
- Ruby-v2.5.1p57
- Rails-v5.2.1
- Postgresql
### Installing
A step by step series of examples that tell you how to get a development env running
1. Clone this repo
2. `cd` to `my-scraper` directory
3. Install dependencies `bundle install`
4. Setup figaro `figaro install`
5. Setup some environment variables in _config/application.yml_ file
```bash
DOMAIN: [your-domain]
SMTP_USER_NAME: [your gmail address]
SMTP_PASSWORD: [your gmail password]
```
__NB: replace everything that is inside []__
6. Prepare database `rake db:setup`
7. Migrate database `rake db:migrate`
8. Start sidekiq `bundle exec sidekiq -q default -q mailers`
9. Start development server `rails s` :sunglasses:
### Docker Folks
1. Install [docker-compose](https://docs.docker.com/compose/install/)
2. Run `docker-compose build`
3. Create Database `docker-compose run app bundle rake db:create`
4. Run migrations `docker-compose run app bundle exec rake db:migrate`
5. Start your app `docker-compose up`
6. Visit [http://localhost:3000](http://localhost:3000)
## Running the tests
`bundle exec rspec`
## Built With
* [Ruby](https://www.ruby-lang.org/en/) - The programming language
* [Rails](https://rubyonrails.org/) - Web framework
* [Postgres](https://www.postgresql.org/) - Database
* [Sidekiq](https://sidekiq.org/) - For background jobs
* [Faraday](https://github.com/lostisland/faraday) - For external HTTP requests
* [Redis](https://redis.io/) - Fast data store
## Authors
* **Gideon Kimutai**
## License
This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details