https://github.com/adhityaramadhanus/inkscraper

Linkedin Jobs Scraper using nodejs,expressjs and mongodb as storage
https://github.com/adhityaramadhanus/inkscraper

expressjs linkedin linkedin-scraper mongodb nodejs

Last synced: 22 days ago
JSON representation

Linkedin Jobs Scraper using nodejs,expressjs and mongodb as storage

• Host: GitHub
• URL: https://github.com/adhityaramadhanus/inkscraper
• Owner: AdhityaRamadhanus
• Created: 2016-06-02T14:57:15.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2024-01-06T14:10:52.000Z (10 months ago)
• Last Synced: 2024-04-15T11:39:26.659Z (7 months ago)
• Topics: expressjs, linkedin, linkedin-scraper, mongodb, nodejs
• Language: JavaScript
• Size: 899 KB
• Stars: 13
• Watchers: 2
• Forks: 4
• Open Issues: 29
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# inkscraper

[![Build Status](https://travis-ci.org/AdhityaRamadhanus/inkscraper.svg?branch=master)](https://travis-ci.org/AdhityaRamadhanus/inkscraper) [![Dependencies Status](https://david-dm.org/adhityaramadhanus/inkscraper.svg)](https://david-dm.org/adhityaramadhanus/inkscraper) [![Code Style](https://img.shields.io/badge/code%20style-standard-green.svg)](https://github.com/feross/standard)



  Installation |

  Documentation |

  License

  



  


  inkscraper is jobs scraper for linkedin that comes with restful api and full-text search.


  Scraping linkedin jobs can be considered an infringement of linkedin TOS, use it carefully.

  inkscraper currently supports:

  - Scrap Listing Page (Job listing, by default this will scrape https://www.linkedin.com/jobs/view-all)

  - Scrap Details Page (Job Details Page)

  - Restful API for jobs scraped from linkedin

  - Full-text search using built-in mongoose (of course built-in mongodb too)

  


Installation

------------

* git clone https://github.com/AdhityaRamadhanus/Linkedin-Scraper.git

* cd Linkedin-Scraper

* npm install

* npm run start-apiserver

* npm run start-scraper

* set .env files (i'm using dotenv, see here https://www.npmjs.com/package/dotenv for documentation)

* Example of .env

```js

NODE_ENV=development

MONGOLAB_URI='mongodb://localhost:27017/linkedin-scraper'

APIDOC=true

```

Documentation

------------

* npm install -g apidoc

* cd Linkedin-Scraper

* npm run gen-doc

* add APIDOC=true in .env

* enjoy, documentation can be found in "/apidoc"

Known Problems

------------

Like i said, scraping linkedin jobs can be considered an infringement of linkedin TOS so sometime you may get 999 status code even if you run this from your local computer

License

----

MIT © [Adhitya Ramadhanus]