Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/adhityaramadhanus/inkscraper

Linkedin Jobs Scraper using nodejs,expressjs and mongodb as storage
https://github.com/adhityaramadhanus/inkscraper

expressjs linkedin linkedin-scraper mongodb nodejs

Last synced: 2 months ago
JSON representation

Linkedin Jobs Scraper using nodejs,expressjs and mongodb as storage

Host: GitHub
URL: https://github.com/adhityaramadhanus/inkscraper
Owner: AdhityaRamadhanus
Created: 2016-06-02T14:57:15.000Z (over 8 years ago)
Default Branch: master
Last Pushed: 2024-01-06T14:10:52.000Z (12 months ago)
Last Synced: 2024-04-15T11:39:26.659Z (9 months ago)
Topics: expressjs, linkedin, linkedin-scraper, mongodb, nodejs
Language: JavaScript
Size: 899 KB
Stars: 13
Watchers: 2
Forks: 4
Open Issues: 29
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# inkscraper
[![Build Status](https://travis-ci.org/AdhityaRamadhanus/inkscraper.svg?branch=master)](https://travis-ci.org/AdhityaRamadhanus/inkscraper) [![Dependencies Status](https://david-dm.org/adhityaramadhanus/inkscraper.svg)](https://david-dm.org/adhityaramadhanus/inkscraper) [![Code Style](https://img.shields.io/badge/code%20style-standard-green.svg)](https://github.com/feross/standard)

Installation |
Documentation |
License

inkscraper is jobs scraper for linkedin that comes with restful api and full-text search.

Scraping linkedin jobs can be considered an infringement of linkedin TOS, use it carefully.

inkscraper currently supports:

- Scrap Listing Page (Job listing, by default this will scrape https://www.linkedin.com/jobs/view-all)
- Scrap Details Page (Job Details Page)
- Restful API for jobs scraped from linkedin
- Full-text search using built-in mongoose (of course built-in mongodb too)

Installation
------------

* git clone https://github.com/AdhityaRamadhanus/Linkedin-Scraper.git
* cd Linkedin-Scraper
* npm install
* npm run start-apiserver
* npm run start-scraper
* set .env files (i'm using dotenv, see here https://www.npmjs.com/package/dotenv for documentation)
* Example of .env
```js
NODE_ENV=development

MONGOLAB_URI='mongodb://localhost:27017/linkedin-scraper'
APIDOC=true
```

Documentation
------------

* npm install -g apidoc
* cd Linkedin-Scraper
* npm run gen-doc
* add APIDOC=true in .env
* enjoy, documentation can be found in "/apidoc"

Known Problems
------------
Like i said, scraping linkedin jobs can be considered an infringement of linkedin TOS so sometime you may get 999 status code even if you run this from your local computer

License
----