https://github.com/miliar/tech-stack-demand

An example microservice pipeline to visualize the tech stack demand on the current job market
https://github.com/miliar/tech-stack-demand

apache-kafka data-pipeline docker docker-compose example flask html-css-javascript indeed jinja2 jobsearch jobseeker kafka microservices neo4j python python3 redis scraper stackexchange tech-stacks

Last synced: 3 months ago
JSON representation

An example microservice pipeline to visualize the tech stack demand on the current job market

• Host: GitHub
• URL: https://github.com/miliar/tech-stack-demand
• Owner: miliar
• Created: 2019-08-23T08:40:44.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2019-09-22T14:27:53.000Z (about 6 years ago)
• Last Synced: 2025-04-09T22:45:03.530Z (6 months ago)
• Topics: apache-kafka, data-pipeline, docker, docker-compose, example, flask, html-css-javascript, indeed, jinja2, jobsearch, jobseeker, kafka, microservices, neo4j, python, python3, redis, scraper, stackexchange, tech-stacks
• Language: Python
• Homepage:
• Size: 16.9 MB
• Stars: 6
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Tech stack demand

An example microservice pipeline to visualize the tech stack demand on the current job market.

#### Show who is using certain technologies:

![Tech stack demand tech demo](demo_pictures/search_by_tech.gif)

#### Search for company:

![Tech stack demand company demo](demo_pictures/search_by_company.gif)

## Installation

* [Install docker/docker-compose](https://docs.docker.com/compose/install/)

* Assure docker has enough resources (Preferences > Advanced; tested with 4 CPUs and 6 GiB RAM) 

* Run `docker-compose up`

* Open in your browser http://localhost:5000/

* Collect new data (reload page to see progress; this can take up to 20 min)

![Tech stack demand collect demo](demo_pictures/collect_new_data.gif)

## How to use the UI

* Check out this [basic action wiki](https://github.com/Nhogs/popoto/wiki/Basic-action)

## Technical sidenotes

### Architecture 

![Tech stack demand architecture](demo_pictures/architecture.png)

#### Browse kafka topics:

* Kafka logs are not mounted

* While collecting new data, check http://localhost:9000/

#### Manage Neo4j data:

* Data mounted to /neo4j_data_loader/neo4j_data

* Open Neo4j browser: http://localhost:7474/

* Login with user: neo4j password: password

#### Run tests:

* `docker-compose -f docker-compose-tests.yml up`

#### Run keywords api update:

* Delete old data saved in `keywords_api/redis_data`

* Run: `docker-compose -f docker-compose-keywords-update.yml up`