Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pavel401/jobsscraper
Jobs Scraper: Streamline job searching by scraping and storing job postings from various websites ina SQLite3 database using this Go application.
https://github.com/pavel401/jobsscraper
gin golang
Last synced: 29 days ago
JSON representation
Jobs Scraper: Streamline job searching by scraping and storing job postings from various websites ina SQLite3 database using this Go application.
- Host: GitHub
- URL: https://github.com/pavel401/jobsscraper
- Owner: Pavel401
- License: gpl-3.0
- Created: 2023-09-04T21:45:58.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-02-07T14:45:40.000Z (11 months ago)
- Last Synced: 2024-04-20T15:21:43.322Z (8 months ago)
- Topics: gin, golang
- Language: Go
- Homepage: https://jobs-scraper-production.up.railway.app/
- Size: 142 MB
- Stars: 8
- Watchers: 1
- Forks: 13
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Contributing: contributing.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Jobs Scraper
[](#contributors-)
## Table of Contents
- [Introduction](#introduction)
- [Mobile App](#mobile-app)
- [Prerequisites](#prerequisites)
- [Endpoints](#endpoints)
- [Contributing](#contributing)
- [License](#license)
- [Contributors](#contributors)
## Introduction
The Jobs Scraper is a powerful Go application designed for scraping job postings from a variety of websites and storing them in an SQLite database. Additionally, it offers endpoints for retrieving and synchronizing job postings, making it a valuable tool for job seekers and developers alike.
## Mobile App
Repo: https://github.com/Pavel401/JobScraper-Mobile
## Prerequisites
Before getting started with the Jobs Scraper, please ensure you have the following prerequisites installed:
- [Go Programming Language](https://golang.org/)
- [Gin Web Framework](https://github.com/gin-gonic/gin)
## How to setup
To run the application, you have to create a `.env` file. To configure you can set environment variables in the file. For example:
```env
SYNC_WITH_SQL_PASSWORD=password
```
```bash
go run main.go
```
## Endpoints
The Jobs Scraper package provides a set of powerful endpoints for scraping, managing, and synchronizing job postings. Here's a brief overview of the available endpoints:
- `/atlassian`, `/amazon`, `/coursera`, `/freshworks`, `/gojek`, `/mpl`: Scrapes job postings from specific websites.
- `/syncwithSql`: Synchronizes job postings with a SQL database.
- `/getallJobsFromSQL`: Retrieves job postings from a SQL database.
- `/`: Serves an HTML file (static/base.html).
## Contributing
Contributions to this project are highly encouraged! If you have ideas for improvements or come across any issues, please don't hesitate to open an issue or submit a pull request on the [GitHub repository](https://github.com/Pavel401/Jobs-Scraper).
This documentation aims to simplify and guide the way beginners make their first contribution. If you are looking to make your first contribution, follow the steps below.
#### If you don't have git on your machine install it.
## Fork this repository
Fork this repository by clicking on the fork button on the top of this page.
This will create a copy of this repository in your account.
## Open in Github Desktop
Click on the green icon indicating "code" and open your Github desktop.
## Clone the repository
Now clone the forked repository to your machine. Go to your GitHub account, open the forked repository, click on the code button and then click on open with Github Desktop ot you can click on _copy to clipboard_ icon if you want to use git bash.
**Note: Further command are for Git bash users not for the Git Desktop Users**
> For Git Desktop users ---> Now just click on Open with VS code and start your contribution.
Example:
## Commit Changes
After you have updated the files,click on 'Commit to main' and then click on push origin.
Example :
Now Come back to Github web and click on contribute to submit your changes for review.
***
## Steps for Git Bash Users
Open a terminal and run the following git command:
```
git clone "url you just copied"
```
where "url you just copied" (without the quotation marks) is the url to this repository (your fork of this project). See the previous steps to obtain the url.
For example:
```
git clone https://github.com/this-is-you/fossc.git
```
Here you're copying the contents of the first-contributions repository on GitHub to your computer.
## Create a branch
Change to the repository directory on your computer (if you are not already there):
```
cd desktop
```
Now create a branch using the `git checkout` command:
```
git checkout -b
```
For example:
```
git checkout -b add-new-file
```
(The name of the branch does not need to have the word _add_ in it, but it's a reasonable thing to include because the purpose of this branch is to add your name to a list.)
## Make necessary changes and commit those changes
Now open add or edit file in a text editor. Add code for any existing algorithm in other language or add some new algorithms. Make sure to update correspond README.md file if needed. Now, save the file.
If you go to the project directory and execute the command `git status`, you'll see there are changes.
Add those changes to the branch you just created using the `git add` command:
```
git add "name of the file you add or edit"
```
Now commit those changes using the `git commit` command:
```
git commit -m "Add message for the change"
```
## Push changes to GitHub
Push your changes using the command `git push`:
```
git push origin
```
replacing `` with the name of the branch you created earlier.
## Submit your changes for review
If you go to your repository on GitHub, you'll see a `Contribute` button. Click on that button.
click on `Open pull request`.
click on `Create pull request`.
click on `Create pull request`.
## Where to go from here?
Congrats! You just completed the standard _fork -> clone -> edit -> pull request_ workflow that you'll encounter often as a contributor!
## Contributors
[](#contributors)
Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
Mabud Alam
💻 🎨 🐛 🚧 👀
Debayan Ghosh
💻
Abhishek Parmar
📖 🎨 🤔
Pinak Dhir
📖
Tholkappiar M
💻
Hemant Chaudhary
💻
Abhinesh
💻
Harmeet Singh
💻 🔌 🛡️
Prapti Sharma
💻
## License
This project is licensed under the [GPL-3.0 license] License - see the [LICENSE](LICENSE) file for details.