https://github.com/euiyounghwang/spark_job_interface_service
spark_job_interface_service
https://github.com/euiyounghwang/spark_job_interface_service
fastapi spark spark-cluster spark-jobs
Last synced: 3 months ago
JSON representation
spark_job_interface_service
- Host: GitHub
- URL: https://github.com/euiyounghwang/spark_job_interface_service
- Owner: euiyounghwang
- Created: 2024-05-28T16:44:45.000Z (about 1 year ago)
- Default Branch: master
- Last Pushed: 2024-10-21T21:16:46.000Z (8 months ago)
- Last Synced: 2025-01-17T19:55:23.909Z (5 months ago)
- Topics: fastapi, spark, spark-cluster, spark-jobs
- Language: Python
- Homepage:
- Size: 140 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# spark_job_interface_service
spark_job_interface_service
Apache Spark is an open-source unified analytics and data processing engine for big data. Its capabilities include near real-time or in-batch computations distributed across various clusters.
Simply put, a Spark Job is a single computation action that gets instantiated to complete a Spark Action.
- This repository searches the list of spark jobs running in the spark cluster currently in service.
- Additionally, they are created in Excel and downloaded.
### Using Poetry: Create the virtual environment in the same directory as the project and install the dependencies:
```bash
python -m venv .venv
source .venv/bin/activate
pip install poetry
# --
poetry config virtualenvs.in-project true
poetry init
poetry add fastapi
poetry add uvicorn
poetry add pytz
poetry add httpx
poetry add requests
poetry add xlsxwriter
poetry add openpyxl
```
or you can run this shell script `./create_virtual_env.sh` to make an environment. then go to virtual enviroment using `source .venv/bin/activate`
### Register Service
- sudo service sparkjob_interface_api status/stop/start/restart
```bash
#-- /etc/systemd/system/sparkjob_interface_api.service
[Unit]
Description=SparkJob Interface Service
[Service]
User=devuser
Group=devuser
Type=simple
ExecStart=/bin/bash /home/devuser/sparkjob_interface_api/service-start.sh
ExecStop= /usr/bin/killall sparkjob_interface_api
[Install]
WantedBy=default.target
# Service command
sudo systemctl daemon-reload
sudo systemctl enable sparkjob_interface_api.service
sudo systemctl start sparkjob_interface_api.service
sudo systemctl status sparkjob_interface_api.service
sudo systemctl stop sparkjob_interface_api.service
sudo service sparkjob_interface_api status/stop/start
```
### Run Custom Promethues Exporter
- Run this command : $ `http://localhost:8003/docs`