https://github.com/scilifelabdatacentre/dc-dynamic
https://github.com/scilifelabdatacentre/dc-dynamic
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/scilifelabdatacentre/dc-dynamic
- Owner: ScilifelabDataCentre
- Created: 2021-02-19T13:14:40.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2024-03-05T08:51:56.000Z (about 1 year ago)
- Last Synced: 2024-03-05T09:53:17.131Z (about 1 year ago)
- Language: Python
- Size: 95.7 KB
- Stars: 2
- Watchers: 7
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
DC dynamic
==========
This repository contains scripts to be run as cron jobs.
`ACCESS_KEY` should be set for the container, defining the access key to use for Blobserver.
No persistent storage is provided, so any generated files should be uploaded somewhere to be saved.
### cron run every 10 minutes
#### freya_runner_every10mins.sh
Runs the scripts created by team Freya on `*/10 * * * *` (Every ten minutes)
Any files put in `$CODE_PATH/output` will be uploaded to Blobserver using the dc-dynamic account.
### cron run hourly
#### slack_runner_hourly.sh
Runs the script `slack_feeds.py` on `40 * * * *` (Every hour at 40th minute)
**slack_feeds.py** - Script to post message about new feeds (events/jobs) on SciLifeLab web to configured slack channel
### cron run daily
#### freya_runner_daily.sh
Runs the script created by team Freya on `0 7 * * *` (07:00 GMT).
Any files put in `$CODE_PATH/output` will be uploaded to Blobserver using the dc-dynamic account.
#### slack_runner_daily.sh
Runs the script `slack_figshare.py` on `0 6 * * *` (06:00 GMT)
**slack_figshare.py** - Script to post message about new items in figshare to configured slack channel
### cron run weekly
#### freya_runner_weekly.sh
Runs the script created by team Freya on `0 4 * * 5 ` (04:00 GMT every Friday morning).
Any files put in `$CODE_PATH/output` will be uploaded to Blobserver using the dc-dynamic account.
## Adding jobs
Add new commands to the above files to have them running on schedule. If you need jobs to run on a different schedule, talk to Sys admins or team Freya.
Make sure that the `requirements.txt` file in dc-dynamic contains all modules required for the native scripts (i.e. residing in dc-dynamic repo) **and** that any changes do not break the current code.
### Uploading files
Th easiest way to upload files is to use curl:
```bash
curl "https://blobserver.dc.scilifelab.se/blob/$filename" -H "x-accesskey: $ACCESS_KEY" --upload-file "$filename"
```
Feel free to set new variables, but remember to tell Sys admins or team Freya if you need them to be set for the cronjobs on the cluster.