https://github.com/fairdataihub/knowmore
Automated Knowledge Discovery Tool for SPARC Datasets
https://github.com/fairdataihub/knowmore
clustering codeathon data data-science fair hackathon knowledge knowledge-graph machine-learning nlp
Last synced: 5 months ago
JSON representation
Automated Knowledge Discovery Tool for SPARC Datasets
- Host: GitHub
- URL: https://github.com/fairdataihub/knowmore
- Owner: fairdataihub
- License: mit
- Created: 2021-09-21T17:35:10.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-05-22T00:41:28.000Z (about 1 year ago)
- Last Synced: 2024-05-22T22:08:16.886Z (about 1 year ago)
- Topics: clustering, codeathon, data, data-science, fair, hackathon, knowledge, knowledge-graph, machine-learning, nlp
- Language: Jupyter Notebook
- Homepage: https://fairdataihub.org/knowmore
- Size: 15.8 MB
- Stars: 3
- Watchers: 2
- Forks: 1
- Open Issues: 15
-
Metadata Files:
- Readme: README.flask.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Overview
# Prerequisites
We recommend using Anaconda to create and manage your development environments for KnowMore. All the subsequent instructions are provided assuming you are using [Anaconda (Python 3 version)](https://www.anaconda.com/products/individual).
# Setup
## Clone repo
Clone the repo and submodules
```
git clone https://github.com/SPARC-FAIR-Codeathon/KnowMore.git --recurse
```
## Setup Flask
### cd into the root folder of this repo
Open Anaconda prompt (Windows) or the system Command line interface then naviguate to the KnowMore folder
```sh
$ cd ./KnowMore
```
### Setup conda env
```sh
$ conda create -n "knowmore-flask-env" python=3.6
$ conda activate knowmore-flask-env
```
### Install Python dependencies
```sh
$ conda install pip
$ pip install -r requirements.txt
```
### Setup env vars
The environment variables required are listed in the table below along with information on how to get them
Suggested name
Value or instructions for obtaining it
Purpose
FLASK_ENV
"development"
Prod or dev
SERVER_NAME
"localhost:5000"
server url
LISTEN_PORT
"5000"
Port for flask app to listen to
CLIENT_URL
"http://localhost:3000"
Sparc-App url
OSPARC_TEST_MODE
false
whether to use test mode, so you don't have to contact osparc to develop your frontend
OSPARC_API_KEY
Contact osparc support
Sending jobs to osparc
OSPARC_API_SECRET
Contact osparc support
Sending jobs to osparc
SECRET_KEY
flask secret key
Each of them can be set in your conda environment as follows
```sh
$ conda env config vars set MY_VAR=value1 MY_OTHER_VAR=value2
```
### start flask server
```sh
$ flask run
```
or if you require remote access: (NOTE untested)
```sh
$ flask run --host=0.0.0.0
```
### View your flask app
http://127.0.0.1:5000/
## Test mode
Want to develop without starting the osparc jobs?
set env var (to anything other than string 'false')
```
OSPARC_TEST_MODE=true
```
This will make it so you don't actually contact osparc, but instead receive sample data back. Helpful for debugging frontend without having to wait for osparc job everytime.
## Run in Docker
### 1) Install Docker
[See Docker's official documentation](https://docs.docker.com/get-docker/).
### 2) Create image and start container
```
docker-compose up -d
```
Note that this will use the env vars as specified in your .env file. Make sure you still set those before building the image.
### 3) Check Container Status
```
docker ps
```
Should get something like this:
```
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
368be64c60c1 knowmore_flaskapp "/entrypoint.sh /sta…" 10 seconds ago Up 9 seconds 80/tcp, 443/tcp, 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp knowmore-flask-web-app
```
Note that this container is exposing port 5000 (the port where flask is listening) to your host.
### 4) Check and follow logs
```
docker logs knowmore-flask-web-app -f
```
### 5) Test the endpoints
- Using browser or `curl` from your host, try out `http://localhost:5000/`.
* Response should be: `status: up`
- Get a sample image: `http://localhost:5000/api/results-images/example-job-id/Plots-PlotID-3.7.png`
* Flask should return the image file.
# Deploy
```
git push heroku main
```
# TODOs
- use production server, rather than dev server
- build a new docker image (current one is outdated)
- upload all files to s3 instead of to local filesystem (especially due to the nature of [Heroku's ephemeral filesystem](https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem))
# Debugging
## Helpful scripts
### Test connection to osparc
```
curl http://127.0.0.1:5000/api/check-osparc-job/123e4567-e89b-12d3-a456-426614174000
# example response: {"error": "{\"errors\":[\"project 123e4567-e89b-12d3-a456-426614174000 not found\"]}", "status_code": 500}
```
### Test out the python methods without using frontend
```
python3 manual-job-starter.py
```
To not create job, but use existing job, pass in a single arg with job uuid of python job (NOT THE MATLAB JOB ID)
e.g.,
```
python3 manual-job-starter.py e9012487-5ff1-4112-aa4f-8165915973fa
```