Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/guardian/pluto-deliverables

Interface that allows for storage and retrieval of finished media products, masters, audio stems, post scripts etc.
https://github.com/guardian/pluto-deliverables

multimedia-tech

Last synced: 2 months ago
JSON representation

Interface that allows for storage and retrieval of finished media products, masters, audio stems, post scripts etc.

Awesome Lists containing this project

README 

        
# pluto-deliverables



An interface that allows for storage and retrieval of finished media

products, masters, audio stems, post scripts etc.



## Running in prexit-local minikube (recommended)

0. Download and set up prexit-local, carefully following the instructions there.

   

1. Build the frontend:

```bash

$ cd frontend

$ yarn install

$ yarn dev

```



This will put webpack into watching mode, rebuilding a dev (debugging) build

of the app bundle every time the frontend source changes.



2. Build the docker image.  You must have minikube set up and running in order for this to work:

```bash

$ cd ${repo-root}

$ eval $(minikube docker-env)

$ docker build . -t guardianmultimedia/pluto-deliverables:DEV

```



3. Check if you already have an instance of pluto-deliverables running:

```bash

$ kubectl get pods | grep deliverables

```



4a. If you do have one running, then check that it is pointing to the "DEV" image:

```bash

$ kubectl describe pod ${podname} | grep -A 5 Image

```



- If you see a number instead of :DEV at the end of the image name, then you must edit the manifest in pluto-dev to change

the `image:` line to `guardianmultimedia/pluto-deliverables:DEV` and add `imagePullPolicy: Never` just below it if that 

is not present already.  Then redeploy as in step 4b.

- If you see "DEV" at the end then you simply need to delete the existing pod:

```bash

$ kubectl delete pod ${podname}

```

and the system will restart with the new "DEV" image you built in step 2.



4b. If you don't have a pod running, or you had to change the manifest in step 4b, you need to redeploy the pluto-deliverables

manifests.  These live in the `kube/pluto-deliverables` directory _of prexit-local_.

```bash

$ cd ${prexit-local}/kube/pluto-deliverables

$ kubectl apply -f .

$ kubectl rollout status deployment/pluto-deliverables

```



Repeat the build image - delete pod steps each time you make a change to see it in the browser.  Remember that you may

need to use the browser tools to temporarily disable cache in order to get the latest javascript code.



## Running locally



pluto-deliverables is a django project, so the usual method of setting up

a django project applies; but there are a couple of specific twists.



### Frontend

The frontend is a standard React/webpack build. The final bundle is directly

output into the django static assets folder at gnm_deliverables/static/app.js.

```bash

$ cd frontend

$ yarn install

$ yarn dev

```



This will put webpack into watching mode, rebuilding a dev (debugging) build

of the app bundle every time the frontend source changes.



### Backend

Once you have the project checked out, you'll need to set up your Python

environment.  The app is developed in Python 3.8 but any recent python

should do.

1. Set up a virtualenv, normally this is put into a venv/ directory in the

project root.  Gitignore is already set up on this folder. Point to the base

python interpreter you want to use.

    ```bash

    $ virtualenv --python=python3.8 venv/

    ```

2. Activate your virtualenv:

    ```bash

    $ source venv/bin/activate

    (venv) $

    ```

3. Install requirements:

    ```bash

    (venv) $ pip -r requirements.txt

    ```

4. **IMPORTANT** pluto-deliverables depends on a library called gnmvidispine

that is linked in as a submodule.  You need to check this out and

install it into your virtualenv too:

    ```bash

    (venv) $ git submodule init

    (venv) $ git submodule update

    (venv) $ cd gnmvidispine

    (venv) $ pip -r requirements.txt

    (venv) $ python ./setup.py install

    (venv) $ cd ..

    ```



5. You need to have a database running in order to use the app:

    ```bash

    $ cd scripts/

    $ ./setup_docker_postgres.sh 

   ```

   This will set up a postgres instance running on port 5432 with appropriate default-configured database, username and password.

   It will fail if you already have something on port 5432; in this case, you can adjust the port in the script and

   set an environment variable to tell the app where to find it. See the `DATABASES` section of `settings.py` for the

   names of the variables to set

   

6. Now you need to make sure that the database is in-sync:

    ```bash

    (venv) $ ./manage.py migrate

    ```



7. Now you are ready to run the dev server:

    ```bash

   (venv) $ ./manage.py runserver 0.0.0.0:9000 

   ```

   If you see an error about a library not found, check that gnmvidispine is 

   correctly installed in your virtualenv as per stage 4.



### Local admin user



When you try to access anything as above, you'll find you get permission denied.

Proper (bearer-token) auth is not configured yet, so the easiest way is to create

a local superuser in the database (e.g. admin/admin):

```bash

(venv) $ ./manage.py createsuperuser

```

In order to log in, with the dev server running, go to `http://localhost:9000/admin` and

enter the credentials.  This gets you to a django admin screen, but also gives

you a session which your browser should then be able to use to access all the endpoints.

Just take your browser back to `http://localhost:9000/` and it should work.



You can use basic-auth for REST only access, e.g.

```bash

$ curl http://localhost:9000/api/bundle/new -X POST -d '{"name":"fred","project_id":12345}' -u admin:admin

```



### Hey, the tests are broken!

You need to set the environment variable CI (to anything, but conventionally 1) in order for the tests to work.

This is because normally the app attempts to connect to rabbitmq at startup and fails if it isn't there.  With CI

set in the environment, this stage is skipped.



### Using bearer auth token locally

1. Set up the [prexit-local](https://gitlab.com/codmill/customer-projects/guardian/prexit-local) development environment

2. Get the key that was generated when Keycloak was set up and save the blob with a PEM header/footer on it like this:

    ```

    -----BEGIN CERTIFICATE-----

    {paste-blob-here}

    -----END CERTIFICATE-----

    ```

   Add this file to the app root i.e.`/gnm_deliverables/{certificate_name}.pem`.

   

   Make sure the `JWT_VALIDATION_KEY` variable has the correct path to the PEM file.

3. Make sure that Docker is pointing to minikube before you build the container 

    ```bash

    $ eval $(minikube docker-env)  

    ```

    Build the docker container: 

    ```bash

    $ docker build . -t guardianmultimedia/pluto-deliverables:DEV

    ```

4. Go to ``https://prexit.local/deliverables/`` 



## Other setup



In order to ingest and proxy media, gnm-deliverables relies on Vidispine.

The following environment variables configure the Vidispine settings, with defaults supplied in `settings.py`:

 - VIDISPINE_URL - location that the _backend_ expects to find Vidispine. In a containerised Kubernetes environment this

 is often different to where a frontend component would expect to find it

 - VIDISPINE_USER - user account for backend access. Admin rights are required; `run_as` is used internally to apply a

 user's permissions to a VS request when performing a request that will be relayed abck to a client

 - VIDISPINE_PASSWORD - password for the user account

 - CALLBACK_ROOT - this is the base url where Vidispine should find the backend, in order to send notifications.  In a

 containerised Kubernetes environment this is normally the name of the service that is fronting the backend (e.g. if the

 backend is deployed behind a service called "pluto-deliverables" on port 9000, then this would be

  `http://pluto-deliverables:9000`)

  

There is another Vidispine-relevant setting in the settings file, which is not configurable in the environment:

 - TRANSCODE_PRESETS - this is a mapping table from media MIME type (expressed as a regex) to required transcode preset.

 

The following environment variables configure the RabbitMQ settings.  The software _works_ without rabbitmq, but you get

a lot of errors cluttering up the logs:

 - RABBITMQ_HOST - hostname that the backend can find rabbitmq at

 - RABBITMQ_PORT - port that rabbitmq is serving amqp on. Defaults to 5672, shouldn't need to change it

 - RABBITMQ_VHOST - rabbitmq virtual host to communicate with

 - RABBITMQ_EXCHANGE - exchange name that we should publish messages to

 - RABBITMQ_USER - user name to access the server with. Must be able to declare exchanges and write data

 - RABBITMQ_PASSWD - password associated with RABBITMQ_USER



## CDS integration



We use a system called the Content Delivery System ([CDS](https://github.com/guardian/content-delivery-system)) for

performing syndication tasks to 3rd party clients.



In order that the user can receive feedback, and endpoint is provided that can receive log updates 

(api/bundle//asset///logupdate) and a CDS method is implemented that sends updates

to this endpoint (notify_pluto_deliverables.rb).  In order to keep communication secure, Basic Auth is implemented (it's

assumed that pluto-deliverables is running behind HTTPS); so the `notify_pluto_deliverables` setup requires a username 

and password.



These can be provided by connecting to any pluto-deliverables instance (e.g. `kubectl -n namespace get pods | grep deliverables`

 to get the pod names then `kubectl exec -n namespace -it {pod-name} -- /bin/sh` to connect) and running from within the

 pod:

 ```

$ ./manage.py create_logs_user {username}

we started up!

Create user cds with password {some-password}. This password won't be shown again, so make sure you make a note of it!

$

```

where `{username}` is the name of the user you want to create.  Conventionally `cds` is used.

If this user exists already, the password is reset; if it does not exist then it is created.

You can't specify the password yourself, a randomised password is generated and shown on the console. Take this and put

it into your CDS configuration so that the routes can communicate with pluto-deliverables.