https://github.com/runwayml/processing-library
RunwayML-for-Processing
https://github.com/runwayml/processing-library
java machine-learning processing runwayml
Last synced: 5 months ago
JSON representation
RunwayML-for-Processing
- Host: GitHub
- URL: https://github.com/runwayml/processing-library
- Owner: runwayml
- License: other
- Archived: true
- Created: 2019-07-19T14:05:44.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2020-04-11T22:18:21.000Z (over 5 years ago)
- Last Synced: 2025-04-07T18:11:46.427Z (6 months ago)
- Topics: java, machine-learning, processing, runwayml
- Language: Java
- Homepage: https://runwayml.github.io/processing-library/
- Size: 30.2 MB
- Stars: 137
- Watchers: 14
- Forks: 24
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: license.txt
Awesome Lists containing this project
README
# RunwayML library for Processing
A library to easily use [RunwayML](https://runwayml.com) with [Processing](https://processing.org/).
## Prerequisites
* [RunwayML](https://runwayml.com/): Download the latest release of [RunwayML](https://runwayml.com/download) and sign up for an account. Visit our [installation guide](https://learn.runwayml.com/#/getting-started/installation) for more details. If you encounter any issues installing RunwayML, feel free to contact us through the [support page](https://support.runwayml.com).
* [Processing](https://processing.org/): Processing version 3.0 or greater is required.
## Installation
1. Download **[RunwayML.zip](https://github.com/runwayml/processing-library/releases/download/latest/RunwayML.zip)**
2. Unzip into **Documents > Processing > libraries**
3. Restart Processing (if it was already running)
[đź“˝ Watch How to Install and Use the RunwayML Processing Library](https://www.youtube.com/watch?v=zGdOKaLOjck&list=PLj598ZXODDO_oWYAiO5c0Ac05IyrPUG8t&index=6&t=0s)
## Testing installation
Before using the library please make sure RunwayML is started and is running one model.
You can test by opening your browser and navigating to [http://localhost:8000/info](http://localhost:8000/info)
to view info about your model.
If you see a JSON string with inputs and outputs information use the Basic Example below !
Feel free to test the Basic Example below.
## Basic Example
For this example run the im2txt model.
This example will print information from RunwayML (e.g. running im2txt)
```processing
// import Runway library
import com.runwayml.*;
// reference to runway instance
RunwayHTTP runway;
void setup() {
// setup Runway HTTP connection using default host (localhost) and port (8000)
runway = new RunwayHTTP(this);
}
void draw() {
}
// this is called when new Runway data is available
void runwayDataEvent(JSONObject runwayData){
println(runwayData);
}
// this is called each time Processing connects to Runway
// Runway sends information about the current model
public void runwayInfoEvent(JSONObject info){
println(info);
}
// if anything goes wrong
public void runwayErrorEvent(String message){
println(message);
}
```
Notice the text running displayed in console.
What if we want to display an image from RunwayML ?
Let's stop im2txt and use CygleGAN.
Notice on info event details about the model: the output in particular is an image with the key "image".
(You can see the information within RunwayML on the top right Network area of the Model).
Images to/from RunwayML are Base64 encoded and the library's `ModelUtils` helps convert between this format and the familiar `PImage` format.
With the CycleGAN model running we can tweak the above example to display an image received from RunwayML:
```processing
// import Runway library
import com.runwayml.*;
// reference to runway instance
RunwayHTTP runway;
// reference to image received
PImage resultImage;
void setup() {
// setup Runway HTTP connection using default host (localhost) and port (8000)
runway = new RunwayHTTP(this);
}
void draw() {
// if there's an image received:
if(resultImage != null){
// display the image
image(resultImage,0,0);
}
}
// this is called when new Runway data is available
void runwayDataEvent(JSONObject runwayData){
// try to access "image" key and convert the received that to a PImage
try{
String base64Image = runwayData.getString("image");
resultImage = ModelUtils.fromBase64(base64Image);
}catch(Exception e){
// print an error message if the above fails
println("error parsing received data: " + runwayData);
e.printStackTrace();
}
}
// this is called each time Processing connects to Runway
// Runway sends information about the current model
public void runwayInfoEvent(JSONObject info){
println(info);
}
// if anything goes wrong
public void runwayErrorEvent(String message){
println(message);
}
```
## Existing Examples
You can access the following examples via **Processing > Examples > Contributed Libraries > RunwayML**
- AdaIN-Style-Transfer 
(content image: "Still Life with fruit dish" by Paul CĂ©zanne,
style image: "Fruit Dish" by Georges Braques)
- Adaptive-Style-Transfer 
(content image: "Fruit Dish" by Georges Braques, inference using CĂ©zanne (landscapes) style)
- Arbitrary-Image-Stylization 
(content image: "Girl with a Mandolin" by Pablo Picasso,
style image: "Man with a guitar" by Georges Braques)
- AttnGAN:
- BigGAN: 
- cnOCR: 
- COCO-SSD: 
(content image: "The Card Players" by Paul CĂ©zanne)
- CycleGAN: 
(content image: "Fruit Dish" by Georges Braques)
- DeepLab: 
(content image: "The Card Players" by Paul CĂ©zanne)
- DeepLabV3: 
(content image: "The Card Players" by Paul CĂ©zanne)
- DeepPrivacy: 
(image credits: "Portrait of Ada Lovelace" by Margaret Sarah Carpenter)
- DenseCap: 
(content image: "The Card Players" by Paul CĂ©zanne)
- DenseDepth: 
(content image: "The Card Players" by Paul CĂ©zanne)
- DensePose: 
(image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)
- DeOldify: 
(content image: Grace Hopper, image credits: Courtesy of the Computer History Museum, source: https://news.yale.edu/2017/02/10/grace-murray-hopper-1906-1992-legacy-innovation-and-service)
- DeRaindrop: 
(content image: "Sudden shower over Shin OĚ„hashi bridge and Atake" by Hiroshige)
- EdgeConnect: 
(content image: "Self-Portrait with Striped Shirt" by Egon Schiele)
- ESRGAN: 
(image credits: SuSuMa original source: https://en.wikipedia.org/wiki/White_tiger#/media/File:White-tiger-2407799_1280.jpg)
- Face-Landmarks: 
(content image: "Celestina" by Pablo Picasso)
- Face-Parser: 
(content image: "Celestina" by Pablo Picasso)
- Face-Recognition: 
(image from 12 Angry Men (1957) directed by Sidney Lumet)
- Fast-Photo-Style 
(content image: "Still Life with fruit dish" by Paul CĂ©zanne,
style image: "Fruit Dish" by Georges Braques)
- Fast-Style-Transfer: 
(content image: "Nude Descending a Staircase, No. 2" by Marcel Duchamp)
- GLOW: 
(content image: "Self-Portrait with Striped Shirt" by Egon Schiele)
- GPT-2: 
- im2txt: 
(content image: "The Card Players" by Paul CĂ©zanne)
- Image-Inpainting-GMCNN: 
(content image: "Study after Velázquez's Portrait of Pope Innocent X" by Francis Bacon)
- Image-Super-Resolution: 
(image credits: SuSuMa original source: https://en.wikipedia.org/wiki/White_tiger#/media/File:White-tiger-2407799_1280.jpg)
- MaskRCNN: 
(image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)
- MobileNet: 
(content image: "Still Life with fruit dish" by Paul CĂ©zanne)
- OpenGPT-2: 
- OpenPifPaf-Pose: 
(image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)
- Photo-Sketch: 
- Pix2Pix: 
(content image: "Broadway Boogie Woogie" by Piet Mondrian)
- Places365: 
(image credits: "London: The Thames from Somerset House Terrace towards the City" by Canaletto)
- PoseNet: 
(image credits: Margaret Hamilton in 1969 with the source code her team developed for the Apollo missions. Photograph: Science History Images/Alamy Stock Photo via theguardian.com)
- SPADE-COCO: 
(uses segmentation based on "The Card Players" by Paul CĂ©zanne)
- SPADE-Landscapes: 
(uses segmentation based on "The Card Players" by Paul CĂ©zanne)
- Style2Paints: 
(content image: "Self-Portrait with Striped Shirt" by Egon Schiele)
- StyleGAN: 
(image credits: "Portrait of Ada Lovelace" by Margaret Sarah Carpenter)
- YOLACT: 
(content image: "The Card Players" by Paul CĂ©zanne)
- UGATIT: 
(content image: "Still Life with fruit dish" by Paul CĂ©zanne)
## Extra platforms
Additionally you can use RunwayML on a machine on the local network and connect from another device capable of running Processing.
### Raspberry PI
As long as you have a Raspberry PI running Processing, all you need to do is specify the IP (and optionally port) of the computer running the RunwayML app: that's it!
(e.g. if your computer's LAN IP is 192.168.0.12 you would initialize Runway like so: `runway = new RunwayHTTP(this,"192.168.0.12",8000);`)
To setup Processing on a Raspberry Pi please follow the official [Get Started tutorial](https://pi.processing.org/get-started/)
### Android
1. please [install Android Mode for Processing](https://android.processing.org/install.html) (if it's not already installed on your system).
2. allow `INTERNET` permissions in **Processing > Android > Sketch Permissions** 
3. run the code (e.g. attnGANAndroid)
The one major caveat at the moment is the Base64 <-> PImage conversion is handled by a separate class: `ModelUtilsAndroid`.
This means on Android
```processing
PImage result = ModelUtils.fromBase64(base64ImageString);
```
becomes:
```processing
PImage result = ModelUtilsAndroid.fromBase64(base64ImageString);
```
(Additionally, it's even possible to write RunwayML Processing sketchs on the Android device using [APDE](https://github.com/Calsign/APDE/releases))
## Dependencies
This Processing Library manages the OSC connection to Runway relying on the [oscP5](http://www.sojamo.de/libraries/oscP5/) library.
Please install oscP5 before using the OSC connection with this library via **Sketch > Import Library... > Add Library... > Contribution Manager Filter > oscP5**
## Contributing
This is still a work in progress. Contributions are welcomed!
## Credits
Special thanks for mentoring and support from [Cris Valenzuela, Anastasis Germanidis](https://runwayml.com) and [Daniel Shiffman](https://github.com/shiffman)
Additional thanks to [Jen Sykes](http://www.gsa.ac.uk/about-gsa/our-people/our-staff/s/sykes,-jen/) for updating the DenseCap and GPT2 examples
Main library development by [George Profenza](https://github.com/orgicus)
Library examples are based partially on [Runway Processing Examples](https://github.com/runwayml/processing) by:
- [Cris Valenzuela](https://github.com/cvalenzuela)
- [Anastasis Germanidis](https://github.com/agermanidis)
- [Gene Kogan](https://github.com/genekogan)
- [maybay21](https://github.com/maybay21)
- [JP Yepez](https://github.com/jpyepez)
- [Joel Matthys](https://github.com/jwmatthys)
- [Alejandro Matamala Ortiz](https://github.com/matamalaortiz)