{"id":26547159,"url":"https://github.com/cpmpercussion/robojam","last_synced_at":"2025-03-22T05:30:12.425Z","repository":{"id":27293195,"uuid":"110691324","full_name":"cpmpercussion/robojam","owner":"cpmpercussion","description":"A Mixture Density RNN for generating musical touchscreen interactions.","archived":false,"fork":false,"pushed_at":"2023-03-25T01:09:22.000Z","size":17071,"stargazers_count":11,"open_issues_count":14,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2023-08-13T13:11:50.698Z","etag":null,"topics":["generation","music","rnn","tensorflow","webapp"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cpmpercussion.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-14T13:06:44.000Z","updated_at":"2023-07-25T14:12:49.000Z","dependencies_parsed_at":"2022-09-10T22:31:07.338Z","dependency_job_id":null,"html_url":"https://github.com/cpmpercussion/robojam","commit_stats":null,"previous_names":[],"tags_count":3,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpmpercussion%2Frobojam","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpmpercussion%2Frobojam/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpmpercussion%2Frobojam/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpmpercussion%2Frobojam/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cpmpercussion","download_url":"https://codeload.github.com/cpmpercussion/robojam/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244912800,"owners_count":20530764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generation","music","rnn","tensorflow","webapp"],"created_at":"2025-03-22T05:30:11.705Z","updated_at":"2025-03-22T05:30:12.414Z","avatar_url":"https://github.com/cpmpercussion.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"RoboJam: A Mixture Density RNN for creating touchscreen performances\n====================================================================\n\n[![DOI](https://zenodo.org/badge/110691324.svg)](https://zenodo.org/badge/latestdoi/110691324)\n\nRoboJam is a Mixture Density RNN and web app for creating and responding to musical touchscreen performances.\nThe RNN design here is a novel application of mixture density network (MDN) to musical touchscreen data.\nThis data consists of a sequence of touch interaction events in the format `[x, y, dt]`. \nThis network learns to predict these events so that a user's interaction can be continued from where they leave off.\nThe web app runs uses Flask with a public API that can be used for interaction with touchscreen music apps running on phones or tablets.\nMore information is in the paper (to be added soon).\n\nHave a look at [how RoboJam is used in a touchscreen app](https://vimeo.com/242251501).\n\nHere's an example:\n\n![](https://github.com/cpmpercussion/robojam/raw/master/notebooks/example_unconditioned_1.png)\n\nData Format.\n------------\n\nTouchscreen performances should be stored in numpy arrays in the following format:\n\n  [x, y, dt]\n  \nWhere `x` and `y` are in [0,1] and `dt` is in [0,5].\n\nTodo:\n-----\n\n- Implement freezing model for more convenient loading in server.\n- Implement restart training from checkpoint\n- Include links to pre-processed data for training and validation.\n\nExamples:\n---------\n\n![](https://github.com/cpmpercussion/robojam/raw/master/notebooks/example_conditioned_1.png)\n\n![](https://github.com/cpmpercussion/robojam/raw/master/notebooks/example_conditioned_2.png)\n\n![](https://github.com/cpmpercussion/robojam/raw/master/notebooks/example_unconditioned_2.png)\n\nInstallation:\n-------------\n\n    pip3 install -r ./requirements.txt\n\nRunning:\n--------\n\n    python3 serve_tiny_performance_mdrnn.py\n\nTesting:\n--------\n\n    curl -i -k -X POST -H \"Content-Type:application/json\" https://127.0.0.1:5000/api/predict -d '{\"perf\":\"time,x,y,z,moving\\n0.005213, 0.711230, 0.070856, 25.524292, 0\\n0.097298, 0.719251, 0.062834, 25.524292, 1\\n0.126225, 0.719251, 0.057487, 25.524292, 1\\n0.194616, 0.707219, 0.045455, 38.290771, 1\\n0.212923, 0.704545, 0.045455, 38.290771, 1\\n0.343579, 0.703209, 0.108289, 38.290771, 1\\n0.495085, 0.701872, 0.070856, 38.290771, 1\\n0.523921, 0.693850, 0.061497, 38.290771, 1\\n0.712066, 0.711230, 0.155080, 38.290771, 1\\n0.730294, 0.717914, 0.155080, 38.290771, 1\\n0.896367, 0.696524, 0.041444, 38.290771, 1\\n1.083786, 0.696524, 0.151070, 38.290771, 1\\n1.301470, 0.684492, 0.049465, 38.290771, 1\\n1.328134, 0.680481, 0.053476, 38.290771, 1\\n1.504139, 0.705882, 0.136364, 38.290771, 1\\n1.527875, 0.712567, 0.120321, 38.290771, 1\\n1.702672, 0.675134, 0.076203, 38.290771, 1\\n1.719294, 0.675134, 0.096257, 38.290771, 1\\n1.901434, 0.715241, 0.145722, 38.290771, 1\\n1.922717, 0.717914, 0.136364, 38.290771, 1\\n2.062994, 0.684492, 0.109626, 38.290771, 1\\n2.091680, 0.680481, 0.129679, 38.290771, 1\\n2.231362, 0.697861, 0.207219, 38.290771, 1\\n2.393213, 0.712567, 0.124332, 38.290771, 1\\n2.525774, 0.632353, 0.149733, 38.290771, 1\\n2.546701, 0.625668, 0.169786, 38.290771, 1\\n2.686487, 0.585561, 0.360963, 38.290771, 1\\n2.715316, 0.580214, 0.387701, 38.290771, 1\\n2.867526, 0.490642, 0.633690, 38.290771, 1\\n2.880361, 0.481283, 0.645722, 38.290771, 1\\n3.054443, 0.319519, 0.689840, 38.290771, 1\\n3.218741, 0.121658, 0.585561, 38.290771, 1\\n3.230362, 0.102941, 0.557487, 38.290771, 1\\n3.391456, 0.089572, 0.534759, 38.290771, 1\"}'\n    curl -i -k -X POST -H \"Content-Type:application/json\" https://138.197.179.234:5000/api/predict -d '{\"perf\":\"time,x,y,z,moving\\n0.005213, 0.711230, 0.070856, 25.524292, 0\\n0.097298, 0.719251, 0.062834, 25.524292, 1\\n0.126225, 0.719251, 0.057487, 25.524292, 1\\n0.194616, 0.707219, 0.045455, 38.290771, 1\\n0.212923, 0.704545, 0.045455, 38.290771, 1\\n0.343579, 0.703209, 0.108289, 38.290771, 1\\n0.495085, 0.701872, 0.070856, 38.290771, 1\\n0.523921, 0.693850, 0.061497, 38.290771, 1\\n0.712066, 0.711230, 0.155080, 38.290771, 1\\n0.730294, 0.717914, 0.155080, 38.290771, 1\\n0.896367, 0.696524, 0.041444, 38.290771, 1\\n1.083786, 0.696524, 0.151070, 38.290771, 1\\n1.301470, 0.684492, 0.049465, 38.290771, 1\\n1.328134, 0.680481, 0.053476, 38.290771, 1\\n1.504139, 0.705882, 0.136364, 38.290771, 1\\n1.527875, 0.712567, 0.120321, 38.290771, 1\\n1.702672, 0.675134, 0.076203, 38.290771, 1\\n1.719294, 0.675134, 0.096257, 38.290771, 1\\n1.901434, 0.715241, 0.145722, 38.290771, 1\\n1.922717, 0.717914, 0.136364, 38.290771, 1\\n2.062994, 0.684492, 0.109626, 38.290771, 1\\n2.091680, 0.680481, 0.129679, 38.290771, 1\\n2.231362, 0.697861, 0.207219, 38.290771, 1\\n2.393213, 0.712567, 0.124332, 38.290771, 1\\n2.525774, 0.632353, 0.149733, 38.290771, 1\\n2.546701, 0.625668, 0.169786, 38.290771, 1\\n2.686487, 0.585561, 0.360963, 38.290771, 1\\n2.715316, 0.580214, 0.387701, 38.290771, 1\\n2.867526, 0.490642, 0.633690, 38.290771, 1\\n2.880361, 0.481283, 0.645722, 38.290771, 1\\n3.054443, 0.319519, 0.689840, 38.290771, 1\\n3.218741, 0.121658, 0.585561, 38.290771, 1\\n3.230362, 0.102941, 0.557487, 38.290771, 1\\n3.391456, 0.089572, 0.534759, 38.290771, 1\"}'\n    curl -i -k -X POST -H \"Content-Type:application/json\" https://138.197.179.234:5000/api/predict -d '{\"perf\":\"time,x,y,z,moving\\n0.002468, 0.106414, 0.122449, 20.000000, 0\\n0.020841, 0.106414, 0.125364, 20.000000, 1\\n0.043218, 0.107872, 0.137026, 20.000000, 1\\n0.065484, 0.107872, 0.176385, 20.000000, 1\\n0.090776, 0.107872, 0.231778, 20.000000, 1\\n0.110590, 0.109329, 0.301749, 20.000000, 1\\n0.133338, 0.115160, 0.357143, 20.000000, 1\\n0.155677, 0.125364, 0.412536, 20.000000, 1\\n0.178238, 0.134111, 0.432945, 20.000000, 1\\n0.516467, 0.275510, 0.180758, 20.000000, 0\\n0.542726, 0.274052, 0.205539, 20.000000, 1\\n0.560772, 0.274052, 0.249271, 20.000000, 1\\n0.583259, 0.282799, 0.316327, 20.000000, 1\\n0.605750, 0.295918, 0.376093, 20.000000, 1\\n0.628259, 0.309038, 0.415452, 20.000000, 1\\n0.653835, 0.316327, 0.432945, 20.000000, 1\\n0.673523, 0.325073, 0.440233, 20.000000, 1\\n1.000294, 0.590379, 0.179300, 20.000000, 0\\n1.022137, 0.593294, 0.183673, 20.000000, 1\\n1.044706, 0.594752, 0.208455, 20.000000, 1\\n1.067020, 0.606414, 0.279883, 20.000000, 1\\n1.091137, 0.626822, 0.355685, 20.000000, 1\\n1.111968, 0.647230, 0.425656, 20.000000, 1\\n1.134535, 0.655977, 0.462099, 20.000000, 1\\n1.156987, 0.657434, 0.485423, 20.000000, 1\\n1.619212, 0.857143, 0.263848, 20.000000, 0\\n1.642492, 0.854227, 0.281341, 20.000000, 1\\n1.663123, 0.851312, 0.320700, 20.000000, 1\\n1.685776, 0.846939, 0.413994, 20.000000, 1\\n1.708192, 0.846939, 0.510204, 20.000000, 1\\n1.730717, 0.858601, 0.591837, 20.000000, 1\\n1.753953, 0.868805, 0.632653, 20.000000, 1\\n1.775862, 0.876093, 0.660350, 20.000000, 1\\n4.376275, 0.542274, 0.860058, 20.000000, 0\\n4.419554, 0.543732, 0.860058, 20.000000, 1\"}'\n    curl -i -k -X POST -H \"Content-Type:application/json\" https://0.0.0.0:5000/api/predict -d '{\"perf\":\"time,x,y,z,moving\\n0.002468, 0.106414, 0.122449, 20.000000, 0\\n0.020841, 0.106414, 0.125364, 20.000000, 1\\n0.043218, 0.107872, 0.137026, 20.000000, 1\\n0.065484, 0.107872, 0.176385, 20.000000, 1\\n0.090776, 0.107872, 0.231778, 20.000000, 1\\n0.110590, 0.109329, 0.301749, 20.000000, 1\\n0.133338, 0.115160, 0.357143, 20.000000, 1\\n0.155677, 0.125364, 0.412536, 20.000000, 1\\n0.178238, 0.134111, 0.432945, 20.000000, 1\\n0.516467, 0.275510, 0.180758, 20.000000, 0\\n0.542726, 0.274052, 0.205539, 20.000000, 1\\n0.560772, 0.274052, 0.249271, 20.000000, 1\\n0.583259, 0.282799, 0.316327, 20.000000, 1\\n0.605750, 0.295918, 0.376093, 20.000000, 1\\n0.628259, 0.309038, 0.415452, 20.000000, 1\\n0.653835, 0.316327, 0.432945, 20.000000, 1\\n0.673523, 0.325073, 0.440233, 20.000000, 1\\n1.000294, 0.590379, 0.179300, 20.000000, 0\\n1.022137, 0.593294, 0.183673, 20.000000, 1\\n1.044706, 0.594752, 0.208455, 20.000000, 1\\n1.067020, 0.606414, 0.279883, 20.000000, 1\\n1.091137, 0.626822, 0.355685, 20.000000, 1\\n1.111968, 0.647230, 0.425656, 20.000000, 1\\n1.134535, 0.655977, 0.462099, 20.000000, 1\\n1.156987, 0.657434, 0.485423, 20.000000, 1\\n1.619212, 0.857143, 0.263848, 20.000000, 0\\n1.642492, 0.854227, 0.281341, 20.000000, 1\\n1.663123, 0.851312, 0.320700, 20.000000, 1\\n1.685776, 0.846939, 0.413994, 20.000000, 1\\n1.708192, 0.846939, 0.510204, 20.000000, 1\\n1.730717, 0.858601, 0.591837, 20.000000, 1\\n1.753953, 0.868805, 0.632653, 20.000000, 1\\n1.775862, 0.876093, 0.660350, 20.000000, 1\\n4.376275, 0.542274, 0.860058, 20.000000, 0\\n4.419554, 0.543732, 0.860058, 20.000000, 1\"}'\n\n# Docker\n\n## Building the Docker Container\n\n    sudo docker build -t robojam:latest .\n    docker tag robojam:latest charlepm/robojam:latest\n    docker push charlepm/robojam:latest\n\n## Running the Docker Container\n\n    docker run -d -p 5000:5000 robojam:latest\n\n## Running the Docker Container in Kubenetes\n\n    kubectl run robojam-cluster --image=charlepm/robojam:latest --port 5000\n    kubectl get pods\n    kubectl expose deployment robojam-cluster --type=LoadBalancer --port 5000 --target-port 5000\n    kubectl get service","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpmpercussion%2Frobojam","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcpmpercussion%2Frobojam","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpmpercussion%2Frobojam/lists"}