{"id":13804718,"url":"https://github.com/MaxAlyokhin/audio-motion-interface","last_synced_at":"2025-05-13T18:32:45.274Z","repository":{"id":101107533,"uuid":"477475334","full_name":"MaxAlyokhin/audio-motion-interface","owner":"MaxAlyokhin","description":"Web synthesizer with smartphone sensors","archived":false,"fork":false,"pushed_at":"2023-11-04T20:44:08.000Z","size":2096,"stargazers_count":46,"open_issues_count":0,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-18T21:44:43.763Z","etag":null,"topics":["art","midi","music","sonification","synthesizer","web","webaudio","websocket"],"latest_commit_sha":null,"homepage":"https://ami.stranno.su","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MaxAlyokhin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-03T21:55:16.000Z","updated_at":"2024-09-24T02:07:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"f50bd4d7-3791-4b68-b00d-909c6f6624c5","html_url":"https://github.com/MaxAlyokhin/audio-motion-interface","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxAlyokhin%2Faudio-motion-interface","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxAlyokhin%2Faudio-motion-interface/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxAlyokhin%2Faudio-motion-interface/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaxAlyokhin%2Faudio-motion-interface/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MaxAlyokhin","download_url":"https://codeload.github.com/MaxAlyokhin/audio-motion-interface/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254003437,"owners_count":21997887,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["art","midi","music","sonification","synthesizer","web","webaudio","websocket"],"created_at":"2024-08-04T01:00:53.148Z","updated_at":"2025-05-13T18:32:44.531Z","avatar_url":"https://github.com/MaxAlyokhin.png","language":"JavaScript","funding_links":[],"categories":["Packages"],"sub_categories":["Apps"],"readme":"# Audio-motion interface (AMI)\n\n*Sonification interface for motion and orientation*\n\n[![Uptime Robot status](https://img.shields.io/uptimerobot/status/m793083121-0b0a0d608a9491e5a58871a4)](https://ami.stranno.su) [![Uptime Robot status](https://img.shields.io/uptimerobot/ratio/m793083121-0b0a0d608a9491e5a58871a4)](https://ami.stranno.su)\n\n**Demo**: https://ami.stranno.su\n\n**Video**: https://youtu.be/H1ryDYgeoOs\n\n\u003e **Note**: [Bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1435625) in Firefox — on default settings there are problems with sound. It is recommended to put the attack value at least 0.1 to fix. Also, Firefox is not recommended because of the slower refresh rate of motion parameters.\n\n![](https://store.stranno.su/ami/design-en.webp)\n\n*\u003ca href=\"README_RU.md\"\u003eЭта страница есть также на русском\u003c/a\u003e*\n\nThe system synthesizes sound based on data from smartphone motion sensors: speed determines the volume, position determines the frequency. In other words, it is a musical instrument (synthesizer) that uses hand gestures instead of keys or strings.\n\nThere are so-called local mode, when the sound is generated by the smartphone and distributed mode, when the smartphone transmits data about the movement to the computer and the computer generates sound.\n\nThe algorithm has a minimal number of internal settings, leaving it up to you to process the sound (it can be both pedals/effects and a variety of DAWs like Ableton, Cubase, FL Studio, etc.).\n\n## Briefly how to use\n\nThe easiest option is to open from your smartphone at https://ami.stranno.su. The smartphone will ask for access to the sensors — it must be allowed. After that it will immediately start generating sound from the built-in speaker with a slight shake. Here it is better to either connect headphones, or connect to a bluetooth speaker, or use a mini-jack to mini-jack cable (or with a jack adapter) to connect to the speakers/amplifier/combo. There are the following disadvantages with this option:\n\n- you are constrained by a cable\n- your smartphone has a noticeable delay\n- you cannot see what you are playing (the note/frequency being generated)\n- it is not very convenient to change the settings\n\nAll these disadvantages are solved by a *distributed mode*. To do so:\n\n- switch the synthesis strategy on the smartphone to the distributed mode\n- enter additionally from the computer to https://ami.stranno.su. The computer will automatically turn on the special data receiver mode. This will display the line \"Connected (1)\" (the number may be higher if someone else has visited the site with you)\n- The smartphone now transmits the motion data to the computer. This is where both the smartphone and the computer start synthesizing sound. The smartphone does this with more delay, so you can hear something like an echo when the sound is on in both devices. Here you can turn down the volume on the smartphone to zero, and connect the computer to your audio system.\n\nFrom the computer, therefore, the sound can be transferred to the DAW via Virtual Audio Cabel (VAC) and processed there, letting the input VAC operating system sounds (as the browser gives sound there), and the output VAC connect to the DAW. Then the sound can be taken either from the mini-jack of the computer, or from an external audio-interface, and from there process further.\n\nSome possible schemes of work:\n- smartphone → built-in speaker\n- smartphone → headphones\n- smartphone → bluetooth-speaker\n- smartphone → padals/effects → speakers/combo\n- smartphone → computer → DAW on computer → padals/effects → speakers/combo\n\nYou can also load presets into the system (you need to download as a file and use the Import button in the system):\n- [electronica.json](https://store.stranno.su/ami/electronica.json)\n- [ambient.json](https://store.stranno.su/ami/ambient.json)\n- [noise.json](https://store.stranno.su/ami/noise.json)\n\n\u003e **Note**: Using https://ami.stranno.su is demo. Its main disadvantage is the synchronization between all users; your sound and your settings can be interrupted by random users. Plus, since the traffic information goes over the internet (at least to Frankfurt, where the server is located, and back), there can be a delay (about 20-100ms, depending on the quality of the connection). To solve all these disadvantages it is recommended to deploy the system locally (see section **[Recommended use (running on a local computer)](#recommended)**).\n\n## Contents\n\n- [Run](#run)\n  - [Simple usage (running the web version at ami.stranno.su)](#simple)\n  - [Recommended usage (running on a local computer)](#recommended)\n  - [Advanced usage (running on local computer + sound processing)](#pro)\n  - [Running the development version](#dev)\n- [Theory and terms](#theory)\n  - [\"Interface\" instead of \"synthesizer\"](#interface)\n  - [Synthesis strategy](#strategy)\n  - [Synthesis point](#point)\n  - [Local mode](#local)\n  - [Distributed mode](#destributed)\n  - [Batch](#batch)\n  - [Motion event](#event)\n  - [Cutoff](#threshold-theory)\n  - [Gesture](#gesture)\n  - [Audio-graph](#graph)\n  - [Used semisphere](#semisphere)\n  - [AMI instance](#instance)\n  - [State](#state)\n- [User guide](#user)\n  - [System control](#user)\n    - [Generated frequency](#generated)\n    - [Note](#note)\n    - [Cutoff](#threshold)\n    - [Speed influences the volume](#influence)\n    - [Release](#release)\n    - [Oscillator amount](#count)\n    - [Latency](#latency)\n    - [Used semisphere](#semisphere-guide)\n    - [Frequency range](#freqs)\n    - [Notes range](#notes-range)\n    - [Frequency generation mode](#generation)\n    - [Tempered mode](#tempered)\n    - [Motion by α/β/γ axes](#motion)\n    - [Smartphone position on the γ axis](#gamma)\n    - [Motion status](#is-motion)\n    - [Maximum value](#max)\n    - [Data receiver mode enabled](#receiver)\n    - [Data source mode enabled](#source)\n    - [Connecting to server](#websocket)\n    - [Connection with server is ready](#websoket-on)\n    - [Connection with server is failed](#websoket-off)\n    - [Connected (х)](#connected)\n    - [Waiting for connections](#waiting)\n    - [Performance saving mode](#lite)\n    - [Cutoff type](#type)\n    - [Sensor timeout](#timeout)\n    - [Attack](#attack)\n    - [Volume](#volume)\n    - [Attenuation to value](#release-value)\n    - [Filter](#filter)\n    - [LFO](#lfo)\n    - [Compressor](#compressor)\n    - [Reset oscillators](#reset)\n    - [Errors](#errors)\n    - [Fullscreen-mode](#fullscreen)\n    - [Audio generation](#audio-generate)\n  - [MIDI](#midi)\n  - [Storing the interface state (settings) (Import / Export)](#state-storing)\n- [Tech guide](#tech)\n- [Secure context](#secure)\n\n## Run\n\u003ca id=\"run\"\u003e\u003c/a\u003e\n\n### Simple usage (running the web version at ami.stranno.su)\n\u003ca id=\"simple\"\u003e\u003c/a\u003e\n\nIn local mode:\n1. Go to https://ami.stranno.su from your smartphone\n\n\u003e **Note**: in local mode, the delay in sound synthesis can be quite noticeable due to the fact that the computing resource of a smartphone is quite limited compared to even the most average laptop\n\nIn distributed mode:\n1. Go to https://ami.stranno.su from your computer\n2. Go to https://ami.stranno.su from your smartphone\n\n(in any order)\n\n\u003e **Note**: in distributed mode, sound synthesis becomes shared by all who are currently logged into the site, and the settings are synchronized between all users. That is, if several people came to the site at the same time and someone changed the synthesis settings, they will be changed **at all participants**; sounds generated by one participant will be played **at all devices of all visitors**\n\n### Recommended usage (running on a local computer)\n\u003ca id=\"recommended\"\u003e\u003c/a\u003e\n\nThe recommended use is to run your own instance of AMI on your computer and work in distributed mode.\n\n\u003e **Note**: smartphone and computer must be connected to the same wifi network. Or you can run a virtual router on your laptop (using a third-party service a la Connectify) and connect your smartphone to your laptop. The most ideally you would run the hotspot on a laptop to even out the latency of the router. **On Windows you can use the \"Mobile Hotspot\" function**.\n\n\u003e **Note**: the latency with this startup option is the shortest possible\n\nThe purpose of both installations is this: Node.js is already in the folders. You need to use it to open the index.js file. This is convenient to do with a script. In MacOS you need to additionally make the script executable with `chmod -R 755 app`.\n\nPerhaps there are easier ways to install. I would be glad to hear your suggestions.\n\nWindows:\n1. [Download archive](https://store.stranno.su/ami/windows/audio-motion-interface.zip)\n2. Unpack\n3. Click on `run.bat`\n\nMacOS:\n1. [Download archive](https://store.stranno.su/ami/macos/audio-motion-interface.zip)\n2. Unpack\n3. Move the folder to Documents*\n4. `cmd` + `Space`\n5. Enter \"terminal.app\", start the Terminal\n6. In the terminal enter `cd` and *drag the folder* \"audio-motion-interface\" from Finder to the terminal. The terminal will automatically insert the path to the folder. You will get something like:\n`cd /User/User Name/Documents/audio-motion-interface`\nPress `Enter`\n7. Input `chmod -R 755 app` and press `Enter`\n8. Right-click the run.command file, then \"open with Terminal\".\n9. Give permission to execute the file. **In the future you can run AMI simply by clicking on run.command**\n\n*In general you can move it to any folder, but then you need to edit the run.command file with any text editor and fix the paths to Node.js and to index.js\n\nOn Linux the installation will be similar to MacOS, only you will need to download Node.js binaries for Linux and put them in the /app/node folder. If Node.js is already installed globally, you just need to run the index.js file with it.\n\n\u003e **Node:** because a self-signed certificate is used to encrypt the traffic the API requires, the browsers will show a warning about an invalid (untrusted) certificate. This is normal for working within local network. Read more in the [Secure context](#secure) section\n\n### Advanced usage (running on local computer + sound processing)\n\u003ca id=\"pro\"\u003e\u003c/a\u003e\n\nIt must be said that the most effective use case is the distributed mode on the local computer. In fact, the smartphone here is used solely as an interface to transmit data from the sensors, and the computer is used as an interface to manage this data.\n\nAfter getting to the computer, this data can be processed in any convenient way. For example, you can send the sound to a DAW (Ableton, Cubase, FL Studio) via [Virtual Audio Cabel](https://vac.muzychenko.net/en/) (VAC) and process it there, letting the operating system sounds *in* of the VAC (since the browser gives the sound there), and *out* of the VAC connect to the DAW. Then the sound can be taken either from the computer's mini-jack or from an external audiointerface, and processed further from there.\n\nIf you do not have an external audiointerface (sound card), you will see a delay in sound processing (at least on Windows). To avoid this, it is recommended to use [ASIO](https://www.asio4all.org/).\n\nIt is also possible to make your local AMI instance available from the Internet without having to deploy it to a remote server. To do this, you need to share your local AMI instance to the Internet using *tunneling*, for example with [ngrok](https://ngrok.com/) (it's free):\n\n`ngrok http https://localhost`\n\nOn the computer, open https://localhost\n\nOn your smartphone, open the link to the tunnel that ngrok generated.\n\n### Running the development version\n\u003ca id=\"dev\"\u003e\u003c/a\u003e\n\nIf you want to refine or rework the code, you must run the required development environment.\n\nFirst run:\n\n1. `git clone https://github.com/MaxAlyokhin/audio-motion-interface.git`\n2. Open folder in terminal\n3. `npm i`\n4. `nodemon index` (or just `node index`)\n5. Open second terminal\n6. `cd client`\n7. `npm i`\n8. `gulp`\n\nFurther launches:\n1. In the first terminal: `nodemon index` (or just `node index`)\n2. In the second terminal: `cd client`.\n3. `gulp`.\n\nThe first terminal is the backend, the second terminal is the frontend.\n\nIt is also necessary to remove the automatic launch of the browser on server restart. To do this, you need to comment out this line in the index.js:\n``` javascript\nserver.listen(443, '0.0.0.0', function () {\n  console.log(`${getDate()} Audio-motion interface is up and running`)\n  lookup(hostname, options, function (err, ips, fam) {\n    ips.forEach(ip =\u003e {\n      if (ip.address.indexOf('192.168') === 0) {\n        address = ip.address\n        console.log(`${getDate()} Opening https://${address} in default browser`)\n        // open(`https://${address}`) \u003c-- This line\n        console.log(`${getDate()} Close terminal for exit from AMI`)\n      } else {\n        address = 'ami.stranno.su'\n      }\n    })\n  })\n})\n```\n\n\u003e **Note**: For development purposes, it is better to globally install Nodemon. That way, it will be responsible for restarting code changes in the backend, and Gulp will be responsible for frontend code changes.\n\nThe repository already contains the private and public keys to run the https server. See the [Secure context](#secure) section below for details.\n\n## Theory and terms\n\u003ca id=\"theory\"\u003e\u003c/a\u003e\n\n### \"Interface\" instead of \"synthesizer\"\n\u003ca id=\"interface\"\u003e\u003c/a\u003e\n\nThe word \"synthesizer\" refers to an electronic musical instrument, often with a keyboard, which generates sound by means of electrical conversions by circuitry (analog) or by means of mathematical calculations by a microprocessor (digital). An important feature here is that such instruments are not acoustic (unlike classical guitars, violins, or woodwinds), but rather provide a *control system for the electric current* supplied to the acoustic system (an acoustic system here means any sound-producing device: speakers, combos, headphones, built-in speakers). Moreover, this is also true for devices that have a sensor as part of them. The electric guitar, for example, is also not an acoustic instrument: its central element is the pickup, and the strings are just a way of forming the signal. Also, any microphone is a sensor, and voice is also simply a way of forming a signal for the microphone to be converted into electricity and to be controlled by it. All such devices are signal (data) shaping tools, and acoustic systems are sonification tools, that is, the translation of non-sound processes (like voltage changes) into sound. Any sound recording process is the reverse of sonification. The keys on a synthesizer or the strings on an electric guitar are simply a part of the control system for the electric current that has migrated into these instruments in its familiar format for musicians.\n\nAn interface is a translator from language to language (in the broad sense of the word). All modern music is a set of ways of *translating* an acoustic signal into an electrical signal and back, or creating an electrical signal from nothing and then manipulating that.\n\nAn interface is a method of access. Any musical instrument, even an acoustic one, is an interface for accessing a particular set of sounds produced by that instrument.\n\nAny sensor is an interface. A smartphone nowadays almost always has an accelerometer and a gyroscope, a motion and orientation sensor, respectively. We can build a sound synthesis and control system based on them, *translating* data from sensors into signals, processing and *translating* into sound. That is, here the system not only and not so much synthesizes sound as it does the translation and interpretation of motion and orientation change processes, so here the word \"synthesizer\" would reflect only part of the system.\n\nThis repository represents a specific implementation of such a system.\n\n### Synthesis strategy\n\u003ca id=\"strategy\"\u003e\u003c/a\u003e\n\nA way of getting data from sensors and determining *where* it will be translated into sound.\n\n### Synthesis point\n\u003ca id=\"point\"\u003e\u003c/a\u003e\n\nA device that acquires motion data and synthesizes sound based on it.\n\n### Local mode\n\u003ca id=\"local\"\u003e\u003c/a\u003e\n\nThe *data source* and *synthesis point* are on the smartphone (combined).\n\n### Distributed mode\n\u003ca id=\"destributed\"\u003e\u003c/a\u003e\n\nThe *data source* is on the smartphone, and the *synthesis point* is on a remote machine (separated). This mode generally allows any number of complex combinations with multiple data sources (smartphones) and multiple synthesis points that are as far apart as you want (when streaming data over the internet) and connected to different audio systems.\n\n### Batch\n\u003ca id=\"batch\"\u003e\u003c/a\u003e\n\nA set of virtual devices [oscillator](https://en.wikipedia.org/wiki/Electronic_oscillator) → [filter](https://en.wikipedia.org/wiki/Low-pass_filter) → [LFO](https://en.wikipedia.org/wiki/Low-frequency_oscillation) → [envelope](https://en.wikipedia.org/wiki/Envelope_(music)).\n\n### Motion event\n\u003ca id=\"event\"\u003e\u003c/a\u003e\n\n[JS-event](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Building_blocks/Events), generated approximately every 16ms (on Chrome and Chromium-based browsers, on the Firefox every 100ms) by the smartphone, containing motion parameters. Events occur even when the device is motionless, in which case the motion parameters are zero.\n\n### Cutoff\n\u003ca id=\"threshold-theory\"\u003e\u003c/a\u003e\n\nThe minimum movement speed at which the system is started.\n\n### Gesture\n\u003ca id=\"gesture\"\u003e\u003c/a\u003e\n\nA set of motion events from above the cutoff to below the cutoff. Each gesture corresponds to its own batch. It is important to note that the word \"gesture\" here has nothing to do with touchscreen gestures or mouse gestures, as it does in other more familiar interfaces; here \"gesture\" is more of a hand gesture in space, or, in a general sense, the process of moving a smartphone in space, *generating* a set of motion events from above the cutoff to below the cutoff.\n\n### Audio-graph\n\u003ca id=\"graph\"\u003e\u003c/a\u003e\n\nA graph is an abstraction that connects *nodes* with *links*. For example, a graph for an electric guitar might be something like this:\n\nguitar → pedal 1 → pedal 2 → pedal 3 → combo\n\nThe AMI consists of virtual devices (nodes) that are connected in a certain way. The overall graph looks something like this:\n\n[oscillator → filter → LFO → envelope] → compressor\n\nThe devices in square brackets (batch) are generated *at each gesture* and connected to the compressor. When the batch is finished, it is removed and disconnected from the compressor.\n\n### Used semisphere\n\u003ca id=\"semisphere\"\u003e\u003c/a\u003e\n\nThe human hand has understandable limitations when rotating the hand: the left hand comfortably rotates from the palm up position to the palm down position clockwise, and the right hand comfortably rotates from the palm up position to the palm down position counterclockwise.\n\nThe smartphone can be rotated along its axis 360 degrees. But in this system, 360 degrees is divided in half: when the smartphone is on the table with the screen up, it is 180 degrees (palm up), and when on the screen (palm down) it is 0. From 0 to 180 you can go two ways: rotating the smartphone counterclockwise and clockwise. To make the system ergonomic, we can divide the 360 degrees into two semispheres, where the right semisphere is convenient for left-handed people and the left semisphere is convenient for right-handed people.\n\n![](https://store.stranno.su/ami/semi-sphere.jpg)\n**The picture in the center conventionally shows the smartphone with the screen down. \"Правша\" in russian means right-handed people and \"Левша\" in russian means left-handed people*\n\n### AMI instance\n\u003ca id=\"instance\"\u003e\u003c/a\u003e\n\nAn AMI instance is one server and clients (computer and smartphone browsers) connected to it. For example, https://ami.stranno.su is one instance of AMI; all clients connecting to it become part of the instance.\n\n### State\n\u003ca id=\"state\"\u003e\u003c/a\u003e\n\nState refers to the entire set of system settings **directly affecting the character of the sound** (that is, apart from user interface settings such as language or dark/light theme).\n\nGestures, passing through the state, turn into sound. Therefore, the state determines the sound.\n\nAt the instance level, state is the same for all clients and is synchronized between all clients. Clients have equal access to state changes.\n\nAll instance-level gestures data is sent to all clients.\n\nThus an instance is a single entity. The server acts as a commutator between clients, sharing gestures and state between all of them.\n\nRead more about state work in the [Storing state (settings) of the interface (Import / Export)](#state-storing)\n\n## User guide\n\u003ca id=\"user\"\u003e\u003c/a\u003e\n\n### System control\n\u003ca id=\"user\"\u003e\u003c/a\u003e\n\nSound synthesis is affected by two parameters — position and speed.\n\n\n\u003ca id=\"generated\"\u003e\u003c/a\u003e\n\n\u003ca id=\"note\"\u003e\u003c/a\u003e\n\nPosition (angle of tilt along the semisphere) determines the frequency displayed in the **Generated frequency** field, as well as below in the **Note** field the hit of this frequency in the nearest note.\n\n\n\u003ca id=\"threshold\"\u003e\u003c/a\u003e\n\n\u003ca id=\"influence\"\u003e\u003c/a\u003e\n\nThe speed of movement affects two factors:\n- turns on the system when the speed exceeds the **Cutoff** defined in the corresponding field\n- affects the volume when **Speed influences the volume** is enabled\n\nIf the cutoff is set to 0, one oscillator will run the whole time.\n\n\n\u003ca id=\"release\"\u003e\u003c/a\u003e\n\nAccordingly, the system turns on when the cutoff is exceeded, creates a batch, and generates sound. When decelerating to a value below the cutoff, the system plans to decay the sound (if the **Release** field is not 0).\n\n\n\u003ca id=\"count\"\u003e\u003c/a\u003e\n\nThe **Oscillator amount** field displays all the batches sounding at the moment.\n\nFor example, if you shake your hand chaotically for some time, the cutoff will be exceeded several times at random, which means that several batches will be generated, which will fade smoothly and their sound will overlap each other. It is better not to bring the number of oscillators, according to current observations, to values higher than 120 pieces, as almost certainly the computing power of the device will end there and the sound will start to stutter, or will disappear at all.\n\nAccording to subjective observations, the optimal cutoff can be between 3 and 7 (the default is 1), then random movements can be eliminated.\n\n\n\u003ca id=\"latency\"\u003e\u003c/a\u003e\n\nThe **Latency** field displays the latency from the motion event to the sound synthesis. On a smartphone, it is equal to the software latency (\u003ca href=\"https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/outputLatency\"\u003e`AudioContext.outputLatency`\u003c/a\u003e). On the desktop, it is equal to the software latency + the transfer time of the motion object from the smartphone to the desktop. In the interface is displayed in the format `device latency + network latency`.\n\nAccording to current observations, the lowest latency is observed on Apple devices (this applies to both IPhone and desktop devices, about 8ms). For example, on the smartphone Huawei Honor 10 device latency is 80ms, on the laptop Huawei Matebook device latency is 40ms; at the same time when starting the system locally via wi-fi router the network latency comes out 4-5ms. So, specifically for these devices, total smartphone latency = 80ms, total laptop latency 40 + 5 = 45ms. That is, comes out a paradoxical situation that on the laptop sound occurs earlier than on the smartphone.\n\nThe most efficient option is to use an Apple laptop and install the system locally, then the latency will be about 5 + 8 = 13ms.\n\n\n\u003ca id=\"semisphere-guide\"\u003e\u003c/a\u003e\n\nSince the frequencies are distributed over a semisphere, there is a field **Used semisphere** that allows you to switch the system for left- or right-handed people.\n\n\n\u003ca id=\"freqs\"\u003e\u003c/a\u003e\n\nThe semisphere contains 1800 divisions (180 degrees * tenths), on which the values specified in the field **Frequency range** are distributed. The values are distributed continuously and exponentially, i.e. there are more hertz for each degree at higher values, which allows to take into account the peculiarities of our hearing and make the frequency distribution across the semisphere even.\n\n\n\u003ca id=\"generation\"\u003e\u003c/a\u003e\n\n\u003ca id=\"tempered\"\u003e\u003c/a\u003e\n\n\u003ca id=\"notes-range\"\u003e\u003c/a\u003e\n\nYou can redistribute notes within a 12-step evenly tempered scale by using the **Frequency generation mode** field by selecting **Tempered mode**. Then the frequency range field will automatically change to **Range of Notes**. By selecting a small range, you can hit the notes you want quite accurately and confidently.\n\n\n\u003ca id=\"motion\"\u003e\u003c/a\u003e\n\nThe **Motion by α/β/γ axes** field shows the speeds of movement by the three coordinates in space.\n\n\n\u003ca id=\"gamma\"\u003e\u003c/a\u003e\n\nThe **Smartphone position on the γ axis** field shows the tilt angle in the semisphere. This tilt angle determines the frequency of the synthesized sound.\n\n\n\u003ca id=\"is-motion\"\u003e\u003c/a\u003e\n\nThe **Motion status** field shows `true` when the cutoff is exceeded and the system is generating sounds in the current batch. In the `false` position, the system is in standby mode when the cutoff is exceeded and generates no sound.\n\n\n\u003ca id=\"max\"\u003e\u003c/a\u003e\n\nThe field **Maximum value** shows the maximum speed of movement for the whole session (i.e. from the moment of opening the tab, to the current moment).\n\n\n\u003ca id=\"receiver\"\u003e\u003c/a\u003e\n\n**Data receiver mode enabled** means the computer is ready to receive data from external smartphones.\n\n\n\u003ca id=\"source\"\u003e\u003c/a\u003e\n\n**Data source mode enabled** means that the smartphone broadcasts its motion data to the remote computer.\n\n\n\u003ca id=\"websocket\"\u003e\u003c/a\u003e\n\n**Connecting to server** — in this status, the system tries to establish a websocket connection with the server through which the data will be broadcast between the smartphone and the computer.\n\n\n\u003ca id=\"websoket-on\"\u003e\u003c/a\u003e\n\n**Connection with server is ready** — it is possible to transmit data between devices.\n\n\n\u003ca id=\"websoket-off\"\u003e\u003c/a\u003e\n\n**Connection with server is failed** — something happened on the server and it is no longer responding, or the device has disconnected from the Internet and lost communication.\n\n\n\u003ca id=\"connected\"\u003e\u003c/a\u003e\n\n**Connected (x)** — number of connected devices, *other than this computer* (this field is displayed only from the desktop).\n\n\n\u003ca id=\"waiting\"\u003e\u003c/a\u003e\n\n**Waiting for connections** — no device is connected apart from this computer (this field is only displayed from the desktop).\n\n\n\u003ca id=\"lite\"\u003e\u003c/a\u003e\n\n**Performance saving mode** — motion events, as well as recalculation of the synthesized sound output values, trigger a very fast update of the data in the interface. This update is quite a costly operation. To save device resources, especially if you hear clicks or sound artifacts at some point, you can enable this mode, but it will turn off all data updates in the interface and you will only have to navigate by ear.\n\n\n\u003ca id=\"type\"\u003e\u003c/a\u003e\n\n**Cutoff type** — with \"full\" type, sound is generated from the point above the cutoff to the point below the cutoff. In the \"Peak\" type, sound is generated only on the point of highest movement speed. Explanation: each gesture slows down at the end of its movement. This leads to, firstly, a difficult system to control, and secondly, in the \"speed influences the volume\" mode, the middle of the sound can be loud and the end very quiet. We can catch the slowdown and interpret it as the end of the gesture. The oscillator will then cut off at the speed peak. In practice, this allows for clearer individual controlled sounds.\n\n\u003cpicture align=\"center\"\u003e\n  \u003cimg src=\"https://store.stranno.su/ami/full-peak.jpg\"\u003e\n\u003c/picture\u003e\n\n\n\u003ca id=\"timeout\"\u003e\u003c/a\u003e\n\n**Sensor timeout** — like the sensor cutoff, this setting allows you to better control your movement and get rid of accidental sounds. It sets a pause after the end of the previous gesture, leveling out the accidental excesses of the cut-off when slowing down the speed of movement.\n\n\n\u003ca id=\"attack\"\u003e\u003c/a\u003e\n\n**Attack** — the time of smooth growth of the volume to the value in the volume field.\n\n\n\u003ca id=\"volume\"\u003e\u003c/a\u003e\n\n**Volume** — the target volume value to which the attack grows and from which the attenuation begins.\n\n\n\u003ca id=\"release-value\"\u003e\u003c/a\u003e\n\n**Attenuation to value** — the volume value to which the sound is attenuated. The default value is `0.0001`, the minimum value (zero cannot be set for mathematical reasons, as the attenuation is exponential). If you set this value higher than the volume, the sound will grow and cut off abruptly, which will create a kind of inside-out effect.\n\n\n\u003ca id=\"filter\"\u003e\u003c/a\u003e\n\n**Filter** — lowpass filter, cuts the upper frequencies starting from the frequency specified in the corresponding field. The **Q-factor** determines the \"power\" of frequency suppression, the breadth of influence on frequencies (of all possible filters lowpass was chosen, because it softens ringing high frequencies, which is very appropriate for such a system. If there is a need for a more sophisticated filtering or a full-fledged equalizer in general, it is necessary to use external solutions, whether DAW or some separate devices).\n\n\n\u003ca id=\"lfo\"\u003e\u003c/a\u003e\n\n**LFO** is the oscillator that controls the main oscillator volume knob. The amplitude is ahead of the depth of volume change, and the frequency is ahead of the volume change rate.\n\n\n\u003ca id=\"compressor\"\u003e\u003c/a\u003e\n\n**Compressor** — by default its influence is minimized. If you want to play a lot of oscillators at once, but do not want to descend into rough noise, you can set, for example, the **Release** field to `0.25`, and the **Threshold** field to `-50`.\n\n\n\u003ca id=\"reset\"\u003e\u003c/a\u003e\n\n**Reset oscillators** — turns off and deletes all the batches that are working. It helps if the abundance of oscillators makes the sound produce artifacts, as well as if you set very high attenuation values and do not want to wait for the sound to fade out.\n\n\n\u003ca id=\"errors\"\u003e\u003c/a\u003e\n\n**Errors** — this field will be removed after the end of testing. Here will be inserted all the errors that occur during the work, which will help to debug the system.\n\n\n\u003ca id=\"fullscreen\"\u003e\u003c/a\u003e\n\n**Fullscreen-mode** — on your smartphone, opposite the text \"Audio-motion interface\", there will be an icon to switch to full-screen mode (Apple devices do not support it). This mode is recommended because it disables the standard browser gestures \"Back\" (when swiping from the left edge to the right) and \"Refresh\" (when swiping from the top edge down), which will give you more confidence in holding the smartphone in your hand without fear of pressing something.\n\n\n\u003ca id=\"audio-generate\"\u003e\u003c/a\u003e\n\n**Audio generation** — indicator showing whether sound is currently being generated.\n\n### MIDI\n\u003ca id=\"midi\"\u003e\u003c/a\u003e\n\nAMI allows you to generate MIDI messages. To enable MIDI, the MIDI regime must be enabled. Automatically selects the first available port and its first channel. When the cutoff is exceeded, a noteOn signal is sent, then in Peak mode the noteOff signal is sent after the time specified in the Duration field; in Full mode, noteOff is sent only when the movement is over + after the time in the Duration field; during the movement, the pressure on the channel changes when speed affects the volume (that is, speed during the movement is equal to the change in the force of the key press). In Peak mode, speed increases Velocity. The MIDI Reset button sends an allSoundOff signal.\n\nMIDI messages can be sent:\n- to neighboring tabs and browser windows if they listen to MIDI (e.g., Web analog [DX7](http://mmontag.github.io/dx7-synth-js))\n- to DAWs and other applications that have virtual synthesizers (that is, AMI can control, for example, a synthesizer in Ableton)\n- to external MIDI-enabled devices connected to your computer\n\nTo send MIDI messages to a DAW on Windows devices, you can use [loopMIDI](https://www.tobias-erichsen.de/software/loopmidi.html).\n\n\u003e **Note**: After any manipulations with MIDI ports (connecting/disconnecting/ reconnecting) you must restart the browser completely, closing all browser windows if there are several\n\n\u003e **Note**: MIDI messages are generated only on the desktop. The smartphone in this mode only sends movement and orientation data\n\n### Saving the state (settings) of the interface\n\u003ca id=\"state-storing\"\u003e\u003c/a\u003e\n\nAt the first start of the application, the default settings are set.\n\nThereafter, any changes to the settings are immediately written to the computer, so your work is automatically restored when you restart.\n\nBut you may have a situation where you need to keep multiple copies of your settings at once. Also, when you clear the browser completely, when you change browsers, or when you start the application from a different domain (e.g. a new address `192.168.X.X`), the settings will be lost. As a solution, the application provides buttons **Import / Export** to save and load settings from an external file in `.json` format.\n\n## Tech guide\n\u003ca id=\"tech\"\u003e\u003c/a\u003e\n\nStack: HTML, Sass, JS, Web Audio API, Device Motion API, Device Orientation API, Socket.io, Express, Gulp.\n\nAMI is essentially a small \"fullstack\" web application with https server on Express and websocket server on Socket.io, handing out a simple frontend in native JS using Web Audio API (WAAPI), Device Motion API (DMAPI) and Device Orientation API (DOAPI). We collect data from DMAPI/DOAPI, tidy it up, and send it directly to WAAPI in local mode and to websocket in distributed mode (and on the remote machine this data is received via websocket and sent there to WAAPI).\n\n\u003cpicture align=\"center\"\u003e\n  \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://store.stranno.su/ami/api-dark.png\"\u003e\n  \u003cimg src=\"https://store.stranno.su/ami/api-en.png\"\u003e\n\u003c/picture\u003e\n\nindex.js is the entry point into the application. Runs Express and Socket.io, distributes the frontend from /client/dist. Frontend builds Gulp in /client folder from /client/src and puts the finished thing in /client/dist. JS is built by Webpack, Sass is compiled to CSS, HTML is built from templates, and the built JS and CSS is injected into `\u003chead\u003e`. BrowserSync starts development server on a separate port, but it will not work backend (but it works live-reload), so it's better to open the address without port (`https://localhost`).\n\nThe application code is commented in many places, so you can learn the nuances directly in the code.\n\nA rough scheme of data flow through the functions after initialization:\n\n\u003cpicture align=\"center\"\u003e\n  \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://store.stranno.su/ami/functions-dark.png\"\u003e\n  \u003cimg src=\"https://store.stranno.su/ami/functions-new.png\"\u003e\n\u003c/picture\u003e\n\nThe [current-device](https://github.com/matthewhudson/current-device) library is used to define the device type - mobile or desktop, which initializes the corresponding mode. On the mobile device each motion event is checked for speed (maximum of three coordinates) and compared to the cutoff, if exceeded, then we create a batch. *If after that* are below the cutoff, then we plan to remove the batch. All elements of a batch are fluffed into arrays, and then deleted from them.\n\nThe application uses an MVC-like architecture (meaning MVC on the client), where the model uses a mutable state instead of the database, and the View will refer not only to the GUI, but also the data from the gyroscope and accelerometer, as well as the synthesized sound. The most important parts of the business logic are in `audio.js`, `motion.js`, and `orientation.js`. The state and almost all of its control is in `settings.js`. The `settings` object contains the entire state of the application. Changes to this object are made through methods of the `mutations` object. The connection between the view (its UI part) and the state is made in the `settingsInit()` function. This object is written to `localStorage`. Each mutation of it causes an update of `localStorage`. The `settings` object, with the exception of the `settings.ui` property, completely defines the sound of the system. The Import / Export buttons allow you to manipulate the state as an external JSON file.\n\nIt is quite possible to build a system with many states and control them through a kind of \"master device\" that mixes \"tracks\" from different devices, but in this case it was decided that each instance of AMI is a separate musical instrument with its own characteristic sound at the moment.\n\n\u003cpicture align=\"center\"\u003e\n  \u003cimg src=\"https://store.stranno.su/ami/mvc.jpg\"\u003e\n\u003c/picture\u003e\n\n### Secure context\n\u003ca id=\"secure\"\u003e\u003c/a\u003e\n\nDue to the fact that the Motion/Orientation API requires secure context (i.e. encryption of traffic), we have to raise the http**s** server. For this purpose, with the help of [OpenSSL](https://en.wikipedia.org/wiki/OpenSSL) were generated public and private keys, with which the traffic between the computer and the smartphone is encrypted ([self-signed certificate](https://en.wikipedia.org/wiki/Self-signed_certificate)). There's not much practical benefit from such encryption, since the data moves within your local network (and if leaked to the Internet, the data about the rotation of your smartphone won't do much harm), but secure context is required by all browsers to transmit data about the movement and position to external devices.\n\nIf necessary, you can generate your own self-signed certificate (see [here](https://stackoverflow.com/questions/21397809/create-a-trusted-self-signed-ssl-cert-for-localhost-for-use-with-express-node) for details).\n\nWhen you publish code on the web, you will most likely have some other infrastructure (nginx + server control panel, etc.) before the Express server, so the need for https-server will most likely disappear and you can change the index.js this way:\n\n``` javascript\nconst fs = require('fs')\nconst os = require('os')\nconst http = require('http') // \u003c-- Changed the package from https to http\nconst express = require('express')\nconst { Server } = require('socket.io')\nconst { lookup } = require('dns')\nconst open = require('open')\n\nconst hostname = os.hostname()\nconst app = express()\nconst server = http.createServer(app) // \u003c-- Removed certificate download\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMaxAlyokhin%2Faudio-motion-interface","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMaxAlyokhin%2Faudio-motion-interface","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMaxAlyokhin%2Faudio-motion-interface/lists"}