{"id":13625425,"url":"https://github.com/ferdavid1/AIMV","last_synced_at":"2025-04-16T06:32:44.921Z","repository":{"id":79815643,"uuid":"554493713","full_name":"ferdavid1/AIMV","owner":"ferdavid1","description":"AI model that generates a Music Video from a .wav file, using Instrument Classification, Speech Transcription, and Video Generation. ","archived":false,"fork":false,"pushed_at":"2022-11-14T09:15:13.000Z","size":65,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-01T22:05:08.698Z","etag":null,"topics":["ai","audio-processing","computer-vision","image-generation","instrument-classification","ml","music-video","natural-language-processing","neural-network","speech-recognition","video-generation"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ferdavid1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-19T22:54:02.000Z","updated_at":"2023-10-12T17:31:44.000Z","dependencies_parsed_at":"2023-12-17T02:02:32.319Z","dependency_job_id":null,"html_url":"https://github.com/ferdavid1/AIMV","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ferdavid1%2FAIMV","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ferdavid1%2FAIMV/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ferdavid1%2FAIMV/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ferdavid1%2FAIMV/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ferdavid1","download_url":"https://codeload.github.com/ferdavid1/AIMV/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223700427,"owners_count":17188319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","audio-processing","computer-vision","image-generation","instrument-classification","ml","music-video","natural-language-processing","neural-network","speech-recognition","video-generation"],"created_at":"2024-08-01T21:01:55.607Z","updated_at":"2024-11-08T14:31:19.154Z","avatar_url":"https://github.com/ferdavid1.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"# AIMV\nAI model that generates a Music Video from a .wav file, using Instrument Classification, Speech Transcription, and Video Generation. \n## *Architecture*:\n### Content Transcription\n  - #### Find Datasets\n    - [Ismir](https://www.ismir.net/resources/datasets/)\n  - #### **Instrument Classification**\n    - [Musical Instrument Identification](https://www.mdpi.com/1424-8220/22/8/3033/pdf?version=1650009477)\n    \n    - [Drum Sound Subtype Classification](https://www.researchgate.net/publication/41538774_Classification_of_snare_drum_sounds_using_neural_networks)\n    \n    - [Timbre Identification](https://iopscience.iop.org/article/10.1088/1742-6596/1856/1/012006/pdf)\n    \n    - [Sound Event Detection](https://arxiv.org/pdf/2107.05463)\n    \n    - [Medley-solos-DB: a cross-collection dataset for musical instrument recognition](https://zenodo.org/record/2582103)\n    \n  - #### Speech Recognition (is speech present?) and subsequent Speech Transcription (if present) using OpenAI Whisper ASR. use --language Spanish if Spanish, etc\n    [Whisper](https://github.com/openai/whisper) uses Transformer Neural Networks\n    \n  - #### Concatenation of the above into a general content transcription\n    For an audio window of length 1 second containing a Vocal saying loudly \"Heart\" over a crash cymbal hit:\n      - Return (loud, heart, crash, splash, synth, hexcolorvalue)\n \n    This translates to:\n      -   Volume relative to the running average volume + Vocal transcription + 3 most confident instrument identifications + color name (from frequency-\u003ehex map)\n      -   Each of the three identifications need to meet a confidence criteria, say 70%, to make the list. Ex: (crash, rimshot, snare) \n      - if less than 3 instruments identified with 70% confidence, i.e. list is (crash), fill the list with the second-and-third most confident guess for those that have been identified with confidence, in the case of crash it could have been a splash instead or a weirdly textured synth. (crash, splash, synth).    \n      - Ex. with 70% confidence, (guitar, vocal) -\u003e (guitar, vocal, bass), bc the \"guitar\" guess has a 2nd-most-confident guess of \"bass\"\n  \n    We then port this to a sentence. \n    \n      - Volume informs the intensity of the image, ie if loud/quiet vocal saying Heart, say \"person screaming/yelling/whispering, action + object.\n     \n      - Action being looking at, running at, holding, etc, the object in question (\"heart\"). Choose action randomly. \n    \n      - Further abstract the crash, splash, drum (transient description) by mapping these to additional, more abstracted related words. \n        - Text similiarity with Gensim [link](https://betterprogramming.pub/introduction-to-gensim-calculating-text-similarity-9e8b55de342d)\n \n        - (crash, splash, synth) -\u003e (impact, wave, tech)\n        - (kick, bass, tom) -\u003e (canon, deep, bounce)\n        - (cello, vocal, guitar) -\u003e (floating, vibrate, pedal)\n        - add these abstracted words to the final text description\n\n      - Category (drum, guitar, violin, vocal, piano, bass) given by the initial transient description determines the verb (hits, picks, plays, sings, plays, plays)\n        - If Drum category and not Kick, assign \"Drum stick strikes\" as the verb\n      - Least likely category gets assigned to the beginning of the metaphorical/abstraction section, with the last element of the abstraction as adjective before it.\n      - If speech present, assign a \"loudly\" or \"softly\" according to volume of the speech\n      - t = transients, a = abstractions\n      - (ifspeech)\"Person(volume) sings, (action) a (transciption), in the background to a (t1, t2) (verb), with a foreground (a3)(t3)(a2)(a1), the scene has a (hexcolorname) color palette.\"\n\n    #### Examples:\n      - (loud, heart, crash, splash, synth, hexcolorvalue, impact, wave, tech) -\u003e \"Person loudly sings, holding a heart, alongside a Crash and Splash hit with a foreground techy synth wave impact, the scene has a (hexcolorname) color palette.\"\n        - Abstraction is the foreground, if speech is present, it is \"alongside\" the background of the instruments. \n      - (loud, N/A, kick, bass, tom, hexcolorvalue, canon, deep, bounce) -\u003e \"Kick hits and Bass plays in the background. In the foreground a bouncey tom deep canon, the scene has a (hexcolorname) color palette\"\n      - (quiet, N/A, cello, vocal, guitar, hexcolorvalue, floating, vibrate, pedal) -\u003e \"Cello plays and Vocal sings in the background. In the foreground a pedaly guitar vibrate floating, the scene has a (hexcolorname) color palette.\"  \n        - Vocal identified as an instrument present, but no intelligible speech was transcribed. Likely a \"hmmmmm\" or an \"oooooooo\" or similar.\n\n### **Video Generation**\n  - [CogVideo](https://github.com/THUDM/CogVideo) uses Transformer Neural Networks\n  - [CogVideoPaper](https://github.com/THUDM/CogVideo)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fferdavid1%2FAIMV","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fferdavid1%2FAIMV","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fferdavid1%2FAIMV/lists"}