{"id":13627778,"url":"https://github.com/Halcyox/XRAgents","last_synced_at":"2025-04-17T00:32:27.322Z","repository":{"id":186433415,"uuid":"525163502","full_name":"Halcyox/XRAgents","owner":"Halcyox","description":"General Purpose 3D Humanoid Simulator integrated with AI Agents.","archived":false,"fork":false,"pushed_at":"2024-04-02T19:11:28.000Z","size":9926,"stargazers_count":4,"open_issues_count":7,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2024-11-08T18:45:21.509Z","etag":null,"topics":["3d-agents","3d-ai","audio2face","auto-animation","gamedev-framework","virtualreality"],"latest_commit_sha":null,"homepage":"https://www.halcyox.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Halcyox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-15T23:01:34.000Z","updated_at":"2024-09-29T11:44:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"3c38eb45-8fb1-4fd9-b1c7-1fe3868e0783","html_url":"https://github.com/Halcyox/XRAgents","commit_stats":null,"previous_names":["halcyox/xragents"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Halcyox%2FXRAgents","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Halcyox%2FXRAgents/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Halcyox%2FXRAgents/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Halcyox%2FXRAgents/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Halcyox","download_url":"https://codeload.github.com/Halcyox/XRAgents/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249293055,"owners_count":21245673,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-agents","3d-ai","audio2face","auto-animation","gamedev-framework","virtualreality"],"created_at":"2024-08-01T22:00:38.393Z","updated_at":"2025-04-17T00:32:27.063Z","avatar_url":"https://github.com/Halcyox.png","language":"Python","funding_links":[],"categories":["All Software"],"sub_categories":["Text Processing/NLP"],"readme":"# Metahumans Halcyox\r\n\r\n## Building the docs\r\n\r\n```sh\r\ncd doc\r\n.\\make.bat html\r\n```\r\n\r\n## Currently being refactored\r\n# Priority Refactor items\r\n* Restructuring the code flow to have one way dependency chain (rather than loop (Setting/Scene circular dependency))\r\n* Defining object-oriented code structure\r\n* Adding documentation\r\n* \r\n\r\n# Non-priority Refactor items\r\n* Encapsulation of data privacy (__id etc.)\r\n* ZODB object saving persistence stuff\r\n* Multithreading\r\n\r\n\r\n\r\n# Next time:\r\n* Work on the batch generation of videos for two characters, and see how far you get.\r\n* Also, work on the front end with Alice.\r\n\r\n## Priority TODO:\r\n\r\n~~* Selection backend for voice options~~\r\n* Set up simple server client app to get people to be able to access a server of talking to ai instance\r\n* Screen record video\r\n* Avatar creation/importing\r\n* Record + stream blendshapes\r\n\r\n~~* Convert input script batch pipeline to work with our video generation scripts~~\r\n* Make VidGen Server -\u003e rendering server distribute accross mult computers\r\n* Methods to upload videos to youtube -\u003e google api integration\r\n\r\n~~* Multi-character (n-entity) scene methods, just give prim paths~~\r\n* Talk to multiple AI simultaneously\r\n\r\n\r\n## Non-Priority TODO:\r\n* emotional speech\r\n* Optimization\r\n* change azure to streaming for lower latency\r\n* Connect your own camera to the output video with the AI\r\n* Unreal integration\r\n* SSML for voice style modification stuff\r\n\r\n## Web TODO:\r\n* Figure out dependencies, Docker stuff, versions\r\n* How to put on Amplify?\r\n* Should we use EC2 instances or what?\r\n* Website should have registration and billing\r\n* Token system for AI voice credits\r\n\r\n## Marketing TODO:\r\n* Affiliate marketing application to get other people to grow our software\r\n\r\n\r\n## LOCAL\r\n\r\n1. mic input\r\n2. speech to text\r\n3. generate response\r\n4. get .wav\r\n5. run sh script\r\n\r\n## SERVER\r\n\r\n### Directory Structure\r\n```\r\nXRAgents\r\n├── LICENSE\r\n├── README.md\r\n├── alphademo\r\n│   ├── videoproxy\r\n│   │   ├── src\r\n│   │   │   ├── main.rs\r\n│   │   │   └── signal.rs\r\n│   │   ├── brws_sess\r\n│   │   ├── Cargo.lock\r\n│   │   └── Cargo.toml\r\n...\r\n├── deps/streaming_server\r\n...\r\n├── xragents\r\n│   ├── anim.py\r\n│   ├── audio.py\r\n│   ├── cast.py\r\n│   ├── nlp.py\r\n│   ├── scenes.py\r\n│   ├── scriptgen.py\r\n│   ├── session.py\r\n│   ├── types.py\r\n│   └── utils.py\r\n└── digital-humans.db\r\n```\r\n\r\n### Example Post Requests\r\n\r\n##### Create Chararacter\r\nhttp://digital-humans.loca.lt/create-character\r\ncharacterName: Tom\r\ncharacterDescription: Tom is a wise old man who genuinely cares about people.\r\n\r\n##### Create Session\r\nhttp://digital-humans.loca.lt/create-session\r\nsessionName: newSession\r\nsessionDescription: The following is a conversation between you and Tom in the bookstore.\r\ncharacterIDList: 1\r\n\r\n##### Get Response\r\nhttp://digital-humans.loca.lt/get-response\r\npromptText: What should I do to heal from heartbreak?\r\nsessionID: 1\r\ncharacterID: 1\r\n\r\n\r\n### Debug (ignore)\r\n```\r\ncurl --location --request POST \"https://eastus.tts.speech.microsoft.com/cognitiveservices/v1\" --header \"Ocp-Apim-Subscription-Key: bfc08e214f6c48cebcde668a433196d3\" --header \"Content-Type: application/ssml+xml\" --header \"X-Microsoft-OutputFormat: audio-16khz-128kbitrate-mono-mp3\" --header \"User-Agent: curl\" --data-raw \"\u003cspeak version='\\''1.0'\\'' xml:lang='\\''en-US'\\''\u003e\u003cvoice xml:lang='\\''en-US'\\'' xml:styledegree='\\''2'\\'' name='\\''en-US-JennyNeural'\\''\u003eI hate you! You ruined my life!\u003c/voice\u003e\u003c/speak\u003e\" \u003e C:\\Users\\phn431\\Desktop\\digital-humans-backend\\wav\\test.mp3\r\n```\r\n\r\n```\r\ncurl --location --request POST \"https://eastus.api.cognitive.microsoft.com/sts/v1.0/issuetoken\" --header \"Ocp-Apim-Subscription-Key: bfc08e214f6c48cebcde668a433196d3\" --header 'Content-Type: application/ssml+xml' --header 'X-Microsoft-OutputFormat: audio-16khz-128kbitrate-mono-mp3' --header 'User-Agent: curl' --data-raw '\u003cspeak version='\\''1.0'\\'' xml:lang='\\''en-US'\\''\u003e\u003cvoice xml:lang='\\''en-US'\\'' xml:style='\\''angry'\\'' xml:styledegree='\\''2'\\'' name='\\''en-US-JennyNeural'\\''\u003eThis is a test\u003c/voice\u003e\u003c/speak\u003e' \u003e C:\\Users\\phn431\\Desktop\\digital-humans-backend\\wav\\test.wav\r\n```\r\n\r\n```\r\ncurl --location --request POST \"https://eastus.tts.speech.microsoft.com/cognitiveservices/v1\" --header \"Ocp-Apim-Subscription-Key: bfc08e214f6c48cebcde668a433196d3\" --header 'Content-Type: application/ssml+xml' --header 'X-Microsoft-OutputFormat: audio-16khz-128kbitrate-mono-mp3' --header 'User-Agent: curl' --data-raw '\u003cspeak version='\\''1.0'\\'' xml:lang='\\''en-US'\\''\u003e\u003cvoice xml:lang='\\''en-US'\\'' xml:gender='\\''Female'\\'' name='\\''en-US-JennyNeural'\\''\u003emy voice is my passport verify me\u003c/voice\u003e\u003c/speak\u003e' \u003e C:\\Users\\phn431\\Desktop\\digital-humans-backend\\wav\\test.wav\r\n```\r\n\r\n```\r\ncurl --location --request POST \"https://eastus.tts.speech.microsoft.com/cognitiveservices/v1\" --header \"Ocp-Apim-Subscription-Key: bfc08e214f6c48cebcde668a433196d3\" --header \"Content-Type: application/ssml+xml\" --header \"X-Microsoft-OutputFormat: audio-16khz-128kbitrate-mono-mp3\" --header \"User-Agent: curl\" --data-raw \"\u003cspeak version='\\''1.0'\\'' xml:lang='\\''en-US'\\''\u003e\u003cvoice xml:lang='\\''en-US'\\'' xml:gender='\\''Female'\\'' name='\\''en-US-JennyNeural'\\''\u003emy voice is my passport verify me\u003c/voice\u003e\u003c/speak\u003e\" \u003e output.mp3\r\n```\r\n\r\n```\r\nkey=\"YourSubscriptionKey\"\r\nregion=\"YourServiceRegion\"\r\n\r\ncurl \"https://eastus.tts.speech.microsoft.com/cognitiveservices/v1\" `\r\n--header \"Ocp-Apim-Subscription-Key: bfc08e214f6c48cebcde668a433196d3\" `\r\n--header 'Content-Type: application/ssml+xml' `\r\n--header 'X-Microsoft-OutputFormat: audio-16khz-128kbitrate-mono-mp3' `\r\n--header 'User-Agent: curl' `\r\n--data-raw '\u003cspeak version='\\''1.0'\\'' xml:lang='\\''en-US'\\''\u003e\r\n    \u003cvoice xml:lang='\\''en-US'\\'' xml:gender='\\''Female'\\'' name='\\''en-US-JennyNeural'\\''\u003e\r\n        my voice is my passport verify me\r\n    \u003c/voice\u003e\r\n\u003c/speak\u003e' \u003e output.mp3\r\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHalcyox%2FXRAgents","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FHalcyox%2FXRAgents","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FHalcyox%2FXRAgents/lists"}