{"id":17722490,"url":"https://github.com/leixingyu/pedagogical-agent","last_synced_at":"2025-03-14T05:31:53.979Z","repository":{"id":45230896,"uuid":"291541698","full_name":"leixingyu/pedagogical-agent","owner":"leixingyu","description":"Unity procedural animation system","archived":false,"fork":false,"pushed_at":"2022-02-27T17:25:57.000Z","size":4104,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-09T23:47:03.849Z","etag":null,"topics":["animation","character","procedural-animation","procedural-generation","unity"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leixingyu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-30T19:54:26.000Z","updated_at":"2023-02-24T04:00:03.000Z","dependencies_parsed_at":"2022-09-01T21:01:55.407Z","dependency_job_id":null,"html_url":"https://github.com/leixingyu/pedagogical-agent","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leixingyu%2Fpedagogical-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leixingyu%2Fpedagogical-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leixingyu%2Fpedagogical-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leixingyu%2Fpedagogical-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leixingyu","download_url":"https://codeload.github.com/leixingyu/pedagogical-agent/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243532517,"owners_count":20306151,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["animation","character","procedural-animation","procedural-generation","unity"],"created_at":"2024-10-25T15:38:29.562Z","updated_at":"2025-03-14T05:31:51.969Z","avatar_url":"https://github.com/leixingyu.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv id=\"top\" align=\"center\"\u003e\n\u003ch1 align=\"center\"\u003ePedagogical Agent (a procedural character animation system)\u003c/h1\u003e\n\n  \u003cp align=\"center\"\u003e\n    A research project during my master's degree at Purdue University; It involves the implementation\n    of character procedural animation used in virtual learning environment.\n    \u003cbr /\u003e\n    \u003ca href=\"https://youtu.be/iXXYbPlqv1s\"\u003eFull Demo\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\n    \u003ca href=\"#overview\"\u003eOverview\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#master-controller\"\u003eMaster Controller\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#xml-reader\"\u003eXML Reader\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#beat-detection\"\u003eBeat Detection\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#emotion-input\"\u003eEmotion Input\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#user-interface\"\u003eUser Interface\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#debug-gui\"\u003eDebug GUI\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#character-setup\"\u003eCharacter Setup\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#scene-structure\"\u003eScene Structure\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#file-structure\"\u003eFile Structure\u003c/a\u003e\u003c/li\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#overview-of-folder\"\u003eOverview of Folder\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#resource-folder\"\u003eResource Folder\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#character-based-assets\"\u003eCharacter-based Assets\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003cli\u003e\u003ca href=\"#character-components\"\u003eCharacter Components\u003c/a\u003e\u003c/li\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#animator-components\"\u003eAnimator Component\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#salsa-3d-plugin\"\u003eSalsa 3D Plugin\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#final-ik-plugin\"\u003eFinal IK Plugin\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#run-time-script\"\u003eRun-time Script\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003cli\u003e\u003ca href=\"#event-system\"\u003eEvent System\u003c/a\u003e\u003c/li\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#system-components\"\u003eSystem Components\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#custom-scripts\"\u003eCustom Scripts\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#run-time-event\"\u003eRun-time Event\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003cli\u003e\u003ca href=\"#plugin\"\u003ePlugin\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n## About The Project\n\n__Project:__ Multimodal Affective Pedagogical Agents for Different Types of Learners\n\n__Faculty:__ Nicoletta Adamo, Purdue (PI), Richard E. Mayer, UCSB (co-PI), Bedrich Benes, Purdue (co-PI)\n\n__Sponsor:__ NSF - IIS - Cyberlearning, award # 1821894 (2018-2021), Purdue Instructional Innovation Grant (2018- 2020)\n\nhttp://hpcg.purdue.edu/idealab/AffectiveAgents/\n\n## Getting Started\n\n\u003e The source project is based on Unity 2018.3.6f1, the build is tested on Win10x64\n\nUnity Components:\n\nDownload the [release package](https://github.com/leixingyu/pedagogical-agent/releases/tag/v0.4.0) which consists of both the final build and source code.\n\nEmotion Recognition:\n\n[__Emotion Recognition Overview__](https://www.notion.so/emotion-recognition-2f2c082af6ba465fbb300c0d8d7e537c)\n|\n[__Emotion Recognition Documentation__](https://www.notion.so/Emotion-Recognition-Deployment-Instruction-6f853779e8664479812f4ee8bb999249)\n\n## Overview\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e\u003c/p\u003e\n\nThe animated agent is controlled by four high-level controller:\n\n\u003cimg src=\"https://i.imgur.com/4sxHtvU.jpg\" alt=\"overview\" height=\"65%\" width=\"65%\"\u003e\n\n### Master Controller\n\n- **Body Gesture**: Mecanim accesses the built-in state-machine in Unity. \nThe procedurally generated body gesture is achieved by pose interpolation – the transitioning between any two predetermined poses sequentially.\n\n    Body Offset generate randomness as well as emotion-based body variation such as leaning forward/backward, body contract/expand. \nIt contains a IK-based shoulder and hand control, a FK-based spine offset and a COG hip control.\n\n\n- **Facial Deformation**: Facial Expression has access to predetermined emotion including happy, bored, angry and content.\nThe component blends naturally between each blendshapes. \n\n    Salsa Plugin enables facial automation including: eye-blink, gaze and lip-sync based on audio input.\nOther subcomponents include eyebrow raise during speech generated algorithmically by [Beat Detection](#beat-detection).\n\n\n- **Hand and Foot**: The agent's hand gesture is layered on top of body gesture when performing actions like pointing. \nIt has pre-determined hand shapes like relax, holding fist, stretched palm.\n\n    The foot position similar to hand shape can be locked to the ground, \nfree (inherit motion from mocap data), and procedural auto-stepping (not-implemented).\n\n### XML Reader \n\nThis XML-based script works like a behavioral markup language that controls the agent during the speech. \nIt specifies events to trigger on certain moments: such as emotional changes, \nor body and hand gesture transition. It also controls when to switch PowerPoint, and when to detect user's [Emotion Input](#emotion-input).\n\n### Beat Detection\n\nThis functionality analyzes the spectrum data of audio input and detects beat in real-time. \nThe detection is based on an algorithm that calculates the average sound energy and triggers when a variance greater\nwhen certain threshold has occurred. \nWhen a beat is detected, a series of Beat Events are invoked. Idle Events are invoked after the Beat Events are completed. (More on [Event System](#event-system))\n\n### Emotion Input\n\nThis is the core mechanic used to communicate with the user. The XML Reader sends requests for emotional state updates. \nThe Emotion Update components communicate with the Python `emotion_recognition` which constantly checks the emotion and stores it in a queue. \nWhen a request for an emotional state update is received, the most frequent emotion in that queue is returned. \nThe Emotion Input triggers emotional based events based on the return results.\n\n## User Interface\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e\u003c/p\u003e\n\n\u003cimg src=\"https://i.imgur.com/THfWylf.png\" alt=\"overview\" height=\"65%\" width=\"65%\"\u003e\n\n| Menu Item  | Operation |\n| ------------- | ------------- |\n| Debug |               Open up the run-time GUI for debugging  |\n| Component Checkbox  | Toggle on/off controller components  |\n|Reload |\t\t\t    Restart the scene\n|Pause/Play |\t\t\tPause/Continue playing the scene\n|Slider |\t\t\t\tAdjust camera distance from full-body to close-up\n|Arrow button |\t\t\tControl slides\n|Character button |\t\tSwitch character and restart the scene\n\n## Debug GUI\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e\u003c/p\u003e\n\n\u003cimg src=\"https://i.imgur.com/WowAc7Y.png\" alt=\"overview\" height=\"65%\" width=\"65%\"\u003e\n\n| Menu Item  | Operation |\n| ------------- | ------------- |\n|Facial Expression| \tChange facial expression\n|Foot lock|\t\tToggle foot position lock to ground or free moving\n|Hand pose|\t\tChange hand pose left or right separately\n|Body parameter| \tModify animation speed, body rotation and blend time\n|Body pose| \t\tSwitch poses\n\n## Character Setup\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e\u003c/p\u003e\n\n### Character Requirement\n- **Joint**: ideally no more than 40 joints\n- **Mesh**: no sub-mesh\n- **Size**: 200mb and less\n\n### Character Clean-up\nThe clean-up process ensures that the `.fbx` is accepted in Unity, \nwhich is basically a transfer from animation rig to a game rig\n\n1. The output should be a character with a single mesh skinned to a single joint hierarchy [H-Anim Standard].\n2. Make sure that all end joints have zeroed-out rotation and unit scaling.\n3. Whenever the character has updated blendshapes, it needs to be re-imported\n\n### Character Template\nThe character template is used in MotionBuilder after identifying key joint components,\nthis will smoothen the retargeting process.\n\n### Character Import\nThe character `.fbx` should be placed in the character folder, \nadjust `Animation Type` to `Humanoid`, make sure `Avatar Definition` is set to `Create From this Model`, then configure the joint as needed.\n\nExisting character settings:\n\n- David Latest version: v2 \nscaling: 1.4\n- Luna Latest version: v3\nscaling: 12\n\n(Note: all scale settings need to be adjusted in scene view, not in character configuration)\n\n## Scene Structure\n\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e\u003c/p\u003e\n\n### Overview of Hierarchy\n\n| Hierarchy  | Description |\n| ------------- | ------------- |\n|Event System| \tScene related control (GUI, audio control, global logic)\n|Character| \t\tCharacter related control components (animation behavior)\n|Environment| \t\t3D environment models\n|Lighting|\t\tLighting components\n|Canvas| \t\tPredefined user interface\n|MainCam|\t\tMaster camera\n\n### Character\n| Hierarchy  | Description |\n| ------------- | ------------- |\n|BaseMesh| \t\tSkinned mesh (has skinned mesh renderer for blendshapes)\n|Skeleton| \t\tCharacter joint hierarchy\n|Foot IKs| \t\tPre-defined IK handle for locking the foot\n\n\n## File Structure\n\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e\u003c/p\u003e\n\n### Overview of Folder\n| Hierarchy  | Description |\n| ------------- | ------------- |\n|Character| \t\tCharacter mask, avatar definition and rig (`.fbx`)\n|Plugins| \t\tAll plugin used (Salsa, Mecanim and Final IK)\n|Resources| \t\tAssets accessed during run-time\n|Scenes|\t\tGame scene (one for each character)\n|Scripts| \t\tAll custom scripts\n|Texture| \t\tMaterials and textures\n\n### Resource Folder\n| Hierarchy  | Description |\n| ------------- | ------------- |\n|Audio|\t\t\tIn-game speech audio file\n|Controller| \t\tAnimator logic controller (mecanim state machine)\n|HandShape|\t\tPredefined hand pose (`.fbx`)\n|MotionLibrary| \tPredefined body pose (`.fbx`)\n|not-in-use| \t\tPowerPoint slide not-in-use\n|Slide| \t\t\tPowerPoint slide texture\n|XML| \t\t\tcharacter behavioral scripts\n\n### Character-based Assets\n\nCharacter sub assets have individual folders, which can be modified (see `Global.cs`)\n\n```csharp\n// character animation clips location\npublic static string lunaAnim = \"MotionLibrary/Luna\";\npublic static string davidAnim = \"MotionLibrary/David\";\n\n// character controller location\npublic static string lunaController  = \"Controller/Luna_controller\";\npublic static string davidController = \"Controller/David_controller\";\n\n// subsequence audio location\npublic static string lunaAudio = \"Audio/Luna\";\npublic static string davidAudio = \"Audio/David\";\n```\n\n## Character Components\n\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e\u003c/p\u003e\n\n### Animator Component\n\n`Animator` component is required for a state machine-based animation transition. \nUsing `MecanimControl.cs` evokes different states in `Animator` for gesture transition.\n\n- Leave `Controller` blank for it will be assigned by `MecanimControl.cs` in run-time\n- Set the `Avatar` to the latest character definition\n- Uncheck `Apply Root Motion`\n\n### Salsa 3D Plugin \n\n**Queue Processor**: Required component to use and monitor Salsa event queue.\n\n**Eyes**: Salsa Eyes component is used to create random eye shifts, eye blinks\n\n- `Properties` control random eye-gaze shift, adjust physical range and frequency\n- `Eyelid properties` controls the random eye-blinks, adjust frequency\n- To use random eye and fix axis, unpack character prefab\n\n**Salsa**: Salsa Lip-sync, used for creating automatic lip-sync based on audio input\n\n- Adjust blendshapes for visemes\n- Adjust visemes trigger threshold\n- Attach corresponding audio file\n\n### Final IK Plugin\n\n**Full body biped IK**: This component is used to create full body IK setup; \nfoot IK is used to lock feet in place; COG/Shoulder/Hand IK can be used to create \nbody offset to alter the character’s gesture style\n\n- Assign corresponding bones\n- Parent foot effector under character hierarchy\n- Set parameters such as iterations (2 is fine)\n\n### Run-time Script\n\n**Mecanim Control**: assigns the appropriate `Controller` to the `Animator`, \ncreates two states, and assigns different clips to these two states during run-time, \nachieving any state to any state transition. (see `MecanimControl.cs`)\n\n**Expression Control**: change character’s blendshape which in result,\ncontrols the facial expression (see `ExpressionControl.cs`)\n\n**Hand Control**: switch a character's hand pose. \nThe hand pose is controlled by a sub-layer (with a character mask) in the animator controller, \nit needs to be manually modified and connected.\n\n**Foot Control**: used to lock or unlock the character’s foot in-place. \nIt is done by accessing the `FinalIK` component.\n\n**Body Effector Setup**: provided by the Final IK package, this component is \nrequired for the following body offset features (`MainOffset` and `SpineOffset`)\n\n**MainOffset**: controls body parts such as Hand and Shoulder by adjusting its \ntranslation value to create body moving outward or inward. (see `mainOffset.cs`)\n\n**SpineOffset**: controls body part mainly Spine by adjusting its rotation value \nto create body leaning forward or backward. (see `spineOffset.cs`)\n\n\n## Event System\n\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e\u003c/p\u003e\n\n### System Components\n\n**Event System, Standalone Input Module \u0026 Base Input (added at run-time)**:\nThese components are required for GUI related controls.\n\n**Audio Source**:\nThis component is responsible for playing audio, it also serves as the source for Salsa Lip-sync, \ncontrolled by `audioControl.cs`.\n\n- `AudioClip` is assigned and adjusted at run-time\n- Output is affected by a certain `Audio Mixer` group, which needs to be manually assigned\n\n### Custom Scripts\n\n**Canvas Manager**:\nresponsible for creating the GUI and connecting its functionality.\n\n**Master Control**:\nis the core and the wrapper of all controls, it initializes most of the run-time components. \n(see `masterControl.cs`)\n\n**XML Reader**:\ntakes an `.xml` file, the file specifies scripted behavior of the character, \nand the behavior will be layered on top.\n\n**Beat Detection (Disabled by default)**:\ntakes the current playing audio from the `AudioSource`, \ndetects its beat and triggers `onBeat()` event. \nThe `onIdle()` event is triggered whenever the beat duration is reached.\n\n**Emotion Input**:\nwhen a check signal is sent by XML Reader,\nthe module requests an emotion state from the `EmotionUpdate.cs` and performs an according reaction.\n\n### Run-time Event\n\n**Slide Control**:\ncontrols the change of PowerPoint slide displayed in the background.\n\n**Audio Control**:\nswitches the audio clip for `AudioSource`, the audio clips are switched based on the change of slide as of now.\n\n**Emotion Update**:\nis added by `EmotionInput.cs`. It communicates with the emotion detection module, \ndetects every 2 seconds for 10 seconds, it then returns the most \nfrequent emotion detected back to Emotion Input components.\n\n\n## Plugin\n\n\u003cp align=\"right\"\u003e\u003ca href=\"#top\"\u003eback to top\u003c/a\u003e\u003c/p\u003e\n\n[Final IK](https://assetstore.unity.com/packages/tools/animation/final-ik-14290)\n\n[Salsa Suite](https://assetstore.unity.com/packages/tools/animation/salsa-lipsync-suite-148442)\n\n[Mecanim Control](https://assetstore.unity.com/packages/tools/animation/mecanim-control-15156)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleixingyu%2Fpedagogical-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleixingyu%2Fpedagogical-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleixingyu%2Fpedagogical-agent/lists"}