{"id":19944244,"url":"https://github.com/lazauk/slm-phi-3-robotics","last_synced_at":"2026-02-13T17:04:35.901Z","repository":{"id":251132327,"uuid":"836457440","full_name":"LazaUK/SLM-Phi-3-Robotics","owner":"LazaUK","description":"Robotics navigation solution (on example of Pioneer 3-DX mobile robot in CoppeliaSim) powered by Phi-3 small language model and AMD AI PC.","archived":false,"fork":false,"pushed_at":"2024-08-01T20:09:56.000Z","size":28,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-30T14:40:03.791Z","etag":null,"topics":["ai","amd","coppeliasim","phi-3","robotics","simulation","slm"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LazaUK.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-31T22:32:33.000Z","updated_at":"2024-08-02T13:11:40.000Z","dependencies_parsed_at":"2024-11-13T00:19:42.277Z","dependency_job_id":null,"html_url":"https://github.com/LazaUK/SLM-Phi-3-Robotics","commit_stats":null,"previous_names":["lazauk/slm-phi-3-robotics"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LazaUK/SLM-Phi-3-Robotics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FSLM-Phi-3-Robotics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FSLM-Phi-3-Robotics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FSLM-Phi-3-Robotics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FSLM-Phi-3-Robotics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LazaUK","download_url":"https://codeload.github.com/LazaUK/SLM-Phi-3-Robotics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LazaUK%2FSLM-Phi-3-Robotics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29412671,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T06:24:03.484Z","status":"ssl_error","status_checked_at":"2026-02-13T06:23:12.830Z","response_time":78,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","amd","coppeliasim","phi-3","robotics","simulation","slm"],"created_at":"2024-11-13T00:19:38.926Z","updated_at":"2026-02-13T17:04:35.883Z","avatar_url":"https://github.com/LazaUK.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Navigating the maze with Phi-3-powered \"System 2\" logic\n\n### Project Overview:\nThis repo demonstrates the use of Microsoft’s **Phi-3** SLM to power so-called “_System 2_” logic of Pioneer 3-DX mobile robot and navigate the maze.\n\nAs humans, we ourselves often operate in System 1 or System 2 modes:\n- \"_System 1_\" is our auto-pilot, e.g. for routine tasks like driving a familiar route;\n- \"_System 2_\" kicks in when something unexpected happens and it requires our conscious decision-making, e.g. braking to avoid hitting an obstacle.\n\nIn the robot's case:\n- \"_System 1_\" is used for a “clear path” situations, which don’t require computational power and thus the robot simply moves forward;\n- \"_System 2_\" is activated when the robot detects an “obstacle”, to feed then its telemetry to Phi-3 model and decide where to turn.\n\n\u003eNote: The maze itself was built in **Coppelia Robotics**’s simulation environment, while the compute power to run this demo was provided by **AMD**’s AI PC.\n\n### Simulation Environment:\n_Pioneer 3-DX_'s virtual robotic solution was provided by Coppelia Robotics in _CoppeliaSim_. Its default configuration was enhanced with 4 additional sensors, used in this demo:\n- 1x Video camera;\n- 3x Proximity sensors.\n\n_CoppeliaSim_'s default 3D scene was also re-designed to build a maze, with internal and external walls made detectable to simulate real-time proximity sensing.\n\n### Solution Implementation:\n1. Programmatic interactions with CoppeliaSim require installation of ZMQ Python package. We'll re-use its _RemoteAPIClient_ class:\n``` Python\nfrom coppeliasim_zmqremoteapi_client import RemoteAPIClient\n```\n2. To run Phi-3 locally on AMD AI PC, you need to use Phi-3 model in ONNX format. Pre-trained versions of Phi-3 models can be found on this [HuggingFace page](https://huggingface.co/microsoft).\n3. This solution was tested on AMD AI PC - **_Minisforum Venus UM790 PRO_** with CPU and GPU, kindly provided by **Hackster** and **AMD** teams. If your AMD hardware comes with NPU, you can use Vitis AI Quantizer to convert the model into INT8 quantisation, required for inference on Ryzen AI.\n4. Phi-3 model is initialised from the provided ONNX files. Prompt generation is enabled by the Helper function:\n``` Python\ndef system2_logic(distance_left, distance_right):\n    prompt = f\"\u003c|system|\u003e\u003cYOUR_SPECIFIC_PROMPT\u003e\u003c|end|\u003e\u003c|assistant|\u003e\"\n    input_tokens = tokenizer.encode(prompt)\n    params.input_ids = input_tokens\n    response = \"\"\n\n    generator = og.Generator(model, params)\n\n    while not generator.is_done():\n        generator.compute_logits()\n        generator.generate_next_token()\n\n        new_token = generator.get_next_tokens()[0]\n        response += tokenizer_stream.decode(new_token)\n\n    del generator\n\n    return response\n```\n5. To handle potential inconsistencies between the model's output and the expected format, a Helper function is provided to use RegEx to extract speed values for the robot's actuators (left and right wheels):\n``` Python\ndef extract_first_list(data):\n    # Join the list into a single string\n    data_str = ''.join(data)\n    \n    # Use regex to find the first list pattern\n    match = re.search(r'\\[\\s*-?\\d+,\\s*-?\\d+\\]', data_str)\n    \n    if match:\n        # Extract the matched string\n        list_str = match.group(0)\n        \n        # Convert the string to a list of integers\n        list_values = eval(list_str)\n        \n        return list_values\n    else:\n        return None\n```\n6. Obstacle detection is driven by the handling of robot's front-centre proximity sensor:\n``` Python\nresponse, distance, _, _, _ = sim.handleProximitySensor(proximitySensorHandle)\n```\n7. If no obstacles detected, the robot is using its _System 1_ logic to maintain the same speed on its left and right wheels:\n``` Python\nsim.setJointTargetVelocity(leftMotorHandle, \u003cREQUIRED_SPEED\u003e)\nsim.setJointTargetVelocity(rightMotorHandle, \u003cREQUIRED_SPEED\u003e)\n```\n8. If the front-centre proximity sensor will detect any objects, the robot will stop immediately, read distance values from its left and right proximity sensors and request instruction from Phi-3 model as per defined _System 2_ logic:\n``` Python\n_, distance_left, _, _, _ = sim.handleProximitySensor(proximitySensorHandleLeft)\n_, distance_right, _, _, _ = sim.handleProximitySensor(proximitySensorHandleRight)\nphi3_recommendations = system2_logic(distance_left, distance_right)\nphi3_recommendations = extract_first_list(phi3_recommendations)\n```\n9. Relevant speed instructions then will be send to left and right wheels to perform turning manoeuvre:\n```\nsim.setJointTargetVelocity(leftMotorHandle, left_wheel_speed)\nsim.setJointTargetVelocity(rightMotorHandle, right_wheel_speed)\n```\n\n### Demo Video:\nYou can find short demo of the working solution [here](https://youtu.be/bX2gU0sx1bE) on YouTube.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flazauk%2Fslm-phi-3-robotics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flazauk%2Fslm-phi-3-robotics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flazauk%2Fslm-phi-3-robotics/lists"}