{"id":23891634,"url":"https://github.com/stephendade/segmav","last_synced_at":"2025-04-15T03:46:13.713Z","repository":{"id":183171537,"uuid":"669722493","full_name":"stephendade/segmav","owner":"stephendade","description":"Semantic segmentation based navigation for Ardupilot","archived":false,"fork":false,"pushed_at":"2023-09-01T08:03:47.000Z","size":308,"stargazers_count":6,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-15T03:46:08.942Z","etag":null,"topics":["ardupilot","jetson","mavlink","semantic-segmentation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stephendade.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-23T08:03:12.000Z","updated_at":"2024-08-18T09:48:55.000Z","dependencies_parsed_at":"2023-11-21T07:03:46.327Z","dependency_job_id":"4717e749-defa-46bb-b846-6491f9548fba","html_url":"https://github.com/stephendade/segmav","commit_stats":{"total_commits":12,"total_committers":1,"mean_commits":12.0,"dds":0.0,"last_synced_commit":"d01e256fb1391cd150aecc2fa0d9a1064239a3a2"},"previous_names":["stephendade/segmav"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephendade%2Fsegmav","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephendade%2Fsegmav/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephendade%2Fsegmav/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stephendade%2Fsegmav/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stephendade","download_url":"https://codeload.github.com/stephendade/segmav/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249003938,"owners_count":21196794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ardupilot","jetson","mavlink","semantic-segmentation"],"created_at":"2025-01-04T13:00:23.445Z","updated_at":"2025-04-15T03:46:13.700Z","avatar_url":"https://github.com/stephendade.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Semantic Segmentation based navigation for ArduPilot\n\n![alt text](screenshot.png \"Title\")\n\nThis program uses an NVIDIA Jetson Nano to provide steering angles to stay on a road or footpath.\n\nSemanatic Segmentation is used to determine where the road or footpath is. Then OpenCV is used for form into\nblobs and determine the angle to the centre of the blob.\n\nThis angle is coded into a MAVLink message and sent to a connected ArduPilot flight controller.\n\nA video demonstration is available at https://youtu.be/aOMq3tztdVY.\n\n## How it works\n\nThis section references the screenshot at the top of this page.\n\nSemantic segmentation is used to separate the captured image into label areas: \"Sky, Grass, Road, etc\".\n\nThe largest area (contour) with a \"road\" label (purple) is then found (red polygon).\n\nThis contour is then split into two - halfway in the horizontal plane (blue boxes).\n\nThe centroid of each half is calculated (white points). \n\nA line is drawn between the two points and the angle calculated (white line).\n\nThe angle is put through a 3 point moving average to smooth out any large changes.\n\nThis angle is encoded into a MAVLink message ``SET_POSITION_TARGET_LOCAL_NED``, in Body (relative) frame and the\nforward velocity and yaw components.\n\n\n## Using\n\n### Hardware\n\nThe following hardware is required:\n- Jetson Nano. Other Jetson boards will likely work, but haven't been tested\n- Tuned ground rover with an ArduPilot flight controller\n- Decent forward-facing camera on the Jetson. I used an IMX219 (Raspberry Pi Camera V2)\n\nA cooling fan may be required on the Jetson in hot environments.\n\nThe flight controller should have a telemetry port connected to the Jetson's UART, with a MAVLink router\nacting as a bridge between the UART and TCP/UDP connections. Software like Rpanion-server, mavlink-router or\nMAVProxy is recommended for this.\n\n## Software\n\nThe Jetson Nano is required to have https://github.com/dusty-nv/jetson-inference installed from source. The \"apt-get\"\nversion is too old to use. When installing, ensure the applicable Pre-Trained Segmentation Models are downloaded. The\nfcn-resnet18-cityscapes-1024x512 dataset is the default (and recommended) used.\n\nThe pymavlink, numpy and opencv python3 libraries are also required:\n\n```\npip3 install numpy pymavlink\nsudo apt-get install python3-opencv \n```\n\n## Running Standalone\n\nThe \"segmav.py\" script can be used on pre-recorded videos on the Jetson, to test reliability before using on a moving vehicle.\n\nThe commandline arguments are:\n\n```\n./segmav.py --headless \u003coptions\u003e input output\n```\n\nThe ``\u003coptions\u003e`` commandline arguments are:\n - Any input or output options from https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md\n - Plus any segnet options from https://github.com/dusty-nv/jetson-inference/blob/master/docs/segnet-console-2.md\n\nWhere the ``input`` and ``output`` video streams are the same\nas https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md\n\nIf the output is a ``file://``, a timestamp will automatically added to the file name\n\nFor example:\n\n```\n./segmav.py --headless --input-codec=H264 file://record-20230422-145739.mp4 rtp://192.168.1.124:5400\n```\n\nWhich will read an input video file, perform processing and output to an RTP stream.\n\nOn Ubuntu, use the following command to view the video:\n\n```\ngst-launch-1.0 udpsrc port=5400 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! rtpjitterbuffer ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! autovideosink sync=false\n```\n\nThe segmav console will output the calculated delta heading to stay on the path or road. The heading is in degrees, negative for\ncounter-clockwise and positive for clockwise. The heading is contstrained to +-90 degrees.\n\n## Running on a live system\n\nUse the \"mavsegmav.py\" script to run on a live system. For example:\n\n```\nmavsegmav.py \u003coptions\u003e input output\n```\n\nWhere the ``input`` and ``output`` video streams are the same\nas https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md\n\nIf the output is a ``file://``, a timestamp will automatically added to the file name\n\nThe ``\u003coptions\u003e`` commandline arguments are:\n - ``--rc``: The RC channel number to monitor for control signals. Defaults to RC10\n - ``--pwmlow``: The RC low PWM value. Defaults to 1000\n - ``--pwmmid``: The RC mid PWM value. Defaults to 1500\n - ``--pwmhigh``: The RC high PWM value. Defaults to 2000\n - ``--device``: MAVLink connection string. Can be serial, TPC or UDP connection\n - ``--baud``: MAVLink baud rate, if using serial connection\n - ``--vel``: Foward velocity setpoint in m/s.\n - Plus any input or output options from https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md\n - Plus any segnet options from https://github.com/dusty-nv/jetson-inference/blob/master/docs/segnet-console-2.md\n\n\nSo, for example:\n\n```\nmavsegmav.py --input-flip=rotate-180 --rc=9 --vel=2.5 --device=/dev/ttyTHS1 --baud=57600 csi://0 rtp://192.168.1.124:5400\n```\n\nWill take in video from the CSI0 port, rotate 180 degrees, process it to determine to correct steering angle and output\nthe MAVLink messages to the ``/dev/ttyTHS1`` UART at 57600 baud. It will monitor RC channel 9 for start and stop commands.\nIt will also command a 2.5m/s forward velocity. The processed video stream will be stream via RTP to ``192.168.1.124:5400``\n\nAn RC controller channel is used to control the state of mavsegmav:\n - On RC LOW (1000 PWM), the system will stop and vision processing and send a halt command to the vehicle\n - On RC MID (1500 PWM), the system will record video to file. This is useful for capturing test datasets\n - On RC HIGH (2000 PWM), the system will send heading and speed commands to the flight controller. The vehicle must be in\n   the Armed state. mavsegmav will automatically switch to guided mode.\n\nIf a buzzer is fitted to the flight controller, a tune will sound each time one of the above actions is switched to.\n\nNote the segmentation may take ~10 seconds to start up or stop after a RC switch action. MAVLink ``STATUSTEXT`` messages will also\nbe sent to any connected ground stations annoncing the start or stop.\n\nFor automatic operation, a supplied systemd service \"segmav.service\" can be used. Ensure to change the path or username details\nto match your system.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephendade%2Fsegmav","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstephendade%2Fsegmav","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephendade%2Fsegmav/lists"}