{"id":15065039,"url":"https://github.com/bhky/nodding-pigeon","last_synced_at":"2025-04-10T13:09:29.216Z","repository":{"id":41410889,"uuid":"487155415","full_name":"bhky/nodding-pigeon","owner":"bhky","description":"Detection and classification of head gestures in videos","archived":false,"fork":false,"pushed_at":"2025-01-24T00:03:08.000Z","size":2673,"stargazers_count":16,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-09T18:50:21.824Z","etag":null,"topics":["gesture-classification","gesture-detection","keras","machine-learning","tensorflow2"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bhky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-30T01:40:25.000Z","updated_at":"2025-04-09T10:21:24.000Z","dependencies_parsed_at":"2024-06-09T08:23:55.778Z","dependency_job_id":"c5aa6bf6-9833-4502-a82a-3a720ed0c065","html_url":"https://github.com/bhky/nodding-pigeon","commit_stats":null,"previous_names":["bhky/head-gesture-detection","bhky/dynamic-gesture-detection"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bhky%2Fnodding-pigeon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bhky%2Fnodding-pigeon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bhky%2Fnodding-pigeon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bhky%2Fnodding-pigeon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bhky","download_url":"https://codeload.github.com/bhky/nodding-pigeon/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248224778,"owners_count":21068075,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gesture-classification","gesture-detection","keras","machine-learning","tensorflow2"],"created_at":"2024-09-25T00:29:57.178Z","updated_at":"2025-04-10T13:09:29.193Z","avatar_url":"https://github.com/bhky.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![logo](logo/nodding-pigeon_logo.png)\n\n[![ci](https://github.com/bhky/nodding-pigeon/actions/workflows/ci.yml/badge.svg)](https://github.com/bhky/nodding-pigeon/actions)\n[![License MIT 1.0](https://img.shields.io/badge/license-MIT%201.0-blue.svg)](LICENSE)\n\n# Introduction\n\nThe **Nodding Pigeon** library provides a pre-trained model and \na simple inference API for detecting **head gestures** in short videos.\nUnder the hood, it uses Google [MediaPipe](https://google.github.io/mediapipe/)\nfor collecting the landmark features.\n\n# Installation\n\nTested for Python 3.8, 3.9, and 3.10.\n\nThe best way to install this library with its dependencies is from PyPI:\n```shell\npython3 -m pip install --upgrade noddingpigeon\n```\nAlternatively, to obtain the latest version from this repository:\n```shell\ngit clone git@github.com:bhky/nodding-pigeon.git\ncd nodding-pigeon\npython3 -m pip install .\n```\n\n# Usage\n\nAn easy way to try the API and the pre-trained model is to\nmake a short video with your head gesture.\n\n## Webcam\n\nThe code snippet below will perform the following:\n- Search for the pre-trained weights file from `$HOME/.noddingpigeon/weights/`,\n  if not exists, the file will be downloaded from this repository.\n- Start webcam.\n- Collect the needed number of frames (default `60`) for the model.\n- End webcam automatically (or you can press `q` to end earlier).\n- Make prediction of your head gesture and print the result to STDOUT.\n```python\nfrom noddingpigeon.inference import predict_video\n\nresult = predict_video()\nprint(result)\n# Example result:\n# {'gesture': 'nodding',\n#  'probabilities': {'has_motion': 1.0,\n#   'gestures': {'nodding': 0.9576354622840881,\n#    'turning': 0.042364541441202164}}}\n```\n\n## Video file\n\nAlternatively, you could provide a pre-recorded video file:\n\n```python\nfrom noddingpigeon.inference import predict_video\nfrom noddingpigeon.video import VideoSegment  # Optional.\n\nresult = predict_video(\n  \"your_head_gesture_video.mp4\",\n  video_segment=VideoSegment.LAST,  # Optionally change these parameters.\n  motion_threshold=0.5,\n  gesture_threshold=0.9\n)\n```\nNote that no matter how long your video is, only the\npre-defined number of frames (`60` for the current model) are used for\nprediction. The `video_segment` enum option controls how the frames \nare obtained from the video, \ne.g., `VideoSegment.LAST` means the last (`60`) frames will be used.\n\nThresholds can be adjusted as needed, see explanation in the \n[head gestures](#head-gestures) section.\n\n## Result format\n\nThe result is returned as a Python dictionary.\n\n```text\n{\n  'gesture': 'turning',\n  'probabilities': {\n    'has_motion': 1.0,\n    'gestures': {\n      'nodding': 0.009188028052449226,\n      'turning': 0.9908120036125183\n    }\n  }\n}\n```\n\n# Head gestures\n\nThe following `gesture` types are available:\n- `nodding` - Repeatedly tilt your head upward and downward.\n- `turning` - Repeatedly turn your head leftward and rightward.\n- `stationary` - Not tilting or turning your head; translation motion is still treated as stationary.\n- `undefined` - Unrecognised gesture or no landmarks detected (usually means no face is shown).\n\nTo determine the final `gesture`:\n- If `has_motion` probability is smaller than `motion_threshold` (default `0.5`),\n  `gesture` is `stationary`. Other probabilities are irrelevant.\n- Otherwise, the largest probability from `gestures` is considered:\n  - If it is smaller than `gesture_threshold` (default `0.9`), `gesture` is `undefined`,\n  - else, the corresponding gesture label is selected (e.g., `nodding`).\n- If no landmarks are detected in the video, `gesture` is `undefined`. \n  The `probabilities` dictionary is empty.\n\n# API\n\n## `noddingpigeon.inference`\n\n### `predict_video`\nDetect head gesture shown in the input video either from webcam or file.\n- Parameters:\n  - `video_path` (`Optional[str]`, default `None`): \n    File path to the video file, or `None` for starting a webcam.\n  - `model` (`Optional[tf.keras.Model]`, default `None`): \n    A TensorFlow-Keras model instance, or `None` for using the default model.\n  - `max_num_frames` (`int`, default `60`):\n    Maximum number of frames to be processed by the model.\n    Do not change when using the default model.    \n  - `video_segment` (`VideoSegment` enum, default `VideoSegment.BEGINNING`):\n    See explanation of [`VideoSegment`](#videosegment).\n  - `end_padding` (`bool`, default `True`): \n    If `True` and `max_num_frames` is set, when the input video has not enough\n    frames to form the feature tensor for the model, padding at the end will be \n    done using the features detected on the last frame.\n  - `drop_consecutive_duplicates` (`bool`, default `True`):\n    If `True`, features from a certain frame will not be used to form the \n    feature tensor if they are considered to be the same as the previous frame.\n    This is a mechanism to prevent \"fake\" video created with static images.\n  - `postprocessing` (`bool`, default `True`):\n    If `True`, the final result will be presented as the Python dictionary\n    described in the [usage](#usage) section, otherwise the raw model output\n    is returned.\n  - `motion_threshold` (`float`, default `0.5`):\n    See the [head gestures](#head-gestures) section.\n  - `gesture_threshold` (`float`, default `0.9`):\n    See the [head gestures](#head-gestures) section.\n- Return:\n  - A Python dictionary if `postprocessing` is `True`, otherwise `List[float]`\n    from the model output.\n\n## `noddingpigeon.video`\n\n### `VideoSegment`\nEnum class for video segment options.\n- `VideoSegment.BEGINNING`: Collect the required frames for the model from the beginning of the video.\n- `VideoSegment.LAST`: Collect the required frames for the model toward the end of the video.\n\n## `noddingpigeon.model`\n\n### `make_model`\nCreate an instance of the model used in this library, \noptionally with pre-trained weights loaded.\n- Parameters:\n  - `weights_path` (`Optional[str]`, default `$HOME/.noddingpigeon/weights/*.h5`): \n    Path to the weights in HDF5 format to be loaded by the model. \n    The weights file will be downloaded if not exists.\n    If `None`, no weights will be downloaded nor loaded to the model.\n    Users can provide path if the default is not preferred. \n    The environment variable `NODDING_PIGEON_HOME` can also be used to indicate\n    where the `.noddingpigeon/` directory should be located.\n- Return:\n  - `tf.keras.Model` object.\n\n# Model training\n\nBrief procedure:\n- Record a few long-ish videos: one for each head gesture done repeatedly \n  with as many variations as possible, and one for stationary.\n- Landmark features in the videos are collected using MediaPipe.\n- During model training, random sub-sequences from the feature collection,\n  correspond to different video segments and gestures, are generated as \n  training samples.\n- This basically means that all samples generated, in each epoch,\n  are very likely not the same as each other.\n  This serves as a good regularization as well.\n- A very simple 1D-convolutional model architecture is used to minimise\n  overfitting.\n\nFor details, see the data collection and model training scripts in the\n[training](https://github.com/bhky/nodding-pigeon/tree/main/training) directory.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbhky%2Fnodding-pigeon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbhky%2Fnodding-pigeon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbhky%2Fnodding-pigeon/lists"}