{"id":17175434,"url":"https://github.com/pantor/griffig","last_synced_at":"2025-04-13T16:30:32.252Z","repository":{"id":55404484,"uuid":"345665955","full_name":"pantor/griffig","owner":"pantor","description":"Robotic Manipulation - Learned in the Real World","archived":false,"fork":false,"pushed_at":"2021-09-15T20:30:16.000Z","size":757,"stargazers_count":48,"open_issues_count":2,"forks_count":10,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-27T07:35:49.846Z","etag":null,"topics":["grasping","robot-learning","robotic-manipulation","robotics"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pantor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-08T13:29:45.000Z","updated_at":"2025-02-15T03:22:31.000Z","dependencies_parsed_at":"2022-08-14T23:40:16.049Z","dependency_job_id":null,"html_url":"https://github.com/pantor/griffig","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pantor%2Fgriffig","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pantor%2Fgriffig/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pantor%2Fgriffig/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pantor%2Fgriffig/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pantor","download_url":"https://codeload.github.com/pantor/griffig/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248743643,"owners_count":21154710,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grasping","robot-learning","robotic-manipulation","robotics"],"created_at":"2024-10-14T23:56:36.878Z","updated_at":"2025-04-13T16:30:31.907Z","avatar_url":"https://github.com/pantor.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1 align=\"center\"\u003eGriffig\u003c/h1\u003e\n  \u003ch3 align=\"center\"\u003e\n    Robotic Manipulation Learned from Imitation and Self-Supervision.\n  \u003c/h3\u003e\n\u003c/div\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/pantor/griffig/actions\"\u003e\n    \u003cimg src=\"https://github.com/pantor/griffig/workflows/CI/badge.svg\" alt=\"CI\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/pantor/griffig/issues\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/issues/pantor/griffig.svg\" alt=\"Issues\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/pantor/griffig/releases\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/v/release/pantor/griffig.svg?include_prereleases\u0026sort=semver\" alt=\"Releases\"\u003e\n  \u003c/a\u003e\n\n  \u003ca href=\"https://github.com/pantor/griffig/blob/master/LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/license-MIT-green.svg\" alt=\"MIT\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\nGriffig is a library (in particular) for 6D robotic grasping, learned from large-scale imitation and self-supervised real-world training. Using an action-centric approach, Griffig does not need object models and requires only a simple depth camera as input. In our [model library](https://griffig.xyz), we publish models pre-trained in densly cluttered bin picking scenarios. Griffig is able to calculate grasp points with high performance (\u003c 70ms), and is yet robust achieving grasp rates as high as 95% for typical use-cases. This repository contains the source code and corresponding library of our upcoming paper *Learning 6D Robotic Grasping using a Fully Convolutional Actor-Critic Architecture*.\n\n[\u003cdiv align=\"center\"\u003e\u003cimg width=\"460\" src=\"https://raw.githubusercontent.com/pantor/griffig/master/doc/systemnew-sm.JPG\"\u003e\u003c/div\u003e](https://griffig.xyz)\n\n\u003cp align=\"center\"\u003e\n  You can find many videos on \u003ca href=\"https://griffig.xyz\"\u003egriffig.xyz\u003c/a\u003e!\n\u003c/p\u003e\n\n## Installation\n\nGriffig is a library for Python 3.7+. You can install Griffig via [PyPI](https://pypi.org/project/griffig/) by\n```bash\n# Install Eigen \u003e3.3, OpenCV \u003e3, and OpenGL (e.g. via apt)\n(sudo) apt install libeigen3-dev libopencv-dev libgl1-mesa-dev libglu1-mesa-dev libegl1-mesa-dev libglew-dev\n\npip install griffig\n```\nOf course, a NVIDIA GPU with corresponding CUDA version is highly recommended. When building from source, you can either call `pip install .` or use CMake directly to build the C++ core of Griffig. We also provide a Docker container to get started more easily.\n\n\n## Tutorial\n\nWe focused on making *Griffig* easy to use! In this tutorial, we use a RGBD pointcloud of the scene to detect a 6D grasp point with an additional pre-shaped gripper stroke. We use a common parallel two-finger gripper and a RealSense D435 camera for recording. Griffig includes a small library of pre-trained models. As with all data-driven methods, make sure to match our robotic system as much as possible. The main output of Griffig is a *grasp point*. Then, the robot should move its gripper to a pre-shaped position and approach the point along a trajectory parallel to its gripper fingers. Be careful of possible collisions that might always happen in bin picking.\n\n[\u003cdiv align=\"center\"\u003e\u003cimg width=\"760\" src=\"https://raw.githubusercontent.com/pantor/griffig/master/doc/input.jpeg\"\u003e\u003c/div\u003e](https://griffig.xyz/dataset/viewer)\n\nA typical scene looks like the color (left) and depth (right) images above. The (orthographic) images are rendered from pointclouds, and show the bin randomly filled with objects of multiple types. Now, we want to find the *best* grasp within the bin. You can find working examples in the corresponding [directory](). At first, we need to import `griffig`, generate a pointcloud, and create the main `Griffig` instance.\n\n```python\nfrom griffig import Affine, Griffig, Gripper, Pointcloud, BoxData\n\n# Griffig requires a RGB pointcloud of the scene\npointcloud = Pointcloud.fromRealsense(camera.record_pointcloud())\n\n# Specify some options\ngriffig = Griffig(\n    model='two-finger',  # Use the default model for a two-finger gripper\n    gripper=Gripper(\n        min_stroke=0.01, # [m]\n        max_stroke=0.10, # [m], to limit pre-shaped stroke\n    ),\n)\n```\n\nNext to the model name (or a path for your own models), we input some details about the robots gripper. In particular, we limit the pre-shaped gripper stroke (or called width). We can now calculate the best possible grasp within the scene. To get the grasp in the global frame in return, we pass Griffig the camera pose of the pointcloud.\n\n```python\ngrasp = griffig.calculate_grasp(pointcloud, camera_pose=camera_pose)\n\n# Make use of the grasp here!\nprint(f'Grasp at {grasp.pose} with probability {grasp.estimated_reward})\n```\nThe grasp pose is given as an [Affx](https://github.com/pantor/affx) affine, which is a very light wrapper around [Eigen's](https://eigen.tuxfamily.org) `Affine3d` class. On top, we can easily generate a heatmap of grasp probabilities as a PIL image to visualize our model.\n\n```python\nheatmap = griffig.calculate_heatmap()\nheatmap.show()\n```\n\nFurthermore, we show the usage of the *Griffig* library in a few more details.\n\n\n### BoxData Class\n\nWe can define a box to avoid grasps outside of the bin (and even worse: grasps of the bin itself). A box can be constructed by its contour given as a polygon in the image frame. To construct a cubic box, we've simplify this method by\n\n```python\nbox_data = BoxData(\n    size=(0.2, 0.3, 0.1),  # (x, y, z) [m]\n    center=(0.0, 0.0, 0.0),  # At the center [m]\n)\n```\nwith the size and center position of the box.\n\n\n### Gripper Class\n\nWe use the gripper class for collision checks and to specify the minimum and maximum gripper stroke\n\n```python\ngripper = Gripper(  # Everything in [m]\n    # Pre-shaped stroke interval\n    min_stroke=0.01,\n    max_stroke=0.10,\n    # Size of a bounding box for optional collision check\n    finger_width=0.02, # Finger width\n    finger_extent=0.008,  # Finger extent (in direction of finger opening/closing)\n    finger_height=0.1,  # Finger height from tip to gripper base\n    # An optional offset for the local grasp pose\n    offset=Affine(z=0.02),\n)\n```\n\n### Griffig Class\n\nThe `Griffig` class is the main interface for grasp calculations. You can create a griffig instance with following arguments:\n\n```python\ngriffig = Griffig(\n    model='two-finger-planar',\n    gripper=gripper,\n    box_data=box_data,  # Might be None\n    avoid_collisions=True,  # If true, check collisions using the given pointcloud and gripper data\n)\n```\n\nGriffig includes a [model library](https://griffig.xyz/model-library) for different tasks and downloads them automatically.\n\n\n### Pointcloud Class\n\nGriffig uses its own pointcloud class as input to its rendering pipeline. Currently, three possible inputs are supported:\n\n```python\n# (1) Input from a realsense frame\npointcloud = Pointcloud(realsense_frame=\u003c...\u003e)\n\n# (2) Input from a ROS Pointcloud2 message\npointcloud = Pointcloud(ros_message=\u003c...\u003e)\n\n# (3) The raw pointer variant...\npointcloud = Pointcloud(type=PointType.XYZRGB, data=cloud_data.ptr())\n\n# Then, we can render the pointcloud as a PIL image\nimage = griffig.render(pointcloud)\nimage.show()\n```\nNote that the pointcloud does only store the pointer to the data, but doesn't hold anything.\n\n\n### Grasp Class\n\nThe calculated grasp contains - of course - information about its grasp pose, but also some more details. At first, we calculate the next grasp based on the current pointcloud input.\n\n```python\ngrasp = griffig.calculate_grasp(pointcloud, camera_pose=camera_pose)  # Get grasp in the global frame using the camera pose\n\nprint(f'Calculated grasp {grasp} in {grasp.calculation_duration} [ms].')  # Calculation duration in [ms]\n```\n\nIf using a GPU, the grasp calculation should not take longer than a few 100ms, and most probably below 70ms! Then, a typical grasp pipeline would look like this:\n\n```python\n# (1) Check if the grasp reward is below a user-defined threshold\nif grasp.estimated_reward \u003c 0.2:  # Grasp probability in [0, 1]\n    print('The bin is probably empty!')\n\napproach_start = grasp.pose * Affine(z=-0.12)  # Approx. finger length [m]\n\n# (2) Move robot to start of approach trajectory\nrobot.move_linear(approach_start)\n\n# (3) Move gripper to pre-shaped grasp stroke\nrobot.move_gripper(grasp.pose.d)\n\n# (4) Move robot to actual grasp pose\nrobot_move_linear(grasp.pose)\n\n# (5) And finally close the gripper\nrobot.close_gripper()\n```\n\nThe robot should have grasped something! If something went wrong, make sure to call `griffig.report_grasp_failure()`, so that griffig will output a different grasp next time.\n\n\n## Development\n\nGriffig is written in C++17 and Python 3.7 (or higher). It is tested on Ubuntu 20.04 against following dependency versions:\n\n- OpenCV 4.5\n- Eigen 3.3.9\n- PyBind11 2.6\n- TensorFlow 2.4\n\nTo build the docker image, call `docker build .`.\n\n\n## License\n\nGriffig is published under the permissive MIT license. Feel free to contact us in case of commercial interest.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpantor%2Fgriffig","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpantor%2Fgriffig","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpantor%2Fgriffig/lists"}