https://github.com/bmw-innovationlab/bmw_omniverse_defect_generation_extension
https://github.com/bmw-innovationlab/bmw_omniverse_defect_generation_extension
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/bmw-innovationlab/bmw_omniverse_defect_generation_extension
- Owner: BMW-InnovationLab
- License: apache-2.0
- Created: 2024-04-02T10:02:01.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-09-25T17:24:46.000Z (about 1 month ago)
- Last Synced: 2025-09-25T19:35:42.310Z (about 1 month ago)
- Language: Python
- Size: 44.7 MB
- Stars: 13
- Watchers: 0
- Forks: 2
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# BMW Defect Generation Extension
This extension for NVIDIA Omniverse allows users to create defects and apply them to prims within an Isaac Sim simulation environment. The extension also provides the means to generate annotated images with customizable domain randomization effects.
|  |  |
|:----------------------------------------------------------:|:-----------------------------------------------------------------:|
# Getting Started
## Adding the Object Parameters
1. **Copy the Target Prim Path**:
- Open the extension within NVIDIA Omniverse.
- Click on the prim(s) to which you wish to apply defects, and then click `Copy`.
- Click `Apply` to add the necessary primvars to the selected prim(s).
## Adding the Defect Parameters
1. **Import the Directory Containing the Defect Folders**:
- Click the `Open` button and select the directory. The folder should have the following structure:
```
data_defects/
├── crack/
├── hole/
└── scratch/
```
2. **Customize and Add the Desired Defects**:
- Click on the dropdown menu for one of the three defect types.
- Choose the semantic label that the defect will be assigned to in the `Defect Semantic` text box.
- Customize the ranges of the width,and length (or radius) randomization for the defects by entering values in the `Min` and `Max` text boxes.
- Customize the ranges of rotation randomization, affecting the `X-axis` in normal mode, and the `X, Y, Z axes` when the `Use Advanced Rotations` checkbox is ticked.
- Determine the number of instances for this specific defect configuration by entering a value in the `Count` text box.
- Click on the `+` sign to add the configured defect to the list.
- The defect information will be indexed by the assigned semantic label.
3. **Import and Export Defect Configuration**:
- The added defects will appear under the corresponding prim path in the `Params` section.
- Click the `Load` button next to a defect object entry to load a defect configuration `JSON` file into the object entry.
- Click the `Select` button next to a defect object entry to make the selected object the current target prim for defect additions.
- Click the `Export` button next to a defect object entry to save the defect configuration of that object into a `JSON` file.
- Click the `Garbage Can` icon in the top right corner of a defect object entry to remove the object from the `Params` list.
- Click the `Garbage Can` button to right the of a defect entry within a defect object to remove that entry from the object's params.
## Adding Randomizer Parameters
- To activate a specific type of randomization tick the checkbox next to the desired randomization type, these include `Light, Camera, Color/Texture` randomizations.
1. **Light Randomizations**:
- Click on the `Light Randomization Parameters` drop down to show/hide the light randomization UI.
- Customize the light intensity, and rotation randomization ranges by changing the `Min` and `Max` text boxes next to the `Light Intensity` and `Light Rotation` labels.
- Modify the RGB color randomization ranges by altering the `Min` and `Max` rows containing a text box for the `R, G, B` channels separately (Note that the ranges are normalized between [0,1]).
- Alter the light scale, and position by changing the `Min` and `Max` text boxes next to the `Light Scale` and `Light Position` labels.
- Specify the number of lights present in the scene by inputting that number in the `Count` text box.
2. **Camera Randomizations**:
* Click on the `Camera Randomization Parameters` drop down to show/hide the camera randomization UI.
* Choose the distance range that the camera will maintain by changing the `Min` and `Max` text boxes next to the `Camera Distance` label (Note that this distance applies only when a `Scattering Prim` is not specified).
* Choose a combination of scattering and look at prims by selecting prims from the scene, and pressing the `Copy` button next to the `Scattering Prim` and `Look At Prim` labels, then finally pressing the `Add` button (Note that the scattering prim and look at prim need to be different).
* Press the `Reset` button to remove the copied scattering and look at prims.
* Designate a range of coordinates for the camera to look at instead of a prim by ticking the checkbox next to the `Use Manual Look At Randomization` then changing the `Min` and `Max` textboxes next to the `X,Y,Z` labels.
* Use the `Reset All` button to delete all camera randomization entries after adding at least one entry.
* Each combination of scattering and look at prim is handled by the extension uniquely, the following is a breakdown of each combination:
* When a `Scattering Prim` and `Look At Prim` are both chosen from the scene using the `Copy` buttons, a camera will be created and randomly scattered on the surface of the scattering prim, while looking at a random point on the look-at prim.
* When a `Scattering Prim` is chosen using the `Copy` button but the user chooses to insert a manual look at range using the `Use Manual Look At Randomization` checkbox, a camera will be created and scattered randomly on the surface of the scattering prim while looking at a random point within the range of look at specified.
* When a `Scattering Prim` is chosen using the `Copy` button but the `Look At Prim` is left empty a camera will be created for every object containing a defect in the scene, these cameras will scatter on the surface of the scattering prim while looking at a random point on their designated look at prim.
* When a `Look At Prim` is chosen using the `Copy` button but the `Scattering Prim` is left empty a camera will be created and scattered inside the volume of a sphere centered on the `Look At Prim`. The camera will then look at a random point on that `Look At Prim`.
* When a look at range is specified using the `Use Manual Look At Randomization` option but the `Scattering Prim` is left empty a camera is created and scattered inside the volume of a sphere centered on the specified look-at range. The camera will then focus on a random point within that range.
* When both the `Scattering Prim` and `Look At Prim` are left empty, a camera will be created for every object containing a defect in the scene. These cameras will be scattered within the volume of spheres centered on the defect objects, while focusing on a random point on each object.
3. **Color Randomizations**:
* Click on the `Color Randomization Parameters` drop down to show/hide the color randomization UI.
* Apply the color randomization to the textures of the specified prims by ticking the checkbox next to the `Texture Color Randomization`.
* Specify the prim for which you want to set the color randomization parameters by selecting the prim and pressing the `Copy` button next to the `Color Prim` label.
* While the prim is still selected, continuously add colors to the randomization pool by using the `RGB Color Picker` and pressing the `Add` button.
* Press the `Reset` button to empty the `Color Prim` textbox.
* Press the `Reset All` button to delete all color randomization entries after adding at least one entry.
4. **Material Randomizations**:
* Click on the `Material Randomization Parameters` drop down to show/hide the material randomization UI.
* Select the prim for which you want to apply material randomizations to in the stage, and press the `Copy` button next to `Material Prim` in the UI.
* To select the materials you want to randomize, there are two options that can be used either separately or together:
- To use materials already found in the stage, check the `Select From Stage` checkbox. A new section called `Materials from stage` will appear, where you can select the materials from the stage and press the `Copy` button.
- To Browse for materials in your nucleus or local directories, simply click the `Browse` button next to `Material Folder` and select the material folder of your choice.
* After selecting the material folder and material prim to apply the randomization on, click the `Add`Button. A UI with the selected Prim Paths and Material folder Paths will appear.
* To remove a material folder for a specific prim, click the `Reset` button.
* To remove all material folders for all prims, click the `Reset All` button.
* Finally, click the `Load Materials` button to load all the selected materials in the stage, and create those that are not in the stage.
**Note**: Material randomization and color randomization can be applied together, meaning the color of the newly selected materials will be randomized as well. To do this, make sure you select `Texture Color Randomization` in the Color Randomization Parameters, and add the colors for the prim you would like to have the randomizations on. It is important to note that randomizing the colors of these materials is done by searching through a list of common color attributes found in *color_attributes.json* in *exts/defect.generation/defect/generation/utils/color_attributes.json*, and applying the selected colors to the found attributes.
* ## Adding the Replicator Parameters
1. **Import the Output Directory**:
- Click the `Open` button and select the directory where you wish the captured images to be written.
2. **Choose the Writer and Annotations**:
- Select the `BMW Format` checkbox if you want the output in the BMW JSON format.
- Select the `Segmentation` checkbox if you want your output to include semantic segmentations.
- Select the `Bounding Box` checkbox if you want your output to include 2D bounding box annotations.
- The output directory will contain a `RenderProduct` folder for each camera capturing within the scene (in case of multiple cameras), as follows:
```
output_directory/
├── RenderProduct_Replicator_1/
│ ├── images/
│ ├── labels/
│ └── semantic_segmentation/
├── RenderProduct_Replicator_2/
│ ├── images/
│ ├── labels/
│ └── semantic_segmentation/
├── RenderProduct_Replicator_3/
│ ├── images/
│ ├── labels/
│ └── semantic_segmentation/
└── ...
```
In case there is only one camera in the scene, the output directory will look as follows:
```
output_directory/
├── images/
├── labels/
├── semantic_segmentation/
```
3. **Select the Replicator Frame and Subframe Count**:
- Enter the number of rendered subframes in the `Render Subframe Count` textbox; to guarantee proper rendering of defects, this value should be between 50 and 100.
- Enter the number of images you want to capture from the scene in the `frame(s)` textbox.
4. **Build the Defect Layer and Run the Scene**:
- Click the `Create Replicator Layer` button to create the replicator xform with all the configured defects.
- Click the `Preview` button to test your scene and the domain randomization.
- Click the `Run for` button to execute the replicator for the number of frames set, which will start capturing images and randomizing the scene and defects.
- Click the `Delete Replicator Layer` button to remove the `Replicator` xform, the `/World/Looks` xform, and every `Projection` xform created in all defect objects.
# Authors
- Joe Khalil, [BMW TechOffice](), Munich, Germany
- Jason El Khoury, [BMW TechOffice](), Munich, Germany
- Angelo Yaghi, [BMW TechOffice](), Munich, Germany
##
- Abboud Hayek, [inmind.ai](https://inmind.ai/), Beirut, Lebanon
- Charbel Bou Maroun, [inmind.ai](https://inmind.ai/), Beirut, Lebanon
- Rachelle Abdel Masih, [inmind.ai](https://inmind.ai/), Beirut, Lebanon