### Install Unity Perception Package

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/SetupSteps.md

Instructions for installing the Unity Perception package via the Package Manager. This involves adding the package by its name, 'com.unity.perception'.

```text
Window → Package Manager
+
Add package by name...
com.unity.perception
Add
```

--------------------------------

### Inspect Prefab for Labels

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This action guides the user to open a specific prefab asset within the Unity project to inspect its components and understand the labels associated with the GameObject. This is crucial for configuring labelers correctly.

```Unity Editor
Navigate to Assets/Samples/Perception/<Perception Package Version>/Tutorial Files/Foreground Objects/Phase 1/Prefabs
Double-click drink_whippingcream_lucerne.prefab
```

--------------------------------

### Add Unity Perception Package from Git URL

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This snippet demonstrates how to add the Unity Perception package to your project using its Git URL via the Unity Package Manager. It also shows how to specify a version or install from a local clone.

```unity-package-manager
com.unity.perception
# To install a specific version: com.unity.perception@0.11.0-preview.1
```

--------------------------------

### Install Jupyter Notebook

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools.md

Installs the Jupyter Notebook environment using pip. This is the first step to enable interactive Python analysis.

```shell
pip install jupyter
```

--------------------------------

### Understanding SOLO Dataset JSON Structure

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This example illustrates the structure of a typical `step0.frame_data.json` file within a Unity Perception generated dataset. It highlights key fields like 'labelId', 'labelName', 'instanceId', 'origin', and 'dimension' used for bounding box annotations.

```json
{
  "captures": [
    {
      "labelId": 1,
      "labelName": "candy_minipralines_lindt",
      "instanceId": 101,
      "origin": [100, 150],
      "dimension": [50, 75]
    }
  ]
}
```

--------------------------------

### Start Jupyter Notebook Server

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools.md

Starts a local Jupyter Notebook server. Once running, you can access the Jupyter interface through your web browser to open and execute notebooks.

```shell
jupyter notebook
```

--------------------------------

### Install OpenEXR Package

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools-fiftyone.md

Installs the OpenEXR Python package using pip. OpenEXR is a required dependency for pysolotools-fiftyone and requires system binaries to be installed first.

```Shell
pip install openexr
```

--------------------------------

### Unity Perception: Scene Setup for Human Pose

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/HumanPose/TUTORIAL.md

This snippet outlines the initial scene setup required for the Human Pose Labeling tutorial. It includes duplicating an existing scene, renaming it, and ensuring the Perception Camera is configured correctly by disabling previous labelers.

```Unity Editor
- 🟢 Action: Open the project you created in the Perception Tutorial steps mentioned above. Duplicate `TutorialScene` and name the new Scene `HPE_Scene`. Open `HPE_Scene`.

We will use this duplicated Scene in this tutorial so that we do not lose our grocery object detection setup from the Perception Tutorial.

- 🟢 Action: If your Scene already contains a Scenario object from the Perception Tutorial, remove all previously added Randomizers from this Scenario.
- 🟢 Action: If your Scene does not already contain a Scenario, create an empty GameObject, name it `Simulation Scenario`, and add a `Fixed Length Scenario` component to it.

Your Scenario should now look like this:

<p align="center">
<img src="../images/Tutorial/fixedscenarioempty.png" width="500"/>
</p>

- 🟢 Action: Select `Main Camera` and in the _**Inspector**_ view of the `Perception Camera` component, **disable** all previously added labelers using the check-mark in front of each. We will be using a new labeler in this tutorial.
```

--------------------------------

### Install pysolotools-fiftyone

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools-fiftyone.md

Installs the pysolotools-fiftyone Python package using pip. This package provides the custom importer for Voxel51 to visualize SOLO datasets.

```Shell
pip install pysolotools-fiftyone
```

--------------------------------

### PerceptionCamera Setup Script

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Features/BoundingBox3DLabeler.md

Illustrates the C# script 'PerceptionCamera' that needs to be added to the main camera in a Unity scene for the BoundingBox3D Labeler to function correctly. This is a crucial step in the setup process.

```csharp
// Add script PerceptionCamera to the Camera in the scene
```

--------------------------------

### Example Scenario JSON Configuration Loading (Bash)

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Randomization/Scenarios.md

Demonstrates how to load a Scenario JSON configuration into a Windows Unity player using a command-line flag. This allows runtime configuration of simulation behavior.

```bash
.\PerceptionTutorial.exe --scenario-config-file scenarioConfiguration.json
```

--------------------------------

### Accessing Perception Settings in Unity

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This snippet shows how to navigate the Unity editor to access the Perception Settings window, which allows users to manage dataset generation locations and other Perception-related configurations.

```unity-editor-menu
Edit → Project Settings → Perception
```

--------------------------------

### Create New Unity Project (HDRP)

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

Instructions for creating a new Unity project using the High Definition Render Pipeline (HDRP) template. This is a prerequisite for using the Perception package, which only supports HDRP.

```Unity
Open Unity and create a new **3D (HDRP)** project.
Name your new project _**Perception Tutorial**_.
```

--------------------------------

### Install Pysolotools using Pip

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools.md

This command installs the Pysolotools Python package using pip. Pysolotools provides tools for processing, analyzing, and converting SOLO datasets.

```Shell
pip install pysolotools
```

--------------------------------

### Clone Pysolotools Repository

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools.md

Clones the Pysolotools Git repository to your local machine and navigates into the cloned directory. This provides access to the example Jupyter notebook and other tools.

```shell
git clone https://github.com/Unity-Technologies/pysolotools
cd pysolotools
```

--------------------------------

### Install Notebook Dependencies

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools.md

Installs essential Python libraries required for the Jupyter notebook to function correctly. These include libraries for plotting, data manipulation, and statistical visualization.

```shell
pip install matplotlib
pip install pandas
pip install seaborn
```

--------------------------------

### Install Pysolotools

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/convert_to_coco.md

Installs the pysolotools Python package using pip. This package provides tools for processing, analyzing, and converting SOLO datasets.

```Shell
pip install pysolotools
```

--------------------------------

### Unity Perception Annotation Example (JSON)

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/PerceptionCamera.md

An example of an annotation in JSON format, demonstrating the structure for environment state metrics, including label hierarchy.

```JSON
{
      "@type": "type.unity.com/unity.solo.EnvironmentStateMetric",
      "id": "environment",
      "sensorId": "",
      "annotationId": "",
      "description": "Captures details about the environment such as label hierarchy",
      "hierarchy": [
        {
          "instance_id": 1,
          "parent_instance_id": null,
          "child_instance_ids": [ 3, 4]
        },
        {
          "instance_id": 5,
          "parent_instance_id": 2,
          "child_instance_ids": []
        },
        {
          "instance_id": 2,
          "parent_instance_id": null,
          "child_instance_ids": [5]
        }
      ]
    }
```

--------------------------------

### Example Metadata JSON Output

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/PerceptionCamera.md

This JSON structure represents an example of the metadata generated by a custom labeling tag. It includes frame information, sequence, step, and a list of metrics with custom values.

```json
{
  "frame": 3,
  "sequence": 1,
  "step": 0,
  "captures": [],
  "metrics": [
    {
      "@type": "type.unity.com/unity.solo.GenericMetric",
      "id": "metadata",
      "sensorId": "",
      "annotationId": "",
      "description": "Metadata labeler",
      "values": [
        {
          "SceneLight-Directional Light": {
            "Color": [
              255,
              244,
              214,
              255
            ],
            "Intensity": 70000.0
          },
          "instances": [
            {
              "instanceId": "1",
              "Values": {
                "Values": [
                  "test1",
                  "test2"
                ]
              }
            },
            {
              "instanceId": "3",
              "SqrMagnitudeToMainCamera": {
                "sqrMagnitudeToMainCamera": 37.0
              },
              "gameObjectName": {
                "name": "Cube"
              }
            },
            {
              "instanceId": "4",
              "transformRecord": {
                "rotationQuaternion": [
                  0.354044646,
                  0.797931969,
                  -0.485905975,
                  0.04303877
                ],
                "localRotationQuaternion": [
                  0.354044646,
                  0.797931969,
                  -0.485905975,
                  0.04303877
                ],
                "rotationEuler": [
                  53.6987267,
                  207.719666,
                  62.0927124
                ],
                "localRotationEuler": [
                  53.6987267,
                  207.719666,
                  62.0927124
                ],
                "parentName": "none",
                "lossyScale": [
                  1.0,
                  0.9999999,
                  1.0
                ],
                "localScale": [
                  1.0,
                  1.0,
                  1.0
                ]
              },
              "gameObjectTag": {
                "unityTag": "Untagged"
              }
            }
          ]
        }
      ]
    }
  ]
}
```

--------------------------------

### Constant Sampler Example

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Randomization/Samplers.md

Demonstrates how to use the Constant Sampler to generate a fixed random value. It involves instantiating the sampler and setting its value.

```csharp
var sampler = new ConstantSampler();

// set sampled value to be 25
sampler.value = 25;

// Generate a sample
var sample = sampler.Sample(); // returns 25
var sample2 = sampler.Sample(); // returns 25
var sample3 = sampler.Sample(); // returns 25
```

--------------------------------

### Assign Label Configs to Perception Camera Labelers

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This describes the process of assigning the created label configurations to the Labeler components attached to the Perception Camera. It involves dragging and dropping assets or using the asset selection window.

```Unity Editor
Select Main Camera → Inspector → Assign TutorialIdLabelConfig to first three Labelers.
Assign TutorialSemanticSegmentationLabelConfig to the fourth Labeler.
```

--------------------------------

### Unity C# Rotation Randomizer Example

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Randomization/Randomizers.md

This C# code example demonstrates a basic Rotation Randomizer for Unity Perception. It shows how to define a custom Randomizer by inheriting from the base Randomizer class, exposing parameters via Unity's Serializable attribute, and implementing the OnIterationStart hook to apply rotations to tagged GameObjects.

```csharp
[Serializable]
[AddRandomizerMenu("Perception/Rotation Randomizer")]
public class RotationRandomizer : Randomizer
{
    public Vector3Parameter rotation = new Vector3Parameter();

    protected override void OnIterationStart()
    {
        var taggedObjects = tagManager.Query<RotationRandomizerTag>();
        foreach (var taggedObject in taggedObjects)
            taggedObject.transform.rotation = Quaternion.Euler(rotation.Sample());
    }
}
```

--------------------------------

### Object Count Labeler Example (JSON)

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/PerceptionCamera.md

Provides an example of the JSON output format for the ObjectCountLabeler, showing the label ID, name, and count for a single object.

```JSON
{
    "label_id": 25,
    "label_name": "drink_whippingcream_lucerne",
    "count": 1
}
```

--------------------------------

### Visualize Datasets with Voxel51 Viewer

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/README.md

This guide explains how to use Voxel51 with a custom SOLO importer to visualize datasets. It walks through the process of integrating the two tools for dataset visualization.

```Python
# Example usage with Voxel51 (requires installation and custom importer)
# import fiftyone as fo
# import fiftyone.zoo as foz
# from pysolotools_fiftyone import SOLOImporter

# dataset = fo.Dataset(name="my-solo-dataset")
# dataset.add_importer(SOLOImporter(dataset_dir="path/to/your/solo/dataset"))
# dataset.load_dataset()
# session = fo.launch_app(dataset)

```

--------------------------------

### Add ForegroundObjectPlacementRandomizer

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This snippet demonstrates how to add the ForegroundObjectPlacementRandomizer component to a Unity Perception scenario. It involves selecting prefabs and setting specific parameters for depth, separation distance, and placement area.

```csharp
// Add ForegroundObjectPlacementRandomizer to your list of Randomizers.
// Select Assets/Samples/Perception/<Perception Package Version>/Tutorial Files/Foreground Objects/Phase 1/Prefabs.
// Set Depth = -3, Separation Distance = 1.5, Placement Area = (5,5).
```

--------------------------------

### FlatEndpoint File Structure Example

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Features/CustomEndpoints.md

Demonstrates the file structure for a hypothetical FlatEndpoint, showing how simulation data (frames, annotations, camera definitions) is organized within a base path and numbered folders.

```yaml
- {BASE_PATH}/FLAT_{num}
  - frame-0.json
  - frame-1.json
  - ...
  - annotation-definition-bbox2d.json
  - annotation-definition-obj-count.json
  - ...
  - camera.sem-seg_1.0.json
```

--------------------------------

### JSON Example: Light Count Metadata

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Features/MetadataLabeler.md

This JSON output demonstrates the metadata generated by the LightCountReporterTag, showing the counts of enabled and disabled lights in the scene.

```json
{
  "@type": "type.unity.com/unity.solo.GenericMetric",
  "id": "metadata",
  "sensorId": "",
  "annotationId": "",
  "description": "Metadata labeler",
  "values": [
    {
      "light_count": {
        "enabled_light_count": 23,
        "disabled_light_count": 3
      }
    }
  ]
}
```

--------------------------------

### Unity Perception: Add BackgroundObjectPlacementRandomizer

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This snippet shows how to add the 'BackgroundObjectPlacementRandomizer' component to a Scenario GameObject in Unity. This randomizer is used to spawn background objects with variations.

```csharp
// Click _**Add Randomizer**_, and from the list choose `BackgroundObjectPlacementRandomizer`.
```

--------------------------------

### Rendered Object Info Labeler Example (JSON)

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/PerceptionCamera.md

Illustrates the JSON output for the RenderedObjectInfoLabeler, detailing the instance ID, resolved label ID, and visible pixel count for a single visible object.

```JSON
{
    "label_id": 24,
    "instance_id": 320,
    "visible_pixels": 28957
}
```

--------------------------------

### Apply Background Randomizers in Unity Perception

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This section details how to add and configure various randomizers for background objects in Unity Perception. It includes adding TextureRandomizer, HueOffsetRandomizer, and RotationRandomizer, and setting their properties.

```Unity C#
// Add TextureRandomizer
// Add HueOffsetRandomizer
// Add RotationRandomizer

// For TextureRandomizer: Add Folder -> Assets/Samples/Perception/<Perception Package Version>/Tutorial Files/Background Textures
// For RotationRandomizer: Ensure min values are 0 and max values are 360 for all ranges
```

--------------------------------

### Unity Perception: Add Fixed Length Scenario Component

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This snippet demonstrates how to add a 'Fixed Length Scenario' component to a GameObject in Unity. This component controls the execution flow and timing of simulation randomizers.

```csharp
// In the Inspector view of the new GameObject, add a new 'Fixed Length Scenario' component.
```

--------------------------------

### Unity Perception Keypoint Annotation Example

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/HumanPose/TUTORIAL.md

This JSON snippet demonstrates the structure of keypoint annotations captured by the Perception Camera. It includes the index, coordinates (x, y), and state for each detected human joint, formatted for compatibility with the COCO dataset.

```json
{
  "pose": "unset",
  "keypoints": [
    {
      "index": 0,
      "x": 0.0,
      "y": 0.0,
      "state": 0
    },
    {
      "index": 1,
      "x": 649.05615234375,
      "y": 300.65264892578125,
      "state": 2
    },
    {
      "index": 2,
      "x": 594.4522705078125,
      "y": 335.8978271484375,
      "state": 2
    },
    {
      "index": 3,
      "x": 492.46444702148438,
      "y": 335.72491455078125,
      "state": 2
    },
    {
      "index": 4,
      "x": 404.89456176757813,
      "y": 335.57647705078125,
      "state": 2
    },
    {
      "index": 5,
      "x": 705.89404296875,
      "y": 335.897705078125,
      "state": 2
    },
    {
      "index": 6,
      "x": 807.74688720703125,
      "y": 335.7244873046875,
      "state": 2
    },
    {
      "index": 7,
      "x": 895.1993408203125,
      "y": 335.57574462890625,
      "state": 2
    },
    {
      "index": 8,
      "x": 612.51654052734375,
      "y": 509.065185546875,
      "state": 2
    },
    {
      "index": 9,
      "x": 608.50006103515625,
      "y": 647.0631103515625,
      "state": 2
    },
    {
      "index": 10,
      "x": 611.7791748046875,
      "y": 797.7828369140625,
      "state": 2
    },
    {
      "index": 11,
      "x": 682.175048828125,
      "y": 509.06524658203125,
      "state": 2
    },
    {
      "index": 12,
      "x": 683.1016845703125,
      "y": 649.64434814453125,
      "state": 2
    },
    {
      "index": 13,
      "x": 686.3271484375,
      "y": 804.203857421875,
      "state": 2
    },
    {
      "index": 14,
      "x": 628.012939453125,
      "y": 237.50531005859375,
      "state": 2
    },
    {
      "index": 15,
      "x": 660.023193359375,
      "y": 237.50543212890625,
      "state": 2
    },
    {
      "index": 16,
      "x": 0.0,
      "y": 0.0,
      "state": 0
    },
    {
      "index": 17,
      "x": 0.0,
      "y": 0.0,
      "state": 0
    }
  ]
}
```

--------------------------------

### Assign Randomizer Tags to Prefabs in Unity Perception

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This snippet explains how to assign specific RandomizerTag components to prefabs to ensure they are targeted by the corresponding randomizers. It covers adding TextureRandomizerTag, HueOffsetRandomizerTag, and RotationRandomizerTag to selected prefabs.

```Unity C#
// Navigate to Prefabs folder
// Select all files
// Add TextureRandomizerTag component
// Add HueOffsetRandomizerTag component
// Add RotationRandomizerTag component
```

--------------------------------

### Add Labeling Component in Unity

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

Users can add the 'Labeling' component to their custom prefabs using the 'Add Component' button in the Unity Inspector. This component is essential for assigning labels to objects within the Perception package.

```unity-editor-inspector
Add Component button in the Inspector
```

--------------------------------

### Unity Perception: Running Simulation for Visualizations

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This snippet outlines the steps to run a Unity Perception simulation after setting up a scene. It details navigating the project, placing prefabs, and initiating the simulation to observe visual outputs like bounding boxes and semantic segmentation.

```Unity Editor Actions
1. Navigate to `Assets/Samples/Perception/<Perception Package Version>/Tutorial Files/Foreground Objects/Phase 1/Prefabs` in the Project tab.
2. Drag and drop a Prefab from this folder into the Scene Hierarchy.
3. Click the play (▷) button in the editor to run the simulation.
```

--------------------------------

### Record Simulation Start Time

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Features/CustomEndpoints.md

Records the start time of a simulation for duration calculation. This method is called when a simulation begins.

```csharp
public void SimulationStarted(SimulationMetadata metadata)
{
    // record when the simulation started so we can use this to calculate
    // duration of the simulation on the SimulationEnded function
    m_SimulationStartTime = DateTimeOffset.Now;
}
```

--------------------------------

### Run pysolotools-fiftyone

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools-fiftyone.md

Executes the pysolotools-fiftyone command-line tool, launching the FiftyOne viewer in the default web browser. Replace '<SOLO_PATH>' with the actual path to your SOLO dataset.

```Shell
pysolotools-fiftyone <SOLO_PATH>
```

--------------------------------

### Create Prefabs from Models in Unity

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This action allows users to quickly generate Unity prefabs from multiple 3D models. After selecting the desired models, users can access this function through the Unity editor's top menu bar, creating prefabs in the same directories as their source models.

```unity-editor-menu
Assets -> Perception -> Create Prefabs from Selected Models
```

--------------------------------

### Unity Perception: Import Human Pose Labeling Samples

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/HumanPose/TUTORIAL.md

This snippet demonstrates the process of importing sample assets for Human Pose Labeling and Randomization within the Unity Perception package. It involves using the Package Manager to locate and import the specific sample bundle.

```Unity Editor
- 🟢 Action: Open _**Package Manager**_ and select the Perception package, which should already be present in the navigation pane to the left side.
- 🟢 Action: From the list of ***Samples*** for the Perception package, click on the ***Import*** button for the sample bundle named _**Human Pose Labeling and Randomization**_.
```

--------------------------------

### Run Docker Container for Dataset Insights

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/DatasetInsights.md

Launches a Docker container for the Dataset Insights notebook, mapping a local download path to the container's data directory and exposing the notebook's web interface.

```Shell
docker run -p 8888:8888 -v <download path>/data:/data -t unitytechnologies/datasetinsights:latest
```

--------------------------------

### JSON Example: Transform Metadata

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Features/MetadataLabeler.md

This JSON structure shows an example of metadata collected for a GameObject's transform, including its rotation, position, and scale, tied to its instance ID.

```json
{
  "@type": "type.unity.com/unity.solo.GenericMetric",
  "id": "metadata",
  "sensorId": "",
  "annotationId": "",
  "description": "Metadata labeler",
  "values": [
    {
      "instances": [
        {
          "instanceId": "2",
          "transformRecord": {
            "rotationEuler": [ 0.0, 180.0, 0.0 ],
            "position": [ 0.0, 1.0, -4.0 ],
            "localScale": [ 1.0, 1.0, 1.0 ],
            ...
          }
        },
        ...
      ]
    }
  ]
}
```

--------------------------------

### Unity Perception: Object Placement Logic

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/FAQ/FAQ.md

This snippet details the core logic for placing objects within a Unity Perception scenario. It includes calculating average bounds size and scale to determine placement density, selecting prefabs, and instantiating them using Poisson disk sampling for non-overlapping placement. It also handles object caching and resetting.

```csharp
public class Randomizer : MonoBehaviour
{
    // ... (other fields and methods)

    protected void Awake()
    {
        // Calculate the average bounds size for the prefabs included in this categorical parameter
        var averageBoundsSize = CalculateAverageBoundsSize();

        // Calculate average scale based on the scale range given
        var averageScale = 1f;
        var sampler = (UniformSampler)scaleParameter.value;
        if (sampler != null)
        {
            averageScale = (sampler.range.minimum + sampler.range.maximum) / 2;
        }

        // Use average bounds size and average scale to guess the maximum number of objects that can be placed without having them overlap.
        // This is a heuristic to help us start the placement process. The actual number of items placed will usually be much smaller.
        m_SelectionPoolSizePerFrame = (int)(placementArea.x * placementArea.y / (averageBoundsSize * averageScale));
    }

    protected void OnIterationStart()
    {
        m_ScaleFactor = scaleParameter.Sample();
        m_SelectedPrefabs.Clear();

        // Select a random number of prefabs for this frame. Placement calculations will be done based on this subset.
        for (var i = 0; i < m_SelectionPoolSizePerFrame; i++)
        {
            var randIndex = (int)Mathf.Round((m_IndexSelector.Sample() * prefabs.categories.Count) - 0.5f);
            m_SelectedPrefabs.Add(prefabs.categories[randIndex].Item1);
        }

        // Calculate the minimum separation distance needed for the selected prefabs to not overlap.
        var separationDistance = CalculateMaxSeparationDistance(m_SelectedPrefabs);
        var seed = SamplerState.NextRandomState();
        var placementSamples = PoissonDiskSampling.GenerateSamples(
            placementArea.x, placementArea.y, separationDistance, seed);
        var offset = new Vector3(placementArea.x, placementArea.y, 0f) * -0.5f;
        foreach (var sample in placementSamples)
        {
            // Pick a random prefab from the selected subset and instantiate it.
            var randIndex = (int)Mathf.Round((m_IndexSelector.Sample() * m_SelectedPrefabs.Count) - 0.5f);
            var instance = m_GameObjectOneWayCache.GetOrInstantiate(m_SelectedPrefabs[randIndex]);
            instance.transform.position = new Vector3(sample.x, sample.y, depth) + offset;
            instance.transform.localScale = Vector3.one * m_ScaleFactor;
        }

        placementSamples.Dispose();
    }

    protected void OnIterationEnd()
    {
        m_GameObjectOneWayCache.ResetAllObjects();
    }

    /// <summary>
    /// Calculates the separation distance needed between placed objects to be sure that no two objects will overlap
    /// </summary>
    /// <returns>The max separation distance</returns>
    float CalculateMaxSeparationDistance(ICollection<GameObject> categories)
    {
        var maxBoundsSize = m_GameObjectBoundsSizeCache.Where(item => categories.Contains(item.Key)).Max(pair => pair.Value);
        return maxBoundsSize * m_ScaleFactor * nonOverlapGuarantee;
    }

    float CalculateAverageBoundsSize()
    {
        foreach (var category in prefabs.categories)
        {
            var prefab = category.Item1;

            prefab.transform.localScale = Vector3.one;
            var renderers = prefab.GetComponentsInChildren<Renderer>();
            var totalBounds = new Bounds();
            foreach (var renderer in renderers)
            {
                totalBounds.Encapsulate(renderer.bounds);
            }

            var boundsSize = totalBounds.size.magnitude;
            m_GameObjectBoundsSizeCache.Add(prefab, boundsSize);
        }

        return m_GameObjectBoundsSizeCache.Values.Average();
    }
}
```

--------------------------------

### Add Automatic Labels to Configurations in Unity Perception

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

After applying automatic labels to prefabs, this action allows users to add these generated labels to their project's label configurations. Users can select specific configurations, like 'TutorialIdLabelConfig' or 'TutorialSemanticSegmentationLabelConfig', and add all associated labels using the 'Add All Labels' button.

```unity-editor-inspector
Click 'Add Automatic Labels of All Selected Assets to Config...'
Click 'Add All Labels' for selected configurations (e.g., TutorialIdLabelConfig, TutorialSemanticSegmentationLabelConfig)
```

--------------------------------

### ColorRandomizer Example

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Randomization/RandomizerTags.md

Demonstrates a ColorRandomizer in Unity Perception that queries for GameObjects with a ColorRandomizerTag and applies a random base color to their MeshRenderer material.

```csharp
[Serializable]
[AddRandomizerMenu("Perception/Color Randomizer")]
public class ColorRandomizer : Randomizer
{
    static readonly int k_BaseColor = Shader.PropertyToID("_BaseColor");

    public ColorHsvaParameter colorParameter;

    protected override void OnIterationStart()
    {
        var taggedObjects = tagManager.Query<ColorRandomizerTag>();
        foreach (var taggedObject in taggedObjects)
        {
            var renderer = taggedObject.GetComponent<MeshRenderer>();
            renderer.material.SetColor(k_BaseColor, colorParameter.Sample());
        }
    }
}
```

--------------------------------

### Create ID Label Config in Unity

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This action demonstrates how to create a new 'ID Label Config' asset within the Unity project's Assets folder. This asset is used to associate labels with unique numerical IDs for object detection.

```Unity Editor
Right-click Assets folder → Create → Perception → ID Label Config
```

--------------------------------

### Configure Dataset Insights Notebook

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/DatasetInsights.md

Demonstrates how to configure the Perception_Statistics.ipynb notebook by modifying data root paths and uncommenting code blocks to input Unity Simulation run details such as execution ID, project ID, annotation definition ID, and access token.

```Python
# data_root = /data/<GUID>  # Remove <GUID> to download directly to the specified path
# Uncomment and fill in the following variables based on your Unity Simulation run:
# run_execution_id = "your_run_execution_id"
# project_id = "your_project_id"
# annotation_definition_id = ""
# access_token = "your_access_token"
```

--------------------------------

### Uniform Sampler Example

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Randomization/Samplers.md

Shows how to use the Uniform Sampler to generate values uniformly distributed within a specified range. It requires setting a FloatRange for the sampler.

```csharp
var sampler = new UniformSampler();

// generate values between the range 5 and 15
sampler.range = new FloatRange(5, 15);

// Generate a sample
var sample = sampler.Sample(); // value between 5 and 15
```

--------------------------------

### Convert Datasets to COCO Format

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/README.md

This guide demonstrates how to use pysolotools to convert a SOLO dataset into the COCO format. It provides the necessary steps for data transformation.

```Python
import pysolotools

# Example usage for COCO conversion
# solo_dataset_path = "path/to/your/solo/dataset"
# coco_output_path = "path/to/save/coco/dataset"
# pysolotools.convert_to_coco(solo_dataset_path, coco_output_path)

```

--------------------------------

### Add Perception Camera Component in Unity

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This snippet describes the process of adding the 'Perception Camera' component to a GameObject in Unity. It involves selecting the camera, navigating to the Inspector tab, clicking 'Add Component', and searching for 'Perception Camera'.

```Unity Editor
Select 'Main Camera' -> Inspector tab -> Add Component -> Search for 'Perception Camera' -> Click to add.
```

--------------------------------

### Activate Conda Environment

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/convert_to_coco.md

Activates the previously created conda environment named 'pysolotools_env'. Activating a virtual environment ensures that the installed packages are isolated to that environment.

```Shell
conda activate pysolotools_env
```

--------------------------------

### Create Semantic Segmentation Label Config in Unity

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/Phase1.md

This action shows how to create a 'Semantic Segmentation Label Config' asset in Unity. This configuration type associates labels with unique colors, which is necessary for the Semantic Segmentation Labeler.

```Unity Editor
Right-click Assets folder → Create → Perception → Semantic Segmentation Label Config
```

--------------------------------

### Normal Sampler Example

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Randomization/Samplers.md

Illustrates the use of the Normal Sampler to generate random samples from a truncated Normal distribution. It requires setting the mean, standard deviation, and a FloatRange for the distribution.

```csharp
var sampler = new NormalSampler();

// in statistics, often denoted by µ
sampler.mean = 10;
// in statistics, often denoted by σ
sampler.stdDev = 2;
// cut-off values below -20 and above 20
sampler.range = new FloatRange(-20, 20);

// Generate a sample
var sample = sampler.Sample();
```

--------------------------------

### Create Custom Randomizers in C# for Unity Perception

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/TUTORIAL.md

Learn to create custom Randomizers by extending provided samples in C#. This allows for complex randomization of simulations to generate diverse training data for computer vision models.

```csharp
using Unity.Perception.Randomization.Parameters;
using Unity.Perception.Randomization.Randomizers;

[Serializable]
public class CustomRandomizer : Randomizer {
    public FloatParameter rotation;

    protected override void OnBeginRandomization() {
        // Apply randomization logic here
        // Example: Randomize object rotation
        // transform.rotation = Quaternion.Euler(0, rotation.Sample(), 0);
    }
}
```

--------------------------------

### Configure Unity IDE for Code Navigation

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/CONTRIBUTING.md

Steps to configure Unity Editor's external tools to generate all .csproj files, enabling better code navigation within IDEs like JetBrains Rider or Visual Studio with ReSharper.

```Unity C#
// In your Unity Editor, navigate to Preferences -> External Tools
// Check 'Generate all .csproj files.'
```

--------------------------------

### Unity Perception Custom Metadata Tag Creation (C#)

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/PerceptionCamera.md

A guide on creating custom metadata tags in Unity Perception by inheriting from BaseMetadataTag and implementing required methods.

```C#
public class CustomMetadataTag : BaseMetadataTag
{
    // Implement required abstract methods
    public override string key { get; }
    public override void GetReportedValues(IMessageBuilder builder)
    {
        // Add any information required here to the builder
        builder.Add("object_name", data_to_be_added>);
    }
    // Optional: public override int instanceId { get; }
}
```

--------------------------------

### Example Pixel Position Annotation JSON

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/PerceptionCamera.md

This JSON object defines an annotation for pixel position, specifying its type, ID, sensor, description, image format, dimensions, and filename.

```json
{
  "@type": "type.unity.com/unity.solo.PixelPositionAnnotation",
  "id": "PixelPosition",
  "sensorId": "camera",
  "description": "Generates a pixelized position image where RGB channels denote the XYZ components of the vector from the camera to a pixel respectively.",
  "imageFormat": "Exr",
  "dimension": [
    1920.0,
    1080.0
  ],
  "filename": "step0.camera.PixelPosition.exr"
}
```

--------------------------------

### Activate Conda Virtual Environment

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools.md

This command activates the previously created Conda virtual environment named 'pysolotools_env'. Activating the environment ensures that subsequent commands and installations are isolated within this environment.

```Shell
conda activate pysolotools_env
```

--------------------------------

### Create PrefabCluster ScriptableObject for Reusable Prefab Lists

Source: https://github.com/unity-technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/FAQ/FAQ.md

Defines a ScriptableObject asset to hold a list of prefabs, making them reusable across different Randomizers. It uses a GameObjectParameter to manage the prefab list and provides a Sample() function.

```C#
using UnityEngine;
using UnityEngine.Perception.Randomization.Parameters;

[CreateAssetMenu(fileName="NewPrefabCluster", menuName="Test/PrefabCluster")]
public class PrefabCluster : ScriptableObject
{
    public GameObjectParameter clusterPrefabs;
}
```