Skip to content

CV Simulation Setup

This document is still work in progress...

This document runs through the setup of Unity simulation in case you do not have a camera but wish to test the CV pipeline. The simulation uses Unity and ROS2 for Unity extension to publish images from virtual camera.

Warning

The Unity assets here are using HDRP (high definition render pipeline), which OpenGL does not support under Linux. Thus, HDRP will be running using Vulkan, and Intel integrated graphics have problems running it. Please use AMD or NVDIA gpus.

Download Required Software

  1. Download Unity Hub for Linux following the official docs here
  2. Create an Unity account (Unity Hub requires it) and setup your student liscence (use your GT email)
  3. In Unity Hub, download Unity Editor 2021.3.11f1 (through my testing, different versions do not seem to matter, so you can try other versions)
  4. Download the simulator project from Github repo. Remember to download as a ZIP and not clone the repo. This will pull all assets on LFS (Git Large File Storage). The compressed project should roughly be 30MB.

Screenshot

  1. Download the ROS2 for Unity extension from Github repo. Use version 1.2.0 Humble Ubuntu 22.04.

Screenshot

  1. Extract the simulation project and ROS2 for Unity zips, and place the Ros2ForUnity folder in the Assets folder of the project.

Screenshot

  1. If you do not have the CV pipeline project cloned already, go to this repo and clone it.

Project Setup

  1. Open a terminal and source the ROS2 environment. You know you have the environment if it displays humble when using the command echo $ROS_DISTRO.
  2. Open Unity Hub by typing unityhub in that terminal, and open the simulation project.
  3. The asset renderings will be incorrect (looking white and without textures), so go to Window > Package Manager, and in the Packages: field, select Unity Registry from the menu.
  4. Search for High Definition RP, and install it.
  5. Since OpenGL does not support HDRP, go to Edit > Project Settings > Player, and under Other Settings uncheck Auto Graphics API and drag Vulkan to the top priority.
  6. To configure all assets to HDRP, go to Window > Rendering > HD Render Pipeline Wizard. In the Configuration Checking section, go to the HDRP tab and click Fix All.
  7. Now all assets should be rendered properly. You may refer to this official doc for more details.

Running the Project

  1. After opening Unity Editor, run the project by clicking the play icon at the center top. An enemy robot should appear on screen. You may use your keyboard and mouse to control the movement of your camera/robot.
  2. In the CV pipeline project, you can comment out the camera node from vision.launch.py. The simulator will create a ROS2UnityCamNode and publish the same topics image_raw and camera_info for downstream nodes to use.

Common Issues

  1. When trying to run the simulation project, console shows UnsatisfiedLinkError: librcl.so.

Solution: Make sure ROS2 environment is sourced when opening Unity. You may also try opening the Unity Editor directly through this path ~/Unity/Hub/Editor/<unity version>/Editor/Unity instead of opening Unity Hub first.