Chapter 4: Simulating Sensors - LiDAR, Depth Cameras, and IMUs

Introduction

Autonomous robots rely on sensors to perceive their environment. LiDAR provides 3D point clouds for mapping and obstacle detection. Depth cameras capture RGBD data for object recognition. IMUs measure acceleration and rotation for odometry and stabilization.

In this chapter, you'll learn how to simulate these essential sensors in Gazebo, configure their parameters to match real hardware, integrate them with ROS 2 topics, and visualize sensor data in RViz.

Why this matters: Accurate sensor simulation enables you to develop and test perception algorithms (SLAM, object detection, navigation) entirely in simulation before deploying to expensive hardware. Poor sensor configuration creates a reality gap that breaks algorithms when transferred to real robots.

Sensor Simulation Overview

Gazebo simulates sensors using plugins - software modules that generate sensor data based on the virtual world. Each sensor type has specific parameters (range, resolution, noise models) that should match your target hardware.

Sensor Categories

Sensor Type	Data Output	Common Use Cases	ROS Message Type
LiDAR	3D point clouds	SLAM, obstacle detection, localization	sensor_msgs/PointCloud2, sensor_msgs/LaserScan
Depth Camera	RGBD images	Object recognition, grasping, dense mapping	sensor_msgs/Image, sensor_msgs/CameraInfo
IMU	Acceleration, angular velocity	Odometry, stabilization, orientation estimation	sensor_msgs/Imu
RGB Camera	Color images	Visual servoing, QR codes, monitoring	sensor_msgs/Image
GPS	Position coordinates	Outdoor navigation	sensor_msgs/NavSatFix

This chapter focuses on LiDAR, depth cameras, and IMUs - the three most common sensors for indoor autonomous robots.

Simulating LiDAR

LiDAR (Light Detection and Ranging) uses laser rays to measure distances, creating 3D point clouds of the environment.

How Gazebo Simulates LiDAR

Gazebo's LiDAR plugin uses ray-casting: it shoots virtual rays in a pattern (horizontal and vertical angles), calculates intersections with objects, and measures distances. This mimics how real LiDAR lasers work.

Adding LiDAR to a Robot (URDF)

Add a LiDAR sensor to your robot's URDF file:

<!-- LiDAR sensor link -->
<link name="lidar_link">
  <visual>
    <geometry>
      <cylinder radius="0.05" length="0.04"/> <!-- Small cylinder for sensor housing -->
    </geometry>
    <material name="black">
      <color rgba="0.1 0.1 0.1 1"/>
    </material>
  </visual>

  <collision>
    <geometry>
      <cylinder radius="0.05" length="0.04"/>
    </geometry>
  </collision>

  <inertial>
    <mass value="0.125"/>
    <inertia ixx="0.001" ixy="0.0" ixz="0.0" iyy="0.001" iyz="0.0" izz="0.001"/>
  </inertial>
</link>

<!-- Joint connecting LiDAR to robot base -->
<joint name="lidar_joint" type="fixed">
  <parent link="base_link"/>
  <child link="lidar_link"/>
  <origin xyz="0.15 0 0.15" rpy="0 0 0"/> <!-- Position: front of robot, 15cm up -->
</joint>

<!-- Gazebo plugin for LiDAR -->
<gazebo reference="lidar_link">
  <sensor type="ray" name="lidar_sensor">
    <pose>0 0 0 0 0 0</pose>
    <visualize>true</visualize> <!-- Show rays in Gazebo GUI -->
    <update_rate>10</update_rate> <!-- 10 Hz -->

    <ray>
      <scan>
        <!-- Horizontal scan -->
        <horizontal>
          <samples>720</samples> <!-- Number of rays per scan -->
          <resolution>1</resolution>
          <min_angle>-3.14159</min_angle> <!-- -180° -->
          <max_angle>3.14159</max_angle>  <!-- +180° (360° coverage) -->
        </horizontal>

        <!-- Vertical scan (for 3D LiDAR) -->
        <vertical>
          <samples>16</samples> <!-- 16 vertical layers -->
          <resolution>1</resolution>
          <min_angle>-0.2618</min_angle> <!-- -15° -->
          <max_angle>0.2618</max_angle>   <!-- +15° -->
        </vertical>
      </scan>

      <range>
        <min>0.1</min> <!-- Minimum range: 10cm -->
        <max>30.0</max> <!-- Maximum range: 30m -->
        <resolution>0.01</resolution> <!-- 1cm resolution -->
      </range>

      <!-- Noise model (realistic sensor errors) -->
      <noise>
        <type>gaussian</type>
        <mean>0.0</mean>
        <stddev>0.01</stddev> <!-- 1cm standard deviation -->
      </noise>
    </ray>

    <!-- ROS 2 plugin -->
    <plugin name="gazebo_ros_lidar" filename="libgazebo_ros_ray_sensor.so">
      <ros>
        <namespace>/robot</namespace>
        <remapping>~/out:=lidar/points</remapping> <!-- Publish to /robot/lidar/points -->
      </ros>
      <output_type>sensor_msgs/PointCloud2</output_type>
      <frame_name>lidar_link</frame_name>
    </plugin>
  </sensor>
</gazebo>

LiDAR Parameters Explained

Parameter	Description	Typical Values	Impact
samples (horizontal)	Rays per horizontal scan	360-1080	Higher = denser point cloud, slower
samples (vertical)	Vertical layers (channels)	1 (2D), 16-64 (3D)	More layers = 3D coverage
min_angle, max_angle	Field of view	-π to π (360°)	Wider = more coverage, more data
min, max (range)	Detection range	0.1m - 100m	Depends on real sensor specs
resolution	Distance measurement precision	0.01m (1cm)	Real sensors: 1-3cm
update_rate	Scans per second (Hz)	5-20 Hz	Higher = more CPU, smoother data
noise.stddev	Measurement error	0.01-0.05m	Matches real sensor noise

Configuring for Different LiDAR Types

2D LiDAR (e.g., SICK TiM, Hokuyo URG):

<vertical>
  <samples>1</samples> <!-- Single horizontal plane -->
  <min_angle>0</min_angle>
  <max_angle>0</max_angle>
</vertical>

3D LiDAR (e.g., Velodyne VLP-16):

<vertical>
  <samples>16</samples> <!-- 16 channels -->
  <min_angle>-0.2618</min_angle> <!-- -15° -->
  <max_angle>0.2618</max_angle>  <!-- +15° -->
</vertical>

Viewing LiDAR Data in RViz

# Source your workspace
source /opt/ros/humble/setup.bash

# Launch RViz
ros2 run rviz2 rviz2

In RViz:

1. Click "Add" button
2. Select "PointCloud2"
3. Set "Topic": /robot/lidar/points
4. Set "Fixed Frame": lidar_link or base_link
5. Adjust "Size": 0.05 for visibility
6. Adjust "Color Transformer": Intensity, X, Y, or Z

You should see a 3D point cloud visualization of the Gazebo environment.

Simulating Depth Cameras

Depth cameras (RGBD cameras) provide color images + depth maps. Popular models include Intel RealSense, Kinect, and Orbbec Astra.

Adding Depth Camera to a Robot

<!-- Depth camera link -->
<link name="camera_link">
  <visual>
    <geometry>
      <box size="0.03 0.1 0.03"/> <!-- Camera housing -->
    </geometry>
    <material name="dark_gray">
      <color rgba="0.3 0.3 0.3 1"/>
    </material>
  </visual>

  <collision>
    <geometry>
      <box size="0.03 0.1 0.03"/>
    </geometry>
  </collision>

  <inertial>
    <mass value="0.1"/>
    <inertia ixx="0.001" ixy="0.0" ixz="0.0" iyy="0.001" iyz="0.0" izz="0.001"/>
  </inertial>
</link>

<!-- Fixed joint to robot -->
<joint name="camera_joint" type="fixed">
  <parent link="base_link"/>
  <child link="camera_link"/>
  <origin xyz="0.2 0 0.1" rpy="0 0 0"/> <!-- Front of robot -->
</joint>

<!-- Gazebo depth camera plugin -->
<gazebo reference="camera_link">
  <sensor type="depth" name="depth_camera">
    <update_rate>30.0</update_rate> <!-- 30 FPS -->
    <camera>
      <!-- Camera intrinsics -->
      <horizontal_fov>1.047</horizontal_fov> <!-- 60° field of view -->
      <image>
        <width>640</width>
        <height>480</height>
        <format>R8G8B8</format> <!-- RGB -->
      </image>
      <clip>
        <near>0.02</near> <!-- Minimum depth: 2cm -->
        <far>10.0</far>   <!-- Maximum depth: 10m -->
      </clip>

      <!-- Noise (realistic sensor imperfections) -->
      <noise>
        <type>gaussian</type>
        <mean>0.0</mean>
        <stddev>0.007</stddev> <!-- ~7mm depth noise -->
      </noise>
    </camera>

    <!-- ROS 2 plugin -->
    <plugin name="depth_camera_controller" filename="libgazebo_ros_camera.so">
      <ros>
        <namespace>/robot</namespace>
        <remapping>image_raw:=camera/color/image_raw</remapping>
        <remapping>depth/image_raw:=camera/depth/image_raw</remapping>
        <remapping>camera_info:=camera/color/camera_info</remapping>
        <remapping>depth/camera_info:=camera/depth/camera_info</remapping>
        <remapping>points:=camera/depth/points</remapping>
      </ros>
      <camera_name>depth_camera</camera_name>
      <frame_name>camera_link</frame_name>
      <hack_baseline>0.07</hack_baseline> <!-- Stereo baseline for point cloud -->
      <min_depth>0.02</min_depth>
      <max_depth>10.0</max_depth>
    </plugin>
  </sensor>
</gazebo>

Published Topics

The depth camera publishes multiple topics:

Topic	Message Type	Description
/robot/camera/color/image_raw	sensor_msgs/Image	RGB color image
/robot/camera/depth/image_raw	sensor_msgs/Image	Depth image (16-bit)
/robot/camera/color/camera_info	sensor_msgs/CameraInfo	Camera calibration
/robot/camera/depth/camera_info	sensor_msgs/CameraInfo	Depth camera calibration
/robot/camera/depth/points	sensor_msgs/PointCloud2	Organized 3D point cloud

Viewing Depth Camera Data

View RGB image:

ros2 run rqt_image_view rqt_image_view /robot/camera/color/image_raw

View depth image:

ros2 run rqt_image_view rqt_image_view /robot/camera/depth/image_raw

View point cloud in RViz:

1. Launch RViz: ros2 run rviz2 rviz2
2. Add PointCloud2 display
3. Set topic: /robot/camera/depth/points
4. Set color transformer: RGB8

Depth Camera Parameters

Parameter	Description	Typical Values
horizontal_fov	Field of view (radians)	0.87 (50°) - 1.57 (90°)
width, height	Image resolution	640x480, 1280x720
near, far	Depth range	0.02m - 10m
update_rate	Frames per second	15-30 FPS
stddev (noise)	Depth measurement error	0.005-0.01m (5-10mm)

Simulating IMU (Inertial Measurement Unit)

IMUs measure linear acceleration and angular velocity, essential for odometry and stabilization.

Adding IMU to a Robot

<!-- IMU link (usually inside robot base) -->
<link name="imu_link">
  <!-- IMU has no visual/collision, it's just a reference frame -->
  <inertial>
    <mass value="0.01"/>
    <inertia ixx="0.00001" ixy="0.0" ixz="0.0" iyy="0.00001" iyz="0.0" izz="0.00001"/>
  </inertial>
</link>

<!-- Fixed joint to robot center -->
<joint name="imu_joint" type="fixed">
  <parent link="base_link"/>
  <child link="imu_link"/>
  <origin xyz="0 0 0.05" rpy="0 0 0"/> <!-- Center of robot mass -->
</joint>

<!-- Gazebo IMU plugin -->
<gazebo reference="imu_link">
  <gravity>true</gravity>
  <sensor name="imu_sensor" type="imu">
    <always_on>true</always_on>
    <update_rate>100</update_rate> <!-- 100 Hz (typical IMU rate) -->

    <visualize>true</visualize>

    <!-- IMU plugin -->
    <plugin filename="libgazebo_ros_imu_sensor.so" name="imu_plugin">
      <ros>
        <namespace>/robot</namespace>
        <remapping>~/out:=imu/data</remapping> <!-- Publish to /robot/imu/data -->
      </ros>
      <frame_name>imu_link</frame_name>

      <!-- Initial orientation -->
      <initial_orientation_as_reference>false</initial_orientation_as_reference>

      <!-- Noise parameters (realistic IMU errors) -->
      <!-- Angular velocity (gyroscope) -->
      <angular_velocity>
        <x>
          <noise type="gaussian">
            <mean>0.0</mean>
            <stddev>2e-4</stddev> <!-- rad/s -->
            <bias_mean>0.0000075</bias_mean>
            <bias_stddev>0.0000008</bias_stddev>
          </noise>
        </x>
        <y>
          <noise type="gaussian">
            <mean>0.0</mean>
            <stddev>2e-4</stddev>
            <bias_mean>0.0000075</bias_mean>
            <bias_stddev>0.0000008</bias_stddev>
          </noise>
        </y>
        <z>
          <noise type="gaussian">
            <mean>0.0</mean>
            <stddev>2e-4</stddev>
            <bias_mean>0.0000075</bias_mean>
            <bias_stddev>0.0000008</bias_stddev>
          </noise>
        </z>
      </angular_velocity>

      <!-- Linear acceleration (accelerometer) -->
      <linear_acceleration>
        <x>
          <noise type="gaussian">
            <mean>0.0</mean>
            <stddev>1.7e-2</stddev> <!-- m/s² -->
            <bias_mean>0.1</bias_mean>
            <bias_stddev>0.001</bias_stddev>
          </noise>
        </x>
        <y>
          <noise type="gaussian">
            <mean>0.0</mean>
            <stddev>1.7e-2</stddev>
            <bias_mean>0.1</bias_mean>
            <bias_stddev>0.001</bias_stddev>
          </noise>
        </y>
        <z>
          <noise type="gaussian">
            <mean>0.0</mean>
            <stddev>1.7e-2</stddev>
            <bias_mean>0.1</bias_mean>
            <bias_stddev>0.001</bias_stddev>
          </noise>
        </z>
      </linear_acceleration>
    </plugin>
  </sensor>
</gazebo>

IMU Data Structure

The sensor_msgs/Imu message contains:

# IMU message fields
header:
  stamp: Time        # Timestamp
  frame_id: string   # Reference frame (imu_link)

orientation: Quaternion  # Orientation (x, y, z, w)
angular_velocity: Vector3  # Gyroscope (rad/s)
linear_acceleration: Vector3  # Accelerometer (m/s²)

# Covariance matrices (3x3, flattened to 9 elements)
orientation_covariance: float[9]
angular_velocity_covariance: float[9]
linear_acceleration_covariance: float[9]

Viewing IMU Data

Echo IMU topic:

ros2 topic echo /robot/imu/data

Plot angular velocity:

ros2 run rqt_plot rqt_plot /robot/imu/data/angular_velocity/z

Visualize orientation in RViz:

1. Add "Imu" display
2. Set topic: /robot/imu/data
3. You'll see arrows representing acceleration and orientation

IMU Noise Parameters

Real IMUs have noise and bias. Match these to your target hardware:

IMU Model	Gyro Noise (rad/s)	Accel Noise (m/s²)	Gyro Bias (rad/s)	Accel Bias (m/s²)
Consumer (smartphone)	0.001	0.05	0.01	0.5
Mid-grade (VN-100)	0.0002	0.017	0.00001	0.1
High-end (tactical)	0.00005	0.005	0.000001	0.01

Integrating Sensors for SLAM

Sensors work together for Simultaneous Localization and Mapping (SLAM):

Example: LiDAR SLAM Setup

# Terminal 1: Launch Gazebo with robot
gazebo robot_world.world

# Terminal 2: Launch SLAM toolbox
ros2 launch slam_toolbox online_async_launch.py

# Terminal 3: Launch RViz
ros2 run rviz2 rviz2 -d slam_config.rviz

SLAM requires:

• LiDAR (/scan or /lidar/points)
• Odometry (/odom) - from wheel encoders or IMU
• TF transforms - robot pose in world

Example: Visual SLAM with Depth Camera

# Launch RTAB-Map (visual SLAM)
ros2 launch rtabmap_ros rtabmap.launch.py \
  rgb_topic:=/robot/camera/color/image_raw \
  depth_topic:=/robot/camera/depth/image_raw \
  camera_info_topic:=/robot/camera/color/camera_info \
  frame_id:=base_link \
  approx_sync:=true

Sensor Fusion: Combining IMU and Odometry

Use robot_localization to fuse IMU and wheel odometry:

# Install robot_localization
sudo apt install ros-humble-robot-localization

# Create config file: ekf.yaml

# ekf.yaml - Extended Kalman Filter configuration
ekf_filter_node:
  ros__parameters:
    frequency: 30.0
    sensor_timeout: 0.1
    two_d_mode: true  # For ground robots

    # Odometry source (wheel encoders)
    odom0: /odom
    odom0_config: [false, false, false,  # x, y, z (position)
                   false, false, false,  # roll, pitch, yaw (orientation)
                   true,  true,  false,  # vx, vy, vz (linear velocity)
                   false, false, true,   # vroll, vpitch, vyaw (angular velocity)
                   false, false, false]  # ax, ay, az (linear acceleration)

    # IMU source
    imu0: /robot/imu/data
    imu0_config: [false, false, false,  # Position
                  true,  true,  true,   # Orientation (roll, pitch, yaw)
                  false, false, false,  # Linear velocity
                  true,  true,  true,   # Angular velocity
                  true,  true,  true]   # Linear acceleration

Launch sensor fusion:

ros2 run robot_localization ekf_node --ros-args --params-file ekf.yaml

Result: Fused odometry published to /odometry/filtered with improved accuracy.

Practical Example: Complete Sensor Suite

Putting it all together - a robot with LiDAR, depth camera, and IMU:

<?xml version="1.0"?>
<robot name="sensor_robot" xmlns:xacro="http://www.ros.org/wiki/xacro">

  <!-- Base link -->
  <link name="base_link">
    <visual>
      <geometry><box size="0.6 0.4 0.2"/></geometry>
    </visual>
    <collision>
      <geometry><box size="0.6 0.4 0.2"/></geometry>
    </collision>
    <inertial>
      <mass value="10.0"/>
      <inertia ixx="0.183" iyy="0.333" izz="0.45"/>
    </inertial>
  </link>

  <!-- LiDAR (from earlier example) -->
  <link name="lidar_link">
    <visual>
      <geometry><cylinder radius="0.05" length="0.04"/></geometry>
    </visual>
    <inertial>
      <mass value="0.125"/>
      <inertia ixx="0.001" iyy="0.001" izz="0.001"/>
    </inertial>
  </link>
  <joint name="lidar_joint" type="fixed">
    <parent link="base_link"/>
    <child link="lidar_link"/>
    <origin xyz="0.15 0 0.15" rpy="0 0 0"/>
  </joint>
  <!-- LiDAR Gazebo sensor (see earlier example) -->

  <!-- Depth Camera -->
  <link name="camera_link">
    <visual>
      <geometry><box size="0.03 0.1 0.03"/></geometry>
    </visual>
    <inertial>
      <mass value="0.1"/>
      <inertia ixx="0.001" iyy="0.001" izz="0.001"/>
    </inertial>
  </link>
  <joint name="camera_joint" type="fixed">
    <parent link="base_link"/>
    <child link="camera_link"/>
    <origin xyz="0.2 0 0.05" rpy="0 0 0"/>
  </joint>
  <!-- Depth camera Gazebo sensor (see earlier example) -->

  <!-- IMU -->
  <link name="imu_link">
    <inertial>
      <mass value="0.01"/>
      <inertia ixx="0.00001" iyy="0.00001" izz="0.00001"/>
    </inertial>
  </link>
  <joint name="imu_joint" type="fixed">
    <parent link="base_link"/>
    <child link="imu_link"/>
    <origin xyz="0 0 0.05" rpy="0 0 0"/>
  </joint>
  <!-- IMU Gazebo sensor (see earlier example) -->

</robot>

Launching the Complete System

# Terminal 1: Gazebo
gazebo sensor_robot_world.world

# Terminal 2: RViz with all sensors
ros2 run rviz2 rviz2

# Add displays in RViz:
# - PointCloud2: /robot/lidar/points
# - PointCloud2: /robot/camera/depth/points
# - Image: /robot/camera/color/image_raw
# - Imu: /robot/imu/data

# Terminal 3: Echo IMU data
ros2 topic echo /robot/imu/data

# Terminal 4: Monitor topics
ros2 topic list

Performance Considerations

Sensor simulation is computationally expensive. Optimize for performance:

1. Reduce LiDAR density: Use fewer samples (360 instead of 720) for faster simulation
2. Lower update rates: 10 Hz for LiDAR, 15 Hz for cameras is often sufficient
3. Disable visualization: Set <visualize>false</visualize> in sensors when not debugging
4. Use 2D LiDAR: If 3D isn't necessary, use single-layer LiDAR (much faster)
5. Optimize depth camera resolution: 320x240 instead of 640x480 reduces load significantly

Summary

In this chapter, you learned:

• Gazebo simulates sensors using plugins that generate realistic data from the virtual world
• LiDAR uses ray-casting to create 3D point clouds with configurable range, resolution, and field of view
• Depth cameras provide RGB images, depth maps, and organized point clouds with camera intrinsics
• IMUs measure linear acceleration and angular velocity with realistic noise and bias models
• Sensor noise parameters should match real hardware specifications to minimize the reality gap
• RViz is the primary tool for visualizing sensor data (point clouds, images, IMU vectors)
• Sensor fusion (e.g., robot_localization) combines IMU and odometry for improved state estimation
• Performance optimization includes reducing sensor update rates, resolution, and sample counts

Module 2 Complete! You now have the skills to create comprehensive digital twins with physics simulation (Gazebo), high-fidelity rendering (Unity), and realistic sensor models (LiDAR, depth cameras, IMUs). These tools enable you to develop and test robotic systems entirely in simulation before deploying to real hardware.

---

Further Reading:

• Gazebo Sensors Tutorial
• ROS 2 Sensor Messages
• robot_localization Documentation
• SLAM Toolbox