Advanced Robotics: Mastering Perception, Kinematics & AI
(Advanced Course):

Home  ›  Robotic Courses  › Advanced Robotics: Mastering Perception, Kinematics & AI

TL;DR — Quick Insights

• This is where robotics gets real — rotation matrices, forward kinematics, PID control, ROS 2 Jazzy Jalisco, and YOLO object detection, all with working Python code you can run today.
• Modules continue from where the Beginner course ends (Modules 1–3). This course covers Modules 4–6: Robotics Math, Robot Perception, and Mastering ROS 2.
• Every module includes runnable Python code — from rotation matrices to YOLO object detection to ROS 2 publisher/subscriber nodes. This is a hands-on course, not slides.
• By the end you will have a working ROS 2 robot setup, understand how a robot sees the world through cameras and LiDAR, and be able to write your own perception and control software.

import numpy as np

# Original position of the robot arm in 3D space
position = np.array([2, 3, 4])   # (x, y, z)

# Rotation angle (30 degrees to the left)
theta = np.radians(30)  # convert degrees to radians

# Rotation matrix around Z-axis
rotation_matrix_z = np.array([
    [np.cos(theta), -np.sin(theta), 0],
    [np.sin(theta),  np.cos(theta), 0],
    [0,              0,             1]
])

# Apply rotation
new_position = rotation_matrix_z @ position

print("Original Position:", position)
print("New Position after 30° rotation around Z-axis:", new_position)

import numpy as np

# Example 1: Drone flying at 5 m/s toward North-East (2D vector)
velocity_2d = np.array([5/np.sqrt(2), 5/np.sqrt(2)])  # split into x and y components
print("Drone velocity vector (2D):", velocity_2d)

# Example 2: Drone moving 5 meters forward (x) and 2 meters up (z) in 3D space
movement_3d = np.array([5, 0, 2])  # [x, y, z]
print("Drone movement vector (3D):", movement_3d)

# Example 3: Adding vectors (current position + movement)
current_position = np.array([0, 0, 0])  # home position
new_position = current_position + movement_3d
print("New Position after movement:", new_position)

import numpy as np
import matplotlib.pyplot as plt

# Define start and goal positions
start = np.array([0, 0])
goal = np.array([10, 0])

# Define obstacle position
obstacle = np.array([5, 0])

# Generate a curved path around the obstacle
t = np.linspace(0, 1, 100)
# Curve using a quadratic "detour" in y-direction
path_x = (1 - t) * start[0] + t * goal[0]
path_y = (1 - t) * start[1] + t * goal[1] + 2 * np.sin(np.pi * t)

# Plot the path
plt.figure(figsize=(6, 4))
plt.plot(path_x, path_y, 'b-', label="Planned Path")
plt.plot(start[0], start[1], 'go', label="Start")
plt.plot(goal[0], goal[1], 'ro', label="Goal")
plt.plot(obstacle[0], obstacle[1], 'ks', label="Obstacle")
plt.legend()
plt.title("Robot Path Planning Example")
plt.xlabel("X")
plt.ylabel("Y")
plt.grid(True)
plt.show()

The image above shows how a vacuum robot can use computer vision to detect the line where the floor meets the wall, with green overlays marking the boundary so the robot doesn’t crash.

This kind of visualization helps beginners understand how edge detection + line detection works in practice. In real robots, the camera feed is processed frame by frame, and the detected lines guide navigation decisions (stop, turn, or follow along the wall).

import cv2
import numpy as np

# Load an image (simulating the robot's camera view)
image = cv2.imread("room.jpg")  # replace with actual camera frame

# Convert to grayscale
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Apply Gaussian blur to reduce noise
blurred = cv2.GaussianBlur(gray, (5, 5), 0)

# Use Canny edge detection to find edges (like floor-wall boundaries)
edges = cv2.Canny(blurred, threshold1=50, threshold2=150)

# Use Hough Line Transform to detect straight lines (walls/floor edges)
lines = cv2.HoughLinesP(edges, 1, np.pi/180, threshold=50, minLineLength=50, maxLineGap=10)

# Draw detected lines on the image
if lines is not None:
    for line in lines:
x1, y1, x2, y2 = line[0]
        cv2.line(image, (x1, y1), (x2, y2), (0, 255, 0), 2)
cv2.imshow("Detected Lines", image)
cv2.waitKey(0)
       

Example: Real-Time Object Detection with YOLOv8 (Ultralytics)

Here is a complete, runnable Python example showing how a security robot detects humans versus boxes in a warehouse using YOLOv8 — the industry standard for real-time robotics object detection in 2026:

Explanation — Model Loading: yolo11n.pt is YOLO11’s nano model — fast enough for real-time inference on a Jetson AGX Orin at 60+ FPS. — Confidence Threshold: Only detections above 50% confidence are acted on — reducing false positives in noisy warehouse environments. — Class Filter: The robot only reacts to person (class 0) and box (class 73 in COCO) — ignoring irrelevant detections. — Bounding Box: xyxy coordinates give the robot the exact pixel region of each detected object for downstream gripper or navigation decisions.

Why This Matters in Robotics — YOLO runs on the edge (directly on the robot’s GPU) — no cloud connection needed, no latency. — Waymo, Boston Dynamics, and Amazon Robotics all use YOLO-family models as the first stage in their perception pipelines. — The output bounding boxes feed directly into the robot’s decision layer — “Is this a human? Stop. Is this a box? Pick it up.”

# Required: pip install ultralytics opencv-python
from ultralytics import YOLO
import cv2

# Step 1: Load YOLOv11 nano model (fast, runs on Jetson AGX Orin at 60+ FPS)
model = YOLO("yolo11n.pt")  # auto-downloads on first run

# Step 2: Open camera feed (0 = USB cam, or replace with "warehouse.mp4")
cap = cv2.VideoCapture(0)

# Step 3: Define classes the robot cares about (COCO dataset labels)
TARGET_CLASSES = {
    0: "Person",   # human worker — robot must stop
    73: "Box"      # cardboard box — robot can pick up
}

print("Robot perception active. Press Q to quit.")

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    # Step 4: Run YOLO inference on the current frame
    results = model(frame, conf=0.5, verbose=False)  # 50% confidence threshold

    # Step 5: Process detections
    for result in results:
        for box in result.boxes:
            class_id = int(box.cls[0])
            confidence = float(box.conf[0])

            # Only process target classes (person or box)
            if class_id not in TARGET_CLASSES:
                continue

            # Step 6: Extract bounding box coordinates (pixel space)
            x1, y1, x2, y2 = map(int, box.xyxy[0])
            label = TARGET_CLASSES[class_id]

            # Step 7: Robot decision logic
            if class_id == 0:  # Person detected
                action = "STOP — Human in path"
                colour = (0, 0, 255)    # Red box
            else:              # Box detected
                action = "PICK UP — Box at target"
                colour = (0, 255, 0)   # Green box

            # Step 8: Draw detection on frame
            cv2.rectangle(frame, (x1, y1), (x2, y2), colour, 2)
            cv2.putText(
                frame,
                f"{label} {confidence:.0%} | {action}",
                (x1, y1 - 10),
                cv2.FONT_HERSHEY_SIMPLEX,
                0.6, colour, 2
            )

            print(f"[DETECTED] {label} | Confidence: {confidence:.1%} | "
                  f"BBox: ({x1},{y1}) → ({x2},{y2}) | Action: {action}")

    # Step 9: Display the frame
    cv2.imshow("Robot Perception — YOLO Object Detection", frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Key Skills demonstrated: YOLOv11 inference, real-time camera loop, class filtering, bounding box extraction, robot decision logic, OpenCV visualisation.

Hardware note: For deployment on a Jetson AGX Orin, replace yolo11n.pt with yolo11n.engine (TensorRT export) for 3× faster inference. See UDHY’s Expert Robotics Course for TensorRT optimisation.

Example: LiDAR Point Cloud Processing with Open3D

Here is a complete Python example showing how a robot processes a raw LiDAR point cloud — filtering noise, detecting ground plane, and isolating obstacles — the exact pipeline used in autonomous vehicle and warehouse robot perception systems:

Explanation — Point Cloud Loading: open3d reads .pcd files directly from Velodyne, Ouster, or RealSense LiDAR sensors. — Voxel Downsampling: Reduces a 100,000-point raw scan to ~5,000 representative points — making real-time processing feasible on embedded hardware. — Statistical Outlier Removal: Eliminates LiDAR noise (rain droplets, dust, sensor artefacts) that would cause phantom obstacle detections. — RANSAC Ground Segmentation: Fits a mathematical plane to the ground surface — everything above the plane is a potential obstacle. — DBSCAN Clustering: Groups the remaining points into individual objects — each cluster is one obstacle (pedestrian, vehicle, box).

Why This Matters in Robotics — This exact 5-step pipeline (downsample → denoise → segment ground → cluster obstacles) runs inside Waymo’s, Aurora’s, and Boston Dynamics’ perception stacks. — Without ground removal, the robot would treat the floor itself as an obstacle and never move. — Each DBSCAN cluster output feeds directly into the object tracking layer — assigning IDs and predicting trajectories for dynamic objects.

# Required: pip install open3d numpy
import open3d as o3d
import numpy as np

# ─────────────────────────────────────────────
# STEP 1: Load raw LiDAR point cloud
# Replace "scan.pcd" with your Velodyne/Ouster/RealSense output file
# ─────────────────────────────────────────────
pcd = o3d.io.read_point_cloud("scan.pcd")
print(f"[LOADED] Raw point cloud: {len(pcd.points):,} points")

# ─────────────────────────────────────────────
# STEP 2: Voxel Downsampling
# Reduce density for real-time processing on embedded hardware (Jetson, etc.)
# voxel_size = 0.05m means one representative point per 5cm cube
# ─────────────────────────────────────────────
pcd_down = pcd.voxel_down_sample(voxel_size=0.05)
print(f"[DOWNSAMPLED] Reduced to: {len(pcd_down.points):,} points")

# ─────────────────────────────────────────────
# STEP 3: Statistical Outlier Removal
# Removes noise — rain, dust, sensor artefacts
# nb_neighbors=20: checks 20 nearest neighbours
# std_ratio=2.0:   removes points > 2 std deviations from mean distance
# ─────────────────────────────────────────────
pcd_clean, inlier_idx = pcd_down.remove_statistical_outlier(
    nb_neighbors=20,
    std_ratio=2.0
)
print(f"[DENOISED]    Clean cloud: {len(pcd_clean.points):,} points")

# ─────────────────────────────────────────────
# STEP 4: Ground Plane Segmentation using RANSAC
# Separates drivable floor from obstacles
# distance_threshold=0.02m: points within 2cm of plane = ground
# ransac_n=3:               minimum points to fit a plane
# num_iterations=1000:      higher = more accurate ground fit
# ─────────────────────────────────────────────
plane_model, ground_idx = pcd_clean.segment_plane(
    distance_threshold=0.02,
    ransac_n=3,
    num_iterations=1000
)
[a, b, c, d] = plane_model
print(f"[GROUND]      Plane equation: {a:.2f}x + {b:.2f}y + {c:.2f}z + {d:.2f} = 0")

# Separate ground and obstacle points
ground_cloud = pcd_clean.select_by_index(ground_idx)
obstacle_cloud = pcd_clean.select_by_index(ground_idx, invert=True)
print(f"[SEGMENTED]   Ground: {len(ground_cloud.points):,} pts | "
      f"Obstacles: {len(obstacle_cloud.points):,} pts")

# ─────────────────────────────────────────────
# STEP 5: DBSCAN Clustering — identify individual obstacles
# Each cluster = one distinct object (pedestrian, vehicle, box, etc.)
# eps=0.3m:         points within 30cm belong to same cluster
# min_points=10:    minimum 10 points to form a valid object
# ─────────────────────────────────────────────
labels = np.array(
    obstacle_cloud.cluster_dbscan(eps=0.3, min_points=10, print_progress=False)
)

num_clusters = labels.max() + 1  # -1 labels = noise (ignored)
print(f"[CLUSTERS]    {num_clusters} obstacle(s) detected")

# ─────────────────────────────────────────────
# STEP 6: Report each detected obstacle
# In a real robot, each cluster feeds into the object tracker
# ─────────────────────────────────────────────
obstacle_points = np.asarray(obstacle_cloud.points)

for cluster_id in range(num_clusters):
    cluster_pts = obstacle_points[labels == cluster_id]
    centroid = cluster_pts.mean(axis=0)
    bbox_size = cluster_pts.max(axis=0) - cluster_pts.min(axis=0)

    print(f"  Obstacle {cluster_id + 1}: "
          f"Centre=({centroid[0]:.2f}m, {centroid[1]:.2f}m, {centroid[2]:.2f}m) | "
          f"Size={bbox_size[0]:.2f}m × {bbox_size[1]:.2f}m × {bbox_size[2]:.2f}m | "
          f"Points={len(cluster_pts)}")

# ─────────────────────────────────────────────
# STEP 7: Visualise — colour ground blue, obstacles red, noise grey
# ─────────────────────────────────────────────
colours = np.zeros((len(obstacle_cloud.points), 3))  # default grey

for cluster_id in range(num_clusters):
    # Assign a unique colour per cluster
    np.random.seed(cluster_id)
    colour = np.random.rand(3)
    colours[labels == cluster_id] = colour

obstacle_cloud.colors = o3d.utility.Vector3dVector(colours)
ground_cloud.paint_uniform_color([0.2, 0.4, 1.0])  # Blue = ground

# Display combined scene
o3d.visualization.draw_geometries(
    [ground_cloud, obstacle_cloud],
    window_name="LiDAR Perception — Ground + Obstacles",
    width=1280, height=720
)

Key Skills demonstrated: Open3D point cloud I/O, voxel downsampling, statistical denoising, RANSAC plane fitting, DBSCAN clustering, 3D visualisation.

Real-world note: In production AV systems, this pipeline runs at 10–20 Hz (once per LiDAR scan rotation). On a Jetson AGX Orin with CUDA acceleration, the full pipeline above completes in under 30ms per frame. For Velodyne HDL-64E data (128,000 points/scan), reduce voxel_size to 0.10 to maintain real-time performance. See the UDHY LiDAR guide for sensor selection.

💡 Running the AI track in parallel?

This course pairs perfectly with Deep Learning for Robotics — Module 5 here (Perception) directly complements that course’s CNN and YOLO content. Running both simultaneously gives you the complete picture: the AI side and the robotics engineering side of the same perception pipeline.

Read alongside this course:

• Sensor Fusion Explained: Cameras, LiDAR & Radar — the multi-sensor stack this course implements
• Why Self-Driving Cars Still Fail — the real-world perception challenges your YOLO model must overcome
• The Complete Guide to AV Teleoperation — human-in-the-loop control that pairs with your ROS 2 learning
• How to Choose the Right LiDAR — hardware guide for the point cloud work in Module 5

Designed by Dr. Dilip Kumar Limbu — Former Principal Research Scientist, A*STAR · Co-Founder, Moovita, Singapore’s first autonomous vehicle company · 30 years building real-world autonomous systems. UDHY.com.