This laboratory introduces students to autonomous driving concepts through hands-on programming of the JetBot robot platform. Building upon the hardware familiarization from Week 1 and the AI/ML concepts from previous weeks, you will now implement practical autonomous navigation capabilities. You'll start with basic motion control to understand how motor commands translate to robot movement, then progress to implementing a collision avoidance system using deep learning. Through data collection, model training, and real-time deployment, you'll experience the complete workflow of developing an autonomous system—from gathering training data to deploying a neural network that enables your JetBot to navigate safely without human intervention.
What You'll Learn
- Basic Robot Motion Control: Program differential drive robots using motor commands to achieve forward, backward, turning, and circular motion patterns.
- Real-Time Data Collection: Capture and label image data from the robot's camera to create training datasets for collision avoidance.
- End-to-End Deep Learning: Train convolutional neural networks to predict safe navigation decisions directly from raw camera images.
- Model Deployment on Edge Devices: Deploy trained models on the Jetson Orin Nano for real-time inference and autonomous decision-making.
- Autonomous Navigation Systems: Implement collision avoidance algorithms that enable the robot to navigate complex environments safely.
- Performance Optimization: Use TensorRT to optimize neural network inference for real-time performance on embedded hardware.
💡 Why This Matters
Autonomous navigation is a fundamental capability for modern robotics applications, from warehouse automation and delivery robots to self-driving vehicles and service robots. The collision avoidance system you'll develop in this lab demonstrates the core principles used in real-world autonomous systems: using deep learning to process sensor data, make real-time decisions, and control actuators. By implementing the complete pipeline—from data collection through model training to deployment—you gain practical experience with the engineering challenges of autonomous systems. These skills are directly applicable to autonomous vehicles, drones, industrial robots, and any system requiring intelligent navigation in dynamic environments. Understanding how to optimize neural networks for edge deployment is particularly valuable as the industry moves toward decentralized, privacy-preserving autonomous systems.
Lab Structure
This laboratory consists of four progressive parts, each building upon previous concepts:
- Part 1: Basic Motion - Understanding differential drive control and programming fundamental movement patterns
- Part 2: Data Collection - Capturing and labeling training images for collision avoidance scenarios
- Part 3: Model Training - Training convolutional neural networks on collected data with visualization
- Part 4: Live Deployment - Implementing real-time collision avoidance with the trained model
By the end of this laboratory session, you will be able to:
-
Control differential drive robots by programming motor speeds and directions to achieve precise movement patterns including forward, backward, rotation, and arc trajectories.
-
Collect and label training data for supervised learning by capturing camera images in various scenarios and categorizing them as "free" or "blocked" for collision avoidance.
-
Train deep learning models for autonomous navigation using PyTorch, implementing data augmentation, loss functions, and training loops for image classification tasks.
-
Deploy neural networks on edge devices for real-time inference, understanding the challenges of latency, power consumption, and computational constraints.
-
Implement collision avoidance systems that process camera images in real-time to make autonomous navigation decisions and control robot actuators.
-
Optimize model performance using TensorRT to accelerate neural network inference through graph optimization, layer fusion, and precision calibration for embedded deployment.
-
Evaluate autonomous system performance by testing the robot in various environments and understanding the relationship between training data quality and system behavior.
Differential Drive Robots
The JetBot uses a differential drive configuration, where two independently controlled wheels provide both propulsion and steering. By varying the speed and direction of each wheel, the robot can achieve various motion patterns. When both wheels move forward at the same speed, the robot moves straight. When one wheel moves faster than the other, the robot turns. When wheels move in opposite directions, the robot rotates in place. This simple yet effective design is widely used in mobile robotics due to its simplicity, reliability, and precise control capabilities.
Collision Avoidance with Deep Learning
Traditional collision avoidance systems rely on explicit sensor data processing and rule-based decision making. However, deep learning enables an end-to-end approach where the robot learns to navigate by example. By collecting images labeled as "free" (safe to move forward) or "blocked" (obstacle present), we can train a convolutional neural network to classify new images in real-time. The network learns to recognize patterns associated with safe and unsafe navigation scenarios, making it robust to various environments and lighting conditions without requiring explicit object detection or distance measurement algorithms.
Real-Time Inference on Edge Devices
Deploying neural networks on edge devices like the Jetson Orin Nano presents unique challenges. Unlike cloud-based inference, edge deployment requires models to run with limited computational resources, minimal latency, and constrained power budgets. The Jetson platform's GPU acceleration enables real-time deep learning inference, but optimization techniques like TensorRT are often necessary to achieve the required frame rates for smooth autonomous navigation. TensorRT optimizes neural networks through techniques like layer fusion, precision calibration (FP16/INT8), and kernel optimization, significantly improving inference speed without sacrificing accuracy.
Key Concepts
- Differential Drive: Two-wheeled robot configuration where independent motor control enables both translation and rotation
- PWM (Pulse Width Modulation): Technique for controlling motor speed by varying the duty cycle of power signals
- End-to-End Learning: Training neural networks to map directly from sensor inputs to control outputs
- Binary Classification: Machine learning task of categorizing inputs into two classes (free vs. blocked)
- Data Augmentation: Techniques like rotation, flipping, and color jittering to artificially expand training datasets
- TensorRT: NVIDIA's inference optimization library for maximizing deep learning performance on their hardware
- Real-Time Systems: Computing systems that must respond to inputs within strict time constraints
- Embedded AI: Deploying machine learning models on resource-constrained devices for on-device inference
💡 Did You Know?
The JetBot collision avoidance approach is similar to NVIDIA's work on autonomous vehicles, where end-to-end deep learning was first demonstrated to successfully navigate complex driving scenarios. The "PilotNet" architecture used in NVIDIA's self-driving car research showed that neural networks could learn to drive by watching human drivers, without explicitly programming rules for every possible scenario. Your JetBot experiments follow this same paradigm at a smaller scale, demonstrating the power of learning-based approaches to autonomous navigation.
Required Knowledge
Before beginning this laboratory, ensure you understand the following concepts:
- Basic Python programming and control flow (loops, conditionals, functions)
- PyTorch tensor operations and neural network training (from Weeks 3-4)
- Convolutional neural networks and image classification (from Week 4)
- Camera interfacing and image capture on Jetson (from Week 7)
- Classification tasks and model evaluation (from Week 8)
📝 Pre-lab Quiz
Test your understanding of autonomous driving and collision avoidance concepts before starting the lab exercises. Click on your chosen answer to check if it's correct.
1. In a differential drive robot like JetBot, what happens when both wheels rotate forward at the same speed?
The robot moves straight forward
The robot turns in a circle
The robot rotates in place
The robot moves backward
2. What is the primary advantage of using end-to-end deep learning for collision avoidance compared to traditional rule-based approaches?
It requires less training data
It can learn complex patterns from examples without explicit programming of detection rules
It always runs faster than rule-based systems
It doesn't require a camera sensor
3. Why is data augmentation important when training a collision avoidance model?
It makes the model run faster
It artificially increases dataset size and helps the model generalize to variations in lighting, angle, and position
It reduces the training time
It eliminates the need for labeled data
4. In the collision avoidance application, what do the two output classes typically represent?
Left turn and right turn
Fast speed and slow speed
Free (safe to move forward) and blocked (obstacle present)
Day and night conditions
5. What is TensorRT and why is it used in this lab?
A programming language for robotics
An inference optimization library that accelerates neural network performance on NVIDIA hardware
A deep learning framework like PyTorch
A motor control library for JetBot
6. How does PWM (Pulse Width Modulation) control motor speed in the JetBot?
By changing the voltage supplied to the motors
By varying the duty cycle of power pulses to effectively control average power delivered
By changing the frequency of the motor's rotation
By adjusting the current flowing through the motors
7. Why is it important to collect diverse training data when building a collision avoidance system?
To make the training process faster
To reduce the model size
To ensure the model can handle various environments, lighting conditions, and obstacle types
To eliminate the need for data augmentation
8. What is the main challenge of deploying deep learning models on edge devices like the Jetson Orin Nano?
Edge devices cannot run neural networks
Balancing computational performance with limited resources, power constraints, and real-time requirements
Edge devices don't have cameras
PyTorch doesn't work on edge devices
9. In the JetBot collision avoidance system, how does the robot make navigation decisions in real-time?
By measuring distance with ultrasonic sensors
By feeding camera images to a trained neural network that classifies scenarios as free or blocked
By following predefined paths programmed in advance
By using GPS coordinates for navigation
10. What advantage does ResNet18 provide when used as the backbone for collision avoidance?
It requires less training data than any other model
It provides a pre-trained architecture with residual connections that can learn complex features efficiently
It is the only model compatible with TensorRT
It doesn't require any fine-tuning for new tasks
⚠️ Safety First!
Before operating the JetBot, ensure you have adequate space and that all obstacles are clearly visible. Start with low motor speeds and gradually increase as you become comfortable with the robot's behavior. Always be ready to manually stop the robot by pressing the emergency stop button or disconnecting power. Keep hands and objects clear of the wheels during operation. Never leave the robot running unattended. When testing collision avoidance, use soft obstacles (like foam blocks or cardboard) to prevent damage to the robot or environment.
In this first part, you'll learn to control the JetBot's movement by programming its differential drive motors. You'll explore how different motor speed combinations produce various motion patterns, from straight-line movement to rotation and circular trajectories. This foundational knowledge is essential for understanding how autonomous navigation systems control robot actuators based on sensor input.
🎯 Key Learning Points
- Understanding differential drive kinematics and motor control
- Programming basic movement patterns (forward, backward, turning, rotation)
- Using the Robot class API to control motors with Python
- Experimenting with motor speed values and observing robot behavior
- Understanding the relationship between motor commands and physical motion
Data collection is a critical step in supervised learning. In this part, you'll use the JetBot's camera to capture training images in two categories: "free" (scenarios where the robot can safely move forward) and "blocked" (scenarios with obstacles). You'll learn the importance of collecting diverse, representative data across different environments, lighting conditions, and obstacle types. The quality and diversity of your training data directly impacts the performance of your collision avoidance system.
🎯 Key Learning Points
- Setting up camera capture and image saving functionality
- Creating labeled datasets with "free" and "blocked" categories
- Understanding the importance of diverse and balanced training data
- Using interactive widgets for efficient data collection
- Best practices for capturing images at various angles, distances, and lighting conditions
- Organizing and managing training datasets
💡 Data Collection Tips
Collect at least 50-100 images per category for effective training. Vary the environment, obstacle types, distances, and lighting conditions. Include scenarios with partial occlusions, different heights, and various backgrounds. The more diverse your training data, the more robust your collision avoidance system will be.
With your collected dataset, you'll now train a convolutional neural network to classify images as "free" or "blocked". You'll implement the complete training pipeline including data loading, augmentation, model architecture setup, training loops, and validation. You'll experiment with different architectures including custom CNNs and pre-trained ResNet18 models. The training process includes real-time visualization of loss curves to monitor model convergence and detect overfitting.
🎯 Key Learning Points
- Loading and preprocessing image datasets with PyTorch DataLoader
- Implementing data augmentation techniques (rotation, flipping, color jittering)
- Training CNNs for binary classification tasks
- Monitoring training progress with loss curves and accuracy metrics
- Using pre-trained models (ResNet18) for transfer learning
- Saving trained models for deployment
- Understanding the tradeoffs between model complexity and inference speed
Training Options
Three training notebooks are available:
- train_model.html: Basic training with simple CNN architecture
- train_model_plot.html: Training with live loss/accuracy visualization
- train_model_resnet18.html: Training with ResNet18 backbone for improved accuracy
Choose based on your learning goals and available computational resources.
Now it's time to deploy your trained model for real-time collision avoidance! You'll load the trained model, capture camera images in real-time, perform inference, and control the robot's motors based on the model's predictions. You'll see how the robot autonomously navigates by making decisions based on what it "sees" through the camera. You'll also explore TensorRT optimization to improve inference speed for smoother, more responsive autonomous behavior.
🎯 Key Learning Points
- Loading trained PyTorch models for inference
- Implementing real-time image capture and preprocessing pipelines
- Running neural network inference on live camera feeds
- Mapping model predictions to motor control commands
- Optimizing models with TensorRT for faster inference
- Testing and debugging autonomous navigation systems
- Evaluating system performance in various environments
Deployment Options
Four deployment notebooks are available:
- live_demo.html: Basic live demo with standard PyTorch inference
- live_demo_resnet18.html: Live demo using ResNet18 model
- live_demo_resnet18_build_trt.html: Convert ResNet18 model to TensorRT format
- live_demo_resnet18_trt.html: Optimized live demo using TensorRT engine
Start with the basic demo, then progress to TensorRT optimization for improved performance.
⚠️ Testing Safety:
Start testing in a controlled, clear area. Gradually introduce obstacles. Monitor the robot's behavior closely and be ready to intervene. Test with different obstacle types, distances, and lighting conditions. Document scenarios where the system performs well and where it struggles.
Hardware Requirements
This laboratory uses the NVIDIA Jetson Orin Nano Developer Kit assembled in Week 1, integrated into the JetBot robot platform. All necessary software has been pre-configured by the lab technician.
- JetBot Platform: Complete robot with Jetson Orin Nano, motors, camera, and chassis
- IMX219 Camera Module: 8MP camera for image capture (installed on JetBot)
- DC Motors (2x): Differential drive motors with wheels
- PiOLED Display: For status information (optional)
- Power Bank: Portable power supply for mobile operation
- Testing Environment: Open space with various obstacles (foam blocks, cardboard, etc.)
Software Environment
- JupyterLab: Interactive development environment (pre-installed)
- PyTorch: Deep learning framework with CUDA support
- torchvision: Computer vision library with transforms and models
- TensorRT: NVIDIA's inference optimization library
- jetbot Python Library: Robot control and utilities
- OpenCV: Computer vision and image processing
- ipywidgets: Interactive widgets for Jupyter notebooks
📁 Exercise Files
All exercise notebooks are available in the week10/Code HTML/ directory:
basic_motion.html - Basic motor control exercisesdata_collection.html - Training data collection interfacetrain_model.html - Basic model trainingtrain_model_plot.html - Training with visualizationtrain_model_resnet18.html - Transfer learning with ResNet18live_demo.html - Basic autonomous navigationlive_demo_resnet18.html - ResNet18 autonomous navigationlive_demo_resnet18_build_trt.html - TensorRT conversionlive_demo_resnet18_trt.html - Optimized TensorRT navigation
Troubleshooting Resources
Submit a lab report documenting your work with the JetBot. Your report should clearly show what you did in each part, include screenshots of your results, and explain what you learned.
⚠️ Submission Deadline:
Submit your completed lab report by [Insert Deadline - Typically 1 week after lab session]. Late submissions will be penalized according to course policy (10% per day, maximum 3 days).
Report Structure (100 points total)
1. Cover Page (5 points)
- Course name and number (ELEC 395)
- Experiment title: "Week 10: Autonomous Driving - JetBot"
- Your name and student ID
- Date of submission
2. Objectives (10 points)
- List the main goals of this lab in your own words
- Explain what you expected to learn from each part
3. Lab Work & Results (60 points)
Document your work for each part. Include:
- Screenshots: Show your code running and the results
- Brief Explanation: Describe what you did and what happened
- Key Observations: What worked well? What was challenging?
For this lab, include:
- Part 1: Screenshots of robot moving in different patterns
- Part 2: Number of images collected and sample images from your dataset
- Part 3: Training accuracy/loss results and any plots generated
- Part 4: Evidence of autonomous navigation (screenshots or describe what happened)
4. Discussion (15 points)
- How well did your collision avoidance system work?
- What types of obstacles did it handle well or poorly?
- How could you improve the system?
5. Challenges (10 points)
- What problems did you encounter during the lab?
- How did you solve them (or attempt to)?
- What would you do differently next time?
📋 Before You Submit
- ✓ All 4 parts are documented with screenshots
- ✓ Screenshots are clear and labeled
- ✓ You explained what you did in your own words
- ✓ You included your observations and challenges
- ✓ Report is saved as PDF
- ✓ File named: Week10_[YourLastName]_[StudentID].pdf
📤 How to Submit
- Format: PDF file
- File Name: Week10_[LastName]_[StudentID].pdf
- Where: Upload to course LMS (Blackboard/Moodle)
- Size Limit: Maximum 50MB
- Optional: You may also include your notebook files (.ipynb) in a ZIP file
📊 How Your Report Will Be Graded
| Section |
Points |
What We're Looking For |
| Cover Page |
5 |
Complete and professional |
| Objectives |
10 |
Clear understanding of lab goals |
| Lab Work & Results |
60 |
All parts completed with clear documentation |
| Discussion |
15 |
Thoughtful analysis of results |
| Challenges |
10 |
Honest reflection on difficulties |
| Total |
100 |
|
Important Notes:
- Include screenshots for every part you completed
- Write in your own words - don't copy from the lab manual
- It's okay if something didn't work perfectly - explain what happened
- Late penalty: 10% per day (maximum 3 days)
- Copying from others = zero credit