In this laboratory experiment, you will implement an autonomous road-following system for the JetBot using regression-based deep learning. Unlike classification tasks that predict discrete categories, regression predicts continuous values - specifically, the X and Y coordinates of a target point on the road that the robot should follow. By training a neural network to estimate where the robot should navigate, you'll enable autonomous path-following behavior that forms the foundation of self-driving systems. This laboratory brings together computer vision, deep learning, and robotics to create an intelligent system capable of navigating paths without explicit programming.
You'll progress through the complete autonomous driving pipeline: collecting training data by manually labeling target points on road images, training a ResNet-18 neural network to predict steering coordinates using regression, and deploying the trained model for real-time autonomous navigation. The regression approach allows the robot to continuously adjust its trajectory based on visual input, following paths with smooth and natural-looking motion. This hands-on experience demonstrates how modern autonomous vehicles use neural networks to perceive their environment and make continuous control decisions.
Autonomous navigation is revolutionizing transportation, logistics, and robotics. Self-driving cars must continuously predict steering angles to follow lanes safely. Warehouse robots need to track paths to transport goods efficiently. Drones require trajectory prediction to navigate complex environments. Agricultural robots must follow crop rows precisely. This laboratory demonstrates the regression-based approach that powers these systems - predicting continuous control values from visual input. By implementing road following on the JetBot, you're learning the core techniques used in full-scale autonomous vehicles, scaled down to an educational platform. The skills you develop here - data collection strategies, regression model training, and real-time inference on edge devices - are directly applicable to industrial autonomous systems, making this laboratory a practical introduction to the technology driving the future of transportation and robotics.
This laboratory consists of three progressive parts that build upon each other to create a complete autonomous driving system:
By the end of this laboratory session, you will be able to:
Regression is a supervised learning technique that predicts continuous numerical values rather than discrete categories. While classification answers questions like "Is there a stop sign?" with yes or no, regression answers "What steering angle is needed?" with a specific numerical value. In autonomous driving, regression enables vehicles to predict continuous control parameters such as steering angles, throttle positions, and trajectory waypoints. This continuous prediction capability is essential for smooth, human-like driving behavior that adapts fluidly to changing road conditions.
The road following task demonstrates end-to-end learning, where a neural network learns to map raw camera images directly to steering coordinates without explicit programming of driving rules. During data collection, human operators label target points on road images indicating where the robot should navigate. The neural network learns patterns in the visual data that correspond to correct navigation targets, effectively learning an implicit model of road-following behavior. This approach mirrors how self-driving car companies like Tesla and Waymo use neural networks to process sensor data and generate driving commands, though production systems employ multiple sensors and redundant safety systems beyond the scope of this educational laboratory.
ResNet-18 (Residual Network with 18 layers) is a convolutional neural network architecture that uses skip connections to enable training of deep networks without gradient vanishing problems. Originally designed for ImageNet classification with 1000 output classes, ResNet-18 can be adapted for regression by replacing its final fully connected layer. For road following, we modify the output layer from 1000 neurons (for classification) to 2 neurons (for X and Y coordinates), transforming the network from a classifier to a regression model. The pre-trained weights from ImageNet provide general feature extractors for visual patterns, which we fine-tune for the specific task of predicting navigation coordinates.
Transfer learning accelerates training and improves performance by leveraging knowledge learned from millions of ImageNet images. The early layers of ResNet-18 detect edges, textures, and basic shapes that are useful for any computer vision task, while later layers learn task-specific features. By fine-tuning a pre-trained model rather than training from scratch, we can achieve strong performance with relatively small datasets - often just a few hundred labeled images. This efficiency is crucial for practical robotics applications where collecting massive datasets is impractical. The regression task uses Mean Squared Error (MSE) loss to minimize the difference between predicted and actual coordinates, encouraging the network to make accurate continuous predictions.
Effective data collection is critical for successful autonomous driving. The "green dot" labeling approach provides intuitive target specification: operators click on camera images to place a marker indicating where the robot should navigate. This target point should be positioned as far along the desired path as the robot can safely travel in a straight line. On straight sections, the target can be placed near the horizon; on sharp curves, it must be closer to avoid steering the robot off the path. This variable target distance naturally encodes the urgency of steering corrections - tight turns require more immediate responses than gentle curves.
Data diversity is essential for model generalization. Collecting images from various positions (centered, left-offset, right-offset) and angles ensures the network learns to handle diverse scenarios. Including examples where the robot starts partially off the path helps it learn recovery behaviors. The "carrot on a stick" metaphor describes the control strategy: the neural network continuously predicts where the carrot (target point) should be, and a simple proportional controller steers the robot toward it. As the robot moves forward, the network continuously updates its prediction, creating smooth path-following behavior. This approach separates high-level path perception (the neural network's job) from low-level motor control (the proportional controller's job), demonstrating a key principle of hierarchical control in robotics.
Autonomous driving requires low-latency inference to respond quickly to changing road conditions. Running neural networks on embedded devices like the Jetson Orin Nano presents computational challenges - models must execute fast enough for real-time control (typically 10-30 frames per second) while maintaining accuracy. PyTorch provides a flexible research framework but is not optimized for embedded inference. TensorRT, NVIDIA's inference optimization engine, addresses this by converting PyTorch models to highly optimized representations that execute significantly faster on Jetson hardware.
TensorRT optimization involves several techniques: layer fusion combines multiple operations into single optimized kernels, precision calibration reduces numerical precision from 32-bit to 16-bit floating point without significant accuracy loss, and kernel auto-tuning selects the fastest implementation for each operation. These optimizations can accelerate inference by 2-5x compared to standard PyTorch execution, enabling higher frame rates for more responsive autonomous navigation. The laboratory provides both standard PyTorch and TensorRT-optimized inference paths, allowing comparison of performance trade-offs. Understanding these optimization techniques is valuable for deploying any neural network on resource-constrained edge devices, from robots to IoT devices.
Complete the following preparation activities before your laboratory session to ensure you understand the fundamental concepts and can work efficiently during the hands-on exercises.
No video tutorials are available for this experiment. Review the reference materials carefully and be prepared to work through the interactive notebooks during the laboratory session.
Test your understanding of regression-based autonomous navigation concepts with these questions:
This laboratory consists of three progressive parts that guide you through building a complete autonomous road-following system. Follow the procedures carefully and document your results at each stage.
In this part, you will collect training data by manually labeling target navigation points on road images captured from the JetBot's camera. You'll use an interactive Jupyter notebook that displays the live camera feed and allows you to click on images to specify where the robot should navigate. Each click saves an image along with the X and Y coordinates of the target point, creating a dataset that maps visual scenes to desired navigation targets.
Collect at least 100-150 images from diverse positions and angles. Include examples where the robot is centered, offset to the left, offset to the right, and at various angles to the path. This diversity is crucial for training a robust navigation model.
With your collected dataset, you'll train a ResNet-18 neural network to predict steering coordinates using regression. The notebook guides you through dataset preparation, model configuration with transfer learning, training loop implementation with Mean Squared Error loss, and model evaluation on test data. You'll monitor training progress through loss curves and save the best-performing model for deployment. The training process typically requires 50-70 epochs and can be completed on the Jetson in 20-30 minutes.
Monitor both training and validation loss. If validation loss stops improving while training loss continues decreasing, you may be overfitting. Consider stopping training early or collecting more diverse data.
Deploy your trained model for real-time autonomous road following. The live demo notebook loads your trained model, captures camera frames, predicts target coordinates, and converts predictions into motor commands for autonomous navigation. You'll implement a proportional controller that steers the robot toward the predicted target point, creating smooth path-following behavior. The laboratory includes both standard PyTorch inference and optimized TensorRT versions for performance comparison.
Always test autonomous navigation in a controlled environment. Start with low speeds and be ready to manually stop the robot if it deviates from the path. Ensure adequate space around the track to prevent collisions.
No software installation required! All necessary libraries and dependencies have been pre-configured by the lab technician on your Jetson Orin Nano. The system is ready for immediate use - simply power on your JetBot and access JupyterLab through your web browser.
JupyterLab Access:
http://jetbot-[YOUR_ROBOT_NUMBER].local:8888 in your web browserweek12 folder to access the laboratory notebooksYour lab report should comprehensively document your experience building and testing the autonomous road-following system. Focus on practical implementation details, experimental observations, and insights gained from hands-on work with regression-based navigation.
ELEC395_Week12_[YourStudentID]_[YourName].pdf| Component | Points | Evaluation Criteria |
|---|---|---|
| Cover Page | 5 | Complete, professional |
| Objectives | 10 | Clear, comprehensive |
| Procedure & Results | 60 | All parts documented, clear screenshots, working system |
| Discussion | 20 | Analysis of results, challenges, and solutions |
| Conclusion | 5 | Reflective, insightful summary |
| Total | 100 |