Robotic Sampling & Collection

Project overview

Our five-person team developed a sampling and collection robot as a Student Innovation Training Program (SITP) project. The project ran from December 2024 to March 2026 and received a “Good” rating on completion.

The project was organized around a complete task: moving through an environment, sensing and mapping the surroundings, reaching an area near a target, identifying an object, and using a manipulator and gripper to collect it. Taking irregular-object sampling in planetary exploration as the application scenario, we built and tested an indoor prototype that brought perception, navigation and grasping onto one physical platform.

I led the project and participated throughout. My main work was deploying and debugging FAST-LIVO2 on edge hardware, integrating the manipulator and gripper, adapting the grasping system and carrying out whole-robot integration tests. Another team member led navigation development; I supported configuration, integration and its connection to grasping.

Hardware and workflow

The robot uses a Scout Mini omnidirectional mobile base and a Jetson Orin NX onboard computer. ROS (Robot Operating System) connects the modules. Environment sensing uses a Livox Mid360 LiDAR, an industrial camera and inertial data. Close-range manipulation uses a six-axis PiPER arm and a Pika gripper with an integrated RealSense D405 depth camera.

The system is organized into four stages:

1. Perception and mapping. FAST-LIVO2 combines LiDAR, visual and inertial data to estimate the robot’s pose and produce a 3D point cloud. The team calibrated the camera intrinsics and LiDAR–camera extrinsics, with an STM32 board coordinating sensor acquisition timing.
2. Movement and navigation. We integrated NEXTE_Sentry_Nav, used OctoMap to convert point-cloud data into a map for navigation, and used move_base to plan movement and send velocity commands to the base. After testing both offline map conversion and live mapping, we adopted the live approach.
3. Recognition and localization. The grasping system builds on AgileX’s GraspGen, using YOLOv8 detection and TensorRT inference. D405 depth data provides the object’s 3D location, which is transformed into the coordinate frame used by the manipulator.
4. Grasping and collection. Inverse kinematics translates target positions into arm motion, followed by gripper control. The system design connects navigation near a target with close-range manipulation. We validated navigation and grasping separately before testing their operation on the assembled robot.

My deployment and integration work

Running perception on edge hardware

The project initially used a Jetson TX2. Algorithm tests exposed a real-time computing bottleneck, leading the team to upgrade the platform. I worked on FAST-LIVO2 environment setup, driver debugging and runtime validation to bring sensor data into the mapping pipeline.

This required coordinating drivers, calibration and runtime libraries. For example, the camera SDK bundled a libusb version that conflicted with system dependencies and disrupted camera and mapping startup. Troubleshooting involved tracing device drivers, shared-library loading and ROS nodes. After resolving compatibility issues and applying the calibration configuration, real-time FAST-LIVO2 mapping worked, and other team members reproduced the setup.

Connecting the arm, gripper and depth camera

I configured PiPER’s CAN communication, verified SDK control and deployed its ROS driver. I also integrated the Pika gripper and D405 depth camera, then adapted the combined URDF (Unified Robot Description Format) model. This included joint limits, gripper linkage and camera coordinates so that the software model matched the physical robot.

The devices also had to operate alongside the mobile base, requiring coordination of CAN interfaces, USB connections and power. I checked device recognition and basic motion first, then the model, sensor data and control nodes before integrating them into the grasping system.

Debugging the grasping pipeline

After deploying GraspGen, object detection initially produced no bounding boxes. Comparing the weights, code and inference-engine build versions led me to a serialization-format mismatch between the TensorRT engine and YOLO plugin. Rebuilding the engine with a matching tensorrtx version restored detection. I then addressed conflicting OpenCV versions and used ROS image topics to inspect detection results.

The detected positions also had to match the physical scene. By comparing real-object positions with RViz, I identified a duplicated optical-frame transformation and corrected the camera orientation in the URDF. For persistent gripper oscillation, I traced the cause to a debugging interface and the inverse-kinematics node both publishing joint states, then adjusted the launch configuration to remove the conflict.

These problems occurred at different layers: inference compatibility, coordinate transforms and control interfaces. I worked through them individually and verified the grasping chain from detection and depth-based localization to arm movement and gripper actuation.

Integration results and next steps

In March 2026, we completed whole-robot integration tests with the mobile base, mapping, navigation, manipulator, gripper and grasping system operating on the same platform. The demonstrations below show mapping, navigation, grasping and combined operation.

Current experiments use a small set of common objects in an indoor environment. Further work includes refining the camera’s translational calibration, adapting data and models for irregular targets such as rocks, and evaluating stability during longer runs.

The project gave me practical experience with Linux, ROS, sensors and robotic manipulation. Starting an individual module was one step in integration; making it work with the others also required consistent timing, coordinates, interfaces and computing resources. Tracing problems through these layers using feedback from the physical robot was a central part of what I learned.

Project demonstrations