🔶Generate Pick Object Pose from Point Cloud Aligned Detection Results

Function Description

This operator combines 3D point cloud information with 2D image detection results to generate pick poses for detected objects. It first aligns the input point cloud list with the input 2D detection result list, then calculates the depth (Z coordinate) of pick points based on aligned point clouds, and combines geometric information from 2D detection results (such as minimum bounding rectangles) to determine the XY coordinates and rotation angle around the Z-axis of pick points, ultimately outputting poses based on the robot base coordinate system.

Usage Scenarios

  • Robot Picking : In vision-guided robot picking tasks, precise 6D pick poses need to be calculated for identified objects so robots can pick accurately.

  • Pose Estimation : Combine category and position information from 2D detection with depth and shape information from 3D point clouds to estimate complete poses of objects in space.

Input/Output

Input

Camera Coordinate System Point Cloud： Input point cloud list, typically point cloud clusters corresponding to objects to be picked after filtering or clustering of point clouds extracted from detection results. Must be point clouds in camera coordinate system without NaN values.

Image： Original image (color or grayscale) corresponding to the point cloud, used for projecting point clouds back to pixel coordinates.

Detection Results： Output result list from 2D detection operators, each result containing object’s bounding box/polygon, score, category, angle and other information.

Camera Intrinsics： 3x3 camera intrinsic matrix.

Camera Distortion： Camera distortion coefficient vector.

Hand-Eye Calibration Matrix： 4x4 transformation matrix from camera coordinate system to robot base coordinate system.

Output

Pick Object Pose Information：

Calculated pick pose list. Each element is a dictionary containing:

  • pose：Pick object pose.

  • score：Confidence score of corresponding 2D detection result.

  • class_name：Category of corresponding 2D detection result.

  • uuid：Unique identifier of corresponding 2D detection result.

  • polygon：Polygon contour of corresponding 2D detection result.

  • object_info：Dictionary containing object dimensions, area and other information, where dimensions and area are actual measurements in mm and mm^2.

  • points_number：Index of corresponding input point cloud in original list.

Parameter Description

Use Detection Result Angle

Parameter Description

Determines whether the rotation angle (around Z-axis) of final output pose directly adopts the angle provided by 2D detection results, or recalculates based on point cloud and minimum bounding rectangle of detection results.

Parameter Adjustment

  • Disabled (Default) : Do not directly use detection result angles. The operator calculates the short edge midpoint connection line of detection result’s minimum bounding rectangle, projects it to camera coordinate system and converts to robot base coordinate system, calculating the angle between this line and base coordinate system X-axis as final pose Rz rotation.

  • Enabled : Directly use the angle value provided by each instance in input "Detection Results" as final pose Rz rotation, generally for rotation detection which would have this value, other detection results have angles of 0.

Alignment Strategy

Parameter Description

Select the method for matching and aligning input point cloud clusters with 2D detection results.

Parameter Adjustment

  • Center Point Alignment (Default)： Calculate distance between center point of point cloud projected to image and detection result mask center point, select closest for matching. Fast computation, suitable for scenarios where objects are well-separated with distinct center points.

  • Mask IoU Alignment： Project point cloud to image to generate mask, calculate IoU with detection result mask, select highest IoU for matching. Considers shape overlap, may be more robust for tightly fitting objects or partial occlusion scenarios, but requires slightly more computation.

Pick Point Z Value Calculation Method

Parameter Description

Select how to calculate the Z coordinate value of final pick pose based on aligned point cloud clusters.

Parameter Adjustment

  • Mean (Default)： Use average Z coordinate value of all points in point cloud cluster. Suitable for good quality point clouds with few outliers.

  • Median： Use median Z coordinate value of all points in point cloud cluster. Not sensitive to outliers, more robust results, recommended when point clouds have high noise levels.

  • Mean of circular region at mask center： At the center of detection result’s minimum bounding rectangle, take a circular region defined by "Circle Radius Ratio", calculate average Z coordinate of point cloud in this region.

Circle Radius Ratio

Parameter Description

Effective when "Pick Point Z Value Calculation Method" selects "Mean of circular region at mask center". Defines the radius of circular region used for calculating Z value, this radius is a ratio relative to half the short edge length of detection result’s minimum bounding rectangle.

Parameter Adjustment

This parameter controls the object surface area size referenced when calculating depth Z value.

  • Smaller ratio means only considering points in very small region at object center, may be more stable but susceptible to noise influence;

  • Larger ratio considers points in larger range, may be smoother but includes more surface variations.

Parameter Range

[0, 1], Default: 0.3