Fusing Point Cloud and Image Yields 6D Pose in a Stacked Bin
After plate cutting, workpieces drop straight into a scrap bin to be sorted and stacked by specification. A single batch can mix dozens of specifications, and similar shapes are hard enough for a person to separate before stacking makes it worse. If mixed stock reaches the fit-up station, the result is wrong assembly and rework, so the bin has become the bottleneck for the whole cutting line.
A kerf a few millimetres wide carries the identity
The difficulty is not a defect but an absence of surface information. The kerf is faint, the surface has no texture or colour variation, and similar specifications differ only in a few millimetres of contour detail, often while pressing on each other. Edge- or texture-based algorithms have nothing to work with. Defining the problem as missing information points to the right remedy: add the missing information instead of making the matcher more sensitive.
Align the modalities before fusing them
A high-precision 3D camera captures point cloud and image synchronously. The two are aligned in a common coordinate frame at the data level, then fused at the feature level. The order cannot be reversed: if each modality keeps its own coordinates, the fused feature is displaced. Once aligned, the point cloud supplies 3D geometry and the image supplies contour and surface detail, so when lighting or a faint kerf degrades one channel, the other still carries evidence.
Attention is a budget spent on corners and holes
An attention mechanism in the matching stage weights kerf lines, corners and holes, and suppresses reflection and texture noise. Near-identical specifications often differ along one short segment of one corner; spreading compute evenly across the image dilutes exactly that evidence. The weighting has a useful side effect: matching tolerates local deformation, so a distorted region does not collapse the whole identification.
Six degrees of freedom is a gripper requirement
Ordinary localization gives three position degrees of freedom, enough for an arm to reach above a part. A 6D pose adds three rotations, which is what tells the gripper at what angle to close. On slender or irregular parts, the wrong orientation does not mean a near miss; it means the part cannot be gripped or slides out. The pose network outputs rotation and translation parameters the robot consumes directly, with no manual conversion in between.
Every pick rearranges the pile
The cell works in a scan, pick, rescan loop. Each removal can make the remaining parts collapse or flip, so the next round rescans and updates poses instead of trusting a static memory of the initial state. Depth information separates upper and lower layers, which prevents the most common stacked-bin error: treating a pinned lower part as the top target and grabbing air, then stopping the line to recover.
The picking policy prefers unoccluded parts with high pose confidence, then updates after each removal until the bin is empty. The order is not convenience. Removing easy parts exposes the pinned ones, so total throughput is higher; reaching for a buried part first usually means moving several others.
Shop light, distortion, and the edge of the method
Overhead lighting plus floor reflection produces highlights and shadows in 2D images, so the system leans on point-cloud geometry and uses image texture as support. Cut outlines deviate from nominal, and attention relaxes the match locally where they do. The cycle is about 11 seconds per part, subject to site acceptance results. Because the stack is insensitive to lighting and surface condition, it also handles parts straight from cutting without a cleaning step.
Three questions from the selection review
Why does cross-modal fusion improve recognition?
Point clouds contribute 3D geometry; images contribute contour and detail. When one fails under poor light or a faint kerf, the other still provides evidence.
How does this compare with plain template matching?
Few specifications, stable shapes and a small field of view favour template matching on cost. Many specifications, visible distortion and stacked picking favour 6D pose with fusion.
Can pose estimation confuse parts pressed together?
Depth layering separates upper and lower surfaces before pose estimation, and a rescan after each pick avoids acting on a stale relationship.