The 22312 robot lining up over the sample field and collecting blocks under Limelight vision. — Featured above: Limelight-guided collection under test — the robot locating a sample and picking it up on its own

Vision Pick and Place Robot

Languages: Java, Python
Tools: Limelight Camera Pipeline
When: 2024–2025

Overview

During the 2024–2025 season as part of FTC Team 22312, I developed a fully autonomous, vision detection and localization algorithm to grab blocks of a specific color — and reject blocks of the wrong color.

This algorithm included 2 main components:

Locating every viewable block
Planning and executing subsystem movements

Team 22312’s FTC robot photographed from above on a competition field: a black chassis numbered 22312, with a scissor-linkage extension, a linear slide, and a yellow intake claw at the front.

1 Vision + Localization

I tuned Limelight 3A's color detection pipeline to segment the image into blocks, and obtained the center pixel values of each block. To account for perspective projection, I calibrated a homographic transformation that converted pixel values into field coordinates. As a result, this pipeline gave me an ArrayList of field coordinates of all visible blocks.

Two camera frames side by side, labelled Before Homography — includes perspective distortion — and After Homography — almost no perspective distortion. The first looks down the field at an angle with yellow blocks foreshortened; the second is rectified to a flat overhead view.

2 Subsystem Movements

To decide which block the robot should target, the robot computes a cost for each block, and select the block with the minimum cost

Cost function:

J = \alpha \sqrt{\Delta x^{2} + \Delta y^{2}} + \beta \left|\tan^{-1}\!\left(\frac{\Delta y}{\Delta x}\right)\right|

$\alpha$: distance weight
$\beta$: angle weight
$\Delta x,\ \Delta y$: block coordinates relative to the front of the robot

To fully utilize the robot's sideways oriented spindles, I designed 2 types of collection modes

Forwards Collection

If the necessary angle of rotation is less than a tuned threshold, directly rotate and extend to the block. This method contacts the block with the front facing spindles.

Sideways Collection

If the angle of rotation is greater than the tuned threshold, rotate to the threshold, extend the collector out, and then rotate the remaining angle. This method contacts the block using the sideways facing spindles.

After computing all the calculations, I used the Roadrunner library to build pre-planned actions controlling each subsystem.