Demo: Table Pick and Place

Description

A pick and place is a common task in manufacturing and production lines. In this demo the camera identifies objects and recognizes object bounding box. The robot's movements are planned accordingly.

 

The objective of this demo is to program robot to pick parts randomly placed on a table after preparing the training data and training.

 

This demo shows how to identify an object and determine its location and orientation.

 
Step by step

We first need to manually prepare training data, using approximately 60 images of an object. For each image, a bounding box of an object needs to be indicated.

The images must then be split in to training and validation sets.

The following steps are explained in subsequent sections:

  • Capturing the images

  • Preparing the training set

  • Preparing the validation set

  • Training the model

  • Testing results

 
To capture the images

  1. Record images using AI Accelerator Dashboard. Every time you tap Save Image, the current camera view is saved as PNG image in a folder pandai_ark/ros/data/images on Compute module.
To prepare the training set

  1. Open the web browser on the compute module and navigate to MakeSense.ai

    This publicly available web site helps to label and annotate images.

  2. Click Get Started and upload 80% of your images.

  3. When the upload is complete, click Object Detection.

  4. You need to specify labels for the objects you want to be recognized. Click the labels list and define at least one label.

    Name the new label object_1.

    You can define multiple labels and map multiple objects on a single image, as well as multiple instances of the same object.

  5. Click Start project.

  6. Specify a bounding box for each image.

    Use the polygon tool and click points to map the bounding box.

  7. Select object label for this new polygon.

  1. Once you annotate all the images in the training set, click Actions and Export Annotations.

  2. Select Single file in COCO JSON format and click Export.

  3. Save the exported file to Downloads.

  4. Inside the /ros/data/datasets create folder rtdetr_active

  5. Inside create "train" folder and copy images of the training set and downloaded coco json file in here. Rename json file to coco_train.json

The training data consists of a combination of the JSON file and images.

 
To prepare the validation set

  1. Repeat the above steps to create the validation set. Starting with uploading remaining 20% of the images to MakeSense.ai

  2. Make sure to use the same label as in the training set.

  3. Create folder validation inside the rtdetr_active

  4. Rename downloaded coco json file to coco_validation.json and copy it together with the images from validation set in to the validation folder.

 
To train the model

  1. Train the model using GUI (you can restart it from run_ark.sh )

  2. Use Train model in rtdetr_active column of the GUI.

The training of the model can take anything from several minutes, depending on the total number of images and labeled objects in each image.
Testing the model

  1. Run the run_ark.sh on the compute module.

  2. Load the model in rtdetr_active column of the GUI.

  3. Open AI Accelerator Dashboard on the robot.

  4. Place an object within camera view.

  5. Select detection2d view from the drop-down.

  6. Tap Detect in the right column.

  7. If an object is recognized a green bounding box will appear in camera view.

 
Example of using object location

A specific robot position stored in this program. Before executing this program check that robot can freely move to detect_wp and poses no risks.

  1. Set program speed to 10%
  2. Select detect_wp and tap Move Here
  3. Verify that there are no obstructions
  4. If necessary, Freedrive robot to a new position and save it as detect_wp

Included withAI AcceleratorSDK you can find example of a robot program using the recognition results.

  1. On the robot open ark_example_detect program, installed during setup.

  2. Place objects within camera view.

  3. Run the program.

  4. Robot moves to waypoint detect_wp and captures an image.

    The function ark_detection_retrieve() returns robot pose matching the bounding box of the recognized object.

  5. If a pre-trained object is recognized, the robot will move to position 150 mm above an object. If multiple objects are recognized, robot will randomly choose one.