Z's learning note

Search

SearchSearch

dlo_detection

Jul 19, 2021, 8 min read

Deformable linear object detection

Fast, accurate thin-structure obstacle detection for autonomous mobile robots

Chen Zhou et al., CVPR workshop 2017

  • Progress 30%, link

  • 要解决的问题：detect thin-structure obstacle (wires, cables, tree branches)

  • 解决方案：edge-based visual odometry techniques Perform thin obstacle detection using video sequences from a monocular camera or a stereo camera pair

  • 具体方案

    1. Edge extraction: DoG (Difference of Gaussians) detector for edge detection + a consequent Canny-style hypothesis linking step
      1. 选择DoG的原因：its good repetitivity
      2. 使用hypothesis linking step的原因：improve the recall of weak edges which are common on thin obstacles
    2. Edge 3D reconstruction
      • Monocular camera: VO (Visual Odometry) algorithm Incorporating IMU data
      • Stereo camera Edge-based stereo matching
    3. Obstacle labeling on edge depth maps

DROAN - Disparity-Space Representation for Obstacle Avoidance: Enabling Wire Mapping amp; Avoidance

Geetesh Dubey et al., IROS 2018

  • CNN

Multi-view Reconstruction of Wires using a Catenary Model

Ratnesh Madaan et al., ICRA 2019

  • model based

Detection and Reconstruction of Wires Using Cameras for Aircraft Safety Systems

Adam Stambler et al., ICRA 2019

  • CNN

Let’s Take a Walk on Superpixels Graphs: Deformable Linear Objects Segmentation and Model Estimation

Daniele De Gregorio et al., ACCV 2018

  • Source code
  • over-segmentation of the source image into superpixels to buid a Region Adjacency Graph (RAG)
  • DLO检测的六个步骤： 0. Endpoints检测，交由外部的算法完成 Fine-tuned YOLOv2 model pre-trained on ImageNet based on Electrical Cable Dataset
    1. Superpixel Segmentation：将原始图片分割为相邻子区域（adjacent sub-regions, superpixels），从而极大的减少了搜索时的分区数量（相比像素图）。并在此基础上由超像素间的相邻关系建立邻接图（adjacency graph），步骤0中检测出的端点作为种子点seeds
    2. Start walks：从种子点出发，沿邻接图搜索（创建一系列搜索方向？）
    3. Extend walks：从当前点出发抵达下一个最优的相邻超像素
    4. Terminate walks
    5. Discard Unlikely Walks
  • 超像素分割的方法：本文使用SLIC（SLIC super-pixel compared to state-of-the-art superpixel methods）

    • Segmentation with high compactness

      During the clustering process the compactness of each cluster can be either increased or reduced to the detriment of visual similarity. In other words we can choose easily to assign more importance to visual consistency of superpixels or to their spatial uniformity.

    • Search

      similar to Region Growing algorithm. restrict the search along a walk applying several model-base constraints rather than relying only on visual similarity only

      • Visual likelihood：使用Color Histogram，HSV色彩空间
      • Curvature likelihood
      • Distance likelihood
      • Estimation of the most likely walk

3D DLO Shape Detection and Grasp Planning from Multiple 2D Views

Alessio Caporali et al., IEEE/ASME AIM 2021

  • In this paper:

An algorithm called ARIADNE [DeGregorio_ACCV2018] is used to detect the DLOs from each input camera image through deep-learning based segmentation and providing a B-Spline representation of the DLO in pixel coordinates.

Through deep-learning based segmentation? 应该是指的用YOLO找出线缆末端

  • 步骤：

    • DeepLabV3+用来移除背景进行二值化，生成binary mask，训练集为合成数据

    • 对binary mask使用SLIC进行超像素化

      • 为啥对二值化之后的图像再进行超像素化？

      The idea of superpixels is to partition the image into local meaningful areas making the further processing easier and faster. In particular, a modified version, called MaskSlic [13], that is capable of applying the superpixel segmentation only on a region of interest (i.e. the foreground pixels) is used, due to the availability of the binary mask previously computed. The application of MaskSlic allows us to exploit completely the result of the initial semantic segmentation.

    • 生成region adjacency graph（RAG）

FASTDLO: Fast Deformable Linear Objects Instance Segmentation

Alessio Caporali et al., RAL2022

FASTDLO focuses directly on solving the intersection areas of the image between multiple DLOs to distinguish the instances, increasing the speed and accuracy of the results.

  • 未使用SLIC?
  • 六个步骤
    1. Background segmentation 通过DCNN进行segmentation，输出binary mask
    2. Skeleton pixels classification 从生成skeleton，对local neighborhoods的像素进行分类
    3. Segments generation 找出产生交叉的位置
    4. Intersections processing A shallow neural network to predict connection probabilities among endpoint-pairs
    5. Informed merging
    6. Intersections layout
  • It (Ariadne+) employs the same segmentation network architecture of the one introduced in Section III-A (background segmentation) to distinguish the DLOs from the scene

配置与运行

  • 依赖的安装参见Ariadne+的配置，此外需要额外安装shapely
    • pip3 install shapely
  • 安装：在根目录下运行pip install .
  • 测试：可以在克隆项目后在根目录下新建”weights”文件夹，注意是复数，用来存放网络参数 run.py为例程，读取weights的目录和图片都是在这个源代码中配置的

Ariadne+: Deep Learning-based Augmented Framework for the Instance Segmentation of Wires

Alessio Caporali et al., IEEE Trans on Industrial Informatics 2022

Ariadne+ [10] was recently introduced as an improved version of Ariadne concerning several aspects: better accuracy and efficiency, ability to consider even more complex scenarios in which the endpoints of the cable were not present in the image. Currently, Ariadne+ represents the state of the art in terms of DLOs instance segmentation. However, its throughput is limited to a few FPS providing a strong limiting factor for its applicability on real-world applications.

  • DeeplabV3进行semantic segmentation
  • Superpixel segmentation

配置与运行

  • 源代码code
  • 环境配置：
    • pip3 install arrow
    • pip3 install termcolor
    • pip3 install igraph
    • pip3 install scikit-image
    • pip3 install pytorch-lightning
      • 1.7.1可以用（之后需将pytorch强制降级为1.8.2）
      • 或者安装1.6.5
    • Pytorch version: 1.8.2可以运行
      • CPU版本
        • pip3 install torch==1.8.2 torchvision==0.9.2 torchaudio===0.8.2 --extra-index-url https://download.pytorch.org/whl/lts/1.8/cpu
      • CUDA 11版本
        • pip3 install torch==1.8.2 torchvision==0.9.2 torchaudio==0.8.2 --extra-index-url https://download.pytorch.org/whl/lts/1.8/cu111
      •  force-install PyTorch 1.8 after installing PL
    • RLException: Roslaunch got a 'No such file or directory' error while attempting to run...
      • 解决方案：run_ariadne_service.py，修改第一行为#！/usr/bin/python3

Deformable Linear Objects 3D Shape Estimation and Tracking from Multiple 2D Views

Alessio Caporali et al., University of Bologna, RAL 2023

A Robust Deformable Linear Object Perception Pipeline in 3D: From Segmentation to Reconstruction

Sun Zhaole, et al., University of Edinburgh, RAL 2024

Backlinks