With the development of novel display technologies such as 3-D graphical applications, virtual and augmented reality, there are new demands for more efficient and natural multi-degree-of-freedom (DOF) interaction devices. However, traditional rigid touchscreens provide only the 2-D position of the touch point as input information, which cannot meet the needs of high DOF interactions, and many of the existing interaction technologies suffer from the problems of low resolution and lack of information. At the same time, non-rigid interaction interfaces provide deformability for input and have been shown to extend the richness of input vocabulary. This article proposes a new non-rigid input device, OneTip, for single-fingertip human-computer interaction with 6-DOF. In terms of design and manufacture, OneTip employs the bio-inspired design of Skin-On interfaces to mimic the sensitivity of human skin and provide 6-DOF interaction capabilities. In terms of sensing, OneTip uses the visuotactile sensing technique based on the marker displacement method to achieve high-resolution and multi-modal measurements. We propose a novel fingertip pose estimation method based on incipient slip detection, a non-learning algorithm that does not require registration and priori information. Experiments show that OneTip had good 6-D pose estimation accuracy, with RMSEs of translation and rotation not exceeding 0.1mm and 2.6°, respectively, within the linear interval. Extensive experiments were also conducted to explore the application of OneTip in typical virtual manipulation tasks and the possibility of combining it with other interaction devices. This work is intended to serve as a reference for other researchers exploring innovative interaction techniques.

Tactile perception has been a hot topic of research in robotics. Robots sense the shape, material, distributed force, slip during contact, and use the multi-modal contact information to control grasping and manipulation. For vision-based tactile sensors, the contact representation and extraction determine the quality of the raw tactile information, and therefore serve a significant role in the robot perception system. This article highlights for the first time the importance of raw representation and extraction in visuotactile perception, and proposes a new multicolor CMP method for enhancing the performance of vision-based tactile sensors. Based on the principle of continuous marker pattern (CMP), the multicolor CMP method is optimized in the pattern and algorithm design. Regarding information representation, we present a new type of marker pattern based on RGB triangles and a preferred layout. In terms of information extraction, we propose a series of extraction strategies with the adaptive growing algorithm (AGA) and the spin-search algorithm (SSA) as the cores. The experiments reveal that the multicolor CMP method achieves improved precision and reliability compared to the former CMP method.

This article presents a detailed review and categorizing of the marker displacement method (MDM) used in vision-based tactile sensors. Vision-based tactile sensors have been proven to be a promising solution for robot tactile perception. Among such sensors, MDM is one of the most commonly used contact characterization and extraction methods. It uses visual approaches to obtain contact deformation and achieve multimodal tactile perception using physical models and post-processing algorithms. In recent years, many tactile sensors using MDM have been developed. However, the existing research does not strictly distinguish between the different types of methods but is uniformly grouped into MDM. Without differentiation, there might be a lack of systematic and comprehensive guidance in analyzing and optimizing the characteristics of MDM and selecting the most suitable method. This article is the first to classify MDM into three typical categories based on the dimensionality perspective: 2D MDM, 2.5D MDM, and 3D MDM. 2D MDM relies only on the monocular camera to acquire the marker array’s 2D displacement field. 2.5D MDM supplements 2D MDM with selected indirect features reflecting the location of the markers in the third dimension. 3D MDM employs a multi-camera system and can obtain the 3D displacement field using the stereo vision method common. Based on the latest literature, we compare the principles, characteristics, advantages and disadvantages, and applications of the three ways in detail. This work can provide a valuable reference for researchers interested in applying MDM in fields such as vision-based tactile sensors.