A-SEE: Active-Sensing End-effector Enabled Probe Self-Normal-Positioning for Robotic Ultrasound Imaging Applications

A. Related Works on Probe Normal Positioning

Maintaining the normal direction with respect to the skin surface is predominantly used as the default optimal probe orientation in RUSS imaging [17]–[20]. The orthogonal probe orientation against the patient body assures more reflected acoustic waves received. It also serves as a good starting point for searching for any anatomical landmark [16]. Preoperative normal angle estimation along a scan trajectory overlaid on a 3D patient body point cloud (obtained from an RGB-D camera) has been implemented in [18]–[22]. The normal vectors for each point in the point cloud can be estimated by fitting a plane to its nearest neighbors. Nevertheless, it is physically challenging to perform real-time normal-based orientation adjustment using this approach for the following reasons: i) The scanned surface may be occluded by the probe, the robot, or other obstacles in the environment, making it difficult to continuously detect the body; ii) Detection algorithms are needed to extract the skin surface from the camera scene, which can be computational-intensive.

To achieve normal positioning in real-time, Tsumura et al. proposed a gantry-like robot with a passive 2 degree-of-freedom (DoF) end-effector that guarantees the normal positioning of the US probe when pressed onto the body [24]–[25]. The end-effector is demonstrated to be useful in both obstetric US and lung US (LUS) examinations. During angle adjustments, there is no delay since the rotational motion is realized mechanically. However, the passive mechanism requires a wide tissue contact area and limits the flexibility of the platform. The US probe is “locked” to the normal direction by default, while the ability to actively tilt the probe to directions other than the normal angle is indispensable in some clinical practices such as liver US examinations [26]. Jiang et al. presented a RUSS that can dynamically rotate the probe to the normal direction for imaging forearm vasculature [20]. The normal positioning task is decomposed into in-plane normal positioning by centering the region with maximum confidence in the US confidence map [27], and out-of-plane normal positioning by executing a fan-shape motion and minimizing the force orthogonal to the probe’s long axis. Although this method allows an active and continuous adjustment, the normal positioning process is separated into two sequential steps, and the out-of-plane adjustment involves extra motion of the probe; hence could compromise the scanning speed. In summary, the real-time normal positioning of the probe is barely explored and is a roadblock toward a fully autonomous RUSS. A new real-time, active control method is desired.

B. Contributions

Despite the previous attempts, the effort of developing a sensing mechanism that is able to perceive the surface angle; thereby providing feedback to guide the robot motion in real-time, is missing. In this work, we aim to move one step forward to autonomous RUSS by addressing the essential problem of how to align the US probe towards the normal direction relative to the patient body in real-time. A novel robot end-effector is proposed where a US probe is encapsulated and surrounded by distance sensors that continuously detect the body surface near the probe. The required rotation adjustments towards perpendicularity can be computed based on the sensed distances from paired sensors. We mount the end-effector to a RUSS and demonstrate real-time tracking of the surface normal direction. A human-shared autonomy is implemented where the probe can be automatically landed on the designated location; the probe’s orientation and contact force are autonomously adjusted during imaging; the probe can be slid on the patient body and be rotated about its long axis via teleoperation.

Our contributions fold into two aspects: i) we propose a compact and cost-effective active-sensing end-effector (A-SEE) design that provides real-time information on the rotation adjustment required for achieving normal positioning. To the best of our knowledge, this is the first work to achieve simultaneous in-plane and out-of-plane probe orientation control on the fly without relying on a passive contact mechanism; ii) we integrate A-SEE with a RUSS (A-SEE-RUSS) and implement a complete US imaging workflow to demonstrate the A-SEE-enabled probe self-normal-positioning capability.

A. Technical Approach Overview

For Task 1), A-SEE consists of four time-of-flight (ToF) laser range finders (VL53L0X, STMicroelectronics, Switzerland) mounted to the sensor ring (see Fig. 2a). A wireless curvilinear US probe (C3HD, Clarius, Canada) is encircled by the sensors. We choose to use laser distance sensors because of the higher accuracy and angular resolution than other distance measuring apparatuses like ultrasonic sensors [29]. The sensors are interfaced with a microcontroller (MEGA2560, Arduino, Italy) which sends the measured distances to a PC (PC-1) through USB serial communication. A clamping mechanism connects A-SEE to a 7-DoF robotic manipulator (Panda, Franka Emika, Germany) together with an RGB-D camera (RealSense D435i, Intel, USA), forming the A-SEE-RUSS hardware. The subject will be captured by the camera when lying on the bed with the robot at its home configuration. The camera and the robot are connected to another PC (PC-2) which receives sensor distance measurements from PC-1 in real-time.

The US imaging procedure using A-SEE-RUSS is organized in the following way: The operator first specifies a landing position on the patient body from the camera view. The 3D pose of the landing position relative to the robot’s coordinate frame can be approximated using the depth information camera and the robot pose measurement. The robot then moves the US probe above the landing pose. A force control strategy will be activated to gradually land the probe on the patient body and keep a constant contact force during imaging (for Task 3)). Once landed on the body, A-SEE-RUSS makes sure the probe is always held orthogonal by concurrently adjusting the in-plane and out-of-plane rotation based on the 4 distance readings (for Task 2)). Meanwhile, the operator can freely change the probe’s position and rotate the probe about its axis through a 3-DoF flight joystick (PXN-2113-SE, PXN, Japan) and collect US images (for Task 4)).

B. Sensor Calibration

Since the probe orientation is adjusted based on the sensor readings, the normal positioning performance depends largely on the distance sensing accuracy of the sensors. The purpose of sensor calibration is to model and compensate for the distance sensing error so that the accuracy can be enhanced. First, we conducted an experiment to test the accuracy of each sensor, where a planar object was placed at different distances (from 50 mm to 200 mm with 10 mm intervals measured by a ruler). The sensing errors were calculated by subtracting the sensor readings from the actual distance. The 50 to 200 mm calibration range is experimentally determined to allow 0 to 60 degrees arbitrary tilting of A-SEE on a flat surface without letting the sensor distance readings exceed this range. Distance sensing beyond this range will be rejected. The results of the sensor accuracy test are shown in Fig. 2b. Black curves indicate that the sensing error changes at different sensing distances with a distinctive distance-to-error mapping for each sensor. A sensor error compensator (SEC) is designed in the form of a look-up table that stores the sensing error versus the sensed distance data. SEC linearly interpolates the sensing error given arbitrary sensor distance input. The process of reading the look-up table is described by $\text{f}:\overrightarrow{d}\in {ℝ}^4\to \overrightarrow{e}\in {ℝ}^4$, where $\overrightarrow{d}$ stores the raw sensor readings; $\overrightarrow{e}$ stores the sensing errors to be compensated. The sensor reading with SEC applied is given by:

where d_min is 50 mm, d_max is 200 mm. With SEC, we repeated the same experiment above. The magenta curves show the sensing accuracy in Fig. 2b. The mean sensing error was 11.03 ± 1.61 mm before adding SEC and 3.19 ± 1.97 mm after adding SEC. A two-tailed t-test (95% confidence level) hypothesizing no significant difference in the sensing accuracy with and without SEC was performed. A p-value of 9.72 × 10⁻⁸ suggests SEC can considerably improve the sensing accuracy. Later in this paper, the effectiveness of SEC will be further demonstrated when testing the probe normal positioning accuracy with the robot arm.

C. US Probe Self-Normal-Positioning

Having accurate distance readings from the sensors in real-time, A-SEE can be integrated with the robot to enable “spontaneous” motion that tilts the US probe towards the normal direction of the skin surface. A moving average filter is applied to the estimated distances to ensure motion smoothness. As depicted in Fig. 3, upon normal positioning of the probe, the distance differences between sensor1, and 3, sensor2, and 4 are supposed to be minimized. Thus, we address Task 2) by simultaneously applying in-plane rotation, which generates angular velocity about the y-axis of F_A–SEE (ω_ny), and out-of-plane rotation, which generates angular velocity about the x-axis of F_A–SEE (ω_nx). The angular velocities about the two axes at timestamp t are given by a PD control law:

where K_p and K_d are empirically tuned control gains; d₁₃(t) = d₃(t) − d₁(t), d₂₄(t) = d₄(t) − d₂(t); Δd₁₃ = d₁₃(t) − d₁₃(t − 1), Δd₂₄ = d₂₄(t) − d₂₄(t − 1); d₁ to d₄ are the filtered distances from sensor 1 to 4, respectively; Δt is the control interval. ω_nx and ω_ny are limited within 0.1 rad/s. The angular velocity adjustment rate can reach to 30Hz.

D. US Probe Contact Force Control

To prevent a loose contact between the probe and the skin that may cause acoustic shadows in the image, a force control strategy is necessary to stabilize the probe by pressing force at an adequate level throughout the imaging process. This control strategy is also responsible for landing the probe gently on the body for the patient’s safety. We formulate a force control strategy to adapt the linear velocity along the z-axis expressed in F_A–SEE. The velocity adaptation is described by a two-stage process that manages the landing and the scanning motion separately: during landing, the probe velocity will decrease asymptotically as it gets closer to the body surface; during scanning, the probe velocity is altered based on the deviation of the measured force from the desired value. Therefore, the velocity at time stamp t is calculated as:

where w is a constant between 0 to 1 to maintain the smoothness of the velocity profile; v is computed by:

where $\overrightarrow{d^{\prime }}$ is the vector of the four sensor readings after error compensation and filtering; ${\overline{F}}_z$ is the robot measured force along the z-axis of F_A–SEE, internally estimated from joint torque readings. It is then processed using a moving average filter; $\tilde{F}$ is the desired contact force; K_p1, K_p2 are the empirically given gains; $\tilde{d}$ is the single threshold to differentiate the landing stage from the scanning stage, which is set to be the length from the bottom of the sensor ring to the tip of the probe (120 mm).

E. Manual Probe Adjustments via Teleoperation

The combination of the self-normal-positioning and contact force control of the probe forms an autonomous pipeline that controls 3-DoF probe motion. A shared control scheme is implemented to give manual control of the translation along the x-, y-axis, and the rotation about the z-axis in concurrence with the three automated DoFs. A 3-DoF joystick is used as the input source, whose movements in the three axes are mapped to the probe’s linear velocity along the x-, y-axis (v_tx, v_ty), and angular velocity about the z-axis (ω_tz), expressed in F_A–SEE.

F. The RUSS Imaging Workflow

An A-SEE-RUSS providing 6-DoF control of the US probe is built by incorporating self-normal-positioning, contact force control, and teleoperation of the probe. A block diagram of the integrated system is found in Fig. 4. To demonstrate the proposed A-SEE-RUSS system, we define a RUSS imaging workflow containing preoperative and intraoperative steps. In the preoperative step, the patient lies on the bed next to the robot with the robot at its home configuration, allowing the RGB-D camera to capture the patient body. The operator selects a region of interest in the camera view (top right corner in Fig. 4) as the initial probe landing position. By leveraging the camera’s depth information, the landing position in 2D image space is converted to ${\text{T}}_{\mathit{\text{land}}}^{\mathit{\text{cam}}}$ representing the 3D landing pose above the patient body relative to F_cam. This is the same approach as we presented in [28]. The landing pose relative to F_base is then obtained by:

where ${\text{T}}_{A-\mathit{\text{SEE}}}^{\mathit{\text{flange}}}$ and ${\text{T}}_{\mathit{\text{cam}}}^{A-\mathit{\text{SEE}}}$ are calibrated from the CAD model of the A-SEE-RUSS. The robot then moves the probe to the landing pose using a velocity-based PD controller [28]. In the intraoperative step, the probe will be gradually attached to the skin using the landing stage force control strategy. Once the probe is in contact with the body, the operator can slide the probe on the body and rotate the probe about its long axis via the joystick. Meanwhile, commanding robot joint velocities generates probe velocities in F_A–SEE, such that the probe will be dynamically held in the normal direction and pressed with constant force. The desired probe velocities are formed as:

Transforming them to velocities expressed in F_base yields:

where ${\text{R}}_{A-\mathit{\text{SEE}}}^{\mathit{\text{base}}}\in SO(3)$ is the rotational component of ${\text{T}}_{A-\mathit{\text{SEE}}}^{\mathit{\text{base}}}\in SE(3)$; [p] is the translational component of ${\text{T}}_{A-\mathit{\text{SEE}}}^{\mathit{\text{base}}}$ represented in skew-symmetric matrix.

Lastly, the joint-space velocity command $\dot{\overrightarrow{q}}$ that will be sent to the robot for execution is obtained by:

where $J{(\overrightarrow{q})}^{†}$ is the Moore-Penrose pseudo-inverse of the robot Jacobian matrix. During the scanning, the US images are streamed and displayed to the operator. The operator decides when to terminate the procedure. The robot will move back to its home configuration after completing the scanning. Readers may refer to the video contained in the supplementary material demonstrating the complete imaging procedure.

A. Validate Self-Normal-Positioning

Two experiments were designed to evaluate the self-normal-positioning performance. The robot tracked the normal direction on a flat surface (expt. A1) and on an uneven surface (expt. A2). Expt. A1 evaluates the effectiveness of SEC and the accuracy of the self-normal-positioning under ideal cases, whereas expt. A2 assesses the feasibility of tracking the normal direction of a human body. The approach vector (z-axis) of the F_A–SEE, ${\overrightarrow{v}}_{ee}$ and the ground truth normal of the scanned surface ${\overrightarrow{v}}_s$ are recorded during experiments. The normal positioning error was used as the evaluation metric, defined as the angle between ${\overrightarrow{v}}_s$ and ${\overrightarrow{v}}_{ee}$. To estimate ${\overrightarrow{v}}_{ee}$, four reflective optical tracking markers were attached to the end-effector at the same distance from the probe tip. These markers defined a rigid body frame (RBF1 in Fig. 5a), whose pose can be tracked in real-time using a motion capture system (Vantage, Vicon Motion Systems Ltd, UK). The z-axis of RBF1 aligned with ${\overrightarrow{v}}_{ee}$. The method to estimate ${\overrightarrow{v}}_s$ varied for each experiment and shall be explained independently. In both experiments, only the rotations about the x-, and y-axis of F_A–SEE were enabled.

In expt. A1, four optical markers on the flat surface formed a rigid body frame (RBF2 in Fig. 5a) with its z-axis perpendicular to the surface, representing ${\overrightarrow{v}}_s$. An operator held the plane against the US probe and changed its rotation in four phases. During each phase, the operator altered ${\overrightarrow{v}}_s$ by tilting the plane to a certain angle. In the meantime, the robot began to track the new ${\overrightarrow{v}}_s$ using the self-normal-positioning feature. Once the robot finished adjusting the angle, the operator moved on to the next phase. The four phases were cycled three times where ${\overrightarrow{v}}_s$ and ${\overrightarrow{v}}_{ee}$ were recorded. The above process was repeated twice with SEC activated and disabled, respectively. The normal positioning errors were calculated for both cases. In the rest experiments, SEC was always turned on.

In expt. A2, we used LUS examination to test the self-normal-positioning on a human body alike surface. LUS procedure requires scanning both the anterior and lateral chests according to one of the recent clinical protocols [30]. Considering the large range of angle adjustment needed to cover the anterior to the lateral chest, LUS examination is a more challenging case than other examination types such as abdominal, cardiac, and spinal US, etc. Thus, it is suitable to verify the self-normal-positioning performance. An upper torso mannequin (Prestan CPR-AED Training Manikin, MCR Medical, USA) was used as the experiment subject. In [30], 12 targets were defined on the mannequin covering the left anterior to left lateral chest. As can be seen from Fig. 5c, targets 1, 4, 7, 10 were in the anterior region; targets 3, 6, 9, 12 were in the lateral region; targets 2, 5, 8, 11 fell in between. For each target, ${\overrightarrow{v}}_s$ can be estimated by attaching a small pad with four optical trackers to the mannequin and tracking the z-axis of the rigid body frame (RBF3 in Fig. 5c) defined by the optical trackers. We manually configured the robot to land the US probe on each target. With the self-normal-positioning enabled, the robot automatically oriented the probe towards ${\overrightarrow{v}}_s$. ${\overrightarrow{v}}_{ee}$ was recorded upon the robot finished the adjustment. This process was repeated three times at each target. The normal positioning errors were calculated accordingly.

B. Validate US Image Quality

A user study was conducted to examine the A-SEE-RUSS’s ability in acquiring good quality images. The study was approved by the institutional research ethics committee at the Worcester Polytechnic Institute (No. IRB-21–0613). Written informed consent was given to the volunteers prior to all test sessions. We recruited three volunteers with different US imaging experiences. A LUS phantom capable of mimicking patient body texture and respiratory motion (COVID-19 Live Lung Ultrasound Simulator, CAE Healthcare^™, USA) was used as the experiment subject. Realistic US images can be acquired from the phantom, which would be leveraged in the US quality assessment. Prior to the study, eight imaging targets were labeled on the phantom surface (see Fig. 6b). The phantom respiratory motion was enabled throughout the imaging acquisition. At the beginning of the study, the US probe was automatically landed near target 1. The volunteers then teleoperated the probe to the center of the target while A-SEE dynamically optimized the probe orientation. Ten images were sampled after the volunteer finished adjusting the probe position. The probe was teleoperated to the next target until all targets were covered. As the control group, each volunteer then manually placed the probe on the targets and was tilted to subjectively find the orientation giving the best image contrast. Ten images were collected from each target at the volunteers’ determined optimal viewing angle.

Since the lung pleural line is often inspected during LUS [30], we used the visibility of this landmark, quantified by the contrast-noise-ratio (CNR), as the image quality measure. Given a rectangular region of interest (ROI) on the landmark, CNR is defined as

where μ_roi and σ_roi are the mean and the standard deviation of the pixel intensities in the ROI, μ_bg and σ_bg are the mean and the standard deviation of the pixel intensities in the image background. The CNR of the pleural line was calculated for the two sets of images collected using freehand scanning and using A-SEE-RUSS, respectively. The ROI area was fixed to ten-by-ten pixels when computing the CNR.

C. Validate Contact Force Control

In the above user study, we further verified the effectiveness of the force control strategy. The force measured at the tip of the probe by the robot was recorded at 30 Hz throughout the study. The desired contact force was empirically set to be a constant of 3.5 N. We use the force control error as the quantification metric, which is calculated by subtracting the measured force from the desired force.

A. Evaludation of Self-Normal-Positioning

The normal positioning error versus time from expt. A1 is shown in Fig. 5b. The error first peaked when the user rotated the plane, then decreased rapidly as the robot aligned the probe to the normal direction. The valley after each peak in the error curve is the minimum normal positioning error achieved, which are 4.17 ± 2.24 degrees with SEC, and 12.51 ± 2.69 degrees without SEC. A two-tailed t-test (95% confidence level) was performed hypothesizing no significant difference in the errors with and without SEC. The t-test returned a p-value of 6.33×10–8. Hence SEC is proved to boost the normal positioning accuracy by rejecting the hypothesis. This accuracy is comparable with the similar-purpose work in [20]. The time to travel from one peak to the next valley in the curve can be interpreted as the response speed of the normal positioning, which was 3.67 ± 0.84 seconds with SEC, and 3.71 ± 1.50 seconds without SEC. Another t-test (95% confidence level) on the response time with and without SEC returned a p-value of 0.67. Therefore, SEC does not affect the response time of self-normal-positioning. Overall, the response speed was sufficient to keep the normal direction while the operator teleoperates the probe during experiments in III.B. and III.C. Assigning more aggressive control gains in (2) will result in a shorter time for the orientation adjustments to cope with unexpected patient movements in actual clinical situations.

Fig. 5d shows the normal positioning errors when scanning a mannequin in expt. A2. The errors when scanning the anterior targets (1, 4, 7, 10), anterior-lateral targets (2, 5, 8, 11), and lateral targets (3, 6, 9, 12) were 28.76 ± 8.84 degrees, 17.80 ± 8.46 degrees, and 8.52 ± 3.85 degrees, respectively. The accuracy when scanning the lateral region was noticeably higher than the anterior targets. As the robot moved from the neck region (targets 1, 2, 3) towards the abdominal region (targets 10, 11, 12), however, no drastic change in the errors was observed. The mean error for all 12 targets was 14.67 ± 8.46 degrees, showing a generally acceptable performance of self-normal-positioning for the anterior targets and the lateral targets.

B. Evaluation of US Image Quality

Fig. 6c–e show the CNR for all images acquired by the volunteers through different means. The mean CNR of all the manually obtained images are 3.01 ± 1.01, 3.00 ± 1.31, and 3.17 ± 1.17, respectively for each volunteer. The average CNR of all the robotically collected images are 2.78 ± 0.90, 2.81 ± 0.97, and 3.06 ± 1.07, respectively for each volunteer. The image quality is more consistent when using A-SEE-RUSS based on the reduced standard deviation in the CNR. Although the images collected using A-SEE-RUSS exhibit lower average CNR, t-tests (95% confidence level) giving p-values of 0.21, 0.28, and 0.53 indicate there is no significant difference statistically in the CNR. Therefore, A-SEE-RUSS and manual scan achieve equivalent image quality. Representative images obtained by volunteer 1 from targets 4 and 6 are shown in Fig. 7. The lung pleural lines are clearly visible in both the robotically and manually scanned images.

C. Evaluation of Contact Force Control

The evaluation of contact force control and US image quality (IV.D.) was based on the LUS phantom to reproduce the patient respiratory. Fig. 6f shows the force control errors when each volunteer was imaging the targets. Positive errors suggest less contact force than the desired value, and negative errors indicate the opposite. The force errors mostly remained within ± 1.0 N, with a mean force error being 0.07 ± 0.49 N. The high standard deviation is caused by the simulated breathing. However, the contact force controller is able to stabilize the force around the desired setpoint with the presence of the respiratory motion.