+
+ + Robotics 52 +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + ☆ UniCal: Unified Neural Sensor Calibration ECCV 2024 +
+
+
+
+
+
+
+
+ Ze Yang, George Chen, Haowei Zhang, Kevin Ta, Ioan Andrei Bârsan, Daniel Murphy, Sivabalan Manivasagam, Raquel Urtasun
+
+
+ Self-driving vehicles (SDVs) require accurate calibration of LiDARs and
+cameras to fuse sensor data accurately for autonomy. Traditional calibration
+methods typically leverage fiducials captured in a controlled and structured
+scene and compute correspondences to optimize over. These approaches are costly
+and require substantial infrastructure and operations, making it challenging to
+scale for vehicle fleets. In this work, we propose UniCal, a unified framework
+for effortlessly calibrating SDVs equipped with multiple LiDARs and cameras.
+Our approach is built upon a differentiable scene representation capable of
+rendering multi-view geometrically and photometrically consistent sensor
+observations. We jointly learn the sensor calibration and the underlying scene
+representation through differentiable volume rendering, utilizing outdoor
+sensor data without the need for specific calibration fiducials. This
+"drive-and-calibrate" approach significantly reduces costs and operational
+overhead compared to existing calibration systems, enabling efficient
+calibration for large SDV fleets at scale. To ensure geometric consistency
+across observations from different sensors, we introduce a novel surface
+alignment loss that combines feature-based registration with neural rendering.
+Comprehensive evaluations on multiple datasets demonstrate that UniCal
+outperforms or matches the accuracy of existing calibration approaches while
+being more efficient, demonstrating the value of UniCal for scalable
+calibration.
+
+
+
+ comment: ECCV 2024. Project page: https://waabi.ai/unical/
+
+
+
+ + ☆ Towards Super-Nominal Payload Handling: Inverse Dynamics Analysis for + Multi-Skill Robotic Manipulation ICRA +
+ +
+ Motion planning for articulated robots has traditionally been governed by
+algorithms that operate within manufacturer-defined payload limits. Our
+empirical analysis of the Franka Emika Panda robot demonstrates that this
+approach unnecessarily restricts the robot's dynamically-reachable task space.
+These results establish an expanded operational envelope for such robots,
+showing that they can handle payloads of more than twice their rated capacity.
+Additionally, our preliminary findings indicate that integrating non-prehensile
+motion primitives with grasping-based manipulation has the potential to further
+increase the success rates of manipulation tasks involving payloads exceeding
+nominal limits.
+
+
+
+ comment: Accepted as an extended abstract to ICRA@40
+
+
+
+ + ☆ Safe Decentralized Multi-Agent Control using Black-Box Predictors, + Conformal Decision Policies, and Control Barrier Functions ICRA 2025 +
+ +
+ We address the challenge of safe control in decentralized multi-agent robotic
+settings, where agents use uncertain black-box models to predict other agents'
+trajectories. We use the recently proposed conformal decision theory to adapt
+the restrictiveness of control barrier functions-based safety constraints based
+on observed prediction errors. We use these constraints to synthesize
+controllers that balance between the objectives of safety and task
+accomplishment, despite the prediction errors. We provide an upper bound on the
+average over time of the value of a monotonic function of the difference
+between the safety constraint based on the predicted trajectories and the
+constraint based on the ground truth ones. We validate our theory through
+experimental results showing the performance of our controllers when navigating
+a robot in the multi-agent scenes in the Stanford Drone Dataset.
+
+
+
+ comment: 6 pages, 1 figure, submitted for ICRA 2025
+
+
+
+ + ☆ Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in + Continuous Environment with Open-Source LLMs +
+
+
+
+
+
+
+
+ Yanyuan Qiao, Wenqi Lyu, Hui Wang, Zixu Wang, Zerui Li, Yuan Zhang, Mingkui Tan, Qi Wu
+
+
+ Vision-and-Language Navigation (VLN) tasks require an agent to follow textual
+instructions to navigate through 3D environments. Traditional approaches use
+supervised learning methods, relying heavily on domain-specific datasets to
+train VLN models. Recent methods try to utilize closed-source large language
+models (LLMs) like GPT-4 to solve VLN tasks in zero-shot manners, but face
+challenges related to expensive token costs and potential data breaches in
+real-world applications. In this work, we introduce Open-Nav, a novel study
+that explores open-source LLMs for zero-shot VLN in the continuous environment.
+Open-Nav employs a spatial-temporal chain-of-thought (CoT) reasoning approach
+to break down tasks into instruction comprehension, progress estimation, and
+decision-making. It enhances scene perceptions with fine-grained object and
+spatial knowledge to improve LLM's reasoning in navigation. Our extensive
+experiments in both simulated and real-world environments demonstrate that
+Open-Nav achieves competitive performance compared to using closed-source LLMs.
+
+
+
+
+ + ☆ Excavating in the Wild: The GOOSE-Ex Dataset for Semantic Segmentation +
+ +
+ The successful deployment of deep learning-based techniques for autonomous
+systems is highly dependent on the data availability for the respective system
+in its deployment environment. Especially for unstructured outdoor
+environments, very few datasets exist for even fewer robotic platforms and
+scenarios. In an earlier work, we presented the German Outdoor and Offroad
+Dataset (GOOSE) framework along with 10000 multimodal frames from an offroad
+vehicle to enhance the perception capabilities in unstructured environments. In
+this work, we address the generalizability of the GOOSE framework. To
+accomplish this, we open-source the GOOSE-Ex dataset, which contains additional
+5000 labeled multimodal frames from various completely different environments,
+recorded on a robotic excavator and a quadruped platform. We perform a
+comprehensive analysis of the semantic segmentation performance on different
+platforms and sensor modalities in unseen environments. In addition, we
+demonstrate how the combined datasets can be utilized for different downstream
+applications or competitions such as offroad navigation, object manipulation or
+scene completion. The dataset, its platform documentation and pre-trained
+state-of-the-art models for offroad perception will be made available on
+https://goose-dataset.de/.
+ \
+
+
+
+ comment: Submitted to IEEE for review
+
+
+
+ + ☆ A POMDP-based hierarchical planning framework for manipulation under + pose uncertainty +
+ +
+ Robots often face challenges in domestic environments where visual feedback
+is ineffective, such as retrieving objects obstructed by occlusions or finding
+a light switch in the dark. In these cases, utilizing contacts to localize the
+target object can be effective. We propose an online planning framework using
+binary contact signals for manipulation tasks with pose uncertainty, formulated
+as a Partially Observable Markov Decision Process (POMDP). Naively representing
+the belief as a particle set makes planning infeasible due to the large
+uncertainties in domestic settings, as identifying the best sequence of actions
+requires rolling out thousands of actions across millions of particles, taking
+significant compute time. To address this, we propose a hierarchical belief
+representation. Initially, we represent the uncertainty coarsely in a 3D
+volumetric space. Policies that refine uncertainty in this space are computed
+and executed, and once uncertainty is sufficiently reduced, the problem is
+translated back into the particle space for further refinement before task
+completion. We utilize a closed-loop planning and execution framework with a
+heuristic-search-based anytime solver that computes partial policies within a
+limited time budget. The performance of the framework is demonstrated both in
+real world and in simulation on the high-precision task of inserting a plug
+into a port using a UR10e manipulator, resolving positional uncertainties up to
+50 centimeters and angular uncertainties close to $2\pi$. Experimental results
+highlight the framework's effectiveness, achieving a 93\% success rate in the
+real world and over 50\% improvement in solution quality compared to greedy
+baselines, significantly accelerating planning and enabling real-time solutions
+for complex problems.
+
+
+
+ comment: Under review (2025 IEEE International Conference on Robotics &
+ Automation)
+
+
+
+ + ☆ Learning from Demonstration with Implicit Nonlinear Dynamics Models +
+ +
+ Learning from Demonstration (LfD) is a useful paradigm for training policies
+that solve tasks involving complex motions. In practice, the successful
+application of LfD requires overcoming error accumulation during policy
+execution, i.e. the problem of drift due to errors compounding over time and
+the consequent out-of-distribution behaviours. Existing works seek to address
+this problem through scaling data collection, correcting policy errors with a
+human-in-the-loop, temporally ensembling policy predictions or through learning
+the parameters of a dynamical system model. In this work, we propose and
+validate an alternative approach to overcoming this issue. Inspired by
+reservoir computing, we develop a novel neural network layer that includes a
+fixed nonlinear dynamical system with tunable dynamical properties. We validate
+the efficacy of our neural network layer on the task of reproducing human
+handwriting motions using the LASA Human Handwriting Dataset. Through empirical
+experiments we demonstrate that incorporating our layer into existing neural
+network architectures addresses the issue of compounding errors in LfD.
+Furthermore, we perform a comparative evaluation against existing approaches
+including a temporal ensemble of policy predictions and an Echo State Networks
+(ESNs) implementation. We find that our approach yields greater policy
+precision and robustness on the handwriting task while also generalising to
+multiple dynamics regimes and maintaining competitive latency scores.
+
+
+
+ comment: 21 pages, 9 figures
+
+
+
+ + ☆ Transparency evaluation for the Kinematic Design of the Harnesses + through Human-Exoskeleton Interaction Modeling +
+ +
+ Lower Limb Exoskeletons (LLEs) are wearable robots that provide mechanical
+power to the user. Human-exoskeleton (HE) connections must preserve the user's
+natural behavior during the interaction, avoiding undesired forces. Therefore,
+numerous works focus on their minimization. Given the inherent complications of
+repeatedly prototyping and experimentally testing a device, modeling the
+exoskeleton and its physical interaction with the user emerges as a valuable
+approach for assessing the design effects. This paper proposes a novel method
+to compare different exoskeleton configurations with a flexible simulation
+tool. This approach contemplates simulating the dynamics of the device,
+including its interaction with the wearer, to evaluate multiple connection
+mechanism designs along with the kinematics and actuation of the LLE. This
+evaluation is based on the minimization of the interaction wrenches through an
+optimization process that includes the impedance parameters at the interfaces
+as optimization variables and the similarity of the LLE's joint variables
+trajectories with the motion of the wearer's articulations. Exploratory tests
+are conducted using the Wearable Walker LLE in different configurations and
+measuring the interaction forces. Experimental data are then compared to the
+optimization outcomes, proving that the proposed method provides contact wrench
+estimations consistent with the collected measurements and previous outcomes
+from the literature. Copyright 2024 IEEE. Personal use of this material is
+permitted. Permission from IEEE must be obtained for all other uses, in any
+current or future media, including reprinting/republishing this material for
+advertising or promotional purposes, creating new collective works, for resale
+or redistribution to servers or lists, or reuse of any copyrighted component of
+this work in other works.
+
+
+
+
+ + ☆ Royal Reveals: LiDAR Mapping of Kronborg Castle, Echoes of Hamlet's + Halls +
+ +
+ This paper presents a large scale dataset from a meticulous 360-degree LiDAR
+(Light Detection and Ranging) scan conducted on Kronborg Castle, a renowned
+Renaissance fortress located in Elsinore (Helsing{\o}r), Denmark, famously
+associated with Shakespeare's "Hamlet." Utilising a vertical mounted, gimbal
+stabilised, 16 channel, 360-degree Velodyne VLP-16 LiDAR scanner, paired with
+an Intel RealSense L515 depth camera. This research offers an unparalleled
+digital representation of the castle's intricate architectural details and
+structural nuances, enabling fellow researchers to conduct experiments
+utilising the data for SLAM (Simultaneous Localisation and Mapping) as well as
+floorplan generation.
+
+
+
+ comment: 4 pages, 4 figures, 3 tables
+
+
+
+ + ☆ A study on the effects of mixed explicit and implicit communications in + human-virtual-agent interactions +
+ +
+ Communication between humans and robots (or virtual agents) is essential for
+interaction and often inspired by human communication, which uses gestures,
+facial expressions, gaze direction, and other explicit and implicit means. This
+work presents an interaction experiment where humans and virtual agents
+interact through explicit (gestures, manual entries using mouse and keyboard,
+voice, sound, and information on screen) and implicit (gaze direction,
+location, facial expressions, and raise of eyebrows) communication to evaluate
+the effect of mixed explicit-implicit communication against purely explicit
+communication. Results obtained using Bayesian parameter estimation show that
+the number of errors and task execution time did not significantly change when
+mixed explicit and implicit communications were used, and neither the perceived
+efficiency of the interaction. In contrast, acceptance, sociability, and
+transparency of the virtual agent increased when using mixed communication
+modalities (88.3%, 92%, and 92.9% of the effect size posterior distribution of
+each variable, respectively, were above the upper limit of the region of
+practical equivalence). This suggests that task-related measures, such as time,
+number of errors, and perceived efficiency of the interaction, have not been
+influenced by the communication type in our particular experiment. However, the
+improvement of subjective measures related to the virtual agent, such as
+acceptance, sociability, and transparency, suggests that humans are more
+receptive to mixed explicit and implicit communications.
+
+
+
+ comment: 22 pages, 12 figures, 4 tables. Under review for International
+ Journal of Social Robotics
+
+
+
+ + ☆ OpenObject-NAV: Open-Vocabulary Object-Oriented Navigation Based on + Dynamic Carrier-Relationship Scene Graph +
+ +
+ In everyday life, frequently used objects like cups often have unfixed
+positions and multiple instances within the same category, and their carriers
+frequently change as well. As a result, it becomes challenging for a robot to
+efficiently navigate to a specific instance. To tackle this challenge, the
+robot must capture and update scene changes and plans continuously. However,
+current object navigation approaches primarily focus on semantic-level and lack
+the ability to dynamically update scene representation. This paper captures the
+relationships between frequently used objects and their static carriers. It
+constructs an open-vocabulary Carrier-Relationship Scene Graph (CRSG) and
+updates the carrying status during robot navigation to reflect the dynamic
+changes of the scene. Based on the CRSG, we further propose an instance
+navigation strategy that models the navigation process as a Markov Decision
+Process. At each step, decisions are informed by Large Language Model's
+commonsense knowledge and visual-language feature similarity. We designed a
+series of long-sequence navigation tasks for frequently used everyday items in
+the Habitat simulator. The results demonstrate that by updating the CRSG, the
+robot can efficiently navigate to moved targets. Additionally, we deployed our
+algorithm on a real robot and validated its practical effectiveness.
+
+
+
+ comment: Project website: https://openobject-nav.github.io/
+
+
+
+ + ☆ Optimum Configuration for Hovering n-Quadrotors carrying a Slung Payload +
+
+
+
+
+
+
+
+ Mohssen E. Elshaar, Pansie A. khodary, Meral L. Badr, Mohamad A. Sayegh, Zeyad M. Manaa, Ayman M. Abdallah
+
+
+ This work proposes a strategy for organising quadrotors around a payload to
+enable hovering without external stimuli, together with a MATLAB software for
+modelling the dynamics of a quadrotor-payload system. Based on geometric
+concepts, the proposed design keeps the payload and system centre of mass
+aligned. Hovering tests that are successful confirm the method's efficiency.
+Moreover, the algorithm is improved to take thrust capacities and propeller
+distances into account, calculating the minimum number of quadrotors needed for
+hovering. The algorithm's effectiveness is demonstrated by numerical examples,
+which reveal that larger quadrotors may require fewer units while smaller ones
+give greater flexibility. Our code can be found at:
+\href{https://github.com/Hosnooo/Swarm-Slung-Payload}{https://github.com/Hosnooo/Swarm-Slung-Payload}
+
+
+
+ comment: accepted for publication at AIAA SCITECH 2025
+
+
+
+ + ☆ Discrete Policy: Learning Disentangled Action Space for Multi-Task + Robotic Manipulation +
+
+
+
+
+
+
+
+ Kun Wu, Yichen Zhu, Jinming Li, Junjie Wen, Ning Liu, Zhiyuan Xu, Qinru Qiu, Jian Tang
+
+
+ Learning visuomotor policy for multi-task robotic manipulation has been a
+long-standing challenge for the robotics community. The difficulty lies in the
+diversity of action space: typically, a goal can be accomplished in multiple
+ways, resulting in a multimodal action distribution for a single task. The
+complexity of action distribution escalates as the number of tasks increases.
+In this work, we propose \textbf{Discrete Policy}, a robot learning method for
+training universal agents capable of multi-task manipulation skills. Discrete
+Policy employs vector quantization to map action sequences into a discrete
+latent space, facilitating the learning of task-specific codes. These codes are
+then reconstructed into the action space conditioned on observations and
+language instruction. We evaluate our method on both simulation and multiple
+real-world embodiments, including both single-arm and bimanual robot settings.
+We demonstrate that our proposed Discrete Policy outperforms a well-established
+Diffusion Policy baseline and many state-of-the-art approaches, including ACT,
+Octo, and OpenVLA. For example, in a real-world multi-task training setting
+with five tasks, Discrete Policy achieves an average success rate that is 26\%
+higher than Diffusion Policy and 15\% higher than OpenVLA. As the number of
+tasks increases to 12, the performance gap between Discrete Policy and
+Diffusion Policy widens to 32.5\%, further showcasing the advantages of our
+approach. Our work empirically demonstrates that learning multi-task policies
+within the latent space is a vital step toward achieving general-purpose
+agents.
+
+
+
+
+ + ☆ Automatic Gain Tuning for Humanoid Robots Walking Architectures Using + Gradient-Free Optimization Techniques +
+
+
+
+
+
+
+
+ Carlotta Sartore, Marco Rando, Giulio Romualdi, Cesare Molinari, Lorenzo Rosasco, Daniele Pucci
+
+
+ Developing sophisticated control architectures has endowed robots,
+particularly humanoid robots, with numerous capabilities. However, tuning these
+architectures remains a challenging and time-consuming task that requires
+expert intervention. In this work, we propose a methodology to automatically
+tune the gains of all layers of a hierarchical control architecture for walking
+humanoids. We tested our methodology by employing different gradient-free
+optimization methods: Genetic Algorithm (GA), Covariance Matrix Adaptation
+Evolution Strategy (CMA-ES), Evolution Strategy (ES), and Differential
+Evolution (DE). We validated the parameter found both in simulation and on the
+real ergoCub humanoid robot. Our results show that GA achieves the fastest
+convergence (10 x 10^3 function evaluations vs 25 x 10^3 needed by the other
+algorithms) and 100% success rate in completing the task both in simulation and
+when transferred on the real robotic platform. These findings highlight the
+potential of our proposed method to automate the tuning process, reducing the
+need for manual intervention.
+
+
+
+
+ + ☆ Pseudo-kinematic trajectory control of tracked vehicles +
+ +
+ Tracked vehicles are used in complex scenarios, where motion planning and
+navigation can be very complex. They have complex dynamics, with many
+parameters that are difficult to identify and that change significantly based
+on the operating conditions. We propose a simple pseudo-kinematic model, where
+the intricate dynamic effects underlying the vehicle's motion are captured in a
+small set of velocity-dependent parameters. This choice enables the development
+of a Lyapunov-based trajectory controller with guaranteed performance and small
+computation time. We demonstrate the correctness of our approach with both
+simulation and experimental data.
+
+
+
+
+ + ☆ From One to the Power of Many: Augmentations for Invariance to + Multi-LiDAR Perception from Single-Sensor Datasets +
+ +
+ Recently, LiDAR perception methods for autonomous vehicles, powered by deep
+neural networks have experienced steep growth in performance on classic
+benchmarks, such as nuScenes and SemanticKITTI. However, there are still large
+gaps in performance when deploying models trained on such single-sensor setups
+to modern multi-sensor vehicles. In this work, we investigate if a lack of
+invariance may be responsible for these performance gaps, and propose some
+initial solutions in the form of application-specific data augmentations, which
+can facilitate better transfer to multi-sensor LiDAR setups. We provide
+experimental evidence that our proposed augmentations improve generalization
+across LiDAR sensor setups, and investigate how these augmentations affect the
+models' invariance properties on simulations of different LiDAR sensor setups.
+
+
+
+
+ + ☆ Analysis of Truncated Singular Value Decomposition for Koopman + Operator-Based Lane Change Model +
+ +
+ Understanding and modeling complex dynamic systems is crucial for enhancing
+vehicle performance and safety, especially in the context of autonomous
+driving. Recently, popular methods such as Koopman operators and their
+approximators, known as Extended Dynamic Mode Decomposition (EDMD), have
+emerged for their effectiveness in transforming strongly nonlinear system
+behavior into linear representations. This allows them to be integrated with
+conventional linear controllers. To achieve this, Singular Value Decomposition
+(SVD), specifically truncated SVD, is employed to approximate Koopman operators
+from extensive datasets efficiently. This study evaluates different basis
+functions used in EDMD and ranks for truncated SVD for representing lane change
+behavior models, aiming to balance computational efficiency with information
+loss. The findings, however, suggest that the technique of truncated SVD does
+not necessarily achieve substantial reductions in computational training time
+and results in significant information loss.
+
+
+
+ comment: Submitted to the 21st International Conference on Informatics in
+ Control, Automation and Robotics (ICINCO 2024)
+
+
+
+ + ☆ Unscented Transform-based Pure Pursuit Path-Tracking Algorithm under + Uncertainty +
+ +
+ Automated driving has become more and more popular due to its potential to
+eliminate road accidents by taking over driving tasks from humans. One of the
+remaining challenges is to follow a planned path autonomously, especially when
+uncertainties in self-localizing or understanding the surroundings can
+influence the decisions made by autonomous vehicles, such as calculating how
+much they need to steer to minimize tracking errors. In this paper, a modified
+geometric pure pursuit path-tracking algorithm is proposed, taking into
+consideration such uncertainties using the unscented transform. The algorithm
+is tested through simulations for typical road geometries, such as straight and
+circular lines.
+
+
+
+ comment: Submitted to the 21st International Conference on Informatics in
+ Control, Automation and Robotics (ICINCO 2024)
+
+
+
+ + ☆ An Epistemic Human-Aware Task Planner which Anticipates Human Beliefs + and Decisions +
+ +
+ We present a substantial extension of our Human-Aware Task Planning
+framework, tailored for scenarios with intermittent shared execution
+experiences and significant belief divergence between humans and robots,
+particularly due to the uncontrollable nature of humans. Our objective is to
+build a robot policy that accounts for uncontrollable human behaviors, thus
+enabling the anticipation of possible advancements achieved by the robot when
+the execution is not shared, e.g. when humans are briefly absent from the
+shared environment to complete a subtask. But, this anticipation is considered
+from the perspective of humans who have access to an estimated model for the
+robot. To this end, we propose a novel planning framework and build a solver
+based on AND-OR search, which integrates knowledge reasoning, including
+situation assessment by perspective taking. Our approach dynamically models and
+manages the expansion and contraction of potential advances while precisely
+keeping track of when (and when not) agents share the task execution
+experience. The planner systematically assesses the situation and ignores
+worlds that it has reason to think are impossible for humans. Overall, our new
+solver can estimate the distinct beliefs of the human and the robot along
+potential courses of action, enabling the synthesis of plans where the robot
+selects the right moment for communication, i.e. informing, or replying to an
+inquiry, or defers ontic actions until the execution experiences can be shared.
+Preliminary experiments in two domains, one novel and one adapted, demonstrate
+the effectiveness of the framework.
+
+
+
+ comment: 15 pages, 4 figures, 1 table
+
+
+
+ + ☆ DynaWeightPnP: Toward global real-time 3D-2D solver in PnP without + correspondences +
+ +
+ This paper addresses a special Perspective-n-Point (PnP) problem: estimating
+the optimal pose to align 3D and 2D shapes in real-time without
+correspondences, termed as correspondence-free PnP. While several studies have
+focused on 3D and 2D shape registration, achieving both real-time and accurate
+performance remains challenging. This study specifically targets the 3D-2D
+geometric shape registration tasks, applying the recently developed Reproducing
+Kernel Hilbert Space (RKHS) to address the "big-to-small" issue. An iterative
+reweighted least squares method is employed to solve the RKHS-based formulation
+efficiently. Moreover, our work identifies a unique and interesting
+observability issue in correspondence-free PnP: the numerical ambiguity between
+rotation and translation. To address this, we proposed DynaWeightPnP,
+introducing a dynamic weighting sub-problem and an alternative searching
+algorithm designed to enhance pose estimation and alignment accuracy.
+Experiments were conducted on a typical case, that is, a 3D-2D vascular
+centerline registration task within Endovascular Image-Guided Interventions
+(EIGIs). Results demonstrated that the proposed algorithm achieves registration
+processing rates of 60 Hz (without post-refinement) and 31 Hz (with
+post-refinement) on modern single-core CPUs, with competitive accuracy
+comparable to existing methods. These results underscore the suitability of
+DynaWeightPnP for future robot navigation tasks like EIGIs.
+
+
+
+
+ + ☆ Exploiting Physical Human-Robot Interaction to Provide a Unique Rolling + Experience with a Riding Ballbot +
+
+
+
+
+
+
+
+ Chenzhang Xiao, Seung Yun Song, Yu Chen, Mahshid Mansouri, João Ramos, Adam W. Bleakney, William R. Norris, Elizabeth T. Hsiao-Wecksler
+
+
+ This study introduces the development of hands-free control schemes for a
+riding ballbot, designed to allow riders including manual wheelchair users to
+control its movement through torso leaning and twisting. The hardware platform,
+Personal Unique Rolling Experience (PURE), utilizes a ballbot drivetrain, a
+dynamically stable mobile robot that uses a ball as its wheel to provide
+omnidirectional maneuverability. To accommodate users with varying torso motion
+functions, the hanads-free control scheme should be adjustable based on the
+rider's torso function and personal preferences. Therefore, concepts of (a)
+impedance control and (b) admittance control were integrated into the control
+scheme. A duo-agent optimization framework was utilized to assess the
+efficiency of this rider-ballbot system for a safety-critical task: braking
+from 1.4 m/s. The candidate control schemes were further implemented in the
+physical robot hardware and validated with two experienced users, demonstrating
+the efficiency and robustness of the hands-free admittance control scheme
+(HACS). This interface, which utilized physical human-robot interaction (pHRI)
+as the input, resulted in lower braking effort and shorter braking distance and
+time. Subsequently, 12 novice participants (six able-bodied users and six
+manual wheelchair users) with different levels of torso motion capability were
+then recruited to benchmark the braking performance with HACS. The indoor
+navigation capability of PURE was further demonstrated with these participants
+in courses simulating narrow hallways, tight turns, and navigation through
+static and dynamic obstacles. By exploiting pHRI, the proposed admittance-style
+control scheme provided effective control of the ballbot via torso motions.
+This interface enables PURE to provide a personal unique rolling experience to
+manual wheelchair users for safe and agile indoor navigation.
+
+
+
+
+ + ☆ Get It For Free: Radar Segmentation without Expert Labels and Its + Application in Odometry and Localization +
+ +
+ This paper presents a novel weakly supervised semantic segmentation method
+for radar segmentation, where the existing LiDAR semantic segmentation models
+are employed to generate semantic labels, which then serve as supervision
+signals for training a radar semantic segmentation model. The obtained radar
+semantic segmentation model outperforms LiDAR-based models, providing more
+consistent and robust segmentation under all-weather conditions, particularly
+in the snow, rain and fog. To mitigate potential errors in LiDAR semantic
+labels, we design a dedicated refinement scheme that corrects erroneous labels
+based on structural features and distribution patterns. The semantic
+information generated by our radar segmentation model is used in two downstream
+tasks, achieving significant performance improvements. In large-scale
+radar-based localization using OpenStreetMap, it leads to localization error
+reduction by 20.55\% over prior methods. For the odometry task, it improves
+translation accuracy by 16.4\% compared to the second-best method, securing the
+first place in the radar odometry competition at the Radar in Robotics workshop
+of ICRA 2024, Japan
+
+
+
+
+ + ☆ BoT-Drive: Hierarchical Behavior and Trajectory Planning for Autonomous + Driving using POMDPs +
+ +
+ Uncertainties in dynamic road environments pose significant challenges for
+behavior and trajectory planning in autonomous driving. This paper introduces
+BoT-Drive, a planning algorithm that addresses uncertainties at both behavior
+and trajectory levels within a Partially Observable Markov Decision Process
+(POMDP) framework. BoT-Drive employs driver models to characterize unknown
+behavioral intentions and utilizes their model parameters to infer hidden
+driving styles. By also treating driver models as decision-making actions for
+the autonomous vehicle, BoT-Drive effectively tackles the exponential
+complexity inherent in POMDPs. To enhance safety and robustness, the planner
+further applies importance sampling to refine the driving trajectory
+conditioned on the planned high-level behavior. Evaluation on real-world data
+shows that BoT-Drive consistently outperforms both existing planning methods
+and learning-based methods in regular and complex urban driving scenes,
+demonstrating significant improvements in driving safety and reliability.
+
+
+
+
+ + ☆ Word2Wave: Language Driven Mission Programming for Efficient Subsea + Deployments of Marine Robots +
+ +
+ This paper explores the design and development of a language-based interface
+for dynamic mission programming of autonomous underwater vehicles (AUVs). The
+proposed 'Word2Wave' (W2W) framework enables interactive programming and
+parameter configuration of AUVs for remote subsea missions. The W2W framework
+includes: (i) a set of novel language rules and command structures for
+efficient language-to-mission mapping; (ii) a GPT-based prompt engineering
+module for training data generation; (iii) a small language model (SLM)-based
+sequence-to-sequence learning pipeline for mission command generation from
+human speech or text; and (iv) a novel user interface for 2D mission map
+visualization and human-machine interfacing. The proposed learning pipeline
+adapts an SLM named T5-Small that can learn language-to-mission mapping from
+processed language data effectively, providing robust and efficient
+performance. In addition to a benchmark evaluation with state-of-the-art, we
+conduct a user interaction study to demonstrate the effectiveness of W2W over
+commercial AUV programming interfaces. Across participants, W2W-based
+programming required less than 10% time for mission programming compared to
+traditional interfaces; it is deemed to be a simpler and more natural paradigm
+for subsea mission programming with a usability score of 76.25. W2W opens up
+promising future research opportunities on hands-free AUV mission programming
+for efficient subsea deployments.
+
+
+
+
+ + ☆ An Augmented Reality Interface for Teleoperating Robot Manipulators: + Reducing Demonstrator Task Load through Digital Twin Control +
+ +
+ Acquiring high-quality demonstration data is essential for the success of
+data-driven methods, such as imitation learning. Existing platforms for
+providing demonstrations for manipulation tasks often impose significant
+physical and mental demands on the demonstrator, require additional hardware
+systems, or necessitate specialized domain knowledge. In this work, we present
+a novel augmented reality (AR) interface for teleoperating robotic
+manipulators, emphasizing the demonstrator's experience, particularly in the
+context of performing complex tasks that require precision and accuracy. This
+interface, designed for the Microsoft HoloLens 2, leverages the adaptable
+nature of mixed reality (MR), enabling users to control a physical robot
+through digital twin surrogates. We assess the effectiveness of our approach
+across three complex manipulation tasks and compare its performance against
+OPEN TEACH, a recent virtual reality (VR) teleoperation system, as well as two
+traditional control methods: kinesthetic teaching and a 3D SpaceMouse for
+end-effector control. Our findings show that our method performs comparably to
+the VR approach and demonstrates the potential for AR in data collection.
+Additionally, we conduct a pilot study to evaluate the usability and task load
+associated with each method. Results indicate that our AR-based system achieves
+higher usability scores than the VR benchmark and significantly reduces mental
+demand, physical effort, and frustration experienced by users. An accompanying
+video can be found at https://youtu.be/w-M58ohPgrA.
+
+
+
+ comment: 6 pages, 4 figures
+
+
+
+ + ☆ Speech to Reality: On-Demand Production using Natural Language, 3D + Generative AI, and Discrete Robotic Assembly +
+ +
+ We present a system that transforms speech into physical objects by combining
+3D generative Artificial Intelligence with robotic assembly. The system
+leverages natural language input to make design and manufacturing more
+accessible, enabling individuals without expertise in 3D modeling or robotic
+programming to create physical objects. We propose utilizing discrete robotic
+assembly of lattice-based voxel components to address the challenges of using
+generative AI outputs in physical production, such as design variability,
+fabrication speed, structural integrity, and material waste. The system
+interprets speech to generate 3D objects, discretizes them into voxel
+components, computes an optimized assembly sequence, and generates a robotic
+toolpath. The results are demonstrated through the assembly of various objects,
+ranging from chairs to shelves, which are prompted via speech and realized
+within 5 minutes using a 6-axis robotic arm.
+
+
+
+ comment: This work has been submitted to the IEEE for possible publication.
+ Copyright may be transferred without notice, after which this version may no
+ longer be accessible. An updated version will replace this version
+
+
+
+ + ☆ Robo-CSK-Organizer: Commonsense Knowledge to Organize Detected Objects + for Multipurpose Robots +
+ +
+ This paper presents a system called Robo-CSK-Organizer that infuses
+commonsense knowledge from a classical knowledge based to enhance the context
+recognition capabilities of robots so as to facilitate the organization of
+detected objects by classifying them in a task-relevant manner. It is
+particularly useful in multipurpose robotics. Unlike systems relying solely on
+deep learning tools such as ChatGPT, the Robo-CSK-Organizer system stands out
+in multiple avenues as follows. It resolves ambiguities well, and maintains
+consistency in object placement. Moreover, it adapts to diverse task-based
+classifications. Furthermore, it contributes to explainable AI, hence helping
+to improve trust and human-robot collaboration. Controlled experiments
+performed in our work, simulating domestic robotics settings, make
+Robo-CSK-Organizer demonstrate superior performance while placing objects in
+contextually relevant locations. This work highlights the capacity of an
+AI-based system to conduct commonsense-guided decision-making in robotics
+closer to the thresholds of human cognition. Hence, Robo-CSK-Organizer makes
+positive impacts on AI and robotics.
+
+
+
+
+ + ☆ AquaMILR+: Design of an untethered limbless robot for complex aquatic + terrain navigation +
+
+
+
+
+
+
+
+ Matthew Fernandez, Tianyu Wang, Galen Tunnicliffe, Donoven Dortilus, Peter Gunnarson, John O. Dabiri, Daniel I. Goldman
+
+
+ This paper presents AquaMILR+, an untethered limbless robot designed for
+agile navigation in complex aquatic environments. The robot features a
+bilateral actuation mechanism that models musculoskeletal actuation in many
+anguilliform swimming organisms which propagates a moving wave from head to
+tail allowing open fluid undulatory swimming. This actuation mechanism employs
+mechanical intelligence, enhancing the robot's maneuverability when interacting
+with obstacles. AquaMILR+ also includes a compact depth control system inspired
+by the swim bladder and lung structures of eels and sea snakes. The mechanism,
+driven by a syringe and telescoping leadscrew, enables depth and pitch
+control-capabilities that are difficult for most anguilliform swimming robots
+to achieve. Additional structures, such as fins and a tail, further improve
+stability and propulsion efficiency. Our tests in both open water and indoor 2D
+and 3D heterogeneous aquatic environments highlight AquaMILR+'s capabilities
+and suggest a promising system for complex underwater tasks such as search and
+rescue and deep-sea exploration.
+
+
+
+
+ + ☆ CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot + Skills using Large Language Models ICRA 2025 +
+ +
+ Curriculum learning is a training mechanism in reinforcement learning (RL)
+that facilitates the achievement of complex policies by progressively
+increasing the task difficulty during training. However, designing effective
+curricula for a specific task often requires extensive domain knowledge and
+human intervention, which limits its applicability across various domains. Our
+core idea is that large language models (LLMs), with their extensive training
+on diverse language data and ability to encapsulate world knowledge, present
+significant potential for efficiently breaking down tasks and decomposing
+skills across various robotics environments. Additionally, the demonstrated
+success of LLMs in translating natural language into executable code for RL
+agents strengthens their role in generating task curricula. In this work, we
+propose CurricuLLM, which leverages the high-level planning and programming
+capabilities of LLMs for curriculum design, thereby enhancing the efficient
+learning of complex target tasks. CurricuLLM consists of: (Step 1) Generating
+sequence of subtasks that aid target task learning in natural language form,
+(Step 2) Translating natural language description of subtasks in executable
+task code, including the reward code and goal distribution code, and (Step 3)
+Evaluating trained policies based on trajectory rollout and subtask
+description. We evaluate CurricuLLM in various robotics simulation
+environments, ranging from manipulation, navigation, and locomotion, to show
+that CurricuLLM can aid learning complex robot control tasks. In addition, we
+validate humanoid locomotion policy learned through CurricuLLM in real-world.
+The code is provided in https://github.com/labicon/CurricuLLM
+
+
+
+ comment: Submitted to ICRA 2025
+
+
+
+ + ☆ iWalker: Imperative Visual Planning for Walking Humanoid Robot +
+ +
+ Humanoid robots, with the potential to perform a broad range of tasks in
+environments designed for humans, have been deemed crucial for the basis of
+general AI agents. When talking about planning and controlling, although
+traditional models and task-specific methods have been extensively studied over
+the past few decades, they are inadequate for achieving the flexibility and
+versatility needed for general autonomy. Learning approaches, especially
+reinforcement learning, are powerful and popular nowadays, but they are
+inherently "blind" during training, relying heavily on trials in simulation
+without proper guidance from physical principles or underlying dynamics. In
+response, we propose a novel end-to-end pipeline that seamlessly integrates
+perception, planning, and model-based control for humanoid robot walking. We
+refer to our method as iWalker, which is driven by imperative learning (IL), a
+self-supervising neuro-symbolic learning framework. This enables the robot to
+learn from arbitrary unlabeled data, significantly improving its adaptability
+and generalization capabilities. In experiments, iWalker demonstrates
+effectiveness in both simulated and real-world environments, representing a
+significant advancement toward versatile and autonomous humanoid robots.
+
+
+
+
+ + ☆ A New 10-mg SMA-Based Fast Bimorph Actuator for Microrobotics IROS 2024 +
+ +
+ We present a new millimeter-scale bimorph actuator for microrobotic
+applications, driven by feedforward controlled shape-memory alloy (SMA) wires.
+The device weighs 10 mg, measures 14 mm in length, and occupies a volume of 4.8
+mm3, which makes it the lightest and smallest fully functional SMA-based
+bimorph actuator for microrobotics developed to date. The experimentally
+measured operational bandwidth is on the order of 20 Hz, and the unimorph and
+bimorph maximum low-frequency displacement outputs are on the order of 3.5 and
+7 mm, respectively. To test and demonstrate the functionality and suitability
+of the actuator for microrobotics, we developed the Fish-&-Ribbon-Inspired
+Small Swimming Harmonic roBot (FRISSHBot). Loosely inspired by carangiformes,
+the FRISSHBot leverages fluid-structure interaction (FSI) phenomena to propel
+itself forward, weighs 30 mg, measures 34 mm in length, operates at frequencies
+of up to 4 Hz, and swims at speeds of up to 3.06 mm/s (0.09 Bl/s). This robot
+is the lightest and smallest swimmer with onboard actuation developed to date.
+
+
+
+ comment: To be presented at the 2024 IEEE/RSJ International Conference on
+ Intelligent Robots and Systems (IROS 2024)
+
+
+
+ + ♻ ☆ Explaining Explaining +
+ +
+ Explanation is key to people having confidence in high-stakes AI systems.
+However, machine-learning-based systems -- which account for almost all current
+AI -- can't explain because they are usually black boxes. The explainable AI
+(XAI) movement hedges this problem by redefining "explanation". The
+human-centered explainable AI (HCXAI) movement identifies the
+explanation-oriented needs of users but can't fulfill them because of its
+commitment to machine learning. In order to achieve the kinds of explanations
+needed by real people operating in critical domains, we must rethink how to
+approach AI. We describe a hybrid approach to developing cognitive agents that
+uses a knowledge-based infrastructure supplemented by data obtained through
+machine learning when applicable. These agents will serve as assistants to
+humans who will bear ultimate responsibility for the decisions and actions of
+the human-robot team. We illustrate the explanatory potential of such agents
+using the under-the-hood panels of a demonstration system in which a team of
+simulated robots collaborate on a search task assigned by a human.
+
+
+
+
+ + ♻ ★ Learning Occlusion-aware Decision-making from Agent Interaction via + Active Perception +
+ +
+ Occlusion-aware decision-making is essential in autonomous driving due to the
+high uncertainty of various occlusions. Recent occlusion-aware decision-making
+methods encounter issues such as high computational complexity, scenario
+scalability challenges, or reliance on limited expert data. Benefiting from
+automatically generating data by exploration randomization, we uncover that
+reinforcement learning (RL) may show promise in occlusion-aware
+decision-making. However, previous occlusion-aware RL faces challenges in
+expanding to various dynamic and static occlusion scenarios, low learning
+efficiency, and lack of predictive ability. To address these issues, we
+introduce Pad-AI, a self-reinforcing framework to learn occlusion-aware
+decision-making through active perception. Pad-AI utilizes vectorized
+representation to represent occluded environments efficiently and learns over
+the semantic motion primitives to focus on high-level active perception
+exploration. Furthermore, Pad-AI integrates prediction and RL within a unified
+framework to provide risk-aware learning and security guarantees. Our framework
+was tested in challenging scenarios under both dynamic and static occlusions
+and demonstrated efficient and general perception-aware exploration performance
+to other strong baselines in closed-loop evaluations.
+
+
+
+
+ + ♻ ☆ Proprioception Is All You Need: Terrain Classification for Boreal + Forests IROS 2024 +
+
+
+
+
+
+
+
+ Damien LaRocque, William Guimont-Martin, David-Alexandre Duclos, Philippe Giguère, François Pomerleau
+
+
+ Recent works in field robotics highlighted the importance of resiliency
+against different types of terrains. Boreal forests, in particular, are home to
+many mobility-impeding terrains that should be considered for off-road
+autonomous navigation. Also, being one of the largest land biomes on Earth,
+boreal forests are an area where autonomous vehicles are expected to become
+increasingly common. In this paper, we address this issue by introducing
+BorealTC, a publicly available dataset for proprioceptive-based terrain
+classification (TC). Recorded with a Husky A200, our dataset contains 116 min
+of Inertial Measurement Unit (IMU), motor current, and wheel odometry data,
+focusing on typical boreal forest terrains, notably snow, ice, and silty loam.
+Combining our dataset with another dataset from the state-of-the-art, we
+evaluate both a Convolutional Neural Network (CNN) and the novel state space
+model (SSM)-based Mamba architecture on a TC task. Interestingly, we show that
+while CNN outperforms Mamba on each separate dataset, Mamba achieves greater
+accuracy when trained on a combination of both. In addition, we demonstrate
+that Mamba's learning capacity is greater than a CNN for increasing amounts of
+data. We show that the combination of two TC datasets yields a latent space
+that can be interpreted with the properties of the terrains. We also discuss
+the implications of merging datasets on classification. Our source code and
+dataset are publicly available online:
+https://github.com/norlab-ulaval/BorealTC.
+
+
+
+ comment: Accepted to the 2024 IEEE/RSJ International Conference on Intelligent
+ Robots and Systems (IROS 2024)
+
+
+
+ + ♻ ★ Universal Trajectory Optimization Framework for Differential Drive Robot + Class +
+
+
+
+
+
+
+
+ Mengke Zhang, Nanhe Chen, Hu Wang, Jianxiong Qiu, Zhichao Han, Qiuyu Ren, Chao Xu, Fei Gao, Yanjun Cao
+
+
+ Differential drive robots are widely used in various scenarios thanks to
+their straightforward principle, from household service robots to disaster
+response field robots. There are several types of driving mechanisms for
+real-world applications, including two-wheeled, four-wheeled skid-steering,
+tracked robots, and so on. The differences in the driving mechanisms usually
+require specific kinematic modeling when precise control is desired.
+Furthermore, the nonholonomic dynamics and possible lateral slip lead to
+different degrees of difficulty in getting feasible and high-quality
+trajectories. Therefore, a comprehensive trajectory optimization framework to
+compute trajectories efficiently for various kinds of differential drive robots
+is highly desirable. In this paper, we propose a universal trajectory
+optimization framework that can be applied to differential drive robots,
+enabling the generation of high-quality trajectories within a restricted
+computational timeframe. We introduce a novel trajectory representation based
+on polynomial parameterization of motion states or their integrals, such as
+angular and linear velocities, which inherently matches the robots' motion to
+the control principle. The trajectory optimization problem is formulated to
+minimize complexity while prioritizing safety and operational efficiency. We
+then build a full-stack autonomous planning and control system to demonstrate
+its feasibility and robustness. We conduct extensive simulations and real-world
+testing in crowded environments with three kinds of differential drive robots
+to validate the effectiveness of our approach.
+
+
+
+ comment: 15 pages, 15 figures
+
+
+
+ + ♻ ☆ AnySkin: Plug-and-play Skin Sensing for Robotic Touch +
+
+
+
+
+
+
+
+ Raunaq Bhirangi, Venkatesh Pattabiraman, Enes Erciyes, Yifeng Cao, Tess Hellebrekers, Lerrel Pinto
+
+
+ While tactile sensing is widely accepted as an important and useful sensing
+modality, its use pales in comparison to other sensory modalities like vision
+and proprioception. AnySkin addresses the critical challenges that impede the
+use of tactile sensing -- versatility, replaceability, and data reusability.
+Building on the simplistic design of ReSkin, and decoupling the sensing
+electronics from the sensing interface, AnySkin simplifies integration making
+it as straightforward as putting on a phone case and connecting a charger.
+Furthermore, AnySkin is the first uncalibrated tactile-sensor with
+cross-instance generalizability of learned manipulation policies. To summarize,
+this work makes three key contributions: first, we introduce a streamlined
+fabrication process and a design tool for creating an adhesive-free, durable
+and easily replaceable magnetic tactile sensor; second, we characterize slip
+detection and policy learning with the AnySkin sensor; and third, we
+demonstrate zero-shot generalization of models trained on one instance of
+AnySkin to new instances, and compare it with popular existing tactile
+solutions like DIGIT and ReSkin. Videos of experiments, fabrication details and
+design files can be found on https://any-skin.github.io/
+
+
+
+
+ + ♻ ☆ Soft Acoustic Curvature Sensor: Design and Development +
+ +
+ This paper introduces a novel Soft Acoustic Curvature (SAC) sensor. SAC
+incorporates integrated audio components and features an acoustic channel
+within a flexible structure. A reference acoustic wave, generated by a speaker
+at one end of the channel, propagates and is received by a microphone at the
+other channel's end. Our previous study revealed that acoustic wave energy
+dissipation varies with acoustic channel deformation, leading us to design a
+novel channel capable of large deformation due to bending. We then use Machine
+Learning (ML) models to establish a complex mapping between channel
+deformations and sound modulation. Various sound frequencies and ML models were
+evaluated to enhance curvature detection accuracy. The sensor, constructed
+using soft material and 3D printing, was validated experimentally, with
+curvature measurement errors remaining within 3.5 m-1 for a range of 0 to 60
+m-1 curvatures. These results demonstrate the effectiveness of the proposed
+method for estimating curvatures. With its flexible structure, the SAC sensor
+holds potential for applications in soft robotics, including shape measurement
+for continuum manipulators, soft grippers, and wearable devices.
+
+
+
+ comment: To appear in Robotics and Automation Letter
+
+
+
+ + ♻ ☆ Deep Bayesian Future Fusion for Self-Supervised, High-Resolution, + Off-Road Mapping +
+
+
+
+
+
+
+
+ Shubhra Aich, Wenshan Wang, Parv Maheshwari, Matthew Sivaprakasam, Samuel Triest, Cherie Ho, Jason M. Gregory, John G. Rogers III, Sebastian Scherer
+
+
+ High-speed off-road navigation requires long-range, high-resolution maps to
+enable robots to safely navigate over different surfaces while avoiding
+dangerous obstacles. However, due to limited computational power and sensing
+noise, most approaches to off-road mapping focus on producing coarse (20-40cm)
+maps of the environment. In this paper, we propose Future Fusion, a framework
+capable of generating dense, high-resolution maps from sparse sensing data (30m
+forward at 2cm). This is accomplished by - (1) the efficient realization of the
+well-known Bayes filtering within the standard deep learning models that
+explicitly accounts for the sparsity pattern in stereo and LiDAR depth data,
+and (2) leveraging perceptual losses common in generative image completion. The
+proposed methodology outperforms the conventional baselines. Moreover, the
+learned features and the completed dense maps lead to improvements in the
+downstream navigation task.
+
+
+
+
+ + ♻ ☆ Learning Adaptive Multi-Objective Robot Navigation Incorporating + Demonstrations +
+ +
+ Preference-aligned robot navigation in human environments is typically
+achieved through learning-based approaches, utilizing user feedback or
+demonstrations for personalization. However, personal preferences are subject
+to change and might even be context-dependent. Yet traditional reinforcement
+learning (RL) approaches with static reward functions often fall short in
+adapting to these varying user preferences, inevitably reflecting
+demonstrations once training is completed. This paper introduces a framework
+that combines multi-objective reinforcement learning (MORL) with
+demonstration-based learning. Our approach allows for dynamic adaptation to
+changing user preferences without retraining. It fluently modulates between
+reward-defined preference objectives and the amount of demonstration data
+reflection. Through rigorous evaluations, including a sim-to-real transfer on
+two robots, we demonstrate our framework's capability to reflect user
+preferences accurately while achieving high navigational performance in terms
+of collision avoidance and goal pursuance.
+
+
+
+
+ + ♻ ☆ TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for + Robotic Manipulation +
+
+
+
+
+
+
+
+ Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, Jian Tang
+
+
+ Vision-Language-Action (VLA) models have shown remarkable potential in
+visuomotor control and instruction comprehension through end-to-end learning
+processes. However, current VLA models face significant challenges: they are
+slow during inference and require extensive pre-training on large amounts of
+robotic data, making real-world deployment difficult. In this paper, we
+introduce a new family of compact vision-language-action models, called
+TinyVLA, which offers two key advantages over existing VLA models: (1) faster
+inference speeds, and (2) improved data efficiency, eliminating the need for
+pre-training stage. Our framework incorporates two essential components to
+build TinyVLA: (1) initializing the policy backbone with robust, high-speed
+multimodal models, and (2) integrating a diffusion policy decoder during
+fine-tuning to enable precise robot actions. We conducted extensive evaluations
+of TinyVLA in both simulation and on real robots, demonstrating that our
+approach significantly outperforms the state-of-the-art VLA model, OpenVLA, in
+terms of speed and data efficiency, while delivering comparable or superior
+performance. Additionally, TinyVLA exhibits strong generalization capabilities
+across various dimensions, including language instructions, novel objects,
+unseen positions, changes in object appearance, background variations, and
+environmental shifts, often matching or exceeding the performance of OpenVLA.
+We believe that \methodname offers an interesting perspective on utilizing
+pre-trained multimodal models for policy learning. Our project is at
+https://tiny-vla.github.io.
+
+
+
+ comment: add more citations
+
+
+
+ + ♻ ☆ FracGM: A Fast Fractional Programming Technique for Geman-McClure Robust + Estimator +
+
+
+
+
+
+
+
+ Bang-Shien Chen, Yu-Kai Lin, Jian-Yu Chen, Chih-Wei Huang, Jann-Long Chern, Ching-Cherng Sun
+
+
+ Robust estimation is essential in computer vision, robotics, and navigation,
+aiming to minimize the impact of outlier measurements for improved accuracy. We
+present a fast algorithm for Geman-McClure robust estimation, FracGM,
+leveraging fractional programming techniques. This solver reformulates the
+original non-convex fractional problem to a convex dual problem and a linear
+equation system, iteratively solving them in an alternating optimization
+pattern. Compared to graduated non-convexity approaches, this strategy exhibits
+a faster convergence rate and better outlier rejection capability. In addition,
+the global optimality of the proposed solver can be guaranteed under given
+conditions. We demonstrate the proposed FracGM solver with Wahba's rotation
+problem and 3-D point-cloud registration along with relaxation pre-processing
+and projection post-processing. Compared to state-of-the-art algorithms, when
+the outlier rates increase from 20% to 80%, FracGM shows 53% and 88% lower
+rotation and translation increases. In real-world scenarios, FracGM achieves
+better results in 13 out of 18 outcomes, while having a 19.43% improvement in
+the computation time.
+
+
+
+ comment: 8 pages, 6 figures
+
+
+
+ + ♻ ☆ Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction + in an Object Categorization Task +
+ +
+ Human intention-based systems enable robots to perceive and interpret user
+actions to interact with humans and adapt to their behavior proactively.
+Therefore, intention prediction is pivotal in creating a natural interaction
+with social robots in human-designed environments. In this paper, we examine
+using Large Language Models (LLMs) to infer human intention in a collaborative
+object categorization task with a physical robot. We propose a novel multimodal
+approach that integrates user non-verbal cues, like hand gestures, body poses,
+and facial expressions, with environment states and user verbal cues to predict
+user intentions in a hierarchical architecture. Our evaluation of five LLMs
+shows the potential for reasoning about verbal and non-verbal user cues,
+leveraging their context-understanding and real-world knowledge to support
+intention prediction while collaborating on a task with a social robot.
+
+
+
+ comment: Accepted at ICSR 2024,14 pages,5 figures,2 tables; work was co-funded
+ by Horizon Europe project TERAIS under Grant agreement number 101079338
+
+
+
+ + ♻ ☆ Commonsense Scene Graph-based Target Localization for Object Search +
+ +
+ Object search is a fundamental skill for household robots, yet the core
+problem lies in the robot's ability to locate the target object accurately. The
+dynamic nature of household environments, characterized by the arbitrary
+placement of daily objects by users, makes it challenging to perform target
+localization. To efficiently locate the target object, the robot needs to be
+equipped with knowledge at both the object and room level. However, existing
+approaches rely solely on one type of knowledge, leading to unsatisfactory
+object localization performance and, consequently, inefficient object search
+processes. To address this problem, we propose a commonsense scene graph-based
+target localization, CSG-TL, to enhance target object search in the household
+environment. Given the pre-built map with stationary items, the robot models
+the room-level knowledge with object-level commonsense knowledge generated by a
+large language model (LLM) to a commonsense scene graph (CSG), supporting both
+types of knowledge for CSG-TL. To demonstrate the superiority of CSG-TL on
+target localization, extensive experiments are performed on the real-world
+ScanNet dataset and the AI2THOR simulator. Moreover, we have extended CSG-TL to
+an object search framework, CSG-OS, validated in both simulated and real-world
+environments. Code and videos are available at
+https://sites.google.com/view/csg-os.
+
+
+
+
+ + ♻ ☆ Query-based Semantic Gaussian Field for Scene Representation in + Reinforcement Learning +
+
+
+
+
+
+
+
+ Jiaxu Wang, Ziyi Zhang, Qiang Zhang, Jia Li, Jingkai Sun, Mingyuan Sun, Junhao He, Renjing Xu
+
+
+ Latent scene representation plays a significant role in training
+reinforcement learning (RL) agents. To obtain good latent vectors describing
+the scenes, recent works incorporate the 3D-aware latent-conditioned NeRF
+pipeline into scene representation learning. However, these NeRF-related
+methods struggle to perceive 3D structural information due to the inefficient
+dense sampling in volumetric rendering. Moreover, they lack fine-grained
+semantic information included in their scene representation vectors because
+they evenly consider free and occupied spaces. Both of them can destroy the
+performance of downstream RL tasks. To address the above challenges, we propose
+a novel framework that adopts the efficient 3D Gaussian Splatting (3DGS) to
+learn 3D scene representation for the first time. In brief, we present the
+Query-based Generalizable 3DGS to bridge the 3DGS technique and scene
+representations with more geometrical awareness than those in NeRFs. Moreover,
+we present the Hierarchical Semantics Encoding to ground the fine-grained
+semantic features to 3D Gaussians and further distilled to the scene
+representation vectors. We conduct extensive experiments on two RL platforms
+including Maniskill2 and Robomimic across 10 different tasks. The results show
+that our method outperforms the other 5 baselines by a large margin. We achieve
+the best success rates on 8 tasks and the second-best on the other two tasks.
+
+
+
+
+ + ♻ ☆ TOP-Nav: Legged Navigation Integrating Terrain, Obstacle and + Proprioception Estimation CoRL 2024 +
+ +
+ Legged navigation is typically examined within open-world, off-road, and
+challenging environments. In these scenarios, estimating external disturbances
+requires a complex synthesis of multi-modal information. This underlines a
+major limitation in existing works that primarily focus on avoiding obstacles.
+In this work, we propose TOP-Nav, a novel legged navigation framework that
+integrates a comprehensive path planner with Terrain awareness, Obstacle
+avoidance and close-loop Proprioception. TOP-Nav underscores the synergies
+between vision and proprioception in both path and motion planning. Within the
+path planner, we present and integrate a terrain estimator that enables the
+robot to select waypoints on terrains with higher traversability while
+effectively avoiding obstacles. In the motion planning level, we not only
+implement a locomotion controller to track the navigation commands, but also
+construct a proprioception advisor to provide motion evaluations for the path
+planner. Based on the close-loop motion feedback, we make online corrections
+for the vision-based terrain and obstacle estimations. Consequently, TOP-Nav
+achieves open-world navigation that the robot can handle terrains or
+disturbances beyond the distribution of prior knowledge and overcomes
+constraints imposed by visual conditions. Building upon extensive experiments
+conducted in both simulation and real-world environments, TOP-Nav demonstrates
+superior performance in open-world navigation compared to existing methods.
+
+
+
+ comment: Published on CoRL 2024
+
+
+
+ + ♻ ☆ Enabling On-Chip High-Frequency Adaptive Linear Optimal Control via + Linearized Gaussian Process +
+ +
+ Unpredictable and complex aerodynamic effects pose significant challenges to
+achieving precise flight control, such as the downwash effect from upper
+vehicles to lower ones. Conventional methods often struggle to accurately model
+these interactions, leading to controllers that require large safety margins
+between vehicles. Moreover, the controller on real drones usually requires
+high-frequency and has limited on-chip computation, making the adaptive control
+design more difficult to implement. To address these challenges, we incorporate
+Gaussian process (GP) to model the adaptive external aerodynamics with linear
+model predictive control. The GP is linearized to enable real-time
+high-frequency solutions. Moreover, to handle the error caused by
+linearization, we integrate end-to-end Bayesian optimization during sample
+collection stages to improve the control performance. Experimental results on
+both simulations and real quadrotors show that we can achieve real-time
+solvable computation speed with acceptable tracking errors.
+
+
+
+
+ + ♻ ☆ Compact 3D Gaussian Splatting For Dense Visual SLAM +
+
+
+
+
+
+
+
+ Tianchen Deng, Yaohui Chen, Leyan Zhang, Jianfei Yang, Shenghai Yuan, Jiuming Liu, Danwei Wang, Hesheng Wang, Weidong Chen
+
+
+ Recent work has shown that 3D Gaussian-based SLAM enables high-quality
+reconstruction, accurate pose estimation, and real-time rendering of scenes.
+However, these approaches are built on a tremendous number of redundant 3D
+Gaussian ellipsoids, leading to high memory and storage costs, and slow
+training speed. To address the limitation, we propose a compact 3D Gaussian
+Splatting SLAM system that reduces the number and the parameter size of
+Gaussian ellipsoids. A sliding window-based masking strategy is first proposed
+to reduce the redundant ellipsoids. Then we observe that the covariance matrix
+(geometry) of most 3D Gaussian ellipsoids are extremely similar, which
+motivates a novel geometry codebook to compress 3D Gaussian geometric
+attributes, i.e., the parameters. Robust and accurate pose estimation is
+achieved by a global bundle adjustment method with reprojection loss. Extensive
+experiments demonstrate that our method achieves faster training and rendering
+speed while maintaining the state-of-the-art (SOTA) quality of the scene
+representation.
+
+
+
+
+ + ♻ ☆ Constraint-Guided Online Data Selection for Scalable Data-Driven Safety + Filters in Uncertain Robotic Systems +
+
+
+
+
+
+
+
+ Jason J. Choi, Fernando Castañeda, Wonsuhk Jung, Bike Zhang, Claire J. Tomlin, Koushil Sreenath
+
+
+ As the use of autonomous robots expands in tasks that are complex and
+challenging to model, the demand for robust data-driven control methods that
+can certify safety and stability in uncertain conditions is increasing.
+However, the practical implementation of these methods often faces scalability
+issues due to the growing amount of data points with system complexity, and a
+significant reliance on high-quality training data. In response to these
+challenges, this study presents a scalable data-driven controller that
+efficiently identifies and infers from the most informative data points for
+implementing data-driven safety filters. Our approach is grounded in the
+integration of a model-based certificate function-based method and Gaussian
+Process (GP) regression, reinforced by a novel online data selection algorithm
+that reduces time complexity from quadratic to linear relative to dataset size.
+Empirical evidence, gathered from successful real-world cart-pole swing-up
+experiments and simulated locomotion of a five-link bipedal robot, demonstrates
+the efficacy of our approach. Our findings reveal that our efficient online
+data selection algorithm, which strategically selects key data points, enhances
+the practicality and efficiency of data-driven certifying filters in complex
+robotic systems, significantly mitigating scalability concerns inherent in
+nonparametric learning-based control methods.
+
+
+
+ comment: The first three authors contributed equally to the work. This work
+ has been submitted to the IEEE for possible publication. Copyright may be
+ transferred without notice, after which this version may no longer be
+ accessible
+
+
+
+ + ♻ ☆ AquaMILR: Mechanical intelligence simplifies control of undulatory + robots in cluttered fluid environments +
+
+
+
+
+
+
+
+ Tianyu Wang, Nishanth Mankame, Matthew Fernandez, Velin Kojouharov, Daniel I. Goldman
+
+
+ While undulatory swimming of elongate limbless robots has been extensively
+studied in open hydrodynamic environments, less research has been focused on
+limbless locomotion in complex, cluttered aquatic environments. Motivated by
+the concept of mechanical intelligence, where controls for obstacle navigation
+can be offloaded to passive body mechanics in terrestrial limbless locomotion,
+we hypothesize that principles of mechanical intelligence can be extended to
+cluttered hydrodynamic regimes. To test this, we developed an untethered
+limbless robot capable of undulatory swimming on water surfaces, utilizing a
+bilateral cable-driven mechanism inspired by organismal muscle actuation
+morphology to achieve programmable anisotropic body compliance. We demonstrated
+through robophysical experiments that, similar to terrestrial locomotion, an
+appropriate level of body compliance can facilitate emergent swim through
+complex hydrodynamic environments under pure open-loop control. Moreover, we
+found that swimming performance depends on undulation frequency, with effective
+locomotion achieved only within a specific frequency range. This contrasts with
+highly damped terrestrial regimes, where inertial effects can often be
+neglected. Further, to enhance performance and address the challenges posed by
+nondeterministic obstacle distributions, we incorporated computational
+intelligence by developing a real-time body compliance tuning controller based
+on cable tension feedback. This controller improves the robot's robustness and
+overall speed in heterogeneous hydrodynamic environments.
+
+
+
+
+ + ♻ ☆ 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple + 3D Representations RSS +
+ +
+ Imitation learning provides an efficient way to teach robots dexterous
+skills; however, learning complex skills robustly and generalizablely usually
+consumes large amounts of human demonstrations. To tackle this challenging
+problem, we present 3D Diffusion Policy (DP3), a novel visual imitation
+learning approach that incorporates the power of 3D visual representations into
+diffusion policies, a class of conditional action generative models. The core
+design of DP3 is the utilization of a compact 3D visual representation,
+extracted from sparse point clouds with an efficient point encoder. In our
+experiments involving 72 simulation tasks, DP3 successfully handles most tasks
+with just 10 demonstrations and surpasses baselines with a 24.2% relative
+improvement. In 4 real robot tasks, DP3 demonstrates precise control with a
+high success rate of 85%, given only 40 demonstrations of each task, and shows
+excellent generalization abilities in diverse aspects, including space,
+viewpoint, appearance, and instance. Interestingly, in real robot experiments,
+DP3 rarely violates safety requirements, in contrast to baseline methods which
+frequently do, necessitating human intervention. Our extensive evaluation
+highlights the critical importance of 3D representations in real-world robot
+learning. Videos, code, and data are available on
+https://3d-diffusion-policy.github.io .
+
+
+
+ comment: Published at Robotics: Science and Systems (RSS) 2024. Videos, code,
+ and data: https://3d-diffusion-policy.github.io
+
+
+
+ + ♻ ☆ AED: Adaptable Error Detection for Few-shot Imitation Policy NeurIPS2024 +
+
+
+
+
+
+
+
+ Jia-Fong Yeh, Kuo-Han Hung, Pang-Chi Lo, Chi-Ming Chung, Tsung-Han Wu, Hung-Ting Su, Yi-Ting Chen, Winston H. Hsu
+
+
+ We introduce a new task called Adaptable Error Detection (AED), which aims to
+identify behavior errors in few-shot imitation (FSI) policies based on visual
+observations in novel environments. The potential to cause serious damage to
+surrounding areas limits the application of FSI policies in real-world
+scenarios. Thus, a robust system is necessary to notify operators when FSI
+policies are inconsistent with the intent of demonstrations. This task
+introduces three challenges: (1) detecting behavior errors in novel
+environments, (2) identifying behavior errors that occur without revealing
+notable changes, and (3) lacking complete temporal information of the rollout
+due to the necessity of online detection. However, the existing benchmarks
+cannot support the development of AED because their tasks do not present all
+these challenges. To this end, we develop a cross-domain AED benchmark,
+consisting of 322 base and 153 novel environments. Additionally, we propose
+Pattern Observer (PrObe) to address these challenges. PrObe is equipped with a
+powerful pattern extractor and guided by novel learning objectives to parse
+discernible patterns in the policy feature representations of normal or error
+states. Through our comprehensive evaluation, PrObe demonstrates superior
+capability to detect errors arising from a wide range of FSI policies,
+consistently surpassing strong baselines. Moreover, we conduct detailed
+ablations and a pilot study on error correction to validate the effectiveness
+of the proposed architecture design and the practicality of the AED task,
+respectively.
+
+
+
+ comment: Accepted to NeurIPS2024
+
+
+
+ + ♻ ☆ Speech-Guided Sequential Planning for Autonomous Navigation using Large + Language Model Meta AI 3 (Llama3) +
+ +
+ In social robotics, a pivotal focus is enabling robots to engage with humans
+in a more natural and seamless manner. The emergence of advanced large language
+models (LLMs) such as Generative Pre-trained Transformers (GPTs) and
+autoregressive models like Large Language Model Meta AI (Llamas) has driven
+significant advancements in integrating natural language understanding
+capabilities into social robots. This paper presents a system for speech-guided
+sequential planning in autonomous navigation, utilizing Llama3 and the Robot
+Operating System~(ROS). The proposed system involves using Llama3 to interpret
+voice commands, extracting essential details through parsing, and decoding
+these commands into sequential actions for tasks. Such sequential planning is
+essential in various domains, particularly in the pickup and delivery of an
+object. Once a sequential navigation task is evaluated, we employ DRL-VO, a
+learning-based control policy that allows a robot to autonomously navigate
+through social spaces with static infrastructure and (crowds of) people. We
+demonstrate the effectiveness of the system in simulation experiment using
+Turtlebot 2 in ROS1 and Turtlebot 3 in ROS2. We conduct hardware trials using a
+Clearpath Robotics Jackal UGV, highlighting its potential for real-world
+deployment in scenarios requiring flexible and interactive robotic behaviors.
+
+
+
+ comment: Accepted at the 16th International Conference on Social Robotics + AI
+
+