Problem

The rapidly expanding retail sector increasingly adopts autonomous mobile robots for efficiency, but these robots face significant challenges in dynamic retail environments. They often struggle to adapt to novel situations, such as newly introduced products or changes in packaging, leading to failures in fully autonomous operation. Existing autonomous manipulation capabilities are often limited to simplified scenarios and cannot generalize to the complexities of a commercial retail store, highlighting a need for more robust and adaptable robotic systems.

Method

This study introduces "GriffinX," an omni-directional dual-arm mobile robot specifically designed for retail store intra-logistic operations. The robot is equipped with two Universal Robots UR5e manipulators, each fitted with a different end-effector: a two-fingered rigid gripper and a custom three-fingered soft gripper, enabling it to handle a wide variety of items. The core of the system is a teleoperation method with shared control, where a Virtual Reality (VR) motion capture system (HTC Vive) captures human operator commands. A constrained optimal control problem is formulated to track operator inputs while providing autonomous assistance to reach target positions, ensuring collision avoidance and kinematic constraints. The cost function balances operator reference tracking and goal-tracking, with an adaptive weighing matrix to smoothly transition between these modes as the end-effector approaches a goal. Collision avoidance uses planner and ellipsoidal constraints, while kinematic constraints are defined for coordinated dual-arm manipulation, particularly for top-down, front, and side grasping approaches. The optimal control problem is solved using Model Predictive Control with the AL-iLQR algorithm.

Results

The proposed system was validated through extensive testing in a mockup retail environment. Experiments demonstrated its efficacy in manipulating various commonly encountered retail items. The robot successfully performed single-arm manipulation for smaller objects using both grippers, closely following operator trajectories before switching to autonomous assistance near the target. It also demonstrated coordinated dual-arm manipulation for longer items, maintaining a nearly constant distance between end-effectors during operations. The custom soft gripper proved effective for grasping a diverse range of objects, including delicate items like fruits and chip packets, primarily through top-down approaches. A comparative analysis showed that the shared control strategy reduced task completion time by an average of 30% and eliminated collisions with obstacles compared to pure teleoperation, highlighting its practical benefits.

Implications

This research offers a promising solution for retail automation by addressing the limitations of fully autonomous robots in dynamic and unpredictable environments. The shared control teleoperation system allows human operators to intervene in complex or novel situations, ensuring task completion and providing valuable feedback for future autonomous system refinement. The "GriffinX" robot, with its versatile manipulation capabilities and robust shared control framework, contributes to enhancing the efficiency and adaptability of robotic systems in retail. Future work aims to leverage operator demonstrations for developing fully autonomous, continually learning systems, and to enhance the user experience with haptic feedback and advanced VR features. Further comprehensive user studies will evaluate the system's overall effectiveness, usability, and repeatability in real-world scenarios.