For over half a century, visions of the future have featured robots serving as domestic assistants. But despite many years of development in the robotics field, except for a few elementary tasks like floor vacuuming, the dream of the domestic robot remains unfulfilled.

Toyota Research Institute (TRI) is working to change all that. This has far reaching implications for society beyond the desire to have clean and tidy homes. If we think more broadly, societies around the world are aging and families face difficult choices regarding how to care for elderly relatives. Robotic assistants have the potential to do far more than clean. They can also assist those with disabilities or age-related challenges so they can age in place with independence, dignity, and joy.

As part of its work to create a domestic robot, TRI is focusing on innovating the machine’s ‘helping hands.’ A domestic robot’s hands (also known as manipulators or grippers) must handle a wide variety of household objects to be useful. These manipulators must be capable of stable grasps, precise placement and safe interactions during inadvertent contact. To be affordable as part of a household robot, the gripper must also be low cost. TRI engineers have designed a manipulator with all these capabilities called the Soft Bubble Gripper.

The Soft Bubble Gripper builds on past achievements by the TRI Manipulation Team. Previously, TRI’s robots gripped objects, sorted them, and even correctly placed items in a dishwasher using conventional, two-fingered grippers guided by external cameras. The grippers relied on these external cameras to do their work. In other words, they had no sense of touch.

Now, TRI has found a way to give these domestic robots a better ability to sense their work with soft grippers that both passively hold objects better and actively sense how much force is applied. Most importantly, the soft grippers accurately measure lateral force – which indicates when an object is about to slip out of a gripper’s grasp.

The gripper’s development team is led by Alex Alspach, who tapped into his soft robot development background to ideate the new gripper design, and Naveen Kuppuswamy, who formulated algorithms to use it. Together with TRI’s Tactile Team, they developed technology that utilizes the robust and compliant nature of air-filled, elastic bubbles for gripping with sensing from cameras on the INSIDE. The cameras show what’s happening from a new perspective inside the grasp, including forces that are usually invisible.

Alspach and Kuppuswamy iterated their design, building on a body of work initiated by other research organizations. Early designs used a robot arm with a single, large, round, soft-bubble on its end effector which then graduated to two arms (a.k.a. “big fingers”) for dexterous, tactile-driven tasks like blindly sorting objects and threading a nut onto a giant bolt.  Now, they use a single arm with two smaller fingers, each using a soft bubble which combines all the advantages of compliant gripping with real-time, real-world tactile sensing. The bubbles feel shapes and forces and recognize the object they are gripping, as well as the forces between the object and the fingers. But none of these advances would move us any closer to making domestic robots a reality if the cost of doing so was prohibitive. So, the team paid careful attention to materials expense and designed the Soft Bubble Gripper with inexpensive materials to eliminate that obstacle.

Here’s What It Can Do

This intriguing combination of force and shape data as visualized by these sensors allows robots equipped with Soft Bubble Grippers to perform a range of seemingly impossible tasks (by hard-gripper standards) that the team at TRI is excited to explore. Thus far, lab-validated capabilities of the Soft Bubble Gripper — as documented in the accompanying video — include:

Robust and Passively-Compliant Grasping. The innovation on the Soft Bubble Gripper begins with the use of the inherently grippy texture and durable elastic properties of latex. The latex is inflated to a degree of softness that optimizes compliance to the shapes of held objects — maximizing the grasp stability. Given the physical properties — its air-filled bubbles and the gummy, high-friction texture of latex — the Soft Bubble Gripper is very reliable when it comes to grabbing and holding onto objects. Passive compliance is a theme in soft robotics and manipulation in general — passive meaning it is controlled by the laws of material physics (not a motor) and compliance to whatever shape is in the gripper. However, as Alspach points out, “Compliance alone doesn’t let you do creative things with the object. This is where the gripper’s other capabilities come into play.”

Recognizing Objects by Touch. Inside the bubble is a low-cost, off-the-shelf Time-of-Flight (ToF) depth sensor/IR camera that uses vision to ‘feel’ what the gripper is holding. It enables the system to recognize objects by their shape and other physical properties and understand what to do with it within about a second. This is analogous to our sensitive fingers searching for house keys buried deep in a purse or backpack; our fingers can’t technically ‘see’ but they provide tactile information we use to build a mental model of what we are touching.

This capability lets the robot perform realistic, useful tasks typically found in a home, because it understands how the object is oriented in its ‘hand’ and what it must do to complete the task. This is not much different from a toddler playing with a shape-sorter toy. When a child is presented with a pile of differently shaped blocks, they randomly grab one and feel its shape and all of its facets. They look at the cut-out holes on the shape sorter box … consider the shape in their hand … and make their selection.

The process that the Soft Bubble Gripper uses to sort a sink full of objects is similar. It measures and recognizes geometric features, then either precisely sets mugs in the dishwasher or drops PET bottles into the recycling bin. It perceptively sorts the shapes, not visually, but through touch.

Shear Force Detection and Interpretation. The Soft Bubble Gripper can also sense when some outside force is trying to take, twist, pull or push the object we’re holding. It does this by using the camera inside the bubbles to measure how a dense dot pattern inside the latex membrane is moving and distorting, and inferring the magnitudes and directions of the forces causing this distortion. This lets it swiftly sense when the object it is lowering has landed on the counter, if it has accidentally bumped into something, or when an object it is handing over has been received. The gripper gives the robot tactile awareness of the outside world and its cohabitants. Depending on the controller, the robot will gently hand you a full wine glass or set it on the table without spilling.

“To achieve a handover, the robot doesn’t need to know it is holding a wine glass,” Kuppuswamy explains, “only that something is pulling on it … exerting an external force that tells it to let go. Because the data is so rich, the controller can be as simple as, “When you feel the tug, let go.”

In another exercise, the robot stacks transparent wine glasses. This would be incredibly difficult for a robot with conventional vision and hard grippers, because it is difficult to ‘see’ transparent objects and wine glasses are fragile. But it is relatively easy with the Soft Bubble Gripper, and the robot blindly stacks stemware with no knowledge of the object height or table position. Shear force — read as directional changes in the dot pattern inside the latex membrane — indicates that one glass has successfully landed on top of another and tells the robot to let go.

“This kind of tactile sensing is absolutely necessary for the uncertainty that you see in people’s homes,” explains Alspach. “We’re not in a controlled factory environment anymore. Our goal is to be helpful to people in their homes, and to do so we have to build a domestic robot with a sense of what it is touching.”

Furthermore, the robot is able to perceive and track changes in rotation of an object in its grasp. This is demonstrated when the robot pushes the bottom of a mug against the kitchen counter to actually flip it into a better position for loading into the dishwasher.

“The ‘a-ha’ moment here,” says Kuppuswamy, “is that instead of putting all our computing energy into how the robot approaches and picks up an object — which is a requirement for blind robots — we fast-forward past that to the actual task. The robot quickly grabs anything in the sink. Then we can focus on and adjust to what’s happening in its hand.  This is quite different from a traditionally slow process needed for a factory robot.”

Operating Blind. Most robots rely heavily on cameras to create a sense of vision.  This means that light is needed to help create this sense of perception.  The technical term for this is ‘visual occlusion.’ Traditionally, robots also have difficulty working around clutter or in confined spaces, where there’s no clear line of sight. Transparent, shiny or dark objects are also quite hard for robots to see. However, since a robot equipped with the Soft Bubble Gripper relies on sensors inside the bubble (and not an external camera) to recognize and manipulate objects, it works equally well in lit or darkened rooms. This also means the technology is ideally suited to situations in which it must reach into cluttered places (such as a sink full of miscellaneous objects and/or water) or has to manipulate in a way that would otherwise block its own view.

“We’re not trying to replace external vision sensors — they’re still important,” Alspach notes. “We’re adding to these systems with even more useful sensory information resulting in increased capabilities and added precision.”

Training through Self-Annotation.  Traditionally, a robotic system guided by computer vision is trained to recognize objects by repeatedly showing it images of a thing in order to establish defined categories — a process known as supervised training. For example, somebody has to show the robot many times over various photos of coffee mugs and tell it, ‘these are all coffee mugs.” As one can imagine, this process of annotation is incredibly time consuming. The team had a breakthrough when they learned how to set up the robot to self-annotate by filling a sink with one type of object and letting the robot repeatedly grasp and drop these objects – reducing the amount of time it takes the robot to ‘learn’ a new object. This process is easily repeated for every new object the team wants the Soft Bubble Gripper to learn. It is important to build tactile sensors that can withstand the many cycles this type of training requires for minimal wear and damage.

Alspach also points out, “By cutting the human out of the loop completely, the grasp data was a better match to the variability the robot would experience in real-world situations.”

Use of Low-Cost Materials. Utilizing materials smartly with simple fabrication techniques, the team designed the Soft Bubble Gripper to be inexpensive to build, operate, maintain and repair.

“An insight we gained,” explains Alspach, “is finding a way to make an inexpensive camera deliver usable depth data at an extremely close range. It’s how we angled the camera that lets us maximize the quality of the depth measurements — so that the camera can view and identify the ‘tactile’ imprint of the gripped object on the soft latex bubble.”

Future Exploration

A long-term goal of the TRI robotics research team is to break the traditional robot controller paradigm and accommodate for an ever-changing environment.

Improving Reflexes. The team is interested in developing a way to replicate real-time course-correcting behavior in robots. They are striving for distributed robotic sensing where information is processed and reacted to ‘locally’ instead of having to refer to a centralized ‘brain’ for every decision. This enables robots to execute their actions much quicker with less forethought. “It’s more like what humans or animals do,” Alspach explains. “In a fraction of a second we react and catch the thing that fell off the table or slipped out of our hand almost without thinking.”

Expanding Softness. Currently, only the robot gripper is soft. But what if this same breakthrough could be applied to more surface areas of the robot? Added softness would bring great advantages for human/robot interaction including working in the home with children, pets or people who may already have physical challenges.

“We aim to build robots that can interact with the world around them comfortably,” says Alspach, drawing on his deep experience in soft robotics. “Softness makes the application of robots more reasonable for homes. Taking it further, we can begin to incorporate lightweight materials, air, smaller electronics and an optimized design to help us reduce weight. It’s the combination of soft and lightweight, along with the ability to sense and react, that contributes to safety.”

Kuppuswamy explains: “It’s actually a really tricky problem. In one sense, it’s extremely hard to build because you are working with frontier materials…. It’s not just classical methods and materials like machining or welding.” Further complicating this is the fact that anything inflatable introduces the factor of continuously dynamic shape-shifting, which destabilizes the usual equations. As Kuppuswamy points out, “Mathematically, soft is hard.”

Publications

More depth on the technical progress and discoveries with a soft gripping can be read in their published paper, “Soft Bubble grippers for robust and perceptive manipulation.” The team’s research was also recognized at ICRA 2020, the annual IEEE International Conference on Robotics and Automation, with a 2019 Robotics and Automation Letters Best Paper award for another publication about the soft gripper project.

TRI welcomes other scientists researching robotic manipulation to build on their work by bringing their own insights and discoveries. To inspire this collaboration and further the research on soft bubble grippers, they hosted VisuoTactile 2020, a workshop of thought-leaders in visuotactile technology at the Robotics Science and Systems (RSS) conference.

Conclusion

The new Soft Bubble Gripper builds upon TRI’s manipulation research toward making human-assist robots reliable and robust. The team’s design is a well-engineered take on a concept for robot grippers. The extreme softness from the stretchy material (low stiffness compared to standard soft grippers) makes this an especially great gripper all by itself, even without the sensing capability. It can conform to all sorts of weirdly shaped objects and get a real stable grasp. The gripper brings together strength and smarts to handle objects that require a delicate touch and a steady hand.

By adding sensing capability, the team has made a great gripper even smarter. Through sensing geometry, camera images, and most excitingly shear forces, we can now ‘see’ objects in the robot hands by estimating the pose of objects and sensing forces on the surface. The additional benefit is relatively low cost due to the use of inexpensive cameras on the inside of the bubble gripper for perception

For roboticists and consumers alike, advances like the Soft Bubble Grippers bring the practical use of robots one step closer to reality, while opening the doors to future applications in robotics more broadly.