How Generative AI Can Accelerate The Deployment Of Autonomous Robots

Generative AI refers to a field of artificial intelligence that uses models and techniques for generating new content. These methods are designed to understand and learn the underlying patterns, structures, and characteristics of the training data, enabling them to generate new content. Generative AI utilizes techniques such as deep neural networks to capture and imitate the statistical distribution of the training data. When these models are trained on large datasets they can generate new examples that exhibit similar characteristics to the data they were trained on. They have been successfully used to create a variety of outputs, including images, texts, audio, and videos. Recent advances in Generative AI (e.g., ChatGPT, DALL-E 2) are expected to revolutionize the future of work in many industry sectors.

Labor shortages are requiring robotic solutions to be deployed rapidly.  Unfortunately, the deployment of autonomous robots takes a significant amount of human effort due to the time needed to write software and test the system. The increasing complexity of robotic systems is aggravating this problem. Unfortunately, the availability of human expertise can become a bottleneck in robot deployment.         

The robotics community has always been at the forefront of leveraging the latest AI advances.  So, a natural question is: how will generative AI concepts and tools be used by the robotics community to accelerate the deployment of the next generation of intelligent robots? There are significant investments in this area due to the enormous amount of interest in generative AI ideas. Recent efforts are showing early signs of success in using generative AI in robotics applications. This blog post highlights eight opportunities for using generative AI in accelerating robot deployment by tackling some of the challenging and time-consuming steps.

1. Generating Robot Motion from Natural Language Description

Robots often need to perform complex motions to successfully execute a task.  Consider the example of sanding where the robot needs to move the sanding tool in a complex motion pattern to produce a scratch-free surface finish. In the past, if a human expert needed a robot to follow a particular type of sanding tool motion, they would use one of the following methods. The first option would be to teleoperate the robot using a tech pendant and specify the tool motion. The second option would be to demonstrate the tool motion using a handheld tool and use a motion capture system to record the tool motion. The robot then attempts to imitate human motion by analyzing the motion capture data. Third, they can program the motion using the motion commands. Unfortunately, all of these approaches are labor intensive and take significant time for complex parts. This can delay the deployment of robots in new applications. Many process experts would prefer a new modality to generate robot motion based on natural language description of the motion. Generative AI now offers the capability to generate code from the text description, which enables humans to communicate with robots in a more natural, time-efficient manner and automatically create robot motion. This means human experts with no programming experience can get the robots to perform the right kind of motion. The elimination of robot programming is expected to remove bottlenecks and can significantly speed up this task.           

2. Task Planning

Many real-world applications require robots to perform complex tasks. For example, consider the task of replacing a motor of a cooling fan in a control box. This requires the top-level task to be decomposed into much simpler subtasks and to determine the sequence of tasks. Once the task sequence is determined, the robot can generate motions for the simple tasks and execute this task. Traditionally, a task planner needs to be developed for each specialized domain and the addition of new objects or processes requires an update to the planner. Task planners often struggle to deal with new failure modes and therefore recovering from failures becomes challenging during task planning. Large Language Models (LLM) have become the foundation of many Generative AI techniques and can be used to identify sequences of atomic tasks needed to perform complex tasks. With the latest advancements in LLMs, we can pose a query such as, “Provide step-by-step directions to obtain a tool from a locked shelf.” and generate a sequence of various subtasks necessary to perform the overall task. An exemplary set of resulting subtasks includes locating the shelf, reaching the shelf, unlocking the shelf, extracting the tool, locking the shelf, and transporting the tool. Once atomic tasks have been identified, the robot can use a motion planner to generate the motion to execute the task.  LLMs can be extremely useful in automatically generating task sequences based on common sense knowledge and using them can eliminate the need for developing domain-specific task planners.  

3. Developing Human-Like Responses to Improve Human-Robot Interaction

In many collaborative tasks, humans will have greater trust in robots if robots exhibit human-like motions to emulate being a team member. Consider the example of a robot that delivers parts and tools to a human performing a complex maintenance task. Humans will be much more comfortable working around this robot if its motions are similar to the movement of a human co-worker. Generative AI has already demonstrated capabilities to generate paintings and writings based on the style of a particular human artist (e.g., Dali) or writer (e.g., Shakespeare). Therefore, we expect that Generative AI can facilitate high-quality imitation learning, where robots learn to mimic human motions. Robots and humans have different morphology, so exactly imitating human motion will not be possible for the robots. By training models on human-motion data or recorded demonstrations, robots will be able to generate their own sequences of actions that closely resemble human behaviors. Therefore, Generative AI can enhance the interaction between robots and humans by enabling robots to exhibit more natural and contextually appropriate responses. By training on human language data, generative models will generate robot speech responses that are coherent, fluent, and aligned with human communication norms. This has the potential to make robot interactions more engaging, intuitive, and effective. In turn, trustworthy robotic co-workers will be deployed ubiquitously once humans grow accustomed to human-like responses.

4. Improving Perception

Robots use perception to build an expressive model of the world and use this to make decisions for autonomous behaviors. Complex environments often create challenges in building complete models due to occlusions and sensor errors. For example, consider the task of a robot scanning a large part to remove the paint. Scanning the part at high speed to obtain fast cycle times may produce some phantom holes in the surface model. These holes can pose challenges for decision-making due to a lack of information. Generative AI can be employed to complete occluded or hidden parts of an object through a technique similar to image inpainting. This technique involves filling in missing or occluded regions of an image with plausible content that blends seamlessly with the surrounding area. Once the Generative AI model is trained, it can be used to complete occluded or hidden parts of the objects. The incomplete object model is fed into the trained model, which then generates predictions for the missing regions based on the learned patterns and context from the training data. This approach can create a visually coherent result by filling in the missing details and enabling a robot to perform autonomous tasks that might otherwise be halted by poor perception.

5. Generating Synthetic Scenarios for Utilizing Reinforcement Learning

Reinforcement learning has emerged as a useful tool for robots to acquire new skills. This type of learning requires the robot to train in a simulation using a trial-and-error approach. Manually generating a large number of scenarios for reinforcement learning is highly time-consuming and may still not cover all the relevant cases or provide adequate diversity. To illustrate this, consider the case of a legged robot that seeks to learn locomotion on challenging terrains to perform inspection tasks. This will require a wide variety of challenging terrains to learn robust and efficient gaits. 

Generative AI technology can be used to generate distinct, synthetic scenarios to aid reinforcement learning in simulations. By enabling the robots to train on a diverse set of examples and scenarios, Generative AI can enable robots to learn new behaviors, strategies, or responses based on the learned patterns and context. This allows robots to develop robust and adaptive behavior when faced with new situations and perform tasks efficiently.

6. Generating Test Plans and Virtual Environments for Testing Autonomous Robots

Autonomous robots need to be rigorously tested to ensure they perform safely in a wide variety of challenging environments. Consider the case of a mobile manipulator performing wind turbine finishing. This robot needs to be carefully tested to make sure that it will not damage the wind turbine. Manually developing test plans is time-consuming for autonomous robots operating in complex environments.   

Generative AI models can learn the structure and logic of the test cases from prior test plans for similar systems and generate new ones based on the learned patterns. This approach can help in automatically generating diverse and valuable test cases to evaluate different aspects of autonomous robots, such as perception, planning, decision-making, state estimation, control, and coordination.

Simulation has emerged as a useful tool to evaluate autonomous robots. Generative AI can be employed to enhance the capabilities of simulation environments used for testing autonomous robots. By generating realistic and diverse synthetic data, such as sensor inputs or environmental conditions, the AI model can augment the existing simulation data and create more challenging and representative test cases to aid in the evaluation of autonomous robot capabilities.

Generative AI models can be used to generate realistic synthetic faults in the behavior of autonomous systems. This enables the evaluation process to assess whether or not the system responds appropriately to faults and remains safe.  

 
7. Failure Detection and Recovery for Resilient Operation

The ability to recover from failure is needed in autonomous robotic systems in challenging missions. Whenever testing reveals that the system is not able to recover from failure, the system needs to be improved. This requires an ability to detect failures and take recovery actions. Take, for example, an assembly operation where a previously installed component is damaged due to a subsequent assembly operation. To proceed with the assembly,  the damaged part will need to be replaced through a series of many disassembly steps. Manually coding all possible contingency actions is intractable. By learning the patterns and characteristics of normal operation, the AI model can generate synthetic data that deviates from the norm. These generated anomalies can be used to train a failure detection and recovery system to detect and respond to unexpected or abnormal situations, helping to improve its robustness and fault tolerance.

Generative models can also learn to generate synthetic sensor data streams that represent typical behavior during normal operation. During robot operation, any deviations from the expected patterns can be detected by comparing the generated sensor data stream with the real-time sensor data. This allows robots to identify faults, anomalies, or malfunctions and take appropriate actions to recover or mitigate the impact.

8. Automating End-Effector Design 

Many material handling tasks in manufacturing require the use of end-effectors and, depending on the task, these end-effectors need to be customized for optimal performance. In the case of picking and transporting a highly-compliant large sheet, designing a complex end-effector can take multiple days. If significant human effort is needed during the end-effector design process, the availability of humans can become a bottleneck.  It will be much better for a Generative AI system to design the end-effector and a human expert to approve the design. Generative AI is showing promise in a wide variety of design tasks and therefore it can be utilized to automate the design of an end-effector for a new task. By reducing the human time required to design new end-effectors tailored to a specific task, autonomous robots can be deployed more rapidly.

Summary 

Deploying autonomous robots in complex applications currently requires significant human effort. The availability of human resources needed to get this accomplished often emerges as a bottleneck and can cause delays in deployment. The emergence of Generative AI is offering new tools to reduce the human expertise needed to deploy autonomous robots and eliminate bottlenecks.

For more information on Robotic Solutions Click Here!

Check out the Forbes Article on this!