Microsoft researchers are exploring the potential of OpenAI’s ChatGPT in instructing robots and drones without the need for extensive programming knowledge. The team from Microsoft Autonomous Systems and Robotics Research conducted research to determine if ChatGPT could apply its reasoning skills beyond text to aid with robotics tasks by considering the physical world and the environment’s context.

ChatGPT’s unique capability of generating code, mostly in Python, for robotics scenarios such as zero-shot planning and code generation, was made possible by providing the system with object-detection and object-distance data through application interfaces. The system’s extensive training in code and written text has enabled it to solve coding problems and debug programs, while also responding to dialogue and seeking clarifications.

As a language-based interface between non-technical users and drones, ChatGPT proved versatile in robotics domains, incorporating natural language and code generation models along with the flexibility of dialogue. The system asked clarification questions when the user’s instructions were ambiguous, and even wrote complex code structures for the drone such as a zig-zag pattern to visually inspect shelves.

Microsoft also tested ChatGPT’s capabilities in using a robotic arm to move blocks around and form the Microsoft logo, writing an algorithm for a drone to reach a point without crashing into obstacles, and deciding where a robot should go based on sensor feedback in real-time.

Similar to Microsoft, researchers at Google Research and Alphabet-owned Everyday Robots have also explored the potential of large language models such as PaLM or Pathways Language Model to aid with robotics challenges. These models helped a robot process open-ended prompts and respond in reasonable ways.

The application of large language models such as ChatGPT and PaLM in robotics can significantly simplify the robotics programming process, making it accessible to non-technical users. The ability to instruct robots and drones without the need for extensive programming knowledge could unlock new applications and use cases for robotics, ultimately leading to more advanced and intelligent robots.