LangCrazyflie | Mikael Yeghiazaryan

Project Description

The goal of this mini-project was to explore the application of artificial intelligence (AI) agents in controlling aerial vehicles. Specifically, I investigated how AI agents can be tasked with navigating aerial vehicles when provided with the necessary tools. While simple navigation tasks, such as “explore the left side of the room,” can demonstrate basic functionality, I wanted to add complexity to make the project more engaging. Therefore, the AI agent was tasked with outlining three shapes of increasing complexity: an ellipse (easy), a heart (medium), and a Lissajous curve (hard).

Method

For this project, I used a Crazyflie drone in a Vicon motion capture environment. To implement the AI agent, I utilized LangGraph, a tool that allows the construction of AI workflows using directed graphs. The architecture of the AI agent’s graph is shown below:

AI agent graph in LangGraph.

The central node of the graph is the “assistant” node, which processes user queries. The “tools” node contains the set of functions required for drone navigation, including the following:

navigate_drone_tool: A tool for directing the drone’s movement.
land_drone_tool: A tool that commands the drone to land safely.
get_drone_position_tool: A tool that retrieves the drone’s current position.
fly_shape_tool: A tool responsible for producing specific shapes and guiding the drone to trace those shapes.

Additionally, the “debugger” node is responsible for debugging the code generated by the shape generation tool. See the implementation code in my GitHub repository.

To generate and trace a trajectory, the AI agent begins by writing a Python function to parameterize the desired shape, enabling the sampling of points along it. If needed, the “debugger” node assists in debugging any flawed code. Once the code is functional, the AI agent samples 20 points along the curve and provides them to the drone for sequential navigation.

Results

In all experiments, the following system prompt was used:

You are a helpful assistant who controls a flying robot. 
Make sure you always fly at 1 meter height, unless explicitly told otherwise. 
The room has the following dimensions: x_min=-1.5, x_max=+1.5, y_min=-1.5, y_max=+1.5.
You can perform the following actions with the robot:
- navigate the robot to a waypoint
- land the robot
- obtain the robot's current position
- fly the robot along a described shape

See the results of the experiments below. See high-resolution videos of the experiments here.

Ellipse (Easy)

The videos above show Vicon-captured data from a top-down view, with the camera positioned to the left. On the left, the video displays the complete trajectory flown by the drone, while the video on the right focuses on the segment of the trajectory where the drone is executing the ellipse shape.

Video of the drone's flight segment tracing an ellipse shape.

See the conversation, including the inner communication of the graph, below:

================================ Human Message =================================

Go to the center of the room.
================================== Ai Message ==================================
Tool Calls:
  navigate_drone_tool
  Args:
    target_coordinate: [0, 0, 1]
================================= Tool Message =================================
Name: navigate_drone_tool

Status: OK
================================== Ai Message ==================================

The robot has navigated to the center of the room at a height of 1 meter.
================================ Human Message =================================

Outline an ellipse.
================================== Ai Message ==================================
Tool Calls:
  fly_shape_tool
  Args:
    shape_description: ellipse
================================= Tool Message =================================
Name: fly_shape_tool

Status: Shape flown successfully
================================== Ai Message ==================================

The robot has successfully outlined an ellipse while maintaining a height of 1 meter.
================================ Human Message =================================

Go to the center of the room and land there.
================================== Ai Message ==================================
Tool Calls:
  navigate_drone_tool
  Args:
    target_coordinate: [0, 0, 1]
================================= Tool Message =================================
Name: navigate_drone_tool

Status: OK
================================== Ai Message ==================================
Tool Calls:
  land_drone_tool
  Args:
================================= Tool Message =================================
Name: land_drone_tool

Status: OK
================================== Ai Message ==================================

The robot has navigated to the center of the room and landed successfully.

Heart (Medium)

Video of the drone's flight segment tracing a heart shape.

See the conversation, including the inner communication of the graph, below:

================================ Human Message =================================

Go to the center of the room.
================================== Ai Message ==================================
Tool Calls:
  navigate_drone_tool
  Args:
    target_coordinate: [0, 0, 1]
================================= Tool Message =================================
Name: navigate_drone_tool

Status: OK
================================== Ai Message ==================================

The robot has been navigated to the center of the room at a height of 1 meter.
================================ Human Message =================================

Outline a heart shape.
================================== Ai Message ==================================
Tool Calls:
  fly_shape_tool
  Args:
    shape_description: heart
================================= Tool Message =================================
Name: fly_shape_tool

Status: Shape flown successfully
================================== Ai Message ==================================

The robot has successfully outlined a heart shape.
================================ Human Message =================================

Go to the center of the room and land there.
================================== Ai Message ==================================
Tool Calls:
  navigate_drone_tool
  Args:
    target_coordinate: [0, 0, 1]
================================= Tool Message =================================
Name: navigate_drone_tool

Status: OK
================================== Ai Message ==================================
Tool Calls:
  land_drone_tool
  Args:
================================= Tool Message =================================
Name: land_drone_tool

Status: OK
================================== Ai Message ==================================

The robot has been navigated to the center of the room and has successfully landed there.

Lissajous Curve (Hard)

Video of the drone's flight segment tracing a Lissajous curve.

See the conversation, including the inner communication of the graph, below:

================================ Human Message =================================

Go to the center of the room. Then, outline a Lissajous curve. You may select the parameters of the curve. Keep it simple but interesting. Let me know the parameters you have selected. After that, go to the center of the room and land there.
================================== Ai Message ==================================
Tool Calls:
  navigate_drone_tool
  Args:
    target_coordinate: [0, 0, 1]
================================= Tool Message =================================
Name: navigate_drone_tool

Status: OK
================================== Ai Message ==================================
Tool Calls:
  fly_shape_tool
  Args:
    shape_description: Lissajous curve with parameters a = 1, b = 2, delta = π/2, over the range of 0 to 2π, centered at (0,0) with a height of 1 meter.
================================= Tool Message =================================
Name: fly_shape_tool

Status: Shape flown successfully
================================== Ai Message ==================================
Tool Calls:
  navigate_drone_tool
  Args:
    target_coordinate: [0, 0, 1]
================================= Tool Message =================================
Name: navigate_drone_tool

Status: OK
================================== Ai Message ==================================
Tool Calls:
  land_drone_tool
  Args:
================================= Tool Message =================================
Name: land_drone_tool

Status: OK
================================== Ai Message ==================================

The drone successfully outlined a Lissajous curve with the parameters \(a = 1\), \(b = 2\), and \(\delta = \pi/2\), over the range of 0 to \(2\pi\), centered at (0,0) while maintaining a height of 1 meter. It has now returned to the center of the room and landed there.

Conclusion

With the appropriate tools, AI agents have the potential to autonomously trace shapes that can be generated programmatically, such as those created using Python code. Additionally, alternative shape generation methods could be employed, such as mimicking a path captured by a camera. While shape tracing itself may not have immediate practical applications, it can serve as a foundational technique for more complex tasks, such as interactive drone formations. Furthermore, introducing a user-friendly interface between humans and robots opens the door for broader accessibility, allowing individuals without coding expertise to effectively utilize robotic systems.