Lab B: Embodied Agent Authentication

Agent type: Vision-Language Model (VLM) agent

Goal

Secure a simulated warehouse cart whose VLM agent receives a camera image, a natural-language mission, and a reported obstacle distance before selecting STOP or PROCEED.

You first impersonate a ROS 2 publisher and observe how untrusted sensor data can induce an unsafe decision. You then use the Secure Swarm Toolkit (SST) to authenticate the legitimate distance and camera sources before their data reaches the VLM agent.

System data flow

Distance, camera, and mission inputs feed the VLM agent, which sends an unchanged STOP or PROCEED decision to the cart. Physical-outcome evaluation uses the actual simulated distance independently and never overrides the model.

Focus

• VLM-based perception and decision-making for an embodied agent
• ROS 2 publisher impersonation and malicious sensor inputs
• Source authentication and protected delivery with SST
• Separation of the model’s decision from independent physical-outcome evaluation

What you will do

1. Run a trusted baseline and inspect the ROS 2 graph.
2. Inject false distance data and measure when the VLM chooses an unsafe action.
3. Sweep malicious inputs to characterize the agent’s behavior.
4. Authenticate sensor sources with SST and repeat the attack.
5. For the CSE 598 extension, analyze a malicious camera source and its secured counterpart.

Learning outcomes

By the end of this project lab, you should be able to:

• explain how unauthenticated sensor publishers can influence a VLM agent;
• distinguish the VLM’s selected action from the independently evaluated physical outcome;
• authenticate legitimate cyber-physical data sources with SST; and
• evaluate both the protection gained and the limits of source authentication.

Project repository

Use the public embodied-ai-lab/embodied-agent-auth template to create your group’s private repository. Read its README.md for the ASU Sol workflow and its ASSIGNMENT.md for the current tasks, deliverables, and rubric.