Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Lab B: Embodied Agent Authentication

Agent type: Vision-Language Model (VLM) agent

Goal

Study source authentication in the control loop of a simulated warehouse cart. A live qwen2.5vl:3b VLM receives a camera image, a natural-language mission, and a reported obstacle distance before selecting STOP or PROCEED.

You first impersonate a ROS 2 publisher and observe how untrusted sensor data can induce an unsafe decision. You then use the Secure Swarm Toolkit (SST) to authenticate the legitimate distance and camera sources before their data reaches the VLM agent.

The cart executes the schema-valid model action unchanged. Only after ROS stops does a separate evaluator use ground truth to judge the recorded execution. Neither the VLM nor the cart receives ground truth.

System data flow

The runtime decision and execution logs feed an independent post-run evaluator. Ground truth reaches only that evaluator, which never feeds back into the control loop. Adapted from the project’s documented data flow.

Focus

  • VLM-based perception and decision-making for an embodied agent
  • ROS 2 publisher impersonation and malicious sensor inputs
  • Source authentication and protected delivery with SST
  • Separation of source identity, sensor truth, model decisions, and post-run outcomes

What you will do

  1. Run a trusted baseline and inspect the ROS 2 graph.
  2. Implement a replacement distance publisher and trace false input through the VLM decision, cart execution, and independently evaluated outcome.
  3. Repeat the attack at seven reported distances with three trials each, keeping other inputs fixed and distinguishing invalid runs from model variability.
  4. Implement the SST channels and demonstrate both authenticated legitimate inputs and rejection of an unregistered replacement source before inference.
  5. For the CSE 598 extension, compare a legitimate red-camera baseline, a malicious green-image replacement, and the SST-protected case.

Missing, stale, invalid, or unauthenticated inputs cause a fail-closed STOP. Successful source authentication establishes who sent protected data; it does not establish that the data is physically true or that the model’s action is safe.

Learning outcomes

By the end of this project lab, you should be able to:

  • explain how unauthenticated sensor publishers can influence a VLM agent;
  • distinguish the VLM’s selected action from the independently evaluated physical outcome;
  • authenticate legitimate physical data sources with SST; and
  • evaluate both the protection gained and the limits of source authentication.

Project repository

Use the public embodied-ai-lab/embodied-agent-auth template to create your group’s private repository. The primary workflow runs ROS 2, SST, and live VLM inference in an ASU Sol GPU allocation. Complete the assigned implementation and rebuild the ROS package before running its graded attack or protected modes.