Overview

The Embodied Artifical Intelligence (Embodied AI) Labs are a core experiential component of the CSE 494/598 Topic: Intelligent and Safe Cyber-Physical Systems (ISCPS) course at Arizona State University.

Students design, implement, and evaluate intelligent agents that interact with the physical world, with a strong emphasis on:

• safety and robustness,
• timing and physical consequences,
• authentication and authorization,
• interpretability of decisions,
• and principled reasoning in cyber-physical environments.

Lab sequence

The three labs progress from language-based reasoning to visual decision-making and finally to physical action generation:

Each project combines a concrete embodied-agent system with reproducible experiments. Students connect model behavior and computer-system mechanisms to observable cyber-physical outcomes rather than evaluating the AI model in isolation.

Start labs

Open the page for your assigned project, then follow the repository’s README.md for setup and ASSIGNMENT.md for graded tasks and deliverables. The public repositories are course templates; students complete their work in private group repositories created from those templates.

Copyright © Hokeun Kim 2026–2026

Lab A: Agentic Driving Coach

Agent type: Large Language Model (LLM) agent

Goal

Place a small local LLM in the control loop of a simulated car approaching a stop sign. The coach reasons about the situation and produces driving guidance or emergency actuation while the system tracks both logical time and real inference latency.

The central question is not only whether an LLM response is sensible, but also whether it arrives in time to affect the physical outcome. Late, malformed, or unsafe responses trigger a deterministic fallback.

System topology

The driver, car, environment, hierarchical coach, inference deadline race, and planner form a closed loop. Driver commands cross 500 ms logical delays, while coach emergency actuation returns over a 200 ms delay. Click the diagram for the full-size topology from the project repository.

Focus

• LLM reasoning inside a cyber-physical control loop
• Xronos reactors, logical time, wall-clock lag, and delayed connections
• Inference deadlines, response validation, and deterministic fallback
• Reproducible comparison of model latency, output quality, and stopping outcome

What you will do

1. Build timing intuition with small Xronos reactor exercises.
2. Run deterministic rule and replay baselines.
3. Compare live local LLMs under a fixed inference deadline.
4. Analyze how driver behavior, model timing, and fallback decisions change the simulated car’s physical outcome.

Learning outcomes

By the end of this lab, you should be able to:

• distinguish logical time from wall-clock inference time, lag, and slack;
• measure latency, deadline misses, and malformed model responses;
• explain why deterministic fallback is needed in an agentic control loop; and
• connect software timing and model behavior to a physical stopping result.

Project repository

Use the public embodied-ai-lab/agentic-driving-coach template to create your group’s private repository. Read its README.md for the ASU Sol workflow and its ASSIGNMENT.md for the current tasks, deliverables, and rubric.

Lab B: Embodied Agent Authentication

Agent type: Vision-Language Model (VLM) agent

Goal

Secure a simulated warehouse cart whose VLM agent receives a camera image, a natural-language mission, and a reported obstacle distance before selecting STOP or PROCEED.

You first impersonate a ROS 2 publisher and observe how untrusted sensor data can induce an unsafe decision. You then use the Secure Swarm Toolkit (SST) to authenticate the legitimate distance and camera sources before their data reaches the VLM agent.

System data flow

Distance, camera, and mission inputs feed the VLM agent, which sends an unchanged STOP or PROCEED decision to the cart. Physical-outcome evaluation uses the actual simulated distance independently and never overrides the model.

Focus

• VLM-based perception and decision-making for an embodied agent
• ROS 2 publisher impersonation and malicious sensor inputs
• Source authentication and protected delivery with SST
• Separation of the model’s decision from independent physical-outcome evaluation

What you will do

1. Run a trusted baseline and inspect the ROS 2 graph.
2. Inject false distance data and measure when the VLM chooses an unsafe action.
3. Sweep malicious inputs to characterize the agent’s behavior.
4. Authenticate sensor sources with SST and repeat the attack.
5. For the CSE 598 extension, analyze a malicious camera source and its secured counterpart.

Learning outcomes

By the end of this project lab, you should be able to:

• explain how unauthenticated sensor publishers can influence a VLM agent;
• distinguish the VLM’s selected action from the independently evaluated physical outcome;
• authenticate legitimate cyber-physical data sources with SST; and
• evaluate both the protection gained and the limits of source authentication.

Project repository

Use the public embodied-ai-lab/embodied-agent-auth template to create your group’s private repository. Read its README.md for the ASU Sol workflow and its ASSIGNMENT.md for the current tasks, deliverables, and rubric.

Lab C: Safe Agent Actuation

Agent type: Vision-Language-Action (VLA) agent

Goal

Place a safety and authorization layer between a VLA model and a simulated ALOHA bimanual robot. A pi0-FAST policy turns a camera observation and language instruction into action tokens, which are decoded into a continuous robot trajectory before any physical risk can be assessed.

The monitor measures the decoded motion, classifies it against a configurable threshold, and requests context-based authorization through the Secure Swarm Toolkit (SST) when an action is high risk. The actuator independently verifies the protected action and either executes it or holds position.

Safety pipeline

Physical risk is assessed only after FAST decoding. Low-risk actions can follow the direct path, while high-risk actions require context-based SST authorization and protected delivery before independent enforcement at the actuator.

Focus

• VLA action generation and FAST decoding
• Motion-risk analysis in physical joint space
• Context-based authorization for high-risk actions
• Enforcement at the actuator and safe rejection of invalid inputs

What you will do

1. Inspect the relationship between VLA action tokens and decoded actions.
2. Implement and evaluate normalized joint-motion risk analysis.
3. Protect, transfer, verify, and enforce high-risk actions with SST.
4. Sweep risk thresholds to study authorization rate and latency tradeoffs.
5. For the CSE 598 extension, evaluate the security window created by session-key reuse.

Learning outcomes

By the end of this lab, you should be able to:

• explain why physical risk is assessed after action decoding rather than from token identifiers;
• classify robot motion using observed state, physical limits, and a threshold;
• enforce authentication and authorization at the actuator boundary; and
• separate motion classification, policy authorization, and physical safety when interpreting results.

All experiments run in simulation; this project does not control a real robot.

Project repository

Use the public embodied-ai-lab/safe-agent-actuation template to create your group’s private repository. Read its README.md for the ASU Sol workflow and its ASSIGNMENT.md for the current tasks, deliverables, and rubric.

Lab Format & Logistics

Format

The Embodied AI Labs balance concepts, implementation, experimentation, and reflection.

Each repository provides a student-facing README.md for setup and commands and an ASSIGNMENT.md for the required experiments, questions, deliverables, and rubric.

• Project setup Create a private group repository from the assigned public template and prepare the documented environment.

• Implementation and checks Complete the assigned components and use the repository’s checks to validate the system incrementally.

• Experimentation and evaluation Run the required scenarios, collect result artifacts, and compare timing, security, safety, or policy behavior.

• Discussion and reflection Explain failure cases, safety boundaries, physical outcomes, and the limits of the evaluation.

Deliverables

The canonical deliverables and grading rubric are in the assigned repository’s ASSIGNMENT.md. Typical deliverables include:

• concise written answers and result tables,
• completed source code and configuration,
• required experiment artifacts, and
• a submission archive generated by the repository’s submission script.

Logistics

The public repositories are templates. Create a private repository for your group, add only your project partners, and do not publish course work in a public repository.

ASU’s Sol supercomputer is the primary platform for compute-intensive project work. Follow the allocation, container, model, and cleanup instructions for your specific project; they are not interchangeable between repositories. Do not run compute-intensive work on a Sol login node.

The course LMS remains authoritative for deadlines, group assignments, and submission. Before uploading, inspect the generated archive and confirm that it contains the current code, answers, and required results.

Contributors

The Embodied AI Labs for CSE 494/598 Topic: Intelligent and Safe Cyber-Physical Systems at Arizona State University are being developed and maintained by:

• Hokeun Kim
• Marten Lohstroh

Additional contributors may be added as the lab sequence evolves.