Decision requested

Adopt one viewer-facing trajectory package contract for recordings produced by agent-authored behavior scripts. The current proposal is the versioned palatial-web-replay-v1 package. Producers should export into this contract rather than introduce a different playback format for each behavior.

“One output type” means one logical package and schema, not one physical file. A package may contain a manifest, binary samples, meshes, rig files, and videos, all referenced and validated by the manifest.

The Viewer should play recorded results; it should not execute agent scripts.

Proposed flow

Asset and behavior script → simulator validation → record time-sampled results → validate target bindings, timebase, units, and resources → export one replay package → register the take → select and replay it in the Viewer.

What the current PRs implement

Backend #4347 and frontend #3065 propose an API-backed recording flow using palatial-web-replay-v1. A take contains a replay.json manifest, float32 motion data, and referenced resources such as STL meshes, a hand URDF, and camera video.

The current Viewer support is narrower than the full set of channels discussed:

Channel Example recorded data Current Viewer support
transform Per-frame position and orientation for an asset part or rigid body Rigid part poses
joint Hand pose and named joint angles; hinge or slider values Hand rig; generic asset joints need confirmation
mesh_points Per-frame mesh vertices and, if needed, topology changes Not supported; meshes are static
force_contact Force or impulse vectors, contact point, and target body Not rendered
state_event Grasped, opened, pour started, success or failure Optional frame markers only; no state timeline
camera Synchronized video, camera pose, and intrinsics Side/top video clips

These PRs are still open and not deployed. The Viewer replays reconstructed recordings; it does not load the chair sample’s raw USD animation directly.

USD sample and format boundary

The chair replay is a useful test case, but its time-sampled USD transforms and cushion deformation do not load directly in the current player. To keep one output contract, producers should export these results into typed channels in the canonical package. Preserving the cushion deformation requires a mesh_points channel and an agreed topology policy. A separate USD playback path would create a second Viewer output path.

Recommended schema direction

Keep one versioned package envelope and make channel types explicit. Each channel should identify its type, target part or prim, encoding, interpolation, units, and sample range. Shared take metadata should define the asset, take and revision IDs, frame count, sample rate, coordinate system, and provenance. Resource references should include paths, media types, sizes, and hashes.

For deformable geometry, confirm whether topology stays fixed during a take. For forces and contacts, define the coordinate frame, units, application point, target body, and sampling rate. Missing or unsupported optional channels should not block playback; the Viewer should identify unsupported channels clearly.

Origami audit and readiness

The backend audit reports archive, binary, hash, and hand-rig checks passed for all 24 recordings. Seventeen takes map to six published assets. Six blue-lid can takes have an Asset ID whose public endpoints return 404; one Timer take has no current asset/revision binding. The Timer should not be attached to a newer model by name.

No live import has happened. Resolve or explicitly exclude those seven takes before batch import. The frontend’s local audit rendered all 24 captures, but the Timer was an unbound renderer fixture; this does not confirm Library admission or deployed playback.

Pipeline team review — please confirm