What we need to deliver

Agree on one viewer-facing trajectory package for Origami and future robot or simulation runs. The current proposal is palatial-web-replay-v1.

The first Origami rollout should use recorded transforms, joints, and camera data. The Viewer plays the recorded results; it does not execute behavior scripts. Future channels such as mesh deformation, forces/contacts, and behavior-state events should extend the same package contract instead of introducing separate trajectory formats.

“One output type” means one logical package and schema. A package may still contain a manifest, binary tracks, meshes, rig files, and video resources.

Producer-confirmed conventions

The producer reports that the current exporter uses:

These should become explicit, agreed contract conventions. The current v1 manifest has fps and camera calibration, but the coordinate and pose conventions are not all explicit validated fields today.

Current v1 manifest example

This example shows how one asset, an animated dial joint, and a camera clip are represented. Track samples live in the referenced binary file. The hash values and resource names below are illustrative; this is a schema example, not an import-ready package.

{
  "schema": "palatial-web-replay-v1",
  "id": "origami_dial_example",
  "asset_id": "0123456789abcdef01234567",
  "frames": 2,
  "fps": 30,
  "task": "Rotate the dial",
  "binary": {
    "file": "files/motion.bin",
    "byte_length": 216,
    "sha256": "0000000000000000000000000000000000000000000000000000000000000000",
    "dtype": "float32",
    "endianness": "little"
  },
  "tracks": [
    { "name": "wrist_pose", "shape": [2, 7], "offset_bytes": 0 },
    { "name": "wrist_valid", "shape": [2], "offset_bytes": 56 },
    { "name": "hand_joints", "shape": [2], "offset_bytes": 64 },
    { "name": "body_pose", "shape": [2, 7], "offset_bytes": 72 },
    { "name": "body_valid", "shape": [2], "offset_bytes": 128 },
    { "name": "dial_pose", "shape": [2, 7], "offset_bytes": 136 },
    { "name": "dial_valid", "shape": [2], "offset_bytes": 192 },
    { "name": "dial_angle", "shape": [2], "offset_bytes": 200 },
    { "name": "dial_angle_valid", "shape": [2], "offset_bytes": 208 }
  ],
  "hand": {
    "side": "right",
    "urdf": "rig/hand.urdf",
    "pose_track": "wrist_pose",
    "valid_track": "wrist_valid",
    "joint_track": "hand_joints",
    "joint_names": ["index_finger"]
  },
  "parts": [
    {
      "id": "timer_body",
      "root": true,
      "recording_mesh": "files/timer_body.stl",
      "recording_mesh_sha256": "1111111111111111111111111111111111111111111111111111111111111111",
      "pose_track": "body_pose",
      "valid_track": "body_valid",
      "library": {
        "asset_id": "0123456789abcdef01234567",
        "part": "timer_body",
        "prims": ["/World/Timer/Body"],
        "mesh_to_library": [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]
      }
    },
    {
      "id": "dial",
      "root": false,
      "recording_mesh": "files/dial.stl",
      "recording_mesh_sha256": "2222222222222222222222222222222222222222222222222222222222222222",
      "pose_track": "dial_pose",
      "valid_track": "dial_valid",
      "library": {
        "asset_id": "0123456789abcdef01234567",
        "part": "dial",
        "prims": ["/World/Timer/Dial"],
        "mesh_to_library": [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]
      },
      "joint": {
        "parent": "timer_body",
        "type": "revolute",
        "unit": "rad",
        "origin": [0, 0, 0],
        "axis": [0, 0, 1],
        "range": [-3.14, 3.14],
        "value_track": "dial_angle",
        "valid_track": "dial_angle_valid"
      }
    }
  ],
  "videos": [
    {
      "slot": "side",
      "file": "files/side.mp4",
      "camera": {
        "K": [[500, 0, 480], [0, 500, 270], [0, 0, 1]],
        "K_for_size": [960, 540],
        "T_table_cam": [
          [1, 0, 0, 0],
          [0, 0, 1, -0.6],
          [0, -1, 0, 0.3],
          [0, 0, 0, 1]
        ]
      }
    }
  ]
}

The referenced files are inventoried and uploaded separately through the backend producer API. The example’s SHA-256 values must be replaced with hashes of the real bytes before import.

Current implementation and gaps

The existing backend and frontend work provides a foundation: