The Fridget Kitchens Dataset

Manipulation data from real kitchens, labelled by the hands that did the work.

Body-worn RGB-D, camera pose, motion and 3D hand pose from real home kitchens. Every step is confirmed or flagged by the person who cooked it, and the whole corpus ships as LeRobot v3.0.

Request access Licensed to select partners
21-joint 3D hands No MANO dependency
Session recording Step 4 of 9
Video 1080p · 30 fps
LiDAR depth 15 Hz · uint16 mm
Camera pose 30 Hz · T[16] K[9]
Motion ~100 Hz · 6-axis
3D hands 21 joints × 2 · per frame
Step labels stamped by the cook
Schematic — what one session carries.

How it works

Cooked, labelled, delivered.

01

Cook

A head-mounted phone, one session per cook. The camera travels with the gaze, and the work stays in frame.

02

Label

The recipe is known before the cook starts and the contributor advances it as they work — so a segment is a labelled dwell inside a step, not a guess from pixels.

03

Deliver

Packaged as LeRobot v3.0 with a Croissant card and the annotation tracks named in meta/info.json — opened with the official loader before it ships.

Modalities

Five synchronised streams. One clock.

Video, depth, pose, motion and 3D hand pose are written together on a single shared clock, so a label placed at a moment in the film lands at the same moment in every other stream.

3D hands

21 joints per hand · per frame · world frame

21 joints per hand in the camera's world frame, per frame, with a per-joint confidence and an inferred flag on every value we filled rather than measured. Solved on our own hardware from LiDAR-anchored 2D landmarks — no MANO model, no research-only licence riding into your training set.

Absent and unmeasured are different words here: a shard with no hands writes no hand columns at all, and a frame with one hand in view says so through present.

observation.hands.left.joints .confidence .inferred .present .wrist_speed

RGB video

1920 × 1080 · 30 fps · HEVC

Written in chunks as the session runs, each with its own SHA-256 and byte count, so the episode can be reassembled and verified independently.

LiDAR depth

256 × 144 · 15 Hz · uint16 mm

ARKit scene depth from the same session as the video — millimetre integers, not a normalised map, with a per-pixel confidence channel beside every frame.

Camera pose

30 Hz · T[16] + K[9]

One sample per RGB frame: the 4 × 4 world transform and 3 × 3 intrinsics, row-major, stamped on the video clock.

Motion (IMU)

~100 Hz · 6 channels

Linear acceleration and gyroscope from device motion, one sample per line, on the same clock as everything else.

Annotations

Layered labels, led by the ones nobody else has.

Most kitchen corpora are labelled after the fact by annotators watching video. Here the contributor marks each step as they work, and reviews their own film before it uploads.

Recipe steps 1 2 3 4
Failure event
Face pauses
Hand pose
Contributor trims
session start session end
step segment derived (telemetry) confirmed by the cook capture paused hand pose present withheld
Schematic of the lane layout — illustrative spans.
01

Failure events

Where the cook went wrong, and who says so. Telemetry proposes a failure window; the contributor confirms, moves or rejects it while the cook is still fresh. Both spans ship, so you can train recovery behaviour instead of a success-only imitator.

step_index span_derived span_confirmed provenance finding_code reason_codes response
02

Step and action segments

Objects carry pre- and post-state across each segment, revisits stay separate visits rather than being merged, and every record declares how its timing was derived and which vocabulary version produced its labels.

imperativeText verb verbFamily objects[] pre → post state tools[] hands motionClass checkpoint tStart / tEnd alignment vocabVersion
03

Per-step human labels

After the cook, the ask is consent plus one confirmation — nothing else is mandatory. The cut is deliberate: a shorter ask produces truer labels, the same reason EPIC-KITCHENS left its contributors undisturbed. What remains is the channel a scraped corpus cannot have: the person who did the work saying whether the cook went as intended, flagging only where it didn't.

stepIndex verdict reason note reviewedAt
04

Per-step telemetry

A per-moment subject-distance track summarised from the centre of the LiDAR frame, plus coverage for when hands are in frame — aggregated per step, so you can filter for segments actually showing manipulation at a workable range. The 3D hand pose itself lives in the frame table; these are the per-step summaries that find it.

depthMedianMm depthValidRatio handsInFrameSec handsCoverage pausedSec timer.deltaSec
05

Exclusion spans

Windows withheld from delivery ship as records inside it, so you can see exactly what was taken out, why, and how it changes the delivered duration. Every reason comes from a closed vocabulary pinned to the packager's types — an unrecognised value fails the build rather than being coerced into a generic trim.

start_sec end_sec reason session_id non_task phone_handling bystander minor

The sim gap

The best kitchen simulator can't cook.

We ran the test ourselves: installed the leading kitchen simulator, rendered its kitchens, and dissected the datasets it ships. These frames are ours, straight from the simulator — the measurement of exactly where simulation stops and this corpus begins.

Simulated robot at a stove: a rigid pan on an unlit burner and a plastic-looking cheese wedge on a plate
"Searing meat", in sim: the pan is a rigid body on an appliance with a state flag. Nothing browns, melts, boils or spills — the food never changes.
Simulated kitchen counter with a single broccoli floret on a cutting board
Prep, in sim: one rigid broccoli, flat lighting, a robot's fixed viewpoint. No knife work, no peels, no wet hands, no judgement calls.
Blurry 256 by 256 pixel eye-in-hand camera frame, the resolution shipped in simulator training data
What sim datasets actually ship: 256 × 256 px RGB at 20 fps, three fixed cameras, no depth channel — even though the simulator could render perfect depth.

Frames rendered by MnM Labs from RoboCasa365 — assets © the RoboCasa authors, CC BY 4.0.

Dimension Kitchen sim ships This corpus ships
Depth None — RGB only LiDAR metric depth, millimetre integers, per-pixel confidence
Food physics Rigid bodies; success is placement plus an appliance flag Real transformations — browning, melting, boiling, spills — the signals world-models are missing
Embodiment A teleoperated robot arm Human ego-view — natural hand strategy, bimanual, uninstructed
Environment Procedural kitchens, flat lighting, spotless counters Real homes — clutter, steam, glare, wear, and the mess of actual dinner
Ego-motion Fixed virtual cameras 6-DoF camera pose at 30 Hz + IMU at ~100 Hz, in-band with the video
Labels Scripted task language, auto-generated Step-aligned language confirmed by the person who cooked

Simulation is where kitchen policies are born — it is not where they learn what a kitchen is. Pretrain in sim. Ground it here.

Delivery

Sold as episodes. One cook, one episode, one schema.

An episode is one complete cook on a contiguous delivered timeline — trimmed windows are physically cut before packaging, and the cut map ships with the data. Episodes declare robot_type: human_ego, matching how they were filmed: worn by the person doing the work, not teleoperated.

Tier 1 · Training format

LeRobot v3.0 shard

Opens in LeRobotDataset() on lerobot ≥ 0.6 — chunked Parquet frame table, HEVC video, depth as lossless uint16-millimetre TIFF, task strings per frame, stats and a Croissant card. Every shard is opened with the official loader before it ships.

Tier 2 · Raw fidelity

Native-rate session bundle

The tracks exactly as captured, at their native rates: 15 Hz depth with per-pixel confidence, ~100 Hz IMU, 30 Hz pose, original HEVC chunks with per-chunk SHA-256 — under the same exclusion list, for teams that want the sensors before our resampling.

Anatomy of one episode

<dataset>/ ├── meta/ │ ├── info.jsonfeature schema · fps · path templates │ ├── tasks.parquetstep imperatives → task_index │ ├── stats.jsonper-feature stats + quantiles │ ├── episodes/chunk-000/*.parquetepisode index + data/video offsets │ ├── metadata.jsonCroissant card (MLCommons) + shard QA │ ├── annotations/ │ │ ├── recipe_steps.jsonlstep + action segments │ │ ├── failure_events.jsonlfailures + provenance ladder │ │ ├── step_annotations.jsonlper-step telemetry + human labels + cue/nav events (derived) │ │ ├── excluded_spans.jsonldeclared, not delivered │ │ └── clock_map.jsoncapture→delivered time, auditable │ └── provenance/<session_id>.jsonhashes · quality verdict ├── data/chunk-000/*.parquetper-frame table, depth embedded └── videos/observation.images.main/ └── chunk-000/*.mp4CFR 30 fps HEVC, one episode per file

One row of the frame table — every stream, one clock

column type what it is
observation.images.mainvideo · HEVC1080p RGB, decoded per frame by the loader
observation.images.depthimage · uint16 TIFFmetric depth, millimetre integers, lossless, embedded in the table
observation.camera_posefloat32[16]4 × 4 world-from-camera transform, row-major (ARKit)
observation.camera_intrinsicsfloat32[9]3 × 3 K matrix per frame
imufloat32[6]acceleration (G) + rotation rate (rad/s), frame-aligned
observation.hands.{left,right}.jointsfloat32[63]21 joints × (x,y,z) in metres, world frame; zeros where the hand is absent
observation.hands.{left,right}.confidencefloat32[21]per-joint solver confidence — shipped so you can filter, not trust blindly
observation.hands.{left,right}.inferredbool[21]true where the solver's priors filled a joint no 2D observation supported
observation.hands.{left,right}.presentboola solved sample exists for this hand at this frame — absent and unmeasured stay distinguishable
observation.hands.{left,right}.wrist_speedfloat32m/s over one solver tick; −1.0 is the undefined sentinel, never a measurement
observation.hands.pose48float32[48]compact view — wrist, 6D rotation and fingertips for both hands, derived from this row's joints
task · task_indexstring · int64the recipe step being executed in this frame, as the cook was told it
timestamp · indicesfloat32 · int64delivered-clock seconds + frame / episode / global row ids

The hand columns are additive and absent-by-shard: a shard with no hands writes no hand columns at all, and is byte-for-byte what it was before they existed. The annotation tracks are line-delimited JSON you can read without our code, named in meta/info.json so they are found rather than discovered.

Tell us what your policy needs to learn.

Send the tasks, the modalities and the annotation depth you are training against. Under NDA we share the full schema, a sample episode tree and a shard quality report.

Request access