DexHOTLearning Hand–Object Configuration Transfer
for Dexterous Manipulation

Zhewei Gui1,3* Anthony Bolton1,3* Jiyao Zhang1,2,3‡
Zeyuan Chen1,4 Qiyang Yan1 Tianyu Wang1,4 Hao Dong1,2,3†

1 CFCS, School of CS, PKU, China2 National Key Laboratory for Multimedia Information Processing, School of CS, PKU, China

3 PrimeBot4 State Key Laboratory of General Artificial Intelligence, BIGAI

Co-first Authors Corresponding Author Project lead

Paper VideoCode
Real-robot demonstrations 2× speed

One policy. No human data for training.Any hand–object configuration transfer.

01

What is hand–object configuration transfer?

Any2Any

Any source. Any target.Illustrated with the wrist held fixed.

Source

Any hand–object configuration

Recorded source configuration: a Sharpa hand holding the scanned drill, with colored finger chains and object-local axes.
Target

Any hand–object configuration

Target configuration: a Sharpa hand holding the scanned drill, with an extended finger alongside neighboring bent fingers, colored finger chains, and object-local axes.

Transfer between feasible hand–object arrangements, with each configuration jointly specifying:

Opaque colored finger skeleton over a translucent Sharpa hand and faint drill, emphasizing finger articulation.
Finger articulationThe hand arrangement
Opaque scanned drill with three colored object-local axes and a faint hand reference, emphasizing pose relative to the wrist.
Object posePosition and orientation relative to the wrist
The recorded drill hand–object configuration combines the finger arrangement and wrist-relative object pose.
Hand–object configurationA joint goal for the hand and object
02

Simulation configuration transfer

Seen-object Any2Any
80.5%40 training objects
Zero-shot unseen-object Any2Any
69.4%80 held-out objects

Any2Any success averages close and far transfers. See how pairs are sampled and evaluated

Far transfer
Policy executionPlanner target · left

Camera

Simulation rollout
Left Planner targetRight Policy execution

Selected successful transfers · choose a source and target to play.

Seen-object examples

Unseen-object examples

03

Real-world configuration transfer

Real-robot Any2Any
61.5%10 objects · 20 uniformly sampled transfers per object

Source and target hand–object configurations are uniformly sampled from each object’s validated pool, excluding near-duplicate pairs. The fingers drive the transfer with a fixed, palm-down wrist.

Seen object
Palm-down Sharpa target hand–object configuration with the scanned medicine bottle
TARGETPalm down · fingers forwardSharpa + scan · traj19

Medicine bottle

Seen objectSuccess rate: 90% (18/20)

Seen objects

Unseen objects

04

Long-horizon tasks from human demonstrations

Long-horizon task execution
90%27 / 30 successful trials
DexHOT · hand policy

Configuration targets

From pickup onward: closed-loop finger control at 30 Hz, with target updates at 1 Hz.

Separate arm controller

Wrist trajectory

The arm follows the demonstration-derived wrist motion; DexHOT controls the hand.

Six demonstration-guided robot tasks shown as four chronological frames and a final close-up of the Sharpa hand–object relationship.

Six tasks, one hand policy.

Frames progress from left to right. Cyan callouts enlarge the final visible Sharpa hand–object relationship for barcode scanning, shelf placement, block assembly, salt shaking, water pouring, and screw driving.

2× speed

Barcode scanning

Demonstration-guided task

Rotate the bottle by 180° to present its barcode to the scanner with an unobstructed hand arrangement.

Tasks

Why object pose is not enough

Object-pose goal only

2× speed
✕ Failed example

Correct object pose. Barcode hidden.

The hand covers the code, so object orientation alone does not provide a scanner-compatible configuration.

Hand–object configuration goal

4× speed
✓ Successful example

Object oriented. Barcode exposed.

The goal also specifies the finger arrangement, keeping the barcode visible to the scanner.

Original playback speeds retained; the two trials are not time-synchronized.

Barcode scanning · task detail

Expose the code, not just rotate the bottle.

A 180° bottle rotation must be paired with a hand arrangement that leaves the barcode uncovered and facing the scanner.

Only the linear barcode in the original scan is boxed, with a schematic line to the scanner-facing bottle side in the real video frame; an arrow indicates the reader direction.
The scan reveals the code; the real frame shows its orientation toward the reader. The correspondence is schematic, not a scanner decode result.

Method

Learn local transfers.
Chain configuration subgoals.

A shared hand policy learns local transfers and reaches distant configurations through hierarchical chaining.

  1. 01

    Synthesize

    Build feasible hand–object configurations across diverse objects.

  2. 02

    Learn to transfer

    Train a shared policy with synthetic local-goal configuration pairs.

  3. 03

    Chain subgoals

    Compose long-horizon behavior using a planner or demonstrations.

DexHOT pipeline: grasp synthesis and local-goal transfer training above; demonstration- or planner-guided hierarchical chaining below.

Benchmark

DexHOTBench

120 objects. 1.2 million feasible configurations. A benchmark for hand–object configuration transfer.

DexHOTBench

120Objects
Mollweide density map of object z-axis directions surrounded by diverse hand–object configurations.
1.2MConfigurations
The 40 training objects.

Training objects

40 objects from 40 categories

The 40 unseen instances from training categories.

Unseen instances

40 held-out objects from training categories

The 40 objects from unseen categories.

Unseen categories

40 objects from 40 held-out categories

DexHOTBench

Diverse configurations across three object splits.

The central Mollweide map shows object z-axis directions relative to the hand across all configuration pools; rotation about the object z-axis is omitted. Colors encode density percentile.

Lower density percentileHigher density percentile

Sample configurations. Evaluate transfer range.

One distance for the full hand–object configuration.

d is the mean Euclidean distance between 28 corresponding hand keypoints, measured in the object frame.

One metric jointly captures differences in:

  • Object position
  • Object orientation
  • Hand joint arrangement

Object position and orientation are relative to the wrist.

  1. 01

    Validated configuration pool

    Feasible hand articulation and wrist-relative object poses, generated for each object.

  2. 02

    Sample source → target pairs

    1,000 fixed pairs per object and distance range; each pair is attempted once.

  3. 03

    Measure configuration distance

    Compare corresponding hand keypoints in the object frame using the same metric across all 120 objects.

Local evaluation

Neighbor

20 < d ≤ 40 mm

Within the local-goal training band.

Any2Any · close

Close transfer

20 < d ≤ 70 mm

Extends beyond the training band.

Any2Any · far

Far transfer

d > 70 mm

Evaluates longer-range configuration changes.

Neighbor and close ranges overlap; their pair sets are sampled independently. Any2Any averages close and far only.

Transfer success rate (%)
Object splitNeighborCloseFarAny2Any average
Seen objects90.985.575.680.5
Unseen instances88.381.467.374.4
Unseen categories81.572.256.764.4
d ≤ 20 mm

Mean keypoint distance within 20 mm.

Maintain this distance to the final target for 10 consecutive control steps.

Citation

BibTeX

@article{gui2026dexhot,
  author    = {Gui, Zhewei and Bolton, Anthony and Zhang, Jiyao and Chen, Zeyuan
               and Yan, Qiyang and Wang, Tianyu and Dong, Hao},
  title     = {{DexHOT}: Learning Hand-Object Configuration Transfer for Dexterous Manipulation},
  journal   = {arXiv preprint arXiv:2611.05119},
  year      = {2026},
  doi       = {10.48550/arXiv.2611.05119},
  url       = {https://arxiv.org/abs/2611.05119},
}

Acknowledgements

We thank Tianhao Wu, Zihan Ding, Junhan Wang, Zimu Han and Yiming Zeng for their insightful discussions, and Hangqi Zhou and Zihao Zhao for their assistance with the hardware. This research was supported by the Beijing Natural Science Foundation (26L080330) and the National Natural Science Foundation of China (62376006).

Figure