Designing reward functions for continuous-control robotics often leads to subtle misalignments or reward hacking, especially in complex tasks. Preference-based RL mitigates some of these pitfalls by learning rewards from comparative feedback rather than hand-crafted signals, yet scaling human annotations remains challenging. Recent work uses VisionLanguage Models (VLMs) to automate preference labeling, but a single final-state image generally fails to capture the agent’s full motion. In this paper, we present a two-part solution that both improves feedback accuracy and better aligns reward learning with the agent’s policy. First, we overlay trajectory sketches on final observations to reveal the path taken, allowing VLMs to provide more reliable preferences—improving preference accuracy by approximately 15–20% in metaworld tasks. Second, we regularize reward learning by incorporating the agent’s performance, ensuring that the reward model is optimized based on data generated by the current policy; this addition boosts episode returns by 20–30% in locomotion tasks. Empirical studies on metaworld demonstrate that our method achieves, for instance, around 70-80% success rate in all tasks, compared to below 50% for standard approaches. These results underscore the efficacy of combining richer visual representations with agent-aware reward regularization.
Drawer Open
Soccer
Sweep Into
Analysis Template
Describe the two images given to you, which includes final frames with a sketch of trajectory for the goal [task description] from the MetaWorld simulation environment. The sketch image on the final frame features a simple, 3D-rendered scene. Notably, a trajectory is sketched and represented by a thin, yellow color. The trajectory’s color uses yellow to indicate temporal progress. Brighter yellow signifies the beginning of the trajectory, while darker brown represents the end, visualizing a temporal progression.
Labeling Template
Based on the text below, answer the following questions:
The goal is [task description]. The most efficient robot arm movement would involve pulling the handle straight back along the linear path of the drawer’s opening direction with a steady and controlled force.
Is the goal better achieved in Image 1 or Image 2?