Here is the rewritten article in HTML:
Introduction
Over the past few years, text and image-based models have seen dramatic performance improvements, primarily due to scaling up model weights and dataset sizes. While the internet provides an extensive database of text and images for LLMs and image generation models, robotics lacks such a vast and diverse qualitative data source and efficient data formats. Despite efforts like Open X, we are still far from achieving the scale and diversity seen with Large Language Models. Additionally, we lack the necessary tools for this endeavor, such as dataset formats that are lightweight, fast to load from, easy to share and visualize online. This gap is what 🤗 LeRobot aims to address.
What’s a dataset in robotics?
In their general form — at least the one we are interested in within an end-to-end learning framework — robotics datasets typically come in two modalities: the visual modality and the robot’s proprioception / goal positions modality (state/action vectors). Here’s what this can look like in practice:
Contribution
We propose a LeRobotDataset format that is simple, lightweight, easy to share (with native integration to the hub) and easy to visualize.
Our datasets are on average 14% the size their original version (reaching up to 0.2% in the best case) while preserving full training capabilities on them by maintaining a very good level of quality. Additionally, we observed decoding times of video frames to follow this pattern, depending on resolution:
Metric
Given those criteria, we chose metrics accordingly.
Criteria
Size compression ratio (lower is better): as mentioned, this is the size of the encoded video over the size of its set of original, unencoded frames.
Loading time
Decoding time: Impacts training time.
Quality
Quality: Impacts training accuracy.
Compatibility
Compatibility: Impacts the ability to easily decode the video and visualize it across devices and platforms.
Metrics
We chose metrics accordingly:
Size compression ratio (lower is better)
Load times ratio (lower is better): this is the time it takes to decode a given frame from a video over the time it takes to load that frame from an individual image.
Quality (mse: lower is better, psnr & ssim: higher is better)
For quality, we looked at 3 commonly used metrics:
MSE (lower is better)
Average Mean Square Error (lower is better): the average mean square error between each decoded frame and its corresponding original image over all requested timestamps, and also divided by the number of pixels in the image to be comparable across different sizes and resolutions.
PSNR (higher is better)
Peak Signal-to-Noise Ratio (higher is better): a measure of the ratio of the peak signal amplitude to the root-mean-square noise amplitude.
SSIM (higher is better)
Structural Similarity Index Measure (higher is better): a measure of the degree to which the decoded frame matches the original image.
Performance
We validated that this new format did not impact performance on trained policies by training some of them on our format. The performances of those policies were on par with those trained on the image versions.
Policies
Policies have also been trained and evaluated on AV1-encoded datasets and compared against our previous reference (h264):
Diffusion policy on pusht dataset
Figure 1: Training curves for Diffusion policy on pusht dataset
ACT policy on an aloha dataset
Figure 2: Training curves for ACT policy on an aloha dataset
Frequently Asked Questions
Q1: What is a LeRobotDataset format?
A1: LeRobotDataset format is a simple, lightweight, easy to share and easy to visualize dataset format designed specifically for robotics.
Q2: What are the criteria for evaluating a dataset format?
A2: The criteria for evaluating a dataset format are: size compression ratio, loading time, quality, and compatibility.
Q3: What are the metrics used to evaluate the LeRobotDataset format?
A3: The metrics used to evaluate the LeRobotDataset format are: size compression ratio, load times ratio, and quality metrics such as MSE, PSNR, and SSIM.
Q4: Has the LeRobotDataset format been validated?
A4: Yes, the LeRobotDataset format has been validated by training policies on it and comparing their performance to those trained on the image versions.
Q5: Are there any future plans for the LeRobotDataset format?
A5: Yes, there are plans to expand the LeRobotDataset format to include more advanced features and to explore the use of video encoding with depth maps.
Conclusion
In conclusion, LeRobotDataset format is a simple, lightweight, easy to share and easy to visualize dataset format designed specifically for robotics. It has been validated through its performance on trained policies and shows promising results.
