Improve loading performance in _absolute_to_relative_idx when remapping indices#3279
Open
hello3x3 wants to merge 1 commit intohuggingface:mainfrom
Open
Improve loading performance in _absolute_to_relative_idx when remapping indices#3279hello3x3 wants to merge 1 commit intohuggingface:mainfrom
_absolute_to_relative_idx when remapping indices#3279hello3x3 wants to merge 1 commit intohuggingface:mainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type / Scope
Summary / Motivation
This PR improves the initialization performance of
DatasetReaderwhenepisodesis specified and index remapping is required. Materializing the entire column as Python / tensor objects becomes a significant bottleneck for large datasets. Use.to_numpy()is much faster.Related issues
absolute_to_reative_idxfor remapping indices to fix delta_timestamp bug #2490What changed
Replaced formatted index column loading with direct Arrow column access in DatasetReader
Optimized remapping dictionary construction for
_absolute_to_relative_idxPreserved existing remapping semantics and public API behavior
No breaking changes
Test
Checklist (required before merge)
pre-commit run -a)pytest)