Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to return indices from index based samplers. #407

Open
Algomancer opened this issue Nov 21, 2024 · 1 comment
Open

Add option to return indices from index based samplers. #407

Algomancer opened this issue Nov 21, 2024 · 1 comment

Comments

@Algomancer
Copy link

Algomancer commented Nov 21, 2024

🚀 The feature

When sampling using an indexed (or potentially time based) sampler, add an option to additionally return the sampled indices. For example in random sampling, it would return the random indices that were generated. This would just be a matter of returning the clip_start_indices within the FrameBatch.

https://github.com/pytorch/torchcodec/blob/main/src/torchcodec/samplers/_index_based.py#L153C9-L153C27

Motivation, pitch

For some of my tasks, i need to know the relative position of a particular frame within the larger context of the video, for example for positional embedding and RoPE offsetting.

I am about to implemented this in my local fork and would happily upstream it if it is desired.

@NicolasHug
Copy link
Member

Thank you for the feature request @Algomancer . That is definitely in scope and it's someone we want to provide (this is a duplicate of #246).

This would just be a matter of returning the clip_start_indices within the FrameBatch.

For consistency with the pts_seconds and duration_seconds fields, I think we would be returning the indices of all the frames, not just the clip starts. I think that would still address your use-case as you'd be able to get the clip start index simply with something like clips.indices[:, 0]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants