Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract features over detections rather than frames in tracker #66

Open
aaprasad opened this issue Jul 18, 2024 · 0 comments
Open

Extract features over detections rather than frames in tracker #66

aaprasad opened this issue Jul 18, 2024 · 0 comments

Comments

@aaprasad
Copy link
Contributor

aaprasad commented Jul 18, 2024

Currently in the tracker we extract the features per frame rather than over all the instances at once. In the GlobalTrackingTransformer, we've refactored it so that we stack the instances and then apply the feature extractor.

Current implementation:

        for frame in frames:
            if frame.has_instances():
                if not self.use_vis_feats:
                    for instance in frame.instances:
                        instance.features = torch.zeros(1, model.d_model)
                    # frame["features"] = torch.randn(
                    #     num_frame_instances, self.model.d_model
                    # )

                # comment out to turn encoder off

                # Assuming the encoder is already trained or train encoder jointly.
                elif not frame.has_features():
                    with torch.no_grad():
                        crops = frame.get_crops()
                        z = model.visual_encoder(crops)

                        for i, z_i in enumerate(z):
                            frame.instances[i].features = z_i

Proposed implementation:

if not self.use_vis_feats:
     for instance in frame.instances:
           instance.features = torch.zeros(1, model.d_model)
else:
     instances = [instance for instance in frame.instances for frame in frames]
     features = self.model.extract_features(instances)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant