Pass last-layer features from PET to metatrain-PET #407
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
abmazitov
reviewed
Nov 26, 2024
| @@ -61,6 +61,12 @@ def __init__(self, model_hypers: Dict, dataset_info: DatasetInfo) -> None: | |||
| self.is_lora_applied = False | |||
| self.checkpoint_path: Optional[str] = None | |||
|
|
|||
| # last-layer feature size (for LLPR module) | |||
| self.last_layer_feature_size = ( | |||
| self.hypers["N_GNN_LAYERS"] * self.hypers["HEAD_N_NEURONS"] * 2 | |||
There was a problem hiding this comment.
Is this correct? In the pure pet code, there code for calculating LLFs looks like this
last_layer_features = []
if "central_token" in result.keys():
predictor_output = central_tokens_predictor(
result["central_token"], central_species
)
last_layer_features.append(predictor_output["features"])
if self.USE_BOND_ENERGIES:
predictor_output = messages_bonds_predictor(
output_messages, mask, nums, central_species, multipliers
)
last_layer_features.append(predictor_output["features"])
last_layer_features = torch.concatenate(last_layer_features, dim=1)
last_layer_features = torch_geometric.nn.global_mean_pool(last_layer_features, batch=batch_dict["batch"])So it's not necessarily multiplied by 2, only if self.USE_BOND_ENERGIES is True
There was a problem hiding this comment.
Thanks a lot, you're right
abmazitov
reviewed
Nov 26, 2024
|
|
||
| if "mtt::aux::last_layer_features" in outputs: | ||
| ll_features = output["last_layer_features"] | ||
| print(ll_features.shape) |
There was a problem hiding this comment.
This is a leftover I suppose
There was a problem hiding this comment.
Whooops thanks!
abmazitov
reviewed
Nov 26, 2024
| else: | ||
| pet_model = PET(ARCHITECTURAL_HYPERS, 0.0, len(all_species)) | ||
| pet_model = pet_model.to(device=device, dtype=dtype) |
There was a problem hiding this comment.
This should be already merged, right?
There was a problem hiding this comment.
This should be gone now that I've updated the branch
frostedoyster
force-pushed
the
pet-ll-features
branch
from
9f37889 to
f243961
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PET can output last-layer features. This passes them through the metatrain interface so that e.g. exported models can be used in chemiscope (after #398) or the LLPR module
Contributor (creator of pull-request) checklist
📚 Documentation preview 📚: https://metatrain--407.org.readthedocs.build/en/407/