Large modular logic refactoring

[Cyrilvallez]

What does this PR do?

This PR largely rework the logic we use in the modular converter. It is (hopefully) clearer and maintainable. Instead of going in all directions, adding stuff, then deleting it if not needed, we now do the following:

• visit all the modular file (record imports/functions/classes/assignments nodes)
  • create function dependency mapping
• for each import coming from another model:
  • visit the corresponding file
  • create function dependency mapping
  • update mapping with function/assignment from the modular (updated/new functions)
  • create the class dependency graph based on merged dependencies
• update dependency graph of the modular with the functions and assignments imported from the other files
• for each class recorded in the modular:
  • if inherithing from class in another file:
    • replace call to super
    • find the dependencies after the node was replaced
    • follow (updated with modular defs) dependency mapping to add all nodes
  • else:
    • only add needed imported functions (and their dependencies)
• determine the needed imports and add them

Note that we now only visit each files once, instead of potentially revisiting them multiple times due to renaming or deleting nodes at the end.
cc @ArthurZucker for the logic design

Still yet to come if the design looks good:

• Unit-tests!!!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Okay! Very nice idea to have a general ModelMapper, and have one for Modular files and one for "other" files.

Let's keep splitting the functionalities, I think creating the module should go outside the visiter, don't necessarily need a class !

Looks good in general! Great work 🔥

[yonigozlan]

Hey, so nice to see modular getting better and better! Just a heads up that I have this relatively short PR out on modular as well #34477 , which adds some functionalities needed for the ColPali PR, so I was wondering if it will be compatible with this refactoring!
Also made a comment there on adding the modular examples to the check_modular_conversion script to traced when some issues are introduced, I think it could be useful to have here also :)

[Cyrilvallez]

Hey @yonigozlan thanks for the heads-up! I added the new TYPE_TO_FILE_TYPE and slightly tweaked how I was handling Annotations, your modular_new_kwargs_model.py now behaves correctly also in this PR.
If you have any issue let me know.

[yonigozlan]

@Cyrilvallez Nice! Thank you!

[ArthurZucker]

Very very nice!
The only things missing now:

• unittests / small examples of the capabilities
• update the documentation a little bit to further explain how we do this but mostly
  To be honest we can merge this will unblock me as well 🤗

[ArthurZucker]

Suggested change

	"ProcessorKwargs": "processing",
	"ImagesKwargs": "processing",
	"TextKwargs": "processing",

MMMM I am not sure about these, we should infer their destination from AlignProcessor that uses them in the signature

[Cyrilvallez]

Are we expecting clashes in the names here? Because infering based on the class using them is not so straightforward since different classes in different file types may use them (e.g. type hints)

[Cyrilvallez]

For what I can see, all of them are only used in "processing" files as of now at least

[yonigozlan]

Hey again @Cyrilvallez @ArthurZucker!
It seems like with this new modular converter, several examples that used to work in examples/modular have now some issues. Some of these examples have functionalities that we would need for the ColPali PR, so it would be great if we could try to keep supporting them.
Also should we add the modular examples to the default call of check_modular_conversion? As otherwise, the modular examples aren't automatically updated when changes are made to modular_model_converter, which makes it hard to trace when issues are introduced

[Cyrilvallez]

Hey @yonigozlan! Thanks for the feedback! I added assignment dependency tracking to correctly follow everything (I previously thought it would never be needed, but it turns out it is in some niche usecases! And good to have it in general anyway).
Everything is now behaving nicely, both in the actual models definitions and examples (the examples were broken already btw). Could you double-check just in case? This highlights that we really need those tests soon!
I'll let @ArthurZucker decide if we want to add the examples in the check_conversion, but I added them to the --files_to_parse all anyway.

[Cyrilvallez]

• Added support for a new important use-case (see [WIP] Emu3: add model #33770): correct dispatch of fully new class. Consider the following modular.py:

    from ..llama.modeling_llama import LlamaModel

    class NewNameTextConfig(PretrainedConfig):
        ...

    class NewNameConfig(PretrainedConfig):
        ...

    class NewNameModel(LlamaModel):
        config = NewNameConfig()
        text_config = NewNameTextConfig()
        ...

we previously had no way of correctly dispatching NewNameTextConfig to configuration_newname.py, without importing it in modeling_newname.py as well as part of the dependencies (because modeling_llama.py only tell us that NewNameConfig will be imported, but nothing about TextConfig).
This is now solved, every fully new class (without exact name match e.g. LlamaConfig <-> NewNameConfig) which does not belong to the correct file type will not be added, and an import will be created instead.

[ArthurZucker]

🥳