Remove explicit product types list from the integrator

[antirotor]

Changelog Description

Removing product type constrain from the integrator allows it to integrate everything without the need to modify code or maintain product types list in the Settings. If instance shouldn't be integrated, just use already present instance.data["integrate"] = False.

Tested

• Substance Painer
• Editorial workflows
• Maya rendering
• Houdini split export/render workflow
• Nuke local/farm render

Note

Closes: #236

Testing notes:

To test it, it needs to be run through different set of scenarios to rule out that there are no instances that shouldn't be integrated. Adding simple creator to Tray publisher should allow immediate publishing (and integration) of that product type

[kalisp]

Maybe it would be a right time to add that to the documentation, most likely https://community.ynput.io/t/dcc-communication-over-websocket-code-duplication/1328 ? (And maybe to docstring of IntegrateAsset class?)

I counted these to limit integration:

• instance.data["integrate"]
• delete in instance.data["representations"]["tags"]
• instance.data["farm"]

I am thinking about good use case when to use integrate to False, maybe Thumbnail instance? (But I think that is not currently integrated because of delete tag)

[BigRoy]

@moonyuet when you're back from vacation - could you check this PR with Substance Painter. I have a feeling some of the "runtime instances" it creates might not work nicely with this. But hey, maybe it does and we are lucky!

@antirotor should we get a 'checklist' in the PR description for some hosts that were tested (and maybe listing some specific publish product types we think are 'special' and may give issues, like e.g. maya renderlayers, substance painters 'split on export' textures, render farm submissions from different hosts (both in local render and farm render mode), etc.)?

[moonyuet]

@moonyuet when you're back from vacation - could you check this PR with Substance Painter. I have a feeling some of the "runtime instances" it creates might not work nicely with this. But hey, maybe it does and we are lucky!

I tested with the code by using PBR template. All image instances are published successfully and I suppose the any instance related to "textureSetMain" would not be published (as I remember it will error out the publisher)

[LiborBatek]

Works ok in SPainter...

[LiborBatek]

As I have been testing it in maya host and render instance....I have created 3 different renderLayers and I have expected that just single publish render instance will appear instead of all 3 as usual.

Is this correct or?

other than that publishing of render instances works fine...

[BigRoy]

As I have been testing it in maya host and render instance....I have created 3 different renderLayers and I have expected that just single publish render instance will appear instead of all 3 as usual.

Is this correct or?

You will get a render instance per renderlayer - that behavior is correct. Each renderlayer also has its own settings on the right hand side. You could e.g. target a specific renderlayer to a different deadline pool or frames per task.

So yes, this behavior is correct.

[MustafaJafar]

So, if I got it correctly,
Then removing families attribute makes the integrator works by default for any instance of any product type.
This includes all existing instances in the scene and all created instances at run time.
except:

1. instances without any representations as they have nothing to integrate.
2. instances with instance.data["integrate"] = False

---

Instances that are associated with local/farm publishing:
This is needed in order not to double publish the instances.

In farm publishing plugins, we can neglect local instances by the following snippet.
Basically, we will use it in deadline plugins.

if not instance.data.get("farm"):
    self.log.debug("Skipping local instance.")
    return

In local publishing plugins, we can neglect farm instances by the following snippet.
Basically, we will use it in extractor plugins.

if instance.data.get("farm"):
    self.log.debug("Should be processed on farm, skipping.")
    return

---

So, This PR works out of the box with the current Houdini state as:

1. Render instances are marked as farm instances and they don't have representation data.
2. Marking Render as reviewable doesn't work therefore render instances won't have a review representation.
3. Instances with local/farm support e.g. pointcache have the above code snippet in their extractors and farm thus they should be fine.
4. we don't have any created instances at run time.

Regarding #328 It should leverage instance.data["integrate"] = False with the original render instance although we may not need to because the original render instance doesn't have any representations.
In local rendering it doesn't have a review representation. Created instances are the ones that have review representation.
In farm rendering it doesn't have a review representation because we only add some review flags and leave it to deadline Houdini submission to do its magic.

So, I'll just mark Houdini split export/render workflow as done.

---

A question: Should we adopt the same idea for farm plugins ? maybe we can add one more flag to determine if it's a render or not render.
farm and render => render submission
farm and not render => cache submission

[ynbot]

Task linked: AY-4138 Drop explicit product type list from integrator

[antirotor]

so, it is totally failing for me for Simple Editorial publish in Tray Publisher. Can anyone please confirm? Complaining from the range of issues - invalid folderEntity, instances without representations and so on. @jakubjezek001 maybe? Could be probably easy and fast for you to confirm.

[jakubjezek001]

It works but @iLLiCiTiT needs to fix client\ayon_core\hosts\traypublisher\plugins\publish\collect_frame_data_from_asset_entity.py in separate PR. I was testing simple Traypublisher Editorial publishing and also CLI CSV publishing.

[BigRoy]

It works but @iLLiCiTiT needs to fix client\ayon_core\hosts\traypublisher\plugins\publish\collect_frame_data_from_asset_entity.py in separate PR. I was testing simple Traypublisher Editorial publishing and also CLI CSV publishing.

@iLLiCiTiT any chance you have time to pick this up? I'd love this PR to get merged for some Houdini stuff I'm working on.
Or @jakubjezek001 do you know what roughly needs fixing so maybe someone else (e.g. me) could take a look and help out?

[antirotor]

Should be fixed in #499 let's test it again.

[antirotor]

I've tested it sucessfully in Tray publisher with Editorial Simple and in Nuke. I think we can merge it.

[BigRoy]

Appreciate the quick tests, thanks so much!