Pype & Avalon - let's sync the cores

[mkolar]

First of all, we would like to say that we truly appreciate the community efforts and a huge part of our current state is owed to the first years of hard work by Marcus and the community he had built around the Avalon and Pyblish projects.

Based on multiple conversations on gitter and a few other places over the past months, we want to propose a possible solution to getting Pype fork of avalon-core and upstream repository in sync once again. I’ll try to explain the current situation, the functional differences and our proposal.

Where we are

Pype’s fork of avalon-core is currently just over 1000 commits ahead of the upstream and a few commits behind (33 at the time of typing this) and we’ve gone through over 200 internal PRs in the past 2 years.

Pype fork of avalon-core is deployed in production at 10 studios with more waiting for availability, but our team is spread quite thin as, apart from supporting our clients, we are also busy working on pype 3.0, which is a major refactor and architectural change to our current code base.

There are plenty of reasons why not that many of our changes made it to the core along the way, but we feel that these are the main ones
Our development is a lot more dynamic than that of the upstream. We are directly funded by our users and very often have to react extremely quickly to production requirements.
On the other hand, thanks to the “fail often, fail early” approach, we believe that we arrive at desired production solutions through iteration and improvements quicker, rather than trying to come up with the “perfect” solution from the start.
Resulting from that, we saw a fairly slow review process of our PRs, which naturally resulted in less push from our end.
Complaints about lower code standard on pype side compared to avalon-core.

There is, of course, a variety of changes in all of that but, really it boils down to this list.

Feature Differences

Master Versions - Option to publish versionless on top of standard incremental versions. We make a “master” version that essentially just points to the latest published explicit version. Scene inventory then allows the artist to switch between explicit and master versions of the subset.

Thumbnails in loader - Mechanism that allows the studio to publish thumbnails and have them loaded back into the loader. Basic disk resolver for the thumbnails is provided, but configs can extend this with more resolvers if there is a preference to store thumbs differently. For instance, serving them from url or database.

Library Loader - essentially the same as a standard loader, but allow an artist to import content from a different project that has been tagged as library

More robust tasks - we tweaked the schema for tasks on the project and assets to be able to store task type information and potentially any other metadata, like task short codes. The key reason for this addition was the ability to filter on various places based on the task type, which is more consistent than just the task name. Type and short codes could of course also be used in path templates.

Faster asset switcher - running Avalon on large CG scenes it was becoming impossible to switch any asset representations due to the way they were queried from DB, we had to change this to be able to run some live productions. That, however, introduces behaviour that doesn’t match 100% the original, so this piece needs to be solved still.

Multiselection loader - see #550

IO tweaks - we can use functions from IO module in different contexts when needed. Very important for example for the library loader.

Hosts Without heavy pype dependency

Photoshop - Production tested, uses Websocket server that would have to be ported to avalon from Pype.

Harmony - Production tested, in the last phase of major refactoring for much more stable and cleaner code after running the current one (similar to the existing PR) in the production for past few months, and gathering tons of feedback. ynput#211

Blender - Limited production testing, Used on pre-production of a large project, abandoned for reasons outside of pipeline scope. The base implementation was stabilised and is ready for configs to extend it with plugins, loaders and such.

Unreal Engine - Beta stage. Limited testing was done, but funding was cut after very basic implementation was finished.

TV Paint - Beta stage, currently in active development. Production testable beta should be out within a week or two. ynput#213

After Effects - in the making. A few weeks from Alpha stage

Features and host with stronger Pype dependency

Compatibility or portability of these to the vanilla core might be difficult or questionable

Hiero/NukeStudio - Production tested, but with a fair amount of pype dependency. Currently lives fully in the pype repository. That being said the base would certainly be transferable to avalon-core.
Family pre-selection in Creator - uses simple mapping between task names and families to pre-select a most likely family for publishing when opening a creator. We use our preset system to load the mapping dictionary.
Project anatomy - probably the only completely custom part that fully overwrites a core functionality. Fairly early on, we bypassed the avalon template system for a lot more robust system of project templates and roots. We support optional keys, template groups, limited nesting of templates and most importantly multiple roots for a single project, so renders can go on a separate file server from the work files for example.

Where to go from here?

We discussed this at length internally and to some degree with other avalon-core maintainers and feel that the best chance of getting aligned is following.

• As a community, we should discuss the individual features and their implementation design we chose. We are happy to explain why we chose particular solutions if there is interest.
• Once the functionality is agreed, we’ll prepare an avalon-core version, cleared of any pype references, that should be testable with other configs.
• This code should go pass through review, however, the PR would be massive, possibly unmergeable. So hopefully at least functional testing (it doesn’t make much sense for us to review our own code).
• We propose this version of Avalon, to become Avalon 6.0

Why 6.0? Some of the changes are small, some are larger. Truth be told, we are unsure if we are able to guarantee 100% backwards compatibility with this kind of merge and bumping to a new major version would allow a cleaner slate.

Even though we would probably still always be somewhat ahead of the upstream, merging back from pype fork would be trivial as most of the necessary structural changes to support our clients are already in place and are production-proven (at least on our end).

We are planning on releasing pype 3.0 hopefully early January and it would be absolutely amazing to be able to continue building it on top of a fresh Avalon 6.0 release. And let other configs benefit from the push that we’ve done on our end too.

Some info for context:
Our avalon fork: https://github.com/pypeclub/avalon-core
Pype config: https://github.com/pypeclub/pype
Documentation (albeit not super complete): https://pype.club/features

[mkolar]

Not sure if @BigRoy, @mottosso, and @tokejepsen have seen this, so this is just a cheeky ping.

[tokejepsen]

I'm stepping out of the conversation as we using Pype, so I'm too biased. Would love to see the cores merge, but we would probably stick with Pype unless there are more development on Avalon for some reason.

[davidlatwe]

How about creating a branch e.g. 5.x for preserving current master, and move Pype cleaned fork into master ? Like Python 2.x & 3.x

[mottosso]

Not sure if BigRoy, mottosso, and tokejepsen have seen this, so this is just a cheeky ping.

Thank you! I had indeed not seen this. But yes, this is great. What I would like to do is merge this as-is, whilst breaking as little as possible of those using "vanilla-avalon".

Ultimately, you are using this not I, so you will be in a much better position to determine the benefits. But from a high-level perspective, I think we could all benefit from unity in the development efforts, and Pype currently has the strongest momentum. Once we're done, I'd like for that momentum to continue from here as the official momentum. In an ideal world, Pype would represent the bare minimum of changes to Avalon that reflect the bespoke and unique character of the Pype clientele, where we'd all benefit from the vast majority of improvements to Avalon as a hole. That was the vision initially and one I think can still come true.

So, how about we make a PR such that you can physically run it and get some idea of what breaks? It may not even be necessary to ensure nothing breaks before we merge, if we follow @davidlatwe suggestion of making a tag we can merge a 6.0 (after tagging the current state as the current 5.x release it is) and rebuild the world from the ashes we're left with. I'd certainly not wish for anyone to sift through the many changes this PR is bound to make. xD

Ideally, "vanilla avalon" users would be able to make the switch as-is, deploy it into production and suffer only a bare minimal of non-essential bugs and missing features. From there, it should become obvious what isn't working and come up with a quick solution. A few weeks of that and this could all be in the rear mirror with a brighter future ahead.

I'd also like for @mkolar and c/o to carry on development here, if possible. The caveat and potential issue being that changes must not only benefit Pype but everyone, something that is understandably tricky given you are not seeing a direct financial incentive of implementing things for others. However, there is indirect financial incentive in that you get back what you invest from others developing things for you (and everyone else). So hopefully the detour turns into a de-detour over time and is ultimately worth it.

Ready, set, ..?

[BigRoy]

How about creating a branch e.g. 5.x for preserving current master, and move Pype cleaned fork into master ? Like Python 2.x & 3.x

I think releasing and maybe preserving a 5.x branch for the time being would be good. Even if only for a month or two.

Not sure if BigRoy, mottosso, and tokejepsen have seen this, so this is just a cheeky ping.

Thanks for that! I had seen it but couldn't find the time to initially reply.

This write-up is excellent and I'm very much in favor in merging as much as possible, and thus doing the switcheroo.

I'd propose to:

• Set up a PR or a branch that we, the vanilla-Avalon users, could pull and run. If specific settings are required for that to run I'd love to hear what exactly is needed so we can update any relevant documentation and test things out on our end.
  Or, would it be that I should be able to pull pypeclub/avalon-core and use it out of the box?
• Once it feels "safe", meaning it runs for vanilla-Avalon users, for about 3 studios/people I'd replace the master branch and leave the old vanilla-avalon 5.x in its own branch for those who might need to quickly access that.

Then we should be good to go.

I'd still love to see a PR to the current master branch just so I could quickly browse the code changes, even if massive amounts.

[BigRoy]

Set up a PR or a branch that we, the vanilla-Avalon users, could pull and run. If specific settings are required for that to run I'd love to hear what exactly is needed so we can update any relevant documentation and test things out on our end.
Or, would it be that I should be able to pull pypeclub/avalon-core and use it out of the box?

@mkolar Is this something you could provide?

[mkolar]

@BigRoy sorry for the silence. we were finishing this baby PYPE 2.14.0 and a bit of a client push.

@mkolar Is this something you could provide?

That is the plan. We're full steam ahead on pype 3.0 which includes cleaning up dependencies. There are certainly a few days of work on making core usable outside of pype and I'm trying to find a gap where we could fit it. More guys in the team now though, so hopefully within the next 2 weeks we could have something usable.