Profile Manager x QGIS Deployment Toolbelt: an impacting contribution proposal

[Guts]

Hello,

I am the main contributor to the QGIS Deployment Toolbelt (QDT) project, a large part of whose functional scope is based on QGIS profiles.
In particular, the tool introduces the concept of a profile descriptor in the form of a profile.json file.

As already mentioned in a mailing list, thank you for your plugin, which we recommend to QDT customers for preparing their profiles.

QDT is still a young project and by now we've reached the point where the need to facilitate profile editing is now paramount.

Developing a QGIS "QDT" plugin seems natural, but at Oslandia our rule is to give priority to existing projects before reinventing the wheel, even if it means making the effort to find a consensus and make compromises.

Would you agree to us investing in your plugin and integrating mechanisms linked to our development standards and QDT?

In concrete terms, this would involve :

• changing the license to make it compatible with that of QGIS/Qt
• industrializing the plugin (CI/CD, tests, linter, formatter...) and adding contributing guidelines
• autogenerated documentation hosted on GH Pages
• enable profile.json generation from the plugin interface
• enable profile.json import from plugin interface
• include a rules editor
• release it as stable

This list is not exhaustive, and the idea is not to do everything at once, but to work with atomic PRs (as far as possible since formatting/lint related work always impact a bunch of files).

This implies a consequent workload for you in reviewing PRs and discussing implementations details.

---

Roadmap

• apply development guidelines and enforcing them using git hooks (through pre-commit) - Tooling: add git hooks (through pre commit microframework) #11
• build and publish documentation on GH Pages
• add CI/CD to check against development guidelines
• add CI/CD to package the plugin on every commit on main branch
• add CI/CD to compute translations
• add a tab to export an existing profile to a new folder using the QDT formalism - feat(qdt export): initialization of qdt export #24
• add ability to update an existing QDT folder by loading it before
• add option to include a default gitignore file and customize it

[kannes]

This sounds very nice and I will try to get the OK for it next week. QDT is a fantastic initiative and we are honored that this plugin is useful for you too. :)

Would you have resources to "take over" my old unfinished PR #8 ? Otherwise I would prefer to find time to get that in a mergeable state first and merge it. It contains some substantial refactoring and fixed many bugs iirc. Unfortunately I had to jump into other projects before getting it done. If I had known that, I would have worked in more sane and atomic small steps... For us that would be enough to drop the experimental flag.

License: Oh ouch, I guess the MIT license is too "open" for a QGIS plugin and we must always stay with the GPL?

[kannes]

Sorry for the late reply! We are happy to collaborate :)
Your proposed chances and additions sound fantastic.

[Guts]

Hello @kannes

It's my turn to apologize for the delay in responding.

First of all, my bandwidth was completely eaten up by other projects. I wanted to wait until I had some idea of how to proceed and how to finance all this.

We've now isolated a small budget and some functional requirements. We're about to get started.

My colleague @jmkerloch and I are about to create some PR in next days/weeks.

QDT is a fantastic initiative and we are honored that this plugin is useful for you too. :)

Thanks!

Would you have resources to "take over" my old unfinished PR #8 ? Otherwise I would prefer to find time to get that in a mergeable state first and merge it. It contains some substantial refactoring and fixed many bugs iirc. Unfortunately I had to jump into other projects before getting it done. If I had known that, I would have worked in more sane and atomic small steps... For us that would be enough to drop the experimental flag.

Is this branch functional or really far from a mergeable state? I would prefer to start from your more recent work because I don't like tue idea to throw it. It has been produced, so let use it. For example, I've changed the base branch to clean2023 for the first PR (introducing git hooks).

If you prefer, we can start from main and cherry-pick some interesting commits only from clean2023.

[jmkerloch]

Hello @kannes

I will work with @Guts to make profile-manager the perfect match with QDT for a set of perfect tools for QGIS profile deployment.

I just checked the #8 MR and I think this should be merged before. This is a nice refactoring work.

I will start from this branch and the first think I want to introduce is CI/CD for automatic release creation.

I will then add documentation generation for GitHub Pages.

[kannes]

Awesome!

I will check that old #8 branch of mine on Wednesday and get it merged. It's definitely a better base than what is in the main branch so basing the PR on it is perfect.

[Guts]

Hey @kannes,

How was it in Bratislava? Not too rainy ☔?

@jmkerloch has started working on, see his first PR: jmkerloch#2.

I've updated this issue description to list roadmap items.

[kannes]

Sorry for the late reply. Unfortunately had I found that #8 is more of a mess than I remembered...

In my tests it did not actually transfer data sources or bookmarks (and probably more) between profiles anymore. I tried to fix it but quickly ran into QGIS-profiles-are-a-mess fatique again and am not sure when I will have proper time and focus for fixing. I remember why I started collecting profile data for automating testing back then. :]

If it is ok with you, I would still be fine with merging it into the main branch anyways so anything you do is already based on it.

That would mean that main would not be usable for a while. The current main version also has its issues (silently failing, not discovering many things, etc) so it is not thaaat much worse. It definitely needs work before a new release and especially before removing the experimental flag. I wish this would have had more rigorous testing and QC when it was initially developed.

I might be able to dedicated two full days on this end of October but can't promise.

[Guts]

No worries @kannes, thanks for taking time to review and follow our work.

After discussion with @jmkerloch, we are Ok to merge your branch into main and start working from it. Don't worry about the "mess", we have 🧼 and 🧹!

I might be able to dedicated two full days on this end of October but can't promise.

Is it time dedicated to review our PRs and merge it or to develop?

[kannes]

Done! 🥳

I might be able to dedicated two full days on this end of October but can't promise.

Is it time dedicated to review our PRs and merge it or to develop?

For fixing those bugs if they are still alive then. My goal for your PRs is to merge them asap so that you are not held back. I will only have time for cursorily reviewing so please don't rely on me for quality assurance ;)

[Guts]

Ok noted. It's really cool from you to trust us. We'll try to do our best!

[kannes]

I have now given @Guts Maintainer and @jmkerloch Writer rights to the repo to accelerate you :)

Thank you for working on this and all the great improvements you have added so far!

I will try to keep checking PRs roughly and at least every other day, but feel welcome to decide on your own that "this is good and it should get in". There probably isn't anything you could do that would go against my/our intentions, so 🚀 !

[Guts]

Woa thanks @kannes for trusting us 🙏.

[Guts]

Hi @kannes

With #24 merged, we would like to release a new plugin's version so i would like to check up last details with you:

• did you set up your plugins.qgis.org credentials in repository actions secrets as discussed in add(ci): GitHub workflow to package plugin on CI/CD #22 (comment)? My access level is not enough to see it but if you need help, tell me, a quick video call can preserve health from CI sick-generator!
• do I have the rights to create and push a git tag to origin/main?

[kannes]

With #24 merged, we would like to release a new plugin's version so i would like to check up last details with you:

I'd be a bit scared of a new public release as currently things are more broken than before (since my half-finished refactoring was merged). Could you check if at least the data source transfer is a quick fix (no matter how dirty for now)? Otherwise I'd be able to work on it October 24/25 and hopefully at least do a quickndirty fix.

• did you set up (...)

Not yet, but on it! I realised that the plugin is still registered under an ex-colleagues account and am trying to get that sorted out... At this opportunity I am also setting up a technical account for our plugins' ownership. I hope this should be sorted out this week.

• do I have the rights to create and push a git tag to origin/main?

You are at maintain level so I would be surprised if not!

[Guts]

I'd be a bit scared of a new public release as currently things are more broken than before (since my half-finished refactoring was merged)

I understand your point but I'm talking about a version flagged as beta in the experimental channel and I'm a pretty fan of getting ASAP end-user feedback after a refactoring. Have I managed to reassure/convince you? 😸 (yes, I invoke a cat to have more chances).

Could you check if at least the data source transfer is a quick fix (no matter how dirty for now)?

Is there an issue tor this one?

Not yet, but on it! I realised that the plugin is still registered under an ex-colleagues account and am trying to get that sorted out... At this opportunity I am also setting up a technical account for our plugins' ownership. I hope this should be sorted out this week.

Ok, thanks for following up our active work and sorry to eat your time when you did not plan it!

[kannes]

I've gotten ownership and set up the secrets :)

Have I managed to reassure/convince you? 😸 (yes, I invoke a cat to have more chances).

You'rrrrr right, let's do it.

Is there an issue tor this one?

Nope, didn't have time for that, sorry.

[Guts]

I've gotten ownership and set up the secrets :)

Yipee! welcome aboard command center boss! 😁

You'rrrrr right, let's do it.

Okay, I push a tag then. Time to really check the CI/CD 🤞!

FYI I'll be off from tomorrow until the 29th October.

[Guts]

@kannes 😢 sadly, it failed during the upload to plugins.qgis.org. Would you accept to add me (OSGeo nickname: geojulien) as maintainer on https://plugins.qgis.org/plugins/profile-manager/ until I can test it locally, please?

[Guts]

Note that the release has been generated with notes and assets as expected: https://github.com/WhereGroup/profile-manager/releases/tag/0.5.0-beta1/

[kannes]

@kannes 😢 sadly, it failed during the upload to plugins.qgis.org. Would you accept to add me (OSGeo nickname: geojulien) as maintainer on https://plugins.qgis.org/plugins/profile-manager/ until I can test it locally, please?

Sure, done!

[Guts]

Ok I get it. The actual plugin's package name is profile-manager and we set profile_manager to be Python compliant. Do you know if it's possible to change for the second one?

[Guts]

To add verbosity to my proposal of renaming downstream, it's a blocking issue since:

• we worked for a project quite fully PEP8 compliant and use absolute imports inside the plugin
• if we change back to profile-manager it will break the imports and tooling

[Guts]

Downstream issue: qgis/QGIS-Django#475

[kannes]

Oh poop... That's an issue I also did not have on my mind.

I don't see an option to change something on the plugins site. Does it fail because the uploaded ZIP is "_" while the site/API expects "-"?

As last resort I think we could deprecate the old https://plugins.qgis.org/plugins/profile-manager/ and go for https://plugins.qgis.org/plugins/profile_manager/ . The switch from experimental to non-experimental could be a good moment. I'd have to check how wide-spread the "-" URL has spread already though.

[Guts]

Maybe you can send an email to require renaming? Since the plugin has never been published outside the experimental channel, it can be acceptable.

Or, as you suggest, we publish a new version on profile-manager which does nothing else than warn the end-user about the new plugin name?

[Guts]

My concern here is to prevent you losing your end-users and maybe customers.

[kannes]

I sent a mail to qgis-dev, let's see what the options are: https://lists.osgeo.org/pipermail/qgis-developer/2024-October/067083.html

[kannes]

I got no feedback of known active users of the existing plugin so risk of breakage is reasonably low.

https://lists.osgeo.org/pipermail/qgis-developer/2024-October/067084.html Lova:

At the moment, it's not possible to change a plugin's directory name (package_name) even through the administration page.

This feature was previously requested with a PR, but it was closed due to potential client-side issues.

https://lists.osgeo.org/pipermail/qgis-developer/2024-October/067089.html Me:

I think we will simply publish a new version of the old plugin to display a message to our (few) existing users and create a "new" plugin for the new version.

https://lists.osgeo.org/pipermail/qgis-developer/2024-October/067090.html Lova:

I think it's better to rename the old plugin first if you want to keep the same name for the new one, as uploading a new plugin with an existing name is not allowed.

It seems like the best way forward is to:

• Rename the old entry on plugins.qgis.org to "Profile Manager (Deprecated)" or similar via Lova or other people with appropriate access
• Release a new (experimental) version of "profile_manager" as new plugin named "Profile Manager" on plugins.qgis.org
• Release a new version of the old codebase ("profile-manager") with a warning message (in the messagebar?) saying that development is stopped for that particular plugin repo entry, that it is called that new name now and that for updates people need to install the "new" Profile Manager plugin. For this we could simply modify the existing .zip, so functionality of that old version stays exactly the same in case people actually use it.

I'd also like to use this opportunity to:

• Rename the GitHub repo to "profile_manager" for consistency

What do you think?

[Guts]

Hi Hannes,

Thanks for the communication work on the dev list. I replied to publicize the issue because the QGIS Django project is a bit unresponsive since they have more important things to worry about (notably the new site https://qgis.org).

I completely agree with your proposals. Just one addition: I suggest declaring the new plugin as a dependency of the old one using plugin_dependencies in the metadata.txt so that the end user is prompted to install the new version automatically.

Similarly, in the info message dialog, you can add a button to open the extension manager directly. Potentially with a search already completed?

[Guts]

hey @kannes,

I'm back. I'll have a look to your PR tomorrow. Regarding this topic of pushing a last version of actual plugin, who's in charge, you or us?

[kannes]

Sorry, was a bit carried away and spent some personal time on a crazy #34 :)

I won't have any significant work time in the next two weeks but I think it should be doable.

Timeline

1. ✔️ Rename https://plugins.qgis.org/plugins/profile-manager/ to "Profile Manager (Deprecated)"
2. ✔️ Rename this GitHub repo to https://github.com/WhereGroup/profile_manager/
3. Create https://plugins.qgis.org/plugins/profile_manager/, named "Profile Manager" with a (still experimental) 0.5.0 release of the latest state from this here repository (best with Big refactoring number 2 #34 merged), pushed via the awesome CI/CD
4. Manually upload a final old version to https://plugins.qgis.org/plugins/profile-manager/ with the message bar message + dialog as above (added simply outside Git) and:
   • name="Profile Manager (deprecated)"
   • version=0.31.1
   • deprecated=True
   • plugin_dependencies=Profile Manager

Does that look correct and free of conflicts? Then I would start by requesting the name change.

Final old version

I prepared a final old version with a message bar message plus instructions dialog for the users:

https://github.com/WhereGroup/profile-manager/tree/final_old_version

Are you sure that plugin_dependencies=Profile Manager would work like that?

[Guts]

Your timeline sounds right, let's deprecate the old man and go younger 🚀!

Are you sure that plugin_dependencies=Profile Manager would work like that?

I made a quick try and it should work:

[kannes]

Excellent! I sent a request to Lova for the renaming.

[kannes]

The renaming of the plugin in the plugins repo is done: https://plugins.qgis.org/plugins/profile-manager/

I have also renamed this GitHub repo now, so we have a dashing ( ;) ) consistency right from the beginning for the new "_" era.

Can you make the initial "profile_manager" release on https://plugins.qgis.org/plugins/ and add me as maintainer? I haven't look at all the fancy CI/CD stuff yet so I think that would be easiest. Current state in main as 0.5.0 I would say.

Once that is out, I would manually bundle the final deprecated version and upload that.

[Guts]

The renaming of the plugin in the plugins repo is done: https://plugins.qgis.org/plugins/profile-manager/

I have also renamed this GitHub repo now, so we have a dashing ( ;) ) consistency right from the beginning for the new "_" era.

🥳 Bravo !

Can you make the initial "profile_manager" release on https://plugins.qgis.org/plugins/ and add me as maintainer? I haven't look at all the fancy CI/CD stuff yet so I think that would be easiest. Current state in main as 0.5.0 I would say.

Done. It seems that credentials which are stored in the CI are those of https://plugins.qgis.org/plugins/user/jkroegerwg/admin. Can you get in touch with him/her to get the plugin's authorship?

[kannes]

It seems that credentials which are stored in the CI are those of https://plugins.qgis.org/plugins/user/jkroegerwg/admin. Can you get in touch with him/her to get the plugin's authorship?

I added you as collaborator, will add our company account some other day.

---

Yay, it's waiting for approval 🥳

[Guts]

[Guts]

The moment to close here has come I think. We've achieved the most of what we announced here and it's quite satisfying. We're not abandoning the ship forever and we'll try to have new opportunities to work on this cool plugin.

Thanks again @kannes for your welcoming behavior, it's such a pleasure to work with you!

[kannes]

Likewise! Thank you for the great collaboration!

See you guys at a FOSS4G event some time 👋