Merge forums, OSQA, MLs to discourse?

[Matthias84]

At #149 we discuss a alternative to OSQA and some already mentioned discourse based forums, which are used by many other FLOSS communities as well.

This step might also allow us to unite different OSM community channels, which some see as ongoing problem for newbies (which one to choose?) and long time contributors (monitor them all, splitted discussions, ...)

Discourse offers possibilities to import discussions from

• for OSQA,
• for fluxBB
• for mailinglists

Would this be a useful second step, if we get a discourse setup running?

[amandasaurus]

Is it possible to use discourse as a mailing list? i.e. to interact with it via email? If not, then you'll never get rid of the mailing lists, and instead you'll just add an additional communication channel.

Have you discussed this with existing heavy OSM Forum users? Do they have any feedback on what Discourse is like compared to the existing OSM Forum software.

As someone who cannot log into the OSM Forum because the software doesn't support my OSM username, changing the software has advantages to me.

[tomhughes]

It has "mailing list mode" yes.

I mean obviously it's not the same as a real mailing list but it's probably the closest anybody has ever come to creating an integrated forum/mailing list system.

[Nakaner]

The other option is to use Mailman 3 which can use authentication via OpenStreetMap (OAuth) and has a simple web frontend to send emails. It is in use by OpenRailwayMap. Try it out.

OpenRailwayMap uses Debian packages (but self built packages with slightly more recent versions and some patches). The packaging can be found at

• https://github.com/fossgis/django_mailman3-debian
• https://github.com/fossgis/hyperkitty-debian
• https://github.com/fossgis/postorius-debian
• https://github.com/fossgis/mailman3-debian

The ansible playbook is available as well.

[tomhughes]

Literally nobody on earth is actually using mailman 3 other than Fedora.

Well OK apparently you've found the one other place that does ;-)

In addition to apparently being a massive PITA to install (though I expect discourse is as well...) it also has a horrible UI to boot.

Even Fedora has now spun up a discourse instance though the internal lists are still using mailman 3 for now.

[Matthias84]

sorry folks, but please no off-topic and keep discussions on pros / cons of discourse at ticket #149 but not here. This is dedicated about merging our internal communication channels.

Let's find out, if unified forums is a chance for our community?

[pnorman]

Have you discussed this with existing heavy OSM Forum users? Do they have any feedback on what Discourse is like compared to the existing OSM Forum software.

I think we need to reach out to those people to get them involved in the discussion.

[Matthias84]

I started topics at forum and at OSQA. My idea is to wait for the WeeklyOSM, but of course anybody can start discussions at the talk ML or other useful places?

[pnorman]

cc @tordanik

[gravitystorm]

This is dedicated about merging our internal communication channels.

FWIW, I'm opposed to merging the forums and the help Q&A site. We definitely have problems with the underlying OSQA software, but the two venues serve completely different use cases.

More generally, I don't think that merging disparate communication channels has a significant benefit. With an ever-expanding community, it's foolish to attempt (or to prioritise) the ability to "monitor all the communications", and different people and different groups of people prefer different styles of communication in different situations (real-time chat, questions and answers, rambling discussions, short-form message broadcasting, long form blogging, etc etc). There's no pressing requirement to combine these different formats together.

So I think before attempting to merge anything, it needs more careful analysis about the style and purpose of communication in each format, and the current and future volumes of communication in the combined format, and whether this then makes sense. I would expect, for example, even a simple analysis to show that the mailing lists and help Q&A to have little overlap in either purpose or style of communication.

[Matthias84]

There is user &ergio which is willing to help on a discourse server!

[mrtngrsbch]

FluxBB and Discourse are not 'completely different' but a technological evolution to the demand of users and administrators.
On the other hand, our Telegram channels are overloaded with chaotic information and very difficult to read and follow.

The forum of the Argentinean community is in 4th place in number of posts and only contains a tiny part of the information that circulates through other channels.

Canada has 38 topics, Spain 91...
It does not take much study to realize that the forum is a cemetery and the different communities are looking for other ways. Moreover, there is already a Discord World!

Times have changed and we must understand the new challenges and offer solutions before the world imposes them.

In Argentina we have created a specific working group for this topic and we are studying all the possibilities to improve the general communication, mainly for the new users who have many difficulties to follow the diversity of channels that we currently have [https://wiki.openstreetmap.org/wiki/ES:Argentina#Contacto].

Currently, our main channel is [https://telegram.me/osm_ar] but we also have themes [Covid, Nature, GLAM] that were created to ease the charge.

Indeed, Telegram is not the appropriate tool for this type of problem, so we are seriously considering migrating the Forum to the Discourse. In this way we hope to be able to channel general questions or queries to the forum and to retain Telegram for more fluid conversations.

We are also considering Telefuel where we already have a Beta installation [https://www.telefuel.com/w/001Jdh8Mw4sgXsTpezCK/] so we can have 'channels' and not migrate quickly to Discord.

In short, to this day we think:

• Preserve the Forum as a historical channel but we would migrate the posts.
• We would make a more complete homepage with a CMS [https://openstreetmap.org.ar/]
• We would keep the Telegram, for the time being

best

[mrtngrsbch]

There is user &ergio which is willing to help on a discourse server!

I just asked the forum admins to offer me an export to import into Discourse...
In the Argentine community we are ready to be the laboratory of this idea
[https://wiki.openstreetmap.org/wiki/Talk:Forum]

[Zverik]

the forum is a cemetery and the different communities are looking for other ways.

Uhm... The forum is the main communication channel for communities in Austria, Belarus, Netherlands, Germany, Russia, Poland, Ukraine and many other countries. If that's not the case for Argentina, that doesn't mean it isn't for other countries. If you contact active members of these communities, I doubt the decision to move to Discourse would pass that easy.

[mrtngrsbch]

I have no doubt that many users will want to continue with the forum and each of them must have their reasons, mainly out of habit.

But if we want to move forward we will have to take some action according to the 21st century

I was just looking at yesterday's stats from our main Telegram: 100 posts!
In the forum: 1

Also... it is not my idea to close the forum! We can leave it to the dinosaurs to play in their park ;-)

Seriously... I just found this thread in the forum [https://forum.openstreetmap.org/viewtopic.php?pid=783989] !
It seems that we are not the only ones looking to leave the cemetery.

[532910]

There is user &ergio which is willing to help on a discourse server!

It's me.

[532910]

Илья, have you ever seen or used some of Discourse instances (don't confuse with Discord)?
It's used on many opensource projects, and it's definitely the best forum engine. So I believe there should be no objection from other members.

[amandasaurus]

Let's not berate people who use the forum. Language like “dinosaurs” doesn't help. (Plus dinosaurs are cool).

[mrtngrsbch]

Let's not berate people who use the forum. Language like “dinosaurs” doesn't help. (Plus dinosaurs are cool).

Hi @rory,
I love dinosaurs ;-)
I'm a 53-year-old dinosaur, who learned computing with a Sinclair Z80!

The problem we have in AR and many other communities is that the forum is not used by anyone!
I don't have any problem with the forum, but it doesn't respond to users' needs, except for dinosaurs ;-)

In other words, we have to find a solution for the new users somehow.

[Matthias84]

PLz. stay ontopic. This is about the idea to shift to discourse and maybe to merge different existing channels into it.

[mmd-osm]

In case you missed it: https://wiki.osmfoundation.org/wiki/Operations/Minutes/2020-07-15#FluxBB_is_old_and_slow

Action item: @iandees to try converting fluxBB DB to go into Discourse.

[iandees]

Hi! More updates also on the forum here: https://forum.openstreetmap.org/viewtopic.php?pid=795873#p795873

I've let this fall as most of my time available to OSM has been on CDN tests. I'll try to make some progress on the remaining pieces of Discourse migration this weekend.

[natrius]

In the german forum there is a thread as well https://forum.openstreetmap.org/viewtopic.php?id=68944

[mrtngrsbch]

In the german forum there is a thread as well https://forum.openstreetmap.org/viewtopic.php?id=68944

It seems to me that it only contributes to the general confusion.
The first step I think should be to offer an alternative to the forum

[mmd-osm]

OAuth 1.0a vs. OAuth 2.0 seems to be one of the issues according to one of the recent forum posts.

Not sure if there are any plans to move OSM website to OAuth 2.0 or if writing a custom OAuth 1.0a plugin for DIscourse is doable? There's also a somewhat stalled issue on OAuth 2.0 for the OSM website out there: openstreetmap/openstreetmap-website#1408

[tomhughes]

I'm currently investigating options for OAuth 2 support.

[tomhughes]

As far as Discourse is concerned - can you confirm if OAuth 2.0 is sufficient? or does it need OpenID Connect?

[iandees]

Discourse has built in support for Oauth2: https://meta.discourse.org/t/oauth2-basic-support/33879

I can also use something like what you did for the existing forum, @tomhughes, but Oauth2 would obviously be better.

[tomhughes]

Right so no need for OpenID Connect but we would need JSON support on the user details API endpoint.

[Matthias84]

The starting point is, that help.osm.org is outdated and doesn't get any more updates.
The forum is ok, but pretty limited compared to modern solutions and for non-tech users

• Image upload
• WYSIWYG Formating
• mobile apps
• ...

Bit wihtin this issues, pls lets focus on the topic of merging non realtime communication channels.

[pnorman]

We will continue to need an email-only way of interacting. If Discourse didn't support email, I doubt we would be considering it. We've had reports from multiple people about Discourse being used as a replacement for mailman2, which is what we use.

Even though we need to continue to support email-only, it's not going to be identical, and some things are going to be different than mailman 2. Some will be better, some will be worse.

For OSQA and FluxBB, the software powering the help and forum sites respectively, there are issues with both. OSQA is unmaintained and the forks of it have died off. We're maintaining a fork of FluxBB, so there are some issues there but no where near as bad as OSQA.

[SomeoneElseOSM]

Why is email-only interaction important?

For me, it's because quite a lot of the time that I might be reading OSM-related stuff I may only have an intermittent network connection or none at all. Not an issue while sat at home due to COVID 19 restrictions, obviously, but likely to become one again soon I hope. I typically read lots of OSM stuff "offline" and then mark it if I want to do something with it later. Email's perfect for this; the other OSM options that I've used (forum/IRC/Telegram/Slack) aren't.

Also - I'm lucky enough to be someone who lives in a country where good internet access is pretty widespread; you don't have to go far in OSM to talk to people who limit their online time or minimise downloaded data column to reduce costs.

[natrius]

So, i made a list on what it means to stay with fluxBB or choose something new.
Discourse:

• picture-upload (in an community where pictures sometimes are needed to prove something, the pictures should stay on the server. Would mean more storage space, of course)
• post-editing with markdown, bbcode and html (not just bbcode)
• reactions on posts (not everybody likes it, but sometimes people want to approve or disapprove a post without posting "+1" all the time)
• mark a post as answer (for help-alike)
• trust-system is available (also for help-alike)
• proper, native mobile support (i made the 'new' theme for the current fluxBB forum and it kinda works, but yeah)
• proper RTL-language support for a global community this is crucial! FluxBB did get an pull request that did not got merged for about 4 years now? Don't know, long time.
• direct answer with e-mail possible
• oauth2 support (tomhughes said already, osm would need to support oauth2 as well)
• integrated updatesystem, not via ftp like fluxbb right now (not useful for a user)
• integrated backup (not useful for a user)
• history for edited posts (that would be sometimes really, really handy)
• and, excluding JS-deniers (how do they use osm btw?)
• migration possible
• constant development

Staying with FluxBB

• excluding RTL-languages
• old software without development
• keeping more channels to talk instead of reducing
• no change, so nobody has to 'learn' something new.

The mainreason i am posting now is: french made their own discourse and migrated the existing content already.
https://forum.openstreetmap.fr/ and i know of other communities who are thinking about that step.

[HarelM]

How can I migrate the Israeli forum like france did? Current communication is mainly telegram...

[mnalis]

@natrius While I don't much care about fluxBB vs. Discourse, and I take your word that Discourse is superior here, I do forsee many problems by trying to cram several different communication channels into one. You know the truism: if it says "one size fits everyone", it doesn't fit anybody.

Here are broad categories currently in use (by OSM community and wider):

• Q&A sites like help.openstreetmap.org: critical thing here is "best answer on top" (ie. sort by upvotes). Like StackExchange and friends. Trying to replace it with forum-alike which has "likes", but sorts chronologically instead by likes and not being split into separate questions (no, topics/subjects of some discussion are not the same as questions, generally speaking) is bad idea. Try imagining stackoverflow.com (let's ignore their political problems for the moment) being replaced by Discourse. Wouldn't work at all. (that's why they've taken the throne there - other people were trying to do the Q&A sites with forums+likes+mark_as_answer, which are still really bad by comparison [i.e. marked as answer is often not the answer you're looking for])

• chat-like sites (like Telegram, Slack, Twitter, Facebook, XMPP etc.): they have way too many messages, which are way too little information, and with too little metadata interconnection (eg. missing full references metadata) to be mixed with forum/mailing-list like solution. Trying to merge those would quickly drive away one group of users (probably the ones who were using mailing lists).

• discussion-oriented sites (like Web Forums - fluxBB&Discourse, Usenet News, Mailing lists, Slashdot, Issue trackers like Github etc). Here I personally prefer:

  • mailing lists for topics I actively follow (completely offline use is the must!) in reasonable mail client which has good support for threading and filtering.
  • Usenet News client (via gmane.io) for long threads I only occasionally need to lookup, due to ability to do reasonable threading and searching subthread for some keywords).
  • Github issue tracker is actually quite usable (as would Discourse be, I imagine - I've only actively used one at Let's encrypt) as long as number of messages in thread is very small - like dozen or two messages at most. With 50+ messages in thread, all "forums" solutions I've found (Discourse included) become horrible waste of time when compared to mailing lists. They lack subthreads that mailing lists/news nicely support so you can't skip over parts which should've been new threads but didn't, they lack reasonable filtering on the subgroup of messages, navigation is just horrible (esp. if it involves who replied to what, limiting/filter by users), they're horribly slow, require online connection etc. All forums suck compared to good e-mail mailing list solution.

Trying to merge more than one group of those is impossible and terrible idea, which is usually self evident quite quickly (sometimes unfortunately only after it is implemented and it's too late to go back - let's try to avoid that future).

But even merging only those discussion-oriented solutions together (which might sometimes be possible - like gmane.io shows for mail/news gatewaying for example) has it's set of problems, some of which some people even see as advantages, like:

• post-editing with markdown, bbcode and html (not just bbcode) - it's all nice only if you can convert them one to another depending on the access method automagically. Eg. please don't force formatting in non-mailing-list-friendly way to mailing list users - eg. I don't read my mails in web browser, so HTML is no-go, and bbcode sucks even more. Simple markdown-alike tagging is OK though (ie. using "*" or "_" around words/phrases to emphasize them is fine if not overdone, but converting plaintext e-mail to XML-alike abomination and claiming "but we have mailing list support" is most definitely not)
• post-editing penalizes e-mail users (as they won't see changes and so confuses the communication), unless you resend messages on each edit, which is bad idea on it's own (as it spams e-mail users if it happens anything but rarely). Also, search should absolutely support finding all historic versions, with appropriate user-defined filters (I've used solutions which only search last revision, and it is extremely frustrating not to be able to find something because someone changed it in the meantime, thus removing critical keyword and hiding needle in the haystack again)

* and, excluding JS-deniers (how do they use osm btw?)

FYI, I rarely use web browser (much less JavaScript!) to either improve or use OSM, and still use it more efficiently than if I did...

So TL;DR - I'm OK with replacing fluxBB web forum with Discourse web forum. What I do have a problem with, is the idea "let's replace all communication channels with Discourse, in order to minimize engineer hours used on supporting the software". While I'm all for lessening the load on the overloaded engineers (I even try to do PR every blue moon, trying to help there, however small), I do not think that it is good enough reason for doing genocide on major parts of the community in the name of mistaken idea that "one size fits all". I'd much rather keep the suboptimal status quo if such outcome is even the possibility.

[natrius]

Just to be clear: What needs to be replaced is the forum and QA. Discourse should mainly be the forum. But, it seems like its possible to integrate QA in a nice way. Your critique [i.e. marked as answer is often not the answer you're looking for] has nothing to do with the software, it has to do with the people picking the answers. But i guess what you mean with that as well is "put the most upvoted at top" and not "put marked at top". Because marked at top is possible https://meta.discourse.org/t/how-to-mark-a-topic-as-resolved/81793

Chat is something else and should not get merged with this. Not at all. (in my opinion).

What i'm suggesting is not "Remove everything and lets Discourse do the job": FluxBB and QA needs to be replaced. Replacing the forum can be achieved with Discourse. It is possible to integrate QA there as well, but maybe there would be some drawbacks with that, i acknowlege that. Integration with mailinglists would just be the cherry on top because there is massive knowlege around mailinglists and i know quite some new people who just don't want to use a mailinglist. If it does not happen, well, there is basically no change from the current situation: there is a forum and there are mailinglists. Let it be.

[iandees]

Tom's work on OAuth in openstreetmap/openstreetmap-website#3177 looks to be merged. Looks like I should be able to test the OAuth 2 connection in my Discourse setup at some point. Has anyone tried that already?

[mmd-osm]

Yup, I got it working already. Relevant bits for Discourse settings are documented here: openstreetmap/openstreetmap-website#1408 (comment)

You probably have to tweak the scope to be "read_prefs", and register a new OAuth2 app on osm.org.

Maybe give it a try, and document what works or does not work. https://forum.openstreetmap.fr/ would need similar settings for their Discourse instance.

[Firefishy]

Progress update:

1. We have a "private" test instance for Discourse up and running, known as "community". Once testing is done, we will reset the instance and publicly launch.
2. Add support for privileged OAuth 2 applications openstreetmap-website#3300 is to allow seamless sign-in to the discourse insance using osm.org credentials.

[Firefishy]

There has been some discussion / planning of the discourse instance in this slack channel: https://osmus.slack.com/archives/C02804ANBC6 ( #discourse-community-rollout ).

[matkoniecz]

https://community.openstreetmap.org/ has this new test instance. Is mailing list interface also enabled on this test instance?

(that would be needed for "Merge (...) MLs to discourse?" part)

[amandasaurus]

No, there is no mailing list interface. I keep asking, because I want to test it. 🙂

…

On Tue, 12 Oct 2021 10:59 +02:00, Mateusz Konieczny ***@***.***> wrote: https://community.openstreetmap.org/ has this new test instance. Is mailing list interface also enabled on this test instance? (that would be needed for "Merge (...) MLs to discourse?" part) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#377 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAMC2TFSF5Z6XYN7SSUELUGP2IFANCNFSM4LONCPHA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Amanda

[mmd-osm]

...which is kind of funny, b/c every time you ask, you get the same answer "not in scope for day 1".

[Matthias84]

Would it be ok to share the link e.g. for the weekly notes newsletter? Or do we wait till there is some kind of preproduction-testing for everone?

[nukeador]

Hi all,

This is Ruben here.

The Ops team is working to have a stable version of the forum live in the coming days and the plan is to post a topic there once it's ready for community discussion on the next steps. Once this is ready, it would be great for people to engage there, I can ping you all here then.

Pointing people to the forum now is not that helpful because we are just finishing the technical testing phase. Once ready, we will enable spaces there for feature and bug report so everyone can be involved.

Thanks!

[nukeador]

Hi all,

I want to make sure you are updated about progress here. There are a couple of technical issues (related with https certificates and backups) to solve before going live and the Ops team has been really busy the past months. I'll post it here once we have clarity on the timeline to have this completed, I so sorry for the delay.

Cheers.

[samwilson]

I'm not trying to hassle anyone at all, nor increase stress anywhere, but... is there any news about when the new Discourse site might be up and running? :-)

Thanks!

[cquest]

I'm not trying to hassle anyone at all, nor increase stress anywhere, but... is there any news about when the new Discourse site might be up and running? :-)

https://community.openstreetmap.org/ is UP !

[nukeador]

We have been doing a "soft" launch for a few days and this week we'll have wider announcements in all OSM channels. Feel free to join the site, explore and share your feedback.

Thanks!

[matkoniecz]

I admit that I still strongly prefer FluxBB, which has much better interface and is simply working while Discourse seems to be some JS monster.

For example search is broken due to some pointless JS magic.

[iandees]

Thanks for everyone working on getting this set up. It feels really fast, I like the options for email communication, and the opportunity for more effective moderation tools.

[Matthias84]

As author of this issue, just once more thank you all for your hard work and effort to give your community forum a new modern home 😃

[grischard]

Linked issues:

#604 for forum migration to discourse
#149 for osqa migration to discourse

[Firefishy]

The forum migration was completed under #604.

#149 is for community.openstreetmap.org (discourse) of OSQA.

Mailinglist migrations are happening in different communities by "migrating" their communications over to community.openstreetmap.org (discourse). Migrating the mailinglist archives across to discourse is very unlikely to happen. The old mailinglist post archive will be kept available. Further discussion on this topic is likely best via community.openstreetmap.org

Considering the above, I'm going to close this ticket as completed.

[Firefishy]

Started a list of retired mailing lists which can be archived: #975