Migrate help.openstreetmap.org from OSQA to static html archive #149
Comments
|
If it's a fork I'm surprised the database can't just be migrated or have their been schema changes on the OSQA side since the fork? |
|
The later probably not, and I wouldn't rule out that we could migrate the mysql DB and then migrate to Postgres. From reading the support forums it is not quite clear what the issue is, but I'll dig a bit more. This http://askbot.org/doc/upgrade.html may be part of the problem. |
|
Sounds sensible to move to something that is actively maintained if it isn't too difficult to do. |
|
OSQA works well enough for me, the glitches that are there don't bother me. I have never used Askbot, it doesn't exactly look like there's a huge developer community behind it apart from the company who runs it - will anything remain if their business tanks? I don't mind staying with OSQA but I don't mind switching either. If that should mean losing our content I think that wouldn't even be too bad, many the answers in OSQA are outdated. We count pull a read-only static image from OSQA for reference purposes and copy over relevant stuff when a question pops up... plus a reboot would create a fair karma playing field where you don't have this woodpeck guy with twice as many karma points as the next contender. |
|
@woodpeck problem is that I need to upgrade that machine to 16.04 at some point but I have no confidence OSQA won't break (it did when we went to 14.04 and we had to patch it). |
|
If it makes sense to move (and I can see why there would, given the issues we had and to some extent still have), I don't see a particular problem with doing so. There are some "old answers" that are worth preserving, but bluntly some sort of static HTML would work for those. Many more old answers "were correct ages ago" but now no longer are. I don't think that a "flattening" of user reputation would be a problem either. |
|
I've been asked to add a link to https://wiki.openstreetmap.org/wiki/Talk:Help.openstreetmap.org#Migration_QSQA_-.3E_ASKBOT here, but the content of the message is small enough to post in full: "Sounds quite good for me. However, if we are migrating to anywhere we rather should think about how to solve the issue of outdated answers. Just deleting the whole thing every 7 years is a waste. There ARE many still good answers (yes, also much irrelevant or even wrong answers nowadays) and I try to take care of them (updating, organizing) as good as possible. I also would like to have incoming link not to be broken if possible (yes, setting it up as static html clone might work). Regarding the "outdated" problem, also see the section above. Regarding possible Karma flattening: that is likely no problem. --Aseerel4c26 (talk) 20:04, 21 February 2017 (UTC)" |
|
Keeping good questions/answers is certainly a good idea. We should make sure people are still able to find them via search engines. |
|
Assuming for a moment that copying over part but not all of existing Qs&As is possible. perhaps we could share the load of judging which questions/answers to keep and which to kill by resetting all scores to 0, and then asking everyone to look through stuff and upvote what they think should be kept... |
|
Yes. Maybe we can go through the top 50 questions here and decide which ones to keep: https://help.openstreetmap.org/questions/?sort=mostvoted If they can't get migrated maybe we could still copy'n'pasting the question and 1-2 good answers and then mark them as community wiki. |
|
I think moving off of qsqa is the only reasonable option given it's unmaintained nature and problems that have required patching, so it's only a question of if askbot the best option. There is information on migrating, so it's just a question of how far back the migrations go. |
|
The Wireshark project solved this by simply archiving their osmqa site. |
|
The problem is that I need to upgrade the machine and doing so will almost certainly break OSQA meaning that even running it in RO mode is likely to be hard. |
|
@tomhughes @gravitystorm @zerebubuth this would seem like a fairly self-contained activity that could be farmed out to an interested person from the wider community that wants to help. Again objections to raising this on the a suitable mailing list (I'm pushing a bit because besides the discussed issues, we should be fixing the gravatar support in some way from a data protection pov, and it doesn't make sense to do this in the old unsupported code). |
|
If you can find somebody that can figure out how to migrate the data then sure. I actually have a chef branch where I started work on code to install and manage askbot. |
|
Is this any help? |
gravitystorm
changed the title
|
How big is the MySQL database? Can one get access to a dump? |
|
It's postgres, not mysql. Ignoring the (vast) session table it looks like there is a little over 100Mb in the tables. If you're volunteering to attempt a conversion then we could probably come to an arrangement to make it available, subject to appropriate agreements around safeguarding any private data. |
|
|
|
Maybe we can try with Discourse - more active development also Postgress and Ruby based. |
|
Hi, about @slachiewicz's suggestion,
See meta.discourse/porting-from-askbot-to-discourse ... Seems that the major investment for us is to enhance this embrionary adaptation. |
|
Discourse is a forum (or a replacement for forums) not a Q+A site. That is kind of evident in the fact that it comes from one of the founders of Stack Overflow and was specifically designed to address a different use case... |
|
As you can see @ppKrauss in the repository, you can boast a large and active number of migration scripts https://github.com/discourse/discourse/tree/master/script/import_scripts |
|
@ppKrauss @slachiewicz pls no. While discourse may have its merits as a forum replacement (just that I've never found any) it is really completely at odds with what the help site is supposed to be doing (as @tomhughes has said). |
|
Some news about our options: meta.discourse community's positions and suggestions. |
|
So, I had a look at AskBot looking at testing a conversion. I can't recommend it because we'll have the same problems as now. All versions of AskBot rely on Django versions that were released 3-4 years ago and are EOL. The commit activity on its repository is minimal. Keeping OSQA on a machine running |
|
As workaround maybe pack all Django OSQA stuff to Docker? |
|
The maintainer of askbot (the modern descendant of osqa) said: "Also I can help you migrate from OSQA if you were interested." So if someone wants to reach out, that is a possible direction. I added continuous integration tests to askbot (PR). It currently has 670 unit tests, running on python 3.11.7 on Ubuntu 22.04, so it is likely in better shape than I thought yesterday. |
|
Sounds like replacement by static HTML site was done instead and this got wontfixed. From looking at https://community.openstreetmap.org/t/archiving-of-help-openstreetmap-org-1st-march-2024/108483 |
grischard
changed the title
|
@matkoniecz thanks, I've edited the title |
|
Seems too bad that there's no place to upvote good questions and good answers, but I understand the desire to just get something done for open discussions. |
Discourse does support this. It's not 100% (see e.g. https://community.openstreetmap.org/t/archiving-of-help-openstreetmap-org-1st-march-2024/108483/6 ) but it is arguably "good enough". |
I appreciate the link, and I have no suggestions, and this looks like a done deal. However, that thread shows that a lot of disagreement about whether that functionality is good enough, and suggests perhaps the new site won't have an effective upvoting Q&A functionality. I'd be happy to be wrong. I have no horse in this race. |
|
Codidact is still an option if folks want, either on our network or self-hosted. (Our self-hosting assistance has gotten much better since that comment I left three years ago.) Q&A (plus articles and you can add other post types), voting, lots of available customization, FOSS -- we're happy to discuss your needs if you're interested. One of our communities, as an example: https://software.codidact.com/ . I am trying to be a good github citizen and don't want to spam. |
I think there's broad consensus on the discourse plugin being not quite as good as OSQA or its forks and alternatives. Ultimately, it's about retiring an antique piece of software, and centralising on one platform to save sysadmin energy and time. If volunteers are willing to help, any plan would have to include setting up the alternative as a new service in chef, running a migration from osqa to it, and maintaining the new platform. |
That makes sense, thanks. I wonder if @evgenyfadeev would be willing to comment on this. He is maintaining ASKBOT, a modern fork of OSQA, and said "I can help you migrate from OSQA if you were interested.. I maintain and support some Q&A sites and the latest code is used in production." But, I don't know him or what he's actually offering. |
|
The ASKBOT avenue was explored years ago, just as trying to find volunteers that would run it. There was never any offers forthcoming and at one point you need to draw a line. |
|
Hi @simonpoole fair enough. I did not see this message years ago;). If you wish to migrate, it can be done. Is the Q&A plugin in discourse fully satisfying your needs in a Q&A app? |
See #149 (comment) and a number of other posts for more context. In any case, doubts about continuing maintenance were just half of the issues, as I already pointed out despite repeated attempts we were not able to find any volunteers to actually support running the site. |
|
@simonpoole If there were funding for ongoing support and interest from your organization, I will take on this and the migration would be free. |
Some of these are due to the Q&A plugin been disabled due to an upstream change. It was re-enabled. Q&A plugin improvements / fixes / issues should be reported upstream https://meta.discourse.org/t/discourse-post-voting/227808 |
|
Solved: appears to be done by this specific user
|
|
There is no migration of questions and answers planned. If we were to consider it someone would have to volunteer to do the coding and testing work. |
|
help.openstreetmap.org login is now disabled. The login session table has also been cleared. |
|
As discussed on the #osm-dev IRC channel, I have launched an
ArchiveBot job to save help.openstreetmap.org to archive.org.
You can follow the job on the ArchiveBot dashboard:
http://archivebot.com/
I've left it at the default of 3 parallel connections with low delay,
if you want them changed, you can ask me here or on #osm-dev or
if I am AFK then ask others on #archivebot on hackint.
https://wiki.archiveteam.org/index.php/ArchiveBot
…--
bye,
pabs
https://bonedaddy.net/pabs3/
|
|
@pabs3 Any chance you could suggest a method for turning the help.openstreetmap.org site static? The site currently runs OSQA which is powered by an ancient version of Django + Python. I'd ideally like to host static content using just apache or nginx. Container is an option. |
|
Something as low-tech as seems to produce quite reasonable reproduction of that one question for quick test. (note that You'll note that external links (like e.g. Obviously inherently dynamic functions (like voting, search, etc.) won't work on static site in any case... 😃 |
|
One should probably set the date on We'll still hit the y2k bug as OSQA uses two-digit years even in those cases, so even with that precaution we'll get things like |
|
As a last resort, once the ArchiveBot job is done and also we have
manually saved the few URLs that failed, then you could just download
the content from archive.org and serve that. I recommend the tool
called wayback-machine-downloader for doing the download. I note the
ArchiveBot job is currently at 300GB (including external URLs) though,
so maybe that isn't really feasible as archive.org downloads are slow.
https://github.com/hartator/wayback-machine-downloader
Searching for a more Django oriented way to do this, I found the
django-bakery project. Seems that using it requires modifying the
OSQA codebase though.
https://palewi.re/docs/django-bakery/
https://github.com/palewire/django-bakery/
…--
bye,
pabs
https://bonedaddy.net/pabs3/
|
|
So far the ArchiveBot job has gotten 369,962.3 MB in 1,168,047
responses with 910,459 URLs in the queue, which is going down.
Another option would be to just redirect all URLs to archive.org,
then you don't have to do any hosting of the site at all.
You may want to keep a copy of the database and code online tho,
potentially published directly in a tarball if you can remove all
the PII and logs before publishing.
…--
bye,
pabs
https://bonedaddy.net/pabs3/
|
|
Thinking about the 300GB number, it probably includes a copy of the OSM
planet exports, I think I saw that linked during the downloads.
So the entire site excluding offsite links would probably be much
smaller. Larger than the size of the database but not too much.
That could make it much more feasible to download from archive.org.
…--
bye,
pabs
https://bonedaddy.net/pabs3/
|
|
FYI, the help.osm.o ArchiveBot (AB) job completed but crashed due to
full disk because I forgot to ignore planet.osm URLs. The results
should be in web.archive.org (the WBM) in a few days time. The AB
job got 444,756.7 MB of data from 3,889,588 requests. Probably some
of those will be off-site requests or planet.osm data though, so the
size of the site in the WBM should be smaller.
However three of the badges URLs took longer to generate than the AB
timeout, so those required manual handling, and the person doing that
has a bit of an upload backlog so it could be some time before they
are included. So you might want to manually save these three timed
out pages in addition to retrieving the AB job data from archive.org:
/badges/1/unknown
/badges/3/unknown
/badges/32/unknown
Happy to help further, I'll be on the #osm-dev IRC channel too.
…--
bye,
pabs
https://bonedaddy.net/pabs3/
|
|
Anyone want to have a go at adapting this to work with help.openstreetmap.org? |
|
I have put out a call for volunteers: https://community.openstreetmap.org/t/can-you-create-a-static-version-of-the-old-help-openstreetmap-org-site/112209/1 |
and others
ASKBOT is an actively maintained fork of qsqa (that was last worked on something like 5 years ago), see:
https://github.com/ASKBOT/askbot-devel and https://askbot.com/
If we want to continue to run our own Q&A site migrating would make sense and would likely address many of the issues we have with QSQA (not just that it is abandonware). The alternative would be to give up running our own system and move to stackexchange, which has IMHO some downsides including loosing the content I believe.
Other issues to consider:
Comments @woodpeck @systemed and anybody else answering questions on help?
The text was updated successfully, but these errors were encountered: