Release 3.2.0

[kelson42]

The 3.2 milestone is huge and has still a lot of work to do. I see work to do for at least 6 months. But the new ZIM support has been implemented and the support of this new type of ZIM files would benefit of an early release. How should we deal with this dilema?

• Update the changelog file
• Further tests
• Rename existing milestones (increment their names), create a new 3.3 milestone and move open issues to this new 3.3
• Create a tag on git
• Generate unsigned extensions (should be automatically triggered on GitHub Actions by the tag creation)
• Upload the unsigned extensions to Stores:
  • Firefox - https://addons.mozilla.org/fr/developers/addon/kiwix-offline/edit
  • Chrome
  • Edge - https://partner.microsoft.com/en-us/dashboard/microsoftedge/
  • Ubuntu touch - https://open-store.io
• Prepare the source code for version 3.3
• When available, copy the signed extension binaries to https://download.kiwix.org/release/browsers/
• Update the kiwix.github.io implementation
• [x ] Publicize these new versions

[Jaifroid]

@kelson42 I think there are quite a lot of things in that milestone that should be bumped off to a later milestone because not so urgent. I'd support an early release with just a few more bugfixes. Ideally I had wanted my tab-opening PR to be in the next release, as it's pretty much finished and working well, but there's an intractable test failure that appears to be an artefact of the testing system that I can't get to the bottom of and is blocking that one. I've implemented it in Kiwix JS Windows instead, where it is working really nicely even on the UWP app.

But I think we could release without that PR, as the no-namespace support is indeed important and was major work that would justify releasing.

[mossroy]

I agree with @Jaifroid : we usually release a new version when there are significant changes, and move the remaining github issues accordingly.
BUT we were waiting for newer ZIM files to test (AFAIK we could only test with old ZIM files converted to the new format). It was also necessary to implement unit/UI tests on a sample one

[kelson42]

@mossroy @Jaifroid Thank you for the quick feedback. Who is going to move the tickets?

[Jaifroid]

Once we've decided on a release, we usually move the tickets just before actual release. @mossroy I guess we could release current master marking the no-namespace support as "preliminary", and say that the implementation could change. If you think it's worth it. We'd have three changes to list: 1. Preliminary support for no-namespace ZIMs; 2. Preliminary support for ZIMs with a version 1 article index; 3. Home key focuses search bar. We could probably add some quick bug-fixes to that, for example the race condition caused by the transition animation.

[mossroy]

Do we now have ZIM files in the newer format, created from scratch? (i.e. not converted from older format). I think it's necessary to test some before releasing.
Do we now have a sample ZIM file in this new format, suitable for unit tests? (like the Ray Charles one)

[mossroy]

@kelson42 : my last questions were for you :-)

[kelson42]

@rgaudin I forward the question to you.

[kelson42]

@mossroy I believe this is the best you can get, see https://mirror.download.kiwix.org/zim/.hidden/dev/libretexts-engineering_en_2021-04.zim

[Jaifroid]

That engineering ZIM wouldn't be suitable for automated tests becuase it is 3.5GB, and the tests need to load the ZIM into memory in order to access it without the "user" having picked the file.

If it looks like the production of no-namespace ZIMs is going to be delayed considerably, I don't think it need be a blocker on us making new releases here, so long as we test manually on the ZIMs we currently have, until we have a reference Ray Charles (or similar) to use. And we mark the feature as "experimental" or "preliminary", and say it is support for the "specification" rather than for any actual ZIM.

Alternatively, we could develop the tests with the current no-namespace "converted" ZIMs we have, and slot in a reference ZIM when available. The risk is having to develop the tests twice if something important changes.

[mossroy]

Can #720 be fixed before considering a release?

It's true that we should release a new version of kiwix-js before the newer ZIM files are publicly available : do we have an approximate time frame for that?

If we only have one such ZIM file privately available (the engineering one above), maybe it's too early to release kiwix-js. I'm not asking for a coverage of all kinds of ZIM files, but one looks too few. We should at least have some mediawiki-based ones (because it's the main use-case), and some other contents (videos, stackoverflow etc).

We also should add unit tests on a reference small one (like Ray Charles) as soon as it's available. I don't think it's relevant to develop tests on converted ZIM files.
Unit tests are there to help us release with confidence, and secure future refactorings. A major change like this one deserves such tests. If such reference file is not available on time, we can consider implementing them after the release, but it would be sad to have to take the risk.

Or are we in a hurry to deploy some of the other changes from kiwix-js changelog? (I don't think so)

[Jaifroid]

Agreed, and yes I can tackle #720 first.

[mossroy]

@kelson42 : do we now have no-ns ZIM files, suitable for more tests before a release? (both manual and automated tests : see above)

[kelson42]

New sotoki can made new zim files. Will try tomorrow to make a run so you can check.

[rgaudin]

@mossroy here's the smallest sotoki Zim built without NS. Uses Illustration API for the Icon as well

beer_stackexchange_com_2021-07.zim

[Jaifroid]

Hmm, the detection of the API is failing badly on this ZIM (with our code). Did something change between the sample ZIMs we tested on and this production ZIM? It seems we can't get the article count or the article pointer position, despite the minor version being 1, hence we fall back to the Version 0 API. Our diagnostic is shown below.

(@mossroy There is also another issue caused by some error in DOM tokens -- to open this ZIM you have to comment out or try/catch around line 1360 of app.js -- you'll see the Exception which gives the line number depending on which branch you're on.)

[kelson42]

@Jaifroid I guess this is a question for @mgautierfr. I can not help here, AFAIK it should have been the same.

[rgaudin]

Well, we've updated the libzim to a more recent version of the master. I don't know the internals of all this so let me know if you need a specific info. from me.

[Jaifroid]

@rgaudin We don't use libzim, hence we have to reproduce the API in JS. But the previous ZIMs we had to work with were all converted using a script, and they all validated fine. This one fails. I need to trace the logic step-by-step and will feed back to @mgautierfr (unless he knows of a change we need to incorporate -- @mgautierfr , please see my comment above #722 (comment)).

[Jaifroid]

I've re-opened #708, where this should be discussed. I think I've determined the issue (an empty v1 directory listing). Please see #708 (comment).

[Jaifroid]

@mossroy Since we have a production ZIM that works with our implementation, might we be in a position to think about hte steps necessary for releasing 3.2.0 with no-namespace support? As production no-namespace ZIMs are close to public release, if not already released publicly, I think we should do this sooner rather than later. It would be good to include #740.

[mossroy]

Yes, it's a good step being done.
@kelson42 : Do we have a small ZIM file to implement unit/UI tests on, like the Ray Charles one? It would at the same time allow to test a mediawiki-based ZIM file (which is the most common use-case).
When do you plan to publicly release the no-namespace ZIM files?

And yes, we should include #740 in 3.2.0 : I'll have a look at it.

[kelson42]

@mossroy We have started a repository to manage a reference test suite of ZIM file here https://github.com/openzim/zim-testing-suite. If you have not what you need there. I would recommend to request/put it there. I think first no-namespace ZIM files will be published in September (as soon as libzim7 will be published which can now happen almost any time).

[mossroy]

@kelson42 , I opened openzim/zim-testing-suite#5 to ask for a small sample ZIM file suitable for our tests (the existing ones are not)
It's sad that this schedule does not give us enough time to implement these automated tests. It would be better to involve us earlier next time a major change is considered and planned.

Anyway, the wikipedia ZIM files from https://github.com/openzim/zim-testing-suite/tree/main/data/nons are enough to check compatibility with wikimedia-based ZIM files : I did not see any issue.

So I would suggest the following plan for kiwix-js :

• finalize and merge Recompile Emscripten binaries, implement WASM with fallback, and add instructions and scripts #740
• manually test, and release version 3.2.0 of kiwix-js
• implement unit and UI tests on the new sample ZIM file as soon as possible
• then work on the changes we planned for the hackathon, with the help/security of these automated tests

[mossroy]

I've added the usual task list on this ticket.
I might handle the milestones & move the issues right now
@Jaifroid : if you're available, you might update the changelog? Else I'll do it myself

[Jaifroid]

I'm happy to do the changelog.

[mossroy]

I moved to a new v3.3 the v3.2 issues we plan to work on at the hackathon.
I moved the other ones to v3.4
I also moved to "backlog" all the jQuery-specific enhancements

[mossroy]

I tested many different ZIM files on my Firefox OS device and in a Firefox extension (both in jQuery mode) and found no issue.
I did not see any issue either on Chromium in SW mode.
But there is a blocking issue in a Chromium extension : the decompressor API says "Error loading ZSTD decompressor!" :

Could not instantiate any ZSTD decoder! RuntimeError: abort(CompileError: WebAssembly.instantiate(): Wasm code generation disallowed by embedder). Build with -s ASSERTIONS=1 for more info.
at A (zstddec-wasm.js:15)
at zstddec-wasm.js:18

and nothing works afterwards.

I seems to come from a CSP constraint

[mossroy]

There is a workaround for this issue that I tested successfully by adding a line in manifest.json : https://stackoverflow.com/questions/48523118/wasm-module-compile-error-in-chrome-extension (works with unsafe-eval or wasm-eval)
Maybe we should open a separate issue for that, to consider compatibility with different versions of Chromium-based browsers (is wasm-eval supported by all of them?), compatibility with Firefox, and security risks that it might raise
I have to leave for now, will come back this afternoon

[mossroy]

Just created #751 for this issue

[Jaifroid]

Good catch. IMHO it's safe to add unsafe-eval because we only execute offline code. We also have a separate CSP for the iframe, which is where ZIM content is executed.

[Jaifroid]

JFYI, I'm on my phone only for much of the rest of today.

[mossroy]

@Jaifroid : if I implement #751 with the unsafe-eval , can I do the release of 3.2.0?
i.e do you think you made enough tests on Windows platforms (including IE11) for the release?

[mossroy]

Let's not rush, and take the time to review #754, including testing it as an Edge extension.
Hopefully the CI instability will be fixed in the following days (it might have blocked our release anyway)

[Jaifroid]

Sorry it turned out I was in a no signal area all day. I've tested the release a lot via the last PR, so happy for you to release when you wish. I'm also happy to wait until we're sure we don't have a blocker with WASM.

[mossroy]

No problem!
You must be overwhelmed by github notifications today : take your time

[mossroy]

The unsigned packages have been nicely generated by GitHub Actions, and uploaded in https://download.kiwix.org/nightly/2021-08-23/
I uploaded the Firefox package in their store : there's a new warning about our unsafe-eval policy : https://addons.mozilla.org/fr/developers/upload/f8bcf19b4fdb45698be20b3a04232e60 , but it has been accepted.
I've also uploaded the package for Ubuntu Touch

[mossroy]

@kelson42 and @Jaifroid : please upload the package https://download.kiwix.org/nightly/2021-08-23/kiwix-chrome-unsigned-extension-3.2.0.zip for the Chrome and Edge stores

Regarding the Firefox store, it would be safer that both of you also could also manage the addon : uploading a new release, or any other need.
I can easily give you rights on this addon, but you first need to create an account on https://accounts.firefox.com/ , and give me the corresponding e-mail address. Please activate 2FA on your accounts for more security.

[mossroy]

@ernesst : Could you please test this new version on your device? It's already uploaded on https://open-store.io/app/kiwix
Could you also give us the "API status" you get at the end of your "Configuration" section? (snapshot or copy/paste)
Last but not least, could you try to activate the ServiceWorker mode in this section, and tell us if there is an error in the API status, and if you can browse ZIM files in this mode? We plan to switch to this mode by default in next release so I'd like to make sure it also works on Ubuntu Touch.

[Jaifroid]

I've uploaded the extension to the Edge Store. Annoyingly, since the last submission the Store has come out of Beta, and it forced me to rebuild the page from scratch. I took the opportunity to update the screenshots. It says it might take 7 business days to review.

[ernesst]

Hello, I don't have currently an headset under ubuntu touch to try the changes. Sorry. Aug 23, 2021 3:19:31 PM Mossroy ***@***.***>:

…

@ernesst[https://github.com/ernesst] : Could you please test this new version on your device? It's already uploaded on https://open-store.io/app/kiwix Could you also give us the "API status" you get at the end of your "Configuration" section? (snapshot or copy/paste) Last but not least, could you try to activate the ServiceWorker mode in this section, and tell us if there is an error in the API status, and if you can browse ZIM files in this mode? We plan to switch to this mode by default in next release so I'd like to make sure it also works on Ubuntu Touch. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub[#722 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ACWP56BHXNEWPLD7YAQ6H3TT6JDGDANCNFSM44NTINPQ]. Triage notifications on the go with GitHub Mobile for iOS[https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675] or Android[https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email]. [data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD8AAAA/CAYAAABXXxDfAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAAmSURBVGiB7cEBDQAAAMKg909tDwcUAAAAAAAAAAAAAAAAAAAAJwY+QwABivJx1AAAAABJRU5ErkJggg==###24x24:true###][Tracking image][https://github.com/notifications/beacon/ACWP56BFVFBA4MFRLBT76BDT6JDGDA5CNFSM44NTINP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGXPEYBA.gif]

[mossroy]

Thanks for your answer @ernesst : is it temporary or permanent?
If it's permanent, do you know someone else who could test our releases?
According to the stats of open-store.io, the previous version (3.1.0) has been downloaded 415 times (surprisingly twice as much as the 207 times of version 3.0.0), which is not negligible.

If it's temporary, we have no rush on our side : the tests can be done later (any rough idea of when you would get a suitable device?)

[kelson42]

Release is waiting validation on Chrome Extension Store. But I have already upload the CRX to download.kiwix.org.

[ernesst]

I don't know when I could reinstall Utouch. I'll try to ask some persons. Aug 24, 2021 1:51:41 PM Mossroy ***@***.***>:

…

Thanks for your answer @ernesst[https://github.com/ernesst] : is it temporary or permanent? If it's permanent, do you know someone else who could test our releases? According to the stats of open-store.io, the previous version (3.1.0) has been downloaded 415 times (surprisingly twice as much as the 207 times of version 3.0.0), which is not negligible. If it's temporary, we have no rush on our side : the tests can be done later (any rough idea of when you would get a suitable device?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub[#722 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ACWP56A275IHZNP3EZWTOSLT6OBUVANCNFSM44NTINPQ]. Triage notifications on the go with GitHub Mobile for iOS[https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675] or Android[https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email]. [data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD8AAAA/CAYAAABXXxDfAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAAmSURBVGiB7cEBDQAAAMKg909tDwcUAAAAAAAAAAAAAAAAAAAAJwY+QwABivJx1AAAAABJRU5ErkJggg==###24x24:true###][Tracking image][https://github.com/notifications/beacon/ACWP56EYVJ5LHW7ZQPSDNODT6OBUVA5CNFSM44NTINP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGXVKIMI.gif]

[kelson42]

Kiwix JS 3.2.0 has finally been published on the Chrome Store

[Jaifroid]

It's also in Edge store here: https://microsoftedge.microsoft.com/addons/detail/kiwix/fbbdjelfpndiajmbaocbfcjdgomfbjgm .

[Jaifroid]

Actually, there now appear to be two Kiwix extensions in the Microsoft Addons Store. Remember I mentioned I had to rebuild the previous version because the previous one had been in the Beta Store? Well it doesn't appear to have taken the place of the old one, so we now have two versions, 3.1 and 3.2. I think I'm going to have to sort this out with Microsoft, because I don't seem to have access to the old version. Grrr. In the meantime, please don't advertise the URL I gave in the post above, because there are 274 users of the old one. Maybe I missed a step somewhere. It would be preferable to update the old version with all those users.

[Jaifroid]

OK, I've sorted out the issue and have managed to submit the package to the correct place now. I'll delete the rogue version.

[kelson42]

Let me know when everything is ready for the annoucement.

[Jaifroid]

@kelson42 From my perspective it's fine for you to announce. The Edge Addon will update in due course (in a few days), but there's no reason to delay the general announcement.

[mossroy]

OK for me to announce too

[mossroy]

The last remaining task is to announce this version, for @kelson42
I'll close this issue, to be able to close the milestone.