Task/anr fix zimfile reader #3937

CalebKL · 2024-07-15T15:28:13Z

This PR fixes the ANR thats being caused while trying to create the zimFileReader object. The UI was being blocked since we were creating the object on IO Thread but we are using runblocking which blocks the main thread. The solution to the problem was to suspend setZimFile and setZimFileDescriptor. Then tie the scope on lifecycleScope on the Fragments. When the zimFileReader is being created and it takes time for the initial load, the progressBar will be visible to the users and when Loading is complete, the progressBar will be Gone.
The solution doesn't block the Main Thread and instead switches the background work to a different thread(IO)

Screenshots

…s to IO when creating zimFileReader

…odelScope

MohitMaliFtechiz

@CalebKL Thanks for the PR and your hard work. But there are some edge cases and crashes scenarios introduced we should fix them.

There is a crash when I frequently changed the zim files. Also, in UI there is old zim files loading when I am switching to other zim file which is bigger in size. I guess because of this, this crash is happening.

2024-07-16 14:20:19.873 12535-12609 libc                    org.kiwix.kiwixmobile                A  Fatal signal 11 (SIGSEGV), code 1, fault addr 0x0 in tid 12609 (ThreadPoolForeg)
2024-07-16 14:20:19.875 12535-12602 ZimFileReader           org.kiwix.kiwixmobile                E  Could not get Item for url = https://kiwix.app/I/Flag_of_Greece.svg.png.webp 
                                                                                                     original exception = java.lang.Exception: Cannot find entry
2024-07-16 14:20:19.876 12535-12602 ZimFileReader           org.kiwix.kiwixmobile                D  getting mimetype for https://kiwix.app/I/Flag_of_Greece.svg.png.webp = null
2024-07-16 14:20:19.877 12535-12821 ZimFileReader           org.kiwix.kiwixmobile                E  Could not get Item for url = https://kiwix.app/I/Political_slogan_by_Red_Guards_on_the_campus_of_Fudan_University_1976.jpg.webp 
                                                                                                     original exception = java.lang.Exception: Cannot find entry
2024-07-16 14:20:19.898 12535-15737 ZimFileReader           org.kiwix.kiwixmobile                E  Could not get Item for url = https://kiwix.app/I/Flag_of_Greece.svg.png.webp 
                                                                                                     original exception = java.lang.Exception: Cannot find entry
2024-07-16 14:20:19.904 12535-12535 AppCompatViewInflater   org.kiwix.kiwixmobile                I  app:theme is now deprecated. Please move to using android:theme instead.
2024-07-16 14:20:19.911 12535-12535 CubicBezierInterpolator org.kiwix.kiwixmobile                D  CubicBezierInterpolator  mControlPoint1x = 0.23, mControlPoint1y = 0.06, mControlPoint2x = 0.09, mControlPoint2y = 0.97
2024-07-16 14:20:19.912 12535-12535 CubicBezierInterpolator org.kiwix.kiwixmobile                D  CubicBezierInterpolator  mControlPoint1x = 0.6, mControlPoint1y = 0.9, mControlPoint2x = 0.8, mControlPoint2y = 1.0
2024-07-16 14:20:19.913 12535-12535 CubicBezierInterpolator org.kiwix.kiwixmobile                D  CubicBezierInterpolator  mControlPoint1x = 0.23, mControlPoint1y = 0.06, mControlPoint2x = 0.09, mControlPoint2y = 0.97
2024-07-16 14:20:19.913 12535-12535 CubicBezierInterpolator org.kiwix.kiwixmobile                D  CubicBezierInterpolator  mControlPoint1x = 0.6, mControlPoint1y = 0.9, mControlPoint2x = 0.8, mControlPoint2y = 1.0
2024-07-16 14:20:20.129 17250-17250 DEBUG                   pid-17250                            A  pid: 12535, tid: 12609, name: ThreadPoolForeg  >>> org.kiwix.kiwixmobile <<<
2024-07-16 14:20:20.132 17250-17250 DEBUG                   pid-17250                            A      #00 pc 0000000000010630  /data/app/org.kiwix.kiwixmobile-G5GGa4VbPmK5Z5zSGKhzQw==/base.apk (offset 0x71f000)
2024-07-16 14:20:20.132 17250-17250 DEBUG                   pid-17250                            A      #01 pc 000000000005be38  /data/app/org.kiwix.kiwixmobile-G5GGa4VbPmK5Z5zSGKhzQw==/oat/arm64/base.odex (offset 0x46000)

MohitMaliFtechiz · 2024-07-16T08:35:02Z

core/src/main/java/org/kiwix/kiwixmobile/core/reader/ZimReaderContainer.kt

    assetFileDescriptorList: List<AssetFileDescriptor>,
    filePath: String? = null
  ) {
-    zimFileReader = runBlocking {
+    zimFileReader = withContext(Dispatchers.IO) {


Please fix the compilation error in the ZimReaderContainer class.

MohitMaliFtechiz · 2024-07-16T08:59:49Z

app/src/main/java/org/kiwix/kiwixmobile/nav/destination/reader/KiwixReaderFragment.kt

-        // top margin for kiwix main app.
-        setTopMarginToWebViews(0)
-        selectTab(currentWebViewIndex)
+      lifecycleScope.launch {


If there is some delay in the creation of the zimFileReader object e.g. for bigger zim files then previously saved tabs are lost.

You can reproduce it, by adding some delay in the creation of the ZimFileReader object.

MohitMaliFtechiz · 2024-07-16T09:11:16Z

core/src/main/java/org/kiwix/kiwixmobile/core/main/CoreReaderFragment.kt

@@ -1721,7 +1733,7 @@ abstract class CoreReaderFragment :
    closeAllTabsButton?.isClickable = isClickable
  }

-  private fun restoreDeletedTabs() {
+  private suspend fun restoreDeletedTabs() {


There is also a leg while restoring all tabs if closed(which is obvious since it is preparing the zimFileReader on the IO thread). But the user is not aware of it so we should show some progress/something so that the user can know about what is going on. Also, if we move to any other fragment while in this process then the application crashes.

XRecorder_16072024_143650.mp4

FATAL EXCEPTION: main Process: org.kiwix.kiwixmobile, PID: 18742 java.util.NoSuchElementException: List is empty. at kotlin.collections.CollectionsKt___CollectionsKt.last(_Collections.kt:418) at org.kiwix.kiwixmobile.core.main.CoreReaderFragment.restoreDeletedTabs(CoreReaderFragment.kt:1746) at org.kiwix.kiwixmobile.core.main.CoreReaderFragment.access$restoreDeletedTabs(CoreReaderFragment.kt:177) at org.kiwix.kiwixmobile.core.main.CoreReaderFragment$restoreDeletedTabs$1.invokeSuspend(Unknown Source:18) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) at android.os.Handler.handleCallback(Handler.java:808) at android.os.Handler.dispatchMessage(Handler.java:101) at android.os.Looper.loop(Looper.java:166) at android.app.ActivityThread.main(ActivityThread.java:7425) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:245) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:921)

MohitMaliFtechiz · 2024-07-16T09:31:43Z

.../src/main/java/org/kiwix/kiwixmobile/core/page/notes/viewmodel/effects/ShowOpenNoteDialog.kt

-            editor.putString(TAG_CURRENT_POSITIONS, "$positions")
-            editor.putInt(TAG_CURRENT_TAB, 0)
-            editor.apply()
+        dialogScope.launch {


There is also a delay in showing the notes dialog, if we open the note from Notes Screen(if note's zim file is not currently set in the zimReaderContainer).

…y and catch blocks to monitor the errors

CalebKL · 2024-07-18T10:21:31Z

@MohitMaliFtechiz When I switch to different zimFiles, I get this crash after numerous attempts.
I am not sure if its the same crash you were referring to earlier.

Cmdline: org.kiwix.kiwixmobile
2024-07-18 12:44:02.286  5611-5611  DEBUG                   pid-5611                             A  pid: 5387, tid: 5606, name: DefaultDispatch  >>> org.kiwix.kiwixmobile <<<
2024-07-18 12:44:02.286  5611-5611  DEBUG                   pid-5611                             A        #00 pc 0000000000010630  /data/app/~~2cz1tgEo0aQgZshuQoQHsg==/org.kiwix.kiwixmobile-ZpQbVVYb8qkV5KrowJ_Pfg==/base.apk!libzim_wrapper.so (offset 0x1b2a000) (Java_org_kiwix_libzim_Archive_getEntryByPath__Ljava_lang_String_2+120) (BuildId: bb96395f5068f1755a93f5daa08b7a934333f07e)
2024-07-18 12:44:02.286  5611-5611  DEBUG                   pid-5611                             A        #02 pc 00000000020ad728  /memfd:jit-cache (deleted) (offset 0x2000000) (org.kiwix.kiwixmobile.core.reader.ZimFileReader.getItem+376)
2024-07-18 12:44:02.286  5611-5611  DEBUG                   pid-5611                             A        #03 pc 000000000214569c  /memfd:jit-cache (deleted) (offset 0x2000000) (org.kiwix.kiwixmobile.core.reader.ZimFileReader.loadContent+204)
2024-07-18 12:44:02.286  5611-5611  DEBUG                   pid-5611                             A        #08 pc 0000000000008cd0  /data/data/org.kiwix.kiwixmobile/code_cache/.overlay/base.apk/classes13.dex (org.kiwix.kiwixmobile.core.reader.ZimFileReader.access$loadContent+0)

MohitMaliFtechiz · 2024-07-18T11:31:42Z

@CalebKL No it is not that crash i have uploaded the crash logs in the review comment.

Also, now on my device, it is crashing immediately when I switch to other zim files(I am using the old version of Android for testing Android 8).

Fatal signal 11 (SIGSEGV), code 1, fault addr 0x0 in tid 11992 (DefaultDispatch)
2024-07-18 16:33:50.291 12077-12077 DEBUG                   pid-12077                            A  pid: 11734, tid: 11992, name: DefaultDispatch  >>> org.kiwix.kiwixmobile <<<
2024-07-18 16:33:50.294 12077-12077 DEBUG                   pid-12077                            A      #00 pc 0000000000010630  /data/app/org.kiwix.kiwixmobile-YaobaHgoDzW2aibsfWri3g==/base.apk (offset 0x71f000)
2024-07-18 16:33:50.294 12077-12077 DEBUG                   pid-12077                            A      #01 pc 000000000005be38  /data/app/org.kiwix.kiwixmobile-YaobaHgoDzW2aibsfWri3g==/oat/arm64/base.odex (offset 0x46000)

In the CoreReaderFragment there is a restore tabs functionality also for opening the previous zim file and restoring the previous tabs, and their article opens in the webView(if we are not opening the new zim file).

Currently, it is somehow restoring the previous zim file and tabs when we open the new file. However, it should not do it according to our openPageInBookFromNavigationArguments method's logic. Can you please check this? It is the main reason for these crashes.

MohitMaliFtechiz · 2024-07-18T11:35:53Z

@MohitMaliFtechiz When I switch to different zimFiles, I get this crash after numerous attempts.

This crash is newer, on which device you are testing.

CalebKL · 2024-07-18T11:42:04Z

@CalebKL No it is not that crash i have uploaded the crash logs in the review comment.

Also, now on my device, it is crashing immediately when I switch to other zim files(I am using the old version of Android for testing Android 8).
Fatal signal 11 (SIGSEGV), code 1, fault addr 0x0 in tid 11992 (DefaultDispatch)
2024-07-18 16:33:50.291 12077-12077 DEBUG                   pid-12077                            A  pid: 11734, tid: 11992, name: DefaultDispatch  >>> org.kiwix.kiwixmobile <<<
2024-07-18 16:33:50.294 12077-12077 DEBUG                   pid-12077                            A      #00 pc 0000000000010630  /data/app/org.kiwix.kiwixmobile-YaobaHgoDzW2aibsfWri3g==/base.apk (offset 0x71f000)
2024-07-18 16:33:50.294 12077-12077 DEBUG                   pid-12077                            A      #01 pc 000000000005be38  /data/app/org.kiwix.kiwixmobile-YaobaHgoDzW2aibsfWri3g==/oat/arm64/base.odex (offset 0x46000)
In the CoreReaderFragment there is a restore tabs functionality also for opening the previous zim file and restoring the previous tabs, and their article opens in the webView(if we are not opening the new zim file).

Currently, it is somehow restoring the previous zim file and tabs when we open the new file. However, it should not do it according to our openPageInBookFromNavigationArguments method's logic. Can you please check this? It is the main reason for these crashes.

let me look into this as well

CalebKL · 2024-07-18T11:42:27Z

@MohitMaliFtechiz When I switch to different zimFiles, I get this crash after numerous attempts.

This crash is newer, on which device you are testing.

I got this crash while testing on my emulator, Pixel 7 pro API34 and also on Samsung S22+

MohitMaliFtechiz · 2024-07-18T12:12:28Z

@CalebKL Thanks, it is a native crash in libzim, but currently, our PR has many bugs and workarounds that can lead to this issue, so probably first we have to fix these things. @mgautierfr can have a better idea about this error when it occurs. Might be the data is corrupted at some point due to these scenarios.

kelson42 · 2024-07-26T10:53:34Z

Any news here?

CalebKL · 2024-07-26T10:59:14Z

Any news here?

There are some fixes that I am working on from @MohitMaliFtechiz reviews. I will complete it ASAP, although this PR will need to wait until the native error(3956) causing the crash is fixed.

MohitMaliFtechiz · 2024-11-22T11:22:42Z

@CalebKL I have found some serious issues in our implementation, especially when working with Coroutines(in a multithreading environment). With the main thread, there are no issues because the implementation was made according to the main thread. But now we are moving to use the Coroutines in our code. The issue we are facing in this PR is related to #4104 which I have already fixed in #4105.

In the meantime, we have also improved our code. So now, we have a better coroutine implementation to create an archive object on the IO thread. Also, we have improved the UI side to properly show users the progress when the archive object is in the creation process: #4074.

So now, you can start working on my review comments.

CalebKL added 6 commits July 15, 2024 17:55

Suspended setZimFile and setZimFileDescriptor and switched dispatcher…

718c519

…s to IO when creating zimFileReader

launched coroutines on lifecycleScope

74ef71e

Passed scope to the constructors and tied the coroutineScope to viewm…

1ac1779

…odelScope

updated errors on the test classes with runblocking

20d7e4c

Passes scope to the test classes to fix errors

bd518f0

Merge branch 'refs/heads/main' into task/anr-fix-zimfile-reader

e148ce1

MohitMaliFtechiz self-requested a review July 15, 2024 17:34

MohitMaliFtechiz requested changes Jul 16, 2024

View reviewed changes

CalebKL added 4 commits July 17, 2024 08:29

fixed compile time errors

b3e4ef2

set setZimFile to null when restoring the tabs

10a813e

wrapped openPageInBookFromNavigationArguments and loadContent into tr…

bc67ec9

…y and catch blocks to monitor the errors

Merge branch 'refs/heads/main' into task/anr-fix-zimfile-reader

fb45bff

CalebKL marked this pull request as draft July 18, 2024 11:40

MohitMaliFtechiz mentioned this pull request Jul 25, 2024

Native crash in getEntryByPath(actualPath), and getData() methods. #3956

Open

Added loading functionality to the notes dialog

eeb184e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task/anr fix zimfile reader #3937

Task/anr fix zimfile reader #3937

CalebKL commented Jul 15, 2024

MohitMaliFtechiz left a comment

MohitMaliFtechiz Jul 16, 2024

MohitMaliFtechiz Jul 16, 2024

MohitMaliFtechiz Jul 16, 2024

MohitMaliFtechiz Jul 16, 2024

CalebKL commented Jul 18, 2024

MohitMaliFtechiz commented Jul 18, 2024

MohitMaliFtechiz commented Jul 18, 2024

CalebKL commented Jul 18, 2024

CalebKL commented Jul 18, 2024 •

edited

Loading

MohitMaliFtechiz commented Jul 18, 2024

kelson42 commented Jul 26, 2024

CalebKL commented Jul 26, 2024

MohitMaliFtechiz commented Nov 22, 2024 •

edited

Loading

Task/anr fix zimfile reader #3937

Are you sure you want to change the base?

Task/anr fix zimfile reader #3937

Conversation

CalebKL commented Jul 15, 2024

MohitMaliFtechiz left a comment

Choose a reason for hiding this comment

MohitMaliFtechiz Jul 16, 2024

Choose a reason for hiding this comment

MohitMaliFtechiz Jul 16, 2024

Choose a reason for hiding this comment

MohitMaliFtechiz Jul 16, 2024

Choose a reason for hiding this comment

MohitMaliFtechiz Jul 16, 2024

Choose a reason for hiding this comment

CalebKL commented Jul 18, 2024

MohitMaliFtechiz commented Jul 18, 2024

MohitMaliFtechiz commented Jul 18, 2024

CalebKL commented Jul 18, 2024

CalebKL commented Jul 18, 2024 • edited Loading

MohitMaliFtechiz commented Jul 18, 2024

kelson42 commented Jul 26, 2024

CalebKL commented Jul 26, 2024

MohitMaliFtechiz commented Nov 22, 2024 • edited Loading

CalebKL commented Jul 18, 2024 •

edited

Loading

MohitMaliFtechiz commented Nov 22, 2024 •

edited

Loading