Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zimsplit (zim-tools_win-x86_64-3.5.0) can not handle well zim.files biger than ca.80gb spliting to Default: 2GB #433

Open
BD-RE opened this issue Nov 11, 2024 · 3 comments

Comments

@BD-RE
Copy link

BD-RE commented Nov 11, 2024

I tried to split wikipedia_en_all_maxi_2024-01.zim with 102 GB (109.885.670.576 Bytes) to Default: 2GB.
Result I got ca. 40 times 2GB files and a bigger leftover.


zim-tools 3.5.0

libzim 9.2.3

  • libzstd 1.5.5
  • liblzma 5.2.11
  • libxapian 1.4.23
  • libicu 74.1.0
@BD-RE
Copy link
Author

BD-RE commented Nov 12, 2024

C:\Users\me>"C:\Users\me\Desktop\offline-wiki burn-projekt\Tools\zim-tools_win-x86_64-3.5.0\zimsplit.exe" "y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zim"
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimaa
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimab
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimac
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimad
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimae
........
log shortened see the last 4 lines:
........
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimbl
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimbm
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimbn
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimbo
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimbp
WARNING: Part y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimbp is bigger that max part size. (3485777929>2147483648)
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimbq
opening new file y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimbr
WARNING: Part y:\kiwix-zim\wikipedia_en_all_maxi_2024-01.zimbr is bigger that max part size. (19317566780>2147483648)
..........
.........dir shortened see the last 4 lines:

12.11.2024 13:30 2.146.437.678 wikipedia_en_all_maxi_2024-01.zimbk
12.11.2024 13:32 2.147.268.773 wikipedia_en_all_maxi_2024-01.zimbl
12.11.2024 13:35 2.146.431.290 wikipedia_en_all_maxi_2024-01.zimbm
12.11.2024 13:37 2.146.400.141 wikipedia_en_all_maxi_2024-01.zimbn
12.11.2024 13:39 1.206.591.345 wikipedia_en_all_maxi_2024-01.zimbo
12.11.2024 13:43 3.485.777.929 wikipedia_en_all_maxi_2024-01.zimbp
12.11.2024 13:43 93.999 wikipedia_en_all_maxi_2024-01.zimbq
12.11.2024 14:06 19.317.566.780 wikipedia_en_all_maxi_2024-01.zimbr

@kelson42
Copy link
Contributor

Yes, here there is nothing we can do as we can not cut the fulltext index...

@BD-RE
Copy link
Author

BD-RE commented Nov 13, 2024

Did I understand well, the full text index is appended at the end of the archive and in this case ca. 19gb?
But in this case (status) it is functional.
If somebody split the left over of the process anyway, he have to join it together to get it functional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants