Compile on 64bit system

[simont77]

Hi,
do you know if this source code can be compiled safely in 64bit mode without affecting the test accuracy?
Thanks

[pierrelecuyer]

There are many places in the code where 32-bit integers must really be 32 bits, otherwise the bit shifts and rotations will not work properly, for example.  We have a 64-bit version in the works, but it has been delayed by all sorts of head winds lately.

-- Pierre L'Ecuyer

On 11/26/2021 11:24 AM, simont77 wrote: Hi, do you know if this source code can be compiled safely in 64bit mode without affecting the test accuracy? Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVHLIYPYDESSQNEQ33ZSCDUN6YEDANCNFSM5I27ZWFA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Pierre L'Ecuyer, Professeur Titulaire CIRRELT, GERAD, and DIRO, Université de Montréal, Canada http://www.iro.umontreal.ca/~lecuyer

[simont77]

Hi again,

actually I conducted some test.

I generated a 2Gbyte file using my own RNG (I limited the file size to 2Gbyte due to the limitation of fstream on 32bit system).
I compiled TestU01 both on a 32bit and 64bit x86 system.
I performed a set of alphabit_file and rabbit_file on both platforms using the 2Gbyte file, setting the number of bits of the test to 1 billion, 2 billion and 16 billion (i.e. in this last case more than 2^32 bit).

Results for the same number of bits where always identical for the 32bit and 64bit platform, apart in some cases for the last significant digit of very small values (e.g. Mu = 9.3132258e-10 vs Mu = 9.3132257e-10), which I suppose is due to floating point rounding error with different machine precision.

So, it seems that the 64bit executable has no issue, at least for alphabit and rabbit test suite. The main reason I would like to move to 64bit is to be able to test random streams (only with alphabit and rabbit) with more than 16 billion bit, and presently the 32bit version it limited to this number due to fstream, while it seems to handle correctly more than 2^32 bit.

What do you think?

Thanks

[aleruggeri87]

Hi,
I am testing an HW RNG too, saving generated random bits on a file and running Alphabit & Rabbit suites.
I had the same issue in reading large files using an old 32-bit Ubuntu 16. However I was able to solve it using a modern 64-bit Ubuntu 20, and executing ./configure --host=i686-linux-gnu "CFLAGS=-m32" "CXXFLAGS=-m32" "LDFLAGS=-m32" which forces the generation of a 32-bit executable. On my system I also need to force the usage of the Large-file support (LFS); I enabled it as in this commit, but probably there are better and portable ways to do that (perhaps via the configure script?).
At this point the fopen() is no more the issue, but there are other points in the code where some long variables (32-bit) are overflowing.
I made this PR, hopefully I spotted and corrected these overflow issues.

[simont77]

I confirm that using your patch I can now test files larger than 2Gbyte even compiling in 32bit! But now I have a doubt: is it meaningful to test such a large amount of data or Alphabit and Rabbit were not designed for that?

[aleruggeri87]

Thank you for testing the patch!

Regarding your question, I had a similar doubt: I've seen a few papers where (mostly) Alphabit was used on large dataset (up to 1Tbit!); however compiling a 32-bit executable lead to the previously discussed issues, while from this thread I understood that the author of the code advise against compiling a 64-bit executable.

I have very little experience on statistical testing of RNG, thus I cannot say if it make sense to analyze such large amount of data... It would be great if Prof. @pierrelecuyer or some other user, with deep knowledge of the code for the various statistical tests performed in the suites, could comment more on that.

[pierrelecuyer]

Sorry for my slow response. Richard Simard, who was maintaining TestU01, has retired long ago, and on my side, I am fighting a pancreas cancer, trying to stay alive. This has slowed me down a lot recently, but the plan to come up with TestU01-64 is still alive; I hope I can get back to it next Summer of Fall. I would also be happy to welcome contributors. On the other hand, I do not plan to make any further changes or maintenance to TestU01-2009. The latter was built for 32-bit computers, a long ago. I am pretty sure some of the code will not work properly if compiled in 64 bits with no change. But perhaps some of the tests will still work correctly.
Regarding the size of files for Alphabit, there is no fundamental reason to limit them to 2 Gbytes. The code is written with 32-bit integers, but the string of bits that is tested can be arbitrarily long. With much larger files, perhaps we could change (increase) the values of L that are used in the tests, to improve the sensitivity, but this is not essential. So I think it is fine to allow larger files.

[t-vi]

Happy new year! I hope this finds you reasonably well.

but the plan to come up with TestU01-64 is still alive; I hope I can get back to it next Summer of Fall. I would also be happy to welcome contributors. On the other hand, I do not plan to make any further changes or maintenance to TestU01-2009.

Is there an articulated plan in terms of goals / tasks / ...to get to TestU01-64?
It seems that TestU01 is still quite the reference for testing PRNGs...