client/server_args with spaces in them?

[agoodm88]

Hi,

Thanks for sharing this interesting work!

I am trying to fuzz a proprietary binary/client where client_args and server_args have spaces in them - for example --server --arg1 value1 --arg2 value2 --arg3

I am fairly sure I am missing something json related here but if I try to run the session with just 1 argument I get expected output, if I try with 2 arguments I get output I would expect if I missed the spaces out.

I am guessing I need to do something different to ensure the spaces are not stripped as if I recall correctly json strips spaces out, any pointers would be appreciated?

Alan

[domenukk]

Just adding multiple args should work, similar to this:

FitM/config/fitm-args.ts3.json

Line 10 in 6e851f3

"server_args": ["license_accepted=1"],

You'd do for example

"server_args": ["--server", "--arg1", "value1", "--arg2", "value2", "--arg3"],

[agoodm88]

Thanks, this does get me further. After completing this I run into criu complaining that it wants netspace to dump a tun link:

(00.012713) Error (criu/tun.c:306): tun: Net namespace is required to dump tun link

So I thought, ok thats easy, run the binary via unshare -n, however now I run into a similar problem to my args. It looks like the spaces are stripped out of the command. I did try ["modifying", "the", "client"], etc lines similarly to the args lines, but it seems it wants a string and not a sequence. I guess this means I cant fuzz this application in this way?

[domenukk]

The arg list gets passed to the final process here:

FitM/src/lib.rs

Line 296 in 6e851f3

.args(cli_args)

It should not strip anything?
If your target really needs a single arg with a space in between, you would want to do it "like this" but that would be very uncommon

[agoodm88]

Its not the arg list I am struggling with now, criu wants the application running in a separate namespace (I believe?) so I tried modifying "client": and "server": lines adding /usr/bin/unshare -n to the start. Like "client": "/usr/bin/unshare -n /path/to/binary",

Upon running it like this I get a "no such file or directory" error in active-state/stdout which suggests to me that the spaces are being stripped out of the binary path?

[domenukk]

Ah yes you want to specify the files to be included in the namespace/snapshot, see

FitM/config/fitm-args.ts3.json

Line 15 in 6e851f3

"server_files": [

Basically the namespace starts a new chroot env and we copy everything we need into it to run

[agoodm88]

I am not sure what you mean here, I tried adding the path to the directory with the binary in it to "server_files": ["/path/to/dir/with/binary"]

But I still see Error while loading /usr/bin/unshare -n /path/to/dir/with/binary/actualbinary: No such file or directory in active-state/stdout so I feel like I am barking up the wrong tree?

[domenukk]

The path inside the namespace might be different, the files get copied in, if that makes sense?
Maybe if you run it with the QEMU_STRACE=1 env variable set (in the config Json) you can see which path it tries to access

[agoodm88]

I am already running with "server_envs": {"QEMU_STRACE": "1"},

Where would I see the errors which might help me diagnose what is wrong?

What is server_files supposed to be set to? Full path to each required binary? Something else?

[domenukk]

You'd see it in the stderr file (or whatever the name is).
unshare -n creates a new namespace, I's not really be surprised if that breaks things. Why do you need it?

[agoodm88]

Because criu is complaining that I need one if I dont use one:

(00.012713) Error (criu/tun.c:306): tun: Net namespace is required to dump tun link

Maybe its not possible to fuzz vpn server/client in this manner?

[domenukk]

I guess the tub device needs to be part of the namespace? It's a rather low level component, I wouldn't know the answer. Likely our network emulator wouldn't work on the tun device, either

[agoodm88]

I am only really interested in fuzzing the negotiation phase of the connection, not the established connection once its created (where data would flow over the tun device). Does this change anything?

[domenukk]

Can you maybe nop out all tun device interactions? That'd probably be the easiest solution.
Else you may need to add the tun device to the namespace FitM creates somehow

[derpsteb]

Heyhey,
cool to see that you are testing the fuzzer :)!

Can you try changing this line to look like this: unsafe { sys_clone(libc::CLONE_NEWNET | libc::CLONE_NEWPID | libc::CLONE_NEWNS | libc::SIGCHLD)? };?
This is the function we use to create the new namespaces in which the targets will get restored. Adding CLONE_NEWNET will spawn the target inside a new net ns. This worked for me in the sense that I could see a different net namespace id inside the target, compared to the net ns on the host. I checked by adding lsns to the restore.sh.tmp. This way the restore script will print the namespace ids before restoring the process.

I am not sure if you really want to do this as your target might now not have access to the tun interface that you possibly created manually in the initial net ns(?).
Otherwise you could try to run unshare -n /bin/bash before starting fitm all together?

[agoodm88]

Thanks for your input. Running the entire session in an 'unshare -n /bin/bash' session, and the edit you posted above both result in things getting further before bailing out. Both methods seem to achieve the same outcome.

It now seems to be bailing out because its trying to read from /dev/urandom which apparently isnt available. I did try adding /dev/urandom to the server_files but this didnt help.

[derpsteb]

My guess would be that something gets messed up with /dev because/while we are creating a new mount ns before restoring. I will have to take another look tomorrow :)

[derpsteb]

Hm. So I have been trying to open /dev/urandom in the restore script while having the above patch in namespacing.rs. Works fine. I also ran bash in it's own namespace like this unshare -n -m -p --fork /bin/bash - no problem opening /dev/urandom.
Can you see the open syscall failing in the strace?

If so you could patch do_openat to gather some information on what is there and what isn't. Not sure what to look for here.

A way to fake /dev/urandom could be to bind mount a text file to that path during our namespace setup function. The mount call would go here. In the lines above you can see how we are mounting the proc-fs. What you would want to do there is:

cat "randomwhoooo" > /some/path
mount --bind /some/path /dev/urandom

sauce

Hope this helps!

[agoodm88]

Thanks, very interesting points. I had presumed that for fuzzing purposes a static set of 'random' data might be better as I've had interesting problems fuzzing stuff that uses random data inputs in the past.

I am only guessing at the issue here. A fresh pair of eyes suggests I actually still have a namespace related problem. I get the same pattern of errors with the patch above, or running inside bash ran via unshare -n.

Here is more information:

sudo rm -rf ./active-state
sudo rm -rf ./cmin-tmp
sudo ./target/release/fitm config/fitm-args.redacted1.json
cwd: "/home/alan/scratch/FitM"

__________________  ___

/ / / _/ |/ /
/ / / / / / / /|/ /
/ / / / / / / / / /
// // // // /_/

File fitm-state.json not found. Restarting from scratch.
No valid state to resume. Starting fresh :)
==== [] Time start init_run: 2022-03-28 21:33:30 ====
[] Init run finished with exit code None
[] Target was killed by signal. Assuming dump success.
[] Init run finished with exit code Some(0)
[!] Unexpected exit status '0' from snapshot creation.
thread 'main' panicked at 'Namespace call failed with error Custom { kind: Other, error: "[!] criu dump failed, check active-state dir." }', src/namespacing.rs:135:31
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.
[*] Init run finished with exit code Some(255)
[!] Unexpected exit status '255' from snapshot creation.
thread 'main' panicked at 'Namespace call failed with error Custom { kind: Other, error: "[!] criu dump failed, check active-state dir." }', src/namespacing.rs:135:31
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.
thread 'main' panicked at '[!] Snapshot in init_run failed. Check latest active-state folder for clues.', src/lib.rs:360:17
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.
make: *** [Makefile:40: run] Error 101

End of criu.log:

(00.014877) 16865 fdinfo 9: pos: 0 flags: 104002/0
(00.014894) Error (criu/tun.c:306): tun: Net namespace is required to dump tun link
(00.014912) ----------------------------------------
(00.014933) Error (criu/cr-dump.c:1351): Dump files (pid: 16865) failed with -1
(00.014981) Waiting for 16865 to trap
(00.015143) Daemon 16865 exited trapping
(00.015156) Sent msg to daemon 3 0 0
(00.015168) Force no-breakpoints restore
(00.015241) 16865 was trapped
(00.015249) 16865 (native) is going to execute the syscall 45, required is 15
(00.015301) 16865 was trapped
(00.015306) - Expecting exit (00.015347) 16865 was trapped (00.015353) 16865 (native) is going to execute the syscall 186, required is 15 (00.015393) 16865 was trapped (00.015397) - Expecting exit
(00.015439) 16865 was trapped
(00.015445) 16865 (native) is going to execute the syscall 1, required is 15
pie: 16865: __fetched msg: 3 0 0
(00.015510) 16865 was trapped
(00.015517) - Expecting exit (00.015571) 16865 was trapped (00.015577) 16865 (native) is going to execute the syscall 186, required is 15 (00.015616) 16865 was trapped (00.015621) - Expecting exit
(00.015661) 16865 was trapped
(00.015667) 16865 (native) is going to execute the syscall 186, required is 15
(00.015706) 16865 was trapped
(00.015710) - Expecting exit (00.015752) 16865 was trapped (00.015758) 16865 (native) is going to execute the syscall 1, required is 15 pie: 16865: 16865: new_sp=0x7f0f2b41fa88 ip 0x7f0f2bef47bc (00.015810) 16865 was trapped (00.015816) - Expecting exit
(00.015857) 16865 was trapped
(00.015863) 16865 (native) is going to execute the syscall 3, required is 15
(00.015925) 16865 was trapped
(00.015929) - Expecting exit (00.015963) 16865 was trapped (00.015969) 16865 (native) is going to execute the syscall 3, required is 15 (00.016031) 16865 was trapped (00.016036) - Expecting exit
(00.016078) 16865 was trapped
(00.016085) 16865 (native) is going to execute the syscall 15, required is 15
(00.016121) 16865 was stopped
(00.016336) Unlock network
(00.016346) Unfreezing tasks into 1
(00.016350) Unseizing 16865 into 1
(00.016385) Error (criu/cr-dump.c:1768): Dumping FAILED.

stderr file:

16865 brk(NULL) = 0x0000004000220000
16865 uname(0x4001a1ff10) = 0
16865 openat(-100,"/usr/lib/x86_64-linux-gnu/coreutils/libstdbuf.so",O_RDONLY|O_CLOEXEC) = 3
16865 read(3,0x1a1eb38,832) = 832
16865 fstat(3,0x0000004001a1e9d0) = 0
16865 mmap(NULL,16488,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE,3,0) = 0x0000004001a4d000
16865 mmap(0x0000004001a4e000,4096,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x1000) = 0x0000004001a4e000
16865 mmap(0x0000004001a4f000,4096,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x2000) = 0x0000004001a4f000
16865 mmap(0x0000004001a50000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x2000) = 0x0000004001a50000
16865 close(3) = 0
16865 access("/etc/ld.so.preload",R_OK) = -1 errno=2 (No such file or directory)
16865 openat(-100,"/etc/ld.so.cache",O_RDONLY|O_CLOEXEC) = 3
16865 fstat(3,0x0000004001a1f4c0) = 0
16865 mmap(NULL,68375,PROT_READ,MAP_PRIVATE,3,0) = 0x0000004001a52000
16865 close(3) = 0
16865 openat(-100,"/lib/x86_64-linux-gnu/libpthread.so.0",O_RDONLY|O_CLOEXEC) = 3
16865 read(3,0x1a1f688,832) = 832
16865 fstat(3,0x0000004001a1f520) = 0
16865 mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000004001a63000
16865 mmap(NULL,132288,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE,3,0) = 0x0000004001a65000
16865 mmap(0x0000004001a6b000,61440,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x6000) = 0x0000004001a6b000
16865 mmap(0x0000004001a7a000,24576,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x15000) = 0x0000004001a7a000
16865 mmap(0x0000004001a80000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x1a000) = 0x0000004001a80000
16865 mmap(0x0000004001a82000,13504,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0x0000004001a82000
16865 close(3) = 0
16865 openat(-100,"/lib/x86_64-linux-gnu/libm.so.6",O_RDONLY|O_CLOEXEC) = 3
16865 read(3,0x1a1f668,832) = 832
16865 fstat(3,0x0000004001a1f500) = 0
16865 mmap(NULL,1581384,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE,3,0) = 0x0000004001a86000
16865 mmap(0x0000004001a93000,651264,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0xd000) = 0x0000004001a93000
16865 mmap(0x0000004001b32000,872448,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0xac000) = 0x0000004001b32000
16865 mmap(0x0000004001c07000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x180000) = 0x0000004001c07000
16865 close(3) = 0
16865 openat(-100,"/lib/x86_64-linux-gnu/libdl.so.2",O_RDONLY|O_CLOEXEC) = 3
16865 read(3,0x1a1f648,832) = 832
16865 fstat(3,0x0000004001a1f4e0) = 0
16865 mmap(NULL,16656,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE,3,0) = 0x0000004001c09000
16865 mmap(0x0000004001c0a000,4096,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x1000) = 0x0000004001c0a000
16865 mmap(0x0000004001c0b000,4096,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x2000) = 0x0000004001c0b000
16865 mmap(0x0000004001c0c000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x2000) = 0x0000004001c0c000
16865 close(3) = 0
16865 openat(-100,"/lib/x86_64-linux-gnu/libc.so.6",O_RDONLY|O_CLOEXEC) = 3
16865 read(3,0x1a1f628,832) = 832
16865 fstat(3,0x0000004001a1f4c0) = 0
16865 mmap(NULL,1837056,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE,3,0) = 0x0000004001c0e000
16865 mprotect(0x0000004001c30000,1658880,PROT_NONE) = 0
16865 mmap(0x0000004001c30000,1343488,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x22000) = 0x0000004001c30000
16865 mmap(0x0000004001d78000,311296,PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x16a000) = 0x0000004001d78000
16865 mmap(0x0000004001dc5000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x1b6000) = 0x0000004001dc5000
16865 mmap(0x0000004001dcb000,14336,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0x0000004001dcb000
16865 close(3) = 0
16865 mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000004001dcf000
16865 arch_prctl(4098,274909166464,274909168800,144,1,64) = 0
16865 mprotect(0x0000004001dc5000,16384,PROT_READ) = 0
16865 mprotect(0x0000004001c0c000,4096,PROT_READ) = 0
16865 mprotect(0x0000004001c07000,4096,PROT_READ) = 0
16865 mprotect(0x0000004001a80000,4096,PROT_READ) = 0
16865 mprotect(0x0000004001a50000,4096,PROT_READ) = 0
16865 mprotect(0x000000400019a000,4096,PROT_READ) = 0
16865 mprotect(0x0000004001a48000,4096,PROT_READ) = 0
16865 munmap(0x0000004001a52000,68375) = 0
16865 set_tid_address(274909167184,274905301944,274909166464,274905471472,274905476752,47) = 16865
16865 set_robust_list(274909167200,24,274909166464,274905471472,274905476752,47) = -1 errno=38 (Function not implemented)
16865 rt_sigaction(32,0x0000004001a20100,NULL) = 0
16865 rt_sigaction(33,0x0000004001a20100,NULL) = 0
16865 rt_sigprocmask(SIG_UNBLOCK,0x0000004001a20278,NULL) = 0
16865 prlimit64(0,3,0,274905301600,0,47) = 0
16865 gettimeofday(0x0000004001a1fd60,NULL) = 0 ({tv_sec = 1648500599,tv_usec = 17000},NULL)
16865 brk(NULL) = 0x0000004000220000
16865 brk(0x0000004000241000) = 0x0000004000241000
16865 openat(-100,"/etc/localtime",O_RDONLY|O_CLOEXEC) = 3
16865 fstat(3,0x0000004001a1fbb0) = 0
16865 fstat(3,0x0000004001a1f9e0) = 0
16865 read(3,0x2204b0,4096) = 3687
16865 lseek(3,-2347,SEEK_CUR) = 1340
16865 read(3,0x2204b0,4096) = 2347
16865 close(3) = 0
16865 gettimeofday(0x0000004001a1fd60,NULL) = 0 ({tv_sec = 1648500599,tv_usec = 27392},NULL)
16865 stat("/etc/localtime",0x0000004001a1fbb0) = 0
16865 gettimeofday(0x0000004001a1fd60,NULL) = 0 ({tv_sec = 1648500599,tv_usec = 27608},NULL)
16865 stat("/etc/localtime",0x0000004001a1fbb0) = 0
16865 gettimeofday(0x0000004001a1fd60,NULL) = 0 ({tv_sec = 1648500599,tv_usec = 27671},NULL)
16865 stat("/etc/localtime",0x0000004001a1fbb0) = 0
16865 gettimeofday(0x0000004001a1fd60,NULL) = 0 ({tv_sec = 1648500599,tv_usec = 27829},NULL)
16865 stat("/etc/localtime",0x0000004001a1fbb0) = 0
16865 gettimeofday(0x0000004001a1fd60,NULL) = 0 ({tv_sec = 1648500599,tv_usec = 27898},NULL)
16865 stat("/etc/localtime",0x0000004001a1fbb0) = 0
16865 gettimeofday(0x0000004001a1fd60,NULL) = 0 ({tv_sec = 1648500599,tv_usec = 27959},NULL)
16865 stat("/etc/localtime",0x0000004001a1fbb0) = 0
16865 fstat(1,0x0000004001a1fca0) = 0
16865 write(1,0x221e20,38) = 38
16865 write(1,0x221e20,57) = 57
16865 write(1,0x221e20,46) = 46
16865 write(1,0x221e20,99) = 99
16865 write(1,0x221e20,99) = 99
16865 write(1,0x221e20,47) = 47
16865 write(1,0x221e20,53) = 53
16865 clock_getres(CLOCK_MONOTONIC_COARSE,0x0000004001a1fda0) = 0 ({tv_sec = 0,tv_nsec = 4000000})
16865 clock_gettime(CLOCK_MONOTONIC,0x0000004001a1fda0) = 0 ({tv_sec = 202141,tv_nsec = 184202960})
16865 epoll_create1(524288) = 3
16865 pipe2(274879600672,524288,524288,-1014,202,1) = 0
16865 write(5,0x1a1fd3b,1) = 1
16865 futex(0x000000400019ee20,FUTEX_PRIVATE_FLAG|FUTEX_WAKE,2147483647,NULL,0x00000000000000ca,202) = 0
16865 pipe2(274879605992,526336,526336,0,202,1) = 0
16865 eventfd2(0,526336) = 8
16865 gettimeofday(0x0000004001a1fd40,NULL) = 0 ({tv_sec = 1648500599,tv_usec = 33448},NULL)
16865 stat("/etc/localtime",0x0000004001a1fb90) = 0
16865 gettimeofday(0x0000004001a1fd10,NULL) = 0 ({tv_sec = 1648500599,tv_usec = 33967},NULL)
16865 stat("/etc/localtime",0x0000004001a1fb60) = 0
16865 socket(PF_INET,SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK,IPPROTO_IP) = 999
16865 setsockopt(999,1,2,274905300316,4,274905299459) = 0
16865 bind(999,{sin_family=AF_INET,sin_port=htons(19655),sin_addr=inet_addr("127.0.0.1")}, 16) = 0
16865 setsockopt(999,1,7,274905300476,4,274905299459) = 0
16865 setsockopt(999,1,8,274905300476,4,274905299459) = 0
16865 openat(-100,"/dev/net/tun",O_RDWR) = 9
16865 ioctl(9,TUNSETIFF,{"redacted-test",4097}) = 0 ({"redacted-test",4097})
16865 epoll_ctl(3,1,9,274905300384,-72340172838076673,-72340172830444956) = 0
16865 epoll_ctl(3,2,9,274905300384,-72340172838076673,-72340172830444956) = 0
16865 ioctl(9,FIONBIO,1) = 0
16865 openat(-100,"/home/alan/scratch/redacted/certs/shared.crt",O_RDONLY) = 10
16865 fstat(10,0x0000004001a1f7b0) = 0
16865 fstat(10,0x0000004001a1f8a0) = 0
16865 lseek(10,0,SEEK_SET) = 0
16865 read(10,0x222e30,1294) = 1294
16865 lseek(10,1294,SEEK_SET) = 1294
16865 close(10) = 0
16865 openat(-100,"/home/alan/scratch/redacted/certs/server.key",O_RDONLY) = 10
16865 fstat(10,0x0000004001a1f7b0) = 0
16865 fstat(10,0x0000004001a1f8a0) = 0
16865 lseek(10,0,SEEK_SET) = 0
16865 read(10,0x222e30,1704) = 1704
16865 lseek(10,1704,SEEK_SET) = 1704
16865 close(10) = 0
16865 openat(-100,"/dev/urandom",O_RDONLY) = 10
16865 read(10,0x1a1fd60,104) = 104
16865 close(10) = 0
16865 epoll_ctl(3,2,9,274905300396,274909142080,274880137616) = -1 errno=2 (No such file or directory)
16865 rt_sigaction(SIGPIPE,0x0000004001a1fbc0,0x0000004001a1fc60) = 0
16865 clock_gettime(CLOCK_MONOTONIC,0x0000004001a1fde0) = 0 ({tv_sec = 202141,tv_nsec = 208895265})
16865 epoll_ctl(3,1,6,274905288100,274905300400,274880137616) = 0
16865 epoll_ctl(3,1,8,274905288100,274905300400,274880137616) = 0
16865 epoll_ctl(3,1,999,274905288100,274905300400,274880137616) = 0
16865 epoll_ctl(3,1,9,274905288100,274905300400,274880137616) = 0
16865 epoll_wait(3,274905288112,1024,4294967295,274905300400,274880137616) = 1
16865 clock_gettime(CLOCK_MONOTONIC,0x0000004001a1ccb0) = 0 ({tv_sec = 202141,tv_nsec = 209940575})

To me this seems the issue is more complicated than I originally thought, perhaps the netspace issue is not resolved though it appears to be getting further when ran with the patch above, or via bash started with unshare -n.

[derpsteb]

Do you want to have a debugging session to take a look? Somewhere, where I can contact you?

[agoodm88]

Sure, do you have an email address or method I can contact you off list?

[derpsteb]

Use this :)
[email protected]

[derpsteb]

After having a debugging session we found that adding an unshare -n here resolves the problem about criu not being able to dump the tun interface. The patch described earlier seemingly doesn't do the same. Have to investigate what else unshare -n does..

[agoodm88]

Hiya,

Many thanks once again for taking the time to look at this at the week end and indeed sharing this interesting project.

I've had this running for a few days now, number of testcases is increasing gradually in the expected manner. No errors to note really as far as I can see. I am still not getting any generation 3 snapshots, but this may be expected?

Since I now have a pile of test cases I am of course now itching to perform some manual triage of these to see if anything exciting is happening. There doesn't seem to be much documentation in regard of where to even begin with this. The test cases look somewhat like a dump of the traffic between the two daemons being tested, however I tried replaying them like a PCAP file, feeding them into restore.sh, etc and am not getting anywhere.

Presumably the best way to triage this further would be to 'replay' the test case, capturing STDOUT/STDERR for client/server and a pcap file of the traffic which results. I can then perform automated analysis of the output looking for anything interesting but since I lack an understanding of what format the testcases are taking I'm not sure where to begin...

alan@debian-KVM:~/scratch/FitM/active-state$ sudo ./restore.sh ../cmin-tmp/imported1000
++ realpath ../cmin-tmp/imported1000

• export INPUT_FILENAME=/home/alan/scratch/FitM/cmin-tmp/imported1000
• INPUT_FILENAME=/home/alan/scratch/FitM/cmin-tmp/imported1000
• ENVFILE=envfile
• env
• sort
• pwd
  /home/alan/scratch/FitM/active-state
  ++ grep -o 'pipe:[.*]'
  ++ head -n 1
  ++ cat ./pipes
• PIPE1='pipe:[34598034]'
  ++ cat ./pipes
  ++ grep -o 'pipe:[.*]'
  ++ head -n 2
  ++ tail -n 1
• PIPE2='pipe:[34598035]'
• echo -n ''
• [[ -z '' ]]
• CRIU_SNAPSHOT_DIR=./snapshot
• [[ -z '' ]]
• exec
• exec
• echo 32768
• ../criu/criu/criu restore -d -vvv -o ../restore.log --images-dir ./snapshot --inherit-fd 'fd[198]:pipe:[34598034]' --inherit-fd 'fd[199]:pipe:[34598035]'