Fix up vmayarascan and vadyarascan to use yarascan properly

[ikelos]

This gets vmayarascan and vadyarascan to use the scanning framework properly, but hands in chunks that are the size of the complete vad.

This resolves issues around plugins needing to know about the version of vad whilst keeping the fix that led to the divergence...

Supercedes #1370.
Closes #1367

[ikelos]

This also fixes issues with the previous commit where windows code from vadyarascan made its way into vmayarascan, suggesting the review process isn't all that thorough...

[gcmoreira]

Thanks GitHub for shuffling the order of the comments in the review :(.
@ikelos the first one should be this #1371 (comment) .. please let me know if anything doesn't make sense.

[ikelos]

Ok, lemme know if that's better... 5:)

[eve-mem]

I've tested this and it seems to fix the issue in #1367

(volatility3) eve@xps:~/Documents/volatility3$ ./vol.py -f linux-sample-1.dmp linux.vmayarascan.VmaYaraScan --pid 1 --yara-string "
ELF"
Volatility 3 Framework 2.12.0
Progress:  100.00               Stacking attempts finished                 
Offset  PID     Rule    Component       Value

0x7f489741c001  1       r1      $a      45 4c 46
0x7f4897620001  1       r1      $a      45 4c 46
0x7f48979ab001  1       r1      $a      45 4c 46
0x7f4897e0b001  1       r1      $a      45 4c 46
0x7f4897e108f4  1       r1      $a      45 4c 46
0x7f4897e24151  1       r1      $a      45 4c 46
0x7f4897e24161  1       r1      $a      45 4c 46
0x7f4897e24583  1       r1      $a      45 4c 46
<snip>

I confirmed those results in volshell - and it all appears to be in order.:

(volatility3) eve@xps:~/Documents/volatility3$ ./volshell.py -f linux-sample-1.dmp -l
Volshell (Volatility 3 Framework) 2.12.0
Readline imported successfully  Stacking attempts finished                 

    Call help() to see available functions

    Volshell mode        : Linux
    Current Layer        : layer_name
    Current Symbol Table : symbol_table_name1
    Current Kernel Name  : kernel

(layer_name) >>> ct(1)
(layer_name_Process1) >>> db(0x7f489741c001, 8)
0x7f489741c001    45 4c 46 02 01 01 00 00                            ELF.....
(layer_name_Process1) >>> rx(b'ELF', 8)
0x7f489741c001    45 4c 46 02 01 01 00 00                            ELF.....

0x7f4897620001    45 4c 46 02 01 01 00 00                            ELF.....

0x7f48979ab001    45 4c 46 02 01 01 00 00                            ELF.....

0x7f4897e0b001    45 4c 46 02 01 01 00 00                            ELF.....

0x7f4897e108f4    45 4c 46 0f 85 6e 02 00                            ELF..n..

0x7f4897e24151    45 4c 46 02 01 01 00 00                            ELF.....

0x7f4897e24161    45 4c 46 02 01 01 03 00                            ELF.....

0x7f4897e24583    45 4c 46 3a 20 20 20 20                            ELF:....

0x7f4897e250e4    45 4c 46 20 66 69 6c 65                            ELF.file

0x7f4897e25109    45 4c 46 20 68 65 61 64                            ELF.head

0x7f4897e25114    45 4c 46 20 66 69 6c 65                            ELF.file

I've also tested it an actual fules file:

(volatility3) eve@xps:~/Documents/volatility3$ ./vol.py -f linux-sample-1.dmp linux.vmayarascan.VmaYaraScan --yara-file ./rules.yar
Volatility 3 Framework 2.12.0
Progress:  100.00               Stacking attempts finished                 
Offset  PID     Rule    Component       Value

0x7fc90a315751  322     RegExpExample2  $re2    62 61 72 72
0x7fc90a315766  322     RegExpExample2  $re2    62 61 72 72
0x7fc90a31577e  322     RegExpExample2  $re2    62 61 72 72
0x7fc90a315793  322     RegExpExample2  $re2    62 61 72 72
0x7fc90a3157ac  322     RegExpExample2  $re2    62 61 72 72
0x7fc90a3157c8  322     RegExpExample2  $re2    62 61 72 72
0x7fc90a3157e7  322     RegExpExample2  $re2    62 61 72 72
<snip>

Then on windows too

yara-string:

(volatility3) eve@xps:~/Documents/volatility3$ ./vol.py -f win-xp-laptop-2005-06-25.img windows.vadyarascan --yara-string "MZ"
Volatility 3 Framework 2.12.0
Progress:  100.00               PDB scanning finished                        
Offset  PID     Rule    Component       Value

0x7c900000      4       r1      $a      4d 5a
0x48580000      448     r1      $a      4d 5a
0x4858aadf      448     r1      $a      4d 5a
0x7c900000      448     r1      $a      4d 5a
0x7c910809      448     r1      $a      4d 5a
0x7c9201ff      448     r1      $a      4d 5a
0xf19a0 504     r1      $a      4d 5a
0x450000        504     r1      $a      4d 5a
0x757aaf        504     r1      $a      4d 5a

yara-file:

(volatility3) eve@xps:~/Documents/volatility3$ ./vol.py -f win-xp-laptop-2005-06-25.img windows.vadyarascan --yara-file ./rules.yar
Volatility 3 Framework 2.12.0
Progress:  100.00               PDB scanning finished                        
Offset  PID     Rule    Component       Value

0x18bfc7        504     RegExpExample2  $re2    62 61 72 57
0x18c034        504     RegExpExample2  $re2    62 61 72 33
0x18c109        504     RegExpExample2  $re2    62 61 72 33
0x81205c        504     RegExpExample2  $re2    62 61 72 33
0x86905d        504     RegExpExample2  $re2    62 61 72 33
0x869d24        504     RegExpExample2  $re2    62 61 72 57
0x869ee5        504     RegExpExample2  $re2    62 61 72 33
0x8a6174        504     RegExpExample2  $re2    62 61 72 57
0x8a62cd        504     RegExpExample2  $re2    62 61 72 33
0x8a8274        504     RegExpExample2  $re2    62 61 72 33
0x8a8d54        504     RegExpExample2  $re2    62 61 72 57
<snip>

For reference rules.yar:

rule RegExpExample2
{
    strings:
        $re1 = /foo/i    // This regexp is case-insentitive
        $re2 = /bar./s   // In this regexp the dot matches everything, including new-line
        $re3 = /baz./is  // Both modifiers can be used together
    condition:
        any of them
}

[ikelos]

Thanks @eve-mem , so does that still solve the problem that the original vadyarascan/vmayarascan changes were made to resolve?

[eve-mem]

e.g. a yara signature that needs the context of the entire VMA to produce a hit?

I haven't yet tested that actually....!

[ikelos]

Ok, I just know we changed it by request because the scanning worked on blocks that were too small for context (ie, the rule needed to act across the whole vad, rather than just 16Mb chunks of it). I want to make sure we're not swapping one bug for another... 5;P

[eve-mem]

That sounds sensible - I'm just running some tests on that now.

[eve-mem]

No it doesn't seem to be working. I created a rule that should hit a specific vma. Using some strings at the start and the end of it.

Although I'm not sure 'all of them' conditions worked before? I have to head out for a while - I can do more testing later.

Extracting the VMAs and running yara on it manually produces a result as expected, but trying to use the vmayarascan plugin it doesn't seem to work. :(

Here's the full testing if you wanted to recreate it:

# run proc maps to extract vmas
(volatility3) eve@xps:~/Documents/volatility3$ python vol.py -f linux-sample-1.dmp linux.proc.Maps --pid 8600 --dump

# use strings and head to to find something unique at the start of the vma.
(volatility3) eve@xps:~/Documents/volatility3$ strings pid.8600.vma.0x7fe78a85f000-0x7fe78a86a000.dmp | head -n 25
&F[9
'0!^a
"[1=/Z
__gmon_start__
__cxa_finalize
_Jv_RegisterClasses
rewind
fopen
__errno_location
fgets_unlocked
__ctype_b_loc
_nss_files_parse_grent
_nss_files_getgrgid_r
__pthread_mutex_lock
__pthread_mutex_unlock
fclose
_nss_files_endgrent
_nss_files_getgrnam_r
strcmp
_nss_files_setgrent
fgetpos
_nss_files_getgrent_r
fsetpos
_nss_files_parse_pwent
_nss_files_getpwuid_r

# use strings and tail to find something unique at the end of the vma.
(volatility3) eve@xps:~/Documents/volatility3$ strings pid.8600.vma.0x7fe78a85f000-0x7fe78a86a000.dmp | tail -n 25
/etc/gshadow
/etc/protocols
/etc/services
/etc/hosts
nss_files/files-hosts.c
buflen >= bufferend - buffer
bufferend <= buffer + buflen
result.h_addr_list[1] == ((void *)0)
(_res_hconf.flags & (1 << 4)) != 0
(bufferend - (char *) 0) % sizeof (char *) == 0
(((uintptr_t) newp) & (__alignof__ (struct hostent_data) - 1)) == 0
(((uintptr_t) tmp_buffer) & (__alignof__ (struct hostent_data) - 1)) == 0
_nss_files_gethostbyname4_r
_nss_files_gethostbyname2_r
_nss_files_gethostbyname_r
/etc/networks
/etc/rpc
/etc/ethers
/etc/netgroup
/etc/publickey
#       :
/etc/aliases
:include:
/lib64/ld-linux-x86-64.so.2
;*3$"

# move the extracted vmas to a folder to make scanning easier
(volatility3) eve@xps:~/Documents/volatility3$ mkdir maps
(volatility3) eve@xps:~/Documents/volatility3$ mv pid*dmp maps/

# use the normal 'yara' tool against the extracted files, note the hit on `pid.8600.vma.0x7fe78a85f000-0x7fe78a86a000.dmp`
(volatility3) eve@xps:~/Documents/volatility3$ yara fullvma.yar maps/
fullvmayarascan maps//pid.8600.vma.0x7fe78a85f000-0x7fe78a86a000.dmp

# attempt the same scan with linux.vmayarascan.VmaYaraScan - no hits... :(
(volatility3) eve@xps:~/Documents/volatility3$ ./vol.py -f linux-sample-1.dmp linux.vmayarascan.VmaYaraScan --yara-file ./fullvma.yar 
Volatility 3 Framework 2.12.0
Progress:  100.00               Stacking attempts finished                 
Offset  PID     Rule    Component       Value

Here is the rule I used for testing:

rule fullvmayarascan
{
    strings:
        $s1 = "_nss_files_parse_grent"
        $s2 = "/lib64/ld-linux-x86-64.so.2"
        $s3 = "(bufferend - (char *) 0) % sizeof (char *) == 0"
    condition:
        all of them
}

[ikelos]

Ok, yep, seems it's still getting cut into 4096 byte blocks to be scanned. I think I'll need to make a BufferedScanner that can rechunk things. I've fixed up #1370 at least to work properly with both (previously it had a mistake of thinking the response was start, end rather than start, size as well as a number of issues with the yara_x return values in particular...

[eve-mem]

Glad you reminded me to even check...

Let me know if there's anything else i can help with on this one.

[gcmoreira]

Great job! It turns out there was more to address.

I've created a Linux test case for the latest bug using the steps provided by @eve-mem above. Feel free to incorporate it (or a similar version) into this PR. You may also want to include more detailed output validation.
We should also add the Windows counterpart.

diff --git a/test/test_volatility.py b/test/test_volatility.py
index 847be88d9..e94761499 100644
--- a/test/test_volatility.py
+++ b/test/test_volatility.py
@@ -14,6 +14,7 @@ import tempfile
 import hashlib
 import ntpath
 import json
+import contextlib
 
 #
 # HELPER FUNCTIONS
@@ -379,6 +381,42 @@ def test_linux_library_list(image, volatility, python):
     assert rc == 0
 
 
+def test_linux_vmayarascan_yara_rule(image, volatility, python):
+    yara_rule_01 = r"""
+        rule fullvmayarascan
+        {
+            strings:
+                $s1 = "_nss_files_parse_grent"
+                $s2 = "/lib64/ld-linux-x86-64.so.2"
+                $s3 = "(bufferend - (char *) 0) % sizeof (char *) == 0"
+            condition:
+                all of them
+        }
+    """
+
+    # FIXME: When the minimum Python version includes 3.12, replace the following with:
+    # with tempfile.NamedTemporaryFile(delete_on_close=False) as fd: ...
+    fd, filename = tempfile.mkstemp(suffix=".yar")
+    try:
+        with os.fdopen(fd, "w") as f:
+            f.write(yara_rule_01)
+
+        rc, out, _err = runvol_plugin(
+            "linux.vmayarascan.VmaYaraScan",
+            image,
+            volatility,
+            python,
+            pluginargs=["--pid", "8600", "--yara-file", filename],
+        )
+    finally:
+        with contextlib.suppress(FileNotFoundError):
+            os.remove(filename)
+
+    out = out.lower()
+    assert out.count(b"\n") > 4
+    assert rc == 0
+
+

Regarding the unit tests, I'm not sure if we should have a separate test for each plugin's functionality or combine everything into one comprehensive test case. I chose the former, but it's up to you. If we go with a combined test, we'll need to merge the above code with the test case in my other PR here.