Extracts web domains and IP address and implements tests #2031

aaronatp · 2024-03-19T18:29:40Z

This PR partially resolves #1907. It extracts web domains and IP addresses, and implements rendering functions and tests.

These changes likely don't require updates to the documentation, but if some users want to, they should be able to repurpose many of the extraction functions fairly easily.

In (-d) mode, this pull request extracts web domains and IP addresses from files and sandbox traces and presents them to the user. In (-v) and (-vv) modes, this pull request also tells the user how many times each web domain and IP address occur, and tries to identify a WinAPI networking function acting on a web domain and IP address for every time they occur. (-v) and (-vv) modes are currently the same.

This PR also implements tests for the part of the code that checks valid web domains, valid IP addresses, and potential WinAPI networking functions.

Example output:

Default (-d) output

Default (-d) output when there are no domains found

Verbose (-v) output

Very verbose (-vv) output

Checklist

No CHANGELOG update needed

No new tests needed

No documentation update needed

…d sandbox traces and presents them to the user. In (-v) and (-vv) modes, this pull request also tries to identify how web domains and IP addresses are used. It checks whether a WinAPI networking function is called soon after the web domain or IP address appears.

github-actions

Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the master (unreleased) section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: [x] No CHANGELOG update needed

CHANGELOG updated or no update needed, thanks! 😄

VascoSch92 · 2024-03-19T18:46:36Z

capa/capabilities/domain_ip_helpers.py

+            raise NotImplementedError("Confirm that argv is an attribute of doc.meta")
+
+    else:
+        print("in 'get_sigpaths_from_doc', run in debug (-d) mode")


this was for debugging? Why are you not using logger here?

Hi @VascoSch92 yes trying to figure out why it's failing to build with PyInstaller 3.11

ah ok easy... I just saw it and I wanted to check :-)

Haha yes thank you for the question though, I appreciate the comments and feedback you're giving me

capa/capabilities/extract_domain_and_ip.py

VascoSch92 · 2024-03-19T18:51:41Z

capa/capabilities/extract_domain_and_ip.py

+    """same as 'formatted_domain_verbose' but without 'ip_address_statement'"""
+    return (
+        f"{ip_addr}\n"
+        + f"    |---- {networking_functions_statement(doc, ip_addr)}"


Suggested change

+ f" |---- {networking_functions_statement(doc, ip_addr)}"

+ f" |---- {networking_functions_statement(doc, ip_addr)}\n"

or?

VascoSch92 · 2024-03-19T18:53:52Z

capa/capabilities/extract_domain_and_ip.py

+        return statement
+
+    else:
+        raise LengthError("'api_functions' contains unexpected data!")


you can never have this error or?

Because the three cases above are already covering all the output of the len function.

If api_functions doesn't implement the length functions, an error will occur on the first if or?

Thank you! Yes I think you are right, I have just taken that out!

capa/capabilities/extract_domain_and_ip.py

VascoSch92 · 2024-03-19T19:01:21Z

capa/render/verbose.py

+
+
+def ceil(num):
+    if isinstance(num, float):


what if num is integer? The return is None. Do you want this behaviour?

but there is a function in the math default package of python which do that not?

This is a good question - it shouldn't be an issue because the input to ceil will always be some multiple of 1/3, so the input's type should always be float, but you're right that it's not explicit - I will add a type hint here. I thought it might be a bit unnecessary to import the math package just for a simple function lol but what do you think?

Just updated, please let me know what you think!

Now the method is better :)

I have a problem with the fact that you implement the same method in two different scripts. If you have a bug in one method you have the same bug in the other, i.e, two bugs.

However, it is improbable that you have a bug in this method ;)

Using math.ceil is not a problem, math is a standard library in python.

But up to you ;)

Co-authored-by: Vasco Schiavo <[email protected]>

aaronatp · 2024-03-20T19:15:54Z

Hey @VascoSch92 looks like most of these tests are passing but GitHub is saying it's missing a code license agreement from you - do you think you could check about that? Would love to commit this code together :)

VascoSch92 · 2024-03-20T19:37:53Z

Hey @VascoSch92 looks like most of these tests are passing but GitHub is saying it's missing a code license agreement from you - do you think you could check about that? Would love to commit this code together :)

Done!

Let me know if there are still problems ;)

You should probably trigger the cla/google tests. I can not do that

mr-tz

nice work with many good parts, let's see how we can incorporate this as part of the main extraction phase and extend the result document with NBI/HBI information initially

mr-tz · 2024-03-22T09:49:16Z

capa/capabilities/extract_domain_and_ip.py

+                        yield feature, addr
+
+
+def default_extract_domain_names(doc: ResultDocument) -> Generator[str, None, None]:


to avoid reprocessing the file here (in generate_insns_from_doc etc.), let's discuss how to move this into the core extraction phase and how we can extend the ResultDocument with HBI/NBI data

@mr-tz Thank you for your feedback! Yes, this is something I thought about too. I'll look into restructuring this program to move the domain/IP extraction to the core extraction phase and then extend ResultDocument with HBI/NBI data

mr-tz · 2024-03-22T09:55:07Z

capa/capabilities/extract_domain_and_ip.py

+      (str): either the formatted IP address, or an error message
+    """
+    try:
+        ip_address = socket.gethostbyname(domain)


we don't want to do this and in most offline analysis systems it won't work anyway

mr-tz · 2024-03-22T09:57:37Z

capa/capabilities/extract_domain_and_ip.py

+        return f"Could not get IP address for {domain.split('/')[0]}\n"
+
+
+def networking_functions_statement(doc: ResultDocument, domain_or_ip: str):


maybe this can be simplified to called networking APIs in "close proximity" to the found string

@mr-tz I think this suggestion could be a good idea. However, when I previously tried simple ways of extracting networking APIs in "close proximity" (like skipping over "push" statements until we get to APIs), I actually got reduced performance. I'll have another look at this and see if I can find a way to make the "close proximity" searching work

mr-tz · 2024-03-22T09:58:58Z

capa/capabilities/extract_domain_and_ip.py

+                yield "Not able to identify the calling function"
+
+
+def potential_winapi_function(string: str) -> bool:


This is a finite list we could generate and embed instead.

@mr-tz thank you for this! Yes, I will do that

capa/render/verbose.py

mr-tz · 2024-03-22T10:00:38Z

tests/test_domain_ip_extractor.py

these tests look great

@mr-tz thank you!

Co-authored-by: Moritz <[email protected]>

github-actions bot previously requested changes Mar 19, 2024

View reviewed changes

aaronatp and others added 4 commits March 19, 2024 13:34

Update domain_ip_helpers.py

999f656

Update extract_domain_and_ip.py

80fc4d4

Update test_domain_ip_extractor.py

2bb09f8

Adds to changelog and fixes string concatenation style error

61c4ad5

Merge branch 'master' into master

627f187

VascoSch92 reviewed Mar 19, 2024

View reviewed changes

aaronatp and others added 14 commits March 19, 2024 14:07

Help debug why it won't build with PyInstaller 3.11

9b41074

Update capa/capabilities/extract_domain_and_ip.py

6236b8f

Co-authored-by: Vasco Schiavo <[email protected]>

Update capa/capabilities/extract_domain_and_ip.py

f9f6bb4

Co-authored-by: Vasco Schiavo <[email protected]>

Update verbose.py

aa5f542

Update vverbose.py

2e9c1a9

Update extract_domain_and_ip.py

11585c3

Fix PyInstaller 3.11 installation error with get_signatures

58b0336

Fix get_extractor_from_doc issue for CAPE

c55169b

Fix 'backend' assignment in get_extractor_from_doc

85dfa55

Fix 'format' in get_extractor_from_doc

5afd175

Merge branch 'master' into master

78d0efc

Reformat imports

681e6c3

Reformat multi-line string

fb3ed8a

Correct Flake8 errors

0443caf

aaronatp added 2 commits March 22, 2024 03:01

Update domain_ip_helpers.py

a000461

Update domain_ip_helpers.py

93943e5

mr-tz reviewed Mar 22, 2024

View reviewed changes

mr-tz mentioned this pull request Mar 22, 2024

Extracts web domain and IP address, implements rendering functions and tests #1944

Closed

3 tasks

Update capa/render/verbose.py

32b4778

Co-authored-by: Moritz <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extracts web domains and IP address and implements tests #2031

Extracts web domains and IP address and implements tests #2031

aaronatp commented Mar 19, 2024 •

edited

Loading

github-actions bot left a comment

VascoSch92 Mar 19, 2024

aaronatp Mar 19, 2024

VascoSch92 Mar 19, 2024

aaronatp Mar 19, 2024

VascoSch92 Mar 19, 2024

VascoSch92 Mar 19, 2024

aaronatp Mar 19, 2024

VascoSch92 Mar 19, 2024

aaronatp Mar 19, 2024

aaronatp Mar 19, 2024

VascoSch92 Mar 19, 2024

aaronatp commented Mar 20, 2024

VascoSch92 commented Mar 20, 2024 •

edited

Loading

mr-tz left a comment

mr-tz Mar 22, 2024

aaronatp Mar 22, 2024

mr-tz Mar 22, 2024

mr-tz Mar 22, 2024

aaronatp Mar 22, 2024

mr-tz Mar 22, 2024

aaronatp Mar 22, 2024

mr-tz Mar 22, 2024

aaronatp Mar 22, 2024

	+ f" \|---- {networking_functions_statement(doc, ip_addr)}"
	+ f" \|---- {networking_functions_statement(doc, ip_addr)}\n"

		yield feature, addr


		def default_extract_domain_names(doc: ResultDocument) -> Generator[str, None, None]:

		return f"Could not get IP address for {domain.split('/')[0]}\n"


		def networking_functions_statement(doc: ResultDocument, domain_or_ip: str):

		yield "Not able to identify the calling function"


		def potential_winapi_function(string: str) -> bool:

Extracts web domains and IP address and implements tests #2031

Are you sure you want to change the base?

Extracts web domains and IP address and implements tests #2031

Conversation

aaronatp commented Mar 19, 2024 • edited Loading

Checklist

github-actions bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronatp commented Mar 20, 2024

VascoSch92 commented Mar 20, 2024 • edited Loading

mr-tz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronatp commented Mar 19, 2024 •

edited

Loading

VascoSch92 commented Mar 20, 2024 •

edited

Loading