Merge pull request #15 from coffeegist/feature/havoc-parsing

Feature/havoc parsing

CHANGELOG.md

@@ -1,4 +1,12 @@
 # Changelog
+## [0.4.3] - 10/30/2024
+### Added
+- Support for pasing ldapsearch BOF results within Havoc log files
+
+### Changed
+- Parsers now can inherit from the `LdapSearchBofParser` (since support for other C2s usually still relies on the same BOF) to cut down on code copypasta
+- The `GenericParser` class (used to parse local group memberships, session data) is now called from main parsers (`LdapSearchBofParser`, `HavocParser`, etc.) to prevent each logfile from being opened, read, formatted, and parsed twice (each file is now read once and just parsed twice, once for LDAP objects and once for local objects)
+
 ## [0.4.2] - 10/24/2024
 ### Fixed
 - Addressed [#12](https://github.com/coffeegist/bofhound/issues/12), an issue with duplicate trusted domain objects

README.md

@@ -15,51 +15,88 @@
 ![PyPi](https://img.shields.io/pypi/v/bofhound?style=for-the-badge)
 </h1>
 
-BOFHound is an offline BloodHound ingestor and LDAP result parser compatible with TrustedSec's [ldapsearch BOF](https://github.com/trustedsec/CS-Situational-Awareness-BOF), the Python adaptation, [pyldapsearch](https://github.com/fortalice/pyldapsearch) and Brute Ratel's [LDAP Sentinel](https://bruteratel.com/tabs/commander/badgers/#ldapsentinel).
+BOFHound is an offline BloodHound ingestor and LDAP result parser compatible with TrustedSec's [ldapsearch BOF](https://github.com/trustedsec/CS-Situational-Awareness-BOF), the Python adaptation, [pyldapsearch](https://github.com/fortalice/pyldapsearch) and Brute Ratel's [LDAP Sentinel](https://bruteratel.com/tabs/commander/badgers/#ldapsentinel). ldapsearch BOF logs can also be parsed from [Havoc](https://github.com/HavocFramework/Havoc) logs.
 
 By parsing log files generated by the aforementioned tools, BOFHound allows operators to utilize BloodHound's beloved interface while maintaining full control over the LDAP queries being run and the spped at which they are executed. This leaves room for operator discretion to account for potential honeypot accounts, expensive LDAP query thresholds and other detection mechanisms designed with the traditional, automated BloodHound collectors in mind.
 
 Check this [PR](https://github.com/trustedsec/CS-Situational-Awareness-BOF/pull/114) to the SA BOF repo for BOFs that collect session and local group membership data and can be parsed by BOFHound.
 
-### Blog Posts
+### References
 
-| Title                                                                                                                                                            | Date         |
-|------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
+Blog Posts:
+
+| Title| Date|
+|------|-----|
+| [*BOFHound: AD CS Integration*](https://medium.com/specter-ops-posts/bofhound-ad-cs-integration-91b706bc7958) | Oct 30, 2024 |
 | [*BOFHound: Session Integration*](https://posts.specterops.io/bofhound-session-integration-7b88b6f18423) | Jan 30, 2024 |
 | [*Granularize Your AD Recon Game Part 2*](https://www.fortalicesolutions.com/posts/granularize-your-active-directory-reconnaissance-game-part-2) | Jun 15, 2022 |
 | [*Granularize Your AD Recon Game*](https://www.fortalicesolutions.com/posts/bofhound-granularize-your-active-directory-reconnaissance-game) | May 10, 2022  |
 
+Presentations:
+
+| Conference| Materials| Date|
+|-----------|----------|-----|
+| *SO-CON 2024*| [Slides](https://github.com/SpecterOps/presentations/blob/main/SO-CON%202024/Matt%20Creel%20%26%20Adam%20Brown%20-%20Manually%20Enumerating%20AD%20Attack%20Paths%20with%20BOFHound/Matt%20Creel%20and%20Adam%20Brown%20-%20Manually%20Enumerating%20AD%20Attack%20Paths%20With%20BOFHound%20-%20SO-CON%202024.pdf) & [Recording](https://www.youtube.com/watch?v=Xxm4YktSKVY)| Mar 11, 2024|
+
 # Installation
 BOFHound can be installed with `pip3 install bofhound` or by cloning this repository and running `pip3 install .`
 
 # Usage
-![](.assets/usage.png)
-
+```
+ Usage: bofhound [OPTIONS]                                                                                  
+                                                                                                            
+ Generate BloodHound compatible JSON from logs written by ldapsearch BOF, pyldapsearch and Brute Ratel's    
+ LDAP Sentinel                                                                                              
+                                                                                                            
+╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --input             -i      TEXT                     Directory or file containing logs of ldapsearch     │
+│                                                      results                                             │
+│                                                      [default: /opt/cobaltstrike/logs]                   │
+│ --output            -o      TEXT                     Location to export bloodhound files [default: .]    │
+│ --properties-level  -p      [Standard|Member|All]    Change the verbosity of properties exported to      │
+│                                                      JSON: Standard - Common BH properties | Member -    │
+│                                                      Includes MemberOf and Member | All - Includes all   │
+│                                                      properties                                          │
+│                                                      [default: Member]                                   │
+│ --parser                    [ldapsearch|BRC4|Havoc]  Parser to use for log files. ldapsearch parser      │
+│                                                      (default) supports ldapsearch BOF logs from Cobalt  │
+│                                                      Strike and pyldapsearch logs                        │
+│                                                      [default: ldapsearch]                               │
+│ --debug                                              Enable debug output                                 │
+│ --zip               -z                               Compress the JSON output files into a zip archive   │
+│ --help                                               Show this message and exit.                         │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
 
 ## Example Usage
 Parse ldapseach BOF results from Cobalt Strike logs (`/opt/cobaltstrike/logs` by default) to /data/
 ```
 bofhound -o /data/
 ```
 
-Parse pyldapsearch logs and only include all properties (vs only common properties)
+Parse pyldapsearch logs and only include all properties (vs other property levels)
 ```
-bofhound -i ~/.pyldapsearch/logs/ --all-properties
+bofhound -i ~/.pyldapsearch/logs/ --properties-level all
 ```
 
 Parse LDAP Sentinel data from BRc4 logs (will change default input path to `/opt/bruteratel/logs`)
 ```
-bofhound --brute-ratel
+bofhound --parser brc4
+```
+
+Parse Havoc loot logs (will change default input path to `/opt/havoc/data/loot`) and zip the resulting JSON files
+```
+bofhound --parser havoc --zip
 ```
 
 # ldapsearch
+Specify `*,ntsecuritydescriptor` as the attributes to return to be able to parse ACL edges. You are missing a ton of data if you don't include this in your `ldapsearch` queries!
 
-## Required Data
+#### Required Data
 The following attributes are required for proper functionality:
-
 ```
 samaccounttype
-dn
+distinguishedname
 objectsid
 ```
 
@@ -70,28 +107,50 @@ ldapsearch (distinguishedname=DC=windomain,DC=local) *,ntsecuritydescriptor
 ``` 
 
 ## Example ldapsearch Queries
-Get All the Data (Maybe Run BloodHound Instead?)
 ```
+# Get All the Data (Maybe Run BloodHound Instead?)
 ldapsearch (objectclass=*) *,ntsecuritydescriptor
-```
 
-Retrieve All Schema Info
-```
-ldapsearch (schemaIDGUID=*) name,schemaidguid -1 "" CN=Schema,CN=Configuration,DC=windomain,DC=local
-```
+# Retrieve All Schema Info
+ldapsearch (schemaIDGUID=*) name,schemaidguid 0 3 "" CN=Schema,CN=Configuration,DC=windomain,DC=local
 
-Retrieve Only the ms-Mcs-AdmPwd schemaIDGUID
-```
-ldapsearch (name=ms-mcs-admpwd) name,schemaidguid 1 "" CN=Schema,CN=Configuration,DC=windomain,DC=local
-```
+# Retrieve Only the ms-Mcs-AdmPwd schemaIDGUID
+ldapsearch (name=ms-mcs-admpwd) name,schemaidguid 1 3 "" CN=Schema,CN=Configuration,DC=windomain,DC=local
 
-Retrieve Domain NetBIOS Names (useful if collecting data via `netsession2/netloggedon2` BOFs)
-```
-ldapsearch (netbiosname=*) * 0 "" "CN=Partitions,CN=Configuration,DC=windomain,DC=local"
+# Retrieve Domain NetBIOS Names (useful if collecting data via `netsession2/netloggedon2` BOFs)
+ldapsearch (netbiosname=*) * 0 3 "" "CN=Partitions,CN=Configuration,DC=windomain,DC=local"
+
+# Unroll a group's nested members
+ldapsearch (memberOf:1.2.840.113556.1.4.1941:=CN=TargetGroup,CN=Users,DC=windomain,DC=local) *,ntsecuritydescriptor
+
+# Query domain trusts
+ldapsearch (objectclass=trusteddomain) *,ntsecuritydescriptor
+
+# Query across a trust
+ldapsearch (objectclass=domain) *,ntsecuritydescriptor 0 3 dc1.trusted.windomain.local "DC=TRUSTED,DC=WINDOMAIN,DC=LOCAL"
+
+#####
+# Queries below populate objects for AD CS parsing
+
+# Query the domain object
+ldapsearch (objectclass=domain) *,ntsecuritydescriptor
+
+# Query Enterprise CAs
+ldapsearch (objectclass=pKIEnrollmentService) *,ntsecuritydescriptor 0 3 “” “CN=Configuration,DC=domain,DC=local”
+
+# Query AIACAs, Root CAs and NTAuth Stores
+ldapsearch (objectclass=certificationAuthority) *,ntsecuritydescriptor 0 3 “” “CN=Configuration,DC=domain,DC=local”
+
+# Query Certificate Templates
+ldapsearch (objectclass=pKICertificateTemplate) *,ntsecuritydescriptor 0 3 “” “CN=Configuration,DC=domain,DC=local”
+
+# Query Issuance Policies
+ldapsearch (objectclass=msPKI-Enterprise-Oid) *,ntsecuritydescriptor 0 3 “” “CN=Configuration,DC=domain,DC=local”
 ```
 
 # Versions
 Check the tagged releases to download a specific version
+- v0.4.0 and onward support parsing AD CS objects and edges
 - v0.3.0 and onward support session/local group data 
 - v0.2.1 and onward are compatible with BloodHound CE
 - v0.2.0 is the last release supporting BloodHound Legacy
@@ -109,4 +168,4 @@ poetry run bofhound --help
 # References and Credits
 - [@_dirkjan](https://twitter.com/_dirkjan) (and other contributors) for [BloodHound.py](https://github.com/fox-it/BloodHound.py)
 - TrustedSec for [CS-Situational-Awareness-BOF](https://github.com/trustedsec/CS-Situational-Awareness-BOF)
-- [P-aLu](https://github.com/P-aLu) for collaboration on bofhoud's [ADCS support](https://github.com/coffeegist/bofhound/pull/8)
+- [P-aLu](https://github.com/P-aLu) for collaboration on bofhoud's [AD CS support](https://github.com/coffeegist/bofhound/pull/8)

bofhound/__main__.py

@@ -9,7 +9,7 @@
 import logging
 import typer
 import glob
-from bofhound.parsers import LdapSearchBofParser, Brc4LdapSentinelParser, GenericParser
+from bofhound.parsers import LdapSearchBofParser, Brc4LdapSentinelParser, HavocParser, ParserType
 from bofhound.writer import BloodHoundWriter
 from bofhound.ad import ADDS
 from bofhound.local import LocalBroker
@@ -23,10 +23,10 @@
 
 @app.command()
 def main(
-    input_files: str = typer.Option("/opt/cobaltstrike/logs", "--input", "-i", help="Directory or file containing logs of ldapsearch results. Will default to [green]/opt/bruteratel/logs[/] if --brute-ratel is specified"),
+    input_files: str = typer.Option("/opt/cobaltstrike/logs", "--input", "-i", help="Directory or file containing logs of ldapsearch results"),
     output_folder: str = typer.Option(".", "--output", "-o", help="Location to export bloodhound files"),
     properties_level: PropertiesLevel = typer.Option(PropertiesLevel.Member.value, "--properties-level", "-p", case_sensitive=False, help='Change the verbosity of properties exported to JSON: Standard - Common BH properties | Member - Includes MemberOf and Member | All - Includes all properties'),
-    brute_ratel: bool = typer.Option(False, "--brute-ratel", help="Parse logs from Brute Ratel's LDAP Sentinel"),
+    parser: ParserType = typer.Option(ParserType.LdapsearchBof.value, "--parser", case_sensitive=False, help="Parser to use for log files. ldapsearch parser (default) supports ldapsearch BOF logs from Cobalt Strike and pyldapsearch logs"),
     debug: bool = typer.Option(False, "--debug", help="Enable debug output"),
     zip_files: bool = typer.Option(False, "--zip", "-z", help="Compress the JSON output files into a zip archive")):
     """
@@ -40,15 +40,32 @@ def main(
 
     banner()
 
-    # if BRc4 and input_files is the default, set it to the default BRc4 logs directory
-    if brute_ratel and input_files == "/opt/cobaltstrike/logs":
-        input_files = "/opt/bruteratel/logs"
-
-    # default to Cobalt logfile naming format
+     # default to Cobalt logfile naming format
     logfile_name_format = "beacon*.log"
-    if brute_ratel:
-        logfile_name_format = "b-*.log"
 
+    match parser:
+
+        case ParserType.LdapsearchBof:
+            logging.debug('Using ldapsearch parser')
+            parser = LdapSearchBofParser
+
+        case ParserType.BRC4:
+            logging.debug('Using Brute Ratel parser')
+            parser = Brc4LdapSentinelParser
+            logfile_name_format = "b-*.log"
+            if input_files == "/opt/cobaltstrike/logs":
+                input_files = "/opt/bruteratel/logs"
+
+        case ParserType.HAVOC:
+            logging.debug('Using Havoc parser')
+            parser = HavocParser
+            logfile_name_format = "Console_*.log"
+            if input_files == "/opt/cobaltstrike/logs":
+                input_files = "/opt/havoc/data/loot"
+
+        case _:
+            raise ValueError(f"Unknown parser type: {parser}")
+
     if os.path.isfile(input_files):
         cs_logs = [input_files]
         logging.debug(f"Log file explicitly provided {input_files}")
@@ -70,18 +87,14 @@ def main(
         logging.error(f"Could not find {input_files} on disk")
         sys.exit(-1)
 
-    parser = LdapSearchBofParser
-    if brute_ratel:
-        logging.debug('Using Brute Ratel parser')
-        parser = Brc4LdapSentinelParser
-
     parsed_ldap_objects = []
     parsed_local_objects = []
     with console.status(f"", spinner="aesthetic") as status:
         for log in cs_logs:
             status.update(f" [bold] Parsing {log}")
-            new_objects = parser.parse_file(log)
-            new_local_objects = GenericParser.parse_file(log)
+            formatted_data = parser.prep_file(log)
+            new_objects = parser.parse_data(formatted_data)
+            new_local_objects = parser.parse_local_objects(formatted_data)
             logging.debug(f"Parsed {log}")
             logging.debug(f"Found {len(new_objects)} objects in {log}")
             parsed_ldap_objects.extend(new_objects)

bofhound/parsers/__init__.py

@@ -1,3 +1,4 @@
 from .ldap_search_bof import LdapSearchBofParser
 from .brc4_ldap_sentinel import Brc4LdapSentinelParser
-from .generic_parser import GenericParser
+from .havoc import HavocParser
+from .parsertype import ParserType

bofhound/parsers/brc4_ldap_sentinel.py

@@ -2,10 +2,9 @@
 import codecs
 import logging
 from datetime import datetime as dt
+from bofhound.parsers import LdapSearchBofParser
 
-# If field is empty, DO NOT WRITE IT TO FILE!
-
-class Brc4LdapSentinelParser():
+class Brc4LdapSentinelParser(LdapSearchBofParser):
     # BRC4 LDAP Sentinel currently only queries attributes=["*"] and objectClass 
     # is always the top result. May need to be updated in the future.
     START_BOUNDARY = '[+] objectclass                        :'
@@ -19,12 +18,25 @@ class Brc4LdapSentinelParser():
     def __init__(self):
         pass #self.objects = []
 
+    #
+    # Legacy, used by test cases for 1 liner
+    #   Removed from __main__.py to avoid duplicating file reads and formatting
+    #
     @staticmethod
     def parse_file(file):
-
         with codecs.open(file, 'r', 'utf-8', errors='ignore') as f:
             return Brc4LdapSentinelParser.parse_data(f.read())
 
+
+    #
+    # Replaces parse_file() usage in __main__.py to avoid duplicate file reads
+    #
+    @staticmethod
+    def prep_file(file):
+        with codecs.open(file, 'r', 'utf-8', errors='ignore') as f:
+            return f.read()
+
+
     @staticmethod
     def parse_data(contents):
         parsed_objects = []

bofhound/parsers/havoc.py

@@ -0,0 +1,13 @@
+import re
+import codecs
+from bofhound.parsers import LdapSearchBofParser
+
+
+class HavocParser(LdapSearchBofParser):
+
+    @staticmethod
+    def prep_file(file):
+        with codecs.open(file, 'r', 'utf-8', errors='ignore') as f:
+            contents = f.read()
+
+        return re.sub(r'\[\d{2}\/\d{2}\/\d{4} \d{2}:\d{2}:\d{2}\] \[\+\] Received Output \[\d+ bytes\]:\n', '', contents)