Skip to content

Rest API

Jump to bottom
Anna-Mariya Mateyna edited this page Mar 18, 2021 · 13 revisions

The FACT Rest API intends to offer close to 100 % functionality of FACT in a script-able and integrate-able interface. The API does not comply with all REST guidelines perfectly, but aims to allow understandable and efficient interfacing.

All data send as message body and all data received from the API is of application/json format. This allows easy integration into most programming languages, most commonly javascript and python. Since json does not support binary data, all binary data is additionally encoded in base64.

The API offers a number of URL parameters to go with GET requests. The parameters for each endpoint are listed via tables at the beginning of the endpoint descriptions below. They come with a usage example, but there will be a handful of usage examples listed after the introduction. Aside from the URL parameters the following descriptions provide information on each endpoint regarding:

  • What it is for
  • How to use it
  • What the response looks like

⚠️ The API is not versioned and thus subject to change. We try to issue timely deprecation warnings before a field or endpoint is to be removed.

Errors

All errors in processing a request to one of the listed endpoints result in an error message of the form:

{
    "error_message": <string>,               # Reason for the error
    "request": {                             # List of (supported) parameters in the request
        "limit": null,
        "offset": null
    },
    "request_resource": "/rest/firmware",    # Targeted endpoint
    "status": 1,                             # 1 signifies error, 0 for all successful requests
    "timestamp": 1587639454                  # Unix timestamp of request
}

Successful messages will basically look the same, aside from status being 0 and no error_message key being present. In place of the error message successes will typically have an additional field containing requested information or processing feedback.

General example

asciicast

Endpoints

  1. Firmware
  2. FileObject
  3. Compare
  4. Binary
  5. Binary Search
  6. Statistics
  7. System Status
  8. Missing Analyses
  9. Authentication

/rest/firmware[/<uid>]

Upload a firmware and query the firmware database.

PUT

⚠️ From version 2.5 there is a new field "device_part".
Post without further parameters. The HTTP body shall contain a json document of the following structure:

{
  "device_name": <string>,
  "device_part": <string>,           # new in FACT 2.5
  "device_class": <string>,
  "file_name": <string>,
  "version": <string>,               # supersedes firmware_version field
  "vendor": <string>,
  "release_date": <string>,
  "tags": <string>,
  "requested_analysis_systems": <list>,
  "binary": <string(base64)>
}

⚠️ Remember: The binary has to be a base64 string representing the raw binary you want to submit. The response is of the form:

{
    "uid": <string>,            # new unique identifier for uploaded firmware
    "status": 0,                # 0 for success, 1 for an error
    "request_resource": "/rest/firmware",
    "timestamp": <integer>,     # linux timestamp of request
    "request": <json_document>  # the input document
}

In case of an error, the uid will not be present, instead an error_message will be given. Also the status field will be 1.

Curl Sample:

curl http://localhost:5000/rest/firmware -X PUT -H "Content-Type: application/json" -d '{"vendor": "AVM", "device_class": "Router", "file_name": "rest_test.txt", "requested_analysis_systems": ["file_type", "file_hashes"], "binary": "dGVzdDEyMzQgdGhpcyBpcyBzb21lIHRlc3QgZmlsZQ==", "device_name": "rest_test", "firmware_version": "1", "release_date": "2011-01-01", "tags": "tag1,tag2"}'

Firmware analysis update

You can use this endpoint with PUT to update your firmware analysis. By using the following parameter, the listed analysis plugins are updated or added for all files in the database:

Parameter Effect Data format Example
update update firmware json list containing plug in names update=["cpu_architecture"]

Curl sample:

curl -X PUT 'http://localhost:5000/rest/firmware/1754689233cfa0d65e27718529c0a94a7b2072775c6099fefc00eb41e70f6f71_73584640' -G --data-urlencode 'update=["software_components", "cpu_architecture"]'

GET

Either request a single firmware by supplying the /[uid] parameter in the URL or browse the complete firmware database.

If a single firmware is requested, the result is of the form:

{
    "firmware": <json_document>,  # meta data and analysis results for the requested firmware
    "status": 0,                  # 0 for success, 1 for an error
    "request_resource": "/rest/firmware",
    "timestamp": <integer>,       # linux timestamp of request
    "request": {
        "uid": <string>    # the requested uid
    }
}

The default is to not aggregate summaries for the requested firmware. If you want to list summaries for the applied analysis plugins you have to add the summary parameter to the request.

Parameter Effect Data format Example
summary include summary in result boolean [true, false] summary=true

Curl Sample: (Single Firmware)

curl 'http://localhost:5000/rest/firmware/e692eca8505b0f4a3572d4d42940c6d5706b8aabec6ad1914bd4d733be9dfecf_25221120' -X GET

To list all firmware in the database, or query a subset, access /rest/firmware without a uid. To subset the database you can either page results using offset and limit or use a mongo syntax query for arbitrary subsets. A primer on the query syntax is embedded into the Web UI on the advanced search page.

Especially for analysis based subsets, matching based on included objects is more useful. To do so, apply the recursive flag. Using recursive you can get all firmware including at least one file matching the given query. This can further be combined with inverted. inverted and recursive combined result in a list of all firmware that does not include a single file matching the given query.

Here is the list of all applicable parameters:

Parameter Effect Data format Example
offset offset of results (paging) integer (0 to ignore) offset=10
limit number of results (paging) integer (0 to ignore) limit=5
query MongoDB style query (see advanced search) json document {"device_class": "Router"}
recursive Requires query. Query for parent firmware of matching objects boolean [true, false] recursive=true
inverted Requires query and recursive. Query for parent firmware that does not include the matching objects boolean [true, false] inverted=true

Responses to firmware database queries look like this:

{
    "uids": <list>,               # list of uids for firmwares that match the request
    "status": 0,                  # 0 for success, 1 for an error
    "request_resource": "/rest/firmware",
    "timestamp": <integer>,       # linux timestamp of request
    "request": {
        "offset": <integer>,      # 0 if not used  
        "limit": <integer>,       # 0 if not used
        "query": <json_document>, # empty if not used 
        "recursive": <boolean>,   # false if not used  
        "inverted": <boolean>     # false if not used  
    }
}

Curl Sample: (Multiple Firmware objects)

curl 'http://localhost:5000/rest/firmware?limit=3&offset=5' -X GET -G --data-urlencode 'query={"device_class": "Router"}'

/rest/file_object[/<uid>]

Browse the file database or request specific file.

GET

The /file_object endpoint works similar to the GET part of the /firmware endpoint. Instead of firmware this endpoint directly requests the included files. It allows to either request a specific file by providing the [uid] parameter in the URL or browse the database by applying query and/or limit and offset.

The only parameter applicable to the single file request is summary, which like for the firmware endpoint, includes summary lists for the applied plugins. This defaults to false.

Parameter Effect Data format Example
summary include summary in result boolean [true, false] summary=true

For possible responses see the /firmware endpoint. The only difference is that the firmware field in the single response is called file_object instead.

Curl Sample: (Single File)

curl 'http://localhost:5000/rest/file_object/6afa145520c15df916818997fd87202e8472141bb2ecf4869710cb026f15f9f8_28' -X GET

The usage of the browsing feature on files is also similar to /firmware. For explanation of the parameters see the documentation above. The supported parameters are:

Parameter Effect Data format Example
offset offset of results (paging) integer (0 to ignore) offset=10
limit number of results (paging) integer (0 to ignore) limit=5
query MongoDB style query (see advanced search) json document {"device_class": "Router"}

Curl Sample: (Multiple Files)

curl 'http://localhost:5000/rest/file_object?limit=100' -G --data-urlencode 'query={"size": 512}' -X GET

/rest/compare[/<compare_id>]

Issue compares and retrieve compare results.

PUT

Start the comparison of a given list of uids. The uid_list shall contain uids of already analysed FileObjects or Firmware objects. The body then contains a json document of the following structure:

{
    "uid_list":  <list>,
    "redo":  <Boolean>     # optional. if 'True' existing comparisons are overwritten
}

The response will contain a status field among others. If the status is 1 an error_message field is added:

{
    "message": <string>,          # optional. If status == 0, a notice is provided
    "status": 0,                  # 0 for success, 1 for an error
    "request_resource": "/rest/compare",
    "timestamp": <integer>,       # linux timestamp of request
    "request": <json_document>,   # see request format
    "error_message": <string>     # optional. Not present in for status == 0
}

GET

A compare result can be requested by GET request on the /rest/compare/<compare_id> endpoint, by providing a semicolon separated list of uids as compare_id (see example below). The response will contain a json_document with the compare result, along with the fields status, timestamp, request_resource and request as meta data.

Curl Sample: (PUT)

curl http://localhost:5000/rest/compare -X PUT -H "Content-Type: application/json" -d '{"uid_list": ["2d63c2bcc2a9e1618c79310fdd2b77a83600a499f77c809a535e045aa499b169_42659840", ...], "redo": false}'

Curl Sample: (GET)

curl 'http://localhost:5000/rest/compare/6a4cba6f73a25f179e4d42460e9383ba499f93e12a9d6d81a7bdf0ea9d936a13_566112;f432dcff65513707659d2d6f3dbd43eefcc580b8c1a48cd3c1ef17a51d71cbb3_564500' -X GET

/rest/binary/<uid>

Request the binary of a given Firmware or FileObject.

GET

Request a binary by providing the uid of the corresponding object in the URL. You can alternatively use the tar parameter on the request to get the target archive as its content repacked into a .tar.gz.

Parameter Effect Data format Example
tar Get tar.gz packed contents of target boolean [true, false] tar=true

The response contains the binary or .tar.gz in base64 encoding as well as the SHA-256 sum of the binary as checksum and the original name of the file:

{  
    "SHA256": <string>,     # SHA-256 checksum of raw binary (not of base64 encoding)
    "binary": <string>,     # base64 representation of the binary
    "file_name": <string>,  # original name of the downloaded file
    "request":{  
        "uid": <string>     # the requested uid
    },
    "request_resource": "/rest/binary",
    "status": 0,            # 0 for success, 1 for an error
    "timestamp": <integer>  # linux timestamp of request
}

Curl Sample:

curl http://localhost:5000/rest/binary/ceb0b13da1e765ec105460cf68367b78ed78b99fcbdd7654999a07b5b87e8f16_31 -X GET

/rest/binary_search[/<search_id>]

Start a binary search on the binary database (or optionally on a single firmware) and fetch the results.

POST

Start a binary search on the binary database by providing a set of YARA rules in the request data. The request data should have the form:

{
  "rule_file": <string>,  # a string containing the YARA rules
  "uid": <string>         # optional: the firmware UID
}

The UID is optional and only needs to be included if the search should be executed on the files of a single firmware.

The response data contains a search_id which is used to fetch the search results. The POST only initiates the search. In order to get the search results, please use the GET interface.

Curl Sample

curl 127.0.0.1:5000/rest/binary_search -X POST -d '{"rule_file": "rule rulename {strings: $a = \"OpenSSL\" condition: $a }", "uid": "715947e9e69aefcde3c09fe0b769a7210b0646b542bd8d958e3902eacb661ca7_2551176"}' -H "Content-Type: application/json"

GET

Get the results of a previously initiated binary search. The search_id (provided by the POST request) is needed to fetch the corresponding search result. The result of the search request can only be fetched once. After this, the search needs to be started again.

The results have the form:

{
    'binary_search_results': {
        '<rule_name_1>': ['<matching_uid_1>', ...],
        '<rule_name_2>': [...], 
        ...
    },
    ...
}

Curl Sample

curl -X GET 127.0.0.1:5000/rest/binary_search/77479d57ad4c24cc471413147e1eb456915cde22ae07a3c2761d95d1eb5d9545_1533566955.8252065

/rest/statistics[/<stat_name>]

Query some or all the statistics of FACT.

GET

Retrieves the statistics from the database as raw JSON data.
To list all the available statistics in the database access /rest/statistics.
The optional parameter stats_name can be used to only retrieve a specifically chosen statistic.

The response is of the form::

{
   "architecture":{
      "cpu_architecture":[[<stat_data>], ...]
   },
   "crypto_material":{
      "crypto_material":[[<stat_data>], ...]
   },
   "elf_executable":{
      "executable_stats":[[<stat_data>], ...]
   },
   "file_type":{
      "file_types":[[<stat_data>], ...],
      "firmware_container":[[<stat_data>], ...]
   },
   "firmware_meta":{
      "device_class":[[<stat_data>], ...],
      "vendor":[[<stat_data>], ...]
   },
   "general":{
      "average_file_size":"<float>",
      "average_firmware_size":"<float>",
      "benchmark":"<float>",
      "creation_time":"<float>",
      "number_of_firmwares":"<int>",
      "number_of_unique_files":"<int>",
      "total_file_size":"<int>",
      "total_firmware_size":"<int>"
   },
   "ips_and_uris":{
      "ips_v4":[[<stat_data>], ...],
      "ips_v6":[[<stat_data>], ...],
      "uris":[[<stat_data>], ...]
   },
   "known_vulnerabilities":{
      "known_vulnerabilities":[[<stat_data>], ...]
   },
   "malware":{
      "malware":[[<stat_data>], ...]
   },
   "release_date":{
      "date_histogram_data":[[<stat_data>], ...]
   },
   "software_components":{
      "software_components":[[<stat_data>], ...]
   },
   "unpacking":{
      "average_packed_entropy":"<float>",
      "average_unpacked_entropy":"<float>",
      "data_loss_file_types":[[<stat_data>], ...],
      "overall_data_loss_ratio":"<float>",
      "overall_unpack_ratio":"<float>",
      "packed_file_types":[[<stat_data>], ...],
      "used_unpackers":[[<stat_data>], ...]
   }
}

#### Curl Sample
```bash
curl -X GET 127.0.0.1:5000/rest/statistics

/rest/status

Request the system status of FACT.

GET

Request a json document showing the system state of FACT, similar to the system health page of the GUI.

The results have the form:

{
    "status": 0,
    "timestamp": 1585810371
    "request_resource": "/rest/status",
    "plugins": {
        "binwalk": {
            "description": "binwalk signature and entropy analysis",
            "version": "0.5.2"
        },
        [...]
    },
    "system_status": {
        "backend": {
            "_id": "backend",
            "last_update": 1585810370.731242,
            "name": "backend",
            "platform": {
                "fact_version": "3.1-dev",
                "os": "Ubuntu 18.04",
                "python": "3.6.9"
            },
            "status": "online",
            "system": {
                "cpu_cores": 8,
                "disk_percent": 29.8,
                "load_average": "0.4, 0.45, 0.33",
                "memory_percent": 18.0,
                [...]
            },
            "unpacking": {
                "unpacking_queue": 0
            }
            "analysis": {
                "analysis_main_scheduler": 0,
                "plugins": {
                    "binwalk": {
                        "active": 0,
                        "queue": 0
                    },
                    [...]
                }
            },
        },
        "database": {
            "_id": "database",
            [...]
        },
        "frontend": {
            "_id": "frontend",
            [...]
        }
    },
}

Curl Sample:

curl http://localhost:5000/rest/status -X GET

/rest/missing

Search the database for missing entries.

GET

Search for:

  • Missing files: Files whose UID is contained in the files_included of some object but that are not found in the database
  • Missing analyses: File objects that are lacking analyses which were performed on their parent firmware

The results have the form:

{
    "missing_files": {
        "<Parent FW/FO UID>": ["<missing file 1 UID>", ...],
        ...
    },
    "missing_analyses": {
        "<Root FW UID>": ["<file 1 missing analysis UID>", ...],
        ...
    },
    "request_resource": "/rest/missing",
    "status": 0,            # 0 for success, 1 for an error
    "timestamp": <integer>  # linux timestamp of request
}

Curl Sample:

curl http://localhost:5000/rest/missing -X GET

Authentication

If the authentication option is enabled in the configuration, each REST request has to be accompanied by an API key in the HTTP header. For more information regarding the authentication in general and which function needs what level of privilege see FACT authentication.

If your user has been given the exemplary key ABCDEFG= a request to the /rest/binary endpoint in accordance to the example given above would look like this:

curl http://localhost:5000/rest/binary/ceb0b13da1e765ec105460cf68367b78ed78b99fcbdd7654999a07b5b87e8f16_31 -X GET --header 'Authorization: ABCDEFG=' -L
FACT Logo

FACT Wiki

Home

Setup

Development

Usage

Clone this wiki locally