Extra Data bug in ACE alert data.json

[seanmcfeely]

We've seen the following bug three times now. I can supply the data.json file in a secure channel if someone wants to take this on.

Command line:

$ ./ace import-alerts ~cybersecurity/6d9de41f-949d-40ce-a77b-7a607aaae0be
+ unable to load json from /home/cybersecurity/6d9de41f-949d-40ce-a77b-7a607aaae0be/data.json: Extra data: line 1 column 832256 (char 832255)
Traceback (most recent call last):
  File "./ace", line 3796, in <module>
    args.func(args)
  File "./ace", line 1740, in import_alerts
    if not alert.load():
  File "/opt/ace/lib/saq/analysis/__init__.py", line 2916, in load
    raise e
  File "/opt/ace/lib/saq/analysis/__init__.py", line 2901, in load
    self.json = json.load(fp)
  File "/usr/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 342, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 832256 (char 832255)

GUI when trying to view alert:

I think the bug may be along these lines: https://stackoverflow.com/questions/48140858/json-decoder-jsondecodeerror-extra-data-line-2-column-1-char-190

[unixfreak0037]

Send me that whole ~cybersecurity/6d9de41f-949d-40ce-a77b-7a607aaae0be directory in slack chat when you get a chance.

[unixfreak0037]

This is a long outstanding bug. I still haven't figured it out, but I'm pretty sure it has something to do with how delayed analysis works. Some hacks were introduced to bypass it but the root cause has not yet been identified and resolved yet.

What is happening is the JSON is getting over-written with a shorter JSON without first truncating the file, so there is extra JSON at the end which invalidates the file.