-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
grep_more_ioc improvements #222
Conversation
…erns for keys in fix_json. Fixes pcdshub#213
…eird float casting
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The regex monster grows! I know I've expressed my distaste for regex in the past, so I won't belabor it much here. It is becoming increasingly difficult to read and understand. Maybe the parsing functions here could get some tests some day soon.
@@ -265,7 +298,7 @@ def find_ioc(hutch: str = None, patt: str = None, | |||
# strip the file information | |||
_temp = re.sub(r'.*cfg\:', '', _temp) | |||
# now convert back to json and load | |||
output = [json.loads(s) for s in fix_json(_temp)] | |||
output = [try_json_loads(s) for s in fix_json(_temp)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As it stands, this could get a bunch of None
values. Are we ok with that getting to the final output? Or do we want to filter that out?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't mind getting a bunch of None values. The dataframe print/manipulation handles that for the most part. Sometimes the negative space is important for how I am perusing the IOC info
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe also importantly, this may print the json error text a bunch of times in a row!
But that could be a feature too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I consider that a feature — in fact it's exactly what I needed to figure out why certain IOCs in RIX were breaking this tool in the first place and realized the problem with my JSONification.
Bethesda would be proud
Co-authored-by: Robert Tang-Kong <[email protected]>
One day I'll burn for my sins — but not today! Tried to at least make the regex consistent — find a procmgr key then json-ify it. Then look for unquoted chars near a procmgr key and quote it too. Maybe I've just gotten Stockholm's syndrome with regex and Perlre by now 🤷 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems reasonable to me
I also fear the regex monster
Co-authored-by: Zachary Lentz <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this looks good otherwise. I yearn for tests in this repo as a whole but that's not on you
Description
Fixed some bugs in
grep_more_ioc
and added some QOL on the search sub-parsers. I'm probably the only person using this tool, but it makes life livable inlas
given the sheer number of IOCs.Motivation and Context
Fixes #213 and #192 and improves functionality for piping output into other commands/scripts.
How Has This Been Tested?
I have ran
../engineering_tools/grep_more_ioc
to use the changes in my fork and can successfully print all IOCs in all hutches, demonstrating #213 has fixed the issue I was seeing before whenAlias
procmngr keys included forbidden characters and digits that weren't being escaped correctly before reading the JSON.I have also demonstrated that the new search subparser arguments work as expected:
Where Has This Been Documented?
This PR.
It is important to add that I have redefined what the
-o
optional flag forsearch
does:Previously,
-o
was--only_search
, meaning skip printing thedataframe
for when you only care about the child ioc search results.We now have the following options:
So if you have any scripts built on the output of
grep_more_ioc <patt> <hutch> search -o <regex>
you need to update it togrep_more_ioc <patt> <hutch> search -s <regex>