Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grep_more_ioc improvements #222

Merged
merged 15 commits into from
Nov 20, 2024
Merged

Conversation

aberges-SLAC
Copy link
Contributor

Description

Fixed some bugs in grep_more_ioc and added some QOL on the search sub-parsers. I'm probably the only person using this tool, but it makes life livable in las given the sheer number of IOCs.

Motivation and Context

Fixes #213 and #192 and improves functionality for piping output into other commands/scripts.

How Has This Been Tested?

I have ran ../engineering_tools/grep_more_ioc to use the changes in my fork and can successfully print all IOCs in all hutches, demonstrating #213 has fixed the issue I was seeing before when Alias procmngr keys included forbidden characters and digits that weren't being escaped correctly before reading the JSON.

I have also demonstrated that the new search subparser arguments work as expected:
image

Where Has This Been Documented?

This PR.
It is important to add that I have redefined what the -o optional flag for search does:

Previously, -o was --only_search, meaning skip printing the dataframe for when you only care about the child ioc search results.

We now have the following options:

    search.add_argument('-s', '--only_search', action='store_true',
                        default=False,
                        help="Don't print the dataframe, just search results.")
    search.add_argument('-o', '--only_results', action='store_true',
                        default=False,
                        help="Only print the results of the regex match. Like"
                        " 'grep -o'.")
    search.add_argument('-n', '--no_color', action='store_true',
                        default=False,
                        help="Don't wrap the search results with a color")

So if you have any scripts built on the output of grep_more_ioc <patt> <hutch> search -o <regex> you need to update it to grep_more_ioc <patt> <hutch> search -s <regex>

Copy link
Contributor

@tangkong tangkong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex monster grows! I know I've expressed my distaste for regex in the past, so I won't belabor it much here. It is becoming increasingly difficult to read and understand. Maybe the parsing functions here could get some tests some day soon.

scripts/grep_more_ioc.py Outdated Show resolved Hide resolved
@@ -265,7 +298,7 @@ def find_ioc(hutch: str = None, patt: str = None,
# strip the file information
_temp = re.sub(r'.*cfg\:', '', _temp)
# now convert back to json and load
output = [json.loads(s) for s in fix_json(_temp)]
output = [try_json_loads(s) for s in fix_json(_temp)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it stands, this could get a bunch of None values. Are we ok with that getting to the final output? Or do we want to filter that out?

Copy link
Contributor Author

@aberges-SLAC aberges-SLAC Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind getting a bunch of None values. The dataframe print/manipulation handles that for the most part. Sometimes the negative space is important for how I am perusing the IOC info

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also importantly, this may print the json error text a bunch of times in a row!
But that could be a feature too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I consider that a feature — in fact it's exactly what I needed to figure out why certain IOCs in RIX were breaking this tool in the first place and realized the problem with my JSONification.

Bethesda would be proud

@aberges-SLAC
Copy link
Contributor Author

The regex monster grows! I know I've expressed my distaste for regex in the past, so I won't belabor it much here. It is becoming increasingly difficult to read and understand. Maybe the parsing functions here could get some tests some day soon.

One day I'll burn for my sins — but not today!

Tried to at least make the regex consistent — find a procmgr key then json-ify it. Then look for unquoted chars near a procmgr key and quote it too.

Maybe I've just gotten Stockholm's syndrome with regex and Perlre by now 🤷

scripts/grep_more_ioc.py Outdated Show resolved Hide resolved
ZLLentz
ZLLentz previously approved these changes Nov 19, 2024
Copy link
Member

@ZLLentz ZLLentz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me
I also fear the regex monster

Copy link
Contributor

@tangkong tangkong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good otherwise. I yearn for tests in this repo as a whole but that's not on you

@aberges-SLAC aberges-SLAC merged commit 0604133 into pcdshub:master Nov 20, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

MNT: Add debug function for json.loads() in find_ioc in grep_more_ioc or fix_json()
3 participants