Conditional Alarm Framework

[slominskir]

We need some way to address alarms that must be wrapped with extra condition logic because they have well known false alarm mitigation logic based on the state of various EPICS PVs.

The most straightforward is just to remove those alarms from the standard set of EPICS alarms and replace them with Python scripts that monitor (wrap around) the original EPICS alarms and then monitor extra conditions as well and only produce JAWS alarms if the special criteria is met. Intercepting all existing alarm notifications and then applying conditions is also possible; so it's a choice between replace producers or introduce global interceptors. The existing JAWS overrides are examples of global interceptors. Filtering via the web Choose options is an example of local interception, and I guess that's a third option, though not a good one as these conditions aren't dynamic based on client/user.

Most of the known example alarms that would benefit from extra conditions apply to a specific class / subset of alarms and have static / fixed extra conditions such as don't alarm if some PV = 1. That condition doesn't change and the set of alarms it applies to doesn't change (and is a small subset of all alarms). It is somewhat inefficient / has a high cost to evaluate every dynamic rule on every alarm notification as it slows down the single alarm stream processor, which is the bottleneck of the system. So I'm leaning towards the producer approach.

This is very much related to: #8.

Known Alarms that would benefit from extra conditional logic:

1. Alarms with Action name like "%Mismatch%": Don't alarm when a magnet is cycling or ramping (TODO: what EPICS PV indicates cycling or ramping?)
2. Alarms with Action name like "%Mismatch%" or "%Communication%": No alarms when the there is a rack power alarm (TODO: sounds like must be able to determine and monitor parent rack alarm, so implies CED lookup else maybe hard code script per rack. This also overlaps with the proposed Masked_by functionality: Flood Control: ability to suppress alarms #11);
3. Alarms with Action name like "%RF Off%": Don't alarm when machine is running at a particular pass and associated PV don't report the same pass number; specifically when (MMSHLAPASS || MMSHLBPASS || MMSHLCPASS || MMSHLDPASS) != pass number. (TODO: Presumably configured pass is stored in a PV somewhere, we need to identify it)
4. Alarms with Action name "ODH Status": Don't alarm if disabled (TODO: I'm assuming disabled here means there is an EPICS PV that tells us this. We need to figure out what the PV is called.)
5. Hysteresis bit set, don’t alarm?

Note: Why can't the IOC that generates the alarm have this logic seems like a reasonable question to ask. It'll probably be twice as much work and twice as much noise over network and in logs for JAWS to intercept it or create a wrapper producer.