Skip to content

Commit

Permalink
ref(crons): Never allow the clock tick consumer to skip ticks (#70872)
Browse files Browse the repository at this point in the history
We SHOULD process every tick. If there is a problem a backlog is
preferred.

We saw this in INC-760, where a query took too long. Luckily the query
timeout is longer than the arroyo configured `max.poll.interval.ms`.
Meaning the consumer was kicked from the consumer group before we could
fail and try to commit
  • Loading branch information
evanpurkhiser authored May 14, 2024
1 parent ff46507 commit d896596
Showing 1 changed file with 5 additions and 8 deletions.
13 changes: 5 additions & 8 deletions src/sentry/monitors/consumers/clock_tick_consumer.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,13 @@ def process_clock_tick(message: Message[KafkaPayload | FilteredPayload]):
assert not isinstance(message.payload, FilteredPayload)
assert isinstance(message.value, BrokerValue)

try:
wrapper: ClockTick = MONITORS_CLOCK_TICK_CODEC.decode(message.payload.value)
ts = datetime.fromtimestamp(wrapper["ts"], tz=timezone.utc)
wrapper: ClockTick = MONITORS_CLOCK_TICK_CODEC.decode(message.payload.value)
ts = datetime.fromtimestamp(wrapper["ts"], tz=timezone.utc)

logger.info("process_clock_tick", extra={"reference_datetime": str(ts)})
logger.info("process_clock_tick", extra={"reference_datetime": str(ts)})

dispatch_check_missing(ts)
dispatch_check_timeout(ts)
except Exception:
logger.exception("Failed to process clock tick")
dispatch_check_missing(ts)
dispatch_check_timeout(ts)


class MonitorClockTickStrategyFactory(ProcessingStrategyFactory[KafkaPayload]):
Expand Down

0 comments on commit d896596

Please sign in to comment.