Skip to content

Commit

Permalink
plots.cursor: fix smallest digit calculation (corner cases) (#377)
Browse files Browse the repository at this point in the history
  • Loading branch information
pmldrmota authored Oct 31, 2023
1 parent a6e3f5f commit b20864d
Showing 1 changed file with 8 additions and 3 deletions.
11 changes: 8 additions & 3 deletions ndscan/plots/cursor.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,14 @@ def update_coords(self, data_coords):
raise NotImplementedError

def set_value(self, value: float, limits: tuple[float, float]):
# We want to be able to resolve at least 1000 points in the displayed
# range.
span = (limits[1] - limits[0]) * self.data_to_display_scale
# Base case: we want to resolve at least milli-units on the data's scale.
span = self.data_to_display_scale
if limits[1] > limits[0]:
# Preferred case: we want to resolve >1000 points in the displayed range.
span *= (limits[1] - limits[0])
elif np.abs(value) > 0:
# Fallback case: we want to resolve >3 significant figures of the value.
span *= value
smallest_digit = np.floor(np.log10(span)) - 3
precision = int(-smallest_digit) if smallest_digit < 0 else 0

Expand Down

0 comments on commit b20864d

Please sign in to comment.