Skip to content

Conversation

@briangow
Copy link
Contributor

@briangow briangow commented Jan 7, 2026

As discussed in #515, an overflow can occur when calculating ADC parameters from limited-precision physical signal data. The root cause of this overflow is precision loss. When computing min/max values during parameter calculation, insufficient floating-point precision at boundaries (e.g. int32 boundaries) leads to gain/baseline values that cause digital samples to overflow by exactly 1 value.

The fix ensures float64 precision is used throughout ADC parameter calculation and conversion:

  • In calc_adc_params(): Convert physical signals to float64 before computing min/max
  • In adc_inplace(): Convert to float64 before performing gain/baseline arithmetic

Users can now safely provide lower-precision physical data (e.g., float32) without validation errors.

This PR includes a test for the overflow which uses the data supplied by @Ivorforce in #515. The test checks that:

  • float32 input data can be written with fmt=32 without overflow
  • digital values stay within the valid range
  • the physical signal round-trips accurately through write/read

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants