Fix ADC parameter calculation overflow errors #553
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed in #515, an overflow can occur when calculating ADC parameters from limited-precision physical signal data. The root cause of this overflow is precision loss. When computing min/max values during parameter calculation, insufficient floating-point precision at boundaries (e.g.
int32boundaries) leads to gain/baseline values that cause digital samples to overflow by exactly 1 value.The fix ensures
float64precision is used throughout ADC parameter calculation and conversion:calc_adc_params(): Convert physical signals tofloat64before computing min/maxadc_inplace(): Convert tofloat64before performing gain/baseline arithmeticUsers can now safely provide lower-precision physical data (e.g.,
float32) without validation errors.This PR includes a test for the overflow which uses the data supplied by @Ivorforce in #515. The test checks that:
fmt=32without overflow