Fixed Point Number - Precision
About
Precision is the maximum number of significant digits (on either side of the radix point) in the fixed point notation
If a value exceeds the precision, then an application such as a database would return an error.
Example
Single number
- The number 00123456.78 has a precision of 8
- 123.45 has a precision of 5
- 1.2345 has also a precision of 5
Decimal and precision
| Precision | Rounded to significant figures | Rounded to decimal places |
| 6 | 12.3450 | 12.345000 |
| 5 | 12.345 | 12.34500 |
| 4 | 12.35 | 12.3450 |
| 3 | 12.3 | 12.345 |
| 2 | 12 | 12.35 |
| 1 | 10 | 12.3 |
| 0 | N/A | 12 |