Skip to content

Decode NUMERIC columns as Decimal without precision loss - #835

Open
aminghadersohi wants to merge 1 commit into
crate:mainfrom
aminghadersohi:numeric-exact-decode
Open

aminghadersohi wants to merge 1 commit into
crate:mainfrom
aminghadersohi:numeric-exact-decode

Conversation

@aminghadersohi

Copy link
Copy Markdown

Fixes #826.

Problem

CrateDB's HTTP endpoint sends every digit of a NUMERIC value, but _json_from_response decodes the body with orjson.loads, which turns every JSON number into a float. SELECT n on a NUMERIC(38, 18) column holding 12345678901234567890.123456789012345678 returns 1.2345678901234567e+19. Writes are already exact, because Decimal is serialized as a string.

Change

After the orjson decode, check the response's col_types. If any column is NUMERIC (22), or an array of NUMERIC ([100, 22]), decode the body a second time with json.loads(..., parse_float=Decimal) and replace only those columns' cells with Decimal values. Integers in those columns also become Decimal.

  • Other columns are unchanged. DOUBLE stays float, and integers stay int.
  • A response without a NUMERIC column (and any bulk or rowless response) is decoded once, as before. orjson's speed is unaffected for them.
  • The type does not depend on a connection option, which NUMERIC reads: full digits arrive, the decode drops them #826 considered. A NUMERIC column always reads as Decimal, the type Decimal writes already use.

Tests

  • test_numeric_columns_decode_without_precision_loss covers a NUMERIC value beyond double precision, a DOUBLE next to it, a NUMERIC array with an int and a null, a null NUMERIC, and integer cells.
  • test_responses_without_numeric_columns_unchanged covers the fast path.
  • tests/client: 137 passed. ruff check and mypy src are clean.
  • Live check against CrateDB 5.10.16 with a NUMERIC(38, 18) column: before the change, the value read back as a float with the fractional digits lost. After it, Decimal('12345678901234567890.123456789012345678').

I left the pre-existing ruff format differences in files this PR doesn't touch as they are.

orjson decodes every JSON number to a float, so NUMERIC values lost the
digits beyond double precision that CrateDB sends. When a response
declares a NUMERIC column, or an array of them, decode the body a second
time with Decimal floats and replace only those columns. Other columns
keep their float values, and responses without NUMERIC columns are
decoded once, as before.

Fixes crate#826
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NUMERIC reads: full digits arrive, the decode drops them

1 participant