Skip to content

fix(plugin-parquet): NUMBER exports as BIGINT and PostgreSQL money as null - #3195

Merged
datlechin merged 4 commits into
mainfrom
fix/parquet-type-mapping
Sep 30, 2026
Merged

datlechin merged 4 commits into
mainfrom
fix/parquet-type-mapping

Conversation

@datlechin

Copy link
Copy Markdown
Member

Summary

  • Oracle, Snowflake and Dameng NUMBER columns exported to Parquet as BIGINT, so every fraction was rounded and values past 2^63 became null. Root cause: number sat in the mapper's integer family, and the base-name match drops the declared (p,s).
  • DECIMAL(p,s) and NUMERIC(p,s) columns exported as DOUBLE, losing digits past about 15 significant ones. Root cause: the mapper ignored the declared precision and scale for the whole exact-numeric family. A declaration now maps to DECIMAL(p,s), or to BIGINT for scale 0 with at most 18 digits. A missing declaration, *, or more than 38 digits stays DOUBLE. SQLite, libSQL, Turso and Cloudflare D1 also keep DOUBLE, because they keep a value such as 19.9875 in a DECIMAL(10,2) column as a plain float and never enforce the scale.
  • Oracle BINARY_FLOAT and BINARY_DOUBLE columns exported as text. Root cause: neither type name was in any numeric family.
  • PostgreSQL money values written as null. Root cause: money mapped to DOUBLE, but PostgreSQL sends it as locale text such as $1,234.56, and TRY_CAST turns that into null. PostgreSQL money now stays text. SQL Server money and smallmoney, which arrive as plain decimal text, map to DECIMAL(19,4) and DECIMAL(10,4).
  • Stopping a multi-table Parquet export between tables left the files already written. Root cause: the loop-head cancellation check sat outside the do/catch that removes those files.

Tests

  • ParquetTypeMapperTests.declaredExactNumericKeepsPrecision: fails without the fix. NUMBER(19) came back BIGINT on main and DOUBLE on the first version of this branch, where it should be DECIMAL(19,0).
  • ParquetTypeMapperTests.undeclaredExactNumericIsDouble: fails without the fix, because bare NUMBER mapped to BIGINT.
  • ParquetTypeMapperTests.exactNumericWithScaleZeroIsAnInteger: fails without the fix, because DECIMAL(10,0) and numeric(5) mapped to DOUBLE.
  • ParquetTypeMapperTests.sqliteFamilyIgnoresDeclaredPrecision: fails on main (number(19) mapped to BIGINT). It also failed on this branch until the SQLite-family check was added.
  • ParquetTypeMapperTests.oracleBinaryFloatsAreDoubles: fails without the fix, because both types mapped to VARCHAR.
  • ParquetTypeMapperTests.postgresMoneyStaysText and sqlServerMoneyIsFixedPoint: both fail without the fix, because money mapped to DOUBLE on every engine.
  • ParquetTypeMapperTests.doubleFamilies: spec change. money and DECIMAL(10,2) leave the list of types pinned to DOUBLE.
  • ParquetExportCancellationTests.stopBetweenTablesRemovesWrittenFiles: fails when the cancellation check sits outside the cleanup. The other three cases pin paths that already worked: a failing table removes earlier files, a finished export keeps every file, and a single table writes to the destination itself.

Docs

User-facing text is covered by the docs rewrite branch.

@datlechin
datlechin merged commit e5423e5 into main Sep 30, 2026
5 checks passed
@datlechin
datlechin deleted the fix/parquet-type-mapping branch September 30, 2026 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant