Skip to content

fix(plugin-mongodb): build the export header from a field census, not the first 200 documents - #3213

Open
datlechin wants to merge 3 commits into
mainfrom
fix/mongo-export-header-sampled
Open

datlechin wants to merge 3 commits into
mainfrom
fix/mongo-export-header-sampled

Conversation

@datlechin

@datlechin datlechin commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Summary

  • MongoDB exports (every format, plus Object Copy and Compare, which read the same stream) drop every top-level field that first appears after the 200th document. Root cause: the stream fixes its header from the first 200 documents and projects every later document onto it, so the header now also takes the fields a server-side census ($objectToArray, $unwind, $group) finds in the documents the stream reads, grouped by an object keyed on the field name so no collation can merge Email with email, and a stream whose census cannot run logs the fields it left out.
  • A field that is null in all of the first 200 documents is exported as VARCHAR, so MQL export writes its later dates, ObjectIds and decimals as plain strings. Root cause: the same 200-document sample fixes each column's type, so a sampled column that held only nulls now takes the type the census found.

Tests

  • MongoStreamProjectionTests/censusFieldJoinsTheHeader: 200 sampled documents without late and a census reporting late as a date give a late column typed TIMESTAMP and a non-null cell for document 201. Fails without the fix (columns ["_id", "name"]).
  • MongoStreamProjectionTests/censusFieldsFollowTheSampledColumns, censusFieldTakesTheMajorityType, censusTypesAColumnTheSampleHeldOnlyNullsIn: census field order, majority typing with nulls not voting, and a sampled all-null column typed by the census. Each fails without the fix.
  • MongoStreamProjectionTests/fieldsOutsideTheHeaderAreReported, fallbackHeaderAnnouncesOnlyTheId: the fields a header left out are reported. They do not compile without the fix.
  • MongoFieldCensusTests/fieldNamesAreGroupedAsObjectKeys, groupRepliesBecomeTallies, missingExampleIsNull: fields are grouped by an object keyed on the field name and read back from the group's name. All three fail with a {name, type} group key. Checked on MongoDB 5.0, 6.0, 7 and 8.2: under a collection collation of {locale: "en", strength: 2} and a statement collation of {locale: "en", strength: 1, alternate: "shifted"}, a {name, type} key returns one group for Email/email and for first_name/firstname, and the object key returns both names.
  • MongoFieldCensusTests (the other cases): the census covers the documents the stream reads (a find's filter, paging, row ceiling and projection; an aggregation's own output; no census for $out or $merge) and keeps the statement's hint, collation and maxTimeMS.
  • MongoStreamProjectionTests/missingCensusKeepsTheSampledColumns, censusLeavesSampledTypesAlone: without a census the header and its types are unchanged.
  • Covering run, 215 of 215 passed: MongoStreamProjectionTests, MongoFieldCensusTests, BsonDocumentFlattenerTests, BsonFieldPathArrayTests, BsonFieldPathTests, MongoDBCollectionSchemaTests, MongoDBNestedValueWriteTests, MongoScriptCommandBuilderTests, MongoScriptCursorOptionsTests, MongoScriptJsonTests.

Docs

The docs rewrite branch covers the user-facing text.

CI fixes

The Package Tests job failed in SyncRecordMapperTests.unknownWireValueFailsClosed: an unknown Safe Mode value decoded as Off instead of Confirm Writes. The sync mapper now uses the model's SafeModeLevel(wireValue:isReadOnly:) decoder. Unknown values require confirmation, legacy read-only restrictions remain, and renaming a connection preserves its unknown wire value. Added regression tests for read-only restrictions and renames.

Integrated main and fixed another inherited build error: SurrealValue.text(_:) called jsonText(length), but main had replaced that helper with a property. Restored the length-aware helper, keeping JSONTruncation for grid display and full JSON for exports. Existing long-structure export and grid-display tests cover both paths.

Follow-up validation on the final merged branch:

  • swift test --package-path Packages/TableProCore: PASS, all Swift Testing and XCTest runs passed.
  • All SurrealDB suites in an isolated SwiftPM harness: PASS. The harness compiles the actual plugin source files, the existing SurrealDBDriverTests.swift suites, and the host's TableFilter model where needed. It adapts imports for the separate test module; it does not run the app's Xcode test host.
  • SwiftLint on changed Swift files: zero violations.
  • Project generation and git diff --check: PASS.
  • verify.sh plugins (AllPlugins): PASS on the merged fix(plugin-surrealdb): BETWEEN, REGEX and IS EMPTY filters become equality checks #3218 branch, including the common SurrealDB helper fix.
  • The lint helper's separate documentation check reports four existing stale paths in CLAUDE.md and the verification guide, which this follow-up leaves unchanged.
  • Reviewed the final diff directly.
  • GitHub CI reruns pending after push.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant