fix(plugin-mongodb): store Set NULL as null, tell a missing field from null, and add Remove Field

[datlechin]

Stacked on #3159 (the grid write serializer), which is stacked on #3149 (the row-write channel). Retarget each to main as the one below it merges. Part of #3132.

Root cause

The grid had one value for a MongoDB field that is null and for one that is missing: .null.

• Read side. BsonDocumentFlattener.flatten returns .null for a key the document lacks and for a key holding null, and nothing else said which it was.
• Write side. MongoDBStatementGenerator read .null as "remove":
  • An update sent $unset.
  • An insert left the field out.
  • A restore left the field out.

So:

• Set NULL deleted the field. On a validator with required: ["deletedAt"] and bsonType: ["date", "null"] the save failed with Document failed validation. Without a validator, {deletedAt: {$exists: true}} stopped matching.
• Duplicating {deletedAt: null} dropped the field.
• Data Rewind restored a null field as missing.
• There was no way to remove a field on purpose, and a removed field looked exactly like a null one.

PluginCellValue is @frozen, so a third state could not be added to the value itself. Presence therefore travels beside the value, and every path that copies, compares, undoes, redoes, rewinds or pastes a row value has to carry it.

Data Rewind had a second, released problem on MongoDB. It compared the rows it read back by position against the recorded columns, but a MongoDB read returns only the fields its documents have. Measured on MongoDB 7.0.43: after $unset of nick, reading that document back returned the columns [_id, name] against a record of [_id, name, nick]. A field added since returned [_id, name, zeta], so the comparison read the wrong field.

Two gates decided what the grid and the inspector offer, and each read less than the server does:

• Which fields take NULL. The validator reader looked at one keyword per field. A nullable bsonType returned early, so bsonType: ["string", "null"] with enum: ["draft"] read as nullable while the server refuses {status: null} with 121. Rules over the whole document were not read at all.
• Which fields the inspector may edit. The inspector marked only generated columns read-only. A MongoDB _id is immutable too, so the inspector took typing, Set NULL and Remove Field on it, and staging then refused the edit and left it pending in the field. The typing and Set NULL part shipped in v0.75.0.

Fix

Absence travels beside the value, never as a sentinel. An engine that cannot tell the two apart never sets it, so its comparisons, statements and menus stay as they were.

PluginKit (additive, kit 33 reused)

• PluginQueryResult.absentCells: row index to the columns that row lacks. An optional stored var, decoded with decodeIfPresent.
• PluginRowChange.absentColumns: the fields an update removes, or a new row leaves out.
• DriverPlugin.supportsFieldRemoval: defaults to false and is carried through the metadata snapshot. A plugin built before it existed offers none of the new UI.
• generateIdentityPreservingInsert(…, absentCells:): its default forwards to the old requirement, so DynamoDB and Spanner are unchanged.

MongoDB driver

• A browse reports the fields each document lacks.
• Set NULL writes $set: {field: null}. Only a field the change removes gets $unset ($unsetField inside the pipeline route).
• New rows, duplicates, pastes and restores write NULL as null and leave out only the missing fields. Each written value still carries the serializer's provenance: typed into the new row, or copied by a duplicate or a paste.
• The empty-name and __proto__ refusals used to say "Set it to NULL". NULL is now stored, and measured through the shell on 7.0.43, insertOne({"": null, "a": 1}) still fails with [22] invalid document for insert: empty key and {"__proto__": null, "b": 2} stores only b. The refusals and the docs now say to remove the field, which leaves it out, and that insert succeeds.

Which fields take NULL

• A field takes NULL only when every keyword of its rule lets null through: its type lists null or is not declared, and any enum lists null too. A combinator on the field counts as refusing it. minLength, pattern and the other type-specific keywords never see null, measured on 7.0.43.
• A rule over the whole document that the reader does not model hides NULL on every field: a combinator, patternProperties, dependencies, a document enum, or a query operator beside $jsonSchema.
• An additionalProperties schema decides NULL for the fields properties does not name.
• validationAction: "warn" and validationLevel: "off" refuse nothing, so every field takes NULL there.
• Being absent from required no longer decides nullability at all.

Grid and inspector

• A missing field draws No Field, which is also its VoiceOver text.
• Remove Field is on the cell's context menu and in the inspector's value menu.
• NULL typed over a missing field is a real edit.
• A new row starts with every field missing. Duplicate and Paste copy which fields were missing and which held null. Pasted text has no such distinction, so a NULL in it leaves the field out.
• Every structured copy builds its clipboard payload in one place, GridRowsClipboardPayload(columns:copying:projection:), so Copy with Headers from the row menu carries missing fields as well. The unused TableViewCoordinator.copyRows(at:) is gone.
• The inspector's pickers route every tag through one FieldPickerSentinel.choose. Choosing the No Field or Multiple values row writes nothing.
• In the inspector, Remove Field on a field every selected row already lacked leaves nothing pending, the same as typing a field's own value back. The inspector-only save carries the removal and each row's missing fields, so it can never write $set: {field: null} for a removal.
• The inspector's read-only fields are now the change manager's own answer, DataChangeManager.unwritableColumns(among:), which is isColumnWritable over every column: generated columns plus the driver's immutable ones. A MongoDB _id is read-only in the inspector and offers no typing, Set NULL or Remove Field. Weaviate's uuid and vector get the same.

Undo, redo and discard

• Undo, redo and discard put presence back.
• Redoing a new, duplicated or pasted row's insertion brings back the fields it lacked. The single-row insertion undo action now carries the row's values and missing fields, as the batch one already did, in place of a side channel that held the values only.

Data Rewind

• Each saved row records which fields it lacked before and after the save.
• Rewinding Remove Field sets the value back. Rewinding a value typed into a missing field removes it again.
• A restored delete keeps its null fields and leaves missing ones out.
• The conflict check compares presence as well as value on every written column. Rewinding Set NULL on a missing field, or Remove Field on a null one, now restores instead of reading as already restored. A field someone removed or added since the save reads as changed since the save.
• On an engine that reports missing fields, the rows read back are matched to the record by field name. A recorded field the read did not return counts as missing. SQL reads stay positional.
• A record saved before this change reads NULL as missing, which is how it was rewound then, and is compared on values alone.

Verified

Everything here ran on this pass, on the rebased and amended commit.

• Restacked onto fix(plugin-mongodb): write binary, nested and specially named fields from the grid as the values they are #3159's head (b4c0a3a46), which is the serializer rebuilt on fix(datagrid): refuse a save that would leave out a change its driver cannot write #3149 plus its round-3 fixes. The commit applied with one string catalog merge. The changed suites plus MongoDBNestedValueWriteTests, MongoDBStatementGeneratorTests and StringCatalogIntegrityTests: 304 of 304. The app, MongoDBDriver and AllPlugins build, and every plugin string is in the catalog.

• Rebase: the two conflicts were resolved keeping the serializer's provenance, field kinds and refusals and this branch's presence rules. Before continuing, 36 suites (this branch's, the MongoDB write suites, row-write coverage, save completion) ran 641 of 641 passed.

• Build: verify.sh build (TablePro and MongoDBDriver) and verify.sh plugins (all 40) pass.

• ABI: against the serializer head it was built on, additions only: 8 lines added, none removed. The kit stays 33, since v0.75.0 ships 32.

• Tests:

  • The suites for this pass's two fixes: 190 of 190 passed, including the 6 new cases.
  • 129 suites across MongoDB, change tracking, pending changes, row operations, rewind, inspector, table rows, the plugin adapter and the metadata registry: 1,613 of 1,613 passed.
  • With the old nullability rule and the old inspector read-only set put back in place, exactly the 6 new cases failed (26 run). Sources were restored afterwards.

• Lint, docs and strings: 0 violations on the 76 Swift files the branch changes. verify.sh docs passes. Both string catalogs verify, and every plugin string is in the catalog.

• Live on MongoDB 7.0.43, with a harness running this branch's driver:

  • Offered NULL against the server's own answer to $set: {s: null}, one collection per validator:

    Validator	NULL offered	Server
    s: ["string","null"] + enum: ["draft"]	no	121
    s: ["string","null"] + enum: ["draft", null]	yes	stored
    s: ["string","null"] + minLength: 3, pattern	yes	stored
    top-level anyOf constraining s	no	121
    {$type: "string"} on s beside $jsonSchema	no	121
    patternProperties: {"^s": string}	no	121
    additionalProperties: string, undeclared u	no	121
    additionalProperties: string, declared t: ["string","null"]	yes	stored
    s: string with validationAction: "warn" or validationLevel: "off"	yes	stored

  • An empty field name holding null fails the insert with [22], and __proto__: null is dropped. The same row with that field removed inserts {"name": "Grace"}.

  • Set NULL sent {"$set": {"deletedAt": null}}, and $exists: true with $type: "null" matches.

  • Remove Field sent {"$unset": {"nick": ""}}, and the field is gone. On the required field the server refuses with 121.

  • Duplicating {deletedAt: null} inserted {"deletedAt": null, "name": "b"}. The old statement, insertOne({"name": "b"}), is refused with 121.

  • Rewinding a delete inserted {"_id": 2, "deletedAt": null, "name": "b"}: null kept, nick still missing.

  • The rewind read-back after a $unset returned [_id, name], and after a new field [_id, name, zeta], against a record of [_id, name, nick]. Reading both reported the missing cells {0: [3], 1: [2]}.

  • Every probe_null-vs-absent_* database was dropped.

• Codex reviews:

  • Round 1 found two P1 and three P2 issues. Rewind compared values only, so a presence-only save read as already restored. Single-row redo lost missing fields. The inspector's Remove Field stayed pending over rows that already lacked the field, and the close-prompt save wrote $set null. The pickers sent the No Field tag as a value. Copy with Headers dropped missing fields. All five were fixed in the previous pass.
  • Round 2 found two P2 issues. NULL was offered where a sibling enum refuses it. Remove Field was offered on _id in the inspector. Both are fixed here, together with the rest of each class.

Deliberately not fixed here

• is NULL filter: it still matches a null field and a missing field alike, as {field: null} does in mongosh. The docs say so and give the $exists: false query. A separate operator would touch every engine's filter bar.
• Rules over the whole document hide NULL everywhere: under a top-level anyOf or a query operator beside $jsonSchema, measured live, fields those rules never name lose Set Value > NULL too, although the server stores null in them. Modeling combinators and match expressions field by field is a larger change. validationLevel: "moderate" is treated as enforced. The docs say both.
• Inspector nullability: Set NULL, Set DEFAULT and SQL functions in the inspector still skip the grid's nullability gate, on every engine. Read-only fields are now gated. On MongoDB the validator refuses a null it does not allow at save.
• Inspector wiring: the inspector's read-only set is built in a SwiftUI view. The tests cover the change manager's answer and the inspector state built from it, not that one call site.
• Set NULL on a stored NULL still marks the inspector field pending, on every engine. It writes the same null again.
• Remove Field on a required field is offered, and the server refuses it at save.
• Copy and export as text: Copy as TSV, Copy Cell, Copy as JSON, Show Row as JSON and exports write NULL or null for a missing field. Presence rides only on the in-app structured clipboard.
• Found while testing: the collection's _id kind is last-writer-wins across every result on the session. A query whose projection turns _id into a string retypes the next _id filter until the collection is browsed again.

No UI test: CI has no MongoDB server, and every path here needs a connected collection with sparse documents. The host side runs through the real MongoDB generator in unit tests.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
TablePro	🟢 Ready	View Preview	Sep 26, 2026, 7:23 PM

💡 Tip: Enable Automations to automatically generate PRs for you.