Skip to content

fix(plugin-mongodb): list views as views, keep index key order and options, and add db.createView to the shell - #3150

Open
datlechin wants to merge 1 commit into
mainfrom
fix/mongodb-views-and-index-order
Open

datlechin wants to merge 1 commit into
mainfrom
fix/mongodb-views-and-index-order

Conversation

@datlechin

Copy link
Copy Markdown
Member

Root cause

Views listed as tables. fetchTables used mongoc_database_get_collection_names_with_opts. In libmongoc 1.28.1 that call keeps each listCollections entry's name and throws its type away. Every name then became PluginTableInfo(type: "table"), so a view was a table everywhere in the app:

  • editable grid, Add Row, Delete, Rename and Truncate were all offered;
  • Edit View Definition never appeared;
  • the server refuses all of those writes with code 166 (CommandNotSupportedOnView).

system.* collections were plain tables the same way. Three things were also broken that only showed once views were reported correctly:

  • fetchIndexes threw 166 into the Structure tab's error banner.
  • fetchTableDDL returned only // Collection: adults.
  • fetchViewDefinition threw unsupportedOperation.

The shell prelude had no DB.prototype.createView, so db.createView resolved to a collection named createView. The New View template failed with TypeError: db.createView is not a function.

Index order and options lost. listIndexes documents went through JSONSerialization into a [String: Any], so the key document lost its order before the driver saw it. As a result:

  • fetchIndexes reported Array(key.keys) in hash order and hard-coded BTREE.
  • fetchTableDDL wrote the key sorted alphabetically.
  • It kept only unique, sparse and a TTL that never matched (a boxed Int32 does not cast to Int), and dropped partial filters, collations, weights and the rest.
  • The capped size hit the same Int32 cast and printed size: 0.

Separately, the shell's own createIndex passed options through a fixed allowlist rebuilt from a Swift dictionary. So even correct DDL would lose wildcardProjection, a 2d index's bits/min/max, and the text-index options when run in a query tab.

Value types lost in generated statements. A bare number in a statement reaches the shell as a JavaScript Number, which the shell sends as an Int32 or Int64 whatever type the server held. So NumberLong(1) and a whole Double such as 1.0 came back as Int32s, 9007199254740993 came back as 9007199254740992, and {"$minKey": 1} and a $timestamp's t/i became $numberInt wrappers the server cannot read.

Wrappers mongosh reads as documents. A regular expression, a Decimal128 NaN or infinity, a symbol and a date outside years 1 to 9999 kept their canonical Extended JSON wrapper. TablePro's host parses that text with libbson and gets the BSON value back, but mongosh sends the object as it is: measured on mongosh 2.10.0, the view DDL failed with unknown operator: $regularExpression and a validator's with $regex has to be a string.

__proto__ members lost. A member named __proto__ written as a plain key, quoted or not, sets the object's prototype in JavaScript and adds no member. The prelude's EJSON.serialize and EJSON.deserialize copied members by assignment, which has the same effect. So:

  • a $match or a partial filter on __proto__ came back empty from Show DDL, measured in TablePro's shell; in mongosh 2.10.0, { $match: { "__proto__": { $eq: 1 } } } stored { $match: {} };
  • an index keyed on __proto__ came back keyed on the remaining fields only;
  • any script that wrote or read a document with a __proto__ field lost the field.

Names from the server ran as statements. Show DDL wrote a server-chosen name into a // comment as it was: // Collection: <name> before this branch, and // View: <name> on it. MongoDB accepts a line feed, a carriage return, U+2028 and U+2029 in a collection or view name, and JavaScript ends a // comment at each of them, so the rest of the name ran as a statement of its own when the DDL was run from a query tab. v0.75.0 also wrote the collMod name and the index name into string literals with no escaping at all.

Edit View Definition's fallback, reachable now that views are views, had two more:

  • Its MongoDB template escaped the quote alone, so a name with a backslash before a quote closed the string and the rest of the name ran.
  • It wrote the error as SQL -- comments, which JavaScript reads as the decrement operator, and it split the error only at a line feed, so a carriage return or U+2028 in the message ended the comment partway through a view's name.

String literals had a quieter gap. MongoScriptJson.jsonString and the literal writer escaped only the C0 controls, and libbson leaves U+2028, U+2029 and U+0085 raw in canonical Extended JSON. JavaScript reads those inside a string, but the editor's statement scanner ends a string at any Character.isNewline, so it could split a statement in the middle of a name. PluginKit's MongoCollectionAccessor tests a name by Character, and Character.isLetter reads only a grapheme's first scalar, so punctuation sharing a grapheme with a letter passed as an identifier.

Fix

The server's catalog documents stay canonical Extended JSON text, in server order, and are never rebuilt from a dictionary. They become shell text only when a statement is written, every value through a constructor TablePro's shell and mongosh both define, and every server-chosen string escaped for the position it lands in.

Connection.

  • listIndexes drains through the shell's canonical cursor reader instead of a dictionary.
  • New listNamespaces(database:named:) reads mongoc_database_find_collections_with_opts, with nameOnly for the full list or a name filter for one entry.

Pure model, unit-tested without a server.

  • MongoDBNamespaceEntry maps view to VIEW, system.* to SYSTEM TABLE, and everything else (including time series) to TABLE. It builds db.createView(...) and the in-place collMod.

  • MongoDBIndexEntry keeps key order. It types the index from its key (BTREE, HASH, FULLTEXT, SPATIAL, 2D, WILDCARD) and lists a text index's weighted fields. It writes createIndex with every option except v, key and ns.

  • MongoDBShellLiteral writes canonical Extended JSON as shell source:

    • an Int32 as a bare number, NumberLong("..."), and Double(1.0) for every whole Double, -0.0 and 1e+20 included;
    • a fraction in its shortest spelling, Infinity and NaN;
    • NumberDecimal("...") for every Decimal128, NaN and the infinities included;
    • ISODate("...") for years 1 to 9999 in the proleptic Gregorian calendar JavaScript uses, and new Date(<ms>) for any other date a JavaScript Date holds;
    • BSONRegExp("pattern", "options"), BSONSymbol("..."), ObjectId, BinData, Timestamp(t, i), MinKey(), MaxKey() and Code.

    Only a DBPointer, undefined and a date more than 100 million days from 1970 keep their wrapper: mongosh has no way to write any of them, and TablePro's shell sends the wrapper to the server as it is. The index key and options, the view pipeline and collation, the validator and the time-series line are all written this way.

  • MongoDBJsonLayout.shellObject writes every shell object literal, with __proto__ as the computed key ["__proto__"], which JavaScript adds as a member like any other. MongoDBJsonLayout.indented keeps a computed key and new Date(...) whole, and a constructor call on one line.

  • MongoDBNamespaceDDL renders the DDL. A view's header is // View:, so MQL export's // Collection: scraper never appends a createView after a view's documents.

  • Collations drop the server's ICU version, so the text also runs on a server with a different ICU build.

Escaping for each position.

  • String: MongoScriptJson.jsonString escapes every C0 and C1 control, U+2028 and U+2029. MongoDBShellLiteral writes each string again through it instead of copying libbson's text, and writes anything that is not a string, a number, true, false or null as the string it spells.
  • Comment: MongoDBShellText.comment writes every DDL comment line with those same characters as escapes.
  • Identifier: MongoDBShellText.collection writes db.<name> only when every scalar is an identifier character and the name is not a db member, and db.getCollection("<name>") otherwise. The plugin keeps its own check rather than fixing PluginKit's, so a plugin release covers apps that already shipped.
  • Member name: MongoDBShellText.memberName, the computed key above.
  • New MongoDBObjectStatements writes Drop, Truncate and the Edit View Definition fallback template, each naming the collection through MongoScriptJson.jsonString, and the driver returns them.

Driver.

  • fetchTables uses the entry types.
  • fetchIndexes treats 166 as no indexes, as libmongoc already does for a missing collection.
  • fetchTableDDL reads one entry and renders it.
  • fetchViewDefinition returns the collMod that Edit View Definition runs, or No view named %@ in this database.
  • prettyJson and the TableProNumberFormatting import are gone.
  • supportsRenameView = false: renameCollection on a view is refused with 166.

Shell.

  • DB.prototype.createView sends create with viewOn and pipeline.
  • createIndex puts every option into the spec verbatim, in written order. commitQuorum, comment, maxTimeMS and writeConcern go on the command instead, because the server refuses maxTimeMS inside a spec.
  • Double(), BSONRegExp() and BSONSymbol() are new. Autocomplete already offered the first two. A whole Double and -0.0 stay Doubles. BSONRegExp is the same definition the shell-number PR stacked on this one adds.
  • NumberDecimal takes NaN, Infinity and -Infinity, the spellings the server writes.
  • EJSON.serialize, EJSON.deserialize, createCollection and createView define a member named __proto__ instead of assigning it.

App. viewDefinitionFallback takes the template and the tab's line comment marker from EditorLanguage.lineCommentMarker, so a MongoDB tab gets //. It splits the error at every Character.isNewline, and a language with no line comment gets the template alone.

Also here:

  • MongoScriptJson decodes JSON escapes in member names. This hunk is byte-identical to Can't create a database collection from the visual editor #3131's.
  • Four lint findings that were already in the touched files are fixed.
  • A CHANGELOG Security entry, since v0.75.0 shipped the unescaped // Collection: header, collMod name and index name.

No PluginKit change, no kit bump. MongoDB is registry-only, so users get this with a plugin-mongodb release.

Verified

Codex review, three rounds, each fixed on this commit.

  • Round 1: [P1] the view and index DDL were written as relaxed Extended JSON, so NumberLong(1) and a whole Double came back Int32s and 9007199254740993 came back 9007199254740992.
  • Round 2: [P1] a view name holding a line terminator ended its // View: comment and ran as a statement.
  • Round 3: [P1] the Edit View Definition fallback escaped only the quote; [P2] a __proto__ member became a prototype; [P2] regex, special Decimal128, far date and symbol wrappers reach mongosh as documents; [P2] the fallback wrote SQL -- comments into a JavaScript tab.

Against this branch's previous commit. I put the previous commit's behaviour behind the new names (its prelude, literal writer, layout, entries, the hand-made escapes for Drop, Truncate and the template, and the old fallback body) and ran the new and changed suites: 45 executed, 24 passed, 21 failed. The 21 failures cover every round 3 finding: Drop, Truncate and fallback-template containment, __proto__ in the literal writer, the layout and the prelude (sent, read back and written again, and as a createView option), the catalog round trip with __proto__, a symbol and far dates added, the no-wrapper check, BSONRegExp/BSONSymbol/NumberDecimal("NaN"), new Date, and the three fallback-comment cases. The three new cases that passed there pin behaviour that did not change: a date past a JavaScript Date, a DBPointer and undefined keep their wrapper, and each editor language's comment marker. On that code Drop sent v\ndb.probe.drop() for a collection named v\r\ndb.probe.drop(), and a name holding a quote and a combining mark sent a drop of probe. The fix was restored byte-identical (cmp of the diff and of the new files).

Tests. 52 suites (every MongoDB suite, MQL export helpers, JavaScriptStatementScannerTests, EditorLanguageTests, PluginTableKindDecoderTests, ObjectRenameEligibilityTests, DatabaseTreeMenuSpecTests and the new ViewDefinitionFallbackTests and MongoScriptPreludeMemberTests): 884 executed, 884 passed. MongoDBSrvHostTests ran 0 cases because it sits behind #if canImport(CLibMongoc).

Builds and checks. MongoDBDriver, AllPlugins (all 40) and the app build. Lint: 0 violations on the 30 changed Swift files. Docs check passes. Every plugin string is in the catalog. git merge-tree is clean against current main (c21dc51) and against #3131's branch (3da4cfd).

Live, MongoDB 7.0.43 and mongosh 2.10.0. A harness links the real plugin sources against libmongoc and splits each text with the app's JavaScriptStatementScanner, running each statement through the driver the way a query tab does. Comparisons are canonical Extended JSON from libbson, in a probe database dropped afterwards.

  • A view whose $match holds BSONRegExp("(?i)x\/y # z", "ix"), BSONRegExp("^a.b", "s"), a symbol, Decimal128 NaN and -Infinity, dates in year 0 and year 10000, and __proto__ at two depths; a collection with a validator of {email: {$regex: /@/i}, __proto__: {$exists: false}}, an index keyed on __proto__ and a partial filter on it.
    • Previous commit: in TablePro the view came back with __proto__ dropped and nested: {}, the index keyed on a alone and the partial filter empty. In mongosh the view failed with unknown operator: $regularExpression and the validator with $regex has to be a string.
    • This commit: the view is identical after Show DDL and after Edit View Definition's collMod, in TablePro and in mongosh. The indexes are identical in both. The collection entry is identical in mongosh apart from the validationLevel and validationAction the server adds on any collMod that sets a validator. TablePro refuses the validator statement, which is the $regex limitation under "Deliberately not fixed here".
  • Round 2's typed seed again (NumberLong 1 and 9007199254740993, Double 1.0, -0.0 and 1e20, Int32, Decimal128 1.50 and NaN, Infinity, dates, ObjectId, BinData, MinKey, MaxKey, a Timestamp, a regex, and 11 indexes: compound, unique sparse, text, 2dsphere, 2d with bounds, wildcard, hashed, hidden, TTL, partial under a collation, and one keyed on a Double): the view is identical after Show DDL and after collMod, and the indexes after dropIndexes plus the DDL, in TablePro and in mongosh.
  • A view holding a DBPointer, an undefined and a date 9e15 ms from 1970: identical after its Show DDL in TablePro; mongosh refuses it with unknown operator: $dbPointer.
  • The fallback template for a view named v\"}); db.probe.drop(); //, run from a tab: the previous commit dropped probe; this one runs as a single collMod, the server answers that no such view exists, and probe stays.
  • Drop of c\r\nx beside c\nx: the previous commit dropped c\nx; this one drops c\r\nx. Truncate of a collection named with a quote and a combining mark, run from a tab: the previous commit dropped probe; this one runs as a single deleteMany and probe stays.

Deliberately not fixed here

  • $regex holding a regular expression. {email: {$regex: /@/i}} is refused by TablePro's shell as a document MongoDB cannot read, on the previous commit and on this one: libbson 1.28's JSON reader takes a $regex key as the legacy wrapper. {email: /@/i} and {email: {$regex: "@", $options: "i"}} both run. A validator written the first way therefore shows in Show DDL as the server holds it, which runs in mongosh and not in a query tab. The fix is in the host's JSON-to-BSON path, which every shell statement uses. The docs page names the limitation.
  • __proto__ in other generated text. MongoDBQueryBuilder (filters), MongoDBStatementGenerator (row edits, Can't create a database collection from the visual editor #3131's) and MQL export still write a __proto__ member as a plain key, so a filter or an edit on that field reaches the server without it. Found by reading; the mechanism is the one measured above. Each emitter needs memberName, and each is its own feature.
  • Rename, collection stats and create on names with a carriage return and line feed. They go through the driver's private escaper, which on this branch still escapes by Character and turns \r\n into \u000a, so Rename of r\r\nx renamed r\nx in the live run. fix(plugin-mongodb): create collections from New Table with their fields as a validator, read the fields back, and write values in their declared types #3145 on main already made that escaper scalar-wise. Changing it here conflicted with that change, so the merge takes main's.
  • The same escaping gap outside this branch's statements.
    • MQL export writes its own // Database: and // Collection: headers through PluginExportUtilities.sanitizeForSQLComment, which replaces a line feed and a carriage return but keeps U+2028 and U+2029 (MQLExportPlugin.swift:73 and :90). Found by reading, not run through mongosh.
    • PluginKit's MongoCollectionAccessor keeps its Character test, and its escapeJSONString leaves U+2028, U+2029 and U+0085 raw. A PluginKit fix reaches plugins only with an app release, which is why the DDL no longer depends on it.
  • Two listCollections readers now that fix(plugin-mongodb): create collections from New Table with their fields as a validator, read the fields back, and write values in their declared types #3145 has merged. Its declaredSchema still reads listCollections through runCommandJson. Moving it onto listNamespaces(named:) needs this branch rebased onto main first. MongoDBIndexKind must stay the inverse of fix(plugin-mongodb): create collections from New Table with their fields as a validator, read the fields back, and write values in their declared types #3145's indexKeyValue(for:): 2D and WILDCARD read back, but reach its modifyIndex refusal as "MongoDB has no 2D index."
  • Two JSON string readers. MongoScriptJson.readString and feat(plugin-mongodb): insert MongoDB documents written as Extended JSON #3140's MongoDocumentText should become one.
  • Shell number gaps. serializeNumber sends a whole number at or above 2^63 as an overflowing $numberLong, EJSON.deserialize revives a whole Double as a plain number, and autocomplete offers Int32, Long and Decimal128, which the shell does not define. The shell-number PR stacked on this one covers them.
  • System collections are still editable and droppable. Edits on system.* are refused with 73 on system.views and system.profile, but succeed on system.buckets.*. Drop is still offered, and dropping system.views deletes every view. system.js loses Truncate although deleteMany works there. Each is a separate decision.
  • Time series. It stays a table, so a cell edit (72) and Rename (166) are offered and refused by the server. A time-series trait is a follow-up.
  • Stale view tabs. Open tabs and recent-table entries saved before this change keep .table until the view is reopened from the sidebar.
  • Export and MCP side effects. Export streams a view's rows, but the progress total leaves views out. MQL export of a view writes its documents, so a restore creates a collection, which the docs note. MCP get_view_definition now returns the collMod.
  • Metadata stats. fetchTableMetadata and fetchDatabaseMetadata still cast $numberInt stats to Int64/Int and get nil.

No UI test: every behaviour here needs a live listCollections and listIndexes from a MongoDB server, and CI has none. The app-side gating a VIEW or SYSTEM TABLE row drives is covered by PluginTableKindDecoderTests, ObjectRenameEligibilityTests and DatabaseTreeMenuSpecTests, and the fallback text by ViewDefinitionFallbackTests. The round-trip and containment tests drive the real shell prelude in-process and split the text with the app's own statement scanner.

@mintlify

mintlify Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
TablePro 🟢 Ready View Preview Sep 26, 2026, 4:37 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

This branch was successfully deployed

1 active deployment
staging - docs — aaf1c954 Deployed Sep 26, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant