fix(plugin-mongodb): send the connection's write concern with every shell write and name the documents a failed write already changed - #3151
Open
datlechin wants to merge 1 commit into
Conversation
…hell write and name the documents a failed write already changed
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
Write Concern was ignored by everything but inserts. The setting reaches libmongoc only through the URI (
w=), and libmongoc applies it only on its CRUD calls. The shell sends every update, replace, delete, findAndModify and bulkWrite as a raw command throughmongoc_client_command_simple, which applies no write concern. The command builders never added one, and the prelude dropped the options argument ofinsertOne,insertMany,insertandbulkWrite, so a statement's ownwriteConcernwas dropped too. Grid edits and deletes run as the same shell statements, so they ignored Write Concern as well, while grid inserts obeyed it.A write that stopped part-way never said so. MongoDB undoes nothing outside a transaction, but every multi-document path lost track of what it had written:
writeCommandthrew the firstwriteErrorsmessage.scriptInsertthrew away the reply that heldinsertedCount.bulkWritelost its running totals at the first throw.mapExecutionErrorreplaced any code-50 message with read-oriented "add an index" text.Measured on 7.0.43: an
updateManythat changed docs 1 and 2 and failed on doc 3 repliesnModified: 0.Found on the way:
remove(filter, {justOne: true})deleted every match, because the prelude only honoured a literaltrue. And aninsertManythat stopped at an oversized document after libmongoc had already sent a batch failed with an empty message: libmongoc 1.28.1 clears its error when the last batch it sent succeeded (mongoc-write-command.c, end of_mongoc_write_opmsg).Fix
Write concern
MongoDBConnection.writeConcernJson(client:)reads the client's write concern from libmongoc, sow,journalandwtimeoutMSfrom a pasted URL all come along. With Default it returns nil and nothing is sent.MongoScriptCommandBuilder.writeConcern(statementOptions:connectionDefault:)takes the statement's own, else the connection's, the way mongosh does. The statement's is rebuilt fromw,jandwtimeout, taking mongosh'sjournal,wtimeoutMSandfsynctoo. One that names none of them ({},null, unknown keys) counts as unset. Measured: 7.0.43 refusesjournalorwtimeoutMSin a command with[40415] IDLUnknownField, and libmongoc 1.28.1 drops them from an insert's options along with the connection's own concern. mongosh 2.10.0 sends{j, wtimeout}and falls back on{}.insertOptions(statementOptions:)passes only the statement's rebuiltwriteConcernandordered. A statement'sw: 0besidej: true(orjournal, orfsync) goes to the insert asw: 1, j: true: libmongoc 1.28.1 refuses the pair with[22] Invalid writeConcern(mongoc_write_concern_is_valid), and the server treats the two the same, since it leaves out a write's reply only for awbelow 1 with neitherjnorfsync(shouldSkipOutput,src/mongo/db/commands/write_commands.cppon v7.0). Commands keep the pair as written.isAcknowledged(writeConcern:): a numericwof 0 or below withoutj: true. Measured: aw: 0update, delete or insert command is answeredn: 0whatever it changed, and a duplicate key it hit is not reported. libmongoc sends an insert with its legacyw: -1without waiting for any answer, and the server drops it (it refuses a negativew). Those writes return{acknowledged: false}with no counts (inserts keep their ids), as mongosh does.findAndModifyis answered in full underw: 0, so it keeps its document. An update or delete withw: -1fails with the server's own refusal, as it does in mongosh.insertOne,insertMany,insertandbulkWrite.removetakestrueor{justOne, …}like mongosh.mongoc_collection_update_onerejects a replacement document on the client, which legacyupdate(q, doc)relies on. The Bulk API rejectsmaxTimeMS, which is how the query timeout bounds a write.Partial writes
MongoWriteFailuregains astage:.document,.unconfirmed,.command,.unansweredor.notSent. It reads the CRUD reply's pluralwriteConcernErrorsanderrorReplies, and a top-levelok: 0. The code always comes from the reply, never frombson_error_t.scriptWriteCommandandscriptInsertreturn aMongoWriteOutcomethat keeps the reply even when the write failed.MongoWriteLedgerunder its activity lock, and marks a write in flight so the silence watchdog can see it. An unacknowledged write counts as one that may have changed documents. Every stage counts what its reply says was written,.notSentincluded: a largeinsertManygoes out in batches, and one that stops at a document it cannot send has already inserted the batches before it. That insert now reports "The insert stopped at a document larger than MongoDB accepts." rather than an empty message.ok: 0and a code in the server'sInterruptioncategory may have written documents first: the server fails the whole batch for exactly those codes (write_ops_exec.cpp,handleError).MongoDBServerErrorCode.interruptionCategoryis the union of that category on every release branch from 4.0 to 9.0, read from each branch'ssrc/mongo/base/error_codes.yml(error_codes.erron 4.0 and 4.2). It adds 91331 (8.0), 10045600 (8.1), 453 (8.2), 471, 473 and 485 (8.3) and 509 (9.0 branch) to the 13 codes 5.1 to 7.3 share.scripts/check-mongodb-interruption-codes.shdiffs the set against every branch, the way the Redis and MySQL curated tables are checked.MongoWriteFailure, its stage crosses the bridge on the exception, and the driver reads it from the error that actually escaped. Before, it matched the last failed write by code and message, so a caught write timeout made a latercounttimeout (same[50] operation exceeded time limit) read as a write.reportedError): timeout wording first, write wording only for a write's own failure, the note after. A write-concern error keeps its "applied" text even when its code is 50. A cancel stays a quiet cancel.use, into the stream. A stream that fails reports the note, and the driver follows theuseon success and on failure.Verified
main..notSentinsert keeping its count,w: 0withj,journalorfsyncon an insert,w: -1, and the seven newer interruption codes) fail 14 of 14 when compiled against the previous commit's sources and pass 14 of 14 against this one.MongoDBDriver(only its 4 existing Sendable-capture warnings),AllPlugins(all 40), the app. Lint: 0 violations on the 19 changed Swift files. Docs checks pass. Plugin localization check passes.shellcheck --severity=warningpasses on the new script, and the script reportsok: 20 codes, the union across v4.0 to v9.0(a copy with 453 swapped for 11000 reports both).Write Concern 2
updateOne,deleteMany,findOneAndUpdate,bulkWritewith{journal: true, wtimeoutMS: 1000}[40415] ... 'WriteConcernOptions.journal' is an unknown field.{j: true, wtimeout: 1000}insertOnewith the same{j: true, wtimeout: 1000}updateOne/insertOnewithwriteConcern: {}w: 2droppedcannot use 'w' > 1 when a host is not replicated{fsync: true, w: 1}{w: 1, j: true}Write Concern Majority with URL
journal=true&wtimeoutMS=4321: update and insert go out as{w: "majority", j: true, wtimeout: 4321}.w: 0andw: -1updateOnew: 0acknowledged: true, matched 0, modified 0, document changed{acknowledged: false}insertManyw: 0with a duplicate firstacknowledged: true,insertedCount: 2, nothing inserted{acknowledged: false, insertedIds}deleteMany,bulkWritew: 0{acknowledged: false}updateManyw: 0, then a duplicateinsertOnew=0,insertOneacknowledged: trueacknowledged: false; a statement's{w: 1}still gets real countsinsertOne/insertManywith{w: 0, j: true},{w: 0, journal: true}[22] Invalid writeConcern{w: 1, j: true}insertOnewith{w: 0, fsync: true}[22] Invalid writeConcernE11000 duplicate key errorinsertManywith{w: -1}acknowledged: true,insertedCount: 2, nothing inserted{acknowledged: false, insertedIds}w=-1,insertOneacknowledged: true, nothing insertedacknowledged: falseinsertOnew: -1, then a duplicateinsertOneupdateOnewith{w: -1}[9] w has to be a non-negative number ...{w: 0, j: true}onupdateOnekeeps real counts, andfindOneAndUpdatewithw: 0returns its document in both.Partial writes, 1 s query timeout
updateManytimeout, then acounttimeoutupdateManyrethrownupdateOne; find(slow)use("b"); db.c.find(), success and timeoutbinsertManyof four 15 MB documents and a 17 MB one (3 inserted)insertManywhose connection a proxy drops after forwarding it (both inserted)The tables above hold each round's before and after. Rechecked on this final build: validator
updateManyfailing on its 3rd document gets the "may" note, ordered and unorderedinsertManywith duplicates report 2, abulkWriteof insert,updateManyover 6, duplicate insert reports 7,remove({...}, {justOne: true})deletes 1, the caught-timeout-then-countstatement reads as a query, a loneupdateManytimeout reads as a write, theupdateOne; find(slow)export carries its count, theuseexport follows the database, and under Write Concern 2 thejournal/wtimeoutMS,{}andfsyncrows behave as tabled. The grid edit and delete rows were not rerun this round.unacknowledgedStopBetweenBatchesMayHaveChanged).Deliberately not fixed here
bulkWrite: operations still run one at a time and stop at the first failure. Documented as a limitation.w: 0changes what a command write reports, not how long it takes:mongoc_client_command_simplealways waits for the server's reply.$merge/$outfailure fromaggregategets no count.aggregatealready applies the write concern through libmongoc.catchblocks see only the server message, with no result counts like mongosh'sMongoBulkWriteError.db.runCommandgo to the server as written. The docs say so.insertManywith an oversized document still inserts the batches libmongoc sent before it; the error now says so and counts them. Refusing the whole call up front needs a size check against the server's limit before the insert, which is its own change.No UI test: CI has no MongoDB server, so a flow that writes to a collection cannot run deterministically there. The grid and shell paths were checked live against 7.0.43 instead.