fix(plugin-weaviate): exports fail past 10,000 objects and the raw filter row is sent as a property - #3217
Open
datlechin wants to merge 4 commits into
Open
fix(plugin-weaviate): exports fail past 10,000 objects and the raw filter row is sent as a property#3217datlechin wants to merge 4 commits into
datlechin wants to merge 4 commits into
Conversation
…ursor so exports finish
…a property named __RAW__
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GET /v1/objectsbyoffset, and Weaviate answers HTTP 422query maximum results exceededonceoffset + limitpassesQUERY_MAXIMUM_RESULTS(10,000 by default). The export now reads the plain first page and then pages by theafteruuid cursor, which the cap does not apply to, and it stops with a clear error on a server older than 1.18, which ignoresafter.__RAW__, and the server rejects that filter. The Weaviate filter builder had no__RAW__branch, so it treated the row as an ordinary text property. It now refuses the row with a message that points to a GraphQLwherein the editor.Tests
swift test --package-path Packages/TableProCore --filter TableProWeaviateCoreTests: 88 of 88 pass.WeaviateObjectPagesTestsuses a stub transport that behaves the way Weaviate 1.32.4 was measured to behave: it answers 422 whenoffset + limit > 10000, returns only the uuids strictly afterafter, and refusesaftercombined with an offset.readsPastQueryMaximum: 12,000 objects drain to 12,000 uuids, in order. Fails without the fix with.api(status: 422, message: "msg:offset or limit code:400 err:query maximum results exceeded").pagesByCursor: pages of 500, 500 and 200.afteris nil, then each page's last uuid, and no request sendsoffset. Fails without the fix.refusesACursorThatDoesNotMove: a server that ignoresafter, as 1.17 does, fails after two requests instead of looping. Fails without the fix.refusesAPageWithoutACursor: a full page whose last object has no id fails after one request. New with the fix, since the offset walk had no cursor to lose.firstPageKeepsTheNilUUID,endsOnEmptyPage,omitsVectorWhenAsked: these guard behavior the offset walk already had, so they pass with or without the fix.WeaviateFilterOperatorTests:rawFilterRowIsRefused: a raw row alone, and a raw row next to a real filter, both throwrawFilterUnsupported. Fails without the fix: no error is thrown.rawFilterRefusalNamesTheEditor: the refusal message. Fails without the fix.Also checked:
WeaviateDriverPluginbuilds,StringCatalogIntegrityTestspasses, andswiftlint lint --strictreports 0 violations on the changed files.Docs
The docs rewrite branch covers the user-facing text: exporting a whole collection needs Weaviate 1.18 or later, and the Raw Filter row is not available for Weaviate.
CI fixes
The Package Tests job failed in
SyncRecordMapperTests.unknownWireValueFailsClosed: an unknown Safe Mode value decoded as Off instead of Confirm Writes. The sync mapper now uses the model'sSafeModeLevel(wireValue:isReadOnly:)decoder. Unknown values require confirmation, legacy read-only restrictions remain, and renaming a connection preserves its unknown wire value. Added regression tests for read-only restrictions and renames.Integrated main and fixed another inherited build error:
SurrealValue.text(_:)calledjsonText(length), but main had replaced that helper with a property. Restored the length-aware helper, keepingJSONTruncationfor grid display and full JSON for exports. Existing long-structure export and grid-display tests cover both paths.Follow-up validation on the final merged branch:
swift test --package-path Packages/TableProCore: PASS, all Swift Testing and XCTest runs passed.SurrealDBDriverTests.swiftsuites, and the host's TableFilter model where needed. It adapts imports for the separate test module; it does not run the app's Xcode test host.git diff --check: PASS.verify.sh plugins(AllPlugins): PASS on the merged fix(plugin-surrealdb): BETWEEN, REGEX and IS EMPTY filters become equality checks #3218 branch, including the common SurrealDB helper fix.