Skip to content

fix(plugin-weaviate): exports fail past 10,000 objects and the raw filter row is sent as a property - #3217

Open
datlechin wants to merge 4 commits into
mainfrom
fix/weaviate-export-paging
Open

datlechin wants to merge 4 commits into
mainfrom
fix/weaviate-export-paging

Conversation

@datlechin

@datlechin datlechin commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Weaviate exports failed after the first 10,000 objects of a collection. The export paged GET /v1/objects by offset, and Weaviate answers HTTP 422 query maximum results exceeded once offset + limit passes QUERY_MAXIMUM_RESULTS (10,000 by default). The export now reads the plain first page and then pages by the after uuid cursor, which the cap does not apply to, and it stops with a clear error on a server older than 1.18, which ignores after.
  • The Raw Filter row was sent to Weaviate as a filter on a property named __RAW__, and the server rejects that filter. The Weaviate filter builder had no __RAW__ branch, so it treated the row as an ordinary text property. It now refuses the row with a message that points to a GraphQL where in the editor.

Tests

swift test --package-path Packages/TableProCore --filter TableProWeaviateCoreTests: 88 of 88 pass.

WeaviateObjectPagesTests uses a stub transport that behaves the way Weaviate 1.32.4 was measured to behave: it answers 422 when offset + limit > 10000, returns only the uuids strictly after after, and refuses after combined with an offset.

  • readsPastQueryMaximum: 12,000 objects drain to 12,000 uuids, in order. Fails without the fix with .api(status: 422, message: "msg:offset or limit code:400 err:query maximum results exceeded").
  • pagesByCursor: pages of 500, 500 and 200. after is nil, then each page's last uuid, and no request sends offset. Fails without the fix.
  • refusesACursorThatDoesNotMove: a server that ignores after, as 1.17 does, fails after two requests instead of looping. Fails without the fix.
  • refusesAPageWithoutACursor: a full page whose last object has no id fails after one request. New with the fix, since the offset walk had no cursor to lose.
  • firstPageKeepsTheNilUUID, endsOnEmptyPage, omitsVectorWhenAsked: these guard behavior the offset walk already had, so they pass with or without the fix.

WeaviateFilterOperatorTests:

  • rawFilterRowIsRefused: a raw row alone, and a raw row next to a real filter, both throw rawFilterUnsupported. Fails without the fix: no error is thrown.
  • rawFilterRefusalNamesTheEditor: the refusal message. Fails without the fix.

Also checked: WeaviateDriverPlugin builds, StringCatalogIntegrityTests passes, and swiftlint lint --strict reports 0 violations on the changed files.

Docs

The docs rewrite branch covers the user-facing text: exporting a whole collection needs Weaviate 1.18 or later, and the Raw Filter row is not available for Weaviate.

CI fixes

The Package Tests job failed in SyncRecordMapperTests.unknownWireValueFailsClosed: an unknown Safe Mode value decoded as Off instead of Confirm Writes. The sync mapper now uses the model's SafeModeLevel(wireValue:isReadOnly:) decoder. Unknown values require confirmation, legacy read-only restrictions remain, and renaming a connection preserves its unknown wire value. Added regression tests for read-only restrictions and renames.

Integrated main and fixed another inherited build error: SurrealValue.text(_:) called jsonText(length), but main had replaced that helper with a property. Restored the length-aware helper, keeping JSONTruncation for grid display and full JSON for exports. Existing long-structure export and grid-display tests cover both paths.

Follow-up validation on the final merged branch:

  • swift test --package-path Packages/TableProCore: PASS, all Swift Testing and XCTest runs passed.
  • All SurrealDB suites in an isolated SwiftPM harness: PASS. The harness compiles the actual plugin source files, the existing SurrealDBDriverTests.swift suites, and the host's TableFilter model where needed. It adapts imports for the separate test module; it does not run the app's Xcode test host.
  • SwiftLint on changed Swift files: zero violations.
  • Project generation and git diff --check: PASS.
  • verify.sh plugins (AllPlugins): PASS on the merged fix(plugin-surrealdb): BETWEEN, REGEX and IS EMPTY filters become equality checks #3218 branch, including the common SurrealDB helper fix.
  • The lint helper's separate documentation check reports four existing stale paths in CLAUDE.md and the verification guide, which this follow-up leaves unchanged.
  • Reviewed the final diff directly.
  • GitHub CI reruns pending after push.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant