v0.8.57: knowledgebase connector improvements, distributed oauth refresh - #8180
Merged
Merged
Conversation
…hes (#8171) * fix(oauth): scope unauthorized_client as terminal to Atlassian refreshes * fix(oauth): look up provider terminal codes in a Map
…l call's delegation (#8174)
… removing a large source cannot time out (#8165) * fix(knowledge): release kept connector documents in the background so removing a large source cannot time out * fix(knowledge): reserve kept connector storage at removal and settle it when the release finishes * fix(knowledge): carry detach reservations through payer moves and refuse credentials to removed connectors * test(workspaces): add knowledge_connector to the payer-move lock-order fixture schema
…ojection fan-out (#8169) * fix(knowledge): refresh unchanged mirrored ACLs without firing the projection fan-out An admin-mode sync rewrote the ACL of every listed document on every page, even when it had not changed. Assigning `acl` fires the document trigger that copies it onto every chunk's search projection rows, and a projection row the backfill has not filled yet is distinct from every ACL, so each unchanged document still rewrote all of its chunks, each one a vector index insert. A single ACL batch could then exceed the statement timeout and fail the whole sync. Unchanged ACLs now only refresh their evidence timestamp, which never assigns `acl`. Documents whose ACL or restrictions actually changed are written in smaller batches, since each one fans out to all of its chunks. * test(knowledge): run the ACL write Postgres test in CI and cover unresolved evidence and multi-batch changes
…jection chunks (#8175) * fix(knowledge): stop document ACL changes from rewriting unfilled projection chunks The document trigger copied a changed ACL onto every enabled chunk whose projected ACL differed, and a chunk the projection backfill has not filled yet (acl IS NULL) always differs. Each such write re-inserts the row into the vector index, so one document ACL change inside a sync or member rewrite could rewrite every unfilled chunk of that document in the writer's own statement. Unfilled chunks now keep a NULL ACL: search already decides them on their document, and the backfill copies the document's current values under a share lock. A document that moves to another source still carries the source onto them, since source filters read it from the row. The new body ships as script migration 0023, which replaces the function in place; 0022 installs the same body on a fresh database. * test(knowledge): expect the projection ACL trigger migration in the upgrade run * chore(ci): create the ACL test database in the projection trigger step itself
…r as out of scope, not listing failures (#8168) * fix(knowledge): treat Google Workspace users without Gmail or Calendar as out of scope, not listing failures Admin-mode Gmail recorded a Directory user without a mailbox, and Calendar recorded a 403 notACalendarUser, as per-user listing failures. Both are standing account properties, so the connector stayed partial, deletion reconciliation never ran, and the scheduler re-probed the same accounts every retry window. - Directory enumeration no longer schedules Gmail users whose mailbox is not set up; the hourly Directory refresh picks them up once provisioned. A partition queued earlier completes with an empty page instead of a failure. - A Calendar 403 whose only reason is notACalendarUser completes the user's partition cleanly and re-probes it no sooner than the Directory refresh (permissions keep their own refresh cadence). Bare forbidden and mixed-reason 403s stay retryable failures. * fix(knowledge): keep a Google Workspace user's visible documents until a missing service outlasts propagation Google applies service and organizational-unit changes within 24 hours, so a missing mailbox or notACalendarUser can be transient for a user whose documents are already indexed, and a mid-listing answer does not prove the whole account lacks the service. - The scheduler skips a service-not-enabled user only on their first provider page, and only when readers see none of their documents or the condition was first observed at least 24 hours ago. Otherwise it is a retained failure, as before, whose first observation is kept in the partition failure; after 24 hours a mid-listing user restarts from their first page. - Permission passes skip on the first page, since a retained failure refreshes nothing. - Gmail Directory enumeration keeps scheduling a user without a mailbox while readers still see their mail; the crawl reports the missing mailbox to the scheduler instead of completing the user. - Visibility is read through doc_acl_gin_idx for the user's token behind an OFFSET 0 fence, bounded by that user's grants. * fix(knowledge): read a user's visible documents from the ACL index and cover the probe in PostgreSQL The visibility probe moves to its own module so it can run against a real database. Planned inline, LIMIT 1 made a sequential scan of the document table look cheaper than doc_acl_gin_idx, because PostgreSQL cannot estimate array overlap. A materialized CTE now reads the user's grants from the index first, bounding the probe by that user's grants. The email goes through userToken, so a mixed-case or padded directory address matches the normalized ACL token. The new PostgreSQL integration test covers fresh, stale and missing permission evidence, another user's grant, another connector, excluded, archived and deleted documents, and email normalization, and runs in the Search progress PostgreSQL CI step.
…document (#8176) * fix(knowledge): read an unfilled keyword candidate's source from its document The Tin keyword page took each candidate's source from its projection row. A row the source and ACL fill has not reached carries a NULL source, so a chunk from a source that needs a live reader proof was never recognized as one: the proof was not resolved, hydration ran without the caller's grants, and the document was hidden from a member who can read it. Until the keyword projection is filled, a page's unfilled rows now take their source from the document, one primary-key read per row of the page after its limit. The excluded-sources filter asks the document the same way, as the vector leg already does, so a denied source's unfilled rows stop taking slots on a rebuilt page. The filled path's statement is unchanged. * test(knowledge): assert the ACL write's fan-out, not unfilled rows the trigger now skips The projection trigger no longer writes an ACL onto chunks the backfill has not filled, so the ACL write tests can no longer see an unchanged write through those rows. They now count the document writes that fire the fan-out trigger, and expect a changed ACL on filled chunks only. * fix(knowledge): read the document only for unfilled rows in the keyword source exclusion
…, not app configuration faults (#8178) * fix(knowledge): unschedule connectors only for credential revocations, not app configuration faults * test(knowledge): mark the revocation fixture as const
…hanged (#8172) * fix(knowledge): bound the members-mode tombstone pass by what a run changed The tombstone phase of the member document lifecycle ran one anti-join over every live document of the connector on every run, and LIMIT could not bound it because almost every document is observed. On a large connector that statement outgrew the statement timeout, rolled back the member's page, and walked the connector toward disabled. Tombstoning is now driven by the documents whose observations the run itself removed (unseen after a complete listing, withdrawn by a change feed, or held by a member being removed), then by a resumable backstop that walks the reconciliation index one page per statement and checks observations only in the UPDATE over that page's live ids. The backstop saves its position on the connector and does a bounded number of pages per run. Resurrection and purge are unchanged. * fix(knowledge): walk the members-mode reconcile by external id and tombstone explicit removals without a completed listing The absence reconcile ordered its resumable walk by source_seen_at, which every member listing rewrites for what it observed. On a connector larger than one run's page budget the walk chased re-stamped documents and a pass never ended, so a document that lost its observers behind the cursor was never revisited. The walk now keys on external id through doc_connector_external_id_idx, which a document never changes, so a pass completes within ceil(documents / budget) runs. Documents a run explicitly unobserved (a complete listing, a change-feed withdrawal, or a member removal) are now tombstoned even when no member with a completed listing remains, as the stale-member sweep already does. Removing the only listed member previously left every document only it observed live and unobserved indefinitely. The reconcile and the purge stay gated on a completed listing. * fix(knowledge): keep member removal ids across interrupted runs and hide the reconcile cursor - A member removal interrupted by the run deadline resumed from its saved document cursor and only handed the lifecycle the documents above it. The resumed walk now re-reads the removed member's observations below the cursor first. They still exist until the member row is deleted. - The absence reconcile writes its cursor once per run instead of once per page, and documents how many runs a pass takes. - Connector presenters drop member_tombstone_cursor alongside the stored API key, so the internal cursor never reaches API responses. - CI runs member-document-lifecycle.integration.ts with the other knowledge PostgreSQL suites. * chore(db): drop the tombstone cursor migration to regenerate it after staging * chore(db): regenerate the tombstone cursor migration on top of staging * chore(db): format the tombstone cursor migration metadata * fix(knowledge): tombstone a removed member's sole documents with the removal itself A membership removal handed the documents its member alone observed to the lifecycle through run memory. If the run stopped after the member row was deleted, the ids were gone, the observations had cascaded away, and with no member left with a completed listing the absence reconcile never ran. Those documents stayed live indefinitely. Each removal page now tombstones the documents no other member observes in the same transaction that advances the removal's checkpoint, so nothing depends on the run surviving to its lifecycle. This applies only where observations decide existence; a corpus owned by a dedicated content credential is left alone, as before. The in-memory re-collection added earlier is removed. The absence reconcile's external_id IS NOT NULL filter is documented. Every writer that sets connector_id copies a required external id, so the filter excludes nothing. * fix(knowledge): resurrect a withdrawn removal's tombstones with the member's restored ACLs A multi-run member removal tombstones, page by page, the documents only that member observes. If directory re-listing withdrew the removal mid-walk, the restarted walk restored the member's token to those ACLs, but the documents stayed deleted until the lifecycle's resurrection phase ran. That phase runs only after the member loop finishes before the deadline. A membership walk that is not a removal now applies the lifecycle's own resurrect predicate to each page, in the same transaction. The predicate is shared through one helper and covers: connector-owned, not excluded, not archived, content present, observed. Only where observations decide existence. * test(knowledge): insert the content-less fixture row with its own typed insert
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 74 files
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
Contributor
 |
Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>
* fix(traces): allow explicitly skipping oversized backfill records * fix(traces): support larger execution trace archives
…urged (#8184) * fix(knowledge): settle detach reservations when a knowledge base is purged - Retention purge settles a detached connector's remaining detach_reserved_bytes (same lock order and settlement branches as the detach job) and zeroes it before the knowledge base delete cascades the connector away - Test drain helper picks the oldest pending outbox row - Unfilled-projection fixture writes its Tin row the way the projection trigger does - Knowledge ACL harness also runs the 0021 projection source/ACL postgres test * fix(knowledge): settle a purged base's detach reservations as one net amount * fix(knowledge): settle a purged base's reservations before deleting its documents * test(memory): always change a byte when tampering with a checkpoint's auth tag * fix(knowledge): settle overdrawn reservations before a purge's documents and the rest after * fix(knowledge): pause a connector detach while its knowledge base is deleted
* fix(knowledge): restore GitLab PAT setup and secret references * fix(knowledge): align secret reference styling and caret tracking * fix(knowledge): reuse shared secret reference highlighting
…8186) * fix(desktop): replace a stale staged update with the newest release * fix(desktop): track Squirrel staging so update refreshes never cancel an install * fix(desktop): clear a cancelled update replacement so refreshes resume
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.