You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Several decision points in the .NET runtime are bounded judgments, not text generation: should a loop continue, has the task been completed, which authorized tools or Skills are relevant, which participant acts next, should a result be escalated. A new category of model answers exactly that shape: it evaluates state against typed questions and returns probabilities directly. TypeSafe AI's Jev is the first such model, and it is fast and cheap enough to run on every loop iteration.
Today the only way to make such a judgment is a generative IChatClient judge (AIJudgeLoopEvaluator), which costs a full model call per iteration and flattens the probability into generated JSON. The provider-neutral abstraction proposed for Microsoft.Extensions.AI in dotnet/extensions#7764 is untriaged and absent from the MEAI version this repository references, so waiting on it blocks the loop integration, provider work, and evaluation frameworks that want one shared contract. This PR makes decision-model inference usable now while keeping Core provider-neutral and keeping the eventual move to MEAI a deletion rather than a migration. Design and alternatives are recorded in ADR-0042.
Description & Review Guide
What are the major changes?
Microsoft.Agents.AI.Abstractions/Decisions/: an [Experimental(MAAI001)] reference copy of the [API Proposal]: Add a provider-neutral abstraction for decision-oriented AI models dotnet/extensions#7764 contract, name for name: IDecisionClient, DecisionRequest (JsonElement state, unique question ids), DecisionOptions, BinaryDecisionQuestion / ChoiceDecisionQuestion / ScoreDecisionQuestion, DecisionResponse, the three answer kinds with range validation, DecisionClientMetadata, and a JsonTypeInfo<TState> convenience overload. Two additions beyond the proposal: DecisionClientException with a DecisionFailureKind and IsTransient, and answer validation. Provider limits are not part of the contract.
Microsoft.Agents.AI/Harness/Loop/DecisionLoopEvaluator: a LoopEvaluator that asks one binary question per iteration, applies a completion threshold (default 0.90), continues with deterministic feedback, rejects non-text request content unless a StateFactory is supplied, and handles provider failure through FailureBehavior (Throw by default, Continue, DeferToNextEvaluator) plus a separate TransientFailureBehavior. Optional ILoggerFactory logs one debug line per decision and a warning when a failure policy is applied; the state is never logged. Ordered before AIJudgeLoopEvaluator it forms a cheap-then-strong cascade using existing LoopAgent semantics, with no new abstraction.
Microsoft.Agents.AI.TypeSafe: TypeSafeDecisionClient over TypeSafe's System One API or OpenRouter's relay; strict parsing, HTTP failure classification, no silent retries, API key never in exception messages, feature index 75 registered.
Harness_Step06_DecisionLoop sample (decision-only loop with a cumulative StateFactory, Jev-then-AI-judge cascade, direct mixed batch of all three primitives), Harness README, solution registration, PublicAPI baselines for every target framework, unit tests for the contract consumer and the provider.
What is the impact of these changes?
Additive and opt-in. No existing agent, evaluator, tool, approval, or workflow behavior changes unless a DecisionLoopEvaluator or IDecisionClient is configured. AIJudgeLoopEvaluator is untouched. Core and Abstractions gain no vendor dependency; the TypeSafe transport is a separate preview package following the existing provider-package precedent.
The Abstractions types are a staging ground: when MEAI ships the abstraction they are deleted and consumers retarget the namespace; the shape is the upstream proposal's so that stays mechanical.
Verified live against Jev jev-1.13.0 with an OpenAI-compatible primary model: the cascade finished a three-part task in four iterations with a single generative judge call; a mixed batch returned a binary probability, a choice distribution, and a weighted score in one call.
What do you want reviewers to focus on?
Whether an experimental reference copy of the MEAI proposal is acceptable in Microsoft.Agents.AI.Abstractions (ADR-0042, Decisions 1 and 2), or should live in a satellite package.
The Microsoft.Agents.AI.TypeSafe package as the provider's home versus a provider-owned package.
Security framing in the XML docs and sample README: a second external inference boundary, minimal state projection, probabilities never used for authorization or approval.
Related: #8545 (.NET decision-model integration request, MEAI-gated), #8556 (Python Jev request), dotnet/extensions#7764 (MEAI proposal this contract mirrors). No other open PR addresses these issues.
Contribution Checklist
The code builds clean without any errors or warnings
All unit tests pass, and I have added new tests where possible
This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.
…sion-model inference
Adds a provider-neutral decision-model contract to Microsoft.Agents.AI.Abstractions
as an experimental reference implementation of the shape proposed in
dotnet/extensions#7764: IDecisionClient, DecisionRequest/Options/Response, the
binary, choice and score question and answer kinds, DecisionClientMetadata, a
JsonTypeInfo-based convenience overload, and a classified DecisionClientException
with IsTransient. Answers validate their probability ranges at construction.
ADR-0042 records the decision: MEAI is the long-term owner; MAF hosts a verbatim,
experimental copy until it ships so the loop evaluator, evaluation frameworks and
providers share one contract, and removes it when MEAI lands.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tion
Adds a LoopEvaluator that asks an IDecisionClient one binary question ("has the
agent fully addressed the original request?") against a minimal, source-generated
text projection of the loop state and applies a configurable completion threshold.
It complements AIJudgeLoopEvaluator: a decision model reports a probability, not
prose, so continuation carries deterministic feedback. Ordered before an AI judge
it forms a cheap-then-strong cascade using existing LoopAgent semantics.
Provider failure is never treated as "incomplete": FailureBehavior (Throw by
default, Continue, DeferToNextEvaluator) and a separate TransientFailureBehavior
for rate limits, overload and outages. Non-text request content is rejected
unless a StateFactory is supplied. Optional ILoggerFactory logs one debug line per
decision and a warning when a failure policy is applied, never the state.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n loop sample
Adds TypeSafeDecisionClient, an IDecisionClient over TypeSafe's System One API
(or OpenRouter's relay of it), mapping binary/choice/score questions to
noul/choice/score with strict parsing, HTTP failure classification, no silent
retries and no API key in exception messages. Provider limits (255 choices, 10
levels) are enforced in the transport, not the contract. Allocates feature index
75 (typesafe) in the .NET feature-usage registry.
Adds the Harness_Step06_DecisionLoop sample: a decision-only loop with a
cumulative StateFactory, a Jev-then-AI-judge cascade, and a direct mixed batch of
all three primitives, against OpenAI or any OpenAI-compatible endpoint.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Sample: the chat API key is bound to the chosen endpoint; a credential for
one service is never sent to the other.
- Answers: distributions supplied at construction are validated (non-blank
keys, unit probabilities); docs state the collection stays mutable to match
the proposed MEAI shape.
- TypeSafe client: a blank per-call ModelId override is rejected before sending.
- TypeSafe protocol: choice and score questions are revalidated at call time
(at least two members, unique non-blank names, no nulls) since their
collections are mutable; a selected choice must be one of the requested
choices, not merely present in the returned map; a score outside 0..N-1 is
rejected.
- DecisionLoopEvaluator: the failure warning logs only the exception type,
kind, status and transience, never the exception message.
- DecisionLoopEvaluatorOptions.CompletionQuestion is string? (null restores
the default); PublicAPI baselines updated.
- ADR-0042: deciders set to the .NET Harness/Loop owners (maintainers may adjust).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This prompt never permits the complete recap that pattern 2 depends on: after the three one-question responses, it still says to answer exactly one unanswered question even though none remain. With the evaluator's latest-response-only projection, the cascade can therefore run to MaxIterations instead of demonstrating a successful final verification. Explicitly instruct the agent to recap all answers once no questions remain.
System One accepts state only as a JSON string, object, or array, but this converts Undefined to null and also sends Null, number, and boolean states. Those requests are guaranteed provider validation failures; reject unsupported ValueKinds locally as InvalidRequest instead.
Token counts are integer fields, but this fallback truncates fractional JSON numbers (for example, 12.9 becomes 12) and accepts negative values, producing inaccurate usage/cost accounting. Require a non-negative Int64 and classify anything else as InvalidResponse.
This claims every listed provider limit is rejected before sending, but the implementation validates only choice and score cardinality; it does not enforce the stated state-token limit. Clarify that state-size limits remain provider-enforced so callers are not promised local rejection.
This issue also appears on line 75 of the same file.
The Probabilities collections on choice and score answers stay mutable to match
the proposed MEAI shape, so the ADR no longer claims an out-of-range value
cannot exist as an object; it states what is validated and who owns the
invariant afterwards, matching the API documentation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
westey (@westey-m)Roger Barreto (@rogerbarreto) — you own the Harness/Loop code this builds on, so I'd value your look at this PR together with ADR-0042 (docs/decisions/0042-decision-model-inference.md); the code is the implementation of that ADR.
Pure move, no behavior or public API change: BinaryDecisionCriteria,
DecisionChoice, DecisionScoreLevel, DecisionFailureKind, and the three answer
types get their own files, matching the surrounding folders and the file
layout of the Microsoft.Extensions.AI staging branch so the two trees diff
file for file.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
I think TypeSafe provider is belongs to Microsoft.Extensions.Ai and in MAF we should use only IDecisionClient from MEAI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
documentationUsage: [Issues, PRs], Target: documentation in the code base and learn docs.NETUsage: [Issues, PRs], Target: .Net
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub Actions
Active
agent-framework-automation
Bot
added
documentation
Motivation & Context
Several decision points in the .NET runtime are bounded judgments, not text generation: should a loop continue, has the task been completed, which authorized tools or Skills are relevant, which participant acts next, should a result be escalated. A new category of model answers exactly that shape: it evaluates state against typed questions and returns probabilities directly. TypeSafe AI's Jev is the first such model, and it is fast and cheap enough to run on every loop iteration.
Today the only way to make such a judgment is a generative
IChatClientjudge (AIJudgeLoopEvaluator), which costs a full model call per iteration and flattens the probability into generated JSON. The provider-neutral abstraction proposed forMicrosoft.Extensions.AIin dotnet/extensions#7764 is untriaged and absent from the MEAI version this repository references, so waiting on it blocks the loop integration, provider work, and evaluation frameworks that want one shared contract. This PR makes decision-model inference usable now while keeping Core provider-neutral and keeping the eventual move to MEAI a deletion rather than a migration. Design and alternatives are recorded in ADR-0042.Description & Review Guide
Microsoft.Agents.AI.Abstractions/Decisions/: an[Experimental(MAAI001)]reference copy of the [API Proposal]: Add a provider-neutral abstraction for decision-oriented AI models dotnet/extensions#7764 contract, name for name:IDecisionClient,DecisionRequest(JsonElementstate, unique question ids),DecisionOptions,BinaryDecisionQuestion/ChoiceDecisionQuestion/ScoreDecisionQuestion,DecisionResponse, the three answer kinds with range validation,DecisionClientMetadata, and aJsonTypeInfo<TState>convenience overload. Two additions beyond the proposal:DecisionClientExceptionwith aDecisionFailureKindandIsTransient, and answer validation. Provider limits are not part of the contract.Microsoft.Agents.AI/Harness/Loop/DecisionLoopEvaluator: aLoopEvaluatorthat asks one binary question per iteration, applies a completion threshold (default 0.90), continues with deterministic feedback, rejects non-text request content unless aStateFactoryis supplied, and handles provider failure throughFailureBehavior(Throw by default, Continue, DeferToNextEvaluator) plus a separateTransientFailureBehavior. OptionalILoggerFactorylogs one debug line per decision and a warning when a failure policy is applied; the state is never logged. Ordered beforeAIJudgeLoopEvaluatorit forms a cheap-then-strong cascade using existingLoopAgentsemantics, with no new abstraction.Microsoft.Agents.AI.TypeSafe:TypeSafeDecisionClientover TypeSafe's System One API or OpenRouter's relay; strict parsing, HTTP failure classification, no silent retries, API key never in exception messages, feature index 75 registered.Harness_Step06_DecisionLoopsample (decision-only loop with a cumulativeStateFactory, Jev-then-AI-judge cascade, direct mixed batch of all three primitives), Harness README, solution registration, PublicAPI baselines for every target framework, unit tests for the contract consumer and the provider.DecisionLoopEvaluatororIDecisionClientis configured.AIJudgeLoopEvaluatoris untouched. Core and Abstractions gain no vendor dependency; the TypeSafe transport is a separate preview package following the existing provider-package precedent.jev-1.13.0with an OpenAI-compatible primary model: the cascade finished a three-part task in four iterations with a single generative judge call; a mixed batch returned a binary probability, a choice distribution, and a weighted score in one call.Microsoft.Agents.AI.Abstractions(ADR-0042, Decisions 1 and 2), or should live in a satellite package.FailureBehaviordefaults toThrowlikeAIJudgeLoopEvaluator;TransientFailureBehaviorcovers rate limits, overload and outages (.NET: [Feature]: Integrate Microsoft.Extensions.AI decision-model inference into agent decision points #8545 asked for "continue" as the default).Microsoft.Agents.AI.TypeSafepackage as the provider's home versus a provider-owned package.Related Issue
Fixes #8562
Related: #8545 (.NET decision-model integration request, MEAI-gated), #8556 (Python Jev request), dotnet/extensions#7764 (MEAI proposal this contract mirrors). No other open PR addresses these issues.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.