Calibrate a model, step by step

Start with the shipped model profile, reproduce one failure, change one setting, and check the actual result.

On this page

Calibration means adapting a particular model’s settings and short instructions to the steps ABot asks it to perform. It is not model training, a replacement system prompt, or a guarantee that a small model will complete every task.

This tutorial follows one model from a working provider connection to a controlled adjustment. Its JSON comes from the public configuration contract. The test task and proposed values are examples for you to evaluate, not a claimed benchmark or a pre-tested model preset.

1. Find the model file you are actually using

In Models, select your model. The editor shows its configuration file path. Start from the generated file and keep a backup before changing it. Do not overwrite a working profile with the placeholder example below.

A typical registration in runtime.config.json looks like this:

{
  "models": {
    "profiles": {
      "default": {
        "configRef": "./models/default.config.json"
      }
    }
  }
}

This is a fragment of the runtime config. default is the profile ID; the referenced file contains the actual model settings. configRef resolves relative to the runtime configuration file, not whichever directory your terminal happens to be in.

The published examples/models/default.config.json is the starting scaffold:

{
  "label": "Replace with your model",
  "provider": "provider",
  "model": "replace-with-model-id",
  "contextWindowTokens": 32768,
  "context": {
    "formatTokenAccounting": {
      "mode": "estimate"
    }
  },
  "calibration": {
    "supervisor.decision": { "format": "json" },
    "worker.decision": { "format": "json" },
    "capability.controls": { "format": "json" },
    "planner.decision": { "format": "json" },
    "planner.graph": { "format": "json" },
    "reviewer.decision": { "format": "json" },
    "degraded.finalization": { "format": "json" },
    "execution.decision": { "format": "json" },
    "auditor.decision": { "format": "json" },
    "context.compact": { "format": "json" },
    "toolPayload.raw": {
      "instructions": [
        "Return only the literal payload text requested for the selected tool.",
        "Do not include JSON wrappers, tool names, params, status text, markdown fences, or commentary."
      ]
    }
  }
}

Read the first fields before touching calibration:

Field	What to put here
`label`	A recognizable name shown in the UI. It does not choose the provider model.
`provider`	An existing key under `models.providers`, such as `ollama` in your installation.
`model`	The exact provider model ID, including the appropriate local tag when relevant.
`contextWindowTokens`	The physical context capacity for that concrete model—not a desired answer length.
`context.formatTokenAccounting`	How structured-format overhead is accounted for. The generic scaffold uses `estimate`; Ollama onboarding can generate `none`. Keep your generated value unless you understand the transport.
`calibration`	Overrides for individual semantic steps, rather than one setting imposed on every call.

The example’s 32768 is not a universal requirement or a capacity upgrade. Replace it when your actual model needs a different declaration. Confirm that ordinary conversation works before continuing; unreachable providers and missing credentials are connection problems, not calibration problems.

2. Choose a route before tuning a step

Use Execution route in the model editor. Supervisor divides work between focused roles; Execution Agent gives the main agent direct tool authority and optional planning/audit advice. Both can use local or hosted models.

For this walkthrough, suppose you chose Supervisor for a local model. This optional fragment goes at the top level of its model file:

{
  "execution": {
    "policy": "supervisor-worker-v1"
  }
}

Execution Agent uses execution-agent-v1. Route selection changes which contracts the model receives; it is not a temperature setting and is not automatically switched mid-request. Keep the route fixed while evaluating a calibration change.

3. Make a baseline you can inspect

Use a small, non-sensitive test folder. Manually create calibration-input.txt with:

Project: Cedar
Owner: Maya
Open tasks: 3
Deadline: not decided

Choose this folder as the conversation’s working folder, ensure the file-reading and file-writing tools are available, and ask:

Read calibration-input.txt. Create calibration-summary.md with one
bullet each for the project, owner, open-task count, and deadline.
Keep unknown values unknown. Do not modify the source file.
Then tell me the output filename and what you verified.

This task deliberately exercises reading, a tool payload, writing, and a final response. It is not guaranteed to call every role. Use a fresh output filename for repeats, or explicitly decide how to handle an existing result; do not accidentally turn the test into an overwrite experiment.

Record the selected profile, route, reasoning mode, prompt, observed error, and output. Open the saved Markdown yourself. Success means the four facts match the source and the reported write actually happened—not just that the final answer sounds confident.

4. Locate the failure before choosing a knob

Look for model.invocation.resolved in the runtime diagnostics. It records the resolved profile/provider/model and modelStep. If necessary, inspect the gateway trace, correlate the request and step, then pair provider.request with provider.response by invocationId.

For the standard local layout, the public troubleshooting guide uses:

tail -n 100 .runtime/shared/logs/model-gateway.jsonl

Use MODEL_GATEWAY_TRACE_FILE instead if your deployment sets it. Logs can contain task content: inspect locally and remove secrets or personal data before sharing excerpts.

Observation	Investigate first
No provider response or connection error	Endpoint, service availability, credentials, then latency.
Invalid decision JSON	The reported decision step, structured format, and sampling.
Correct decision, wrong tool inputs	`capability.controls` and the actual selected operation.
Extra prose inside a file payload	`tool_payload.raw`, mapped to the `toolPayload.raw` calibration slot.
Correct artifact, confusing final answer	`supervisor.response` or `execution.response`, not the Worker decision.
Context admission or truncated output	Real model capacity, provider allocation, and observed output limits.

Do not diagnose a step from the last sentence in chat alone. The error may have happened earlier, or the tool may have failed for a reason unrelated to model output.

5. Change one decision setting

Suppose your baseline shows malformed output at worker.decision. This is an illustrative failure, not a report of a test run performed for this guide.

The starter contains this entry inside calibration:

{
  "worker.decision": {
    "format": "json"
  }
}

Replace only that entry, preserving every other calibration slot:

{
  "worker.decision": {
    "format": "json",
    "generation": {
      "temperature": 0.15
    }
  }
}

0.15 is a candidate low-sampling setting for this experiment, not a recommended value for every model. A slot overrides the model’s base generation settings only for calls routed to that slot. It does not flatten the whole model’s writing style. Omitted settings continue to inherit from the base profile.

In the UI, expand Invocation calibration → worker.decision and edit Temperature, then save the model. Use Apply & restart when the Runtime is idle to activate saved settings. This applies saved settings across environments; finish other active work first. Saving a file alone is not proof that active requests are using it.

Repeat the same test and compare the actual result. If nothing improves, revert this change before trying another explanation. For an Execution Agent failure, investigate execution.decision; changing worker.decision will not tune its main decision call.

6. Add a short instruction only for a specific problem

If the remaining failure is extra prose around the structured decision, an additional experiment is:

{
  "worker.decision": {
    "format": "json",
    "generation": {
      "temperature": 0.15
    },
    "instructions": [
      "Return only the decision object required by the supplied schema.",
      "Do not surround the object with Markdown fences or explanatory text."
    ]
  }
}

Again, this replaces just the matching entry under the model’s calibration. Retest separately from the temperature change so you can identify which adjustment helped. Do not add these lines when the observed problem is actually missing evidence, an invalid path, or unavailable tools.

instructions is a JSON array of short strings for this model and selected slot. Edit it in Raw JSON, keeping the whole file valid, and use Save file before applying the saved configuration. The provider adapters include applicable calibration guidance as system-level instruction content. That does not give the text authority to invent actions, bypass approval, or replace the runtime’s schema. Keep format: "json" for structured decision slots; do not copy it onto raw-text authoring just to make every output look consistent.

The shipped toolPayload.raw instructions show a different requirement: literal payload text with no wrappers. They are not a generic “always return JSON” prompt.

7. Use instructionRefs for a reusable methodology

A model-specific correction belongs in calibration.instructions. A reusable step methodology belongs in Markdown referenced by steps.<step>.instructionRefs in request-runner.config.json.

For example, create methodologies/result-checklist.md beside that request-runner file:

Lead with the observed result.
Distinguish completed actions from suggested next steps.
Mention an output filename only when the supplied evidence supports it.
If verification was not performed, say so briefly.

Append the path to the relevant existing response step. For the shipped layout, this fragment preserves its two standard references and adds the tutorial file:

{
  "steps": {
    "supervisor.response": {
      "instructionRefs": [
        "../methodologies/response-ux.md",
        "../methodologies/memory-informed-response.md",
        "./methodologies/result-checklist.md"
      ]
    }
  }
}

Merge this into the existing request-runner steps object; do not replace the entire file. Paths resolve relative to that file. Preserve your installation’s actual existing references if they differ. Add the reference to execution.response as well if you want the same presentation guidance on the EA route.

This guidance is appended to the runtime-owned system message for that step. It changes the requested presentation, not the available actions or evidence. It cannot cause a missing file to exist.

Do not assume a response calibration entry will apply just because it has the right name. The shipped runner explicitly maps supervisor.response, worker.result, and execution.response to the base default profile, while tool_payload.raw maps to toolPayload.raw. Schema version 2 resolves other omitted mappings by matching the step ID. Inspect existing mappings before changing advanced routing.

8. Understand the remaining controls

Setting	Meaning and common mistake
`generation.temperature`	Sampling control. Tune it per step when supported; low values are not proof of correctness.
`generation.topP`	Another sampling control. Do not change it alongside temperature in the same comparison.
`generation.reasoningEffort`	Requested reasoning effort: `none`, `minimal`, `low`, `medium`, `high`, or `xhigh`. Actual support depends on the model/provider.
`supportsThinking`	Enables ABot’s reasoning controls for the profile. Unchecked means provider-default behavior, not “thinking disabled.”
`options`	Provider-specific options. An Ollama option is not automatically valid for OpenAI.
`contextWindowTokens`	Physical capacity for the profile, shared across calibrated steps. A calibration overlay cannot enlarge it.
`stepDefaults.timeoutMs` / step `timeoutMs`	Request-runner latency limits. Longer timeouts do not improve reasoning.
`calibration.<slot>.timeoutMs`	Optional model-specific timeout for that mapped slot. It overrides the request-runner timeout for that invocation.

With active reasoning on a supported OpenAI profile, ABot omits temperature and top-P from the provider payload. Verify the resolved call before concluding that a sampling edit had an effect. Explicit request reasoning overrides also take precedence over profile reasoning settings.

For Ollama, options.num_ctx may select a lower stable provider allocation; effective admission uses the lower allocation/capacity. Increasing the JSON declaration does not create more RAM. Provider-native output controls also differ from context headroom: outputReserveTokens reserves input space and is not an output-token cap. Prefer the default output handling until traces show why an override is needed.

9. Keep only changes that survive retesting

Repeat the baseline with a fresh conversation and equivalent inputs. Then test nearby variations: a different owner, an unknown task count, a missing source file, and an unavailable write tool. Expected behavior includes reporting inability honestly, not just succeeding on the happy path.

Keep a small comparison record: profile and route, one changed field, test prompt, actual artifact, error if any, and elapsed time if useful. Do not treat one successful run as universal QA. Retain a change only when it improves the observed problem without breaking nearby tasks; otherwise restore the backup.

After calibration, return to ordinary work and monitor real results. More instructions are not automatically better: they consume context and can conflict. The aim is the smallest model-specific adjustment that measurably helps while leaving ABot’s contracts intact.