ask-agent

Agent runtime for the ask-rb ecosystem. The core agent loop: think → call tools → execute → feed back → repeat.

Use ask-agent when you want to add AI capabilities to your app for your users — chatbots, automated workflows, coding assistants. Bring your own tools, persistence, and UI. Works in any Ruby app.

Use ask-rails-harness when you want to give AI agents access to your Rails app — internal admin tools, ops dashboards, dev assistants. Ships with Rails-aware tools (database, filesystem, logs) and an admin chat UI at /ask. Rails 7.1+ only.

Use ask-rails when you want to add AI capabilities to your Rails app for your users. Provides generators, file conventions, and a railtie for building agents with the ask-rb ecosystem. Rails 7.1+ only.

Ported from RubyLLM::Conductor to Ask::Agent namespace.

Installation

gem "ask-agent"

Components

Component	Purpose
`Session`	Full agent loop — message → tool calls → results → follow-up
`Loop`	Turn management with loop detection and max-turn guard
`ToolExecutor`	Parallel/sequential tool execution with retry and abort
`Compactor`	Context window management (proactive + overflow)
`Hooks`	Before/after tool lifecycle callbacks
`Events`	Streaming events for monitoring
`Telemetry`	File-backed telemetry for error tracking
`Reflector`	Assistant response self-evaluation
`Evaluator`	Independent response evaluation with structured rubric, separate model, isolated context
`MetaAgent`	Self-improvement from telemetry analysis

Quick Start

session = Ask::Agent::Session.new(
  model: "gpt-4o",
  tools: [Ask::Tools::Shell::Bash]
)

response = session.run("List the current directory")
puts response

If a model is registered under one provider but served by another — for example, deepseek-v4-flash served by opencode_go — pass the provider: override:

session = Ask::Agent::Session.new(
  model: "deepseek-v4-flash",
  provider: :opencode_go,
  tools: [Ask::Tools::Shell::Bash]
)

The provider: parameter tells the agent which provider to use, regardless of which provider the model is registered under in the catalog.

Cost & Token Tracking

Every session tracks cumulative token usage and cost:

session.run("Write a poem")
session.total_input_tokens   # => 150
session.total_output_tokens  # => 320
session.total_cost           # => 0.0015

Turn and session events carry the same data:

session.on(Ask::Agent::Events::TurnEnd) do |event|
  puts "Turn #{event.turn_number}: #{event.input_tokens} in / #{event.output_tokens} out / $#{event.cost}"
end

session.on(Ask::Agent::Events::SessionEnd) do |event|
  puts "Session total: #{event.input_tokens} in / #{event.output_tokens} out / $#{event.cost}"
end

Instrumentation

The agent emits ActiveSupport::Notifications events via ask-instrumentation:

chat.ask — on each completion (model, provider, tokens, cost)
chat.stream.ask — on each streaming completion

Subscribe from anywhere in your app:

Ask::Instrumentation.subscribe("chat.ask") do |event|
  Rails.logger.info "LLM call: #{event.payload[:model]} cost=$#{event.payload[:cost]}"
end

The ask-monitoring Rails engine hooks into these automatically for its dashboard.

Audit Log

The agent can log all lifecycle events (tool calls, errors, token usage, turns) to a configurable audit log. This is useful for debugging, compliance, and visibility into what your agents are doing.

Configuration

# Global — all sessions use this adapter
Ask::Agent.configure do |c|
  c.audit_log = { adapter: :active_record }
end

# Per-session override
session = Ask::Agent::Session.new(
  model: "gpt-4o",
  audit_log: { adapter: :file, path: "tmp/my_agent_audit.jsonl" }
)

Built-in Adapters

Adapter	Description
`:active_record`	Writes to an `ask_audit_logs` table. Auto-creates the table on first write. For Rails: run `rails generate ask_rails:install` for a proper migration.
`:file`	Appends JSON lines to a file. Good for development.
Custom	Any object implementing `#write(entry)` — see `Ask::Agent::Extensions::AuditLog::Adapter`.

Logged Events

Event	Stored Data
`session_start`	—
`session_end`	turn_count, tool_calls_made, input_tokens, output_tokens, cost
`turn_end`	turn_number, tool_results_count, tokens, cost
`tool_execution_start`	name, args (sensitive fields redacted)
`tool_execution_end`	name, id, duration_ms, is_error, result
`error`	message, recoverable
`max_turns_exceeded`	max_turns
`loop_detected`	tool_name, repeated_count
`compaction_end`	tokens_before, tokens_after
`evaluation_blocked`	feedback, scores

Sensitive arguments (password, token, api_key, sql, command) are redacted automatically.

Migration

If you’re using Rails, generate the audit log migration:

rails generate ask_rails:install

This creates db/migrate/create_ask_audit_logs.rb with the correct table schema.

Evaluator

New in ask-agent 0.15.0

Independent response evaluation with generator/evaluator separation. The evaluator uses a separate model (different from the session’s model) and an isolated context to judge the agent’s output — preventing the anti-pattern of a model grading its own work.

This is different from Reflector (which has the same model evaluate its own output). The Evaluator uses an independent model and a structured rubric, so the agent that writes the code is never the one that checks it.

Quick Start

session = Ask::Agent::Session.new(
  model: "gpt-4o",
  evaluator: { model: "claude-sonnet-4", goal: "Write an email validator" }
)
session.run("Write a function that validates email addresses")

Verdicts

Verdict	What Happens
`:accept`	Output passes — falls through to the reflector for backward compatibility
`:revise`	Evaluator provides actionable feedback; session runs another turn with feedback injected into system context
`:block`	Output is fundamentally wrong — session returns a blocked message and emits `EvaluationBlocked`

Configuration

# Set a global default evaluator model
Ask::Agent.configure do |c|
  c.default_evaluator_model = "claude-sonnet-4"
end

# Then enable with the default
session = Ask::Agent::Session.new(model: "gpt-4o", evaluator: true)

# Or pass nothing (default) — no evaluation, backward compatible
session = Ask::Agent::Session.new(model: "gpt-4o")

Custom Rubric

The default rubric evaluates five dimensions (correctness, completeness, verification, scope, clarity). Pass a custom rubric for domain-specific evaluation:

evaluator = Ask::Agent::Evaluator.new(
  model: "claude-sonnet-4",
  rubric: [
    Ask::Agent::Evaluator::Dimension.new(
      name: "performance",
      description: "Is the implementation efficient?",
      weight: 2
    )
  ]
)

result = evaluator.evaluate(
  goal: "Write an email validator",
  response: agent_output
)

result.accept?  # => true
result.revise?  # => false
result.block?   # => false
result.scores   # => { performance: 2 }
result.feedback # => "" or "Add unicode character handling"

Events

The evaluator emits its own events during evaluation:

session.on_event do |event|
  case event
  when Ask::Agent::Events::EvaluationStart
    puts "Evaluating against: #{event.dimensions.join(', ')}"
  when Ask::Agent::Events::EvaluationDelta
    print event.content
  when Ask::Agent::Events::EvaluationEnd
    puts "Decision: #{event.decision}"
    puts "Scores: #{event.scores}"
  when Ask::Agent::Events::EvaluationBlocked
    puts "Blocked: #{event.feedback}"
  end
end

Rate-Limit Handling

When a provider returns RateLimitError, the agent retries up to 3 times with exponential backoff. If the provider includes a Retry-After header, that value is used instead. No configuration needed.

Prompt Caching

Prompt caching saves up to 90% on input token costs for repeated conversation prefixes. It works directly through provider-native caching APIs — no proxy server needed.

Enabled by default. All sessions automatically send cache-control hints to supporting providers:

Anthropic — Caches system prompt and last user message context. Response metadata includes cache_creation_input_tokens and cache_read_input_tokens.
OpenAI — Automatic for prompts exceeding ~1024 tokens. Response metadata includes cached_tokens.

Providers that don’t support caching (Google, Mistral, Ollama, etc.) safely ignore the parameter.

# Disable if needed
Ask::Agent.configure do |c|
  c.prompt_caching = false
end

Persistence (State)

By default sessions run entirely in memory. Pass a state: adapter to persist conversations across restarts — every turn is saved immediately, so a crash mid-conversation doesn’t lose progress.

Quick Start

require "ask-state-providers"

store = Ask::State::Providers::SQLite.new  # or Redis, Postgres, MySQL

session = Ask::Agent::Session.new(
  model: "gpt-4o",
  tools: [Ask::Tools::Shell::Bash],
  state: store
)

session.run("Investigate the error")
# Every turn is persisted automatically to the store

Backends

Any Ask::State::Adapter works. The ask-state-providers gem ships four:

Backend	Class	Best For
SQLite	`Ask::State::Providers::SQLite`	CLI tools, single-user agents, local dev
Redis	`Ask::State::Providers::Redis`	Distributed multi-process deployments
PostgreSQL	`Ask::State::Providers::Postgres`	Rails apps with an existing DB pool
MySQL	`Ask::State::Providers::MySQL`	Teams already running MySQL

# SQLite — zero config
store = Ask::State::Providers::SQLite.new

# Redis — for distributed setups
store = Ask::State::Providers::Redis.new(url: ENV["REDIS_URL"])

# PostgreSQL — reuse your existing connection
store = Ask::State::Providers::Postgres.new(url: ENV["DATABASE_URL"])

Save & Resume

session = Ask::Agent::Session.new(model: "gpt-4o", state: store)
session.run("Analyze the logs")
session_id = session.id  # save this somewhere

# Later, in a different process or after a restart:
restored = Ask::Agent::Session.load(session_id, adapter: store)
restored.run("What else should I check?")  # picks up where it left off

A session is persisted after every LLM turn and all its metadata (model, tools, turn count, token usage) is preserved across loads.

Backward Compatibility

The old persistence: keyword still works but is deprecated:

session = Ask::Agent::Session.new(model: "gpt-4o", persistence: store)

Prefer state: — it’s shorter and matches the Ask::State::Adapter naming.

Custom Adapter

Any object responding to get(key), set(key, value), and delete(key) works as a state adapter. This makes it trivial to persist to any backend:

class RedisAdapter
  def get(key)  = redis.get(key).then { |v| v ? JSON.parse(v) : nil }
  def set(key, value, ttl: nil) = redis.set(key, JSON.generate(value), ex: ttl)
  def delete(key) = redis.del(key)
end

session = Ask::Agent::Session.new(model: "gpt-4o", state: RedisAdapter.new)

Events

session.on_event do |event|
  case event
  when Ask::Agent::Events::TextDelta
    print event.content
  when Ask::Agent::Events::ToolExecutionStart
    puts "Running #{event.name}..."
  end
end

Sub-Agent Delegation

New in ask-agent 0.19.0

Delegate sub-tasks to specialized sub-agents using Ask::Agent::SubAgent. The coordinator agent sees it as a regular tool — when called, a fresh sub-agent session runs independently with its own model, tools, and instructions.

Ask::Agent::SubAgent satisfies the tool duck type directly (name, description, params_schema, call) — pass it in the tools array just like any other tool. No wrapping or factory needed.

From a filesystem definition

If you already have an agent defined in agents/<name>/agent.rb, reference it by name:

# agents/web_search/agent.rb defines model, tools, instructions
search = Ask::Agent::SubAgent.new("web_search")

coordinator = Ask::Agent::Session.new(
  model: "gpt-4o",
  tools: [search, Ask::Tools::Shell::Bash]
)

This is the same definition convention used by Ask::Agent.new("name").

Inline configuration

search = Ask::Agent::SubAgent.new(
  name: "web_search",
  description: "Search the web for current information",
  model: "gpt-4o-mini",                              # cheaper model for search
  tools: [Ask::Tools::WebSearch],
  system_prompt: "You are a research assistant."
)

review = Ask::Agent::SubAgent.new(
  name: "code_review",
  description: "Review code for bugs, style issues, and security problems",
  model: "claude-sonnet-4",                           # better at code review
  tools: [Ask::Tools::Shell::Read, Ask::Tools::Shell::Grep],
  system_prompt: "You are a senior code reviewer. Be thorough."
)

coordinator = Ask::Agent::Session.new(
  model: "gpt-4o",
  tools: [search, review, Ask::Tools::Shell::Bash]
)

coordinator.run("Find the latest Rails release and check our Gemfile")

With a custom provider

review = Ask::Agent::SubAgent.new(
  name: "code_review",
  model: "claude-sonnet-4",
  provider: :anthropic,
  tools: [Ask::Tools::Shell::Read, Ask::Tools::Shell::Grep],
  system_prompt: "You are a senior code reviewer."
)

What happens at runtime

The coordinator LLM decides to call web_search with task: "Latest Rails version"
A fresh sub-agent session starts with gpt-4o-mini + WebSearchTool
The sub-agent searches, processes results, and returns a concise answer
The coordinator receives this as a normal tool response and continues

Error isolation

If a sub-agent fails (provider outage, rate limit, max turns exceeded), the error returns as a regular tool error message. The coordinator can decide to retry, rephrase, or skip — the main conversation isn’t disrupted.

# Multiple sub-agents, each with distinct identity
tools = [
  Ask::Agent::SubAgent.new(name: "data_analysis", model: "gpt-4o", tools: [Analyzer]),
  Ask::Agent::SubAgent.new(name: "fact_check",    model: "gpt-4o-mini", tools: [WebSearch])
]

Provider-Executed Tools

Some LLM providers offer built-in tools that run on their infrastructure — web search, file search, code execution. These tools don’t need local execution; the provider handles them and returns results directly in the response.

Pass Ask::ProviderTool objects alongside regular tools in the Session constructor:

session = Ask::Agent::Session.new(
  model: "gpt-4o",
  tools: [
    Bash, Read,
    Ask::ProviderTool.web_search(search_context_size: "high"),
    Ask::ProviderTool.file_search(vector_store_ids: ["vs_abc"])
  ]
)

session.run("Search for recent security advisories and check our config")

The agent loop automatically detects provider-executed results and adds them to the conversation without attempting local execution. Regular user-defined tools continue to run locally as before.

Available provider tools:

Factory method	Provider	What it does
`Ask::ProviderTool.web_search`	OpenAI	Search the internet for current information
`Ask::ProviderTool.file_search`	OpenAI	Search through uploaded files in a vector store
`Ask::ProviderTool.code_interpreter`	OpenAI	Execute Python code in a sandboxed environment

Custom provider tools can be created directly:

Ask::ProviderTool.new(
  id: "openai.web_search",
  name: "web_search",
  args: { search_context_size: "high" }
)

Middleware (LLM Call Pipeline)

Middleware wraps every provider.chat(...) call with cross-cutting behavior — retry, logging, default params, and more. Configure globally; applies to all Chat and Session instances automatically.

Ask::Agent.configure do |c|
  c.middleware.use :retry_on_failure, max_retries: 5
  c.middleware.use :log_calls, logger: Rails.logger
  c.middleware.use :default_settings, temperature: 0.7
end

Built-in Middleware

Middleware	Key	What it does
RetryOnFailure	`:retry_on_failure`	Retries on `RateLimitError`, `ServerError`, `ServiceUnavailable` with exponential backoff + jitter. Does not retry on `Unauthorized`, `ModelNotFound`, or `ConfigurationError`. Respects `retry_after` from provider errors.
LogCalls	`:log_calls`	Logs every LLM call: model, tool count, message count, duration, token usage. Custom logger support.
DefaultSettings	`:default_settings`	Injects `temperature`, `max_tokens`, `top_p`, etc. into every provider call. User-supplied values take precedence.

Custom Middleware

class MyMiddleware < Ask::Agent::Middleware::Base
  def around_request(provider, request)
    Rails.logger.info "Calling #{request[:model]} with #{request[:messages].length} messages"
    start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
    result = yield
    elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start
    Rails.logger.info "Completed in #{elapsed.round(3)}s"
    result
  end
end

Ask::Agent.configure { |c| c.middleware.use MyMiddleware }

Stream Transforms

Stream transforms process each raw Ask::Chunk through a chain before yielding ChatChunks to your block. Useful for extracting thinking tokens, buffering text, or parsing streaming JSON.

Ask::Agent.configure do |c|
  c.stream_transforms.use :thinking_separator
  c.stream_transforms.use :text_buffer, min_size: 100
end

Built-in Transforms

Transform	Key	What it does
ThinkingSeparator	`:thinking_separator`	Splits chunks with both `thinking` and visible `content` into two separate chunks, so you can handle reasoning independently.
TextBuffer	`:text_buffer`	Buffers rapid text deltas until they reach `min_size` characters. Reduces UI flicker and log noise. Auto-flushes before non-content chunks and at stream end.
ExtractJson	`:extract_json`	Accumulates the response and attempts JSON parsing. Check `#extracted_json` and `#json?` after the stream completes.

Custom Transform

class FilterTransform < Ask::Agent::StreamTransforms::Base
  def call(chunk, &block)
    block.call(chunk) unless chunk.content == "drop_me"
  end
end

Ask::Agent.configure { |c| c.stream_transforms.use FilterTransform }

Extensions

PermissionGate — Require approval for destructive tools
RateLimiter — Prevent runaway tool calls
AuditLog — Immutable, append-only tool call log

Scheduler (Recurring Agent Runs)

Schedule agents to run on cron schedules or recurring intervals. Tasks run in background threads managed by rufus-scheduler.

Ask::Agent.configure do |c|
  c.scheduler.every "5 minutes", name: "health-check" do
    Ask::Agent::Session.new(model: "gpt-4o").run("Check server health")
  end

  c.scheduler.cron "0 9 * * 1-5", name: "morning-report" do
    session = Ask::Agent::Session.new(model: "gpt-4o")
    session.run("Generate daily report and send to team")
  end
end

# Start the scheduler (background thread)
Ask::Agent::Scheduler.start

# Manage at runtime
Ask::Agent::Scheduler.running?          # => true
Ask::Agent::Scheduler.jobs              # list of scheduled jobs
Ask::Agent::Scheduler.job_by_name("health-check")

# Graceful shutdown
Ask::Agent::Scheduler.stop

Task names are optional but recommended — they let you find and manage jobs at runtime.

Agent Definitions (Convention-Based)

Define reusable agents in agents/<name>/agent.rb or app/agents/<name>/agent.rb. The directory name becomes the agent name. Instructions auto-load from a sibling instructions.md.

agents/
├── health_check/
│   ├── agent.rb           → Definition subclass
│   └── instructions.md    → auto-loaded system prompt
├── daily_report/
│   ├── agent.rb
│   └── instructions.md
└── shared/
    └── tools/             → shared across all agents

# agents/health_check/agent.rb
class HealthCheckAgent < Ask::Agent::Definition
  model "gpt-4o"
  tools :bash, :read, :grep
  schedule "every 5 minutes"
end

Create sessions from definitions:

agent = Ask::Agent.new("health_check")
agent.run("Check server health")

# List all discovered definitions
Ask::Agent.definitions.each do |name, (klass, dir)|
  puts "#{name}: #{klass.model}"
end

CLI

The askr CLI is installed with the gem:

askr list                    # List all agents
askr run health_check        # Run an agent
askr run health_check "..."  # Run with a prompt
askr schedule                # Start the scheduler
askr new deploy_bot          # Scaffold a new agent

Skills

Skills are markdown files with step-by-step methodology that agents can load on demand. They follow the SKILL.md convention (markdown with YAML frontmatter containing name and description).

Where Skills Live

Skills are discovered from these locations (highest priority first):

Location	Scope	Description
`agents/<name>/skills/`	Per-agent	Skills only available to one agent
`agents/shared/skills/`	Project-wide	Shared across all agents
`app/agents/shared/skills/`	Rails project	Rails variant of shared skills
`.agents/skills/`	Legacy	Backward compatibility path
`~/.config/ask/skills/`	User	Personal skills across projects
Installed gems	Global	Skills shipped with ask-* gems
Built-in	Built-in	`skill.design`, `skill.compose`

agents/
├── health_check/
│   ├── agent.rb
│   ├── instructions.md
│   └── skills/
│       └── nginx_debug/SKILL.md    ← only for health_check
├── daily_report/
│   └── agent.rb
└── shared/
    ├── tools/
    └── skills/
        └── rails_debug/SKILL.md    ← for all agents

Skills follow a progressive disclosure pattern: names and descriptions are listed in the system prompt so the model knows they exist, and the full instructions are loaded only when the model calls the load_skill tool or the host calls session.skill(name).

Configuration

```ruby
Ask::Agent.configure do |c|
  c.default_model = "claude-sonnet-4"
  c.default_max_turns = 50
  c.parallel_tool_execution = true
  c.prompt_caching = false    # disable provider-native prompt caching
  c.middleware.use :log_calls
  c.stream_transforms.use :thinking_separator
end
```