Observability & Events

Two layers give you visibility into your agents: an in-process event stream from the agent loop, and ActiveSupport::Notifications events from ask-instrumentation that any tool can subscribe to.

gem "ask-instrumentation"

Agent events

Every session publishes lifecycle events as it runs. Subscribe with on_event, or filter by event class with on:

session = Ask::Agent::Session.new(model: "deepseek-v4-flash")

session.on_event do |event|
  case event
  when Ask::Agent::Events::TextDelta
    print event.content                 # stream text to the user
  when Ask::Agent::Events::ThinkingDelta
    print event.content                 # reasoning tokens, if the model emits them
  when Ask::Agent::Events::ToolExecutionStart
    puts "\n[Running #{event.name}...]"
  when Ask::Agent::Events::ToolExecutionEnd
    puts "\n[#{event.name} finished in #{event.duration_ms}ms]"
  when Ask::Agent::Events::Error
    puts "Error: #{event.error}"
  end
end

session.run("Run `ruby -v` and answer with only the version string.")

Available events

Event	Fired When	Data
`SessionStart`	The session begins	—
`SessionEnd`	The session finishes	`result`, `turn_count`, `tool_calls_made`, `input_tokens`, `output_tokens`, `cost`
`TurnStart` / `TurnEnd`	Each agent turn	`turn_number`, `tool_results`, tokens, `cost`
`MessageStart` / `MessageEnd`	Each LLM call within a turn	`tool_calls`
`TextDelta`	A text chunk streams	`content`
`ThinkingDelta`	A reasoning chunk streams	`content`
`ToolCallDelta`	Tool call arguments stream	`name`, `arguments`, `id`
`ToolExecutionStart` / `ToolExecutionUpdate` / `ToolExecutionEnd`	A tool runs	`name`, `id`, `arguments`, `partial_result`, `result`, `is_error`, `duration_ms`
`CompactionStart` / `CompactionEnd`	Context is compacted	`tokens_before`, `tokens_after`, `summary`
`LoopDetected`	The agent repeats itself	`tool_name`, `repeated_count`
`MaxTurnsExceeded`	The turn limit is hit	`max_turns`
`EvaluationStart` / `EvaluationDelta` / `EvaluationEnd` / `EvaluationBlocked`	The evaluator runs	`dimensions`, `decision`, `feedback`, `scores`
`ReflectionStart` / `ReflectionDelta` / `ReflectionEnd`	The reflector runs	`reflection_number`, `decision`, `feedback`
`Error`	An error occurs	`error`, `recoverable`

Cost and token tracking

Sessions accumulate usage as they run. These are real numbers from the provider responses, not estimates:

session.run("Write a Ruby method that computes factorials")
session.total_input_tokens   # => 150
session.total_output_tokens  # => 320
session.total_cost           # => 0.0015

The same numbers ride on TurnEnd and SessionEnd events, so you can log or charge per turn:

session.on(Ask::Agent::Events::TurnEnd) do |event|
  Rails.logger.info "Turn #{event.turn_number}: #{event.input_tokens} in / #{event.output_tokens} out / $#{event.cost}"
end

Instrumentation events

ask-instrumentation wraps ActiveSupport::Notifications and emits one event per LLM operation, named {operation}.ask:

Event	Fired When
`chat.ask`	A chat completion finishes
`chat.stream.ask`	A streaming chat completes
`tool.ask`	A tool executes
`tool_call.ask` / `tool_result.ask`	Tool call and result round-trip
`embedding.ask`	Embeddings are generated
`image.ask`	An image is generated

Subscribe from anywhere — a Rails initializer, a background job, a plain Ruby script:

Ask::Instrumentation.subscribe("chat.ask") do |event|
  Rails.logger.info "LLM call: #{event.payload[:provider]} #{event.payload[:model]} " \
                    "#{event.duration}ms cost=$#{event.payload[:cost]}"
end

Attach metadata that flows through every event in a block:

Ask::Instrumentation.with_metadata(user_id: current_user.id, session_id: session.id) do
  response = session.run("Summarize this article")
end

Instrument your own code the same way:

Ask::Instrumentation.instrument("chat.ask", provider: "openai", model: "deepseek-v4-flash") do
  provider.chat(messages, model: "deepseek-v4-flash")
end

The ask-monitoring Rails engine subscribes to these events for its dashboard, ask-opentelemetry turns them into spans, and ask-observability turns them into Prometheus metrics. They all work with any provider.

Prometheus metrics with ask-observability

ask-observability is the infra-observability twin of the dashboard — where the dashboard answers “what is the app spending?” inside the product, metrics answer “is the service healthy?” in Prometheus, OpenObserve, or Grafana.

gem "ask-observability"

require "ask/observability"

Ask::Observability.install  # in plain Ruby; the Rails railtie does it for you

Every instrumentation event then maintains:

ask_llm_calls_total{provider,model,kind}
ask_llm_tokens_total{provider,model,kind,direction}
ask_llm_duration_seconds{provider,model,kind}
ask_llm_errors_total{provider,kind}

Rails auto-mounts /metrics (tune with metrics_path), bootstraps OpenTelemetry export to the OTLP endpoint, and switches logs to JSON. Run rails generate ask:observability:install for a config file.

For the full setup — configuration, with_context correlation, and how the four gems compose (ask-instrumentation → ask-opentelemetry / ask-observability / ask-monitoring) — see ask-observability on GitHub.

Telemetry

The agent ships a file-backed telemetry log for error tracking. It’s on by default; configure the directory through the session:

telemetry = Ask::Agent::Telemetry.new(dir: "log/ask/")
session = Ask::Agent::Session.new(model: "deepseek-v4-flash", telemetry: telemetry)

Every error and notable lifecycle event is appended as JSON lines. The MetaAgent component reads this same log to propose improvements (see The Agent Loop).