Agent Playtest

an LLM agent drives a running SimVX game, headless.

📄 Docs only

Tags: ai

AgentSession wraps a scene in a steppable, headless session that an agent observes and acts on. run_agent(session, client, goal=...) hands the model the session’s tools and loops observe, decide, act and step until the model stops talking or max_turns runs out:

• observe reads the game as text: the scene tree, one node’s properties, a search or a snapshot.

• send_input presses keys, clicks, moves the mouse or drives a controller, through the same input path a player’s hardware uses, so the game’s own actions answer.

• set_state sets a Property directly, or pauses and resumes, for setup and probing.

• watch records every emit of a node’s signal without changing the game; each step then lists the emits since the last one, so the agent reads what happened instead of polling for it.

• step advances N frames and reports terminated, with the outcome as reason, once the session’s terminal_fn names one for the root.

dispatch(session, tool, arguments) is the call the loop makes for each tool the model picks, and anything else may make it too: without an endpoint this example drives the same tools from a fixed script. With SIMVX_LLM_BASE_URL, SIMVX_LLM_MODEL and SIMVX_LLM_API_KEY set, OpenAICompatibleClient.from_env() connects to any OpenAI-compatible endpoint and the model plays.

The game is a toy so the loop is easy to follow: holding the score action (Space) adds a point every frame and emits scored, and reaching 5 points wins. The session and its tools are the same for any game.

Source

# /// simvx
# web = { disabled = true, reason = "requires a local or remote LLM endpoint; not available in the browser runtime" }
# ///
"""Agent Playtest: an LLM agent drives a running SimVX game, headless.

`AgentSession` wraps a scene in a steppable, headless session that an agent observes and acts on.
`run_agent(session, client, goal=...)` hands the model the session's tools and loops observe,
decide, act and step until the model stops talking or `max_turns` runs out:

- `observe` reads the game as text: the scene tree, one node's properties, a search or a snapshot.
- `send_input` presses keys, clicks, moves the mouse or drives a controller, through the same
  input path a player's hardware uses, so the game's own actions answer.
- `set_state` sets a `Property` directly, or pauses and resumes, for setup and probing.
- `watch` records every emit of a node's signal without changing the game; each `step` then
  lists the emits since the last one, so the agent reads what happened instead of polling for it.
- `step` advances N frames and reports `terminated`, with the outcome as `reason`, once the
  session's `terminal_fn` names one for the root.

`dispatch(session, tool, arguments)` is the call the loop makes for each tool the model picks,
and anything else may make it too: without an endpoint this example drives the same tools from a
fixed script. With `SIMVX_LLM_BASE_URL`, `SIMVX_LLM_MODEL` and `SIMVX_LLM_API_KEY` set,
`OpenAICompatibleClient.from_env()` connects to any OpenAI-compatible endpoint and the model plays.

The game is a toy so the loop is easy to follow: holding the `score` action (Space) adds a point
every frame and emits `scored`, and reaching 5 points wins. The session and its tools are the same for any game.
"""

import asyncio
import os

from simvx.ai import AgentSession, OpenAICompatibleClient, dispatch, run_agent
from simvx.core import Input, Key, Node2D, Property, Signal


class Scorer(Node2D):
    input_context = {"score": [Key.SPACE]}
    score = Property(0)
    scored = Signal(int)

    def on_update(self, dt):
        if Input.pressed("score"):
            self.score += 1
            self.scored.emit(self.score)


def won(root: Scorer) -> str | None:
    return "win" if root.score >= 5 else None


async def agent_run(session: AgentSession) -> None:
    client = OpenAICompatibleClient.from_env()
    print(f"Driving model {client.model!r} at {client.base_url}\n")
    result = await run_agent(session, client, goal="Make the Scorer reach 5 points, then report how.", max_turns=15)
    print(result.final_text)
    print(f"\nturns={result.turns} tool_calls={result.tool_calls} terminated={result.terminated} {result.reason!r}")


def scripted_run(session: AgentSession) -> None:
    print("No SIMVX_LLM_BASE_URL set: driving the same tools from a script.\n")
    print(dispatch(session, "observe", {"kind": "describe"})["tree"], "\n")
    print(dispatch(session, "watch", {"op": "add", "path": "Scorer", "signal": "scored"}))
    print(dispatch(session, "send_input", {"kind": "key", "key": "space", "mode": "down"}))
    for _ in range(10):
        out = dispatch(session, "step", {"frames": 1})
        scored = [emit["args"][0] for emit in out["result"]["emissions"]]
        print(f"  frame={out['result']['frame']} scored={scored} terminated={out['terminated']}")
        if out["terminated"]:
            print(f"\nReached terminal state: {out['reason']!r}")
            break


if __name__ == "__main__":
    session = AgentSession(Scorer(name="Scorer"), terminal_fn=won)
    if os.environ.get("SIMVX_LLM_BASE_URL"):
        asyncio.run(agent_run(session))
    else:
        scripted_run(session)