{
"cells": [
{
"cell_type": "markdown",
"id": "06e7f13d",
"metadata": {},
"source": [
"# S3E Tutorial: Teaching Images to Speak PDDL\n",
"\n",
"\n",
" ![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
\n",
"\n",
"\n",
"Welcome in. In this notebook, you are going to teach a vision-language pipeline to look at an image, answer planning-relevant questions about it, and hand back a symbolic state that a planner can actually use. That is the core idea behind S3E. This package was originally built to support the [S3E paper](https://openreview.net/forum?id=gw4hYNFUIC), but the code has since been redesigned to be modular and reusable for a variety of applications.\n",
"\n",
"We will start with the friendliest possible version of the workflow, where everything is deterministic and easy to inspect. Then, once the moving parts make sense, we will swap in a real small HuggingFace model and watch the same pipeline run for real.\n",
"\n",
"By the end, you should have a feel for both the *shape* of the library and the *rhythm* of using it."
]
},
{
"cell_type": "markdown",
"id": "5117977a",
"metadata": {},
"source": [
"## Package Overview\n",
"\n",
"Here is the whole game in one sentence: S3E turns *what a planner cares about* into *questions a model can answer*, then turns the answers back into planner-friendly state. We call this **Semantic Symbolic State Estimation** (S3E).\n",
"\n",
"The package is built as concentric layers, and each one is usable on its own:\n",
"\n",
"1. **Backends** (`s3e.backends`): a uniform `VLMBackend` interface over HuggingFace, OpenAI, and vLLM models, all returning a `VLMOutput`.\n",
"2. **Engine** (`s3e.engine`): `QueryEngine` takes images, free-form queries, and an *answer space* (`BinaryAnswers`, `CategoricalAnswers`) and returns `Prediction`s. No PDDL anywhere.\n",
"3. **Calibration** (`s3e.calibration`): collect scores on labeled examples once, then fit and apply a `PlattCalibrator` offline, without a model in the loop.\n",
"4. **State-estimation facade** (`s3e.estimator`): `SemanticStateEstimator` takes grounded predicates (an explicit list, or grounded from a PDDL domain and problem by `from_pddl`), translates them into queries, and drives a `QueryEngine`.\n",
"\n",
"If you only remember a handful of names from the repo, make them these: `SemanticStateEstimator`, `QueryEngine`, `TemplateTranslator`, `VLMBackend`, and `PredictionSet`. Everything else in the tutorial will orbit around those."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ffde6577",
"metadata": {
"execution": {
"iopub.execute_input": "2026-08-30T14:09:28.946186Z",
"iopub.status.busy": "2026-08-30T14:09:28.945951Z",
"iopub.status.idle": "2026-08-30T14:09:29.957303Z",
"shell.execute_reply": "2026-08-30T14:09:29.956111Z"
},
"tags": [
"skip-execution"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install -q \"s3e[pddl,hf,calibration]\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "28a4623c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:14:53.455741Z",
"iopub.status.busy": "2026-10-07T07:14:53.455119Z",
"iopub.status.idle": "2026-10-07T07:15:32.852465Z",
"shell.execute_reply": "2026-10-07T07:15:32.849926Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Repository root: /path/to/s3e\n"
]
}
],
"source": [
"import os\n",
"import sys\n",
"import tempfile\n",
"import textwrap\n",
"import warnings\n",
"from pathlib import Path\n",
"from pprint import pprint\n",
"from unittest.mock import patch\n",
"\n",
"from PIL import Image, ImageDraw\n",
"\n",
"warnings.filterwarnings(\"ignore\", message=\".*IProgress not found.*\")\n",
"from tqdm import TqdmWarning\n",
"warnings.filterwarnings(\"ignore\", category=TqdmWarning)\n",
"\n",
"\n",
"def find_repo_root(start: Path) -> Path:\n",
" for candidate in [start, *start.parents]:\n",
" if (candidate / \"pyproject.toml\").exists() and (candidate / \"s3e\").exists():\n",
" return candidate\n",
" return start\n",
"\n",
"\n",
"REPO_ROOT = find_repo_root(Path.cwd())\n",
"if str(REPO_ROOT) not in sys.path:\n",
" sys.path.insert(0, str(REPO_ROOT))\n",
"\n",
"from s3e import (\n",
" BinaryAnswers,\n",
" CalibrationExample,\n",
" CalibrationSet,\n",
" CategoricalAnswers,\n",
" HuggingFaceVLM,\n",
" IdentityTranslator,\n",
" LLMTranslator,\n",
" PlattCalibrator,\n",
" PredictionSet,\n",
" PrewrittenTranslator,\n",
" QueryEngine,\n",
" SemanticStateEstimator,\n",
" TemplateTranslator,\n",
" VLMBackend,\n",
" VLMOutput,\n",
")\n",
"from s3e.pddl import (\n",
" bool_constant_to_fnode,\n",
" convert_state_dict_to_up_compatible,\n",
" get_object_names_dict,\n",
" get_pddl_strings,\n",
" ground_predicate_str_to_fnode,\n",
" ground_predicates,\n",
" parse_domain_problem,\n",
")\n",
"from s3e.translation.cache import load_cache, make_cache_key, save_cache\n",
"\n",
"COLOR_BLOCKSWORLD_DOMAIN = \"\"\"\n",
"(define (domain color-blocksworld)\n",
" (:requirements :typing)\n",
" (:types block)\n",
" (:predicates\n",
" (on ?x - block ?y - block)\n",
" (clear ?x - block)\n",
" )\n",
")\n",
"\"\"\"\n",
"\n",
"COLOR_BLOCKSWORLD_PROBLEM = \"\"\"\n",
"(define (problem color-bw-2)\n",
" (:domain color-blocksworld)\n",
" (:objects blue orange - block)\n",
" (:init (on blue orange) (clear blue))\n",
" (:goal (on orange blue))\n",
")\n",
"\"\"\"\n",
"\n",
"COLOR_BLOCKSWORLD_PROBLEM_3 = \"\"\"\n",
"(define (problem color-bw-3)\n",
" (:domain color-blocksworld)\n",
" (:objects blue orange green - block)\n",
" (:init (on blue orange) (clear blue) (clear green))\n",
" (:goal (on green blue))\n",
")\n",
"\"\"\"\n",
"\n",
"\n",
"def make_color_blocksworld_image() -> Image.Image:\n",
" image = Image.new(\"RGB\", (256, 256), \"white\")\n",
" draw = ImageDraw.Draw(image)\n",
" draw.rectangle((78, 60, 178, 130), fill=\"deepskyblue\", outline=\"black\", width=3)\n",
" draw.rectangle((78, 130, 178, 200), fill=\"orange\", outline=\"black\", width=3)\n",
" return image\n",
"\n",
"\n",
"def make_angled_blocksworld_image() -> Image.Image:\n",
" image = Image.new(\"RGB\", (256, 256), \"white\")\n",
" draw = ImageDraw.Draw(image)\n",
"\n",
" # Bottom orange block seen from a slight right-side angle.\n",
" draw.polygon([(78, 130), (178, 130), (208, 110), (108, 110)], fill=\"#ffcc80\", outline=\"black\")\n",
" draw.polygon([(178, 130), (208, 110), (208, 180), (178, 200)], fill=\"#d9822b\", outline=\"black\")\n",
" draw.polygon([(78, 130), (178, 130), (178, 200), (78, 200)], fill=\"orange\", outline=\"black\")\n",
"\n",
" # Top blue block from the same camera angle.\n",
" draw.polygon([(110, 60), (210, 60), (240, 40), (140, 40)], fill=\"#b3e5ff\", outline=\"black\")\n",
" draw.polygon([(210, 60), (240, 40), (240, 110), (210, 130)], fill=\"#2a99c9\", outline=\"black\")\n",
" draw.polygon([(110, 60), (210, 60), (210, 130), (110, 130)], fill=\"deepskyblue\", outline=\"black\")\n",
" return image\n",
"\n",
"\n",
"TEMPLATE_MAP = {\n",
" \"on\": \"Is the {0} block on top of the {1} block?\",\n",
" \"clear\": \"Is the {0} block clear?\",\n",
"}\n",
"\n",
"\n",
"def rounded(mapping: dict, digits: int = 4) -> dict:\n",
" \"\"\"Display helper: round probabilities to keep printed output compact.\"\"\"\n",
" return {key: round(value, digits) for key, value in mapping.items()}\n",
"\n",
"\n",
"print(f\"Repository root: {REPO_ROOT}\")"
]
},
{
"cell_type": "markdown",
"id": "021378f3",
"metadata": {},
"source": [
"## 1. Start from a planning problem\n",
"\n",
"Before we ask a model anything, we need to tell it what world it lives in. In S3E, that world comes from a PDDL domain and problem.\n",
"\n",
"For this tutorial, we are keeping the world tiny on purpose: two colored blocks, where `blue` sits on `orange`, and `blue` is clear. Small enough to understand instantly, but rich enough to exercise the full pipeline.\n",
"\n",
"The `s3e.pddl` helpers parse the PDDL with [Unified Planning](https://github.com/aiplan4eu/unified-planning) and enumerate every grounded predicate over the problem's objects. This is exactly what `SemanticStateEstimator.from_pddl` does for you later, so it is worth seeing once in the open."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "78efdebf",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:32.858271Z",
"iopub.status.busy": "2026-10-07T07:15:32.857639Z",
"iopub.status.idle": "2026-10-07T07:15:33.419762Z",
"shell.execute_reply": "2026-10-07T07:15:33.417712Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Objects grouped by type:\n",
"{'block': ['blue', 'orange']}\n",
"\n",
"All grounded predicates that the estimator will consider:\n",
"['on(blue,blue)',\n",
" 'on(blue,orange)',\n",
" 'on(orange,blue)',\n",
" 'on(orange,orange)',\n",
" 'clear(blue)',\n",
" 'clear(orange)']\n"
]
}
],
"source": [
"up_problem = parse_domain_problem(COLOR_BLOCKSWORLD_DOMAIN, COLOR_BLOCKSWORLD_PROBLEM)\n",
"grounded_predicates = ground_predicates(up_problem)\n",
"\n",
"assert up_problem.name == \"color-bw-2\"\n",
"assert len(grounded_predicates) == 6\n",
"\n",
"print(\"Objects grouped by type:\")\n",
"pprint(get_object_names_dict(up_problem))\n",
"print()\n",
"print(\"All grounded predicates that the estimator will consider:\")\n",
"pprint(grounded_predicates)"
]
},
{
"cell_type": "markdown",
"id": "44ac09e4",
"metadata": {},
"source": [
"That same helper also accepts real `.pddl` files, which is nice because it lets you prototype with strings and then graduate to normal project files without rewriting the rest of your code."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "82838ed1",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.424208Z",
"iopub.status.busy": "2026-10-07T07:15:33.424003Z",
"iopub.status.idle": "2026-10-07T07:15:33.453094Z",
"shell.execute_reply": "2026-10-07T07:15:33.450676Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"String input and file-path input both parsed successfully.\n"
]
}
],
"source": [
"with tempfile.TemporaryDirectory() as tmpdir:\n",
" domain_path = Path(tmpdir) / \"domain.pddl\"\n",
" problem_path = Path(tmpdir) / \"problem.pddl\"\n",
" domain_path.write_text(textwrap.dedent(COLOR_BLOCKSWORLD_DOMAIN).strip() + \"\\n\")\n",
" problem_path.write_text(textwrap.dedent(COLOR_BLOCKSWORLD_PROBLEM).strip() + \"\\n\")\n",
"\n",
" up_problem_from_files = parse_domain_problem(str(domain_path), str(problem_path))\n",
"\n",
"assert up_problem_from_files.name == up_problem.name\n",
"print(\"String input and file-path input both parsed successfully.\")"
]
},
{
"cell_type": "markdown",
"id": "0669ff9b",
"metadata": {},
"source": [
"## 2. Create an image observation\n",
"\n",
"Now let's give the estimator something to look at. In a real system, this image might come from a robot camera, a simulator, or a dataset. Here we draw the scene directly so the notebook stays self-contained and easy to replay.\n",
"\n",
"We will make two views of the same stack: one straight-on, and one sketched from a slight right-side angle."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "605bd6ef",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.457517Z",
"iopub.status.busy": "2026-10-07T07:15:33.457291Z",
"iopub.status.idle": "2026-10-07T07:15:33.487955Z",
"shell.execute_reply": "2026-10-07T07:15:33.485867Z"
}
},
"outputs": [
{
"data": {
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAEAAQADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD3+iiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAoorn/G3if/AIQ7whfa/wDY/tn2Xy/3Hm+Xu3SKn3sHGN2enagDoKK8Pj+P2rzRJLF8PL543AZWW6chgehB8nkU7/hfOtf9E51D/wACX/8AjNX7KfZmXt6S+0vvR7dRXiP/AAvnWv8AonOof+BL/wDxmj/hfOtf9E51D/wJf/4zT9lP+V/cL6xR/mX3o9uorxH/AIXzrX/ROdQ/8CX/APjNH/C+da/6JzqH/gS//wAZo9lP+V/cH1ij/MvvR7dRXiP/AAvnWv8AonOof+BL/wDxmj/hfOtf9E51D/wJf/4zR7Kf8r+4PrFH+Zfej26ivEf+F861/wBE51D/AMCX/wDjNH/C+da/6JzqH/gS/wD8Zo9lP+V/cH1ij/MvvR7dRXiP/C+da/6JzqH/AIEv/wDGaP8AhfOtf9E51D/wJf8A+M0eyn/K/uD6xR/mX3o9uorj/hx46/4WB4euNW/s77B5N21t5Xn+bnCI27O1f7+MY7V2FZmwUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXn/xt/wCSQ67/ANu//pRHXoFef/G3/kkOu/8Abv8A+lEdAHnfh7/kWtK/684f/QBWlWb4e/5FrSv+vOH/ANAFaVfa0vgj6I/K8R/Fl6v8wooorQxCiiigAooooAKKKKACiiigDS/Zx/5J5qH/AGFZP/RUVewV4/8As4/8k81D/sKyf+ioq9gr4Y/WgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACvP8A42/8kh13/t3/APSiOvQK8/8Ajb/ySHXf+3f/ANKI6APO/D3/ACLWlf8AXnD/AOgCtKs3w9/yLWlf9ecP/oArSr7Wl8EfRH5XiP4svV/mFFFFaGIUUUUAFFFFABRRRQAUUUUAaX7OP/JPNQ/7Csn/AKKir2CvH/2cf+Seah/2FZP/AEVFXsFfDH60FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAV5/8AG3/kkOu/9u//AKUR16BXn/xt/wCSQ67/ANu//pRHQB534e/5FrSv+vOH/wBAFaVZvh7/AJFrSv8Arzh/9AFaVfa0vgj6I/K8R/Fl6v8AMKKKK0MQooooAKKKKACiiigAooooA0v2cf8Aknmof9hWT/0VFXsFeP8A7OP/ACTzUP8AsKyf+ioq9gr4Y/WgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACvP/jb/wAkh13/ALd//SiOvQK8/wDjb/ySHXf+3f8A9KI6APO/D3/ItaV/15w/+gCtKs3w9/yLWlf9ecP/AKAK0q+1pfBH0R+V4j+LL1f5hRRRWhiFFFFABRRRQAUUUUAFFFFAGl+zj/yTzUP+wrJ/6Kir2CvH/wBnH/knmof9hWT/ANFRV7BXwx+tBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFef8Axt/5JDrv/bv/AOlEdegV5/8AG3/kkOu/9u//AKUR0AfPNh8K9c1HTrW+hu9OWK5iSZA8jggMARnCdeasf8Ke8Qf8/mmf9/ZP/iK9W8L/APIpaN/14wf+i1rWr4ern2MjUlFNaN9D6GGW0HFN3+88T/4U94g/5/NM/wC/sn/xFH/CnvEH/P5pn/f2T/4ivbKKz/1gxvdfcV/ZmH7P7zxP/hT3iD/n80z/AL+yf/EUf8Ke8Qf8/mmf9/ZP/iK9soo/1gxvdfcH9mYfs/vPE/8AhT3iD/n80z/v7J/8RR/wp7xB/wA/mmf9/ZP/AIivbKKP9YMb3X3B/ZmH7P7zxP8A4U94g/5/NM/7+yf/ABFH/CnvEH/P5pn/AH9k/wDiK9soo/1gxvdfcH9mYfs/vPE/+FPeIP8An80z/v7J/wDEUf8ACnvEH/P5pn/f2T/4ivbKKP8AWDG919wf2Zh+z+8o/s4/8k81D/sKyf8AoqKvYK8f/Zx/5J5qH/YVk/8ARUVewV94fOBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFef/G3/AJJDrv8A27/+lEdegV5/8bf+SQ67/wBu/wD6UR0Acv4X/wCRS0b/AK8YP/Ra1rVk+F/+RS0b/rxg/wDRa1rV+W4j+LL1f5n2FP4F6BRRRWRoFFFFABRRRQAUUUUAFFFFAFH9nH/knmof9hWT/wBFRV7BXj/7OP8AyTzUP+wrJ/6Kir2Cv1c+LCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK8/+Nv8AySHXf+3f/wBKI69Arz/42/8AJIdd/wC3f/0ojoA5fwv/AMilo3/XjB/6LWtasnwv/wAilo3/AF4wf+i1rWr8txH8WXq/zPsKfwL0CiiisjQKKKKACiiigAooooAKKKKAKP7OP/JPNQ/7Csn/AKKir2CvH/2cf+Seah/2FZP/AEVFXsFfq58WFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAV5/8AG3/kkOu/9u//AKUR16BXn/xt/wCSQ67/ANu//pRHQBy/hf8A5FLRv+vGD/0Wta1ZPhf/AJFLRv8Arxg/9FrWtX5biP4svV/mfYU/gXoFFFFZGgUUUUAFFFFABRRRQAUUUUAUf2cf+Seah/2FZP8A0VFXsFeP/s4/8k81D/sKyf8AoqKvYK/Vz4sKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArz/wCNv/JIdd/7d/8A0ojr0Cq99YWep2clnf2kF3ayY3wzxiRGwQRlTwcEA/hQB89aD8QfC9l4d0y1uNT2TwWkUci/Z5TtZUAIyFx1FaH/AAsrwj/0Fv8AyWl/+Jr13/hBPB//AEKmh/8Aguh/+Jo/4QTwf/0Kmh/+C6H/AOJrwJ8O4WcnJylr5r/I9KOaVopJJfj/AJnkX/CyvCP/AEFv/JaX/wCJo/4WV4R/6C3/AJLS/wDxNeu/8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTU/wCreE/ml96/yH/a1bsvx/zPIv8AhZXhH/oLf+S0v/xNH/CyvCP/AEFv/JaX/wCJr13/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mj/VvCfzS+9f5B/a1bsvx/wAzyL/hZXhH/oLf+S0v/wATR/wsrwj/ANBb/wAlpf8A4mvXf+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJo/1bwn80vvX+Qf2tW7L8f8zyL/hZXhH/AKC3/ktL/wDE0f8ACyvCP/QW/wDJaX/4mvXf+EE8H/8AQqaH/wCC6H/4mj/hBPB//QqaH/4Lof8A4mj/AFbwn80vvX+Qf2tW7L8f8zyL/hZXhH/oLf8AktL/APE0f8LK8I/9Bb/yWl/+Jr13/hBPB/8A0Kmh/wDguh/+Jo/4QTwf/wBCpof/AILof/iaP9W8J/NL71/kH9rVuy/H/M8//Zx/5J5qH/YVk/8ARUVewVT03SdN0a3a30vT7SxgZy7R2sKxKWwBkhQBnAAz7CrlfQnmBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQB//2Q==",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAIAAADTED8xAAAHB0lEQVR4Ae3Wwc0cRwxEYa+hxBSH43McDm0MnXQmBG4NyU8nHfpnF1/1w+zneZ6//EPgKoG/ry5ubwR+ESCAd3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO1295AngDpwkQ4HT9lieAN3CaAAFO12/5H3cQfD6fO8v++abP8/z5kPdP8AV4f0cSNhIgQCNco99P4NBPoN9l/Hfi4/5739L/ft76oegLUHodDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdEgAAlXA5vI0CAbY3ap0SAACVcDm8jQIBtjdqnRIAAJVwObyNAgG2N2qdE4PM8T+kP5h7+fD5zw38/+ZGH4Qvw/aflxhcRIMCLyhDl+wQI8H3mbnwRAQK8qAxRvk/gx/evjN/4/BuP8N4An3/em60jmS9AB1UzxxAgwJiqBO0gQIAOqmaOIUCAMVUJ2kGAAB1UzRxDgABjqhK0gwABOqiaOYYAAcZUJWgHAQJ0UDVzDAECjKlK0A4CBOigauYYAgQYU5WgHQQI0EHVzDEECDCmKkE7CBCgg6qZYwgQYExVgnYQIEAHVTPHECDAmKoE7SBAgA6qZo4hQIAxVQnaQYAAHVTNHEOAAGOqErSDAAE6qJo5hgABxlQlaAcBAnRQNXMMAQKMqUrQDgIE6KBq5hgCBBhTlaAdBAjQQdXMMQQIMKYqQTsIEKCDqpljCBBgTFWCdhAgQAdVM8cQIMCYqgTtIECADqpmjiFAgDFVCdpBgAAdVM0cQ4AAY6oStIMAATqomjmGAAHGVCVoBwECdFA1cwwBAoypStAOAgTooGrmGAIEGFOVoB0ECNBB1cwxBAgwpipBOwgQoIOqmWMIEGBMVYJ2ECBAB1UzxxAgwJiqBO0gQIAOqmaOIUCAMVUJ2kGAAB1UzRxDgABjqhK0gwABOqiaOYYAAcZUJWgHAQJ0UDVzDAECjKlK0A4CBOigauYYAgQYU5WgHQQI0EHVzDEECDCmKkE7CBCgg6qZYwgQYExVgnYQ+DzP0zH3hTM/n88LU7020pGH4Qvw2hco2DcIEOAblN3xWgKHfgK9tgPBggR8AYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECRAgCN/VeQIEyHcgQZAAAYLwXZ0nQIB8BxIECfwPXfMa9//dSAsAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"scene_image = make_color_blocksworld_image()\n",
"scene_image"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "6b0c3481",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.492382Z",
"iopub.status.busy": "2026-10-07T07:15:33.492191Z",
"iopub.status.idle": "2026-10-07T07:15:33.500917Z",
"shell.execute_reply": "2026-10-07T07:15:33.498890Z"
}
},
"outputs": [
{
"data": {
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAEAAQADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD3+iiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAornNQ8UK88thoax3t2jGKa4DK1vZuDgiUhgWcYP7tecgBjGGDVizeHbLUn+0a7DDq903Ja7j3xRk9RFGxKxL0GF5IUbixG46QpSlqY1K0YadTvaK8+/wCEN8Lf9C3o/wD4Axf/ABNH/CG+Fv8AoW9H/wDAGL/4mtPq77mX1pdj0GivPv8AhDfC3/Qt6P8A+AMX/wATR/whvhb/AKFvR/8AwBi/+Jo+rvuH1pdj0GivPv8AhDfC3/Qt6P8A+AMX/wATR/whvhb/AKFvR/8AwBi/+Jo+rvuH1pdj0GivPv8AhDfC3/Qt6P8A+AMX/wATR/whvhb/AKFvR/8AwBi/+Jo+rvuH1pdj0GivPv8AhDfC3/Qt6P8A+AMX/wATR/whvhb/AKFvR/8AwBi/+Jo+rvuH1pdj0GivMtS0HwRpFr9p1DRdDt4idql7KLLtgkKo25Zjg4UZJ7CuC1ez0jW8xQ+GtN02wOPkW0jS5cj+9Ihwi57KcnAy2CUqo4WcnZEzx0IK8j6Korn/AAJ/yTzw1/2CrX/0UtdBXKdoUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUVheMtbufDvhW71Kyghmu1aKKFJmITfJIsalsc4BcEgckDGR1ppNuyE2oq7NDU9WsNGtRcahdR28bNsTcfmkcgkIijl3ODhVBJ7A1zNzc6pr+RcefpWmn/l1jk2Xbkd3michFz/AAocnapL4Zo64i28Ta4l01/eaTZ3eoSKVeZ9Vl2ICQSsSGEiNOF4HJ2qWLMN1XP+E01r/oA6f/4M3/8AjFehTwFVayizx6ub4d6QmvvO1hhitoI4IIkihjUIkaKFVFAwAAOgA7U+uH/4TTWv+gDp/wD4M3/+MUf8JprX/QB0/wD8Gb//ABiuj6vV/lZy/X8N/OvvO4orh/8AhNNa/wCgDp//AIM3/wDjFH/Caa1/0AdP/wDBm/8A8Yo+r1f5WH17Dfzr7zuKK4f/AITTWv8AoA6f/wCDN/8A4xR/wmmtf9AHT/8AwZv/APGKPq9X+Vh9ew386+87iiuH/wCE01r/AKAOn/8Agzf/AOMUf8JprX/QB0//AMGb/wDxij6vV/lYfXsN/OvvO4orh/8AhNNa/wCgDp//AIM3/wDjFH/Caa1/0AdP/wDBm/8A8Yo+r1f5WH17Dfzr7zuK5bW/GdvaPJZ6UEvb5co7qwaG2cHBEhBBLDB+ReeADsDBq5i+8V6vrj3Nm4XTYLeRY5Us53MkjbVcETDYVXDrwFByv3iCQaMcaRRrHGioiAKqqMAAdABThQb+LQKuLjFe5r+Q6aS4vLtry+uHuLps/MSdkYOMrGhJCLwvA5O0Fix5ooorrjFRVkefKTk7yZ7F4E/5J54a/wCwVa/+ilroK5/wJ/yTzw1/2CrX/wBFLXQV80fZhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXI/Ez/kRrj/AK/LL/0qirrq5H4mf8iNcf8AX5Zf+lUVaUvjj6oxxH8KXo/yPO6KKK+1PysKKKKACiiigAooooAKKKKACiiigDDtf+Qnq/8A19L/AOiIquVTtf8AkJ6v/wBfS/8AoiKrleY936v8z6SHwR9F+SCiiigo9i8Cf8k88Nf9gq1/9FLXQVz/AIE/5J54a/7BVr/6KWugr5k+0CiiigAooooAKKKKACiiigAooooAKKKKACuR+Jn/ACI1x/1+WX/pVFXXVyPxM/5Ea4/6/LL/ANKoq0pfHH1RjiP4UvR/ked0UUV9qflYUUUUAFFFFABRRRQAUUUUAFFFFAGHa/8AIT1f/r6X/wBERVcqna/8hPV/+vpf/REVXK8x7v1f5n0kPgj6L8kFFFFBR7F4E/5J54a/7BVr/wCilroK5/wJ/wAk88Nf9gq1/wDRS10FfMn2gUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVyPxM/5Ea4/6/LL/wBKoq66uA+JuvaZ/wAI/No6XQlv/tllvhhVpDD/AKREw80qCI8jld5G7tmrpfHH1RlX/hS9H+RxlFFFfbH5UFFFFABRRRQAUUUUAFFFFABRRRQBh2v/ACE9X/6+l/8AREVXKp2v/IT1f/r6X/0RFVyvMe79X+Z9JD4I+i/JBRRUc88NtC008qRRL955GCqO3JNBSV9EezeBP+SeeGv+wVa/+ilroKw/BcE1r4F8PW9xFJDPFplskkcilWRhEoIIPIIPGK3K+ZPtAooooAKKKKACiiigAooooAKKKKACq99f2emWcl5f3cFpax43zTyCNFyQBljwMkgfjWVqvii3srl7Cxi/tHU0x5ltDKoFvkZVpmJ/dqcjsWIyVVtpxhpaXFzeJf6td/a71P8AViMNHBDwRmOIswDYLZckt8zDcFIUc1fFQo6PV9jalQlU22J7vWNS10GO0WbTNMb5XklXbdXCnvHhswqRjlh5nzMMRMoY854wtYbLwcYIFKot5aHlixYm6jJZmPLMSSSxJJJJJJNdTXOeOv8AkVn/AOvyz/8ASmKvOo4idbFU3J/aX5nTiKUaeFqW/lf5HOUUUV+pn4sFFFFABRRRQAUUUUAFFFFABRRRQBh2v/IT1f8A6+l/9ERVcrPSeG2vtZmnlSKJbpNzyMFUfuYhyTXb+HvAWo6yyXOrLLpunZB+zsMXFypGeobMI6Dkb+WGIyAx8etXhSu5d3+Z9ZhcNUrxioLpHX5I53T7O91m+ex0i2+13Me3zgHCpbg9GkY/dHOcAFiASqtg49R8MeA7LQZkvruX7fqqbgtwVKRxg8fu4tzBTjI3EljuYZCnaOi07TrTSNPhsLCBYbaEYRASe+SSTySSSSTkkkkkk1arx6+KnV02R9DhsFToa7vuFFFFcx2hRRRQAUUUUAFFFFABRRRQAVzPxAubq28GXRsrua0mmmtrfz4CBIiyzxxsVJBw212weoPI5FdNXK/EX/kTn/6/rH/0rhrOs3GnJrsy6avNJnC2un6jZW6wQeJdUVFyeY7ZixJyWZjDlmJJJYkkkkkkmpvJ1f8A6GfU/wDvza//ABmrlFfm7x+Ierl+R9V9XproU/J1f/oZ9T/782v/AMZqtfaVfalam2u/EepyQl0fb5dsPmRg6nIhzwyg/hWrRTjmGJi7qWvyFLDUpJpq6Zg/8I3L/wBB7U/++bf/AONUf8I3L/0HtT/75t//AI1W9RXX/b+af8/5fecX9h5Z/wA+If8AgKMH/hG5f+g9qf8A3zb/APxqj/hG5f8AoPan/wB82/8A8areoo/t/NP+f8vvD+w8s/58Q/8AAUYP/CNy/wDQe1P/AL5t/wD41R/wjcv/AEHtT/75t/8A41W9RR/b+af8/wCX3h/YeWf8+If+Aowf+Ebl/wCg9qf/AHzb/wDxqj/hG5f+g9qf/fNv/wDGq3qKP7fzT/n/AC+8P7Dyz/nxD/wFHExWF4+oalbnW7/ZbXCxIQkGSDFG/P7vrlz+GKn/ALKuv+g5qH/fEH/xqrNv/wAhrXP+vxP/AEnhq3XuRzfH8sX7aWy6vsjCOS5dr+4hu/sruZf9lXX/AEHNQ/74g/8AjVH9lXX/AEHNQ/74g/8AjValFP8AtfH/APP6X3sr+xMt/wCfEP8AwFHXfDjwfpdro2meJpTPd6vqFpFdvPcOCI2khQEIigIvGRuxuwSCxBxXf1z/AIE/5J54a/7BVr/6KWugr6SUnJ3bPDjCMFyxVkFFFFIoKKKKACiiigAooooAKKKKACiiigArlfiL/wAic/8A1/WP/pXDXVVyvxF/5E5/+v6x/wDSuGssR/Cl6P8AIun8a9Tl6KKK/LT7EKKKKACiiigAooooAKKKKACiiigDmbf/AJDWuf8AX4n/AKTw1bqpb/8AIa1z/r8T/wBJ4at19FH4I+i/JHNHr6v8woooplHpvgT/AJJ54a/7BVr/AOilroK5/wACf8k88Nf9gq1/9FLXQV9ofJBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXK/EX/kTn/wCv6x/9K4a6quV+Iv8AyJz/APX9Y/8ApXDWWI/hS9H+RdP416nL0UUV+Wn2IUUUUAFFFFABRRRQAUUUUAFFFFAHM2//ACGtc/6/E/8ASeGrdVLf/kNa5/1+J/6Tw1br6KPwR9F+SOaPX1f5hRRRTKPTfAn/ACTzw1/2CrX/ANFLXQVz/gT/AJJ54a/7BVr/AOilroK+0PkgooooAKKKKACiiigAooooAKKKKACiiigArlfiL/yJz/8AX9Y/+lcNdVXK/EX/AJE5/wDr+sf/AErhrLEfwpej/Iun8a9Tl6KKK/LT7EKKKKACiiigAooooAKKKKACiiigDmbf/kNa5/1+J/6Tw1bqpb/8hrXP+vxP/SeGrMkiQxPLK6pGilmdjgKB1JPYV9FD4I+i/JHNHZ+r/MdTEaS4vVsLG3kvb9k8xbWBl37O7HcQFXtuYgZwM5IB1dD8Nan4kCTxM1hpbAMt66K7XCk4/crnjjJDuMfdIV1bI9M0fRdP0Gy+yabAYoi5di0jSO7HuzsSzHAAyScAAdABXrYXLZT96rovxPPxOYRh7tPV/gReGtNm0bwrpGl3DRtPZWUNvI0ZJUsiBSRkA4yPQVqUUV754gUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVyvxF/5E5/+v6x/wDSuGuqqvfWFnqdnJZ39pBd2smN8M8YkRsEEZU8HBAP4VFSPPBx7lRfLJM81orsv+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJr5b/AFY/6e/+S/8ABPY/tf8Aufj/AMA42iuy/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+Jo/1Y/6e/wDkv/BD+1/7n4/8A42iuy/4QTwf/wBCpof/AILof/iaP+EE8H/9Cpof/guh/wDiaP8AVj/p7/5L/wAEP7X/ALn4/wDAONorsv8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/iaP9WP+nv8A5L/wQ/tf+5+P/AONorsv+EE8H/8AQqaH/wCC6H/4mj/hBPB//QqaH/4Lof8A4mj/AFY/6e/+S/8ABD+1/wC5+P8AwDjaK7L/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mj/Vj/p7/AOS/8EP7X/ufj/wDyS0aS48T63YWNvJe37XSyLawMu/Z9nhyx3EBV7bmIGcDOSAfR/D/AICjtJor/W5Y7y9jdZIYotywW5HPQn96wbkOwGNqlVQ5z1Gm6TpujW7W+l6faWMDOXaO1hWJS2AMkKAM4AGfYVcr3sLgKdBLq0kr+iscFfGVKumyCiiiu45AooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigD/9k=",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAIAAADTED8xAAAJGklEQVR4Ae3d0Y0cRRCHcS9yFgRgCV6R/EwAROAAnAVCZEEAjoAAeEbi1ZYIgDiWPa1V6jv2uOm62Zndqp8fUO26e6b/X/VXPN35cDwe3/iDQFcC33QNLjcCDwQI4B60JkCA1u0XngDuQGsCBGjdfuEJ4A60JkCA1u0XngDuQGsCBGjdfuEJ4A60JkCA1u0XngDuQGsCBGjdfuEJ4A60JkCA1u0XngDuQGsCBGjdfuEJ4A60JkCA1u0XngDuQGsCBGjdfuEJ4A60JkCA1u0XngDuQGsCBGjdfuEJ4A60JkCA1u0XngDuQGsCBGjdfuEJ4A60JkCA1u0XngDuQGsCBGjdfuEJ4A60JkCA1u0XngDuQGsCBGjdfuEJ4A60JkCA1u0X/i0Et0/gcDjc/iFv54RT//Q1AW6ncRdOcr76v//jHzO/AOfJVz99mxkTBHiC8YY+nm6/q7+wH6fb/91vf54Wf/n4fuGW8zICTOHaaLHBvxz0efCfb//yXbGSAIHiVgqDf3knYvAv3/JkJQGeANnzo8G/nP4rB3+8iACBYufC4F/egNcP/ngXAQLFboXBvxz9WoM/3kiAQLFPYfAv577i4I+XEiBQbF0Y/MuJrz7449UECBSbFgb/ctzXGPzxdgIEio0Kg3856OsN/jgDAQLFFoXBv5zyVQd/HIMAgeK6hcG/nO8Ggz8OQ4BAccXC4F8Od5vBH+chQKC4SmHwL8e65eCPUxEgUKxfGPzLmW48+ONgBAgUaxYG/3Kauwz+OB4BAsVqhcG/HOVegz9OSIBAsUJh8C+HuO/gj3MSIFC8tjjf/tNTzq197eOq70//CMu6YAiwKs8//PDuMp4/Zn5+d9mj51b5tShzvKwuRoAAxRoqzhwBAszxsroYAQIUa6g4cwQIMMfL6mIECFCsoeLMESDAHC+rixEgQLGGijNHgABzvKwuRoAAxRoqzhwBAszxsroYAQIUa6g4cwQIMMfL6mIECFCsoeLMESDAHC+rixEgQLGGijNHgABzvKwuRoAAxRoqzhwBAszxsroYAQIUa6g4cwQIMMfL6mIECFCsoeLMESDAHC+rixEgQLGGijNHgABzvKwuRoAAxRoqzhwBAszxsroYAQIUa6g4cwQIMMfL6mIECFCsoeLMESDAHC+rixEgQLGGijNHgABzvKwuRoAAxRoqzhwBAszxsroYAQIUa6g4cwQIMMfL6mIECFCsoeLMESDAHC+rixEgQLGGijNHgABzvKwuRoAAxRoqzhwBAszxsroYAQIUa6g4cwQIMMfL6mIECFCsoeLMESDAHC+rixEgQLGGijNH4O3ccqsvETgcDpe+9t0dECDAq5p0vvrHv345PeXww8N//dmLwJeP70+vPh6PUwcgwBSuR4tPt/989R9968MeBE63f/bqn49JgEy7xsGf2W/PegRygz/eT4BAsbQw+JeSuv669OCPoxEgULxcGPwvM9pqxSsHfxyTAIHihcLgfwHQhn/9+sEfhyVAoHi2MPifRbP5X6w1+OPgBAgUlwuD/zKXPb5dcfDH8QkQKJ4WBv9TIvt9Xn3wRxQCBIpHhcH/CMeuH64x+CMQAQLF18Lgf0pkv8/XG/yRiQCB4qEw+B/h2PXDVQd/JCPAVxQGf9yJ3YsNBn9kJMADCoM/LsTuxTaDP2J2F8Dgj6uwe7Hl4I+wrQUw+OMe7F5sPPgjb1MBDP64AbsXuwz+SN1RAIM/2r97sdfgj+C9BDD4o/G7F/sO/oh/yP0cTey/o+J8++/owLWPeiMXr9f/AY6fal+q1dIdPrz5+9fvV3vc8KB3P3++kat/PpRfizI0R9mPAAH69VzigQABBhjKfgQI0K/nEg8ECDDAUPYjQIB+PZd4IECAAYayHwEC9Ou5xAMBAgwwlP0IEKBfzyUeCBBggKHsR4AA/Xou8UCAAAMMZT8CBOjXc4kHAgQYYCj7ESBAv55LPBAgwABD2Y8AAfr1XOKBAAEGGMp+BAjQr+cSDwQIMMBQ9iNAgH49l3ggQIABhrIfAQL067nEAwECDDCU/QgQoF/PJR4IEGCAoexHgAD9ei7xQIAAAwxlPwIE6NdziQcCBBhgKPsRIEC/nks8ECDAAEPZjwAB+vVc4oEAAQYYyn4ECNCv5xIPBAgwwFD2I0CAfj2XeCBAgAGGsh8BAvTrucQDAQIMMJT9CBCgX88lHggQYICh7EeAAP16LvFAgAADDGU/AgTo13OJBwIEGGAo+xEgQL+eSzwQIMAAQ9mPwNt+kSXeh8C7nz+fXnw8Hvd5/TNvJcAzYHy9KoHT7b+1q3/OR4BV++xh/yFwm4M/jkmAQKFYn8DNDv6ISoBAoViTwI0P/ohKgEChWI3A7Q/+iEqAQKFYgcC9DP6ISoBAoXgtgTsa/BGVAIFCkSdwd4M/ohIgUCiSBO5x8EdUAgQKxTSB+x38EZUAgUIxR+CuB39EJUCgUCwlUGDwR1QCBArFIgI1Bn9EJUCgULxAoNLgj6gECBSK/yNQbPBHVAIECsVlAiUHf0QlQKBQXCBQdfBH1MNt/phCnG/F4nA4HD+t+LzKjzp8+Jqu/PXoJUDlO7t2tvJX/wyskQBr3xDPq0DAb4Wo0EUZ0gQIkEZnYwUCBKjQRRnSBAiQRmdjBQIEqNBFGdIECJBGZ2MFAgSo0EUZ0gQIkEZnYwUCBKjQRRnSBAiQRmdjBQIEqNBFGdIECJBGZ2MFAgSo0EUZ0gQIkEZnYwUCBKjQRRnSBAiQRmdjBQIEqNBFGdIECJBGZ2MFAgSo0EUZ0gQIkEZnYwUCBKjQRRnSBAiQRmdjBQIEqNBFGdIECJBGZ2MFAgSo0EUZ0gQIkEZnYwUCBKjQRRnSBAiQRmdjBQIEqNBFGdIECJBGZ2MFAgSo0EUZ0gQIkEZnYwUCBKjQRRnSBAiQRmdjBQIEqNBFGdIECJBGZ2MFAgSo0EUZ0gQIkEZnYwUCBKjQRRnSBAiQRmdjBQIEqNBFGdIECJBGZ2MFAgSo0EUZ0gQIkEZnYwUCBKjQRRnSBAiQRmdjBQIEqNBFGdIECJBGZ2MFAgSo0EUZ0gQIkEZnYwUCBKjQRRnSBAiQRmdjBQIEqNBFGdIECJBGZ2MFAv8C5+pTM0xcmAYAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"angled_scene_image = make_angled_blocksworld_image()\n",
"angled_scene_image"
]
},
{
"cell_type": "markdown",
"id": "4bc57157",
"metadata": {},
"source": [
"## 3. The PDDL-free core: a `QueryEngine` and a deterministic backend\n",
"\n",
"Let's put training wheels on first. Instead of jumping straight into a real model, we will use a tiny custom `VLMBackend` that already knows the answer for this toy scene, and drive it with a bare `QueryEngine`: no PDDL, no predicates, just images and questions.\n",
"\n",
"That buys us two things immediately: the tutorial becomes perfectly runnable, and you get to see the exact contract a custom backend has to satisfy. If you ever want to plug in your own model wrapper later, this is the seam you will use.\n",
"\n",
"A backend implements one method, `query(images, prompt, system_prompt=None, generate=False, interest_tokens=None, **inference_kwargs)`, and returns a `VLMOutput`. Two arguments deserve attention:\n",
"\n",
"- `generate`: `False` means \"score the next token\" (`token_probs`), `True` means \"write an answer\" (`text`).\n",
"- `interest_tokens`: when the engine scores by log-probabilities, it tells the backend exactly which token strings it cares about (the answer space's tokens). The backend then reports probability mass for *those strings only*, and flags in `argmax_in_interest` whether the model's single most likely token was one of them. Real backends use this to skip decoding the rest of the vocabulary."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "a43f4737",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.505428Z",
"iopub.status.busy": "2026-10-07T07:15:33.505224Z",
"iopub.status.idle": "2026-10-07T07:15:33.514998Z",
"shell.execute_reply": "2026-10-07T07:15:33.513047Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Prompt the backend actually received:\n",
" Answer yes or no only: Is the blue block on top of the orange block?\n",
"Interest tokens the engine asked for:\n",
" ['yes', ' yes', 'Yes', ' Yes', 'YES', ' YES', 'no', ' no', 'No', ' No', 'NO', ' NO']\n",
"\n",
"One Prediction, piece by piece:\n",
" masses: {'yes': 0.85, 'no': 0.15000000000000002}\n",
" null_mass: 0.0\n",
" unassigned_mass: 0.0\n",
" probability: 0.85\n",
" answer: True\n",
"\n",
"Whole PredictionSet as probabilities:\n",
"{'Is the blue block clear?': 0.85,\n",
" 'Is the blue block on top of the orange block?': 0.85,\n",
" 'Is the orange block on top of the blue block?': 0.15}\n",
"\n",
"...and thresholded into a state:\n",
"{'Is the blue block clear?': True,\n",
" 'Is the blue block on top of the orange block?': True,\n",
" 'Is the orange block on top of the blue block?': False}\n"
]
}
],
"source": [
"def binary_output(yes_prob, interest_tokens=None, yes=\"yes\", no=\"no\"):\n",
" \"\"\"Build a VLMOutput that spends `yes_prob` on the yes token, the rest on no.\"\"\"\n",
" probs = {yes: yes_prob, no: 1.0 - yes_prob}\n",
" if interest_tokens is None:\n",
" return VLMOutput(token_probs=probs)\n",
" # The interest_tokens contract: exactly those keys, unknown strings get 0.0.\n",
" return VLMOutput(\n",
" token_probs={token: probs.get(token, 0.0) for token in interest_tokens},\n",
" argmax_in_interest=True,\n",
" )\n",
"\n",
"\n",
"class TutorialSceneVLM(VLMBackend):\n",
" \"\"\"A tiny deterministic backend for the tutorial notebook.\"\"\"\n",
"\n",
" def __init__(self):\n",
" self.received_prompts = []\n",
" self.received_system_prompts = []\n",
"\n",
" def query(self, images, prompt, system_prompt=None, generate=False, interest_tokens=None, **inference_kwargs):\n",
" self.received_prompts.append(prompt)\n",
" self.received_system_prompts.append(system_prompt)\n",
"\n",
" truth_map = {\n",
" \"Is the blue block on top of the orange block?\": 0.85,\n",
" \"Is the orange block on top of the blue block?\": 0.15,\n",
" \"Is the blue block clear?\": 0.85,\n",
" \"Is the orange block clear?\": 0.15,\n",
" }\n",
"\n",
" yes_prob = 0.15\n",
" for query_text, probability in truth_map.items():\n",
" if query_text in prompt:\n",
" yes_prob = probability\n",
" break\n",
"\n",
" if generate:\n",
" return VLMOutput(token_probs=None, text=\"yes\" if yes_prob >= 0.5 else \"no\")\n",
" return binary_output(yes_prob, interest_tokens)\n",
"\n",
"\n",
"tutorial_vlm = TutorialSceneVLM()\n",
"engine = QueryEngine(\n",
" tutorial_vlm,\n",
" prompt_template=\"Answer yes or no only: {query}\",\n",
" system_prompt=\"You are looking at a stack of colored blocks.\",\n",
")\n",
"\n",
"questions = [\n",
" \"Is the blue block on top of the orange block?\",\n",
" \"Is the orange block on top of the blue block?\",\n",
" \"Is the blue block clear?\",\n",
"]\n",
"predictions = engine.ask([scene_image], questions)\n",
"\n",
"assert isinstance(predictions, PredictionSet)\n",
"assert list(predictions) == questions\n",
"\n",
"first = predictions[\"Is the blue block on top of the orange block?\"]\n",
"print(\"Prompt the backend actually received:\")\n",
"print(\" \", tutorial_vlm.received_prompts[0])\n",
"print(\"Interest tokens the engine asked for:\")\n",
"print(\" \", engine.answers.interest_tokens)\n",
"print()\n",
"print(\"One Prediction, piece by piece:\")\n",
"print(\" masses: \", first.masses)\n",
"print(\" null_mass: \", first.null_mass)\n",
"print(\" unassigned_mass: \", first.unassigned_mass)\n",
"print(\" probability: \", first.probability)\n",
"print(\" answer: \", first.answer)\n",
"print()\n",
"print(\"Whole PredictionSet as probabilities:\")\n",
"pprint(rounded(predictions.probabilities()))\n",
"print()\n",
"print(\"...and thresholded into a state:\")\n",
"pprint(predictions.to_state())"
]
},
{
"cell_type": "markdown",
"id": "168e3afc",
"metadata": {},
"source": [
"A few things to notice. `engine.ask` returned a `PredictionSet`, an ordered mapping from query to `Prediction`. Each `Prediction` stores only the raw data (per-option `masses`, the mass on an explicit null option, and whatever `unassigned_mass` the model spent elsewhere); everything else (`probability`, `answer`, `has_answer`, `null_dominated`, `matched`, `distribution()`, `score`) is derived lazily and cached, so nothing re-runs inference. `probability` is simply `T / (T + F)`, the true mass over the true and false masses, so it prints as a clean `0.85`; the `rounded` helper from the setup cell only keeps longer numbers short.\n",
"\n",
"Also notice the interest tokens: `BinaryAnswers()` defaults to the labels `yes`/`no`, and each label auto-expands into case and leading-space variants (`yes`, ` yes`, `Yes`, ...). That is what lets real tokenizers, which love a leading space, work without hand-written token lists."
]
},
{
"cell_type": "markdown",
"id": "4d403b16",
"metadata": {},
"source": [
"## 4. Answer spaces: yes/no is just the default\n",
"\n",
"An **answer space** says what counts as an answer and how to score the model's output. `BinaryAnswers` has boolean semantics (a true label and a false label). `CategoricalAnswers` scores any number of labeled options. Both accept an optional explicit null option (`null_tokens=...`) so a model can abstain.\n",
"\n",
"Models do not always answer in the vocabulary you hoped for, either. Sometimes `yes` and `no` are great; sometimes a different pair of words is easier to elicit. Below, the same symbolic task is answered with `present`/`absent`, and a second backend answers a categorical question about color."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "9b35a25b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.519453Z",
"iopub.status.busy": "2026-10-07T07:15:33.519224Z",
"iopub.status.idle": "2026-10-07T07:15:33.537317Z",
"shell.execute_reply": "2026-10-07T07:15:33.535075Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"present/absent answer space, logprobs scoring:\n",
"{'Is the blue block clear?': 0.9,\n",
" 'Is the blue block on top of the orange block?': 0.9,\n",
" 'Is the orange block on top of the blue block?': 0.1}\n",
"\n",
"The same backend and answer space, text_match scoring:\n",
"{'Is the blue block clear?': 1.0,\n",
" 'Is the blue block on top of the orange block?': 1.0,\n",
" 'Is the orange block on top of the blue block?': 0.0}\n",
"\n",
"Categorical answer: blue\n",
"Categorical distribution: {'blue': 0.7, 'orange': 0.2, 'green': 0.1}\n"
]
}
],
"source": [
"class PresentAbsentVLM(VLMBackend):\n",
" def query(self, images, prompt, system_prompt=None, generate=False, interest_tokens=None, **inference_kwargs):\n",
" is_true = \"blue block on top of the orange\" in prompt or \"blue block clear\" in prompt\n",
" if generate:\n",
" return VLMOutput(token_probs=None, text=\"present\" if is_true else \"absent\")\n",
" return binary_output(0.9 if is_true else 0.1, interest_tokens, yes=\"present\", no=\"absent\")\n",
"\n",
"\n",
"class TopBlockColorVLM(VLMBackend):\n",
" def query(self, images, prompt, system_prompt=None, generate=False, interest_tokens=None, **inference_kwargs):\n",
" probs = {\"blue\": 0.7, \"orange\": 0.2, \"green\": 0.1}\n",
" if generate:\n",
" return VLMOutput(token_probs=None, text=\"I think it is blue.\")\n",
" return VLMOutput(\n",
" token_probs={token: probs.get(token, 0.0) for token in interest_tokens},\n",
" argmax_in_interest=True,\n",
" )\n",
"\n",
"\n",
"present_absent = QueryEngine(\n",
" PresentAbsentVLM(),\n",
" answers=BinaryAnswers(\"present\", \"absent\"),\n",
" prompt_template=\"Answer present or absent only: {query}\",\n",
")\n",
"present_absent_predictions = present_absent.ask([scene_image], questions)\n",
"\n",
"color_engine = QueryEngine(\n",
" TopBlockColorVLM(),\n",
" answers=CategoricalAnswers([\"blue\", \"orange\", \"green\"]),\n",
")\n",
"color_prediction = color_engine.ask([scene_image], [\"What color is the top block?\"])[\"What color is the top block?\"]\n",
"\n",
"# scoring is a per-call knob too: text_match asks the backend to generate\n",
"# text (generate=True) and scores the answer option the reply starts with.\n",
"text_match_predictions = present_absent.ask([scene_image], questions, scoring=\"text_match\")\n",
"\n",
"assert present_absent_predictions.to_state()[\"Is the blue block on top of the orange block?\"] is True\n",
"assert color_prediction.answer == \"blue\"\n",
"\n",
"print(\"present/absent answer space, logprobs scoring:\")\n",
"pprint(rounded(present_absent_predictions.probabilities()))\n",
"print()\n",
"print(\"The same backend and answer space, text_match scoring:\")\n",
"pprint(rounded(text_match_predictions.probabilities()))\n",
"print()\n",
"print(\"Categorical answer:\", color_prediction.answer)\n",
"print(\"Categorical distribution:\", color_prediction.distribution())"
]
},
{
"cell_type": "markdown",
"id": "f2e376f8",
"metadata": {},
"source": [
"Here is the practical takeaway: use `logprobs` when the backend is actually surfacing the answer tokens you care about, and switch to `text_match` when you can strongly coerce the model into a short, exact answer. Neither is universally better; the right choice depends on how your model behaves. Under `text_match`, the matched option gets all the mass, so probabilities collapse to exactly `0.0`/`1.0`.\n",
"\n",
"`text_match` is strict on purpose: the reply has to *start* with one of the answer space's tokens, as a whole word (case-sensitive against the token variants; surrounding whitespace is ignored). `present`, `Absent.` and `PRESENT` all match; `I think it is present`, `Answer: present` and `**present**` match nothing, so prompt for a bare answer. A query whose reply matches nothing, or whose top answer is the null option, has **no answer** (`has_answer` is `False`): its `probability` is exactly `0.5`, its `distribution()` is uniform, and its `answer` falls to the first option. The engine also emits one `UnmatchedAnswerWarning` per call that had unmatched replies, so they never slip by silently.\n",
"\n",
"`probability` and `score` are only defined for binary answer spaces; a categorical `Prediction` gives you `answer` (the argmax label) and `distribution()` instead."
]
},
{
"cell_type": "markdown",
"id": "d2821837",
"metadata": {},
"source": [
"## 5. Build the estimator: predicates in, state out\n",
"\n",
"Now let's put the PDDL back. `SemanticStateEstimator.from_pddl` grounds the domain and problem into predicates (section 1), runs them through a *translator* to produce queries, and wires up a `QueryEngine` (section 3) underneath. The estimator's contract is \"predicates in, state out\".\n",
"\n",
"`TemplateTranslator` is the comfortable default here. It turns planner-shaped strings like `on(blue,orange)` into questions a model has a fighting chance of answering sensibly. In practice, that is usually what you want."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "a39ab8ad",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.541466Z",
"iopub.status.busy": "2026-10-07T07:15:33.541279Z",
"iopub.status.idle": "2026-10-07T07:15:33.821613Z",
"shell.execute_reply": "2026-10-07T07:15:33.819346Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Predicate -> query mapping:\n",
"{'clear(blue)': 'Is the blue block clear?',\n",
" 'clear(orange)': 'Is the orange block clear?',\n",
" 'on(blue,blue)': 'Is the blue block on top of the blue block?',\n",
" 'on(blue,orange)': 'Is the blue block on top of the orange block?',\n",
" 'on(orange,blue)': 'Is the orange block on top of the blue block?',\n",
" 'on(orange,orange)': 'Is the orange block on top of the orange block?'}\n",
"\n",
"System prompt handed to the engine:\n",
"A curious human is asking an artificial intelligence assistant yes or no questions. The assistant answers with one of two responses: YES or NO. The assistant's response should not include any additional text.\n",
"Additional Instructions and clarifications:\n",
"Focus only on the visible colored blocks.\n"
]
}
],
"source": [
"tutorial_vlm = TutorialSceneVLM()\n",
"estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=tutorial_vlm,\n",
" translator=TemplateTranslator(TEMPLATE_MAP),\n",
" prompt_template=\"Answer yes or no only: {query}\",\n",
" additional_instructions=\"Focus only on the visible colored blocks.\",\n",
")\n",
"\n",
"assert estimator.predicates == grounded_predicates\n",
"assert isinstance(estimator.engine, QueryEngine)\n",
"\n",
"print(\"Predicate -> query mapping:\")\n",
"pprint(estimator.queries)\n",
"print()\n",
"print(\"System prompt handed to the engine:\")\n",
"print(estimator.engine.system_prompt)"
]
},
{
"cell_type": "markdown",
"id": "635e3a3d",
"metadata": {},
"source": [
"### The three views of one prediction pass\n",
"\n",
"This is the part of the library you will touch most often, so it is worth slowing down for a second. There are three ways to look at the same prediction pass, depending on how much uncertainty and debugging detail you want to keep around.\n",
"\n",
"- Call the estimator directly for a clean symbolic state.\n",
"- Call `estimate()` for a `PredictionSet` with full per-predicate detail; `probabilities()` and `to_state()` derive from it.\n",
"- Pass `keep_raw=True` to `estimate()` when you are tuning prompts or sanity-checking a backend, and inspect the normalized `VLMOutput` on each prediction's `.raw`.\n",
"\n",
"`confidence` is an acceptance threshold on `P(true)`: a predicate is `True` when `P(true) >= confidence` and `False` otherwise, so you always get a complete state. At the default `0.5` every predicate with an answer is decided by its argmax; raise the bar and a predicate has to earn its `True`, so anything the model is not sure enough about comes back `False`. A prediction with no answer (section 4) sits at exactly `P(true) = 0.5`: `True` at the default, `False` at any stricter threshold. The state alone cannot tell \"confidently false\" from \"not confident enough\"; for that, keep the `PredictionSet` and look at `probabilities()`, or filter with `results.where(lambda p: p.confident(0.9))`. (`confidence` applies to binary answer spaces only.)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "00d679fb",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.826396Z",
"iopub.status.busy": "2026-10-07T07:15:33.826169Z",
"iopub.status.idle": "2026-10-07T07:15:33.835826Z",
"shell.execute_reply": "2026-10-07T07:15:33.833846Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Boolean state at the default confidence=0.5:\n",
"{'clear(blue)': True,\n",
" 'clear(orange)': False,\n",
" 'on(blue,blue)': False,\n",
" 'on(blue,orange)': True,\n",
" 'on(orange,blue)': False,\n",
" 'on(orange,orange)': False}\n",
"\n",
"Probabilities:\n",
"{'clear(blue)': 0.85,\n",
" 'clear(orange)': 0.15,\n",
" 'on(blue,blue)': 0.15,\n",
" 'on(blue,orange)': 0.85,\n",
" 'on(orange,blue)': 0.15,\n",
" 'on(orange,orange)': 0.15}\n",
"\n",
"The same predictions at a stricter confidence=0.9 (0.85 does not reach 0.9, so nothing is True):\n",
"{'clear(blue)': False,\n",
" 'clear(orange)': False,\n",
" 'on(blue,blue)': False,\n",
" 'on(blue,orange)': False,\n",
" 'on(orange,blue)': False,\n",
" 'on(orange,orange)': False}\n",
"\n",
"Predicates the model is not 90% sure of, either way: 6 of 6\n",
"\n",
"Raw backend output for on(blue,orange):\n",
"VLMOutput(token_probs={'yes': 0.85, ' yes': 0.0, 'Yes': 0.0, ' Yes': 0.0, 'YES': 0.0, ' YES': 0.0, 'no': 0.15000000000000002, ' no': 0.0, 'No': 0.0, ' No': 0.0, 'NO': 0.0, ' NO': 0.0}, text=None, argmax_in_interest=True, reasoning=None)\n"
]
}
],
"source": [
"state = estimator([scene_image])\n",
"results = estimator.estimate([scene_image], keep_raw=True)\n",
"probabilities = results.probabilities()\n",
"strict_state = results.to_state(confidence=0.9)\n",
"unsure = results.where(lambda prediction: not prediction.confident(0.9))\n",
"\n",
"assert isinstance(state, dict)\n",
"assert all(isinstance(value, bool) for value in state.values())\n",
"assert all(isinstance(value, bool) for value in strict_state.values())\n",
"assert all(0.0 <= value <= 1.0 for value in probabilities.values())\n",
"assert isinstance(results[\"on(blue,orange)\"].raw, VLMOutput)\n",
"assert state[\"on(blue,orange)\"] is True\n",
"assert state[\"on(orange,blue)\"] is False\n",
"assert state[\"on(blue,blue)\"] is False\n",
"assert not any(strict_state.values()) # 0.85 does not reach 0.9\n",
"assert len(unsure) == len(results)\n",
"assert estimator([scene_image], confidence=0.9) == strict_state\n",
"\n",
"print(\"Boolean state at the default confidence=0.5:\")\n",
"pprint(state)\n",
"print()\n",
"print(\"Probabilities:\")\n",
"pprint(rounded(probabilities))\n",
"print()\n",
"print(\"The same predictions at a stricter confidence=0.9 (0.85 does not reach 0.9, so nothing is True):\")\n",
"pprint(strict_state)\n",
"print()\n",
"print(f\"Predicates the model is not 90% sure of, either way: {len(unsure)} of {len(results)}\")\n",
"print()\n",
"print(\"Raw backend output for on(blue,orange):\")\n",
"print(results[\"on(blue,orange)\"].raw)"
]
},
{
"cell_type": "markdown",
"id": "1febe6df",
"metadata": {},
"source": [
"A good mental shortcut is this: the boolean state is just the probability state after someone has drawn a line in the sand. If your downstream code can benefit from uncertainty, hold onto the `PredictionSet` for as long as you can.\n",
"\n",
"Predictions are also plain data. `PredictionSet.to_dict()` / `PredictionSet.from_dict()` round-trip through JSON with no backend involved, which is how you keep prediction details around for offline analysis or recalibration later."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "6ef017d8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.840348Z",
"iopub.status.busy": "2026-10-07T07:15:33.840152Z",
"iopub.status.idle": "2026-10-07T07:15:33.856372Z",
"shell.execute_reply": "2026-10-07T07:15:33.854298Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Serialized 6 predictions into 2902 bytes of JSON and back.\n",
"Restored on(blue,orange) masses: {'yes': 0.85, 'no': 0.15000000000000002}\n"
]
}
],
"source": [
"import json\n",
"\n",
"payload = json.dumps(results.to_dict())\n",
"restored = PredictionSet.from_dict(json.loads(payload))\n",
"\n",
"assert restored.probabilities() == results.probabilities()\n",
"assert restored[\"on(blue,orange)\"].raw is None # raw backend output is never serialized\n",
"\n",
"print(f\"Serialized {len(results)} predictions into {len(payload)} bytes of JSON and back.\")\n",
"print(\"Restored on(blue,orange) masses:\", restored[\"on(blue,orange)\"].masses)"
]
},
{
"cell_type": "markdown",
"id": "df9be76a",
"metadata": {},
"source": [
"### Querying a subset of predicates\n",
"\n",
"Sometimes you do not need every grounded predicate evaluated. Maybe you already know most of the state and only need to re-check a few fluents, or maybe your domain is large and you want to save on VLM calls. `estimate()` accepts an optional `predicates` argument that restricts which grounded predicates are queried. Pass `None` (the default) to query everything, or pass a list of predicate strings to query only those."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "0dfda69e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.860745Z",
"iopub.status.busy": "2026-10-07T07:15:33.860570Z",
"iopub.status.idle": "2026-10-07T07:15:33.876892Z",
"shell.execute_reply": "2026-10-07T07:15:33.875007Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Filtered boolean state (2 of 6 predicates):\n",
"{'clear(blue)': True, 'on(blue,orange)': True}\n",
"\n",
"Filtered probabilities:\n",
"{'clear(blue)': 0.85, 'on(blue,orange)': 0.85}\n",
"\n",
"Filtered raw output:\n",
" on(blue,orange): VLMOutput(token_probs={'yes': 0.85, ' yes': 0.0, 'Yes': 0.0, ' Yes': 0.0, 'YES': 0.0, ' YES': 0.0, 'no': 0.15000000000000002, ' no': 0.0, 'No': 0.0, ' No': 0.0, 'NO': 0.0, ' NO': 0.0}, text=None, argmax_in_interest=True, reasoning=None)\n",
" clear(blue): VLMOutput(token_probs={'yes': 0.85, ' yes': 0.0, 'Yes': 0.0, ' Yes': 0.0, 'YES': 0.0, ' YES': 0.0, 'no': 0.15000000000000002, ' no': 0.0, 'No': 0.0, ' No': 0.0, 'NO': 0.0, ' NO': 0.0}, text=None, argmax_in_interest=True, reasoning=None)\n"
]
}
],
"source": [
"subset = [\"on(blue,orange)\", \"clear(blue)\"]\n",
"\n",
"filtered = estimator.estimate([scene_image], predicates=subset, keep_raw=True)\n",
"\n",
"assert list(filtered) == subset\n",
"\n",
"print(\"Filtered boolean state (2 of 6 predicates):\")\n",
"pprint(filtered.to_state())\n",
"print()\n",
"print(\"Filtered probabilities:\")\n",
"pprint(rounded(filtered.probabilities()))\n",
"print()\n",
"print(\"Filtered raw output:\")\n",
"for predicate, prediction in filtered.items():\n",
" print(f\" {predicate}: {prediction.raw}\")"
]
},
{
"cell_type": "markdown",
"id": "c724eb37",
"metadata": {},
"source": [
"## 6. Calibrating probabilities with held-out examples\n",
"\n",
"Raw model probabilities are often useful, but they are not automatically trustworthy. A model that says `0.8` does not necessarily mean \"80% of the time this fluent is true.\" S3E can fit a small post-hoc Platt scaling layer on held-out examples so that downstream thresholds and planning decisions operate on better-calibrated numbers.\n",
"\n",
"Calibration is a self-contained pipeline over prediction data, and it separates the expensive step from the cheap one:\n",
"\n",
"1. `CalibrationSet.collect(estimator, examples)` queries the VLM once per labeled example and records a grouped log-odds `score` plus the true `label` for every predicate the model answered (a prediction with no answer is skipped: its probability stays `0.5` whatever a calibrator says, so it is not a training point). This is the only step that needs a model.\n",
"2. `PlattCalibrator.fit(data, scope=...)` fits the sigmoid. `scope` controls grouping: `\"global\"` (one fit for everything), `\"lifted\"` (one per predicate name, so every `on(...)` shares a fit), or `\"grounded\"` (one per fully-grounded predicate).\n",
"3. `calibrator.apply(results)` returns a new, calibrated `PredictionSet`; `estimator.estimate(images, calibrator=...)` does the same inline.\n",
"\n",
"Both the data and the fitted calibrator save to JSON, so you can refit with a different scope tomorrow without touching the model again. Fitting requires the optional extra (`pip install 's3e[calibration]'`) and `scoring=\"logprobs\"`."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "3d2f7d8a",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:33.881767Z",
"iopub.status.busy": "2026-10-07T07:15:33.881522Z",
"iopub.status.idle": "2026-10-07T07:15:34.085976Z",
"shell.execute_reply": "2026-10-07T07:15:34.084718Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"First collected sample: CalibrationSample(predicate='clear(blue)', score=1.386294361116141, label=True, problem=None)\n",
"Global calibrator groups: ['__global__']\n",
"Lifted calibrator groups: ['clear', 'on']\n",
"\n",
"Raw probabilities:\n",
"{'clear(blue)': 0.7,\n",
" 'clear(orange)': 0.3,\n",
" 'on(blue,blue)': 0.05,\n",
" 'on(blue,orange)': 0.6,\n",
" 'on(orange,blue)': 0.2,\n",
" 'on(orange,orange)': 0.05}\n",
"\n",
"Global Platt-scaled probabilities:\n",
"{'clear(blue)': 0.8376448556124211,\n",
" 'clear(orange)': 0.2222486486029375,\n",
" 'on(blue,blue)': 0.007896955077302354,\n",
" 'on(blue,orange)': 0.7081478869691008,\n",
" 'on(orange,blue)': 0.10220937327764033,\n",
" 'on(orange,orange)': 0.007896955077302354}\n",
"\n",
"Lifted-fluent Platt-scaled probabilities:\n",
"{'clear(blue)': 0.7886829110663371,\n",
" 'clear(orange)': 0.34560451632924843,\n",
" 'on(blue,blue)': 0.01447525743604522,\n",
" 'on(blue,orange)': 0.6115560324656291,\n",
" 'on(orange,blue)': 0.1144130324303443,\n",
" 'on(orange,orange)': 0.01447525743604522}\n"
]
}
],
"source": [
"def calibration_prompt(predicate: str) -> str:\n",
" name, arguments = predicate[:-1].split(\"(\", 1)\n",
" return TEMPLATE_MAP[name].format(*arguments.split(\",\"))\n",
"\n",
"\n",
"CALIBRATION_LABELS_2OBJ = {\n",
" \"clear(blue)\": True,\n",
" \"clear(orange)\": False,\n",
" \"on(blue,blue)\": False,\n",
" \"on(blue,orange)\": True,\n",
" \"on(orange,blue)\": False,\n",
" \"on(orange,orange)\": False,\n",
"}\n",
"\n",
"CALIBRATION_LABELS_3OBJ = {\n",
" \"clear(blue)\": True,\n",
" \"clear(green)\": True,\n",
" \"clear(orange)\": False,\n",
" \"on(blue,blue)\": False,\n",
" \"on(blue,green)\": False,\n",
" \"on(blue,orange)\": True,\n",
" \"on(green,blue)\": False,\n",
" \"on(green,green)\": False,\n",
" \"on(green,orange)\": False,\n",
" \"on(orange,blue)\": False,\n",
" \"on(orange,green)\": False,\n",
" \"on(orange,orange)\": False,\n",
"}\n",
"\n",
"CALIBRATION_SCORES = {\n",
" example_id: {\n",
" calibration_prompt(predicate): score\n",
" for predicate, score in predicate_scores.items()\n",
" }\n",
" for example_id, predicate_scores in {\n",
" 1: {\n",
" \"clear(blue)\": 0.80,\n",
" \"clear(orange)\": 0.20,\n",
" \"on(blue,blue)\": 0.05,\n",
" \"on(blue,orange)\": 0.70,\n",
" \"on(orange,blue)\": 0.20,\n",
" \"on(orange,orange)\": 0.05,\n",
" },\n",
" 2: {\n",
" \"clear(blue)\": 0.75,\n",
" \"clear(green)\": 0.70,\n",
" \"clear(orange)\": 0.25,\n",
" \"on(blue,blue)\": 0.05,\n",
" \"on(blue,green)\": 0.25,\n",
" \"on(blue,orange)\": 0.65,\n",
" \"on(green,blue)\": 0.10,\n",
" \"on(green,green)\": 0.05,\n",
" \"on(green,orange)\": 0.15,\n",
" \"on(orange,blue)\": 0.15,\n",
" \"on(orange,green)\": 0.10,\n",
" \"on(orange,orange)\": 0.05,\n",
" },\n",
" 3: {\n",
" \"clear(blue)\": 0.70,\n",
" \"clear(orange)\": 0.30,\n",
" \"on(blue,blue)\": 0.05,\n",
" \"on(blue,orange)\": 0.60,\n",
" \"on(orange,blue)\": 0.20,\n",
" \"on(orange,orange)\": 0.05,\n",
" },\n",
" }.items()\n",
"}\n",
"\n",
"\n",
"class CalibrationTutorialVLM(VLMBackend):\n",
" \"\"\"Reads an example id out of the image and answers from a score table.\"\"\"\n",
"\n",
" def query(self, images, prompt, system_prompt=None, generate=False, interest_tokens=None, **inference_kwargs):\n",
" example_id = images[0].getpixel((0, 0))[0]\n",
" return binary_output(CALIBRATION_SCORES[example_id][prompt], interest_tokens)\n",
"\n",
"\n",
"def calibration_demo_image(example_id: int):\n",
" return [Image.new(\"RGB\", (2, 2), color=(example_id, 0, 0))]\n",
"\n",
"\n",
"calibration_estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=CalibrationTutorialVLM(),\n",
" translator=TemplateTranslator(TEMPLATE_MAP),\n",
")\n",
"\n",
"held_out_examples = [\n",
" CalibrationExample(\n",
" images=calibration_demo_image(1),\n",
" state_dict=CALIBRATION_LABELS_2OBJ,\n",
" ),\n",
" CalibrationExample(\n",
" images=calibration_demo_image(2),\n",
" state_dict=CALIBRATION_LABELS_3OBJ,\n",
" problem=COLOR_BLOCKSWORLD_PROBLEM_3, # a different problem instance, same domain\n",
" ),\n",
"]\n",
"\n",
"with tempfile.TemporaryDirectory() as tmpdir:\n",
" # Expensive step: query the VLM once per example, then save the scores.\n",
" data = CalibrationSet.collect(calibration_estimator, held_out_examples)\n",
" data.save(Path(tmpdir) / \"calibration-data.json\")\n",
"\n",
" # Cheap and VLM-free from here on.\n",
" data = CalibrationSet.load(Path(tmpdir) / \"calibration-data.json\")\n",
" global_calibrator = PlattCalibrator.fit(data, scope=\"global\")\n",
" lifted_calibrator = PlattCalibrator.fit(data, scope=\"lifted\")\n",
" lifted_calibrator.save(Path(tmpdir) / \"platt-lifted.json\")\n",
" lifted_calibrator = PlattCalibrator.load(Path(tmpdir) / \"platt-lifted.json\")\n",
"\n",
"assert len(data.samples) == 18\n",
"assert calibration_estimator.predicates == grounded_predicates # restored after collect()\n",
"\n",
"raw_results = calibration_estimator.estimate(calibration_demo_image(3))\n",
"global_results = global_calibrator.apply(raw_results)\n",
"lifted_results = calibration_estimator.estimate(calibration_demo_image(3), calibrator=lifted_calibrator)\n",
"\n",
"print(\"First collected sample:\", data.samples[0])\n",
"print(\"Global calibrator groups:\", global_calibrator.group_keys())\n",
"print(\"Lifted calibrator groups:\", lifted_calibrator.group_keys())\n",
"print()\n",
"print(\"Raw probabilities:\")\n",
"pprint(rounded(raw_results.probabilities()))\n",
"print()\n",
"print(\"Global Platt-scaled probabilities:\")\n",
"pprint(global_results.probabilities())\n",
"print()\n",
"print(\"Lifted-fluent Platt-scaled probabilities:\")\n",
"pprint(lifted_results.probabilities())"
]
},
{
"cell_type": "markdown",
"id": "ca5cf1aa",
"metadata": {},
"source": [
"Two details are easy to miss. First, the second example carried its own `problem`: `CalibrationSet.collect` re-grounded the estimator against it while collecting and restored the original problem afterwards, which is how you pool calibration data across problem instances of one domain. Second, a calibrated `PredictionSet` still has the original `masses`; only `probability` is overridden, so you can always look back at what the model actually said."
]
},
{
"cell_type": "markdown",
"id": "763dac3a",
"metadata": {},
"source": [
"## 7. Choosing a translator strategy\n",
"\n",
"Think of translators as the bridge between planner language and model language. The planner thinks in predicates. The model would usually prefer a question. The translator is the part that keeps those two worlds from talking past each other.\n",
"\n",
"The practical rule of thumb is simple: start with `TemplateTranslator`, drop to `IdentityTranslator` only when literal predicates are what you want, reach for `PrewrittenTranslator` when every prompt needs careful hand-authoring, and use `LLMTranslator` when you want the library to draft those prompts for you.\n",
"\n",
"One subtle but important detail: translator choice also changes the estimator's default answer space and system prompt. Literal predicate prompting scores `true`/`false` and, via `from_pddl`, gets a domain-aware system prompt that spells out the PDDL domain and objects; natural-language prompting scores `yes`/`no`."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "705f2a07",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:34.098703Z",
"iopub.status.busy": "2026-10-07T07:15:34.098071Z",
"iopub.status.idle": "2026-10-07T07:15:34.214366Z",
"shell.execute_reply": "2026-10-07T07:15:34.213062Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Identity translator example:\n",
"[('on(blue,blue)', 'on(blue,blue)'), ('on(blue,orange)', 'on(blue,orange)')]\n",
"\n",
"Prewritten translator example:\n",
"[('on(blue,blue)', 'Is the blue block stacked on itself?'),\n",
" ('on(blue,orange)', 'Is the blue block on top of the orange block?')]\n",
"\n",
"Answer labels without translation: ['true', 'false']\n",
"Answer labels with translation: ['yes', 'no']\n",
"\n",
"Domain-aware system prompt for identity translation:\n",
"The following is a PDDL domain\n",
"(define (domain color-bw-2-domain)\n",
" (:requirements :strips :typing)\n",
" (:types block)\n",
" (:predicates \n",
" (on ?x - block ?y - block)\n",
" (clear ?x - block)\n",
" )\n",
")\n",
"\n",
"Here are the names of all the objects in the current problem, sorted by their type:\n",
"block type: ['blue', 'orange']\n",
"Given a grounded predicate with concrete variables, state whether the statement is true or false.\n",
"Respond only with a \"true\" or \"false\" response and nothing else.\n"
]
}
],
"source": [
"identity_estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=TutorialSceneVLM(),\n",
")\n",
"\n",
"prewritten_queries = {\n",
" \"on(blue,blue)\": \"Is the blue block stacked on itself?\",\n",
" \"on(blue,orange)\": \"Is the blue block on top of the orange block?\",\n",
" \"on(orange,blue)\": \"Is the orange block on top of the blue block?\",\n",
" \"on(orange,orange)\": \"Is the orange block stacked on itself?\",\n",
" \"clear(blue)\": \"Is the blue block clear?\",\n",
" \"clear(orange)\": \"Is the orange block clear?\",\n",
"}\n",
"prewritten_estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=TutorialSceneVLM(),\n",
" translator=PrewrittenTranslator(prewritten_queries),\n",
")\n",
"\n",
"assert isinstance(identity_estimator.translator, IdentityTranslator)\n",
"\n",
"print(\"Identity translator example:\")\n",
"pprint(list(identity_estimator.queries.items())[:2])\n",
"print()\n",
"print(\"Prewritten translator example:\")\n",
"pprint(list(prewritten_estimator.queries.items())[:2])\n",
"print()\n",
"print(\"Answer labels without translation:\", identity_estimator.engine.answers.labels)\n",
"print(\"Answer labels with translation: \", prewritten_estimator.engine.answers.labels)\n",
"print()\n",
"print(\"Domain-aware system prompt for identity translation:\")\n",
"print(identity_estimator.engine.system_prompt)"
]
},
{
"cell_type": "markdown",
"id": "c7833fcd",
"metadata": {},
"source": [
"### A quick note on backend kwargs\n",
"\n",
"There are two different kwargs channels, and keeping them separate will save you a lot of confusion:\n",
"\n",
"- `vlm_kwargs` are for backend construction time (model/client setup). They only apply when `vlm` is a model string; passing them alongside an already-constructed backend instance raises, rather than silently dropping them.\n",
"- `inference_kwargs` are for per-query runtime inference options. Set defaults on the estimator, and override them per call with `estimate(..., inference_kwargs=...)`.\n",
"\n",
"For custom backends, this means your `query(...)` or `query_batch(...)` implementation should accept `generate`, `interest_tokens`, and `**inference_kwargs`, and apply `inference_kwargs` during the actual model call.\n",
"\n",
"In practical terms: `scoring=\"logprobs\"` drives `generate=False` with `interest_tokens` set, while `scoring=\"text_match\"` drives `generate=True` with `interest_tokens=None`."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "38943e7b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:34.217302Z",
"iopub.status.busy": "2026-10-07T07:15:34.216966Z",
"iopub.status.idle": "2026-10-07T07:15:34.315014Z",
"shell.execute_reply": "2026-10-07T07:15:34.313104Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"First logprobs call (generate should be False, interest tokens set):\n",
"{'generate': False,\n",
" 'inference_kwargs': {'temperature': 0.2},\n",
" 'interest_tokens': ['yes',\n",
" ' yes',\n",
" 'Yes',\n",
" ' Yes',\n",
" 'YES',\n",
" ' YES',\n",
" 'no',\n",
" ' no',\n",
" 'No',\n",
" ' No',\n",
" 'NO',\n",
" ' NO']}\n",
"Per-call override wins over the estimator default: {'temperature': 0.0}\n",
"output: {'on(blue,blue)': 0.8, 'on(blue,orange)': 0.8, 'on(orange,blue)': 0.8, 'on(orange,orange)': 0.8, 'clear(blue)': 0.8, 'clear(orange)': 0.8}\n",
"\n",
"First text_match call (generate should be True, no interest tokens):\n",
"{'generate': True,\n",
" 'inference_kwargs': {'max_new_tokens': 4},\n",
" 'interest_tokens': None}\n",
"output: {'on(blue,blue)': 1.0, 'on(blue,orange)': 1.0, 'on(orange,blue)': 1.0, 'on(orange,orange)': 1.0, 'clear(blue)': 1.0, 'clear(orange)': 1.0}\n"
]
}
],
"source": [
"class DebugKwargsVLM(VLMBackend):\n",
" \"\"\"Tiny backend that records mode, interest tokens, and inference kwargs.\"\"\"\n",
"\n",
" def __init__(self):\n",
" self.calls = []\n",
"\n",
" def query(self, images, prompt, system_prompt=None, generate=False, interest_tokens=None, **inference_kwargs):\n",
" self.calls.append({\n",
" \"generate\": generate,\n",
" \"interest_tokens\": interest_tokens,\n",
" \"inference_kwargs\": dict(inference_kwargs),\n",
" })\n",
" if generate:\n",
" return VLMOutput(token_probs=None, text=\"yes\")\n",
" return binary_output(0.8, interest_tokens)\n",
"\n",
"\n",
"logprobs_debug_vlm = DebugKwargsVLM()\n",
"logprobs_debug_estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=logprobs_debug_vlm,\n",
" translator=TemplateTranslator(TEMPLATE_MAP),\n",
" inference_kwargs={\"temperature\": 0.2},\n",
")\n",
"out_prob = logprobs_debug_estimator.estimate([scene_image]).probabilities()\n",
"_ = logprobs_debug_estimator.estimate([scene_image], inference_kwargs={\"temperature\": 0.0})\n",
"\n",
"text_match_debug_vlm = DebugKwargsVLM()\n",
"text_match_debug_estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=text_match_debug_vlm,\n",
" translator=TemplateTranslator(TEMPLATE_MAP),\n",
" scoring=\"text_match\",\n",
" inference_kwargs={\"max_new_tokens\": 4},\n",
")\n",
"out_text = text_match_debug_estimator.estimate([scene_image]).probabilities()\n",
"\n",
"assert logprobs_debug_vlm.calls[0][\"generate\"] is False\n",
"assert logprobs_debug_vlm.calls[-1][\"inference_kwargs\"] == {\"temperature\": 0.0}\n",
"assert text_match_debug_vlm.calls[0][\"generate\"] is True\n",
"\n",
"print(\"First logprobs call (generate should be False, interest tokens set):\")\n",
"pprint(logprobs_debug_vlm.calls[0])\n",
"print(\"Per-call override wins over the estimator default:\", logprobs_debug_vlm.calls[-1][\"inference_kwargs\"])\n",
"print(\"output:\", rounded(out_prob))\n",
"print()\n",
"print(\"First text_match call (generate should be True, no interest tokens):\")\n",
"pprint(text_match_debug_vlm.calls[0])\n",
"print(\"output:\", rounded(out_text))"
]
},
{
"cell_type": "markdown",
"id": "3dfe9155",
"metadata": {},
"source": [
"## 8. Multiple scenes, averaging, and batching\n",
"\n",
"This is where the cost model starts to matter. Every grounded predicate is a question, and every image is another piece of evidence, so runtime can grow faster than you expect.\n",
"\n",
"S3E gives you a few levers to manage that. A *scene* is a list of images shown to the model together in one query. To combine evidence from several scenes of the same state, `estimate_averaged(scenes)` estimates each scene separately and averages the stored masses (`PredictionSet.average` does the same for any prediction sets over the same keys). And `batch_size` controls how many predicate queries travel together in one backend call.\n",
"\n",
"The examples below use two views of the same scene: one straight-on, one from a different angle."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "615be83c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:34.319186Z",
"iopub.status.busy": "2026-10-07T07:15:34.318946Z",
"iopub.status.idle": "2026-10-07T07:15:34.365412Z",
"shell.execute_reply": "2026-10-07T07:15:34.363817Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Head-on view: 0.9\n",
"Angled view: 0.6\n",
"\n",
"Averaged probabilities across both views:\n",
"{'clear(blue)': 0.75,\n",
" 'clear(orange)': 0.25,\n",
" 'on(blue,blue)': 0.25,\n",
" 'on(blue,orange)': 0.75,\n",
" 'on(orange,blue)': 0.25,\n",
" 'on(orange,orange)': 0.25}\n"
]
}
],
"source": [
"class ViewAwareVLM(VLMBackend):\n",
" def query(self, images, prompt, system_prompt=None, generate=False, interest_tokens=None, **inference_kwargs):\n",
" image = images[0]\n",
" angled_view = image.getpixel((225, 90)) != (255, 255, 255)\n",
" if \"blue block on top of the orange block\" in prompt or \"blue block clear\" in prompt:\n",
" yes_prob = 0.6 if angled_view else 0.9\n",
" else:\n",
" yes_prob = 0.4 if angled_view else 0.1\n",
" return binary_output(yes_prob, interest_tokens)\n",
"\n",
"\n",
"average_estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=ViewAwareVLM(),\n",
" translator=TemplateTranslator(TEMPLATE_MAP),\n",
")\n",
"head_on = average_estimator.estimate([scene_image])\n",
"angled = average_estimator.estimate([angled_scene_image])\n",
"averaged = average_estimator.estimate_averaged([[scene_image], [angled_scene_image]])\n",
"\n",
"assert abs(averaged[\"on(blue,orange)\"].probability - 0.75) < 1e-9\n",
"assert averaged.probabilities() == PredictionSet.average([head_on, angled]).probabilities()\n",
"\n",
"print(\"Head-on view: \", round(head_on[\"on(blue,orange)\"].probability, 4))\n",
"print(\"Angled view: \", round(angled[\"on(blue,orange)\"].probability, 4))\n",
"print()\n",
"print(\"Averaged probabilities across both views:\")\n",
"pprint(rounded(averaged.probabilities()))"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "a02a87f9",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:34.368834Z",
"iopub.status.busy": "2026-10-07T07:15:34.368655Z",
"iopub.status.idle": "2026-10-07T07:15:34.686258Z",
"shell.execute_reply": "2026-10-07T07:15:34.685267Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"With 6 grounded predicates and batch_size=2, the estimator made 3 batch calls.\n"
]
}
],
"source": [
"class BatchCountingVLM(VLMBackend):\n",
" def __init__(self):\n",
" self.batch_calls = 0\n",
"\n",
" def query(self, images, prompt, system_prompt=None, generate=False, interest_tokens=None, **inference_kwargs):\n",
" raise RuntimeError(\"query() should not be used when query_batch() is overridden\")\n",
"\n",
" def query_batch(self, images, prompts, system_prompt=None, generate=False, interest_tokens=None, **inference_kwargs):\n",
" self.batch_calls += 1\n",
" return [\n",
" binary_output(0.8 if \"blue block\" in prompt else 0.2, interest_tokens)\n",
" for prompt in prompts\n",
" ]\n",
"\n",
"\n",
"batch_vlm = BatchCountingVLM()\n",
"batch_estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=batch_vlm,\n",
" translator=TemplateTranslator(TEMPLATE_MAP),\n",
" batch_size=2,\n",
")\n",
"_ = batch_estimator.estimate([scene_image])\n",
"\n",
"assert batch_vlm.batch_calls == 3\n",
"print(\"With 6 grounded predicates and batch_size=2, the estimator made\", batch_vlm.batch_calls, \"batch calls.\")"
]
},
{
"cell_type": "markdown",
"id": "23b0946f",
"metadata": {},
"source": [
"### Getting the most out of HuggingFace batching\n",
"\n",
"`batch_size` controls how many prompts the `QueryEngine` sends to `vlm.query_batch(...)` at once. The toy backend above only counts those calls, but `HuggingFaceVLM` uses the batch for real work. For each batch, it renders the prompt text for every query, calls the HuggingFace processor once with `text=[...]` and the same image set for every row, then runs one batched model call.\n",
"\n",
"That applies to both scoring modes:\n",
"\n",
"- `scoring=\"logprobs\"`: one batched forward pass, then the mass on each interest token is gathered per prompt.\n",
"- `scoring=\"text_match\"`: one batched `generate(...)` call, then generated answers are decoded back into one `VLMOutput` per prompt.\n",
"\n",
"A practical HuggingFace setup usually looks like this:\n",
"\n",
"```python\n",
"hf_vlm = HuggingFaceVLM(TINY_HF_VLM_ID, device_map=device)\n",
"\n",
"hf_estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=hf_vlm,\n",
" translator=TemplateTranslator(TEMPLATE_MAP),\n",
" prompt_template=\"Answer with exactly one word, yes or no: {query}\",\n",
" batch_size=6, # all 6 predicates in this toy problem fit in one backend batch\n",
")\n",
"```\n",
"\n",
"To make batching pay off in real runs:\n",
"\n",
"- Start with a modest `batch_size` such as 4 or 8, then increase it until GPU/CPU memory becomes the bottleneck.\n",
"- If all grounded predicates fit comfortably, set `batch_size` at least as large as the predicate count so the engine can use one backend batch.\n",
"- In logprobs mode the engine asks the backend for exactly the answer space's tokens (`interest_tokens`), so the masses are exact and nothing else is detokenized. `HuggingFaceVLM(num_logprobs=...)` only matters when you query the backend directly without interest tokens: finite values return only the most likely tokens, while `None` returns all vocabulary probabilities, which is useful for deep debugging but slower.\n",
"- Do not loop over predicates manually. Let `estimate(...)` or `estimator(...)` batch the prompts for you; use `predicates=[...]` only when you intentionally want a smaller subset.\n",
"- Predicates that translate to the same query are asked once and share the prediction.\n",
"- Several images in one scene are attached to every prompt row in the batch. `estimate_averaged` runs separate estimations per scene and averages the results, which can be more robust but costs more model calls.\n",
"\n",
"The batching path is deliberately model-agnostic. It does not use KV-cache tricks or model-family-specific assumptions, and if a processor cannot handle batched multimodal inputs, the backend falls back to per-prompt processing for correctness.\n",
"\n",
"And when careful batching still is not enough -- large predicate counts, multiple GPUs going to waste -- that is your cue to jump to section 12 and let vLLM do the heavy lifting."
]
},
{
"cell_type": "markdown",
"id": "d645803a",
"metadata": {},
"source": [
"## 9. Swapping to a new planning problem\n",
"\n",
"Imagine your planner moves on to a new problem instance: same domain, different objects, different grounded predicates. You do not necessarily need to rebuild everything from scratch.\n",
"\n",
"`set_problem(...)` is the quick way to re-ground the planning problem and refresh the query set; the engine and its backend are untouched, so a loaded model stays loaded. The catch is that it leaves your prompt template and answer space alone. If those settings were generic, great. If they were tightly tied to the old problem, rebuild the estimator and save yourself a confusing afternoon."
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "c6fed627",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:34.689751Z",
"iopub.status.busy": "2026-10-07T07:15:34.689592Z",
"iopub.status.idle": "2026-10-07T07:15:34.761718Z",
"shell.execute_reply": "2026-10-07T07:15:34.760945Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Grounded predicates: 6 -> 12\n",
"A few queries after the swap:\n",
"[('on(blue,blue)', 'Is the blue block on top of the blue block?'),\n",
" ('on(blue,orange)', 'Is the blue block on top of the orange block?'),\n",
" ('on(blue,green)', 'Is the blue block on top of the green block?'),\n",
" ('on(orange,blue)', 'Is the orange block on top of the blue block?'),\n",
" ('on(orange,orange)', 'Is the orange block on top of the orange block?')]\n"
]
}
],
"source": [
"swap_demo = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=TutorialSceneVLM(),\n",
" translator=TemplateTranslator(TEMPLATE_MAP),\n",
")\n",
"backend_before = swap_demo.engine.backend\n",
"before_count = len(swap_demo.queries)\n",
"swap_demo.set_problem(COLOR_BLOCKSWORLD_DOMAIN, COLOR_BLOCKSWORLD_PROBLEM_3)\n",
"after_count = len(swap_demo.queries)\n",
"\n",
"assert before_count == 6\n",
"assert after_count == 12\n",
"assert swap_demo.engine.backend is backend_before\n",
"\n",
"print(f\"Grounded predicates: {before_count} -> {after_count}\")\n",
"print(\"A few queries after the swap:\")\n",
"pprint(list(swap_demo.queries.items())[:5])"
]
},
{
"cell_type": "markdown",
"id": "56b5b18d",
"metadata": {},
"source": [
"## 10. Real HuggingFace integration with a small instruct model\n",
"\n",
"Time to take the training wheels off. Everything up to this point was about learning the workflow cleanly; now we plug in a real `HuggingFaceVLM` and run the same pipeline for real.\n",
"\n",
"For this notebook, I picked `HuggingFaceTB/SmolVLM-256M-Instruct`. It is not a powerhouse, but it *does* follow simple instructions, and it is still small enough to be a reasonable CPU-first demo.\n",
"\n",
"Notice that we do not hand-write any token lists: the default `BinaryAnswers()` already covers `yes`, ` Yes`, `YES`, and friends, and the backend gathers mass at the matching vocabulary ids. `argmax_in_interest` on the raw output tells you whether the model's top token was actually one of them.\n",
"\n",
"Set expectations accordingly: this section is about integration and behavior, not about showing off world-class visual reasoning."
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "742be0d6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:34.765228Z",
"iopub.status.busy": "2026-10-07T07:15:34.765083Z",
"iopub.status.idle": "2026-10-07T07:15:42.451796Z",
"shell.execute_reply": "2026-10-07T07:15:42.450856Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loading the tiny HuggingFace VLM (first run may download weights)...\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r",
"Loading weights: 0%| | 0/471 [00:00 ... ` for the Qwen3 family) and only then reply. Some open that block by default, Qwen3.5-4B and larger through their chat template and Mistral's reasoning models through their default system prompt; others, such as Gemma 4, think when asked to.\n",
"\n",
"That collides with `scoring=\"logprobs\"`, which reads the probability of the *very next token* as the answer. When the prompt ends inside an open `` block, the next token is the first word of the reasoning, not `yes` or `no`.\n",
"\n",
"To see it happen, we will use `Qwen/Qwen3.5-0.8B` (about 1.7 GB to download). Its chat template leaves thinking off, but the template variable `enable_thinking=True` makes it behave the way its larger siblings do by default. The two renderings differ only in their last few characters:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "fbb94fc2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:42.456020Z",
"iopub.status.busy": "2026-10-07T07:15:42.455769Z",
"iopub.status.idle": "2026-10-07T07:15:43.387056Z",
"shell.execute_reply": "2026-10-07T07:15:43.385413Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"thinking off (default): ...'>assistant\\n\\n\\n\\n\\n'\n",
"thinking on: ...'<|im_start|>assistant\\n\\n'\n"
]
}
],
"source": [
"from transformers import AutoProcessor\n",
"\n",
"THINKING_VLM_ID = \"Qwen/Qwen3.5-0.8B\"\n",
"THINKING_ON = {\"enable_thinking\": True} # what Qwen3.5-4B and larger do by default\n",
"\n",
"thinking_processor = AutoProcessor.from_pretrained(THINKING_VLM_ID)\n",
"conversation = [\n",
" {\"role\": \"user\", \"content\": [{\"type\": \"image\"}, {\"type\": \"text\", \"text\": \"Is the blue block clear?\"}]}\n",
"]\n",
"for label, options in [(\"thinking off (default):\", {}), (\"thinking on:\", THINKING_ON)]:\n",
" rendered = thinking_processor.apply_chat_template(\n",
" conversation, add_generation_prompt=True, tokenize=False, **options\n",
" )\n",
" print(f\"{label:24} ...{rendered[-30:]!r}\")"
]
},
{
"cell_type": "markdown",
"id": "b4634330",
"metadata": {},
"source": [
"With thinking on, the prompt ends inside an open `` block; with thinking off, it carries an empty block that is already closed.\n",
"\n",
"First, what happens when the backend ignores the block. `thinking_delimiters=None` switches S3E's thinking handling off, so the backend scores the very next token, as it would for any other model. `chat_template_kwargs` passes variables to the chat template, here turning thinking on:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "52da595d",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:43.390890Z",
"iopub.status.busy": "2026-10-07T07:15:43.390667Z",
"iopub.status.idle": "2026-10-07T07:15:48.517029Z",
"shell.execute_reply": "2026-10-07T07:15:48.515105Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Thinking on, block ignored:\n",
" on(blue,blue) P(true) = 0.077 mass on yes/no = 2.9e-06 argmax_in_interest = False\n",
" on(blue,orange) P(true) = 0.117 mass on yes/no = 3.0e-06 argmax_in_interest = False\n",
" on(orange,blue) P(true) = 0.099 mass on yes/no = 2.5e-06 argmax_in_interest = False\n",
" on(orange,orange) P(true) = 0.099 mass on yes/no = 2.2e-06 argmax_in_interest = False\n",
" clear(blue) P(true) = 0.079 mass on yes/no = 5.5e-06 argmax_in_interest = False\n",
" clear(orange) P(true) = 0.081 mass on yes/no = 4.8e-06 argmax_in_interest = False\n"
]
}
],
"source": [
"import gc\n",
"import time\n",
"\n",
"# Four model loads ahead; keep their progress bars out of the output.\n",
"hf_logging.disable_progress_bar()\n",
"# Qwen3.5 is trained in bfloat16, which float16 (the GPU default) can overflow.\n",
"thinking_dtype = (\n",
" torch.bfloat16 if device == \"cuda\" and torch.cuda.is_bf16_supported() else torch.float32\n",
")\n",
"\n",
"\n",
"def run_thinking_estimator(predicates=None, **vlm_options):\n",
" \"\"\"Estimate the toy scene with a fresh Qwen3.5-0.8B backend, then free it.\n",
"\n",
" Returns the predictions and the seconds the estimate itself took.\n",
" \"\"\"\n",
" vlm = HuggingFaceVLM(\n",
" THINKING_VLM_ID,\n",
" device_map=device,\n",
" torch_dtype=thinking_dtype,\n",
" # Qwen3.5 answers the same padded or not; skipping the one-time check\n",
" # keeps the timings comparable.\n",
" skip_pad_invariance_check=True,\n",
" **vlm_options,\n",
" )\n",
" estimator = SemanticStateEstimator.from_pddl(\n",
" COLOR_BLOCKSWORLD_DOMAIN,\n",
" COLOR_BLOCKSWORLD_PROBLEM,\n",
" vlm=vlm,\n",
" translator=TemplateTranslator(TEMPLATE_MAP),\n",
" prompt_template=\"Answer with exactly one word, yes or no: {query}\",\n",
" batch_size=6,\n",
" )\n",
" # Pay the one-time costs before timing: the first reply on the device, and\n",
" # the token index the first scored query builds over the whole vocabulary.\n",
" vlm.query_batch([scene_image], [\"Describe the image.\"] * 6, generate=True, max_new_tokens=1)\n",
" vlm.unsupported_interest_tokens([\"yes\"])\n",
" start = time.perf_counter()\n",
" results = estimator.estimate([scene_image], predicates=predicates, keep_raw=True)\n",
" seconds = time.perf_counter() - start\n",
" del vlm, estimator # free the weights before the next configuration loads\n",
" gc.collect()\n",
" if device == \"cuda\":\n",
" torch.cuda.empty_cache()\n",
" return results, seconds\n",
"\n",
"\n",
"block_ignored, _ = run_thinking_estimator(\n",
" chat_template_kwargs=THINKING_ON, thinking_delimiters=None\n",
")\n",
"\n",
"assert all(prediction.unassigned_mass > 0.9 for prediction in block_ignored.values())\n",
"\n",
"print(\"Thinking on, block ignored:\")\n",
"for predicate, prediction in block_ignored.items():\n",
" print(\n",
" f\" {predicate:18} P(true) = {prediction.probability:.3f} \"\n",
" f\"mass on yes/no = {1 - prediction.unassigned_mass:.1e} \"\n",
" f\"argmax_in_interest = {prediction.raw.argmax_in_interest}\"\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "0f88a893",
"metadata": {},
"source": [
"That is a silent failure, and exactly what the `keep_raw=True` habit from section 10 is for. Almost no probability lands on `yes` or `no` (a few millionths, and `argmax_in_interest` is `False`), yet every predicate still gets a probability: `T / (T + F)` of two tiny masses, which is noise. The resulting state calls every predicate `False`, including the two that hold, and no warning fires, because the masses are tiny but not exactly zero.\n",
"\n",
"`HuggingFaceVLM` and `VLLMBackend` take a `thinking` flag instead. With the default `thinking=False`, the model answers without reasoning. Qwen3.5's template has a switch, so the backend simply turns it off: it finds it when it is constructed by rendering a probe conversation with each known template variable (`enable_thinking` here). Models without a switch get a fallback: the backend watches for a thinking block (the prompt ends by opening one, or the model's most likely next token opens one), closes it at once, and scores the answer after the empty block.\n",
"\n",
"We can watch the fallback by forcing the template's thinking on ourselves, since `chat_template_kwargs` override the switch. For Qwen3.5 the closed block is exactly the prompt its template renders with thinking off, so the two give the same answers. `VLMOutput.reasoning` records what happened: `\"\"` for a block closed without reasoning, `None` when there was no block."
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "8086165f",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:48.520720Z",
"iopub.status.busy": "2026-10-07T07:15:48.520482Z",
"iopub.status.idle": "2026-10-07T07:15:58.486804Z",
"shell.execute_reply": "2026-10-07T07:15:58.484954Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"P(true) block ignored block closed thinking off\n",
"on(blue,blue) 0.0765 0.4622 0.4622\n",
"on(blue,orange) 0.1169 0.7662 0.7662\n",
"on(orange,blue) 0.0989 0.4859 0.4859\n",
"on(orange,orange) 0.0988 0.3891 0.3891\n",
"clear(blue) 0.0792 0.6684 0.6684\n",
"clear(orange) 0.0814 0.4740 0.4740\n",
"\n",
"Seconds per predicate: thinking off 0.05, block closed 0.15\n"
]
}
],
"source": [
"# thinking=False (the default) closes the block the forced-on template opens...\n",
"closed, closed_seconds = run_thinking_estimator(chat_template_kwargs=THINKING_ON)\n",
"# ...and otherwise turns the template's switch off.\n",
"thinking_off, off_seconds = run_thinking_estimator()\n",
"\n",
"for predicate in closed:\n",
" assert abs(closed[predicate].probability - thinking_off[predicate].probability) < 1e-4\n",
"assert all(prediction.raw.reasoning == \"\" for prediction in closed.values())\n",
"assert all(prediction.raw.reasoning is None for prediction in thinking_off.values())\n",
"\n",
"print(f\"{'P(true)':18} {'block ignored':>13} {'block closed':>12} {'thinking off':>12}\")\n",
"for predicate in closed:\n",
" print(\n",
" f\"{predicate:18} {block_ignored[predicate].probability:13.4f} \"\n",
" f\"{closed[predicate].probability:12.4f} {thinking_off[predicate].probability:12.4f}\"\n",
" )\n",
"print()\n",
"print(\n",
" f\"Seconds per predicate: thinking off {off_seconds / len(thinking_off):.2f}, \"\n",
" f\"block closed {closed_seconds / len(closed):.2f}\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "774852e8",
"metadata": {},
"source": [
"### Letting the model think\n",
"\n",
"Closing the block throws the model's reasoning away. `thinking=True` keeps it: the backend turns the switch on, the model reasons first, greedily and for up to `max_thinking_tokens` tokens (1024 by default, after which the block is closed for it), and the answer is scored after its reasoning. The reasoning lands in `VLMOutput.reasoning`. (A warning says when a model offers no way to think.) We cap the budget at 256 tokens and compare the answers with the scene's true state:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "e74205c5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:15:58.490408Z",
"iopub.status.busy": "2026-10-07T07:15:58.490181Z",
"iopub.status.idle": "2026-10-07T07:16:14.359054Z",
"shell.execute_reply": "2026-10-07T07:16:14.357201Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"P(true) thinking off thinking truth\n",
"on(blue,blue) 0.4622 0.0409 False\n",
"on(blue,orange) 0.7662 0.9993 True\n",
"on(orange,blue) 0.4859 0.9985 False\n",
"on(orange,orange) 0.3891 0.1019 False\n",
"clear(blue) 0.6684 0.4833 True\n",
"clear(orange) 0.4740 0.2743 False\n",
"\n",
"Thinking off: 6 of 6 predicates right at confidence 0.5\n",
"Thinking: 4 of 6 predicates right at confidence 0.5\n",
"Seconds per predicate: thinking off 0.05, block closed 0.15, thinking 1.91\n",
"\n",
"Reasoning behind on(blue,orange):\n",
" The user wants me to answer a yes/no question based on an image.\n",
"\n",
" 1. **Analyze the image:**\n",
" * There is a blue square block.\n",
" * There is an orange square block below it.\n",
" * The blue block is positioned directly on top of the orange block.\n",
"\n",
" 2. **Analyze the question:** \"Is the blue block on top of the orange block?\"\n",
"\n",
" 3. **Formulate the answer:**\n",
" * The visual evidence clearly shows the blue block is on top of the orange block.\n",
" * The user requested the answer be exactly one word: \"yes\" or \"no\".\n",
" * The answer is \"yes\".\n",
"\n",
" 4. **Final check:** Does the response incl ...\n",
"\n",
"Reasoning behind on(orange,blue):\n",
" The user wants me to answer a yes/no question based on an image.\n",
"\n",
" 1. **Analyze the image:**\n",
" * There is a blue square.\n",
" * There is an orange square.\n",
" * The orange square is positioned directly on top of the blue square.\n",
"\n",
" 2. **Analyze the question:** \"Is the orange block on top of the blue block?\"\n",
"\n",
" 3. **Formulate the answer:**\n",
" * The visual evidence shows the orange block is indeed on top of the blue block.\n",
" * The user requested the answer be exactly one word: \"yes\" or \"no\".\n",
" * The answer is \"yes\".\n",
"\n",
" 4. **Final check:** Does the response include any additional t ...\n"
]
}
],
"source": [
"thought, thought_seconds = run_thinking_estimator(thinking=True, max_thinking_tokens=256)\n",
"truth = CALIBRATION_LABELS_2OBJ # the toy scene's true state, from section 6\n",
"\n",
"assert all(prediction.raw.reasoning for prediction in thought.values())\n",
"\n",
"print(f\"{'P(true)':18} {'thinking off':>12} {'thinking':>9} {'truth':>6}\")\n",
"for predicate in thought:\n",
" print(\n",
" f\"{predicate:18} {thinking_off[predicate].probability:12.4f} \"\n",
" f\"{thought[predicate].probability:9.4f} {str(truth[predicate]):>6}\"\n",
" )\n",
"print()\n",
"for name, results in [(\"Thinking off\", thinking_off), (\"Thinking\", thought)]:\n",
" right = sum(results.to_state()[predicate] == label for predicate, label in truth.items())\n",
" print(f\"{name}: {right} of {len(truth)} predicates right at confidence 0.5\")\n",
"print(\n",
" f\"Seconds per predicate: thinking off {off_seconds / len(thinking_off):.2f}, \"\n",
" f\"block closed {closed_seconds / len(closed):.2f}, thinking {thought_seconds / len(thought):.2f}\"\n",
")\n",
"for predicate in [\"on(blue,orange)\", \"on(orange,blue)\"]:\n",
" print()\n",
" print(f\"Reasoning behind {predicate}:\")\n",
" print(textwrap.indent(thought[predicate].raw.reasoning[:600].rstrip() + \" ...\", \" \"))"
]
},
{
"cell_type": "markdown",
"id": "560b6ce5",
"metadata": {},
"source": [
"### Accuracy versus cost\n",
"\n",
"Thinking trades compute for a chance at better answers. On questions that take several steps (counting, spatial relations among many objects, partial occlusion), a model that reasons first can answer better than one that has to commit to its first token, and models trained to reason may lose accuracy when you turn their thinking off. Whether that pays off for your domain is an empirical question, and the run above shows both sides of it:\n",
"\n",
"- **Cost.** Every grounded predicate is its own question. With thinking off, each one costs a single forward pass. For a model without a switch, closing the block adds two short passes per query (a few greedy steps that find the whitespace after the block, then the scoring pass): about 3× in the run saved here, on one GPU. Thinking adds a generation of up to `max_thinking_tokens` tokens per predicate: about 40× here, and more with a larger budget or a more talkative model. Multiply that by the 110 predicates of a 10-block blocksworld, observed at every step of a plan, and thinking can dominate the run time of the whole pipeline. Batching helps (the HuggingFace backend reasons for a whole batch at once, and vLLM schedules requests continuously), but it cannot remove the extra tokens.\n",
"- **Accuracy.** In the run saved here, thinking made the model right and nearly certain about `on(blue,orange)` (0.7662 to 0.9993), but just as certain, and wrong, about `on(orange,blue)` (0.4859 to 0.9985): its reasoning describes the orange block on top. With thinking, 4 of the 6 predicates come out right at confidence 0.5; the hesitant thinking-off answers get all 6. A 0.8B model is a weak reasoner and larger models usually gain more from thinking, but the lesson holds at any size: reasoning can talk a model into a wrong answer as easily as into a right one.\n",
"- **Confidence.** After reasoning, the model has usually committed, so its probabilities sit near 0 or 1. For a planner, a confident mistake is worse than a hesitant one: a stricter `confidence` threshold no longer catches it. A calibrator fitted on held-out examples (section 6) is how you find out how often those near-certain answers are actually right.\n",
"- **Budget.** Greedy reasoning can repeat itself until the budget runs out, and an answer read after a block closed at the budget follows unfinished reasoning. A tighter budget saves time but may cut the reasoning off before it reaches a conclusion.\n",
"\n",
"A practical rule of thumb:\n",
"\n",
"1. Keep the default `thinking=False`. It turns thinking off through the template's switch where there is one, which costs nothing (and here matched closing the block exactly), and closes the block at a modest cost where there is not. Models that never think pay nothing either way.\n",
"2. Use `thinking=True` only where you have measured that it pays: collect labeled examples with thinking on and off (the `CalibrationSet` workflow from section 6 gathers exactly this data), compare their accuracy and calibration, and weigh the difference against the per-predicate cost on your hardware.\n",
"\n",
"The same arguments work on `VLLMBackend`. In text mode (`scoring=\"text_match\"`), `thinking=False` generates the reply after a closed block, and `thinking=True` cuts the block out of the reply and keeps it in `reasoning`; give the model enough `max_new_tokens` (`max_tokens` with vLLM) to finish reasoning, or the reply is `None` and the query has no answer. The README's [Thinking models](https://github.com/CLAIR-LAB-TECHNION/s3e#thinking-models) section lists the supported model families and the formats S3E does not cover."
]
},
{
"cell_type": "markdown",
"id": "6090fccd",
"metadata": {},
"source": [
"## 12. Scaling up with vLLM: many GPUs, one estimator\n",
"\n",
"Before moving on, it is worth pausing on what `estimate` actually does on a *real* problem: it asks one question per grounded predicate. Our toy scene has 6 predicates. A 10-block blocksworld already has 110, and a realistic robotics domain can reach thousands. With plain transformers, every estimator batch becomes one giant forward pass, so activation memory grows linearly with `batch_size`, and however many GPUs you have, that forward pass has a single-GPU-sized ceiling.\n",
"\n",
"This is exactly the workload [vLLM](https://docs.vllm.ai/) was built for, and S3E ships a backend for it. You construct `VLLMBackend(...)` explicitly and pass the instance as `vlm=`:\n",
"\n",
"- The backend loads the model **once** into a persistent engine and, by default, shards it across **every visible GPU** (tensor parallelism). A multi-GPU machine behaves like one big GPU; you never assign layers or devices yourself.\n",
"- Each estimator batch becomes a single `chat` call, and vLLM's continuous batching schedules it internally. You can hand it hundreds of prompts at once without doing any memory math.\n",
"\n",
"One installation note before we run it. The standard `pip install s3e[vllm]` wheel is built for CUDA GPUs; vLLM also publishes a separate [CPU build](https://docs.vllm.ai/en/latest/getting_started/installation/cpu.html) for machines without one (slower, but the same API). The demo cell asks the installed build what it can actually run on -- GPU build with a visible GPU, or CPU build -- and politely skips itself if the answer is neither. Nothing later in the notebook depends on it. If it does run, expect the first call to take a little longer: the engine may download weights and JIT-compile kernels once."
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "28620caf",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-07T07:16:14.363128Z",
"iopub.status.busy": "2026-10-07T07:16:14.362931Z",
"iopub.status.idle": "2026-10-07T07:17:27.768298Z",
"shell.execute_reply": "2026-10-07T07:17:27.766106Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING 10-07 10:16:20 [vllm.py:942] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING 10-07 10:16:20 [vllm.py:960] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"(EngineCore pid=4018190) [transformers] Model config: pad_token_id must be `None` or an integer within the vocabulary (between 0 and 31999), got 128002. This may result in unexpected behavior.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"(EngineCore pid=4018190) WARNING 10-07 10:17:13 [vllm.py:942] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none\n",
"(EngineCore pid=4018190) WARNING 10-07 10:17:13 [vllm.py:960] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"(EngineCore pid=4018190) \r",
"Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00