Changelog

All notable changes to S3E are documented here. The format follows Keep a Changelog, and the project uses semantic versioning (pre-1.0: minor versions may contain breaking changes). Entries for 0.1.0–0.4.1 were reconstructed from the commit history and PyPI release dates when this file was introduced.

Unreleased

Added

  • Runnable examples in examples/: custom_backend.py implements a VLMBackend and runs the whole pipeline offline, including held-out calibration; blocksworld_benchmark.py measures a model on rendered Blocksworld scenes (accuracy, Brier score, ECE, unanswered rate, time per query, and peak memory per scoring mode, plus held-out metrics before and after offline Platt scaling) and writes per-predicate results with provenance to JSON.

  • Documentation: the walkthrough notebook rendered as the tutorial, a user guide, and pages for the examples, the architecture and extension points, related software, and citing; doctested examples on the answer-space, result, translator, and calibrator APIs.

  • SECURITY.md, GOVERNANCE.md, and the release process in CONTRIBUTING.md.

  • Continuous integration on macOS and Windows and at the lowest supported dependency versions, a coverage report, ruff, packaging and CITATION.cff checks, and a weekly workflow that runs the real-model tests, the README Quick Start, and the tutorial notebook on CPU.

  • SPDX license headers in every Python source file.

  • Thinking-model support in HuggingFaceVLM and VLLMBackend: a thinking flag. thinking=False (the default) turns thinking off through the chat template’s switch, found by rendering (enable_thinking, thinking, or Seed’s thinking_budget), and otherwise closes any thinking block at once, so the answer follows an empty block, in logprobs and text mode alike. thinking=True turns the switch on and lets the model reason first (up to max_thinking_tokens in logprobs mode), with a warning when it cannot. Blocks are found by their delimiters, among those of known model families (<think>, Gemma 4’s <|channel>, Mistral’s [THINK], Seed’s <seed:think>) or given (thinking_delimiters). The block’s text is in the new VLMOutput.reasoning, and a chat_template_kwargs argument passes other template variables.

  • A thinking-models section in the walkthrough notebook: Qwen3.5-0.8B with its thinking block ignored, closed, and thought through, and the tradeoff between the accuracy reasoning can add and the time it costs.

Changed

  • Runtime dependencies and extras declare minimum supported versions, which CI tests on Python 3.10 (all but vLLM’s, which needs a GPU), e.g. transformers>=4.56 (for from_pretrained(dtype=...)), openai>=1.66 (for the Responses API used by LLMTranslator), and torch>=2.2. The dev extra adds ruff and pytest-cov; the docs extra uses myst-nb.

  • Breaking: TemplateTranslator.translate raises ValueError, naming the template and predicate, instead of KeyError or IndexError when a placeholder has no matching argument.

  • Breaking: the backend helper modules moved to s3e.backends.utils, so that s3e.backends holds only the backends and their interface: s3e.backends.resolve and s3e.backends.token_index are now s3e.backends.utils.resolve and s3e.backends.utils.token_index. resolve_backend and OPENAI_MODEL_IDENTIFIER are still exported by s3e.backends.

Fixed

  • Thinking models (e.g. Qwen3.5-4B and larger, Mistral reasoning models, and Gemma 4 with thinking enabled) no longer break the HuggingFace and vLLM backends: logprobs mode read the token that opens the reasoning, putting almost no mass on the answer tokens without a warning, and text mode returned the reasoning as the reply.

  • HuggingFaceVLM loads the processor from the same Hub snapshot as the model: revision, cache_dir, token, and the other file-selection kwargs are forwarded to it, so a pinned revision no longer pairs with the processor from main.

  • Translation-cache file names are valid on Windows: <>:"|?* and control characters become _. Cache files whose model id or inference kwargs contain those characters get new names, so their translations are regenerated once.

  • Calibration data, Platt calibrators, and translation caches are read and written as UTF-8 on every platform, not in the locale’s encoding.

0.5.0 - 2026-10-05

Added

  • JOSS submission materials: paper/paper.md and paper/paper.bib, with a GitHub Actions workflow that builds the draft PDF.

  • Sphinx API reference under docs/ (Read the Docs configuration in .readthedocs.yaml) and docstrings for the remaining public calibration, answer-space, and serialization APIs.

  • Continuous integration running the fast test suite on Python 3.10–3.14 and building the documentation.

  • Community files: CONTRIBUTING.md, CODE_OF_CONDUCT.md, issue and pull request templates, and CITATION.cff.

  • .mailmap consolidating the maintainer’s commit identities.

  • MANIFEST.in so the source distribution ships the full test suite.

  • Prediction.has_answer and Prediction.matched, and an UnmatchedAnswerWarning emitted once per QueryEngine.ask call when some queries match no answer option (a text_match reply that starts with no answer token, or zero mass on every answer token).

Changed

  • Breaking: to_state and SemanticStateEstimator.__call__ always return a complete dict[str, bool]: a predicate is True iff P(true) >= confidence, never None. Prediction.answer is never None.

  • Breaking: a prediction has no answer when the explicit null option is its top answer or no answer token received any mass. Its P(true) is 0.5 (a uniform distribution for categorical spaces), even when calibrated, so it is True at the default confidence of 0.5 and False above it.

  • Breaking: for binary spaces, Prediction.answer is probability >= 0.5, so it follows a calibrated probability; previously it was the argmax of the raw masses. Categorical ties, including the uniform distribution of a prediction with no answer, go to the first listed option.

  • Breaking: text_match scoring matches only when the reply starts with an answer token (as a whole word, longest token first, with the null option competing); previously a token anywhere in the reply matched. Replies such as Answer: yes, **Yes**, or a leading <think> block now match nothing.

  • Prediction.probability is T / (T + F) without smoothing, so a matched text_match reply gives exactly 1.0. Calibration scores are unchanged.

  • Prediction.probability raises ValueError for non-binary answer spaces even when a calibrated probability is set.

Fixed

  • CalibrationSet.collect skips predictions with no answer, which are not valid training points.

  • PredictionSet.average averages calibrated probabilities only over members that have an answer.

0.4.1 - 2026-08-30

Architecture redesign into independently usable layers. (The version was bumped to 0.4.0 during development, but 0.4.0 was never published to PyPI; its changes ship in 0.4.1.)

Added

  • QueryEngine: images + free-form queries + an answer space → predictions, with no PDDL involved.

  • Answer spaces (BinaryAnswers, CategoricalAnswers, AnswerOption) with multi-token surface forms and an optional explicit null option.

  • Lazy, immutable Prediction / PredictionSet result objects with JSON round-tripping and PredictionSet.average for multi-view estimation.

  • s3e.calibration package: Calibrator interface, PlattCalibrator, and CalibrationSet / CalibrationExample / CalibrationSample.

  • Public resolve_backend() and helpful ImportErrors that name the extra to install for each optional dependency.

  • dev-gpu extra; contract tests for the local backends (HuggingFace, vLLM) and end-to-end workflow tests.

Changed

  • Breaking: SemanticStateEstimator is now a thin facade (“predicates in, state out”); build it from PDDL with from_pddl or from an explicit predicate list.

  • Breaking: s3e.vlm renamed to s3e.backends; vLLM is selected by passing a VLLMBackend instance instead of a flag.

  • Breaking: torch, torchvision, transformers, accelerate, and unified-planning moved from the core dependencies into the hf and pddl extras, so pip install -U s3e alone no longer brings the HuggingFace backend or PDDL support; install s3e[all] (or the extras you need). import s3e no longer pulls in torch, Unified Planning, scikit-learn, or openai.

Removed

  • Breaking: StateEstimator, ProbabilisticStateEstimator, PredicatePredictionDetails, and PlattCalibrationSample, superseded by QueryEngine, Prediction / PredictionSet, and CalibrationSample.

  • Breaking: estimator methods swap_problem, estimate_prediction_details, estimate_probabilities, estimate_raw, and the *_platt_scaling* family, superseded by set_problem, estimate, and the s3e.calibration package.

  • Breaking: modules s3e.semantic_state_estimator and s3e.state_estimator (now s3e.estimator), s3e.cache (now s3e.translation.cache), and PlattScalingProfile (now PlattCalibrator).

Fixed

  • LLMTranslator puts the domain text, not a .pddl file path, into its prompt.

  • OpenAIVLM honors generate mode instead of always requesting log-probabilities, and fails clearly when log-probabilities are missing.

  • Translation cache keys strip path separators from every part of the name, not only from the model id.

  • Only boolean fluents are grounded as predicates; the Platt sigmoid is overflow-safe; and configuration is validated before any model is loaded.

0.3.2 - 2026-08-04

Added

  • interest_tokens contract for VLM backends: backends report probability mass for exactly the requested answer tokens, matched by token id in the HuggingFace and vLLM backends.

Changed

  • Breaking for custom backends: the estimator now passes interest_tokens= to every backend’s query, so VLMBackend subclasses must accept that argument.

  • The SemanticStateEstimator.use_vllm attribute was removed; the use_vllm constructor flag still selects vLLM (and is ignored, with a warning, for OpenAI models).

  • s3e.__version__ is read from the installed package metadata.

Fixed

  • Padding issues in multi-prompt HuggingFace batches.

0.3.1 - 2026-07-20

Changed

  • Requesting vLLM together with an OpenAI model now warns and is ignored instead of raising.

0.3.0 - 2026-07-15

Added

  • VLLMBackend for local, single-node multi-GPU inference (requires vllm>=0.11.0, the vllm extra), selected with SemanticStateEstimator(..., use_vllm=True).

Changed

  • Breaking: estimator configuration arguments are keyword-only.

  • HuggingFace backend: uses dtype instead of the deprecated torch_dtype, runs under torch.inference_mode, and keeps only next-token logits to save memory.

0.2.0 - 2026-05-12

Added

  • Batched HuggingFace inference for both log-probability queries and text generation.

  • A calibration-data workflow: collect labeled scores once, save and load them, and fit Platt scaling from the saved data without querying the VLM.

  • s3e.__version__.

Changed

  • The HuggingFace backend returns probabilities over the full vocabulary by default.

0.1.0 - 2026-05-07

First release on PyPI. Development started in August 2024 as the implementation behind the S3E workshop paper (AAAI 2025 Workshop LM4Plan). The repository became public at github.com/guyazran/s3e in February 2026 and moved to github.com/CLAIR-LAB-TECHNION/s3e in April 2026.

Added

  • SemanticStateEstimator: PDDL grounding (via Unified Planning), per-predicate probabilities from answer-token masses, prediction details, querying a subset of predicates, and averaging across scenes.

  • HuggingFace and OpenAI VLM backends.

  • Identity, template, prewritten, and LLM-generated predicate translation.

  • Optional null-token abstention.

  • Platt-scaling calibration (global or lifted scope) with save and load.