# liquid-spec **Repository Path**: mirrors_Shopify/liquid-spec ## Basic Information - **Project Name**: liquid-spec - **Description**: liquid-spec is a test suite for the Liquid language. - **Primary Language**: Unknown - **License**: MIT - **Default Branch**: main - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2025-07-25 - **Last Updated**: 2026-09-26 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # liquid-spec [![CI](https://github.com/Shopify/liquid-spec/actions/workflows/ruby.yml/badge.svg)](https://github.com/Shopify/liquid-spec/actions/workflows/ruby.yml) **liquid-spec is both an acceptance suite and an implementation system for [Liquid](https://github.com/Shopify/liquid).** It can verify an existing parser and renderer, but it is also designed to guide a human or coding agent from an empty project to a production-ready Liquid implementation, one observable behavior at a time. The suite contains thousands of executable examples drawn from Shopify's reference implementation, a curated beginner ramp, parser-error matrices, Dawn theme fixtures, and production recordings. The adapter boundary works with any implementation strategy and, over JSON-RPC, any programming language. ## More Than a Conformance Suite A conventional conformance suite answers **“is this implementation compatible?”** liquid-spec also answers: - **What is the smallest useful behavior to implement next?** - **Why does Liquid behave this way?** - **Which earlier behavior did this change regress?** - **How far has the implementation progressed from a text passthrough to production compatibility?** Every spec has a `complexity` from 0 to 1000. Foundation cases come first: empty input, literal text, and object output. Variables, filters, control flow, loops, scopes, partials, parser modes, compatibility quirks, and production recordings follow. The runner reports the lowest-complexity failures as **“Next best specs to work on”** and shows an implementation hint with each failure. That turns the suite into an executable curriculum: ```text run liquid-spec │ ▼ read the first failure + hint ──► implement the general rule ▲ │ └──────── rerun every earlier spec ◄───┘ ``` `Complexity level cleared: 70 of 1000` means all exercised levels before the first failing level are solid. It is deliberately more honest than a raw pass count: a toy renderer may accidentally pass hundreds of empty-output cases without clearing the beginner ramp. ## Why the Agent Loop Works liquid-spec gives an agent the ingredients that open-ended “implement Liquid” prompts normally lack: 1. **A stable, observable contract.** Specs describe templates, input values, filesystems, expected output, and expected errors—not a required internal architecture. An AST interpreter, bytecode VM, compiler, or transpiler can follow the same path. 2. **A prerequisite-ordered search space.** Complexity scoring reduces a language-sized task to a small next step. Later behavior is introduced after the semantics it builds on. 3. **Just-in-time implementation guidance.** Curated failures include actionable hints, and `liquid-spec docs` supplies deeper implementer guides for values, grammar, scopes, filters, loops, partials, parsing, and quirks. 4. **A tight verification loop.** Every change is immediately checked against all earlier behavior. The agent cannot make apparent progress by silently trading one feature for another. 5. **Reference and production evidence.** `liquid-spec tools eval --compare` answers ambiguous questions against Shopify/liquid, while integration tests and production recordings prevent a classroom-only implementation from looking complete. 6. **Explicit scope.** Feature gates distinguish portable Liquid, legacy parser modes, Ruby-specific behavior, and Shopify extensions. Unsupported features are visible debt, not noise hidden among failures. 7. **Machine-readable operation.** `--json`, name/suite filters, inspection, and focused eval commands let an agent gather precise evidence and iterate without scraping an unstructured test log. `liquid-spec init` makes this workflow agent-ready. It generates disposable adapter shims plus an `AGENTS.md` that explains the loop, hard rules, architecture advice, protocol, feature gates, and documentation commands. Your Liquid package remains a standalone library; the adapter exists only to let liquid-spec exercise it. ## How Acceptance Testing Works ```text YAML behavior specs (template, environment, filesystem, expected output/error) │ ▼ liquid-spec runner │ compile + render calls │ ┌────────────┴────────────┐ ▼ ▼ direct Ruby adapter JSON-RPC adapter (TypeScript, Rust, Go, Python, ...) │ │ └────────────┬────────────┘ ▼ your Liquid engine ``` For each case, liquid-spec compiles the source, renders it with the recorded environment and filesystem, then compares output or errors with the accepted behavior. The bridge is small enough that results describe the engine rather than the test integration. ## Installation Add to your Gemfile: ```ruby gem "liquid-spec", git: "https://github.com/Shopify/liquid-spec" ``` Then run: ```bash bundle install ``` Or install directly from GitHub: ```bash gem install specific_install gem specific_install https://github.com/Shopify/liquid-spec ``` ## Quick Start ```bash # Generate both adapter choices and an agent implementation guide. liquid-spec init # Read the curriculum, then run the adapter for your language. liquid-spec docs curriculum liquid-spec run specs/adapter.rb # Ruby liquid-spec run specs/adapter-jsonrpc.rb \ --command="your-liquid-jsonrpc-server" # any other language ``` The first failure is the next lesson. Implement the behavior described by its hint and rerun the same command. Re-running `init` leaves identical generated files alone; when a target differs, it asks before overwriting and defaults to **No**. ## Full CLI Example: Ask an Agent to Build Liquid in TypeScript The complete bootstrap is intentionally small. Start in an empty directory: ```bash mkdir liquid-typescript cd liquid-typescript gem install specific_install gem specific_install https://github.com/Shopify/liquid-spec # Creates specs/adapter.rb, specs/adapter-jsonrpc.rb, and AGENTS.md. # The JSON-RPC adapter is the bridge the TypeScript implementation will use. liquid-spec init codex -p "/goal Implement a full production-ready Liquid implementation in TypeScript. \ Read AGENTS.md first. Use specs/adapter-jsonrpc.rb only as the test bridge; build the \ engine as a standalone TypeScript library. Ask liquid-spec for guidance on the next \ steps, implement the general behavior behind each lowest-complexity failure, and rerun \ the suite after every change. Do not special-case specs or hide required behavior with \ missing_features. Keep going until liquid-spec reports Complexity level cleared: \ 1000 of 1000 for every applicable suite." ``` That is enough context because `liquid-spec init` wrote the detailed operating manual into `AGENTS.md`. The agent can discover and use the whole feedback loop itself: ```bash # After setting DEFAULT_COMMAND in specs/adapter-jsonrpc.rb to the TypeScript server: liquid-spec run specs/adapter-jsonrpc.rb liquid-spec tools inspect specs/adapter-jsonrpc.rb -n "the_failing_spec" liquid-spec docs curriculum liquid-spec docs core-abstractions cat scratch.yml | liquid-spec tools eval specs/adapter-jsonrpc.rb --compare liquid-spec run specs/adapter-jsonrpc.rb --json ``` The generated JSON-RPC adapter initially opts out of capabilities that cannot be carried portably or have not been wired yet, such as Ruby-only values, standard test drops, and Shopify extensions. Reaching `1000 of 1000` means every spec selected for the adapter passes; a production target also requires reviewing `missing_features` and removing each entry that belongs in that target. Use `liquid-spec tools features` to audit that scope. For a human-driven implementation, use exactly the same loop: run, read the first hint, implement, and rerun. `liquid-spec docs json-rpc-protocol` documents the four subprocess methods (`initialize`, `compile`, `render`, and `quit`). ## Writing an Adapter An adapter is a small Ruby file that tells liquid-spec how to use your implementation: ```ruby #!/usr/bin/env ruby require "liquid/spec/cli/adapter_dsl" # Load your implementation; ctx carries compiled state between callbacks. LiquidSpec.setup do |ctx| require "my_liquid" end # Declare what your adapter can't handle (default: run everything) LiquidSpec.configure do |config| # Parse modes this adapter implements. Highest strictness is the default. config.error_modes = [:strict2] # Raised render errors are the default; :inline is optional. config.render_error_modes = [:raise] config.missing_features = [:shopify_tags, :shopify_filters] end # Parse template source and retain the result in the adapter context. LiquidSpec.compile do |ctx, source, options| # options includes: :line_numbers, :error_mode ctx[:template] = MyLiquid::Template.parse(source, **options) end # Render the template stored by compile or load_artifact. LiquidSpec.render do |ctx, assigns, options| # assigns = variables hash # options includes: :registers, :strict_errors, :error_mode, :exception_renderer ctx[:template].render(assigns, **options) end ``` The `options` hash in render includes: - `:registers` - Host context for filters/drops, including `:file_system`, `:template_factory`, and `:current_time`; it is not template-visible like assigns Specs without `error_mode` run once using the adapter's highest supported parse mode (`strict2`, then `strict`, then `lax`). A spec declaring multiple modes runs at its highest supported strict mode; explicitly declaring `lax` adds one separate lax run. Inline-error specs run only when `render_error_modes` includes `:inline`; raised render errors are the normal path. - `:strict_errors` - If true, raise errors; if false, render them inline - `:exception_renderer` - Custom exception handler (optional) ### JSON-RPC adapters for non-Ruby implementations Use `liquid-spec init --jsonrpc my_adapter.rb` when your Liquid engine is written in Rust, Go, Python, Node.js, or another language. The generated Ruby adapter launches your server as a subprocess and talks JSON-RPC over stdin/stdout. Key setup points: - Your server implements `initialize`, `compile`, `render`, and `quit`. - Server debug logs go to stderr; stdout must contain only newline-delimited JSON-RPC messages. - The adapter controls spec selection with `config.error_modes`, `config.render_error_modes`, and `config.missing_features`; server-reported `features` are informational. - Minimal JSON-RPC adapters should usually opt out of unsupported or non-portable features such as `:drops`, `:ruby_types`, `:ruby_drops`, `:drop_class_output`, `:self_environment_shadowing`, `:binary_data`, and `:template_factory`, plus Shopify-specific features. - Remove `:drops` when the engine supports the standard test-drop library. Bidirectional runtime objects can use the protocol's `drop_get`, `drop_call`, and `drop_iterate` callbacks. - Read `docs/json-rpc-protocol.md` for the exact message format and error-handling rules. ```bash liquid-spec init --jsonrpc my_adapter.rb liquid-spec run my_adapter.rb --command="./my-liquid-server" liquid-spec run my_adapter.rb --json --list-passed > results.json ``` ### Optional: compiled-artifact protocol Some Liquid implementations compile source once, store executable bytecode or an equivalent compiled representation in a shared cache, and load those bytes in application processes that never receive the source. `compile` and `render` alone cannot measure that important production path: parsing again is not an artifact-cache hit, while repeatedly rendering a resident template omits the load and first-use costs. If your implementation has such a persistent compiled format, declare both hooks: ```ruby # Called immediately after LiquidSpec.compile, before the template is rendered. # Return the exact binary String you would put in memcache, a database, etc. LiquidSpec.dump_artifact do |ctx| ctx[:template].to_artifact end # Called with the artifact bytes and no source recompilation. # Restore all adapter state expected by the regular LiquidSpec.render hook. LiquidSpec.load_artifact do |ctx, bytes, _options| ctx[:template] = MyLiquid::Artifact.load(bytes) end ``` The contract is deliberately production-oriented: 1. `dump_artifact` receives the state produced by `compile` and must return a binary-safe `String`. It is invoked before any validation render, because rendering is allowed to mutate template runtime state. 2. The returned bytes must contain all immutable compile-time information needed to load the template without its source. Do not capture assigns, observed values, request objects, or render-time object shapes. 3. `load_artifact` must leave `ctx` in the same renderable state that `compile` would. The source is unavailable. Its third argument is the runtime options Hash that the following `render` hook will receive; most loaders ignore it. 4. Assigns, registers, and runtime filesystems are still supplied to the normal `render` hook for every call; they are not part of the artifact. With both hooks, `--bench` validates the dump → load → render roundtrip, reports raw artifact bytes and steady-state load diagnostics, and measures atomic source compile + first-render and artifact-load + first-render workflows with 10 interleaved samples in the adapter process. Each sample invokes compile or load before its first render, but process-level runtime, JIT, and global caches remain warm. The harness intentionally does not include process startup or IPC. `--bench` warns when either hook is missing because artifact size and load+first-render results will be omitted. Implementations without a persistent compiled format may leave the hooks absent; the warning documents that the benchmark is then limited to source compile and resident render paths. ### Optional: local suites Projects can ship their own spec/benchmark suites alongside their adapter: any `./specs//suite.yml` directory in the invoking project is discovered next to the gem's builtin suites and selected the same way (`-s `). Set `timings: true` in `suite.yml` to make it benchmarkable with `liquid-spec bench ADAPTER -s `, and `default: false` to keep it out of regular runs. ## Test Suites | Suite | Tests | Description | |-------|-------|-------------| | **basics** | 955 | Essential Liquid features - start here! Ordered by complexity with implementation hints | | **liquid_ruby** | 2,106 | Core Liquid specs from [Shopify/liquid](https://github.com/Shopify/liquid) integration tests | | **liquid_ruby_lax** | 121 | Lax-mode reference behavior | | **parser_errors** | 1,905 | Strict parser error compatibility and mutation matrices | | **partials** | 12 | Include/render focused compatibility specs and timings | | **benchmarks** | 10 | Storefront, dynamic-partial, `{% liquid %}`, and Shopify-theme performance cases | | **shopify_production_recordings** | 2,260 | Recorded behavior from Shopify's production Liquid compiler | | **shopify_theme_dawn** | 26 | Real-world templates from [Shopify Dawn](https://github.com/Shopify/dawn) theme | ### The Basics Suite If you're building a new Liquid implementation, **start with the basics suite**. It runs first and covers all fundamental features from the [official Liquid documentation](https://shopify.github.io/liquid/). Specs are ordered by complexity so you can implement features progressively. The goal is a smooth ramp: a toy renderer should pass only the trivial first specs, then fail on a small, actionable next behavior. | Complexity | Features | |------------|----------| | 0-1 | Empty template and literal passthrough | | 5-20 | First object output, literal strings/numbers/booleans/nil | | 30-50 | Variables, missing variables, very simple filters, assign | | 55-65 | Basic if/else/unless and simple boolean composition | | 70-100 | Gentle loops, comparisons, forloop basics, capture, simple case/when | | 105-150 | Common filters/tags, comments/raw, interrupts, loop modifiers, whitespace control | | 160-220 | Generated filter breadth, truthy/falsy edges, cycle/tablerow, first partials/filesystem | | 230-400 | Long-tail standard behavior and parser/scope/filesystem edge cases | | 500-900 | Mature compatibility: parser mutations, resource-limit accounting, recursion/deep nesting, date/time/Ruby quirks | | 1000 | Production recordings and unscored specs | Each non-trivial spec includes a detailed `hint` explaining how the feature should be implemented. If the first failure is surprising or unactionable, the spec probably needs a better hint or a higher complexity score. **Read `Complexity level cleared`, not just total passes.** A naive adapter that always returns `""` can accidentally pass many later specs whose expected output is empty, but its cleared complexity level should remain at 0. The complexity-level line tells you how far the implementation progressed through the ordered curriculum. ### Feature-Based Suite Selection Suites marked as default run unless you select one explicitly with `-s`. Declare what your adapter cannot handle to skip specs that require those capabilities: ```ruby LiquidSpec.configure do |config| config.error_modes = [:strict2] config.render_error_modes = [:raise] # Run everything (default — empty denylist) config.missing_features = [] # Skip Shopify-specific specs (for adapters without Shopify extensions) config.missing_features = [:shopify_tags, :shopify_objects, :shopify_filters] end ``` Do not list parsing or inline-error tags manually in `missing_features`; they are derived from the two positive error-mode declarations. ## Generated Adversarial Coverage After the recorded ramp passes, generate nearby cases and compare them directly with Shopify/liquid: ```bash liquid-spec tools mutate adapter.rb --around=for_loops --limit=100 liquid-spec tools fuzz adapter.rb --seed=1234 --rounds=500 --minimize liquid-spec tools stress adapter.rb --depth=64 --repetitions=100 ``` `mutate` deterministically changes existing specs; `fuzz` reproducibly chains random mutations; `stress` generates bounded valid nesting and repetition. Differences are saved as runnable YAML regression specs by default. These commands cover whitespace controls, literal boundaries, lookups, filters, conditionals, loop options, malformed block structure, opaque bodies, Unicode, and newlines. This is differential corpus mutation, not native coverage-guided fuzzing. See `liquid-spec docs adversarial` for comparison semantics, seed selection, JSON output, minimization, and how to curate a generated discovery into the permanent suite. ## CLI Reference ```bash liquid-spec COMMAND [options] Core commands: liquid-spec init [FILE] Generate adapters and AGENTS.md liquid-spec docs [NAME] Read implementer documentation liquid-spec run ADAPTER Run the acceptance ramp liquid-spec bench [ADAPTER] Benchmark implementations Tool collection (`liquid-spec tools help`): liquid-spec tools inspect ADAPTER Inspect matching specs in detail liquid-spec tools eval ADAPTER Evaluate one YAML spec liquid-spec tools matrix Compare adapters side-by-side liquid-spec tools test Exercise bundled example adapters liquid-spec tools features Audit feature tags and scope liquid-spec tools report Analyze benchmark results liquid-spec tools check Run every verifier liquid-spec tools mutate ADAPTER Deterministic differential mutations liquid-spec tools fuzz ADAPTER Seeded differential fuzz-style testing liquid-spec tools stress ADAPTER Bounded structural stress Run options: -n, --name PATTERN Only run specs matching PATTERN -s, --suite SUITE Run specific suite (liquid_ruby, benchmarks, etc.) -c, --compare Compare output against reference liquid-ruby -v, --verbose Show detailed output -l, --list List available specs --list-suites List available test suites --max-failures N Stop after N failures (default: 5) --no-max-failures Run all specs without stopping --list-passed List specs that passed after the run (ramp/debug audits) --json Output a single JSON summary (for tools) --jsonl Output one JSON event per line (for benchmark streaming/tools) -h, --help Show help Examples: liquid-spec run my_adapter.rb # Run all applicable specs liquid-spec run my_adapter.rb -n for_tag # Run specs matching 'for_tag' liquid-spec run my_adapter.rb -s liquid_ruby # Run only liquid_ruby suite liquid-spec run my_adapter.rb --compare # Compare against reference liquid-spec run my_adapter.rb --no-max-failures # See all failures liquid-spec bench my_adapter.rb -n storefront # Benchmark adapter vs reference liquid-spec tools check # Run every spec verifier liquid-spec tools features # Audit supported feature scope liquid-spec tools inspect my_adapter.rb -n "case" # Debug specific specs liquid-spec tools mutate my_adapter.rb --around=for_loops liquid-spec tools fuzz my_adapter.rb --seed=1234 --json ``` ### Auditing the Ramp with Dumb Adapters When changing complexity scores or adding early specs, test the harness with intentionally bad adapters: - an adapter that returns the template source unchanged - an adapter that always returns `""` - an adapter that raises during compile or render Use `--list-passed` to see accidental passes and `--json` for machine-readable analysis: ```bash liquid-spec run /tmp/echo_adapter.rb --list-passed liquid-spec run /tmp/empty_adapter.rb --json --list-passed > empty-results.json ``` A source-echo adapter should only pass raw-text specs before failing on first object output. An always-empty adapter may pass many empty-output specs, so judge progress by `Complexity level cleared` (or JSON `max_complexity_reached`), not by total passes. ### Matrix Command The `matrix` command runs specs across multiple adapters simultaneously and shows differences between implementations. This is useful for comparing behavior across different Liquid implementations or configurations. ```bash liquid-spec tools matrix [options] Options: --all Run all default adapters from examples/ --adapters=LIST Comma-separated list of adapters --reference=NAME Reference adapter (default: liquid_ruby) -n, --name PATTERN Filter specs by name pattern -s, --suite SUITE Spec suite to run --max-failures N Stop after N differences (default: 10) --no-max-failures Show all differences -v, --verbose Show detailed output Examples: # Compare the default bundled adapters liquid-spec tools matrix --all # Compare specific adapters liquid-spec tools matrix --adapters=liquid_ruby,liquid_ruby_lax # Compare adapters on specific tests liquid-spec tools matrix --adapters=liquid_ruby,my_adapter.rb -n truncate ``` Output shows which adapters produce different results for each spec: ``` Running 100 specs: ....F....F.. done ====================================================================== DIFFERENCES ====================================================================== ---------------------------------------------------------------------- 1. TruncateTest#test_truncate_with_custom_ellipsis Template: {{ text | truncate: 10, "..." }} Adapters: liquid_ruby Output: "Hello w..." Adapters: my_adapter Output: "Hello wo..." ====================================================================== ``` ### Benchmarking liquid-spec includes a benchmark suite for measuring and comparing implementation performance. Benchmarks measure **compile** and **render** times separately, with statistical analysis including mean, standard deviation, and min/max ranges. #### Single Adapter Benchmarks Run benchmarks against a single adapter to measure its performance: ```bash liquid-spec bench examples/liquid_ruby.rb -n leaderboard ``` Abbreviated output: ```text liquid_ruby — Benchmarks Ruby 4.x (no-jit) │ 1 specs │ 5s/spec Benchmark 1/1: liquid_tag_leaderboard Parse (mean ± σ): 270µs ± 38µs [893 allocs, 9.3k runs] Render (mean ± σ): 564µs ± 116µs [926 allocs, 4.4k runs] Range (min … max): 530µs … 3.79ms [4.4k runs] Cold (@1 / @10): 664µs / 618µs (1.2x vs warm) 1 passed ``` Benchmarks report parse/render distributions, allocation counts, cold-render behavior, and iteration totals. GC is disabled during timing to reduce measurement jitter. #### Multi-Adapter Performance Comparison Compare performance across different implementations using the core `bench` command: ```bash liquid-spec bench --adapters=liquid_ruby,liquid_ruby_lax ``` Each benchmark runs against all adapters. The command prints their measurements together, then reports geometric-mean parse and render comparisons across common specs. When a comparison mixes inline adapters with JSON-RPC adapters, `bench` warns that subprocess and protocol overhead make the timings non-equivalent. ```text Benchmark 1/10: storefront_product_page liquid_ruby Parse ... Render ... liquid_ruby_lax Parse ... Render ... → liquid_ruby is 1.08x faster ────────────────────────────────────────────────────────────────────── Comparison (10 common specs, reference: liquid_ruby) Parse (geometric mean): liquid_ruby is 1.05x faster than liquid_ruby_lax Render (geometric mean): liquid_ruby_lax ≈ liquid_ruby ``` #### Benchmark Specs The benchmark suite currently includes 10 realistic templates: | Benchmark | Description | |-----------|-------------| | `bench_dynamic_partials` | Data-selected partials, loops, and three levels of nested includes | | `bench_liquid_tag_inventory_report` | Inventory aggregation entirely inside a `{% liquid %}` block | | `bench_liquid_tag_leaderboard` | Nested-loop ranking and formatting in `{% liquid %}` syntax | | `bench_storefront_product_page` | Standard-Liquid product page with variants, reviews, and partials | | `bench_storefront_collection_page` | Standard-Liquid collection browsing and product grids | | `bench_storefront_cart_page` | Standard-Liquid cart totals, discounts, and line items | | `bench_storefront_order_email` | Standard-Liquid transactional order email | | `bench_storefront_cms_page` | Standard-Liquid content-management page | | `shopify_theme_full_page` | Shopify-shaped Dream theme layout using portable Liquid | | `shopify_theme_product_page` | Shopify-shaped Dream product page using portable Liquid | Selecting these specs through `liquid-spec run ADAPTER -s benchmarks` checks correctness without collecting timings; use `liquid-spec bench` for performance measurements. #### Profiling with StackProf Use `liquid-spec bench --profile` to generate StackProf profiles for detailed performance analysis: ```bash # Single adapter profiling liquid-spec bench examples/liquid_ruby.rb --profile # Multi-adapter profiling liquid-spec bench --adapters=liquid_ruby,liquid_ruby_lax --profile ``` A single-adapter profile directory contains four dumps: ```text /tmp/liquid-spec-profile-20260710_155637/ ├── compile_cpu.dump ├── compile_object.dump ├── render_cpu.dump └── render_object.dump ``` Inspect one with `stackprof /tmp/liquid-spec-profile-20260710_155637/render_cpu.dump`. In multi-adapter bench mode, each adapter gets a separate directory such as `/tmp/liquid-spec-profile-{timestamp}-liquid_ruby/`; the command prints every path. Profile types: - `*_cpu.dump` - CPU time profiles (where time is spent) - `*_object.dump` - Object allocation profiles (where allocations happen) ### Quick Testing with `eval` The `eval` tool lets you quickly test individual templates. Specs are passed via YAML on stdin or `--spec=FILE`; add `--compare` to compare with the reference liquid-ruby implementation and fill omitted expectations: ```bash liquid-spec tools eval examples/liquid_ruby.rb --compare <