Hermetic Liquid Template Recording and Replay System

[tobi]

This PR implements a comprehensive hermetic recording system for Liquid templates that enables perfect deterministic replay of template executions with massive size optimizations.

What This Is

The Hermetic Template Recorder captures complete template execution state - including all variable access patterns, filter calls, and file system interactions - and stores it in optimized JSON recordings that can be replayed with perfect fidelity. This enables:

• Multi implementation testing - This makes it trivial to verify that multiple liquid implementations render the same outputs with the ability to selectively turn on/off filters as they are getting implemented in new versions.
• Production sampling - Record real production liquid runs, then re-run them later for testing
• Deterministic template testing - Record once, replay identically every time
• Performance analysis - Isolate template rendering from data fetching
• Template debugging - Inspect exact variable access patterns and filter chains
• CI/CD optimization - Use recordings instead of complex database setups

Key Features

🎯 Perfect Roundtrip Fidelity

• Zero differences between original and replayed output
• Captures both Liquid Drop objects and plain Hash/Array variables
• Hermetic replay uses pure data structures (no Drop object reconstruction)

📦 Massive Size Optimization

• Semantic key-based filter recording reduces file sizes by ~95% compared to full object space dump
• Smart data extraction from Drop objects
• Efficient JSON serialization of complex nested structures

🛠 Developer-Friendly CLI Tools

# Record template execution
bundle exec ruby bin/liquid-record output.json vogue product

# Verify perfect roundtrip
bundle exec ruby bin/liquid-record --verify output.json vogue product
✅ Verification PASSED - outputs match perfectly!

🔧 TrackableHash/Array Wrappers

• Intercepts property access on non-Drop objects
• Captures complete variable interaction patterns
• Maintains data type integrity throughout recording/replay cycle

How to Use

Basic Recording

# Record template execution

Liquid::TemplateRecorder.record(/tmp/recording.json) do
  # ...
  template = Liquid::Template.parse(source).render(assigns)
  # .... 
end

CLI Workflow

# Record a template with variables
bundle exec ruby bin/liquid-record recording.json template_dir variable_set

# Verify the recording works perfectly
bundle exec ruby bin/liquid-record --verify recording.json template_dir variable_set

# Replay without original data/database
replayer = Liquid::TemplateRecorder.replay_from('recording.json')
output = replayer.render

Integration Testing

# Unit test roundtrip functionality
def test_cli_roundtrip
  # Record template
  system("bundle exec ruby bin/liquid-record test.json dropify product")
  
  # Verify perfect replay
  replayer = Liquid::TemplateRecorder.replay_from('test.json')
  assert_equal expected_output, replayer.render
end

Technical Implementation

Drop-Free Replay Architecture

During recording, the system extracts complete underlying data from Liquid Drop objects and stores it as plain JSON. During replay, only Hash/Array structures are used - no Drop objects are reconstructed. This ensures:

• No dependency on original object models
• Faster replay performance
• Perfect data isolation

Semantic Key Generation

Filter calls are identified by semantic keys like product.title|upcase[0] instead of sequential indices, enabling:

• Resilient replay across template modifications
• Massive file size reduction through deduplication
• Better debugging and inspection capabilities

Smart Data Merging

The system merges original template assigns with dynamically accessed Drop properties using intelligent conflict resolution that preserves Hash structures and prevents type conversion issues.

Test Plan

• Unit tests for CLI roundtrip workflow
• Integration tests for recording/replay cycles
• Verification of zero-diff output matching
• Performance testing with complex nested data structures
• Edge case handling for circular references and deep nesting

The system achieves perfect hermetic recording with 55KB JSON files containing complete execution state, enabling reliable template testing and analysis workflows.

• make theme_runner actually useful outside of the performance benchmarks
• the performance runner wasn't actually working before
• also fix profiler
• Apply suggestions from code review
• memory-profiler update
• more profile cleanups
• Implement hermetic template recording and replay system
• Add ProductDrop class for proper Liquid Drop object handling
• Reorder JSON structure and add CLI tools for better usability
• Implement TrackableHash/Array wrappers for hermetic variable recording
• Fix loop event processing to prevent Hash-to-Array conversion
• Add comprehensive CLI roundtrip unit test
• Update replayer and file system components for hermetic recording
• Update integration tests and test helper for hermetic recording

[Copilot]

Pull Request Overview

This PR implements a comprehensive hermetic recording system for Liquid templates that enables perfect deterministic replay of template executions with massive size optimizations. The system captures complete template execution state - including all variable access patterns, filter calls, and file system interactions - and stores it in optimized JSON recordings for testing, debugging, and performance analysis.

Key changes:

• Implements hermetic template recording and replay with drop-free architecture
• Adds semantic key-based filter recording system for ~95% size reduction
• Provides TrackableHash/Array wrappers for comprehensive variable interaction capture

Reviewed Changes

Copilot reviewed 31 out of 33 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/unit/template_recorder_unit_test.rb	Comprehensive unit tests for template recorder functionality
test/unit/memory_file_system_unit_test.rb	Unit tests for in-memory file system component
test/unit/json_schema_unit_test.rb	Unit tests for JSON schema validation and serialization
test/unit/event_log_unit_test.rb	Unit tests for event logging and path parsing
test/unit/cli_roundtrip_test.rb	CLI integration tests for recording/replay workflow
test/unit/binding_tracker_unit_test.rb	Unit tests for object binding and loop context tracking
test/test_helper.rb	Minor parameter update for file system compatibility
test/integration/template_recorder_integration_test.rb	Integration tests for complex template scenarios
test/integration/tags/include_tag_test.rb	Updates file system interface for context parameter
test/integration/profiler_test.rb	Updates file system interface for context parameter
test/integration/error_handling_test.rb	Updates file system interface for context parameter
performance/theme_runner.rb	Refactored to support individual test execution and recording
performance/shopify/vision.database.yml	Database formatting improvements for better readability
performance/shopify/database.rb	Adds ProductDrop class for proper Liquid Drop object handling
performance/profile.rb	Enhanced profiling with better output and flame graph support
performance/memory_profile.rb	Updates to use refactored theme runner methods
performance/benchmark.rb	Updates to use refactored theme runner methods and YJIT detection
lib/liquid/template_recorder/*	Core recorder implementation with replayer, memory filesystem, and schema
lib/liquid/strainer_template.rb	Adds filter call recording hooks
lib/liquid/tags/for.rb	Adds loop event recording for proper iteration tracking
lib/liquid/partial_cache.rb	Updates file system interface for context parameter
lib/liquid/file_system.rb	Adds file read recording hooks and context parameter
lib/liquid/drop.rb	Adds drop property access recording hooks
lib/liquid.rb	Adds template recorder module require
RECORDER_IMPLEMENTATION_PLAN.md	Comprehensive implementation documentation

Comments suppressed due to low confidence (6)

test/unit/template_recorder_unit_test.rb:252

• This puts statement in a test method will produce output during test execution. Consider using a proper assertion or removing debug output.

    puts "Replayed #{found_items} out of #{expected_items.length} expected items" if found_items < expected_items.length

test/unit/template_recorder_unit_test.rb:423

• This puts statement in a test method will produce output during test execution. Consider using a proper assertion or removing debug output.

      puts "Root recorded as array due to loop recording behavior"

test/unit/template_recorder_unit_test.rb:499

• This puts statement in a test method will produce output during test execution. Consider using a proper assertion or removing debug output.

      puts "Items available for replay: #{root_vars.map { |item| item['name'] }.join(', ')}"

test/unit/template_recorder_unit_test.rb:522

• This puts statement in a test method will produce output during test execution. Consider using a proper assertion or removing debug output.

      puts "Verify mode failed as expected: #{e.message}"

test/integration/template_recorder_integration_test.rb:133

• This puts statement in a test method will produce output during test execution. Consider using a proper assertion or removing debug output.

      puts "Files captured: #{data['file_system'].keys}" unless files_captured

performance/theme_runner.rb:29

• [nitpick] The parameter name 'strictness' is unclear - consider a more descriptive name like 'strict_options' or 'strict_filters'.

    @strictness = strictness

[Copilot]

Version comparison using string equality may fail for semantic versioning. Consider using proper version comparison logic.

Suggested change

	recorded_version = @data['engine']['liquid_version']
	current_version = Liquid::VERSION
	recorded_version = Gem::Version.new(@data['engine']['liquid_version'])
	current_version = Gem::Version.new(Liquid::VERSION)

[Copilot]

The smart_merge method performs deep merging which could be expensive for large data structures. Consider optimizing for common cases or adding size limits.

Suggested change

	# Calculate the size of a nested data structure
	def calculate_size(data)
	case data
	when Hash
	data.sum { |_, v| calculate_size(v) } + data.size
	when Array
	data.sum { |v| calculate_size(v) } + data.size
	else
	1
	end
	end

[Copilot]

Creating a new Set for each call may be inefficient for deeply nested structures. Consider reusing the visited set or using a different circular reference detection approach.

Suggested change

	def self.ensure_serializable(obj, visited = Set.new)
	def self.ensure_serializable(obj, visited = nil)
	visited ||= Set.new

[Copilot]

The parse_path method is complex with multiple state variables and nested conditions. Consider breaking it into smaller methods or using a more structured parsing approach.

[isaacbowen]

:))))))))))))))))

[ggmichaelgo]

Suggested change

	segment.each_with_index do |item, index|
	# Recording hook - item binding
	if (recorder = context.registers[:recorder])
	recorder.for_item(index, item)
	end
	recorder = context.registers[:recorder]
	segment.each_with_index do |item, index|
	# Recording hook - item binding
	recorder.for_item(index, item) if recorder

[ggmichaelgo]

Suggested change

	# Recording hook - loop exit
	if (recorder = context.registers[:recorder])
	recorder.for_exit
	end
	# Recording hook - loop exit
	recorder.for_exit if recorder

[ianks]

This is really cool. I spent some time integrating this into our storefront to test it out, and here are some things that came up:

1. Multi-template architectures
Our storefront uses JSON templates that render multiple section templates in a single request. The current recorder design captures a single template source, but in practice we often have dozens of section templates rendering together. It might be helpful to support recording multiple templates in a single recording session, perhaps with a templates array that tracks each parsed template along with its name/path.

2. Template parsing flow variations
We noticed the recorder hooks into Liquid::Template.parse (class method), but in our codebase we create template instances first and then call parse on them:

template = Liquid::Template.new
template.parse(source, options)

The instance method interception works, but the RecordingTemplate wrapper approach might need some adjustments for this pattern.

3. Section rendering context
For applications using section-based architectures, it would be valuable if the recording could capture which sections were rendered and in what order. This would make it easier to trace execution flow in complex page layouts.

4. File path resolution
The file system recording works well. One thing we noticed is that the paths in our file system reads are often relative to different roots, so having some metadata about the base path context would help with replay accuracy.

5. Template source capture timing
When templates are parsed through adapter layers or caching mechanisms, the source might not be available at the Liquid::Template.parse call site. Consider capturing source at multiple points in the template lifecycle to handle these cases.

The JSON schema is well structured and the recorder's approach to tracking variable access and filter calls is solid. Would be happy to discuss extending this to handle multi-template scenarios if you're interested.