This PR adds --var template variable support to the generate command, matching the functionality already available in the run command. This enables users to create flexible, reusable prompt files for test generation scenarios while preventing conflicts with reserved variables only where necessary.

Problem

The generate command previously only supported a fixed input variable when templating prompt files, while the run command supported custom template variables via --var name=value flags. This inconsistency made it difficult to create reusable prompt templates for test generation.

Solution

Added complete --var template variable support to the generate command by:

1. Added CLI flag support: The --var flag can now be used multiple times to pass custom template variables
2. Template variable parsing: Implemented shared util.ParseTemplateVariables() function used by both run and generate commands
3. Command-specific reserved key validation: Prevents input from being used as a template variable key in the generate command only to avoid conflicts with the built-in variable
4. Pipeline integration: Template variables are applied during test execution in runSingleTestWithContext()
5. Backward compatibility: Existing input variable is preserved alongside custom variables

Usage

# Single variable
gh models generate --var name=Alice prompt.yml

# Multiple variables  
gh models generate --var name=Alice --var topic="machine learning" prompt.yml

# Reserved variable protection (generate command only)
gh models generate --var input=test prompt.yml  # Error: 'input' is a reserved variable name
gh models run --var input=test prompt.yml       # Works fine - input allowed in run command

Example prompt file with template variables:

name: "Customer Support Template"
messages:
  - role: system
    content: "You are a helpful assistant for {{name}}."
  - role: user
    content: "Tell me about {{topic}} in {{style}} style."

Implementation Details

• Shared utility: Both run and generate commands use util.ParseTemplateVariables() for consistent behavior
• Command-specific validation: Only the generate command validates against using "input" as a template variable to prevent conflicts with its built-in variable
• Template data structure: {"input": inputValue, "name": "customValue", ...}
• Error handling: Validates variable format, detects duplicate keys, prevents reserved key usage in generate command only
• Integration point: Variables are applied during groundtruth generation when running individual tests

Testing

• Added comprehensive unit tests for template variable parsing covering various test cases
• Added integration test verifying template variable application in the pipeline
• Added specific test verifying run command accepts "input" as template variable
• Added specific test verifying generate command rejects "input" as template variable
• All existing tests continue to pass

This change enables more flexible and parameterized prompt files for test generation while maintaining full backward compatibility and preventing variable conflicts only where necessary.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.