Skip to content

generate command #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 157 commits into from
Aug 4, 2025
Merged

generate command #79

merged 157 commits into from
Aug 4, 2025

Conversation

pelikhan
Copy link
Contributor

@pelikhan pelikhan commented Jul 24, 2025

Implement PromptPex strategy to generate tests for prompts automatically.


🚀 PromptPex Power-Up: Smarter Prompt Test Generation, Output Cleaning, and Dev Experience Improvements

This PR supercharges the gh models CLI extension with a suite of new features and quality-of-life upgrades focused on prompt test generation, output handling, and developer workflow. Highlights include:

  • 🧪 PromptPex-Based Test Generation

    • New generate command leveraging the PromptPex framework for systematic, rules-driven prompt test creation.
    • Extensive documentation and examples to guide users through advanced prompt testing workflows.
  • 🧹 Model Output Cleaning Utilities

    • Added helpers to normalize and sanitize LLM-generated outputs, removing formatting artifacts and detecting refusal responses.
    • Comprehensive unit tests ensure robust text cleaning.
  • 🧠 Context Persistence & Effort Configuration

    • Support for loading, saving, and merging prompt test contexts, enabling session persistence and collaborative editing.
    • Configurable "effort levels" to control the intensity and scope of test generation runs.
  • 🏆 Rules-Based Output Evaluation

    • Integrated rules compliance evaluator for automated, granular assessment of LLM outputs against prompt rules.
  • 🔬 Advanced Parsing & Robustness

    • Improved JSON extraction and normalization from LLM responses, handling markdown code blocks, JS-style concatenations, and malformed content.
    • Flexible rules parsing from varied list formats.
  • 🛠️ Utility Functions & CLI Enhancements

    • New helpers for string slicing, map merging, and template variable parsing.
    • Enhanced CLI output formatting and verbose logging for better feedback.
  • 🧪 Expanded Unit Test Coverage

    • Table-driven and scenario-based tests for flag parsing, context management, error handling, and end-to-end prompt rendering.
  • 📝 Developer Experience & CI Improvements

    • Updated Makefile with new linting targets and improved build workflows.
    • Rich project and AI agent instructions for onboarding and code consistency.
  • 🗂️ Prompt File Handling Upgrades

    • Cleaner YAML outputs with omitempty struct tags.
    • Round-trip file save support and improved type safety for test data.

These changes collectively deliver smarter, more reliable prompt test automation, improved output handling, and a better developer experience for working with LLM prompts and tests.

AI-generated content by prd may be incorrect.

pelikhan added 30 commits July 21, 2025 13:41
- Implement tests for Float32Ptr to validate pointer creation for float32 values.
- Create tests for ExtractJSON to ensure correct extraction of JSON from various input formats.
- Add tests for cleanJavaScriptStringConcat to verify string concatenation handling in JavaScript context.
- Introduce tests for StringSliceContains to check for string presence in slices.
- Implement tests for MergeStringMaps to validate merging behavior of multiple string maps, including overwrites and handling of nil/empty maps.
… tests in export_test.go

- Changed modelParams from pointer to value in toGitHubModelsPrompt function for better clarity and safety.
- Updated the assignment of ModelParameters to use the value directly instead of dereferencing a pointer.
- Introduced a new test suite in export_test.go to cover various scenarios for GitHub models evaluation generation, including edge cases and expected outputs.
- Ensured that the tests validate the correct creation of files and their contents based on the provided context and options.
- Added NewPromptPex function to create a new PromptPex instance.
- Implemented Run method to execute the PromptPex pipeline with context management.
- Created context from prompt files or loaded existing context from JSON.
- Developed pipeline steps including intent generation, input specification, output rules, and tests.
- Added functionality for generating groundtruth outputs and evaluating test results.
- Implemented test expansion and rating features for improved test coverage.
- Introduced error handling and logging throughout the pipeline execution.
- Implemented TestCreateContext to validate various prompt YAML configurations and their expected context outputs.
- Added TestCreateContextRunIDUniqueness to ensure unique RunIDs are generated for multiple context creations.
- Created TestCreateContextWithNonExistentFile to handle cases where the prompt file does not exist.
- Developed TestCreateContextPromptValidation to check for valid and invalid prompt formats.
- Introduced TestGithubModelsEvalsGenerate to test the generation of GitHub Models eval files with various scenarios.
- Added TestToGitHubModelsPrompt to validate the conversion of prompts to GitHub Models format.
- Implemented TestExtractTemplateVariables and TestExtractVariablesFromText to ensure correct extraction of template variables.
- Created TestGetMapKeys and TestGetTestScenario to validate utility functions related to maps and test scenarios.
…se and restore its implementation; remove obsolete promptpex.go and summary_test.go files
…covering various scenarios and error handling
…neFlags function and update flag parsing to use consistent naming
Co-authored-by: Sarah Vessels <[email protected]>
Copy link
Contributor Author

@pelikhan pelikhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot Analyze comments around documentation and apply fixes.

@pelikhan
Copy link
Contributor Author

I address a few comments.

Copilot

This comment was marked as outdated.

@Copilot Copilot AI review requested due to automatic review settings August 2, 2025 16:26
Copilot

This comment was marked as outdated.

Copy link
Collaborator

@sgoedecke sgoedecke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to approve again after merge conflicts are fixed

@Copilot Copilot AI review requested due to automatic review settings August 4, 2025 09:01
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a comprehensive test generation feature for the GitHub Models CLI extension using the PromptPex methodology. The changes add an automated system for creating robust test cases from prompt files, along with significant infrastructure improvements for model output handling and developer workflows.

  • PromptPex-based test generation: New generate command that systematically creates test cases by analyzing prompt intent, input specifications, and output rules
  • Enhanced prompt file format: Added support for test data persistence, evaluators, and cleaner YAML output with omitempty tags
  • Improved developer tooling: Better HTTP logging, template variable parsing utilities, and expanded CLI functionality

Reviewed Changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
cmd/generate/ Complete implementation of PromptPex test generation pipeline with LLM interactions, parsing, and evaluation
pkg/util/util.go Moved template variable parsing to shared utility for reuse across commands
pkg/prompt/prompt.go Enhanced prompt file structure with test data support and save functionality
internal/azuremodels/ Added HTTP logging capabilities for debugging API interactions
cmd/run/ Updated to use shared template variable parsing utility
examples/ Added example prompt file for test generation

@pelikhan
Copy link
Contributor Author

pelikhan commented Aug 4, 2025

@sgoedecke I added a 'min' effort level that aggressively disables the amount of tests generated. I think we should have eval and generate in the integration tests as well.

@pelikhan pelikhan requested a review from sgoedecke August 4, 2025 09:09
@sgoedecke sgoedecke merged commit c6a6821 into main Aug 4, 2025
5 checks passed
@sgoedecke sgoedecke deleted the pelikhan/promptpex branch August 4, 2025 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants