Skip to content

generate command #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 128 commits into
base: main
Choose a base branch
from
Draft

generate command #79

Changes from 1 commit
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
871788d
plumbing for commands
pelikhan Jul 21, 2025
9e82844
bringing promptpex
pelikhan Jul 21, 2025
d8fcb9d
Add comprehensive Copilot instructions for AI coding agents
pelikhan Jul 21, 2025
35870b9
Add unit tests for utility functions in generate package
pelikhan Jul 21, 2025
3ea7a6e
Enhance ApplyEffortConfiguration to handle nil options gracefully
pelikhan Jul 21, 2025
ef7d089
Refactor PromptPexContext to use ChatMessage from azuremodels and rem…
pelikhan Jul 21, 2025
96f9183
Implement GitHub Models evaluation file generation and enhance Prompt…
pelikhan Jul 21, 2025
37b761c
Fix dereferencing of Frontmatter fields in GitHub Models prompt gener…
pelikhan Jul 21, 2025
74d048b
Refactor model parameters handling in export.go and add comprehensive…
pelikhan Jul 22, 2025
61a43ca
feat: Implement PromptPex command handler with pipeline execution
pelikhan Jul 22, 2025
ee90766
clea content
pelikhan Jul 22, 2025
e7d4a17
Add comprehensive tests for prompt generation and context creation
pelikhan Jul 22, 2025
1c936c0
refactor: Remove obsolete export_test_new.go file
pelikhan Jul 22, 2025
292917a
refactor: Remove obsolete output options and related tests from Promp…
pelikhan Jul 22, 2025
e9c6668
feat: Add GenerateSummary function and corresponding tests for prompt…
pelikhan Jul 22, 2025
5c5a167
feat: Implement runPipeline function and refactor GenerateSummary for…
pelikhan Jul 22, 2025
b4b662f
refactor: Rename parseTestsFromLLMResponse to ParseTestsFromLLMRespon…
pelikhan Jul 22, 2025
393020f
test: Add comprehensive tests for ParseTestsFromLLMResponse function …
pelikhan Jul 22, 2025
6458590
feat: Implement generate command with comprehensive options and add s…
pelikhan Jul 22, 2025
cdc38f1
refactor: Consolidate command-line flag definitions into AddCommandLi…
pelikhan Jul 22, 2025
bbdd748
test: Add comprehensive tests for NewGenerateCommand and flag parsing…
pelikhan Jul 22, 2025
7dc3d7d
test: Enhance TestGenerateCommandWithValidPromptFile with detailed mo…
pelikhan Jul 22, 2025
e812aec
move test to common fodler
pelikhan Jul 22, 2025
341442f
feat: Update generate command description to include evaluations for …
pelikhan Jul 22, 2025
da294e2
fix: Clarify command description to specify the use of PromptPex meth…
pelikhan Jul 22, 2025
50b853f
fix: Update build instructions to include 'make build' command
pelikhan Jul 22, 2025
5018380
refactor: Rename runPipeline to RunTestGenerationPipeline and add Ren…
pelikhan Jul 23, 2025
9391f0d
Merge remote-tracking branch 'origin/main' into pelikhan/promptpex
pelikhan Jul 23, 2025
f3f320b
refactor: Update test prompt from sentiment analysis to joke analysis
pelikhan Jul 23, 2025
7ab63bc
fix: Disable usage help for pipeline failures in generate command
pelikhan Jul 23, 2025
f726d7c
feat: Add verbose flag for detailed output and update test prompts
pelikhan Jul 23, 2025
e5dd291
refactor: Remove logging functions and implement callModelWithRetry f…
pelikhan Jul 23, 2025
239eed7
fix: Remove unnecessary blank line in callModelWithRetry function
pelikhan Jul 23, 2025
1a9c9ab
feat: Add LLM connection test to RunTestGenerationPipeline
pelikhan Jul 23, 2025
e0f0311
Refactor pointer creation to use util.Ptr for consistency
pelikhan Jul 23, 2025
26f6ee3
refactor: Update PromptPexOptions to use pointers for consistency acr…
pelikhan Jul 23, 2025
c6da6de
add logging
pelikhan Jul 23, 2025
0a15cec
fix incorrect check for completions
pelikhan Jul 23, 2025
130bc53
refactor: Remove LLM connection test from RunTestGenerationPipeline
pelikhan Jul 23, 2025
06a9caa
refactor: Improve output formatting in GenerateSummary for better rea…
pelikhan Jul 23, 2025
e6f4173
Initial plan
Copilot Jul 24, 2025
59fad69
Update interface and all calls to support HTTP logging filename param…
Copilot Jul 24, 2025
fca40f4
Add tests for HTTP logging filename feature and remove DEBUG env var …
Copilot Jul 24, 2025
a46680e
Refactor HTTP log to use Context instead of function parameters
Copilot Jul 24, 2025
3cdc67e
Group HTTP log writes into single fprintf call for better performance
Copilot Jul 24, 2025
d19f533
Remove test files as requested
Copilot Jul 24, 2025
7903afa
Update internal/azuremodels/azure_client.go
pelikhan Jul 24, 2025
6dda3c5
Add --http-log CLI flag to replace DEBUG environment variable for HTT…
pelikhan Jul 24, 2025
26df5e4
Remove redundant log formatting line in GetChatCompletionStream method
pelikhan Jul 24, 2025
f797a5a
Refactor PromptPexContext by removing WriteResults field and update r…
pelikhan Jul 24, 2025
9e8a469
Update http-log flag description for clarity and adjust log formattin…
pelikhan Jul 24, 2025
cbf9d78
Remove http-log flag from eval and run commands; reset log file in Wi…
pelikhan Jul 24, 2025
1ac704e
Rename CreateContext to CreateContextFromPrompt for clarity and updat…
pelikhan Jul 24, 2025
34d55a2
Remove RateTests option and related logic from PromptPexOptions and c…
pelikhan Jul 24, 2025
2d7ef3f
Refactor PromptPexOptions and PromptPexContext by removing unused fie…
pelikhan Jul 24, 2025
582c004
Remove SplitRules field from EffortConfiguration and related tests fo…
pelikhan Jul 24, 2025
ba24a7a
Update copilot instructions for clarity and organization; enhance pro…
pelikhan Jul 24, 2025
3b99ef5
Add PromptHash to PromptPexContext and implement hash computation for…
pelikhan Jul 24, 2025
527b564
Implement ComputePromptHash function and update context handling; enh…
pelikhan Jul 24, 2025
440914f
Refactor PromptPexOptions and related structures to consolidate model…
pelikhan Jul 24, 2025
9e1c074
Refactor context creation to enhance clarity; update model handling i…
pelikhan Jul 24, 2025
d3b430a
Implement ParseRules function to clean up rules text; add tests for I…
pelikhan Jul 24, 2025
fce67b8
Refactor rules handling in tests to remove leading/trailing whitespac…
pelikhan Jul 24, 2025
e2c28d3
Refactor InverseRules to use a slice instead of a string; update rela…
pelikhan Jul 24, 2025
f183206
Refactor context and test handling to unify naming conventions; repla…
pelikhan Jul 24, 2025
f5bc450
Remove context_test.go file to streamline test suite and eliminate ob…
pelikhan Jul 24, 2025
97a945f
Refactor PromptPexOptions and related tests to remove evals and model…
pelikhan Jul 24, 2025
6f22c37
Refactor PromptPexOptions and related configurations to remove Compli…
pelikhan Jul 24, 2025
4e17f45
Remove Compliance field from default options tests to align with rece…
pelikhan Jul 24, 2025
1d06bed
Remove Compliance field from EffortConfiguration and related types; u…
pelikhan Jul 24, 2025
2192d9e
Add TestExpansion field to PromptPexModelAliases; update related func…
pelikhan Jul 24, 2025
12e866e
Refactor PromptPexContext to use pointers for RunID, PromptHash, Inte…
pelikhan Jul 24, 2025
7ca45de
Enhance context creation by adding session file support; implement lo…
pelikhan Jul 24, 2025
d3d51c6
Add session file support to context creation; implement context loadi…
pelikhan Jul 24, 2025
fa86243
Update .gitignore to include generate.json files; remove test_generat…
pelikhan Jul 24, 2025
4632732
Refactor CreateContextFromPrompt to remove sessionFile parameter; upd…
pelikhan Jul 24, 2025
b083161
Refactor output logging to use box formatting; enhance intent, input …
pelikhan Jul 24, 2025
28f5a44
Move box formatting constants to the top of render.go for better visi…
pelikhan Jul 24, 2025
0fe3ff6
Refactor context merging logic for improved readability; add system p…
pelikhan Jul 24, 2025
4a55194
Add UnBacket and UnXml functions; update ParseRules and add tests for…
pelikhan Jul 24, 2025
ded2220
Refactor output rule and inverse rule rendering to use WriteEndListBo…
pelikhan Jul 24, 2025
3aadaa3
Refactor regex patterns in ParseRules for improved accuracy; remove o…
pelikhan Jul 24, 2025
cbbccc2
Enhance session file handling in CreateContextFromPrompt; improve err…
pelikhan Jul 24, 2025
9469f8c
Refactor generateTests function to simplify empty tests check; add gu…
pelikhan Jul 24, 2025
df5d94b
Refactor output rule and test generation messages for improved clarit…
pelikhan Jul 24, 2025
3bb8a18
Refactor runSingleTestWithContext to simplify message handling; repla…
pelikhan Jul 24, 2025
52eed37
Refactor runSingleTestWithContext and rendering functions for improve…
pelikhan Jul 24, 2025
39249c3
Save context after generating groundtruth in generateGroundtruth func…
pelikhan Jul 24, 2025
d79901a
Update .gitignore to include all generate.json files in subdirectories
pelikhan Jul 24, 2025
b1b4f24
Refactor CreateContextFromPrompt to use handler's promptFile; add Sav…
pelikhan Jul 24, 2025
fc6800d
usebuiltin templating
pelikhan Jul 24, 2025
507ec74
Refactor RunTestGenerationPipeline to handle context saving errors; s…
pelikhan Jul 24, 2025
e668ba8
Refactor effort configuration and remove test expansions; update comm…
pelikhan Jul 24, 2025
8e5d8f8
Remove unused command-line flags for tests and verbosity from generat…
pelikhan Jul 24, 2025
23ab2d7
render reasoning and ground truth
pelikhan Jul 24, 2025
39f24ea
Refactor context handling in CreateContextFromPrompt; streamline sess…
pelikhan Jul 24, 2025
af399ab
Refactor output rendering; replace WriteToOut calls with WriteToParag…
pelikhan Jul 24, 2025
98146ad
Enhance session file handling in CreateContextFromPrompt; update logg…
pelikhan Jul 24, 2025
d719935
Fix session file checks in CreateContextFromPrompt and SaveContext to…
pelikhan Jul 24, 2025
5127d94
Refactor WriteStartBox method to accept a subtitle parameter for impr…
pelikhan Jul 24, 2025
f57bf34
Refactor groundtruth model handling and update command-line flag desc…
pelikhan Jul 24, 2025
eba1adc
wire up ci-lint
pelikhan Jul 24, 2025
2d031ec
Update command examples in NewGenerateCommand for consistency and acc…
pelikhan Jul 24, 2025
8b12281
Refactor PromptPex model aliases and remove unused TestExpansion fiel…
pelikhan Jul 24, 2025
9eb7803
Refactor EffortConfiguration and PromptPexOptions by removing TestGen…
pelikhan Jul 24, 2025
025e32e
Refactor PromptPex model handling by changing pointer fields to value…
pelikhan Jul 25, 2025
7571825
Refactor PromptPexTest struct by changing pointer fields to values; u…
pelikhan Jul 25, 2025
0615664
Refactor PromptPexContext by changing RunID and PromptHash fields to …
pelikhan Jul 25, 2025
178d921
Refactor PromptPexOptions and related logic by changing pointer field…
pelikhan Jul 25, 2025
59ca252
Refactor test_generate.yml by nesting temperature under modelParamete…
pelikhan Jul 25, 2025
2831dd9
Fix JSON field names in PromptPexTest and test generation output for …
pelikhan Jul 25, 2025
a0cda99
Add model key parsing in callModelWithRetry for improved error handling
pelikhan Jul 25, 2025
d4a8976
Add IntentMaxTokens and InputSpecMaxTokens to PromptPexOptions; updat…
pelikhan Jul 25, 2025
2018522
Add support for custom instructions in generation phases; update flag…
pelikhan Jul 25, 2025
e9adb0f
Add test generation feature using PromptPex methodology; include adva…
pelikhan Jul 25, 2025
7defd59
Enhance README.md with detailed explanation of Inverse Output Rules a…
pelikhan Jul 25, 2025
29074f6
Add Intent node to PromptPex mermaid diagram for clarity in output rules
pelikhan Jul 25, 2025
c058406
Refactor command-line flags and update test generation examples for c…
pelikhan Jul 25, 2025
5bc1b87
Refactor test input handling in ParseTestsFromLLMResponse and update …
pelikhan Jul 25, 2025
36fd696
Validate effort level in ParseFlags and add comprehensive tests for v…
pelikhan Jul 25, 2025
4a3285e
Add evaluator rules compliance functionality and update related struc…
pelikhan Jul 25, 2025
9c13267
Refactor effort configuration structure and update related logic for …
pelikhan Jul 25, 2025
4b18ed0
Update Makefile to use correct path for Go linter; enhance error hand…
pelikhan Jul 25, 2025
45d8915
add pull request description script
pelikhan Jul 25, 2025
8f7da6c
Update cmd/generate/parser.go
pelikhan Jul 25, 2025
376135e
Update cmd/generate/generate.go
pelikhan Jul 25, 2025
1a6090e
Update cmd/generate/README.md
pelikhan Jul 25, 2025
d21cd6c
Remove test data from test_generate.yml to streamline example usage
pelikhan Jul 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update cmd/generate/README.md 
Co-authored-by: Copilot <[email protected]>
  • Loading branch information
@pelikhan @Copilot
pelikhan and Copilot authored Jul 25, 2025
commit 1a6090e06f93328ab6bc34a44100a3fce1264a5e
2 changes: 1 addition & 1 deletion cmd/generate/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ This command is based on [PromptPex](https://github.com/microsoft/promptpex), a
- Source https://github.com/microsoft/promptpex/tree/dev
- Agentic implementation plan: https://github.com/microsoft/promptpex/blob/dev/.github/instructions/implementation.instructions.md

In a nutshel, read https://microsoft.github.io/promptpex/reference/test-generation/
In a nutshell, read https://microsoft.github.io/promptpex/reference/test-generation/

Loading