Skip to content

CFNv2: Standardise on test reporting #12984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Aug 12, 2025
Merged

CFNv2: Standardise on test reporting #12984

merged 7 commits into from
Aug 12, 2025

Conversation

simonrw
Copy link
Contributor

@simonrw simonrw commented Aug 9, 2025 •
edited
Loading

Motivation

It would be good going forward to be able to automatically parse the number of tests executed and skipped so that we can easily report progress.

Changes

  • Standardise the skip decorator format and structure so we can automatically parse the values in a plugin
  • Add plugin in conftest.py which adds a reporting line for number of skips if PROVIDER_OVERRIDE_CLOUDFORMATION=engine-v2, for example (at time of creating):
CFNV2: skipped 99 v1 skips and 63 v2 skips
- CFNV2(Transform): 26
- CFNV2(LanguageExtensions): 6
- CFNV2(Resolve): 5
- CFNV2(Engine): 4
- CFNV2(DescribeStacks): 3
- CFNV2(Validation): 2
- CFNV2(Other): 2
- CFNV2(Imports): 2
- CFNV2(GetTemplate): 2
- CFNV2(Metadata,Other): 1
- CFNV2(Test): 1
- CFNV2(Deletion): 1
- CFNV2(DescribeStack): 1
- CFNV2(AWS::Include): 1
- CFNV2(UpdateStack): 1
- CFNV2(ResourceProvider): 1
- CFNV2(Provider): 1
- CFNV2(DescribeStackResource): 1
- CFNV2(Macros): 1
- CFNV2(Fn::Transform): 1

See an example from this PR's CI here

We can then read this value from the latest CI run to see progress

Note

See how we technically support more of the test suite with the v2 provider than the v1 provider!

@simonrw simonrw added the semver: patch Non-breaking changes which can be included in patch releases label Aug 9, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

Test Results - Preflight, Unit

22 106 tests  ±0   20 371 ✅ ±0   6m 33s ⏱️ +5s
     1 suites ±0    1 735 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit f95de6c. ± Comparison against base commit 6cc56f7.

♻️ This comment has been updated with latest results.

@simonrw simonrw marked this pull request as ready for review August 9, 2025 14:27
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 7s ⏱️ +2s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit f95de6c. ± Comparison against base commit 6cc56f7.

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

LocalStack Community integration with Pro

  2 files    2 suites   22m 27s ⏱️
555 tests 330 ✅ 225 💤 0 ❌
557 runs  330 ✅ 227 💤 0 ❌

Results for commit f95de6c.

♻️ This comment has been updated with latest results.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2025 •
edited
Loading

Test Results (amd64) - Integration, Bootstrap

  5 files  ±    0    5 suites  ±0   33m 56s ⏱️ - 1h 46m 22s
579 tests  - 4 408  355 ✅  - 4 040  224 💤  - 368  0 ❌ ±0 
585 runs   - 4 408  355 ✅  - 4 040  230 💤  - 368  0 ❌ ±0 

Results for commit f95de6c. ± Comparison against base commit 6cc56f7.

This pull request removes 4408 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

@coveralls
Copy link

Coverage Status

coverage: 66.371% (-20.5%) from 86.848%
when pulling f95de6c on cfn/v2/reporting
into 6cc56f7 on main.

pinzon
pinzon approved these changes Aug 12, 2025
View reviewed changes
Copy link
Member

@pinzon pinzon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. We still have a bit of skipped tests

@simonrw simonrw merged commit a1ae8f0 into main Aug 12, 2025
39 of 40 checks passed
@simonrw simonrw deleted the cfn/v2/reporting branch August 12, 2025 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pinzon pinzon pinzon approved these changes

@dominikschubert dominikschubert Awaiting requested review from dominikschubert dominikschubert is a code owner

Assignees
No one assigned
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@simonrw @coveralls @pinzon