Skip to content

LocalStack SDK for integration tests #12991

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

LocalStack SDK for integration tests #12991

wants to merge 3 commits into from

Conversation

giograno
Copy link
Member

Motivation

In #12972 @viren-nadkarni suggested using the LocalStack SDK for our own integration tests.

Changes

This PR:

@giograno giograno added this to the Playground milestone Aug 11, 2025
@giograno giograno self-assigned this Aug 11, 2025
@giograno giograno added the semver: patch Non-breaking changes which can be included in patch releases label Aug 11, 2025
Copy link

S3 Image Test Results (AMD64 / ARM64)

    2 files    2 suites   8m 25s ⏱️
  515 tests 465 ✅  50 💤 0 ❌
1 030 runs  930 ✅ 100 💤 0 ❌

Results for commit 30cee15.

Copy link

Test Results - Preflight, Unit

22 063 tests  ±0   20 329 ✅ ±0   6m 25s ⏱️ -6s
     1 suites ±0    1 734 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit 30cee15. ± Comparison against base commit e82a823.

Copy link

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 5s ⏱️ -1s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit 30cee15. ± Comparison against base commit e82a823.

Copy link

Test Results (amd64) - Integration, Bootstrap

    5 files      5 suites   2h 20m 36s ⏱️
4 985 tests 4 394 ✅ 591 💤 0 ❌
4 991 runs  4 394 ✅ 597 💤 0 ❌

Results for commit 30cee15.

Copy link

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 42m 32s ⏱️ - 4m 21s
4 626 tests ±0  4 187 ✅ ±0  439 💤 ±0  0 ❌ ±0 
4 628 runs  ±0  4 187 ✅ ±0  441 💤 ±0  0 ❌ ±0 

Results for commit 30cee15. ± Comparison against base commit e82a823.

Copy link
Member

@alexrashed alexrashed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While it is intriguing to use the SDK directly here to test the features of LocalStack that are exposed via the SDK, I think we have to be very careful with dependencies like this.
Let's discuss this a bit more in depth before introducing this dependency! :)

@@ -116,6 +116,7 @@ test = [
"aws-cdk-lib>=2.88.0",
"websocket-client>=1.7.0",
"localstack-snapshot>=0.1.1",
"localstack-sdk-python"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: This effectively implements a circular dependency:

  • The SDK is directly generated from the OpenAPI specs.
  • The OpenAPI specs are released with every release of LocalStack Pro.
  • The SDK is defined as a dependency of LocalStack Pro's test dependencies here.

In my opinion, this could make it very much more tedious to implement changes when changing the API endpoint or adding new ones.

But what exactly is the motivation behind using the SDK here directly?

  • Is it to save some lines of code? Maybe we should then restructure the code or the API to make sure the API is very easily usable.
  • Or is it to verify that the SDK works? I think that should be verified directly in the repo of the SDK. If you want to do regular integration tests, you could run a scheduled integration test pipeline there, ensuring that the latest SDK still integrates well with latest?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants