Allow stack traces in HTTP responses only if `INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE` is enabled #12938

sannya-singal · 2025-07-31T05:58:24Z

Motivation

As a part of the initiative for improving Localstack error messages, this PR aims to conditionally remove stack traces from HTTP responses. A new environment variable INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is introduced, which when enabled, allows stack traces to be included in the HTTP response.
This is expected to be always enabled during test runs, where it improves LocalStack devX, unless its explicitly disabled. This flag is for internal use only. The stack traces wouldn't appear in the HTTP responses for users unlike before when DEBUG is enabled.

Current Behaviour:

sannyasingal@Sannyas-MacBook-Pro ~ % awslocal organizations describe-organization

An error occurred (InternalError) when calling the DescribeOrganization operation (reached max retries: 4): exception while calling organizations.DescribeOrganization: Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/rolo/gateway/chain.py", line 166, in handle
    handler(self, self.context, response)
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/handlers/service.py", line 112, in __call__
    handler(chain, context, response)
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/handlers/service.py", line 82, in __call__
    skeleton_response = self.skeleton.invoke(context)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/skeleton.py", line 154, in invoke
    return self.dispatch_request(serializer, context, instance)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/skeleton.py", line 168, in dispatch_request
    result = handler(context, instance) or {}
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/forwarder.py", line 133, in _call
    return handler(context, req)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/skeleton.py", line 118, in __call__
    return self.fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/organizations/provider.py", line 304, in describe_organization
    a = org[0]
        ~~~^^^
TypeError: 'NoneType' object is not subscriptable

Expected Behaviour:

sannyasingal@Sannyas-MacBook-Pro ~ % awslocal organizations describe-organization

An error occurred (InternalError) when calling the DescribeOrganization operation (reached max retries: 4): exception while calling organizations.DescribeOrganization: 'NoneType' object is not subscriptable

Changes

This PR excludes stack traces from HTTP responses when DEBUG=1 mode is enabled (for DEBUG=0 we already do not log stack trace in the response) and rather allow stack traces when INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is enabled.

Testing

The responses to:

local test runs by enabling and disabling INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE have been tested locally by deliberately introducing an error by assigning a = request["KeyId"][100] in describe_key handler and run awslocal kms describe-key --key-id=411cff4c-e1d0-49cc-9ec7-73ac91049d86 command ensuring expected stack trace behaviour for both cases.
CI run with a failing operation https://github.com/localstack/localstack/actions/runs/16820674492/job/47646766927?pr=12938 ensuring expected stack trace behaviour when enabled.

TODO

test CI run with a failing operation

localstack-bot · 2025-07-31T05:58:34Z

Currently, only patch changes are allowed on main. Your PR labels (semver: minor) indicate that it cannot be merged into the main at this time.

localstack-bot · 2025-07-31T05:58:34Z

Currently, only patch changes are allowed on main. Your PR labels (semver: minor) indicate that it cannot be merged into the main at this time.

github-actions · 2025-07-31T06:08:44Z

Test Results - Preflight, Unit

22 063 tests ±0 20 329 ✅ ±0 6m 22s ⏱️ +2s
1 suites ±0 1 734 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit c28d16c. ± Comparison against base commit 2d08a27.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-31T06:17:52Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 16s ⏱️ +11s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit c28d16c. ± Comparison against base commit 2d08a27.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-31T06:47:55Z

Test Results (amd64) - Integration, Bootstrap

5 files 5 suites 2h 30m 54s ⏱️
4 983 tests 4 362 ✅ 589 💤 32 ❌
4 989 runs 4 362 ✅ 595 💤 32 ❌

For more details on these failures, see this check.

Results for commit e152ef5.

♻️ This comment has been updated with latest results.

coveralls · 2025-07-31T06:52:47Z

coverage: 83.175% (+31.8%) from 51.395%
when pulling 08fc74c on rm-stack-traces-http-resp
into 476e7f3 on main.

github-actions · 2025-07-31T07:00:02Z

LocalStack Community integration with Pro

2 files ±0 2 suites ±0 1h 45m 40s ⏱️ + 1m 18s
4 624 tests +3 4 155 ✅ - 31 437 💤 +2 32 ❌ +32
4 626 runs +3 4 155 ✅ - 31 439 💤 +2 32 ❌ +32

For more details on these failures, see this check.

Results for commit e152ef5. ± Comparison against base commit 2d08a27.

This pull request removes 1 and adds 4 tests. Note that renamed tests count towards both.

tests.aws.services.cloudformation.resources.test_cdk.TestCdkInit ‑ test_cdk_bootstrap_redeploy

tests.aws.services.cloudformation.resources.test_cdk.TestCdkInit ‑ test_cdk_bootstrap[28]
tests.aws.services.cloudformation.resources.test_cdk.TestCdkInit ‑ test_cdk_bootstrap_redeploy[v20]
tests.aws.services.cloudformation.resources.test_cdk.TestCdkInit ‑ test_cdk_bootstrap_redeploy[v28]
tests.aws.services.kms.test_kms.TestKMS ‑ test_stack_trace

♻️ This comment has been updated with latest results.

thrau

general question: if we remove this, how does the devx look like for localstack developers? the stacktrace is added when DEBUG=1 is set so the stack traces appear in test failures directly, which has helped (at least me) in the past to find issues very quickly.

bentsku · 2025-07-31T16:28:39Z

general question: if we remove this, how does the devx look like for localstack developers? the stacktrace is added when DEBUG=1 is set so the stack traces appear in test failures directly, which has helped (at least me) in the past to find issues very quickly.

That's a good point: it returns the error directly in the client call, which allows the test annotation to show the error directly and not have to go through the logs of the running LocalStack instance.

I think there's a fine line with the customer running with DEBUG=1 and what we show. But returning stack traces in error is IMHO a behavior change that is a bit more intrusing that just the debug level.
Maybe this should be a different flag, and not only rely on the logging level, but with another environment variable that would be set to true while running tests? Something that we could directly automatically enable via a fixture? That way, we'd keep the internal DevX while still prevent customers to see those stack traces.

thrau · 2025-08-01T06:35:51Z

that's fair. so perhaps instead of removing the behavior (returning stack traces) all together, we could tie it to our tests being run. for example os.environ[constants.ENV_INTERNAL_TEST_RUN], will be set for integration tests and we could use that instead of config.DEBUG

github-actions · 2025-08-06T06:55:46Z

S3 Image Test Results (AMD64 / ARM64)

2 files 2 suites 8m 32s ⏱️
515 tests 465 ✅ 50 💤 0 ❌
1 030 runs 930 ✅ 100 💤 0 ❌

Results for commit c28d16c.

♻️ This comment has been updated with latest results.

sannya-singal added this to the 4.8 milestone Jul 31, 2025

sannya-singal self-assigned this Jul 31, 2025

sannya-singal added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Jul 31, 2025

sannya-singal requested a review from k-a-il July 31, 2025 08:54

sannya-singal marked this pull request as ready for review July 31, 2025 08:54

sannya-singal requested a review from thrau as a code owner July 31, 2025 08:54

k-a-il requested a review from bentsku July 31, 2025 08:56

thrau reviewed Jul 31, 2025

View reviewed changes

sannya-singal marked this pull request as draft August 1, 2025 10:23

sannya-singal force-pushed the rm-stack-traces-http-resp branch from f103562 to 5c4c022 Compare August 6, 2025 07:31

sannya-singal added 4 commits August 7, 2025 12:22

Remove stack traces from HTTP responses

86f543d

omit DEBUG and use INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE

fdabdab

add fixture to enable stack traces

fc7056c

change scope for enable_stack_trace_for_tests

0fde5fd

sannya-singal force-pushed the rm-stack-traces-http-resp branch from 5c4c022 to 0fde5fd Compare August 7, 2025 06:55

add a failing test to validate stack trace

8131028

sannya-singal changed the title ~~Remove stack traces from HTTP responses~~ Allow stack traces in HTTP responses only if INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is enabled Aug 7, 2025

sannya-singal added 3 commits August 7, 2025 17:25

test CI run with stack trace disabled

756aa0b

add error to code

a64151d

enable INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE

e152ef5

remove testing code

c28d16c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Allow stack traces in HTTP responses only if `INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE` is enabled #12938

Allow stack traces in HTTP responses only if `INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE` is enabled #12938

sannya-singal commented Jul 31, 2025 •

edited

Loading

Uh oh!

localstack-bot commented Jul 31, 2025

Uh oh!

localstack-bot commented Jul 31, 2025

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

coveralls commented Jul 31, 2025

Uh oh!

github-actions bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

thrau left a comment

Uh oh!

bentsku commented Jul 31, 2025 •

edited

Loading

Uh oh!

thrau commented Aug 1, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 6, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Allow stack traces in HTTP responses only if INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is enabled #12938

Are you sure you want to change the base?

Allow stack traces in HTTP responses only if INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is enabled #12938

Conversation

sannya-singal commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Testing

TODO

Uh oh!

localstack-bot commented Jul 31, 2025

Uh oh!

localstack-bot commented Jul 31, 2025

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

coveralls commented Jul 31, 2025

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

thrau left a comment

Choose a reason for hiding this comment

Uh oh!

bentsku commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thrau commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

S3 Image Test Results (AMD64 / ARM64)

Uh oh!

Uh oh!

Allow stack traces in HTTP responses only if `INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE` is enabled #12938

Allow stack traces in HTTP responses only if `INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE` is enabled #12938

sannya-singal commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

bentsku commented Jul 31, 2025 •

edited

Loading

thrau commented Aug 1, 2025 •

edited

Loading

github-actions bot commented Aug 6, 2025 •

edited

Loading