Allow stack traces in HTTP responses only if `INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE` is enabled #12938

sannya-singal · 2025-07-31T05:58:24Z

Motivation

As a part of the initiative for improving Localstack error messages, this PR aims to conditionally remove stack traces from HTTP responses. A new environment variable INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is introduced, which when enabled, allows stack traces to be included in the HTTP response.
This is expected to be always enabled during test runs, where it improves LocalStack developers devX, unless its explicitly disabled. This flag is for internal use only. The stack traces wouldn't appear in the HTTP responses for users unlike before when DEBUG is enabled.

Current Behaviour:

sannyasingal@Sannyas-MacBook-Pro ~ % awslocal organizations describe-organization

An error occurred (InternalError) when calling the DescribeOrganization operation (reached max retries: 4): exception while calling organizations.DescribeOrganization: Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/rolo/gateway/chain.py", line 166, in handle
    handler(self, self.context, response)
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/handlers/service.py", line 112, in __call__
    handler(chain, context, response)
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/handlers/service.py", line 82, in __call__
    skeleton_response = self.skeleton.invoke(context)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/skeleton.py", line 154, in invoke
    return self.dispatch_request(serializer, context, instance)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/skeleton.py", line 168, in dispatch_request
    result = handler(context, instance) or {}
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/forwarder.py", line 133, in _call
    return handler(context, req)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/skeleton.py", line 118, in __call__
    return self.fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/pro/core/services/organizations/provider.py", line 304, in describe_organization
    a = org[0]
        ~~~^^^
TypeError: 'NoneType' object is not subscriptable

Expected Behaviour:

sannyasingal@Sannyas-MacBook-Pro ~ % awslocal organizations describe-organization

An error occurred (InternalError) when calling the DescribeOrganization operation (reached max retries: 4): exception while calling organizations.DescribeOrganization: 'NoneType' object is not subscriptable

Changes

This PR excludes stack traces from HTTP responses when DEBUG=1 mode is enabled (for DEBUG=0 we already do not log stack trace in the response) and rather allow stack traces when INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is enabled.
Related PR in pro https://github.com/localstack/localstack-pro/pull/5003

Testing

The responses to:

local test runs by enabling and disabling INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE have been tested locally by deliberately introducing an error by assigning a = request["KeyId"][100] in describe_key handler and run awslocal kms describe-key --key-id=411cff4c-e1d0-49cc-9ec7-73ac91049d86 command ensuring expected stack trace behaviour for both cases.
CI run with a failing operation https://github.com/localstack/localstack/actions/runs/16820674492/job/47646766927?pr=12938 ensuring expected stack trace behaviour when enabled.

TODO

test CI run with a failing operation

localstack-bot · 2025-07-31T05:58:34Z

Currently, only patch changes are allowed on main. Your PR labels (semver: minor) indicate that it cannot be merged into the main at this time.

localstack-bot · 2025-07-31T05:58:34Z

Currently, only patch changes are allowed on main. Your PR labels (semver: minor) indicate that it cannot be merged into the main at this time.

github-actions · 2025-07-31T06:08:44Z

Test Results - Preflight, Unit

22 063 tests ±0 20 329 ✅ ±0 6m 21s ⏱️ -1s
1 suites ±0 1 734 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit df061e4. ± Comparison against base commit 2d08a27.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-31T06:17:52Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 13s ⏱️ +8s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit df061e4. ± Comparison against base commit 2d08a27.

♻️ This comment has been updated with latest results.

github-actions · 2025-07-31T06:47:55Z

Test Results (amd64) - Integration, Bootstrap

5 files 5 suites 2h 20m 29s ⏱️
4 982 tests 4 393 ✅ 589 💤 0 ❌
4 988 runs 4 393 ✅ 595 💤 0 ❌

Results for commit df061e4.

♻️ This comment has been updated with latest results.

coveralls · 2025-07-31T06:52:47Z

coverage: 83.175% (+31.8%) from 51.395%
when pulling 08fc74c on rm-stack-traces-http-resp
into 476e7f3 on main.

github-actions · 2025-07-31T07:00:02Z

LocalStack Community integration with Pro

2 files ±0 2 suites ±0 1h 42m 29s ⏱️ - 1m 53s
4 623 tests +2 4 186 ✅ ±0 437 💤 +2 0 ❌ ±0
4 625 runs +2 4 186 ✅ ±0 439 💤 +2 0 ❌ ±0

Results for commit df061e4. ± Comparison against base commit 2d08a27.

This pull request removes 1 and adds 3 tests. Note that renamed tests count towards both.

tests.aws.services.cloudformation.resources.test_cdk.TestCdkInit ‑ test_cdk_bootstrap_redeploy

tests.aws.services.cloudformation.resources.test_cdk.TestCdkInit ‑ test_cdk_bootstrap[28]
tests.aws.services.cloudformation.resources.test_cdk.TestCdkInit ‑ test_cdk_bootstrap_redeploy[v20]
tests.aws.services.cloudformation.resources.test_cdk.TestCdkInit ‑ test_cdk_bootstrap_redeploy[v28]

♻️ This comment has been updated with latest results.

thrau

general question: if we remove this, how does the devx look like for localstack developers? the stacktrace is added when DEBUG=1 is set so the stack traces appear in test failures directly, which has helped (at least me) in the past to find issues very quickly.

bentsku · 2025-07-31T16:28:39Z

general question: if we remove this, how does the devx look like for localstack developers? the stacktrace is added when DEBUG=1 is set so the stack traces appear in test failures directly, which has helped (at least me) in the past to find issues very quickly.

That's a good point: it returns the error directly in the client call, which allows the test annotation to show the error directly and not have to go through the logs of the running LocalStack instance.

I think there's a fine line with the customer running with DEBUG=1 and what we show. But returning stack traces in error is IMHO a behavior change that is a bit more intrusing that just the debug level.
Maybe this should be a different flag, and not only rely on the logging level, but with another environment variable that would be set to true while running tests? Something that we could directly automatically enable via a fixture? That way, we'd keep the internal DevX while still prevent customers to see those stack traces.

thrau · 2025-08-01T06:35:51Z

that's fair. so perhaps instead of removing the behavior (returning stack traces) all together, we could tie it to our tests being run. for example os.environ[constants.ENV_INTERNAL_TEST_RUN], will be set for integration tests and we could use that instead of config.DEBUG

github-actions · 2025-08-06T06:55:46Z

S3 Image Test Results (AMD64 / ARM64)

2 files 2 suites 8m 52s ⏱️
515 tests 465 ✅ 50 💤 0 ❌
1 030 runs 930 ✅ 100 💤 0 ❌

Results for commit df061e4.

♻️ This comment has been updated with latest results.

bentsku

I have a question regarding adding the flag to our config.py file instead of directly manipulating environment variables, not sure if I miss some context. Maybe it is because it is internal only?

For the other "test only" variables like LOCALSTACK_INTERNAL_TEST_RUN and LOCALSTACK_INTERNAL_TEST_COLLECT_METRIC, they don't change LocalStack behavior, only the tests behavior, which is maybe why they are not added to config.py.

Here we're directly directing LocalStack behavior, so I think we might be better if it is a proper config flag, what do you think?

As a side-note, it could also allow us to keep the current behavior until the next release (where we would only flip the flag) to not modify behavior suddenly, but to be honest I'm not sure it is worth it, I don't see what it would bring to our users, so I think we can change it in latest 😄

bentsku · 2025-08-08T09:10:13Z

localstack-core/localstack/aws/handlers/service.py

+            if (
+                os.environ.get(constants.INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE)
+                in constants.TRUE_STRINGS
+            ):


question: was there a specify reason that would prevent us from adding INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE to config like other configuration variables? It would more easily discoverable, and we could still patch it in the fixture with monkeypatch.setattr(config, "INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE", True)

But maybe I am missing something?

Thanks for raising this 🙌 There was no strong reason not to add it to config.py, the flag was introduced mainly for internal use so took this approach. But I agree that making it a proper config flag would improve discoverability also since this flag affects the LocalStack behaviour. I will go ahead and move it to config.py with a default of 0, and patch it to 1 in the fixture and also make relevant changes in pro 👍

k-a-il

LGTM 👍

bentsku · 2025-08-08T09:28:21Z

tests/conftest.py

+@pytest.fixture(autouse=True)
+def enable_stack_trace_for_tests(monkeypatch):
+    """
+    Ensure stack traces are enabled in HTTP responses during test sessions.
+
+    This is useful for debugging purposes.
+    """
+    monkeypatch.setenv(INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE, "1")


nit: we could create a session scope fixture too with MonkeyPatch directly: pytest-dev/pytest#1872 (comment)

See here where we do it for a class fixture:

localstack/tests/aws/scenario/bookstore/test_bookstore.py

Lines 50 to 60 in 4464cd2

@pytest.fixture(scope="class")

def patch_opensearch_strategy(self):

"""patching the endpoint strategy for opensearch to path, to make the endpoint resolution in the lambda easier"""

from _pytest.monkeypatch import MonkeyPatch

from localstack import config

mpatch = MonkeyPatch()

mpatch.setattr(config, "OPENSEARCH_ENDPOINT_STRATEGY", "path")

yield mpatch

mpatch.undo()

bentsku

LGTM, thanks for addressing the comments 🚀

bentsku · 2025-08-08T11:55:13Z

localstack-core/localstack/testing/pytest/in_memory_localstack.py

@@ -71,6 +71,8 @@ def pytest_runtestloop(session: Session):

    # configure
    os.environ[ENV_INTERNAL_TEST_RUN] = "1"
+    localstack_config.INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE = True


question: do we need this if we have the fixture? I guess it cannot hurt, so I'm fine leaving it like that 👍

sannya-singal added this to the 4.8 milestone Jul 31, 2025

sannya-singal self-assigned this Jul 31, 2025

sannya-singal added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Jul 31, 2025

sannya-singal requested a review from k-a-il July 31, 2025 08:54

sannya-singal marked this pull request as ready for review July 31, 2025 08:54

sannya-singal requested a review from thrau as a code owner July 31, 2025 08:54

k-a-il requested a review from bentsku July 31, 2025 08:56

thrau reviewed Jul 31, 2025

View reviewed changes

sannya-singal marked this pull request as draft August 1, 2025 10:23

sannya-singal force-pushed the rm-stack-traces-http-resp branch from f103562 to 5c4c022 Compare August 6, 2025 07:31

sannya-singal added 4 commits August 7, 2025 12:22

Remove stack traces from HTTP responses

86f543d

omit DEBUG and use INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE

fdabdab

add fixture to enable stack traces

fc7056c

change scope for enable_stack_trace_for_tests

0fde5fd

sannya-singal force-pushed the rm-stack-traces-http-resp branch from 5c4c022 to 0fde5fd Compare August 7, 2025 06:55

add a failing test to validate stack trace

8131028

sannya-singal changed the title ~~Remove stack traces from HTTP responses~~ Allow stack traces in HTTP responses only if INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is enabled Aug 7, 2025

sannya-singal added 3 commits August 7, 2025 17:25

test CI run with stack trace disabled

756aa0b

add error to code

a64151d

enable INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE

e152ef5

remove testing code

c28d16c

sannya-singal marked this pull request as ready for review August 8, 2025 07:39

sannya-singal requested a review from dominikschubert as a code owner August 8, 2025 07:39

bentsku reviewed Aug 8, 2025

View reviewed changes

k-a-il approved these changes Aug 8, 2025

View reviewed changes

bentsku reviewed Aug 8, 2025

View reviewed changes

fix reviews

df061e4

sannya-singal requested a review from bentsku August 8, 2025 10:46

bentsku approved these changes Aug 8, 2025

View reviewed changes

	@pytest.fixture(scope="class")
	def patch_opensearch_strategy(self):
	"""patching the endpoint strategy for opensearch to path, to make the endpoint resolution in the lambda easier"""
	from _pytest.monkeypatch import MonkeyPatch

	from localstack import config

	mpatch = MonkeyPatch()
	mpatch.setattr(config, "OPENSEARCH_ENDPOINT_STRATEGY", "path")
	yield mpatch
	mpatch.undo()

Uh oh!

Allow stack traces in HTTP responses only if INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is enabled #12938

Are you sure you want to change the base?

Allow stack traces in HTTP responses only if INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE is enabled #12938

Conversation

sannya-singal commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Testing

TODO

Uh oh!

localstack-bot commented Jul 31, 2025

Uh oh!

localstack-bot commented Jul 31, 2025

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

coveralls commented Jul 31, 2025

Uh oh!

github-actions bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

thrau left a comment

Choose a reason for hiding this comment

Uh oh!

bentsku commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thrau commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

S3 Image Test Results (AMD64 / ARM64)

Uh oh!

bentsku left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bentsku Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

sannya-singal Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

k-a-il left a comment

Choose a reason for hiding this comment

Uh oh!

bentsku Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

bentsku left a comment

Choose a reason for hiding this comment

Uh oh!

bentsku Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Allow stack traces in HTTP responses only if `INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE` is enabled #12938

Allow stack traces in HTTP responses only if `INCLUDE_STACK_TRACES_IN_HTTP_RESPONSE` is enabled #12938

sannya-singal commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

github-actions bot commented Jul 31, 2025 •

edited

Loading

bentsku commented Jul 31, 2025 •

edited

Loading

thrau commented Aug 1, 2025 •

edited

Loading

github-actions bot commented Aug 6, 2025 •

edited

Loading

bentsku left a comment •

edited

Loading