bug: Localstack CloudWatch Start Query Error with Grafana #12185
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
AWS logs.StartQuery => 500 (InternalError)
Getting Internal exception for localstack cloud watch start query using Grafana, although it's working fine with awslocal logs get-log-events. Not sure if it's an issue with Grafana Query.
Expected Behavior
Should be able to query localstack cloudwatch logs using Grafana.
How are you starting LocalStack?
Docker compose.
Steps To Reproduce
Docker compose Localstack and Grafana
services:
localstack:
container_name: localstack
image: localstack/localstack
ports:
- 4566:4566
- 4510-4559:4510-4559
environment:
- PERSISTENCE=1
- AWS_ACCESS_KEY_ID=local
- AWS_SECRET_ACCESS_KEY=local
- DEBUG=1
grafana:
container_name: grafana
image: grafana/grafana
ports:
- 3000:3000
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_DISABLE_LOGIN_FORM=true
- GF_USERS_DEFAULT_THEME=lightCreate log group and log streams and put event.
# awslocal logs create-log-group --log-group-name local-log-group --region=ap-south-1
# awslocal logs create-log-stream --log-group-name local-log-group --log-stream-name local-log-stream --region=ap-south-1
# awslocal logs describe-log-groups --region=ap-south-1
{
"logGroups": [
{
"logGroupName": "local-log-group",
"creationTime": 1737816952895,
"metricFilterCount": 0,
"arn": "arn:aws:logs:ap-south-1:000000000000:log-group:local-log-group:*",
"storedBytes": 0
}
]
}
# awslocal logs describe-log-streams --region=ap-south-1 --log-group-name local-log-group
{
"logStreams": [
{
"logStreamName": "local-log-stream",
"creationTime": 1737816959145,
"arn": "arn:aws:logs:ap-south-1:000000000000:log-group:local-log-group:log-stream:local-log-stream",
"storedBytes": 0
}
]
}
# awslocal logs put-log-events --region=ap-south-1 --log-group-name local-log-group --log-stream-name local-log-stream --log-events timestamp=1737817421872,message=Hello
{
"nextSequenceToken": "00000000000000000000000000000000000000000000000000000004"
}
# awslocal logs put-log-events --region=ap-south-1 --log-group-name local-log-group --log-stream-name local-log-stream --log-events timestamp=1737817437715,message=World
{
"nextSequenceToken": "00000000000000000000000000000000000000000000000000000005"
}
# awslocal logs get-log-events --region=ap-south-1 --log-group-name local-log-group --log-stream-name local-log-stream
{
"events": [
{
"timestamp": 1737817421872,
"message": "Hello",
"ingestionTime": 1737817449731
},
{
"timestamp": 1737817437715,
"message": "World",
"ingestionTime": 1737817458357
}
],
"nextForwardToken": "f/00000000000000000000000000000000000000000000000000000001",
"nextBackwardToken": "b/00000000000000000000000000000000000000000000000000000000"
}
Open Grafana and add Cloudwatch datasource with Localstack endpoint.
Explore and select log group and execute query. (No data found)
Localstack Exception / Error
2025-01-25 20:43:21 2025-01-25T15:13:21.538 INFO --- [ asgi_gw_0] localstack.request.aws : AWS logs.StartQuery => 500 (InternalError)
2025-01-25 20:43:21 2025-01-25T15:13:21.830 DEBUG --- [ asgi_gw_2] rolo.gateway.wsgi : POST 192.168.1.34:4566/
2025-01-25 20:43:21 2025-01-25T15:13:21.831 ERROR --- [ asgi_gw_2] l.aws.handlers.logging : exception during call chain
2025-01-25 20:43:21 Traceback (most recent call last):
2025-01-25 20:43:21 File "/opt/code/localstack/.venv/lib/python3.11/site-packages/rolo/gateway/chain.py", line 166, in handle
2025-01-25 20:43:21 handler(self, self.context, response)
2025-01-25 20:43:21 File "/opt/code/localstack/localstack/aws/handlers/service.py", line 113, in __call__
2025-01-25 20:43:21 handler(chain, context, response)
2025-01-25 20:43:21 File "/opt/code/localstack/localstack/aws/handlers/service.py", line 83, in __call__
2025-01-25 20:43:21 skeleton_response = self.skeleton.invoke(context)
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/localstack/aws/skeleton.py", line 154, in invoke
2025-01-25 20:43:21 return self.dispatch_request(serializer, context, instance)
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/localstack/aws/skeleton.py", line 168, in dispatch_request
2025-01-25 20:43:21 result = handler(context, instance) or {}
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/localstack/aws/forwarder.py", line 143, in _call
2025-01-25 20:43:21 return fallthrough_handler(context, req)
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/localstack/services/moto.py", line 88, in _proxy_moto
2025-01-25 20:43:21 return call_moto(context)
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/localstack/services/moto.py", line 49, in call_moto
2025-01-25 20:43:21 return dispatch_to_backend(context, dispatch_to_moto, include_response_metadata)
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/localstack/aws/forwarder.py", line 188, in dispatch_to_backend
2025-01-25 20:43:21 http_response = http_request_dispatcher(context)
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/localstack/services/moto.py", line 119, in dispatch_to_moto
2025-01-25 20:43:21 response = dispatch(request, raw_url, request.headers)
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/.venv/lib/python3.11/site-packages/moto/core/responses.py", line 270, in dispatch
2025-01-25 20:43:21 return cls()._dispatch(*args, **kwargs)
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/.venv/lib/python3.11/site-packages/moto/core/responses.py", line 478, in _dispatch
2025-01-25 20:43:21 return self.call_action()
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/.venv/lib/python3.11/site-packages/moto/core/responses.py", line 575, in call_action
2025-01-25 20:43:21 response = method()
2025-01-25 20:43:21 ^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/.venv/lib/python3.11/site-packages/moto/logs/responses.py", line 416, in start_query
2025-01-25 20:43:21 query_id = self.logs_backend.start_query(
2025-01-25 20:43:21 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-25 20:43:21 File "/opt/code/localstack/.venv/lib/python3.11/site-packages/moto/logs/models.py", line 1202, in start_query
2025-01-25 20:43:21 for log_group_name in log_group_names:
2025-01-25 20:43:21 TypeError: 'NoneType' object is not iterable
Environment
- OS: Mac
- LocalStack:
LocalStack version: 4.0.4.dev135
LocalStack build date: 2025-01-24
LocalStack build git hash: 179031879
Anything else?
https://community.grafana.com/t/grafana-cloudwatch-loggroups-query-error/87779/3 tried but isn't working.