Skip to content

Secret Manager: Solve the issue for rotate secret after sub-sequent r… #12391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

mabuaisha
Copy link
Contributor

@mabuaisha mabuaisha commented Mar 16, 2025

Motivation

This PR is added in order to address the issue raised by our community 12144. The issue happens when making subsequent rotate-secret calls without specifying RotationLambdaARN and RotationRules cause ResourceNotFoundException

Changes

  • Read from rotation configurations for a given secret from the secret store when they are not provided by user
  • Reading from secret store (current rotation configuration) before rotation happened helped us to fetch these configuration in order to invoke the target lambda function

Testing

In order to test this you can run localstack locally and run the following command

Create role for lambda function

awslocal iam create-role --role-name lambda-rotation-role --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "lambda.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}'

Attach policy to role

awslocal iam put-role-policy --role-name lambda-rotation-role --policy-name secret-rotation-policy --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:PutSecretValue"
      ],
      "Resource": "*"
    }
  ]
}'

Create lambda function

awslocal lambda create-function --function-name rotate-secret-function --runtime python3.9 --role arn:aws:iam::000000000000:role/lambda-rotation-role --handler rotate_function.lambda_handler --timeout 30 --zip-file fileb://rotate_function.zip

Create secret

bash
awslocal secretsmanager create-secret --name MySecretKey --secret-string "initialValue"

First rotation

awslocal secretsmanager rotate-secret --secret-id MySecretKey --rotation-lambda-arn arn:aws:lambda:us-east-1:000000000000:function:rotate-secret-function --rotation-rules AutomaticallyAfterDays=1

Second rotation

awslocal secretsmanager rotate-secret --secret-id MySecretKey

Copy link
Contributor

@localstack-bot localstack-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welcome to LocalStack! Thanks for raising your first Pull Request and landing in your contributions. Our team will reach out with any reviews or feedbacks that we have shortly. We recommend joining our Slack Community and share your PR on the #community channel to share your contributions with us. Please make sure you are following our contributing guidelines and our Code of Conduct.

@localstack-bot
Copy link
Contributor

localstack-bot commented Mar 16, 2025

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@simonrw simonrw added the semver: patch Non-breaking changes which can be included in patch releases label Mar 18, 2025
@simonrw simonrw self-requested a review March 18, 2025 16:57
@mabuaisha
Copy link
Contributor Author

I have read the CLA Document and I hereby sign the CLA

localstack-bot added a commit that referenced this pull request Mar 21, 2025
@k-a-il k-a-il added this to the Playground milestone Mar 24, 2025
Comment on lines 4586 to 4625
"tests/aws/services/secretsmanager/test_secretsmanager.py::TestSecretsManager::test_rotate_secret_with_lambda_success[False]": {
"recorded-date": "15-03-2025, 08:34:38",
"recorded-content": {
"rotate_secret_immediately_1": {
"ARN": "arn:<partition>:secretsmanager:<region>:111111111111:secret:<SecretId-0idx><ArnPart-0idx>",
"Name": "<SecretId-0idx>",
"VersionId": "<version_uuid:2>",
"ResponseMetadata": {
"HTTPHeaders": {},
"HTTPStatusCode": 200
}
},
"describe_secret_rotated_1": {
"ARN": "arn:<partition>:secretsmanager:<region>:111111111111:secret:<SecretId-0idx><ArnPart-0idx>",
"CreatedDate": "datetime",
"Description": "testing rotation of secrets",
"LastChangedDate": "datetime",
"Name": "<SecretId-0idx>",
"NextRotationDate": "datetime",
"RotationEnabled": true,
"RotationLambdaARN": "<lambda-arn:1>",
"RotationRules": {
"AutomaticallyAfterDays": 1
},
"VersionIdsToStages": {
"<version_uuid:1>": [
"AWSCURRENT"
]
},
"ResponseMetadata": {
"HTTPHeaders": {},
"HTTPStatusCode": 200
}
},
"list_secret_versions_rotated_1_1": {
"ARN": "arn:<partition>:secretsmanager:<region>:111111111111:secret:<SecretId-0idx><ArnPart-0idx>",
"Name": "<SecretId-0idx>",
"Versions": [
{
"CreatedDate": "datetime",
"KmsKeyIds": [
"DefaultEncryptionKey"
],
"VersionId": "<version_uuid:1>",
"VersionStages": [
"AWSCURRENT"
]
}
],
"ResponseMetadata": {
"HTTPHeaders": {},
"HTTPStatusCode": 200
}
}
}
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I don't think this is a valid test snapshot. I can't see where this function was run with the parameter False in the code

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right, I think I missed this while I was doing some refactoring and forgot to change it back again. It should not be False, it should be None indeed, I can update this

"tests/aws/services/secretsmanager/test_secretsmanager.py::TestSecretsManager::test_rotate_secret_with_lambda_success": {
"last_validated_date": "2024-03-15T08:12:22+00:00"
},
"tests/aws/services/secretsmanager/test_secretsmanager.py::TestSecretsManager::test_rotate_secret_with_lambda_success[False]": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per my previous comment, can we remove this validation if it is not a currently tracked test combination?

Copy link
Contributor Author

@mabuaisha mabuaisha Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as I mentioned before, you are right. This should be None not False. The test for False is already existed

Copy link
Contributor

@macnev2013 macnev2013 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @mabuaisha 🚀

The PR looks good to me overall - I just have some minor concerns/suggestion.

VersionStage="AWSPENDING",
RemoveFromVersionId=token,
)
logger.info(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we can use logger.debug instead of logger.info here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason why I choose the info is because in the same function for AWSCURRENT and others log levels we are also using info so I thought this was added in purpose, so I stick to use info in this case

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only in the lambda function body though right @macnev2013?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that’s right @simonrw. This can be ignored.

@@ -782,7 +788,7 @@ def backend_rotate_secret(
pass

secret.rotation_lambda_arn = rotation_lambda_arn
secret.auto_rotate_after_days = rotation_rules.get(rotation_days, 0)
secret.auto_rotate_after_days = rotation_period
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to add default value in case rotation_period is not set

Suggested change
secret.auto_rotate_after_days = rotation_period
secret.auto_rotate_after_days = rotation_period or 0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this in purpose without or 0 because usually the default value for https://github.com/localstack/moto/blob/localstack/moto/secretsmanager/models.py#L118 based on this logic usually is 0 in case its not set.

For extra check, we can still also fallback to 0
secret.auto_rotate_after_days = rotation_period or 0

I will update this

Comment on lines 120 to 122
sm_snapshot.add_transformer(
sm_snapshot.transform.key_value("RotationLambdaARN", "lambda-arn")
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we could move this to localstack.testing.snapshots.transformer_utility.TransformerUtility.secretsmanager_api since it's a geneic transformer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely, I can move it there to make it a generic transformer since RotationLambdaARN could be user later on with multiple tests. define it there could be useful later on

Comment on lines 753 to 756
else:
# Resolve auto_rotate_after_days and fallback to previous value
# if its missing from the current request
rotation_period = secret.auto_rotate_after_days
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could cause 'rotation_period' variable to be referenced before assignment here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah thats right, I will init rotation_period = 0

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it always 0 or can we move the rotation_period = secret.auto_rotate_after_days above the if block and have that as the default, then remove the else branch?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can remove the else branch and make it the default one. However, I will fallback to 0 like this

rotation_period = secret.auto_rotate_after_days or 0

That way even if previous value for for auto_rotate_after_days that is part of secret is not integer, we are make sure always if the value is not set the default one is 0

raise InvalidRequestException(
"An error occurred (InvalidRequestException) when calling the RotateSecret operation: You tried to \
perform the operation on a secret that's currently marked deleted."
)
# Resolve rotation_lambda_arn and fallback to previous value if its missing
# from the current request
rotation_lambda_arn = rotation_lambda_arn or secret.rotation_lambda_arn
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the secret is being rotated first time and we haven't specified the rotation_lambda_arn?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, this was an existing behavior even before I introduce my fix. In case for the first time rotation we did not specify rotation_lambda_arn then it will raise error, check this code here


    try:
        lm_client = connect_to(region_name=self.region_name).lambda_
        lm_client.get_function(FunctionName=rotation_lambda_arn)
    except Exception:
        raise ResourceNotFoundException("Lambda does not exist or could not be accessed")

This will raise error whenever we invoked it from the cli

An error occurred (ResourceNotFoundException) when calling the RotateSecret operation: Lambda does not exist or could not be accessed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What error message does AWS return in this case? Does it match the error we raise in this circumstance? I suspect not.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right. I will update the code because I need to add a check earlier to raise an error that match AWS one

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error mainly look like this from AWS

        "Error": {
          "Code": "InvalidRequestException",
          "Message": "No Lambda rotation function ARN is associated with this secret."
        },

Comment on lines 178 to 199
assert response["VersionIdsToStages"][initial_secret_version] == ["AWSPREVIOUS"]
assert response["VersionIdsToStages"][rot_res["VersionId"]] == ["AWSCURRENT"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: we usually don't use assertion in the helper methods.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a pattern you followed when you write tests ? or you are talking in general ?. The reason why I add this method is to avoid duplication for multiple tests and I thought usually this could be re-used only within multiple tests for assertions. However, I'm open to change this if you think this could create confusion or violates any sort of patterns/convention you usually followed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we normally prefer two things:

  1. assertions in the test body, not in helper methods
  2. fixtures over helper methods

I personally find it clearer for the assertion failure to occur in the test body so it's clear what's being tested at first glance in the test code itself, rather than having to go diving into helper methods. It also makes the helper methods more flexible, as the assertions may not be consistent from test to test.

For the second item, we often create fixtures that look like:

def my_fixture(aws_client, snapshot):  # other fixtures
    def inner(name: str, *args, **kwargs):  # variables taken from the test body
        # do something with both arguments and fixtures, e.g.:
        return aws_client.sts.get_caller_identity()
    
    return inner

then in the test we only need to include the fixture and use the "inner" function, which simplifies the usage of that fixture:

def test_foo(my_fixture):
    identity = my_fixture("name")

See how we could call AWS methods without passing in the aws_client.

These are just preferences and I'm not asking you to refactor the helper methods into fixtures, but perhaps you could consider moving the assertions to the test bodies?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense, I think you raised a valid point here. I will do some refactoring to this

Comment on lines 633 to 639
def test_rotate_secret_multiple_times_with_lambda_success(
self,
sm_snapshot,
secret_name,
create_secret,
create_lambda_function,
aws_client,
set_rotation_on_second_run,
):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be worth testing - what happens if the function_arn is not provided for the first time rotation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing, I can add this as a new test since it will fail if we did not specify the function_arn. We do not have a test for this case but the rotation backend can handle that as expected based on my previous comment. I will add a test to catch this

@mabuaisha mabuaisha requested a review from steffyP as a code owner March 27, 2025 16:42
@mabuaisha mabuaisha force-pushed the secret-manager-rotation-secret-issue branch 4 times, most recently from 1e21cf0 to 1d2726b Compare March 27, 2025 19:12
@mabuaisha
Copy link
Contributor Author

@macnev2013 Thanks a lot for taking the time to review my PR. I replied to your comments and pushed changes as a result of your comments. Let me know if need to discuss this further

Copy link
Contributor

@simonrw simonrw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope have added some clarity and context to some of the points from @macnev2013 particularly around the use of helper methods and assertions.

I have an issue with test_first_rotate_secret_with_missing_lambda_arn: you are skipping verification of the two fields that matter for this test. It is specifically around capturing the error that is presented, but you have skipped the wording.

I think this may cause confusion to our users who get an invalid response. I would like to see this implemented correctly.

raise InvalidRequestException(
"An error occurred (InvalidRequestException) when calling the RotateSecret operation: You tried to \
perform the operation on a secret that's currently marked deleted."
)
# Resolve rotation_lambda_arn and fallback to previous value if its missing
# from the current request
rotation_lambda_arn = rotation_lambda_arn or secret.rotation_lambda_arn
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What error message does AWS return in this case? Does it match the error we raise in this circumstance? I suspect not.

Comment on lines 178 to 199
assert response["VersionIdsToStages"][initial_secret_version] == ["AWSPREVIOUS"]
assert response["VersionIdsToStages"][rot_res["VersionId"]] == ["AWSCURRENT"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we normally prefer two things:

  1. assertions in the test body, not in helper methods
  2. fixtures over helper methods

I personally find it clearer for the assertion failure to occur in the test body so it's clear what's being tested at first glance in the test code itself, rather than having to go diving into helper methods. It also makes the helper methods more flexible, as the assertions may not be consistent from test to test.

For the second item, we often create fixtures that look like:

def my_fixture(aws_client, snapshot):  # other fixtures
    def inner(name: str, *args, **kwargs):  # variables taken from the test body
        # do something with both arguments and fixtures, e.g.:
        return aws_client.sts.get_caller_identity()
    
    return inner

then in the test we only need to include the fixture and use the "inner" function, which simplifies the usage of that fixture:

def test_foo(my_fixture):
    identity = my_fixture("name")

See how we could call AWS methods without passing in the aws_client.

These are just preferences and I'm not asking you to refactor the helper methods into fixtures, but perhaps you could consider moving the assertions to the test bodies?

Comment on lines 704 to 697
assert "RotationEnabled" not in describe_secret
assert "RotationRules" not in describe_secret
assert "RotationLambdaARN" not in describe_secret
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: these assertions are not required giving we are matching the snapshot above


@markers.snapshot.skip_snapshot_verify(paths=["$..Error", "$..Message"])
@markers.aws.validated
def test_first_rotate_secret_with_missing_lambda_arn(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is a start, however it does not execute the rotation itself, therefore testing the error message given by AWS if a lambda function arn is not specified. This will help clarify the error handling here.

Here is a demonstration of the confusion that extracting these helper methods causes. I (incorrectly) believed that you hadn't tried to rotate the lambda function, but when I looked in the snapshot I saw that the _setup_invalid_rotation_secret method tries to rotate a secret without specifying a function ARN.

I would find it much clearer if the test was more like:

  • create secret (using our create_secret fixture)
  • try to rotate secret, wrapped in pytest.raises(...)
  • capturing a snapshot of the error message
  • (optional) a describe secret call, however this is covered by other more specific tests

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I have been more lean toward refactor this since it created this confusion. Thanks for raising this

Comment on lines 3791 to 3789
"AWSPREVIOUS"
"AWSCURRENT"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q: do you understand why this changed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My assumption for this. The list is not deterministic as I noticed that sometimes version1 could be the previous version and version 2 is the current version and vice versa. Thats the reason why sometimes version 1 could be previous and sometimes it could be current. The same thing for version 2 as well


@markers.snapshot.skip_snapshot_verify(paths=["$..Error", "$..Message"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: you have skipped the key part of this test => the error message. Can you remove this verify and implement the correct behaviour?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing, I will remove the check it. The only thing that need to be updated is to add a check that raise an error InvalidRequestException that match AWS one

VersionStage="AWSPENDING",
RemoveFromVersionId=token,
)
logger.info(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only in the lambda function body though right @macnev2013?

@mabuaisha mabuaisha force-pushed the secret-manager-rotation-secret-issue branch from b8e3ea6 to b14f7f2 Compare March 29, 2025 10:01
@mabuaisha mabuaisha force-pushed the secret-manager-rotation-secret-issue branch 2 times, most recently from 5baccec to 9aaf9c2 Compare March 29, 2025 10:04
… missing function arn for the first time rotation
@mabuaisha mabuaisha force-pushed the secret-manager-rotation-secret-issue branch from 9aaf9c2 to 8b3a1a3 Compare March 30, 2025 11:46
@simonrw simonrw merged commit 727155e into localstack:master Apr 4, 2025
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants