Motivation

With the requirement of having more up-to-date feature documentation, this PR is exploring one approach, where we the documentation will live inside the code base.

This is only a first PoC. It includes the entire feature catalog for KMS, and only a few selected features for cloudwatch.
We added a script that is creating a json-output with the data from this PoC.

Basic Idea

For this PoC the main idea is to have python classes that will document certain features.
E.g. it provides base class ServiceFeature, which will then be enhanced by service specific features, like KmsFeature. The KmsFeatures in turn can further be split into sub classes, and you can add as many class-attributes as required (e.g. general docs description, limitations, implementation status etc).

This part of the feature catalog documentation is static, e.g. the classes itself already document the behavior.

Decorators

The classes can be used as decorators for functions or api-operations. In that case, we can collect the data dynamically, e.g. by iterating through all service-packages and collect the function/operation names and map them to the specific features.

In a similar way, we are suggesting two approaches to document specific limitations for an api-operation.
This could be useful in case we use this docs also for creating the api-coverage docs, and also if you have some specific limitations that you feel shouldn't be part of the python-docs limitation, as they are very specific.

The suggested approaches:

1. Using sub classes for each limitation

There is a class ApiCoverage, which should be used a base class for any additional api-limitation that you want to document, e.g.

Downside: for every limitation, you have to create a new subclass, maintaining may be more tedious
Advantage: code is not cluttered too much in case there are a lot of limitations

2. Using a generic class to document limitation

Using a decorator ApiLimitation that takes an input argument limitation that can be set dynamically for as many operations as required.

Downside: if there are a lot of limitations, this may clutter the code (one way to improve this could be to use string constants in that case, that are defined in another file)
Advantage: no need to recreate subclasses

Here is a sample on how this would be applied for both approaches:

General Evaluation

Advantages:

• code and documentation are close, whenever there is a minor or major label on the PR we can assume that the docs should also be adapted
• Flexible model, we can create machine-readable output from the data and then further use it to create documents, official docs etc
• api-operations or functions can be marked to be part of a feature, but it's not required
• we can add feature documentation for different provider implementations (e.g. v1, v2)

Disadvantages:

• using decorator pattern feels a bit abusive, as we are not really adding functionality to the code
• keeping things up-to-date may still be tricky

Further improvement ideas

• define json-schema and validate that certain attributes are part of each feature definition
  • this could be helpful when we want to further parse the output for docs generation etc.
  • could be run in a workflow

Changes

• PoC for feature catalog
• added script playground.py to parse the data:

{
  "ServiceFeature": {
    "CloudWatchFeature": {
      "docs": {
        "implementation_status": "ImplementationStatus.PARTIALLY_IMPLEMENTED"
      },
      ....
      "Alarm": {
        "docs": {
          "general_docs": "Alarms are used to set thresholds for metrics and trigger actions",
          "supported": "SupportStatus.SUPPORTED_PARTIALLY_EMULATED",
          "implementation_status": "ImplementationStatus.PARTIALLY_IMPLEMENTED",
          "aws_docs_link": "https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html"
        },
        "CompositeAlarm": {
          "docs": {
            "supported": "SupportStatus.SUPPORTED_MOCKED_ONLY",
            "aws_docs_link": "https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Create_Composite_Alarm.html"
          },
          "operations": [
            "localstack.services.cloudwatch.provider_v2.CloudwatchProvider.put_composite_alarm"
          ]
        },
        "MetricAlarm": {
          "docs": {
            "supported": "SupportStatus.SUPPORTED",
            "supported_triggers": [
              "SNS",
              "Lambda"
            ]
          },
          "operations": [
            "localstack.services.cloudwatch.provider_v2.CloudwatchProvider.put_metric_alarm"
          ]
        },
        "operations": [
          "localstack.services.cloudwatch.provider_v2.CloudwatchProvider.delete_alarms",
          "localstack.services.cloudwatch.provider_v2.CloudwatchProvider.describe_alarm_history",
          "localstack.services.cloudwatch.provider_v2.CloudwatchProvider.describe_alarms",
          "localstack.services.cloudwatch.provider_v2.CloudwatchProvider.describe_alarms_for_metric",
          "localstack.services.cloudwatch.provider_v2.CloudwatchProvider.disable_alarm_actions",
          "localstack.services.cloudwatch.provider_v2.CloudwatchProvider.enable_alarm_actions",
          "localstack.services.cloudwatch.provider_v2.CloudwatchProvider.set_alarm_state",
          "localstack.services.cloudwatch.provider_v2.delete_alarms_helper"
        ]
      }
    },
    "KMSFeature": {
      "docs": {
        "implementation_status": "ImplementationStatus.PARTIALLY_IMPLEMENTED"
      },
        "Provisioning": {
          "docs": {
            "general_docs": "Manage the creation and modification of the keys.",
            "implementation_status": "ImplementationStatus.FULLY_IMPLEMENTED",
            "support_type": "SupportStatus.SUPPORTED",
            "aws_docs_link": "https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html",
            "api_operations": [
              "CreateKey"
            ]
          },
          "operations": [
            "localstack.services.kms.provider.KmsProvider.create_key",
            "localstack.services.kms.provider.KmsProvider.update_key_description"
          ],
          "api_limitations": {
            "localstack.services.kms.provider.KmsProvider.create_key": "Status 'Updating' is not supported."
          }
        },
    ...
    }