Rework RegionBackend #6444
Rework RegionBackend #6444
Conversation
|
I love the direction this is taking @viren-nadkarni , really cool stuff! I particularly like the idea of scoping attributes explicitly via descriptors. the naming is also really clear and easy to understand. i played around a bit with my own version of this and noticed how you totally nailed all the abstractions! here are some of my other initial thoughts:
i would love to introduce a type of API: class S3Store(Store):
buckets: Dict[str, Bucket] = GlobalAttribute()
something_local: Dict[str, str] = LocalAttribute()
store = S3Store()
store.buckets["my-bucket"] # totally works because buckets is global
store.something_local["foo"] # doesn't work (region/account context missing)
view = store.view(region="us-east-1", account="000...")
view.something_local["foo"] # works!
view.buckets["my-bucket"] # also works!i have some code that makes this work, but the more i look at this though i'm unsure whether it's really better :-) |
viren-nadkarni
force-pushed
the
provider-stores
branch
from
189c8c7 to
f0fc9ba
Compare
|
Thanks for the feedback @thrau! Wrt to nested dicts, we could certainly use aggregate keys, and it would improve the performance too. However, I have a feeling that there might be benefits in using the nested structure just as Moto v4. Cloud pods and persistence mechanism logic would see dramatic simplification. For the other points, they look promising candidates for following iterations! |
There was a problem hiding this comment.
This is great, as I already commented :-)
My only concern would be the similar naming of BaseStore and AccountRegionStore, which somehow indicates that one is derived by the other, where really BaseStore is the schema of the store, whereas AccountRegionStore is the actual instance managing the data. I'm wondering whether we could improve the naming to better reflect this. Either the Store is a schema/descriptor, or the AccountRegionStore is a view.
There's a slight mismatch there also in terms of functionality. Since the RegionStore/AccountRegionStore are managing the data of the store, it feels to me a bit unintuitive that the BaseStore is also doing state management (via reset), which is in the current structure something i would expect the AccountRegionStore to do.
I'm happy to merge this and then iterate afterwards when we start using it.
Also start calling encapsulations as Bundles
|
@thrau I see what you actually meant by your original comment! Let's keep the using store for the model defintions and the encapsulating views as bundles -- thoughts? Your other point about reset logic also makes sense 👍 It's been moved to RegionBundle |
Summary
This PR introduces Stores, which supercede RegionBackends.
Stores provide in-memory storage for all provider data. They are analogous to the LocalStack's RegionBackend or Moto's BackendDict.
The terminology used, Store, is to avoid confusion with Backend which is used by Moto. The usage pattern is similar to Moto's where a singleton tracks all the stores for a service (see example below with
sqs_stores)This rework is influenced by Moto v4's approach of using nested dicts to delimit data by AWS account IDs and regions. The usage pattern is inspired by SQLAlchemy and its use of descriptor protocols.
Problem with RegionBackend
RegionBackend relies on voodoo magic to support multi-accounts. It is complex and breaks serialisation, causing compatibiltiy issues with persistence and cloud pods.
Usage pattern
Followup in separate PRs