chore(eco): Refactors organization report building #96917

GabeVillalobos · 2025-07-31T22:44:33Z

Continuation of #96869, merge that first

Includes the following refactors:

Adds a new OrganizationReportContextFactory to clean up the prepare_organization_report task.
Moves is_empty check logic to OrganizationReportContext class, from the dedicated util where it lived before.
Adds stronger typing for ProjectContext checks when building org contexts.

Why these changes?

This is a precursor for adding metrics, SLOs, and logging for organiztion reporting logic.

This also will precede SLOs for individual email sending, which will come in a later PR as well.

…zation scheduling

codecov · 2025-07-31T22:53:19Z

Codecov Report

❌ Patch coverage is 97.68786% with 4 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/tasks/summaries/weekly_reports.py	95.16%	3 Missing ⚠️
...s/summaries/organization_report_context_factory.py	98.82%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #96917    +/-   ##
========================================
  Coverage   80.68%   80.69%            
========================================
  Files        8498     8506     +8     
  Lines      374269   374689   +420     
  Branches    24290    24290            
========================================
+ Hits       301996   302361   +365     
- Misses      71896    71951    +55     
  Partials      377      377

GabeVillalobos · 2025-07-31T22:54:06Z

src/sentry/tasks/summaries/organization_report_context_factory.py

+        with sentry_sdk.start_span(op="weekly_reports.user_project_ownership"):
+            for project_id, user_id in OrganizationMember.objects.filter(
+                organization_id=ctx.organization.id,
+                teams__projectteam__project__isnull=False,
+                teams__status=TeamStatus.ACTIVE,
+            ).values_list("teams__projectteam__project_id", "user_id"):
+                if user_id is not None:
+                    ctx.project_ownership.setdefault(user_id, set()).add(project_id)


Each of these chunks is copied pretty much verbatim from the prepare_organization_report task. It just allows us to individually wrap chunks in SLOs if we decide to go this route.

I could also see a scenario where we run each of these steps on a per project context basis and remove all of the nested mutability of passing around a partially populated OrganozationReportContext object. Future goal though 😅

GabeVillalobos · 2025-07-31T22:54:38Z

src/sentry/tasks/summaries/utils.py

+        """
+        Returns True if every project context is empty.
+        """
+        return all(
+            project_ctx.check_if_project_is_empty()
+            for project_ctx in self.projects_context_map.values()
+        )


Replaced the util for this with an actual helper method for simplicity.

GabeVillalobos · 2025-07-31T22:56:06Z

src/sentry/tasks/summaries/weekly_reports.py

+@dataclass
+class WeeklyReportProgressTracker:
+    """
+    This class is used to track the last processed org ID for a given
+    weekly report. It can either be configured with an explicit start time and
+    watermark TTL, or it will assume beginning of day, with a 7 day TTL.
+    """
+
+    beginning_of_day_timestamp: float
+    duration: int
+    _redis_connection: LocalClient
+
+    REPORT_REDIS_CLIENT_KEY: Final[str] = "weekly_reports_org_id_min"
+
+    def __init__(self, timestamp: float | None = None, duration: int | None = None):
+        if timestamp is None:
+            # The time that the report was generated
+            timestamp = floor_to_utc_day(timezone.now()).timestamp()
+
+        self.beginning_of_day_timestamp = timestamp
+
+        if duration is None:
+            # The total timespan that the task covers
+            duration = ONE_DAY * 7
+
+        self.duration = duration
+        self._redis_connection = redis.clusters.get("default").get_local_client_for_key(
+            self.REPORT_REDIS_CLIENT_KEY
+        )
+
+    @property
+    def min_org_id_redis_key(self) -> str:
+        return f"{self.REPORT_REDIS_CLIENT_KEY}:{self.beginning_of_day_timestamp}"
+
+    def get_last_processed_org_id(self) -> int | None:
+        min_org_id_from_redis = self._redis_connection.get(self.min_org_id_redis_key)
+        return int(min_org_id_from_redis) if min_org_id_from_redis else None
+
+    def set_last_processed_org_id(self, org_id: int) -> None:
+        self._redis_connection.set(self.min_org_id_redis_key, org_id)
+
+    def delete_min_org_id(self) -> None:
+        self._redis_connection.delete(self.min_org_id_redis_key)
+
+


This is part of the parent PR, ignore pls

GabeVillalobos · 2025-07-31T22:56:29Z

src/sentry/tasks/summaries/weekly_reports.py

-    if timestamp is None:
-        # The time that the report was generated
-        timestamp = floor_to_utc_day(timezone.now()).timestamp()
-
-    if duration is None:
-        # The total timespan that the task covers
-        duration = ONE_DAY * 7
-
-    batch_id = str(uuid.uuid4())
-
-    def min_org_id_redis_key(timestamp: float) -> str:
-        return f"weekly_reports_org_id_min:{timestamp}"
-
-    redis_cluster = redis.clusters.get("default").get_local_client_for_key(
-        "weekly_reports_org_id_min"
-    )
-
-    min_org_id_from_redis = redis_cluster.get(min_org_id_redis_key(timestamp))
-    minimum_organization_id = int(min_org_id_from_redis) if min_org_id_from_redis else None
+    batching = WeeklyReportProgressTracker(timestamp, duration)
+    minimum_organization_id = batching.get_last_processed_org_id()

    organizations = Organization.objects.filter(status=OrganizationStatus.ACTIVE)

-    for organization in RangeQuerySetWrapper(
-        organizations,
-        step=10000,
-        result_value_getter=lambda item: item.id,
-        min_id=minimum_organization_id,
-    ):
-        # Create a celery task per organization
-        logger.info(
-            "weekly_reports.schedule_organizations",
-            extra={
-                "batch_id": str(batch_id),
-                "organization": organization.id,
-                "minimum_organization_id": minimum_organization_id,
-            },
-        )
-        prepare_organization_report.delay(
-            timestamp, duration, organization.id, batch_id, dry_run=dry_run
-        )
-        redis_cluster.set(min_org_id_redis_key(timestamp), organization.id)
+    with WeeklyReportSLO(
+        operation_type=WeeklyReportOperationType.SCHEDULE_ORGANIZATION_REPORTS
+    ).capture() as lifecycle:
+        try:
+            batch_id = str(uuid.uuid4())


Same with these changes.

GabeVillalobos added 7 commits July 31, 2025 11:50

chore(eco): Refactors report tracker, adds failure metrics for organi…

5990678

…zation scheduling

Renames operation types, removes cancelled from failure types

d9e4e04

Adds explicit handling for timeouts

db1798e

Reorders private/public data in progress tracker

b37a0a6

Fix redis key when obtaining a client

7101357

Fixes default param error

73c319a

Adds context factory, updates existing code to use it

b476ecf

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 31, 2025

GabeVillalobos requested review from kcons and a team July 31, 2025 22:45

vercel bot deployed to Preview July 31, 2025 22:46 View deployment

GabeVillalobos commented Jul 31, 2025

View reviewed changes

Adds metrics for email delivery, updates tags to include dry runs

d367885

vercel bot deployed to Preview August 1, 2025 18:50 View deployment

vercel bot deployed to Preview August 1, 2025 19:01 View deployment

Changes dry run tag to strings

e54c100

GabeVillalobos force-pushed the gv/refactor-organization-report-context-building branch from 685b49f to e54c100 Compare August 1, 2025 19:02

vercel bot deployed to Preview August 1, 2025 19:05 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

chore(eco): Refactors organization report building #96917

chore(eco): Refactors organization report building #96917

GabeVillalobos commented Jul 31, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jul 31, 2025 •

edited

Loading

Uh oh!

GabeVillalobos Jul 31, 2025

Uh oh!

GabeVillalobos Jul 31, 2025

Uh oh!

GabeVillalobos Jul 31, 2025

Uh oh!

GabeVillalobos Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

chore(eco): Refactors organization report building #96917

Are you sure you want to change the base?

chore(eco): Refactors organization report building #96917

Conversation

GabeVillalobos commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why these changes?

Uh oh!

codecov bot commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

GabeVillalobos Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

GabeVillalobos Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

GabeVillalobos Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

GabeVillalobos Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GabeVillalobos commented Jul 31, 2025 •

edited

Loading

codecov bot commented Jul 31, 2025 •

edited

Loading