[--Dashboard-- data source] Implement AdHoc filtering #108011

samjewell · 2025-07-11T10:34:12Z

What is this feature?

Implementing AdHoc filtering for the --Dashboard-- data source, behind an experimental feature toggle.

AdHoc.filtering.for.Dashboard.ds_small.mp4

I've kept the scope purposefully small here in order to keep this PR small, and get it in front of our stakeholders as soon as possible. It includes support only for:

String and numeric fields
Operators = and !=

For cases other than these we simply reject all rows for now (to clearly indicate to the user that the newly added filter is not supported). We can extend support or add error-messages at a later date, as we find they are needed.

We also haven't implemented getTagKeys or getTagValues, again to keep scope small. We anticipate these filters being applied primarily by people clicking on data within panels, and not by hand-crafting the filter text in the filter input itself at the top of the dashboard (which is what getTagKeys/Values would enable).

Why do we need this feature?

BI dashboarding tools such as Tableau and Looker allow for "Panel to Panel filtering", which is shown here in the video

Tableau-example-small.mp4

So users can click a value in a panel to immediately filter the whole dashboard (and all other panels) to the value they clicked. They can then continue clicking and filtering by clicking on other panels, and other values in those panels, to keep drilling and drilling down to the small set of data that they want to study.

AdHoc filters are a good fit for this because they scale well:

They apply to all panels on a dashboard (which are powered by the single datasource they are scoped to), and
They can contain multiple filter key-value pairs in a single filter, and
They require zero per-panel configuration: They apply automatically.

BUT
Until now they've only been implemented for:

Table cells (will be addressed in another PR), and
A limited number of Observability datasources: Prom, Loki, Elastic, Tempo, Pyroscope, and a few others (which we address in THIS PR)

Our stakeholder wants Panel-to-Panel filtering to work with their BigQuery data from their data warehouse. But we can't implement AdHoc filtering in BigQuery itself because we can't rely on a column being present in another panel, even if it is powered by the same underlying table. If we try to filter to Customer BBC for example, by landing a WHERE 'customer' = 'bbc' clause into our panel query, we expect it to simply fail with a syntax error such as customer column not found for many of the panels on the dashboard, where for example the Customer column will not be present in the outermost query. It may have been aggregated away already, or never even queried, given there are a great many tables in many typical SQL databases encountered in the wild.

The dashboards we anticipate in response to this are as follows:

The designer configures a primary query or queries that will power the dashboard. These are unfiltered

^ This primary query will exist as a table panel that contains the full dataset. At this point we know which columns are present in that dataset, and can rely on them always being present. We won't ever get "column not found" errors. Then we can use the --Dashboard-- datasource to power every other panel on the dashboard, and then (through this PR) enable Ad-hoc filtering on all of those panels.

Who is this feature for?

BI users, building dashboards for other viewer-users, where the viewer-users need to filter and drill the dashboard quickly and easily on a variety of dimensions.

Which issue(s) does this PR fix?:

None

Special notes for your reviewer:

Please check that:

It works as expected from a user's perspective.
If this is a pre-GA feature, it is behind a feature toggle.
No docs, as this is experimental

samjewell · 2025-07-14T16:18:38Z

Here's the dashboard-JSON that I've been using to test out this work:

dashboard.json

Import it from the /dashboard/import URL, and then choose a TestData datasource to get it working locally.

It'll ask you for a Prom datasource too - I've just been running Prom locally, which I installed via Homebrew on my mac.

packages/grafana-ui/src/components/Table/CellActions.tsx

public/app/plugins/datasource/dashboard/datasource.ts

packages/grafana-ui/src/components/Table/CellActions.tsx

public/app/plugins/datasource/dashboard/datasource.ts

alexjonspencer1 · 2025-07-14T21:19:25Z

Looking good, @samjewell! Left some comments for you. The one blocking change would be applying this to TableNextGen instead of old table.

Also, when I pulled the branch down locally to review and test (pulled it down fresh), there were a bunch of changes to sync. Could you double-check that everything’s committed and pushed cleanly on your end? I'd love to test with a clean branch! 😺

pkg/services/featuremgmt/toggles_gen.json

@leeoniya

…ce`" This reverts commit 307fbf3. As per PR feedback from @leeoniya and @alexjonspencer1 #108011 (comment) > these changes should instead go into TableCellActions.tsx of TableNG So removing the changes from here. In fact TableCellActions.tsx already has this change so no further work is needed. It's here: https://github.com/grafana/grafana/blob/main/packages/grafana-ui/src/components/Table/TableNG/Cells/TableCellActions.tsx#L67-L78

As per PR feedback @ #108011 (comment)

Starting with string-fields only

But retain maintainability/extensability

ephemeral-instances-bot · 2025-07-22T09:46:22Z

Preparing your instance. A comment containing your instance's url will be added to this PR when the instance is ready.
Your instance will be ready in ~10 minutes. Follow the workflow progress
Slack channel: #proj-ephemeral-hg-instances
Building instance with sj/dashboard-ds-implements-adhoc-filtering oss branch and main enterprise branch. How to choose a branch

ephemeral-instances-bot · 2025-07-22T10:03:57Z

Your instance can be accessed at: https://ephemeral15111821108011samjew.grafana-dev.net
The instance is not using the CDN assets.
How to access / How to update instance config / How to build a specific branch

public/app/plugins/datasource/dashboard/datasource.ts

@mdvictor

Thanks to @mdvictor for code-review

alexjonspencer1 · 2025-07-22T17:19:14Z

NICE! I like where this PR is at - awesome adding some tests. They seem great to me. Just two items and I'll be more than happy to ✅ this PR.

Would you mind addressing Victor and Kristina's comment as part of this PR? Making the matchingRows param optional? I don't think it should be too big of a lift!
Have you had the chance to test this on a frame with 1K + rows by chance? I'd be curious how the filtering impacts performance.

samjewell · 2025-07-22T22:23:30Z

Would you mind addressing Victor and Kristina's comment as part of this PR? Making the matchingRows param optional? I don't think it should be too big of a lift!

Done ✅

Have you had the chance to test this on a frame with 1K + rows by chance? I'd be curious how the filtering impacts performance.

Also done ✅ . It feels immediate to me, on a table which is 17k rows, 88 columns, 1.5M cells

alexjonspencer1

Looks good to me! The RevOps folks will be so hyped on this 🚀 AWESOME work!

mdvictor

I was also wondering about performance on a large dataset, thanks for bringing that up @alexjonspencer1 and for checking it out @samjewell!

LGTM!

samjewell · 2025-07-23T10:45:26Z

I wonder if this could potentially be simplified a bit in the query by moving some of it in the upcoming getFiltersApplicability method here.

From what I gather you basically check which filters are applicable in the dataframe before actually filtering the data and I think that part might potentially move out of query.

Hey @mdvictor
I started to look at this, since both this and your PR have merged to main.

I saw your PRs #107775 and #106756, and specifically your comment in the PR description:

We want to see, when carrying filters from one dashboard to another, which of them are still applicable and which are not

I'm a little curious about how this plays out in practice, so I wanted to ask: Which datasources is this relevant for? I was thinking about Prom and Loki, and my understanding is that ALL filters for Prom or Loki that are valid on one dashboard should remain valid on another dashboard - that validity isn't affected by the dashboard, but only by the data source ID, as it's the datasource which contains the labels and their values. WDYT? Is my understanding correct?

I'll just feel more confident implementing this once I know the use-case we are building it for.

EDIT:
I'm also wondering how I can see this in the UI? Has Dashboards implemented any UI to surface the reason to end users? Is there an additional feature-toggle I need to enable to see these for myself?

mdvictor · 2025-07-23T12:02:33Z

Hey @samjewell,
We are planning to start with Prometheus for this. So there, for example, you could have some filters and then, through a link, go to another potentially unrelated dashboard and carry those filters with you in the URL. That's when getFiltersApplicability would verify based on the given filters and queries which of them can be applied to the queries and which wouldn't, thus returning the list of filters where non-applicable ones would be crossed out and offer you a visual cue on which are not to be applied, instead of applying everything , showing them all in the filters bar and then on query time result in panels that show No Data.

This is the continuation PR of this. Because I've renamed the DS method currently there is no UI but it will come after grafana/scenes#1181 gets merged.

github-actions bot added the area/frontend label Jul 11, 2025

samjewell force-pushed the sj/dashboard-ds-implements-adhoc-filtering branch from 2edc201 to 9199493 Compare July 11, 2025 10:36

samjewell self-assigned this Jul 11, 2025

github-actions bot added the area/backend label Jul 14, 2025

samjewell marked this pull request as ready for review July 14, 2025 14:37

samjewell requested review from grafanabot and a team as code owners July 14, 2025 14:37

samjewell requested review from mdvictor, kaydelaney, leeoniya, fastfrwrd and drew08t and removed request for a team July 14, 2025 14:37

github-actions bot added this to the 12.1.x milestone Jul 14, 2025

samjewell added no-backport Skip backport of PR no-changelog Skip including change in changelog/release notes labels Jul 14, 2025

samjewell requested a review from alexjonspencer1 July 14, 2025 14:40

samjewell force-pushed the sj/dashboard-ds-implements-adhoc-filtering branch from b9aa2e6 to 307fbf3 Compare July 14, 2025 15:08

alexjonspencer1 reviewed Jul 14, 2025

View reviewed changes

packages/grafana-ui/src/components/Table/CellActions.tsx Outdated Show resolved Hide resolved

alexjonspencer1 reviewed Jul 14, 2025

View reviewed changes

public/app/plugins/datasource/dashboard/datasource.ts Outdated Show resolved Hide resolved

leeoniya reviewed Jul 14, 2025

View reviewed changes

packages/grafana-ui/src/components/Table/CellActions.tsx Outdated Show resolved Hide resolved

alexjonspencer1 reviewed Jul 14, 2025

View reviewed changes

public/app/plugins/datasource/dashboard/datasource.ts Show resolved Hide resolved

alexjonspencer1 reviewed Jul 14, 2025

View reviewed changes

public/app/plugins/datasource/dashboard/datasource.ts Show resolved Hide resolved

grafana deleted a comment from ephemeral-instances-bot bot Jul 15, 2025

samjewell requested a review from alexjonspencer1 July 18, 2025 13:40

alexjonspencer1 reviewed Jul 21, 2025

View reviewed changes

pkg/services/featuremgmt/toggles_gen.json Outdated Show resolved Hide resolved

samjewell added a commit that referenced this pull request Jul 22, 2025

Refactor to use Grafana's existing value matcher architecture

d6b943d

As per PR feedback @ #108011 (comment)

samjewell added 5 commits July 22, 2025 10:26

Implement AdHoc filtering on --Dashboard-- data source

e61b9b6

Starting with string-fields only

Add support for numeric fields

ce1e61c

Refactor to remove code duplication

abe9dda

But retain maintainability/extensability

Add feature toggle, and move feature behind toggle

e8f1289

Improvements thanks to @alexjonspencer1 code-review

79ac3e8

samjewell force-pushed the sj/dashboard-ds-implements-adhoc-filtering branch from 195db46 to 79ac3e8 Compare July 22, 2025 09:31

mdvictor reviewed Jul 22, 2025

View reviewed changes

public/app/plugins/datasource/dashboard/datasource.ts Outdated Show resolved Hide resolved

samjewell added 2 commits July 22, 2025 12:12

Also gate the filtering logic behind feature toggle

62a3b5d

Thanks to @mdvictor for code-review

Add tests to test the public API

12335fb

Make the second parameter of reconstructDataFrame optional

be7360d

samjewell requested review from alexjonspencer1, mdvictor and gelicia July 22, 2025 22:24

alexjonspencer1 approved these changes Jul 22, 2025

View reviewed changes

mdvictor approved these changes Jul 23, 2025

View reviewed changes

samjewell merged commit aedd7b6 into main Jul 23, 2025
112 of 113 checks passed

samjewell deleted the sj/dashboard-ds-implements-adhoc-filtering branch July 23, 2025 08:12

samjewell mentioned this pull request Jul 23, 2025

Remove "Filter for/out value" to see CI failures #108499

Draft

samjewell mentioned this pull request Jul 23, 2025

[--Dashboard-- data source] Implement getFiltersApplicability() #108517

Open

3 tasks

samjewell mentioned this pull request Jul 23, 2025

[AdHoc] Add filters to tooltips, starting with BarChart #107874

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[--Dashboard-- data source] Implement AdHoc filtering #108011

[--Dashboard-- data source] Implement AdHoc filtering #108011

Uh oh!

samjewell commented Jul 11, 2025 •

edited

Loading

Uh oh!

samjewell commented Jul 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexjonspencer1 commented Jul 14, 2025

Uh oh!

Uh oh!

ephemeral-instances-bot bot commented Jul 22, 2025

Uh oh!

ephemeral-instances-bot bot commented Jul 22, 2025

Uh oh!

Uh oh!

alexjonspencer1 commented Jul 22, 2025

Uh oh!

samjewell commented Jul 22, 2025 •

edited

Loading

Uh oh!

alexjonspencer1 left a comment •

edited

Loading

Uh oh!

mdvictor left a comment

Uh oh!

Uh oh!

samjewell commented Jul 23, 2025 •

edited

Loading

Uh oh!

mdvictor commented Jul 23, 2025

Uh oh!

Uh oh!

[--Dashboard-- data source] Implement AdHoc filtering #108011

[--Dashboard-- data source] Implement AdHoc filtering #108011

Uh oh!

Conversation

samjewell commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this feature?

Why do we need this feature?

Who is this feature for?

Which issue(s) does this PR fix?:

Special notes for your reviewer:

Uh oh!

samjewell commented Jul 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexjonspencer1 commented Jul 14, 2025

Uh oh!

Uh oh!

ephemeral-instances-bot bot commented Jul 22, 2025

Uh oh!

ephemeral-instances-bot bot commented Jul 22, 2025

Uh oh!

Uh oh!

alexjonspencer1 commented Jul 22, 2025

Uh oh!

samjewell commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexjonspencer1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdvictor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

samjewell commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdvictor commented Jul 23, 2025

Uh oh!

Uh oh!

samjewell commented Jul 11, 2025 •

edited

Loading

samjewell commented Jul 22, 2025 •

edited

Loading

alexjonspencer1 left a comment •

edited

Loading

samjewell commented Jul 23, 2025 •

edited

Loading