Skip to content

Transform lego explorations #61459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 196 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
196 commits
Select commit Hold shift + click to select a range
b5e258f
Empty
ranquild Jul 21, 2025
1434d05
Empty
ranquild Jul 21, 2025
2f13cb0
Basic UI
ranquild Jul 21, 2025
87ed3b5
Add metabase-enterprise.transform module
metamben Jul 21, 2025
117e923
Refer to module with plural
metamben Jul 21, 2025
64f799a
add dummy execute function
wzimrin Jul 21, 2025
09ec301
Add a turn into transform modal
ranquild Jul 21, 2025
9c30b51
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 21, 2025
5f60c16
Transforms page
ranquild Jul 21, 2025
316c16f
Return an entry from the list endpoint
metamben Jul 21, 2025
fa0c36e
Transforms page
ranquild Jul 21, 2025
29ebc2e
basic transform execution
wzimrin Jul 21, 2025
d87a7ba
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 21, 2025
7620abe
Fix POST input schema
metamben Jul 21, 2025
7a9c5ba
some untested dummy routes
wzimrin Jul 21, 2025
717b6bd
use compile-with-inline-parameters
wzimrin Jul 21, 2025
d37fdca
actually fetch the query from the result of compile
wzimrin Jul 21, 2025
ec44b10
make execute actually execute
wzimrin Jul 21, 2025
eb4e329
Transform migration + toucan model
ericnormand Jul 21, 2025
f1fdf99
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 21, 2025
f276dab
h2 doesn't support transforms
wzimrin Jul 21, 2025
9320a8f
Use toucan to implement API endpoints
ericnormand Jul 21, 2025
eba9714
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 21, 2025
fccfa94
Placate clj-kondo
metamben Jul 21, 2025
27edaf2
Delete table.
ericnormand Jul 21, 2025
9b8f61d
Check native query execution right when executing a transform
metamben Jul 21, 2025
a8d1dcc
[ci skip] Enable selective updates
metamben Jul 21, 2025
87c5dd8
[ci skip] Prepend schema to table if provided
metamben Jul 21, 2025
1964438
Merge branch 'master' into poc-transform-table
metamben Jul 21, 2025
0648c78
Transforms page
ranquild Jul 22, 2025
d495f23
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 22, 2025
aaee089
Require metabase-enterprise.transforms.models.transform in init
metamben Jul 22, 2025
a550e69
[skip ci] Drop target database-id, check if the target table exists
metamben Jul 22, 2025
b7247b2
[ci skip] Better api parameter handling
ericnormand Jul 22, 2025
38c5fa5
[ci skip] more parsing
ericnormand Jul 22, 2025
febfce8
Fix
ranquild Jul 22, 2025
e64ecbb
Fix table exists check
metamben Jul 22, 2025
0e1450f
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 22, 2025
bb85233
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 22, 2025
706d7f5
Fix
ranquild Jul 22, 2025
d989b5b
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 22, 2025
9bf91fe
[ci skip] Merge branch 'poc-transform-table' of github.com:metabase/m…
ericnormand Jul 22, 2025
9967f4f
[ci skip] Transforms api tests
ericnormand Jul 22, 2025
d87da10
[skip ci] Fix table existence check
metamben Jul 22, 2025
c54de85
[skip ci] Require superuser
metamben Jul 22, 2025
bb438ce
[skip ci] Remove superfluous require
metamben Jul 22, 2025
d49b886
[ci skip] Use crowberto in tests and fix api
ericnormand Jul 22, 2025
821df47
[skip ci] Sync transform output table on execution
metamben Jul 22, 2025
9e4bca9
Basic page
ranquild Jul 22, 2025
1d9c6c7
[skip ci] Sync transform on create an update too
metamben Jul 22, 2025
1e498de
[ci skip] lego definition (interpreter) for transform
ericnormand Jul 22, 2025
d2f02f7
[ci skip] Add plan
ericnormand Jul 22, 2025
7443a2d
[ci skip] Add provisional transfer code
ericnormand Jul 22, 2025
2585f35
[ci skip] test edn, json, yaml versions
ericnormand Jul 22, 2025
f7d947f
basic transfers work
wzimrin Jul 22, 2025
4615194
Basic page
ranquild Jul 23, 2025
b340f8e
Basic page
ranquild Jul 23, 2025
f941af8
Basic page
ranquild Jul 23, 2025
f41c0b1
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 23, 2025
397e369
Basic page
ranquild Jul 23, 2025
a254fc1
Basic page
ranquild Jul 23, 2025
a753f2e
use original case
wzimrin Jul 23, 2025
259ca02
Basic page
ranquild Jul 23, 2025
cbaa883
Basic page
ranquild Jul 23, 2025
5322f76
Basic page
ranquild Jul 23, 2025
c85ec06
Basic page
ranquild Jul 23, 2025
3815358
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 23, 2025
6bdeb7f
Basic page
ranquild Jul 23, 2025
e02d4d0
Basic page
ranquild Jul 23, 2025
658c2f1
Basic page
ranquild Jul 23, 2025
05005f9
Basic page
ranquild Jul 23, 2025
042ae3a
Add execute
ranquild Jul 23, 2025
99cbfac
Add execute
ranquild Jul 23, 2025
4e24289
update call to execute
wzimrin Jul 23, 2025
663d0d2
Modal
ranquild Jul 23, 2025
13f74c8
[ci skip] REVERT LATER Only run transforms api tests for drivers
ericnormand Jul 23, 2025
41390fd
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 23, 2025
0ade82a
add necessary require to transfers.execute
wzimrin Jul 23, 2025
b8a263a
Modal
ranquild Jul 23, 2025
5f576ea
Modal
ranquild Jul 23, 2025
380fe7c
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 23, 2025
ac3bfcd
Modal
ranquild Jul 23, 2025
d22050e
[skip ci] Remove bad require
metamben Jul 23, 2025
29d4658
Modal
ranquild Jul 23, 2025
29a6635
Modal
ranquild Jul 23, 2025
ea07fdf
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 23, 2025
be2337d
[ci skip] Add hippie parse
ericnormand Jul 23, 2025
5b2fc3d
Move actions under src
ericnormand Jul 23, 2025
b9291ce
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 23, 2025
bbe4767
Modal
ranquild Jul 23, 2025
fd48958
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 23, 2025
9beb14a
[skip ci] transfers can be called via legos/execute!
wzimrin Jul 23, 2025
ff1f127
update execute! schema for transfer
wzimrin Jul 23, 2025
e015457
[ci skip] Add tests for hippie parse
ericnormand Jul 23, 2025
f711c11
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 23, 2025
c8b01f0
[skip ci] Add description to transforms
metamben Jul 23, 2025
2383c6d
Modal
ranquild Jul 23, 2025
8f57302
Fix actions test
ericnormand Jul 23, 2025
7dd7c25
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 23, 2025
5803b4f
[skip ci]
ranquild Jul 23, 2025
bd2cc27
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 23, 2025
bbe8bc7
[ci skip] Fix actions test
ericnormand Jul 23, 2025
ff73867
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 23, 2025
6c74aa8
[skip ci]
ranquild Jul 23, 2025
0504b74
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 23, 2025
d1f0e0e
[skip ci]
ranquild Jul 23, 2025
f52fce7
[skip ci]
ranquild Jul 23, 2025
a2aed5c
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 23, 2025
416f44d
[skip ci] Add database_id to transform and filter the list by it
metamben Jul 23, 2025
a4e8364
[skip ci]
ranquild Jul 23, 2025
75994b3
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 23, 2025
6f29ee6
[skip ci]
ranquild Jul 23, 2025
b806d8e
[skip ci] Merge list-transform-test vars
metamben Jul 23, 2025
c3b6f4e
[skip ci] Hydrate target table at the top level in the get endpoint
metamben Jul 23, 2025
0416b84
[skip ci] Delete transform created in tests
metamben Jul 23, 2025
aa05a56
[skip ci]
ranquild Jul 23, 2025
e42d842
[skip ci]
ranquild Jul 23, 2025
d89c57c
[ci skip] return non-nil for success
ericnormand Jul 23, 2025
caaa2e8
[skip ci]
ranquild Jul 23, 2025
3bbe285
[ci skip] tests for transfers lego
ericnormand Jul 23, 2025
90c0d5a
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 23, 2025
4dfa96b
[skip ci]
ranquild Jul 23, 2025
a92f7f4
clean up naming and module refs
wzimrin Jul 23, 2025
cff26c6
fix actions ref [skip ci]
wzimrin Jul 23, 2025
a2b0c4e
[skip ci]
ranquild Jul 23, 2025
6f78bc4
[skip ci]
ranquild Jul 23, 2025
721394d
[skip ci]
ranquild Jul 23, 2025
8e83fbb
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 23, 2025
b59e4aa
[skip ci] Fix target table existence check and add cleanup to tests
metamben Jul 23, 2025
10d49ab
[ci skip] Add rich comment
ericnormand Jul 23, 2025
7337c26
[ci skip] save transfer.yaml example
ericnormand Jul 23, 2025
611d221
[ci skip] print progress messages
ericnormand Jul 23, 2025
1079267
[ci skip] 6-step analysis plan!
ericnormand Jul 23, 2025
c3edf1d
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 23, 2025
03b9b74
[skip ci]
ranquild Jul 24, 2025
e1494a1
[skip ci]
ranquild Jul 24, 2025
dc093d4
[skip ci]
ranquild Jul 24, 2025
2ed5047
[skip ci]
ranquild Jul 24, 2025
38d0b3e
[skip ci]
ranquild Jul 24, 2025
0a4c544
[skip ci]
ranquild Jul 24, 2025
f7380aa
[skip ci]
ranquild Jul 24, 2025
d53506d
[skip ci]
ranquild Jul 24, 2025
1cb1106
[skip ci]
ranquild Jul 24, 2025
fb159d2
[skip ci]
ranquild Jul 24, 2025
e284745
[skip ci]
ranquild Jul 24, 2025
6ae2316
[skip ci]
ranquild Jul 24, 2025
c29b56e
[skip ci]
ranquild Jul 24, 2025
caa0810
[skip ci]
ranquild Jul 24, 2025
f956bca
[skip ci]
ranquild Jul 24, 2025
ef1fe0d
[skip ci]
ranquild Jul 24, 2025
071489c
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 24, 2025
b8ba1b4
kick ci
ericnormand Jul 24, 2025
aa01429
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 24, 2025
9814449
kick ci
ericnormand Jul 24, 2025
a39502c
[ci skip] Multi-file plans implemented
ericnormand Jul 24, 2025
4306da5
[skip ci]
ranquild Jul 24, 2025
f060565
kick ci
ericnormand Jul 24, 2025
ed65b84
[ci skip] two more yaml plans generated by claude
ericnormand Jul 24, 2025
504b183
[skip ci]
ranquild Jul 24, 2025
cd68161
[skip ci]
ranquild Jul 24, 2025
8b46adb
[skip ci]
ranquild Jul 24, 2025
2b07b83
[skip ci]
ranquild Jul 24, 2025
7949fe6
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 24, 2025
6b28bce
Add delete confirmation
ranquild Jul 24, 2025
0ee16ea
Add delete confirmation
ranquild Jul 24, 2025
776c225
Add delete confirmation
ranquild Jul 24, 2025
f6e00a4
Add delete confirmation
ranquild Jul 24, 2025
c1ea464
Add delete confirmation
ranquild Jul 24, 2025
5a1ebe9
Add delete confirmation
ranquild Jul 24, 2025
f686175
Add delete confirmation
ranquild Jul 24, 2025
eb087c4
[skip ci]
ranquild Jul 24, 2025
b4bba9e
[skip ci]
ranquild Jul 24, 2025
cbfd98f
[skip ci]
ranquild Jul 24, 2025
6f2dff8
Fix tests
ericnormand Jul 24, 2025
8bfed17
Merge branch 'poc-transform-table' of github.com:metabase/metabase in…
ericnormand Jul 24, 2025
9368abe
Better feature name
ericnormand Jul 24, 2025
068fed4
Run tests again with new impl
ericnormand Jul 24, 2025
fa59b19
Better tests
ericnormand Jul 24, 2025
22d1f30
Fix tests and driver feature
ericnormand Jul 24, 2025
2318965
Hack for clickhouse (but it works!)
ericnormand Jul 24, 2025
02e1014
Disable transforms/basic feature for redshift
ericnormand Jul 24, 2025
0eb562c
VERY minimal python execution
wzimrin Jul 24, 2025
6143a75
[skip test] Add support for transform scheduling
metamben Jul 24, 2025
a7ab600
[skip ci]
ranquild Jul 24, 2025
d409ac1
[skip ci]
ranquild Jul 25, 2025
7544897
[skip ci]
ranquild Jul 25, 2025
29a500c
[skip ci]
ranquild Jul 25, 2025
4d86edd
Merge remote-tracking branch 'origin/poc-transform-table' into poc-tr…
ranquild Jul 25, 2025
0aa0fc2
[skip ci]
ranquild Jul 25, 2025
16fa5c0
[skip ci]
ranquild Jul 25, 2025
d690b9b
[skip ci]
ranquild Jul 25, 2025
c83231a
[skip ci]
ranquild Jul 25, 2025
614f574
[skip ci]
ranquild Jul 25, 2025
c1b1137
run CI
perivamsi Jul 25, 2025
e437887
Merge branch 'master' into poc-transform-table
perivamsi Jul 25, 2025
0aaa386
Fix create and delete issues
metamben Jul 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[skip ci]
  • Loading branch information
ranquild committed Jul 23, 2025
commit a4e8364b2202bceaae2fac4d83d9f67e0d25267c
6 changes: 3 additions & 3 deletions enterprise/frontend/src/metabase-enterprise/api/transform.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import type {
CreateTransformRequest,
ListTransformsRequest,
Transform,
TransformId,
UpdateTransformRequest,
Expand All @@ -17,11 +18,10 @@ import {

export const transformApi = EnterpriseApi.injectEndpoints({
endpoints: (builder) => ({
listTransforms: builder.query<Transform[], void>({
query: (params) => ({
listTransforms: builder.query<Transform[], ListTransformsRequest>({
query: () => ({
method: "GET",
url: "/api/ee/transform",
params,
}),
providesTags: (transforms = []) => provideTransformListTags(transforms),
}),
Expand Down
6 changes: 6 additions & 0 deletions frontend/src/metabase-types/api/transform.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import type { DatabaseId } from "metabase-types/api/database";

import type { DatasetQuery } from "./query";

export type TransformId = number;
Expand All @@ -22,6 +24,10 @@ export type TransformTarget = {
table: string;
};

export type ListTransformsRequest = {
database_id?: DatabaseId;
};

export type CreateTransformRequest = {
name: string;
source: TransformSource;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -130,13 +130,20 @@ export function useTableLoader(path: TreePath) {
return [];
}

if (transformsRef.current.isError) {
const newArgs = {
database_id: databaseId,
};

if (
transformsRef.current.isError &&
_.isEqual(transformsRef.current.originalArgs, newArgs)
) {
// Do not refetch when this call failed previously.
// This is to prevent infinite data-loading loop as RTK query does not cache error responses.
return [];
}

const response = await fetchTransforms();
const response = await fetchTransforms(newArgs, true);
return response?.data?.map((transform) =>
node<TransformNode>({
type: "transform",
Expand Down
7 changes: 6 additions & 1 deletion frontend/src/metabase/plugins/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ import type {
Group,
GroupPermissions,
GroupsPermissions,
ListTransformsRequest,
ModelCacheRefreshStatus,
ParameterId,
Pulse,
Expand Down Expand Up @@ -794,12 +795,16 @@ export type UseLazyListTransformsData = {
};

export type UseLazyListTransformsState = {
originalArgs?: ListTransformsRequest;
isFetching: boolean;
isError: boolean;
};

export type UseLazyListTransformsResult = [
() => Promise<UseLazyListTransformsData>,
(
args: ListTransformsRequest,
preferCacheValue?: boolean,
) => Promise<UseLazyListTransformsData>,
UseLazyListTransformsState,
];

Expand Down