MQE: Count samples saved on the CSE #12169
Open
+442
−123
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Konstantinov-Innokentii
changed the title
Konstantinov-Innokentii
added
the
changelog-not-needed
Konstantinov-Innokentii
force-pushed
the
count_samples_saved_by_cse
branch
from
53ce5ba to
2335849
Compare
When detecring a duplicate expression, set the sampleCountMultiplicator on the Vector/Matrix selector in the duplicated path. Signed-off-by: Innokentii Konstantinov <[email protected]>
Signed-off-by: Innokentii Konstantinov <[email protected]>
Konstantinov-Innokentii
force-pushed
the
count_samples_saved_by_cse
branch
from
2335849 to
ed3cbe6
Compare
Signed-off-by: Innokentii Konstantinov <[email protected]>
Signed-off-by: Innokentii Konstantinov <[email protected]>
Signed-off-by: Innokentii Konstantinov <[email protected]>
56quarters
approved these changes
Jul 24, 2025
| @@ -42,6 +42,20 @@ type Selector struct { | |||
| series *seriesList | |||
|
|
|||
| seriesIdx int | |||
|
|
|||
| // Sample count factor for stats counting when applying CSE. | |||
There was a problem hiding this comment.
Suggested change
| // Sample count factor for stats counting when applying CSE. | |
| // Sample count factor for stats counting at each step when applying common sub-expression elimination. |
| if testCase.skipCompareWithPrometheus == "" { | ||
| require.Equal(t, testCase.expectedTotalSamples, prometheusSamplesStats.TotalSamples, "invalid test case: expected total samples does not match value from Prometheus' engine") | ||
| require.Equal(t, testCase.expectedTotalSamplesPerStep, prometheusSamplesStats.TotalSamplesPerStep, "invalid test case: expected per stepsamples does not match value from Prometheus' engine") | ||
| } |
There was a problem hiding this comment.
Should we log the reason for skipping here?
} else {
t.Log(testCase.skipCompareWithPrometheus)
}
lamida
reviewed
Jul 27, 2025
| @@ -375,3 +377,23 @@ func (p *path) String() string { | |||
| b.WriteRune(']') | |||
| return b.String() | |||
| } | |||
|
|
|||
| // setSampleCountFactor sets SampleCountFactor on the VectorSelector and MatrixSelector nodes. | |||
| // It sets it only if multipilcator was not set before to avoid overwriting it for nested duplicate expressions. | |||
There was a problem hiding this comment.
Nits. I think multiplier is a more common word for this.
Suggested change
| // It sets it only if multipilcator was not set before to avoid overwriting it for nested duplicate expressions. | |
| // It sets it only if multiplier was not set before to avoid overwriting it for nested duplicate expressions. |
Comment on lines
+82
to
+84
| if m.SampleCountFactor != 0 { | ||
| selector.SetSampleCountFactor(m.SampleCountFactor) | ||
| } |
There was a problem hiding this comment.
What if we just push the non zero check != 0 inside SetSampleCountFactor method itself? With that we can remove this if branch and also the one in vector_selector.go.
Comment on lines
+50
to
+52
| func (s *Selector) SetSampleCountFactor(factor uint32) { | ||
| s.sampleCountFactor = factor | ||
| } |
There was a problem hiding this comment.
As I mentioned in another comment, since we have setter method here, just add logic to set only for non-zero factor. Hence we can remove non-zero check in the caller.
Suggested change
| func (s *Selector) SetSampleCountFactor(factor uint32) { | |
| s.sampleCountFactor = factor | |
| } | |
| func (s *Selector) SetSampleCountFactor(factor uint32) { | |
| if factor != 0 { | |
| s.sampleCountFactor = factor | |
| } | |
| } |
| return q.Stats().Samples | ||
| } | ||
|
|
||
| testCases := map[string]struct { |
There was a problem hiding this comment.
All the testCases below are for instant query. Should we have testCase for range query too? So that we can see the case where totalSamples is not equal with totalSamplesPerStep.
| @@ -103,7 +103,7 @@ func (m *RangeVectorSelector) NextStepSamples() (*types.RangeVectorStepData, err | |||
| m.stepData.RangeStart = rangeStart | |||
| m.stepData.RangeEnd = rangeEnd | |||
|
|
|||
| m.Stats.IncrementSamplesAtTimestamp(m.stepData.StepT, int64(m.stepData.Floats.Count())+m.stepData.Histograms.EquivalentFloatSampleCount()) | |||
| m.Stats.IncrementSamplesAtTimestamp(m.stepData.StepT, (int64(m.stepData.Floats.Count())+m.stepData.Histograms.EquivalentFloatSampleCount())*int64(m.Selector.GetSampleCountFactor())) | |||
There was a problem hiding this comment.
With some castings, it takes sometime to see what is being computed. What if we extract some of the operation to local variables to make it clearer?
Suggested change
| m.Stats.IncrementSamplesAtTimestamp(m.stepData.StepT, (int64(m.stepData.Floats.Count())+m.stepData.Histograms.EquivalentFloatSampleCount())*int64(m.Selector.GetSampleCountFactor())) | |
| floatCount := int64(m.stepData.Floats.Count()) | |
| histogramCount := m.stepData.Histograms.EquivalentFloatSampleCount() | |
| sampleFactor := int64(m.Selector.GetSampleCountFactor()) | |
| totalSamples := (floatCount + histogramCount) * sampleFactor | |
| m.Stats.IncrementSamplesAtTimestamp(m.stepData.StepT, totalSamples)``` |
Comment on lines
+384
to
+398
| switch selector := node.(type) { | ||
| case *core.VectorSelector: | ||
| if selector.SampleCountFactor == 0 { | ||
| selector.SampleCountFactor = factor | ||
| } | ||
| return | ||
| case *core.MatrixSelector: | ||
| if selector.SampleCountFactor == 0 { | ||
| selector.SampleCountFactor = factor | ||
| } | ||
| return | ||
| } | ||
| for _, child := range node.Children() { | ||
| setSampleCountFactor(child, factor) | ||
| } |
There was a problem hiding this comment.
Instead of having two empty returns, what if we just add default clause to catch all?
Suggested change
| switch selector := node.(type) { | |
| case *core.VectorSelector: | |
| if selector.SampleCountFactor == 0 { | |
| selector.SampleCountFactor = factor | |
| } | |
| return | |
| case *core.MatrixSelector: | |
| if selector.SampleCountFactor == 0 { | |
| selector.SampleCountFactor = factor | |
| } | |
| return | |
| } | |
| for _, child := range node.Children() { | |
| setSampleCountFactor(child, factor) | |
| } | |
| switch selector := node.(type) { | |
| case *core.VectorSelector: | |
| if selector.SampleCountFactor == 0 { | |
| selector.SampleCountFactor = factor | |
| } | |
| case *core.MatrixSelector: | |
| if selector.SampleCountFactor == 0 { | |
| selector.SampleCountFactor = factor | |
| } | |
| default: | |
| for _, child := range node.Children() { | |
| setSampleCountFactor(child, factor) | |
| } | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
When detecting a duplicate expression, set the sampleCountMultiplicator on the Vector/Matrix selector in the duplicated path. It's needed to preserve "original" number of samples processed, as there were no CSE applied. The multiplication is set on a planning stage - code
Note that during this work, I found an edge case, when MQE returns 0 samples processed when CSE is applied - plan is to tackle it separately. Here is a skipped test covering it.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]. If changelog entry is not needed, please add thechangelog-not-neededlabel to the PR.about-versioning.mdupdated with experimental features.