Cherry pick Rows out in EXPLAIN ANALYZE #670

robozmey · 2024-10-15T08:34:50Z

Cherry pick yezzey-gp/ygp@1d41230

Change logs

Add "Rows out" print in cdbexplain_showExecStats

set gp_enable_explain_rows_out=on;

drop table if exists tt; create table tt (a int, b int) distributed randomly;

explain (analyze,verbose) insert into tt select * from generate_series(1,1000)a,generate_series(1,1000)b;

 Insert  (cost=0.00..495560.34 rows=333334 width=8) (actual time=3.829..1148.458 rows=333741 loops=1)
   Output: generate_series_1.generate_series, generate_series.generate_series, "outer".ColRef_0002, generate_series_1.generate_series
   Executor Memory: 1kB  Segments: 3  Max: 1kB (segment 0)
   Rows out: 333333.33 rows avg x 3 workers, 333741 rows max (seg1), 333054 rows min (seg2).
   ->  Redistribute Motion 3:3  (slice1; segments: 3)  (cost=0.00..479935.34 rows=333334 width=12) (actual time=3.707..761.241 rows=333741 loops=1)
         Output: generate_series_1.generate_series, generate_series.generate_series, "outer".ColRef_0002
         Rows out: 333333.33 rows avg x 3 workers, 333741 rows max (seg1), 333054 rows min (seg2).
         ->  Result  (cost=0.00..479922.82 rows=333334 width=12) (actual time=0.226..882.580 rows=1000000 loops=1)
               Output: generate_series_1.generate_series, generate_series.generate_series, 1
               Rows out: 333333.33 rows avg x 3 workers, 1000000 rows max (seg1), 0 rows min (seg0).
               ->  Result  (cost=0.00..479918.82 rows=333334 width=8) (actual time=0.225..631.202 rows=1000000 loops=1)
                     Output: generate_series_1.generate_series, generate_series.generate_series
                     One-Time Filter: (gp_execution_segment() = 1)
                     Rows out: 333333.33 rows avg x 3 workers, 1000000 rows max (seg1), 0 rows min (seg0).
                     ->  Nested Loop  (cost=0.00..479898.05 rows=333334 width=8) (actual time=0.220..386.554 rows=1000000 loops=1)
                           Output: generate_series_1.generate_series, generate_series.generate_series
                           Join Filter: true
                           Rows out: 333333.33 rows avg x 3 workers, 1000000 rows max (seg1), 0 rows min (seg0).
                           ->  Function Scan on pg_catalog.generate_series generate_series_1  (cost=0.00..0.00 rows=334 width=4) (actual time=0.102..0.333 rows=1000 loops=1)
                                 Output: generate_series_1.generate_series
                                 Function Call: generate_series(1, 1000)
                                 work_mem: 40kB  Segments: 1  Max: 40kB (segment 1)
                                 Rows out: 333.33 rows avg x 3 workers, 1000 rows max (seg1), 0 rows min (seg0).
                           ->  Function Scan on pg_catalog.generate_series  (cost=0.00..0.00 rows=334 width=4) (actual time=0.000..0.092 rows=999 loops=1001)
                                 Output: generate_series.generate_series
                                 Function Call: generate_series(1, 1000)
                                 work_mem: 40kB  Segments: 1  Max: 40kB (segment 1)
                                 Rows out: 333333.67 rows avg x 3 workers, 1000001 rows max (seg1), 0 rows min (seg0).
 Planning time: 5.981 ms
   (slice0)    Executor memory: 87K bytes avg x 3 workers, 87K bytes max (seg0).
   (slice1)    Executor memory: 97K bytes avg x 3 workers, 172K bytes max (seg1).  Work_mem: 40K bytes max.
 Memory used:  128000kB
 Optimizer: Pivotal Optimizer (GPORCA)
 Execution time: 1180.349 ms
(34 rows)

explain (analyze,verbose,format text) select * from tt where a > b;

                                                          QUERY PLAN                                                           
-------------------------------------------------------------------------------------------------------------------------------
 Gather Motion 3:1  (slice1; segments: 3)  (cost=0.00..431.00 rows=1 width=8) (actual time=2.534..223.679 rows=499500 loops=1)
   Output: a, b
   ->  Seq Scan on public.tt  (cost=0.00..431.00 rows=1 width=8) (actual time=0.145..127.350 rows=166742 loops=1)
         Output: a, b
         Filter: (tt.a > tt.b)
         Rows out: 166500.00 rows avg x 3 workers, 166742 rows max (seg0), 166340 rows min (seg2).
 Planning time: 2.578 ms
   (slice0)    Executor memory: 183K bytes.
   (slice1)    Executor memory: 43K bytes avg x 3 workers, 43K bytes max (seg0).
 Memory used:  128000kB
 Optimizer: Pivotal Optimizer (GPORCA)
 Execution time: 245.018 ms
(12 rows)

Why are the changes needed?

"Rows out" is useful for auto explain

Does this PR introduce any user-facing change?

If yes, please clarify the previous behavior and the change this PR proposes.

How was this patch tested?

This feature tests in regression gp_explain test

Contributor's Checklist

Here are some reminders and checklists before/when submitting your pull request, please check them:

Make sure your Pull Request has a clear title and commit message. You can take git-commit template as a reference.
Sign the Contributor License Agreement as prompted for your first-time contribution(One-time setup).
Learn the coding contribution guide, including our code conventions, workflow and more.
List your communication in the GitHub Issues or Discussions (if has or needed).
Document changes.
Add tests for the change
Pass make installcheck
Pass make -C src/test installcheck-cbdb-parallel
Feel free to request cloudberrydb/dev team for review and approval when your PR is ready🥳

* Add "Rows out" print in cdbexplain_showExecStats set gp_enable_explain_rows_out=on; explain (analyze,verbose,format text) select * from tt where a > b; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------- Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..431.00 rows=1 width=8) (actual time=2.534..223.679 rows=499500 loops=1) Output: a, b -> Seq Scan on public.tt (cost=0.00..431.00 rows=1 width=8) (actual time=0.145..127.350 rows=166742 loops=1) Output: a, b Filter: (tt.a > tt.b) Rows out: 166500.00 rows avg x 3 workers, 166742 rows max (seg0), 166340 rows min (seg2). ...

CLAassistant · 2024-10-15T08:35:04Z

All committers have signed the CLA.

github-actions

Hiiii, @robozmey welcome!🎊 Thanks for taking the effort to make our project better! 🙌 Keep making such awesome contributions!

src/backend/utils/misc/guc_gp.c

fanfuxiaoran · 2024-10-16T06:30:09Z

src/backend/cdb/cdbvars.c

@@ -277,6 +277,7 @@ int			gp_hashagg_groups_per_bucket = 5;
 int			gp_motion_slice_noop = 0;

 /* Cloudberry Database Experimental Feature GUCs */
+bool		gp_enable_explain_rows_out = false;


It is better to send a param to explain command than using a guc to control if print out the "rows out"

Rows out feature created for auto_explain. If we add "rows out" as parameter of explain, then we'll need change auto_explain source, and auto_explain wont be compatible with vanilla

auto_explain wont be compatible with vanilla

Hmm... cannot understand. Could you give more details? Is auto_explain aslo included in vanilla?

auto_explain is included in vanilla https://github.com/greenplum-db/gpdb-archive/tree/main/contrib/auto_explain

Also GUC gp_enable_explain_rows_out is made by analogy with GUC gp_enable_explain_allstat

src/include/cdb/cdbvars.h

src/backend/commands/explain_gp.c

avamingli

Do we have a Parallel Plan test(ex: cbdb_parallel.sql) to ensure the output is as expected?

avamingli · 2024-11-07T12:26:23Z

src/backend/commands/explain_gp.c

+                                 ntuples_imin);
+        }
+        else {
+            // ExplainOpenGroup("Rows Out", NULL, false, es);


Please remove unused codes

robozmey · 2024-11-21T09:07:21Z

Do we have a Parallel Plan test(ex: cbdb_parallel.sql) to ensure the output is as expected?

I'm not sure what we need Parallel Plan test, but I expanded gp_explain.sql test

avamingli · 2024-11-26T06:01:18Z

Do we have a Parallel Plan test(ex: cbdb_parallel.sql) to ensure the output is as expected?

I'm not sure what we need Parallel Plan test, but I expanded gp_explain.sql test

In GPDB, there is one process executed the Slice on each segment. And explain use the statistic of that gang of QEs to compute the results.
In CBDB, we have Parallel feature. For a parallel plan, there may be multiple QEs execute the same Slice on each segment.
We call them parallel workers.

See CBDB style README https://github.com/apache/cloudberry/blob/11333c0b4d3a4962b0a6610ceb5b6d7a12e45ec4/src/backend/optimizer/README.cbdb.parallel for more details.
And most cases could be found in https://github.com/apache/cloudberry/blob/11333c0b4d3a4962b0a6610ceb5b6d7a12e45ec4/src/test/regress/sql/cbdb_parallel.sql

This is a significant difference between CBDB and GPDB.

In parallel plan cases, what do the statistic results should be? Do current codes compute results right or it's already as expected?
Maybe the current codes don't need to do anything else more, but we should be clear about that.
In any cases, there should be test cases to verify the explain output.

fanfuxiaoran · 2024-11-29T09:52:37Z

In parallel plan cases, what do the statistic results should be? Do current codes compute results right or it's already as expected?
Maybe the current codes don't need to do anything else more, but we should be clear about that.

Yes, agree! Looks we don't a lot things for the parallel workers. Cloudberry uses motion to gather the slice result both from workers on different segments and from the parallel worker on same segment . So the EXPLAIN ANALYZE command can still work properly to collect results from all workers.

But we should confirm the EXPLAIN ANALYZE output format for the plan contains parallel workers. what should it be like?

 Rows out: 333333.33 rows avg x 3 workers, 1000000 rows max (seg1), 0 rows min (seg0).

Do we need to add the parallel worker info into this line? like

 Rows out: 333333.33 rows avg x 3 workers from xx segments?, 1000000 rows max (seg1 worker?), 0 rows min (seg0 worker?).

avamingli · 2024-12-02T08:52:41Z

@fanfuxiaoran Thanks for your analysis and verification.

Do we need to add the parallel worker info into this line? like
 Rows out: 333333.33 rows avg x 3 workers from xx segments?, 1000000 rows max (seg1 worker?), 0 rows min (seg0 wo

Sounds good, with a worker info is more reasonable of parallel plan.

github-actions bot reviewed Oct 15, 2024

View reviewed changes

robozmey changed the title ~~Rows out в EXPLAIN ANALYZE~~ Rows out in EXPLAIN ANALYZE Oct 15, 2024

robozmey changed the title ~~Rows out in EXPLAIN ANALYZE~~ Cherry pick Rows out in EXPLAIN ANALYZE Oct 15, 2024

my-ship-it requested a review from gongxun0928 October 15, 2024 08:48

reshke reviewed Oct 15, 2024

View reviewed changes

src/backend/utils/misc/guc_gp.c Outdated Show resolved Hide resolved

Fix GUC description

e1320a9

fanfuxiaoran reviewed Oct 16, 2024

View reviewed changes

robozmey added 4 commits October 24, 2024 14:57

Merge branch 'main' into rows-out

87928a5

Use CdbExplain_NodeSummary to get max and avg

ecca72c

Add min to CdbExplain_Agg

cbff7f0

Add min to CdbExplain_DepStatAcc

c0448fe

fanfuxiaoran reviewed Oct 25, 2024

View reviewed changes

src/backend/commands/explain_gp.c Outdated Show resolved Hide resolved

robozmey added 3 commits October 25, 2024 14:05

Remove rshmin, rsimin, nsimin from CdbExplain_DepStatAcc

490521e

Change GUC gp_enable_explain_rows_out description

0b7ab04

Merge branch 'main' into rows-out

e617d39

fanfuxiaoran previously approved these changes Nov 4, 2024

View reviewed changes

my-ship-it previously approved these changes Nov 5, 2024

View reviewed changes

avamingli reviewed Nov 7, 2024

View reviewed changes

Remove unused code

d6aa761

robozmey dismissed stale reviews from my-ship-it and fanfuxiaoran via d6aa761 November 20, 2024 02:11

robozmey added 2 commits November 21, 2024 11:11

Fix test

07a73a3

Expand test for rows out

d0193b8

my-ship-it added the cherry-pick cherry-pick upstream commts label Dec 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry pick Rows out in EXPLAIN ANALYZE #670

Cherry pick Rows out in EXPLAIN ANALYZE #670

robozmey commented Oct 15, 2024 •

edited

Loading

CLAassistant commented Oct 15, 2024 •

edited

Loading

github-actions bot left a comment

fanfuxiaoran Oct 16, 2024

robozmey Oct 24, 2024

fanfuxiaoran Oct 29, 2024 •

edited

Loading

robozmey Oct 31, 2024

avamingli left a comment

avamingli Nov 7, 2024

robozmey Nov 11, 2024

robozmey commented Nov 21, 2024

avamingli commented Nov 26, 2024 •

edited

Loading

fanfuxiaoran commented Nov 29, 2024

avamingli commented Dec 2, 2024

Cherry pick Rows out in EXPLAIN ANALYZE #670

Are you sure you want to change the base?

Cherry pick Rows out in EXPLAIN ANALYZE #670

Conversation

robozmey commented Oct 15, 2024 • edited Loading

Change logs

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Contributor's Checklist

CLAassistant commented Oct 15, 2024 • edited Loading

github-actions bot left a comment

Choose a reason for hiding this comment

fanfuxiaoran Oct 16, 2024

Choose a reason for hiding this comment

robozmey Oct 24, 2024

Choose a reason for hiding this comment

fanfuxiaoran Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

robozmey Oct 31, 2024

Choose a reason for hiding this comment

avamingli left a comment

Choose a reason for hiding this comment

avamingli Nov 7, 2024

Choose a reason for hiding this comment

robozmey Nov 11, 2024

Choose a reason for hiding this comment

robozmey commented Nov 21, 2024

avamingli commented Nov 26, 2024 • edited Loading

fanfuxiaoran commented Nov 29, 2024

avamingli commented Dec 2, 2024

robozmey commented Oct 15, 2024 •

edited

Loading

CLAassistant commented Oct 15, 2024 •

edited

Loading

fanfuxiaoran Oct 29, 2024 •

edited

Loading

avamingli commented Nov 26, 2024 •

edited

Loading