Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](paimon)Upgrade the Paimon version to 1.0.0 #46990

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[opt](paimon)Upgrade the Paimon version to 1.0.0 #46990

wants to merge 2 commits into from
+22 −7

Conversation

wuwenchi
Copy link
Contributor

What problem does this PR solve?

Problem Summary:

Upgrade the Paimon version to 1.0

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • [] No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32924 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed5afca7df8f5ec13ad0c7c8dc64606a0ba0c27f, data reload: false

------ Round 1 ----------------------------------
q1	17580	6234	6130	6130
q2	2057	317	173	173
q3	10407	1285	738	738
q4	10212	897	446	446
q5	7519	2211	1969	1969
q6	207	186	155	155
q7	877	757	632	632
q8	9231	1360	1198	1198
q9	5232	4915	4838	4838
q10	6782	2298	1849	1849
q11	470	292	269	269
q12	355	356	222	222
q13	17755	3703	3096	3096
q14	229	237	205	205
q15	588	515	522	515
q16	619	615	583	583
q17	572	862	325	325
q18	6989	6550	6493	6493
q19	1200	947	544	544
q20	316	328	197	197
q21	2876	2236	2027	2027
q22	374	329	320	320
Total cold run time: 102447 ms
Total hot run time: 32924 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6261	6292	6277	6277
q2	245	337	236	236
q3	2282	2654	2372	2372
q4	1397	1829	1394	1394
q5	4385	4793	4839	4793
q6	197	181	149	149
q7	2062	2043	1856	1856
q8	2657	2786	2675	2675
q9	7409	7223	7354	7223
q10	3022	3324	2790	2790
q11	612	516	504	504
q12	692	782	622	622
q13	3464	3984	3303	3303
q14	277	293	271	271
q15	572	528	543	528
q16	648	686	662	662
q17	1220	1746	1250	1250
q18	7760	7627	7349	7349
q19	815	1139	1074	1074
q20	1986	1991	1890	1890
q21	5871	5164	4860	4860
q22	593	580	577	577
Total cold run time: 54427 ms
Total hot run time: 52655 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188631 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed5afca7df8f5ec13ad0c7c8dc64606a0ba0c27f, data reload: false

query1	976	392	374	374
query2	6525	2076	1995	1995
query3	6701	218	218	218
query4	33915	23493	23575	23493
query5	4373	648	452	452
query6	301	203	186	186
query7	4619	490	316	316
query8	284	229	224	224
query9	9543	2730	2750	2730
query10	495	313	262	262
query11	17900	15237	14980	14980
query12	164	107	110	107
query13	1663	528	404	404
query14	10436	6851	6892	6851
query15	219	203	186	186
query16	7886	636	463	463
query17	1635	738	567	567
query18	2056	413	312	312
query19	232	192	159	159
query20	120	115	112	112
query21	220	126	105	105
query22	4333	4361	4239	4239
query23	34470	33414	33284	33284
query24	6544	2255	2266	2255
query25	483	450	379	379
query26	1198	270	156	156
query27	1986	479	343	343
query28	5082	2471	2488	2471
query29	648	542	424	424
query30	228	198	155	155
query31	962	874	788	788
query32	71	64	59	59
query33	519	352	306	306
query34	734	869	517	517
query35	781	803	705	705
query36	1000	1054	947	947
query37	120	127	77	77
query38	4236	4168	4143	4143
query39	1428	1385	1403	1385
query40	199	114	103	103
query41	61	49	52	49
query42	127	107	104	104
query43	529	526	489	489
query44	1410	836	819	819
query45	177	173	166	166
query46	871	1020	661	661
query47	1817	1835	1758	1758
query48	384	404	329	329
query49	783	499	400	400
query50	622	639	398	398
query51	6887	6930	6713	6713
query52	106	102	93	93
query53	230	256	199	199
query54	487	505	399	399
query55	85	81	82	81
query56	270	272	282	272
query57	1151	1173	1079	1079
query58	256	241	244	241
query59	3117	3078	2866	2866
query60	283	275	259	259
query61	120	122	121	121
query62	898	798	723	723
query63	223	209	197	197
query64	4037	1032	713	713
query65	3268	3130	3190	3130
query66	1082	418	321	321
query67	15872	15500	15559	15500
query68	4918	839	550	550
query69	465	302	259	259
query70	1227	1123	1113	1113
query71	399	289	276	276
query72	5844	3857	3822	3822
query73	660	754	366	366
query74	10113	9089	8773	8773
query75	3222	3162	2663	2663
query76	3287	1167	783	783
query77	473	399	294	294
query78	9987	10070	9282	9282
query79	2557	830	625	625
query80	1710	570	445	445
query81	579	280	234	234
query82	396	204	124	124
query83	266	189	158	158
query84	249	107	76	76
query85	816	361	299	299
query86	476	312	310	310
query87	4423	4442	4337	4337
query88	3810	2215	2175	2175
query89	400	330	292	292
query90	1845	191	192	191
query91	145	139	112	112
query92	78	58	55	55
query93	2013	930	534	534
query94	729	413	298	298
query95	353	273	260	260
query96	496	616	287	287
query97	2763	2875	2752	2752
query98	231	207	200	200
query99	1422	1499	1374	1374
Total cold run time: 286790 ms
Total hot run time: 188631 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed5afca7df8f5ec13ad0c7c8dc64606a0ba0c27f, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.41	0.40	0.42
query6	1.18	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.51	0.51
query10	0.55	0.58	0.54
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	2.78	2.74	2.73
query15	0.91	0.83	0.83
query16	0.37	0.38	0.38
query17	1.05	1.05	1.07
query18	0.23	0.21	0.21
query19	1.92	1.78	1.93
query20	0.01	0.01	0.01
query21	15.39	0.95	0.59
query22	0.76	0.83	0.66
query23	15.24	1.44	0.57
query24	3.34	0.64	1.85
query25	0.26	0.12	0.20
query26	0.23	0.14	0.14
query27	0.07	0.05	0.05
query28	13.64	1.53	1.05
query29	12.56	3.93	3.23
query30	0.25	0.10	0.06
query31	2.83	0.61	0.38
query32	3.23	0.54	0.46
query33	3.03	3.02	3.03
query34	16.80	5.23	4.57
query35	4.58	4.53	4.60
query36	0.65	0.50	0.48
query37	0.11	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.27 s
Total hot run time: 31.04 s

morningman
morningman reviewed Jan 15, 2025
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java
return hadoopAuthenticator.doAs(() -> catalog.tableExists(Identifier.create(dbName, tblName)));
return hadoopAuthenticator.doAs(() -> {
try {
catalog.getTable(Identifier.create(dbName, tblName));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The API is changed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman left review comments

@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wuwenchi @hello-stephen @doris-robot @morningman