-
Notifications
You must be signed in to change notification settings - Fork 418
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize the code path in createFileset
and optimize path.
#6562
Comments
I would like to work on it. |
OK, just go ahead. |
@Abyss-lord |
@yuqi1129 Yes, I can finish it in two days |
Abyss-lord
added a commit
to Abyss-lord/gravitino
that referenced
this issue
Mar 14, 2025
…Fileset and optimize path Optimize the code path in createFileset and optimize path by using future.
Abyss-lord
added a commit
to Abyss-lord/gravitino
that referenced
this issue
Mar 14, 2025
…Fileset and optimize path make thread pool static remove shutdown logic in finally block.
Abyss-lord
added a commit
to Abyss-lord/gravitino
that referenced
this issue
Mar 14, 2025
…Fileset and optimize path Set the number of core threads and queue size, and set the idle time.
Abyss-lord
added a commit
to Abyss-lord/gravitino
that referenced
this issue
Mar 14, 2025
…Fileset and optimize path Optimize the code path in createFileset and optimize path by using future.
Abyss-lord
added a commit
to Abyss-lord/gravitino
that referenced
this issue
Mar 14, 2025
…Fileset and optimize path make thread pool static remove shutdown logic in finally block.
Abyss-lord
added a commit
to Abyss-lord/gravitino
that referenced
this issue
Mar 14, 2025
…Fileset and optimize path Set the number of core threads and queue size, and set the idle time.
Abyss-lord
added a commit
to Abyss-lord/gravitino
that referenced
this issue
Mar 17, 2025
…Fileset and optimize path set queue size from 100 to 500
Abyss-lord
added a commit
to Abyss-lord/gravitino
that referenced
this issue
Mar 17, 2025
…Fileset and optimize path fix some bugs in the code. 1. set min thread to Math.max(Math.min(Runtime.getRuntime().availableProcessors() * 2, 100), 4) 2. set max thread to Math.max(Runtime.getRuntime().availableProcessors() * 4, 400).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code in HadoopCatalogOperations#createFileset
These two lines will repeatedly get and initialize file system and can be merged into one
This code can be replaced to Java
Future
mechanism to reduce the time taken in poll status.There may be other minor points to improve.
The text was updated successfully, but these errors were encountered: