forked from Velocidex/velociraptor
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Distributed notebook processor (Velocidex#3043)
This introduces an experimental support for distributing notebook processing. Notebook queries can be very CPU intensive and use a lot of resources. They currently run within the GUI master node without bounds. This means that if many users are running expensive queries at the same time the server stability may be affected. This PR changes notebook queries to run in dedicated workers. The master node will start a certain number of workers in all cases but the minions will also start workers. When a user submits a query to the notebook cell the GUI will send the query to one of the available workers. By default remote workers are preferred if available but if there are no minions, the local worker will be tasked. This also has the effect of limiting the total number of worker threads on local deployments as well. For example if two local threads are started then issuing the third concurrent query will fail. This increases server stability by imposing a reasonable limit. A Future PR will leverage this mechansims for sever artifact runers as well.
- Loading branch information
Showing
53 changed files
with
3,800 additions
and
1,508 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.