Skip to content

Commit

Permalink
Distributed notebook processor (Velocidex#3043)
Browse files Browse the repository at this point in the history
This introduces an experimental support for distributing notebook
processing. Notebook queries can be very CPU intensive and use a lot of
resources. They currently run within the GUI master node without bounds.
This means that if many users are running expensive queries at the same
time the server stability may be affected.

This PR changes notebook queries to run in dedicated workers. The master
node will start a certain number of workers in all cases but the minions
will also start workers. When a user submits a query to the notebook
cell the GUI will send the query to one of the available workers. By
default remote workers are preferred if available but if there are no
minions, the local worker will be tasked.

This also has the effect of limiting the total number of worker threads
on local deployments as well. For example if two local threads are
started then issuing the third concurrent query will fail. This
increases server stability by imposing a reasonable limit.

A Future PR will leverage this mechansims for sever artifact runers as
well.
  • Loading branch information
scudette authored Oct 23, 2023
1 parent 553d40d commit 4df78a6
Show file tree
Hide file tree
Showing 53 changed files with 3,800 additions and 1,508 deletions.
1 change: 1 addition & 0 deletions .github/workflows/windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ jobs:
echo "VSStest2 with more data" > c:\Users\test2.txt
echo Clearing the event logs
wevtutil.exe cl System
wevtutil.exe cl security
echo Create second service.
sc.exe create TestingDetection2 binPath="%COMSPEC% /Q /c echo 'COMSPEC testing 2"
Expand Down
20 changes: 20 additions & 0 deletions api/mock/api_mock.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1,202 changes: 606 additions & 596 deletions api/proto/api.pb.go

Large diffs are not rendered by default.

28 changes: 14 additions & 14 deletions api/proto/api.pb.gw.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions api/proto/api.proto
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import "csv.proto";
import "download.proto";
import "completions.proto";
import "vfs_api.proto";
import "scheduler.proto";

package proto;

Expand Down Expand Up @@ -547,6 +548,9 @@ service API {
// Push monitoring event to the server.
rpc WriteEvent(VQLResponse) returns (google.protobuf.Empty) {}

// Scheduler endpoint for minion scheduling
rpc Scheduler(stream ScheduleRequest) returns (stream ScheduleResponse) {}

// Remote data store access.
rpc GetSubject(DataRequest) returns (DataResponse) {}
rpc SetSubject(DataRequest) returns (DataResponse) {}
Expand Down
70 changes: 70 additions & 0 deletions api/proto/api_grpc.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 4df78a6

Please sign in to comment.