Improve Proto Generation: Download proto file directly instead of via submodule (#39)

Author: berndverst

CHANGELOG.md

@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Protos are compiled with gRPC 1.62.3 / protobuf 3.25.X instead of the latest release. This ensures compatibility with a wider range of grpcio versions for better compatibility with other packages / libraries ([#36](https://github.com/microsoft/durabletask-python/pull/36)) - by [@berndverst](https://github.com/berndverst)
 - Http and grpc protocols and their secure variants are stripped from the host name parameter if provided. Secure mode is enabled if the protocol provided is https or grpcs ([#38](https://github.com/microsoft/durabletask-python/pull/38) - by [@berndverst)(https://github.com/berndverst)
+- Improve ProtoGen by downloading proto file directly instead of using submodule ([#39](https://github.com/microsoft/durabletask-python/pull/39) - by [@berndverst](https://github.com/berndverst)
 
 ### Updates
 

Makefile

@@ -11,7 +11,8 @@ install:
 	python3 -m pip install .
 
 gen-proto:
-	cp ./submodules/durabletask-protobuf/protos/orchestrator_service.proto durabletask/internal/orchestrator_service.proto
+	curl -o durabletask/internal/orchestrator_service.proto https://raw.githubusercontent.com/microsoft/durabletask-protobuf/refs/heads/main/protos/orchestrator_service.proto
+	curl -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/microsoft/durabletask-protobuf/commits?path=protos/orchestrator_service.proto&sha=main&per_page=1" | jq -r '.[0].sha' >> durabletask/internal/PROTO_SOURCE_COMMIT_HASH
 	python3 -m grpc_tools.protoc --proto_path=.  --python_out=. --pyi_out=. --grpc_python_out=. ./durabletask/internal/orchestrator_service.proto
 	rm durabletask/internal/*.proto
 

README.md

@@ -161,19 +161,13 @@ The following is more information about how to develop this project. Note that d
 
 ### Generating protobufs
 
-Protobuf definitions are stored in the [./submodules/durabletask-proto](./submodules/durabletask-proto) directory, which is a submodule. To update the submodule, run the following command from the project root:
-
-```sh
-git submodule update --init
-```
-
-Once the submodule is available, the corresponding source code can be regenerated using the following command from the project root:
-
 ```sh
 pip3 install -r dev-requirements.txt
 make gen-proto
 ```
 
+This will download the `orchestrator_service.proto` from the `microsoft/durabletask-protobuf` repo and compile it using `grpcio-tools`. The version of the source proto file that was downloaded can be found in the file `durabletask/internal/PROTO_SOURCE_COMMIT_HASH`.
+
 ### Running unit tests
 
 Unit tests can be run using the following command from the project root. Unit tests _don't_ require a sidecar process to be running.

durabletask/internal/PROTO_SOURCE_COMMIT_HASH

@@ -0,0 +1 @@
+443b333f4f65a438dc9eb4f090560d232afec4b7

durabletask/internal/orchestrator_service_pb2.py

@@ -63,16 +63,18 @@ class ActivityRequest(_message.Message):
     def __init__(self, name: _Optional[str] = ..., version: _Optional[_Union[_wrappers_pb2.StringValue, _Mapping]] = ..., input: _Optional[_Union[_wrappers_pb2.StringValue, _Mapping]] = ..., orchestrationInstance: _Optional[_Union[OrchestrationInstance, _Mapping]] = ..., taskId: _Optional[int] = ..., parentTraceContext: _Optional[_Union[TraceContext, _Mapping]] = ...) -> None: ...
 
 class ActivityResponse(_message.Message):
-    __slots__ = ("instanceId", "taskId", "result", "failureDetails")
+    __slots__ = ("instanceId", "taskId", "result", "failureDetails", "completionToken")
     INSTANCEID_FIELD_NUMBER: _ClassVar[int]
     TASKID_FIELD_NUMBER: _ClassVar[int]
     RESULT_FIELD_NUMBER: _ClassVar[int]
     FAILUREDETAILS_FIELD_NUMBER: _ClassVar[int]
+    COMPLETIONTOKEN_FIELD_NUMBER: _ClassVar[int]
     instanceId: str
     taskId: int
     result: _wrappers_pb2.StringValue
     failureDetails: TaskFailureDetails
-    def __init__(self, instanceId: _Optional[str] = ..., taskId: _Optional[int] = ..., result: _Optional[_Union[_wrappers_pb2.StringValue, _Mapping]] = ..., failureDetails: _Optional[_Union[TaskFailureDetails, _Mapping]] = ...) -> None: ...
+    completionToken: str
+    def __init__(self, instanceId: _Optional[str] = ..., taskId: _Optional[int] = ..., result: _Optional[_Union[_wrappers_pb2.StringValue, _Mapping]] = ..., failureDetails: _Optional[_Union[TaskFailureDetails, _Mapping]] = ..., completionToken: _Optional[str] = ...) -> None: ...
 
 class TaskFailureDetails(_message.Message):
     __slots__ = ("errorType", "errorMessage", "stackTrace", "innerFailure", "isNonRetriable")
@@ -421,14 +423,16 @@ class OrchestratorRequest(_message.Message):
     def __init__(self, instanceId: _Optional[str] = ..., executionId: _Optional[_Union[_wrappers_pb2.StringValue, _Mapping]] = ..., pastEvents: _Optional[_Iterable[_Union[HistoryEvent, _Mapping]]] = ..., newEvents: _Optional[_Iterable[_Union[HistoryEvent, _Mapping]]] = ..., entityParameters: _Optional[_Union[OrchestratorEntityParameters, _Mapping]] = ...) -> None: ...
 
 class OrchestratorResponse(_message.Message):
-    __slots__ = ("instanceId", "actions", "customStatus")
+    __slots__ = ("instanceId", "actions", "customStatus", "completionToken")
     INSTANCEID_FIELD_NUMBER: _ClassVar[int]
     ACTIONS_FIELD_NUMBER: _ClassVar[int]
     CUSTOMSTATUS_FIELD_NUMBER: _ClassVar[int]
+    COMPLETIONTOKEN_FIELD_NUMBER: _ClassVar[int]
     instanceId: str
     actions: _containers.RepeatedCompositeFieldContainer[OrchestratorAction]
     customStatus: _wrappers_pb2.StringValue
-    def __init__(self, instanceId: _Optional[str] = ..., actions: _Optional[_Iterable[_Union[OrchestratorAction, _Mapping]]] = ..., customStatus: _Optional[_Union[_wrappers_pb2.StringValue, _Mapping]] = ...) -> None: ...
+    completionToken: str
+    def __init__(self, instanceId: _Optional[str] = ..., actions: _Optional[_Iterable[_Union[OrchestratorAction, _Mapping]]] = ..., customStatus: _Optional[_Union[_wrappers_pb2.StringValue, _Mapping]] = ..., completionToken: _Optional[str] = ...) -> None: ...
 
 class CreateInstanceRequest(_message.Message):
     __slots__ = ("instanceId", "name", "version", "input", "scheduledStartTimestamp", "orchestrationIdReusePolicy", "executionId", "tags")
@@ -856,8 +860,12 @@ class StartNewOrchestrationAction(_message.Message):
     def __init__(self, instanceId: _Optional[str] = ..., name: _Optional[str] = ..., version: _Optional[_Union[_wrappers_pb2.StringValue, _Mapping]] = ..., input: _Optional[_Union[_wrappers_pb2.StringValue, _Mapping]] = ..., scheduledTime: _Optional[_Union[_timestamp_pb2.Timestamp, _Mapping]] = ...) -> None: ...
 
 class GetWorkItemsRequest(_message.Message):
-    __slots__ = ()
-    def __init__(self) -> None: ...
+    __slots__ = ("maxConcurrentOrchestrationWorkItems", "maxConcurrentActivityWorkItems")
+    MAXCONCURRENTORCHESTRATIONWORKITEMS_FIELD_NUMBER: _ClassVar[int]
+    MAXCONCURRENTACTIVITYWORKITEMS_FIELD_NUMBER: _ClassVar[int]
+    maxConcurrentOrchestrationWorkItems: int
+    maxConcurrentActivityWorkItems: int
+    def __init__(self, maxConcurrentOrchestrationWorkItems: _Optional[int] = ..., maxConcurrentActivityWorkItems: _Optional[int] = ...) -> None: ...
 
 class WorkItem(_message.Message):
     __slots__ = ("orchestratorRequest", "activityRequest", "entityRequest", "healthPing", "completionToken")