Dataflow: ListJobs API is missing name request parameter

[kberezin-nshl]

ListJobsRequest protobuf is missing name parameter, which is clearly present in the Dataflow API documentation.

I also tested it manually using CURL and it works.

[suztomo]

Would you share your curl command?

[kberezin-nshl]

Sure:

curl -X GET \
	"https://dataflow.googleapis.com/v1b3/projects/PRJ-NAME-HERE/locations/us-east1/jobs?name=foo-bar-baz" \
	-H "Content-Type: application/json" \
	-H "Authorization: Bearer $(gcloud auth print-access-token)" \

[kberezin-nshl]

@suztomo hey, any news on this? I believe it's just a simple one-liner fix, right?

[suztomo]

Try public final Job getJob(GetJobRequest request). The symbol "name" has special meaning in the underlying framework of the libraries in this repository (Protobuf/gRPC, not JSON/HTTP). The identifier of the job is job ID:

google-cloud-java/java-dataflow/google-cloud-dataflow/src/main/java/com/google/dataflow/v1beta3/JobsV1Beta3Client.java

Line 384 in b68c98c

public final Job getJob(GetJobRequest request) {

   * try (JobsV1Beta3Client jobsV1Beta3Client = JobsV1Beta3Client.create()) {
   *   GetJobRequest request =
   *       GetJobRequest.newBuilder()
   *           .setProjectId("projectId-894832108")
   *           .setJobId("jobId101296568")
   *           .setView(JobView.forNumber(0))
   *           .setLocation("location1901043637")
   *           .build();
   *   Job response = jobsV1Beta3Client.getJob(request);
   * }

[suztomo]

We found a stale library generation definiton (proto files). If getJob doesn't suit your needs, let me know.

[kberezin-nshl]

Yeah, that's the whole culprit here, that I don't know the job ID yet and I am trying to find it out.

Basically, here's my scenario (oversimplified):

• we have stable and unique Job names, which guarantees that whenever we try to request the same job twice (for example, due to PubSub message double-delivery, which happens from time to time), we'll get 409 from Dataflow
• unfortunately, with that 409 we don't get a job info (Job protobuf), just an error message, but we still need that info to continue the execution correctly
• so whenever I receive 409, I go ahead and try to find this job using List endpoint
• without being able to supply the name filter, I have to navigate through all the results in the output and potentially even paginate through them to find my job

It is not the end of the world, of course, but having a more straightforward approach, especially considering that the API already supports that, would be great.

[suztomo]

I'll work on fixing the stale library definition. I'll update you here.