Skip to content

feat(glue-alpha): pair workerType and numberOfWorkers into a required workerConfiguration #38576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
otaviomacedo wants to merge 2 commits into
base: main
Choose a base branch
from
+95 −128
Open

feat(glue-alpha): pair workerType and numberOfWorkers into a required workerConfiguration #38576

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
79a5d04
feat(glue-alpha): pair workerType and numberOfWorkers into a required…
otaviomacedo Aug 17, 2026
c81143f
Merge branch 'main' into otaviom/glue/worker-configuration
mergify[bot] Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions packages/@aws-cdk/aws-glue-alpha/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,10 @@ new glue.PySparkEtlJob(stack, 'PySparkETLJob', {
script,
glueVersion: glue.GlueVersion.V5_1,
continuousLogging: { enabled: false },
workerType: glue.WorkerType.G_2X,
workerConfiguration: {
workerType: glue.WorkerType.G_2X,
numberOfWorkers: 2,
},
maxConcurrentRuns: 100,
timeout: cdk.Duration.hours(2),
connections: [glue.Connection.fromConnectionName(stack, 'Connection', 'connectionName')],
Expand All @@ -127,7 +130,6 @@ new glue.PySparkEtlJob(stack, 'PySparkETLJob', {
SecondTagName: 'SecondTagValue',
XTagName: 'XTagValue',
},
numberOfWorkers: 2,
maxRetries: 2,
});
```
Expand Down Expand Up @@ -174,7 +176,10 @@ new glue.PySparkStreamingJob(stack, 'PySparkStreamingJob', {
script,
glueVersion: glue.GlueVersion.V5_1,
continuousLogging: { enabled: false },
workerType: glue.WorkerType.G_2X,
workerConfiguration: {
workerType: glue.WorkerType.G_2X,
numberOfWorkers: 2,
},
maxConcurrentRuns: 100,
timeout: cdk.Duration.hours(2),
connections: [glue.Connection.fromConnectionName(stack, 'Connection', 'connectionName')],
Expand All @@ -184,7 +189,6 @@ new glue.PySparkStreamingJob(stack, 'PySparkStreamingJob', {
SecondTagName: 'SecondTagValue',
XTagName: 'XTagValue',
},
numberOfWorkers: 2,
maxRetries: 2,
});
```
Expand Down Expand Up @@ -229,7 +233,10 @@ new glue.PySparkFlexEtlJob(stack, 'pySparkFlexEtlJob', {
script,
glueVersion: glue.GlueVersion.V5_1,
continuousLogging: { enabled: false },
workerType: glue.WorkerType.G_2X,
workerConfiguration: {
workerType: glue.WorkerType.G_2X,
numberOfWorkers: 2,
},
maxConcurrentRuns: 100,
timeout: cdk.Duration.hours(2),
connections: [glue.Connection.fromConnectionName(stack, 'Connection', 'connectionName')],
Expand All @@ -239,7 +246,6 @@ new glue.PySparkFlexEtlJob(stack, 'pySparkFlexEtlJob', {
SecondTagName: 'SecondTagValue',
XTagName: 'XTagValue',
},
numberOfWorkers: 2,
maxRetries: 2,
});
```
Expand Down
19 changes: 1 addition & 18 deletions packages/@aws-cdk/aws-glue-alpha/lib/jobs/job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { lit } from 'aws-cdk-lib/core/lib/helpers-internal';
import type * as constructs from 'constructs';
import type { Code } from '../code';
import type { IConnection } from '../connection';
import type { MetricType, WorkerType, GlueVersion } from '../constants';
import type { MetricType, GlueVersion } from '../constants';
import { JobState } from '../constants';
import { warnOnPlaintextSecrets } from '../private/secret-detection';
import type { ISecurityConfiguration } from '../security-configuration';
Expand Down Expand Up @@ -343,23 +343,6 @@ export interface JobProps {
*/
readonly description?: string;

/**
* Number of Workers (optional)
* Number of workers for Glue to use during job execution
*
* @default 10
*/
readonly numberOfWorkers?: number;

/**
* Worker Type (optional)
* Type of Worker for Glue to use during job execution
* Enum options: Standard, G_1X, G_2X, G_025X. G_4X, G_8X, Z_2X
*
* @default WorkerType.G_1X
*/
readonly workerType?: WorkerType;
Comment on lines -352 to -361

@otaviomacedo otaviomacedo Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced with WorkerConfiguration.


/**
* Max Concurrent Runs (optional)
* The maximum number of runs this Glue job can concurrently run
Expand Down
4 changes: 2 additions & 2 deletions packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-etl-job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,8 @@ export class PySparkEtlJob extends SparkJob {
pythonVersion: PythonVersion.THREE,
},
glueVersion: props.glueVersion ?? GlueVersion.V4_0,
workerType: props.workerType ?? WorkerType.G_1X,
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers : 10,
workerType: props.workerConfiguration?.workerType ?? WorkerType.G_1X,
numberOfWorkers: props.workerConfiguration?.numberOfWorkers ?? 10,
maxRetries: props.jobRunQueuingEnabled ? 0 : props.maxRetries,
jobRunQueuingEnabled: props.jobRunQueuingEnabled ? props.jobRunQueuingEnabled : false,
notificationProperty: props.notifyDelayAfter ? { notifyDelayAfter: props.notifyDelayAfter.toMinutes() } : undefined,
Expand Down
4 changes: 2 additions & 2 deletions packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-flex-etl-job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,8 +97,8 @@ export class PySparkFlexEtlJob extends SparkJob {
pythonVersion: PythonVersion.THREE,
},
glueVersion: props.glueVersion ? props.glueVersion : GlueVersion.V5_0,
workerType: props.workerType ? props.workerType : WorkerType.G_1X,
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers : 10,
workerType: props.workerConfiguration?.workerType ?? WorkerType.G_1X,
numberOfWorkers: props.workerConfiguration?.numberOfWorkers ?? 10,
maxRetries: props.maxRetries,
executionProperty: props.maxConcurrentRuns ? { maxConcurrentRuns: props.maxConcurrentRuns } : undefined,
notificationProperty: props.notifyDelayAfter ? { notifyDelayAfter: props.notifyDelayAfter.toMinutes() } : undefined,
Expand Down
4 changes: 2 additions & 2 deletions packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-streaming-job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,8 @@ export class PySparkStreamingJob extends SparkJob {
pythonVersion: PythonVersion.THREE,
},
glueVersion: props.glueVersion ? props.glueVersion : GlueVersion.V4_0,
workerType: props.workerType ? props.workerType : WorkerType.G_1X,
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers : 10,
workerType: props.workerConfiguration?.workerType ?? WorkerType.G_1X,
numberOfWorkers: props.workerConfiguration?.numberOfWorkers ?? 10,
maxRetries: props.jobRunQueuingEnabled ? 0 : props.maxRetries,
jobRunQueuingEnabled: props.jobRunQueuingEnabled ? props.jobRunQueuingEnabled : false,
executionProperty: props.maxConcurrentRuns ? { maxConcurrentRuns: props.maxConcurrentRuns } : undefined,
Expand Down
20 changes: 12 additions & 8 deletions packages/@aws-cdk/aws-glue-alpha/lib/jobs/ray-job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import { CfnJob } from 'aws-cdk-lib/aws-glue';
import type * as iam from 'aws-cdk-lib/aws-iam';
import { ValidationError } from 'aws-cdk-lib/core';
import { memoizedGetter, lit } from 'aws-cdk-lib/core/lib/helpers-internal';
import { memoizedGetter } from 'aws-cdk-lib/core/lib/helpers-internal';
import { addConstructMetadata } from 'aws-cdk-lib/core/lib/metadata-resource';
import { propertyInjectable } from 'aws-cdk-lib/core/lib/prop-injectable';
import type { Construct } from 'constructs';
Expand All @@ -24,6 +23,15 @@ export interface RayJobProps extends JobProps {
*/
readonly runtime?: Runtime;

/**
* The number of workers allocated when a job runs.
*
* Ray jobs only support the Z.2X worker type, so the worker type is not configurable.
*
* @default 3
*/
readonly numberOfWorkers?: number;

/**
* Specifies whether job run queuing is enabled for the job runs for this job.
* A value of true means job run queuing is enabled for the job runs.
Expand Down Expand Up @@ -104,10 +112,6 @@ export class RayJob extends Job {
...observabilityMetricsArgs,
};

if (props.workerType && props.workerType !== WorkerType.Z_2X) {
throw new ValidationError(lit`RayJobsOnlySupportZ2XWorkerType`, 'Ray jobs only support Z.2X worker type', this);
}

this.resource = new CfnJob(this, 'Resource', {
name: props.jobName,
description: props.description,
Expand All @@ -118,8 +122,8 @@ export class RayJob extends Job {
runtime: props.runtime ? props.runtime : Runtime.RAY_TWO_FOUR,
},
glueVersion: GlueVersion.V4_0,
workerType: props.workerType ? props.workerType : WorkerType.Z_2X,
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers: 3,
workerType: WorkerType.Z_2X,
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers : 3,
maxRetries: props.jobRunQueuingEnabled ? 0 : props.maxRetries,
jobRunQueuingEnabled: props.jobRunQueuingEnabled ? props.jobRunQueuingEnabled : false,
executionProperty: props.maxConcurrentRuns ? { maxConcurrentRuns: props.maxConcurrentRuns } : undefined,
Expand Down
11 changes: 3 additions & 8 deletions packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-etl-job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import { CfnJob } from 'aws-cdk-lib/aws-glue';
import { ValidationError } from 'aws-cdk-lib/core';
import type * as cdk from 'aws-cdk-lib/core';
import { memoizedGetter, lit } from 'aws-cdk-lib/core/lib/helpers-internal';
import { memoizedGetter } from 'aws-cdk-lib/core/lib/helpers-internal';
import { addConstructMetadata } from 'aws-cdk-lib/core/lib/metadata-resource';
import { propertyInjectable } from 'aws-cdk-lib/core/lib/prop-injectable';
import type { Construct } from 'constructs';
Expand Down Expand Up @@ -98,10 +97,6 @@ export class ScalaSparkEtlJob extends SparkJob {
...this.nonExecutableCommonArguments(props),
};

if ((!props.workerType && props.numberOfWorkers !== undefined) || (props.workerType && props.numberOfWorkers === undefined)) {
throw new ValidationError(lit`WorkerTypeAndNumberOfWorkersMustBothBeSet`, 'Both workerType and numberOfWorkers must be set', this);
}

this.resource = new CfnJob(this, 'Resource', {
name: props.jobName,
description: props.description,
Expand All @@ -111,8 +106,8 @@ export class ScalaSparkEtlJob extends SparkJob {
scriptLocation: this.codeS3ObjectUrl(props.script),
},
glueVersion: props.glueVersion ? props.glueVersion : GlueVersion.V4_0,
workerType: props.workerType ? props.workerType : WorkerType.G_1X,
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers : 10,
workerType: props.workerConfiguration?.workerType ?? WorkerType.G_1X,
numberOfWorkers: props.workerConfiguration?.numberOfWorkers ?? 10,
maxRetries: props.jobRunQueuingEnabled ? 0 : props.maxRetries,
jobRunQueuingEnabled: props.jobRunQueuingEnabled ? props.jobRunQueuingEnabled : false,
notificationProperty: props.notifyDelayAfter ? { notifyDelayAfter: props.notifyDelayAfter.toMinutes() } : undefined,
Expand Down
4 changes: 2 additions & 2 deletions packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-flex-etl-job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,8 @@ export class ScalaSparkFlexEtlJob extends SparkJob {
scriptLocation: this.codeS3ObjectUrl(props.script),
},
glueVersion: props.glueVersion ? props.glueVersion : GlueVersion.V5_0,
workerType: props.workerType ? props.workerType : WorkerType.G_1X,
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers : 10,
workerType: props.workerConfiguration?.workerType ?? WorkerType.G_1X,
numberOfWorkers: props.workerConfiguration?.numberOfWorkers ?? 10,
maxRetries: props.maxRetries,
executionProperty: props.maxConcurrentRuns ? { maxConcurrentRuns: props.maxConcurrentRuns } : undefined,
notificationProperty: props.notifyDelayAfter ? { notifyDelayAfter: props.notifyDelayAfter.toMinutes() } : undefined,
Expand Down
11 changes: 3 additions & 8 deletions packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-streaming-job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import { CfnJob } from 'aws-cdk-lib/aws-glue';
import { ValidationError } from 'aws-cdk-lib/core';
import { memoizedGetter, lit } from 'aws-cdk-lib/core/lib/helpers-internal';
import { memoizedGetter } from 'aws-cdk-lib/core/lib/helpers-internal';
import { addConstructMetadata } from 'aws-cdk-lib/core/lib/metadata-resource';
import { propertyInjectable } from 'aws-cdk-lib/core/lib/prop-injectable';
import type { Construct } from 'constructs';
Expand Down Expand Up @@ -90,10 +89,6 @@ export class ScalaSparkStreamingJob extends SparkJob {
...this.nonExecutableCommonArguments(props),
};

if ((!props.workerType && props.numberOfWorkers !== undefined) || (props.workerType && props.numberOfWorkers === undefined)) {
throw new ValidationError(lit`WorkerTypeAndNumberRequired`, 'Both workerType and numberOfWorkers must be set', this);
}

this.resource = new CfnJob(this, 'Resource', {
name: props.jobName,
description: props.description,
Expand All @@ -103,8 +98,8 @@ export class ScalaSparkStreamingJob extends SparkJob {
scriptLocation: this.codeS3ObjectUrl(props.script),
},
glueVersion: props.glueVersion ? props.glueVersion : GlueVersion.V4_0,
workerType: props.workerType ? props.workerType : WorkerType.G_1X,
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers : 10,
workerType: props.workerConfiguration?.workerType ?? WorkerType.G_1X,
numberOfWorkers: props.workerConfiguration?.numberOfWorkers ?? 10,
maxRetries: props.jobRunQueuingEnabled ? 0 : props.maxRetries,
jobRunQueuingEnabled: props.jobRunQueuingEnabled ? props.jobRunQueuingEnabled : false,
executionProperty: props.maxConcurrentRuns ? { maxConcurrentRuns: props.maxConcurrentRuns } : undefined,
Expand Down
29 changes: 29 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/jobs/spark-job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { lit } from 'aws-cdk-lib/core/lib/helpers-internal';
import { addConstructMetadata } from 'aws-cdk-lib/core/lib/metadata-resource';
import type * as constructs from 'constructs';
import type { Code } from '../code';
import type { WorkerType } from '../constants';
import type { JobProps } from './job';
import { Job } from './job';

Expand Down Expand Up @@ -90,10 +91,38 @@ export interface SparkUILoggingLocation {
readonly prefix?: string;
}

/**
* The worker configuration for a Spark job.
*
* The worker type and the number of workers are set together: providing this
* configuration requires both values, so a Spark job can never be given one
* without the other.
*/
export interface WorkerConfiguration {

@gasolima gasolima Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm always confused when nesting vs flat for me this against guideline, can you explain to me why did you go with this choice and also update the guideline for other to understand?

@otaviomacedo otaviomacedo Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default, we should prefer flat interfaces, to make it more ergonomic for other jsii languages, such as Java. So, if it's just to organize the fields more nicely, it's not worth it. But here, the nested interface serves another ergonomic purpose: to make sure that invalid states are unrepresentable. In this case, workerType and numberOfWorkers must either both be set or neither. If they are two independent optional props in a flat interface, nothing prevents you from setting one and not the other. If they are required properties in a nested interface, you can't do that.

/**
* The type of predefined worker that is allocated when a job runs.
*
* Enum options: Standard, G_1X, G_2X, G_025X, G_4X, G_8X, Z_2X
*/
readonly workerType: WorkerType;

/**
* The number of workers of the given `workerType` that are allocated when a job runs.
*/
readonly numberOfWorkers: number;
}

/**
* Common properties for different types of Spark jobs.
*/
export interface SparkJobProps extends JobProps {
/**
* The worker type and the number of workers allocated when a job runs.
*
* @default - the job runs with the G_1X worker type and 10 workers.
*/
readonly workerConfiguration?: WorkerConfiguration;

@gasolima gasolima Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By moving it to SparkJobProps instead of leaving it to JobProps means this property will not be anymore in PythonShellJobProps, is that intended?

@otaviomacedo otaviomacedo Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but PythonShellJobProps never reads them. What controls its capacity is maxCapacity.


/**
* Enables the Spark UI debugging and monitoring with the specified props.
*
Expand Down
3 changes: 1 addition & 2 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job-pyspark-etl.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,7 @@ new glue.PySparkEtlJob(stack, 'OverridePySparkETLJob', {
role: iam_role,
description: 'Optional Override PySpark ETL Job',
glueVersion: glue.GlueVersion.V5_1,
numberOfWorkers: 20,
workerType: glue.WorkerType.G_1X,
workerConfiguration: { workerType: glue.WorkerType.G_1X, numberOfWorkers: 20 },
timeout: cdk.Duration.minutes(15),
jobName: 'Optional Override PySpark ETL Job',
defaultArguments: {
Expand Down
3 changes: 1 addition & 2 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job-pyspark-flex-etl.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,7 @@ new glue.PySparkFlexEtlJob(stack, 'OverridePySparkFlexEtlJob', {
role: iam_role,
description: 'Optional Override PySpark Flex Etl Job',
glueVersion: glue.GlueVersion.V3_0,
numberOfWorkers: 20,
workerType: glue.WorkerType.G_1X,
workerConfiguration: { workerType: glue.WorkerType.G_1X, numberOfWorkers: 20 },
timeout: cdk.Duration.minutes(15),
jobName: 'Optional Override PySpark Flex Etl Job',
defaultArguments: {
Expand Down
3 changes: 1 addition & 2 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job-pyspark-streaming.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,7 @@ new glue.PySparkStreamingJob(stack, 'OverridePySparkStreamingJob', {
role: iam_role,
description: 'Optional Override PySpark Streaming Job',
glueVersion: glue.GlueVersion.V3_0,
numberOfWorkers: 20,
workerType: glue.WorkerType.G_1X,
workerConfiguration: { workerType: glue.WorkerType.G_1X, numberOfWorkers: 20 },
timeout: cdk.Duration.minutes(15),
jobName: 'Optional Override PySpark Streaming Job',
defaultArguments: {
Expand Down
3 changes: 1 addition & 2 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job-scalaspark-etl.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,7 @@ new glue.ScalaSparkEtlJob(stack, 'OverrideScalaSparkETLJob', {
role: iam_role,
description: 'Optional Override ScalaSpark ETL Job',
glueVersion: glue.GlueVersion.V3_0,
numberOfWorkers: 20,
workerType: glue.WorkerType.G_1X,
workerConfiguration: { workerType: glue.WorkerType.G_1X, numberOfWorkers: 20 },
timeout: cdk.Duration.minutes(15),
jobName: 'Optional Override ScalaSpark ETL Job',
defaultArguments: {
Expand Down
3 changes: 1 addition & 2 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job-scalaspark-flex-etl.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,7 @@ new glue.ScalaSparkFlexEtlJob(stack, 'OverrideScalaSparkFlexEtlJob', {
role: iam_role,
description: 'Optional Override ScalaSpark Flex Etl Job',
glueVersion: glue.GlueVersion.V3_0,
numberOfWorkers: 20,
workerType: glue.WorkerType.G_1X,
workerConfiguration: { workerType: glue.WorkerType.G_1X, numberOfWorkers: 20 },
timeout: cdk.Duration.minutes(15),
jobName: 'Optional Override ScalaSpark Flex Etl Job',
defaultArguments: {
Expand Down
3 changes: 1 addition & 2 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.job-scalaspark-streaming.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,7 @@ new glue.ScalaSparkStreamingJob(stack, 'OverrideScalaSparkStreamingJob', {
role: iam_role,
description: 'Optional Override ScalaSpark Streaming Job',
glueVersion: glue.GlueVersion.V3_0,
numberOfWorkers: 20,
workerType: glue.WorkerType.G_1X,
workerConfiguration: { workerType: glue.WorkerType.G_1X, numberOfWorkers: 20 },
timeout: cdk.Duration.minutes(15),
jobName: 'Optional Override ScalaSpark Streaming Job',
defaultArguments: {
Expand Down
6 changes: 2 additions & 4 deletions packages/@aws-cdk/aws-glue-alpha/test/integ.workflow.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,14 @@ const OutboundJob = new glue.PySparkEtlJob(stack, 'OutboundJob', {
script: script,
role,
glueVersion: glue.GlueVersion.V4_0,
workerType: glue.WorkerType.G_2X,
numberOfWorkers: 2,
workerConfiguration: { workerType: glue.WorkerType.G_2X, numberOfWorkers: 2 },
});

const InboundJob = new glue.PySparkEtlJob(stack, 'InboundJob', {
script: script,
role,
glueVersion: glue.GlueVersion.V4_0,
workerType: glue.WorkerType.G_2X,
numberOfWorkers: 2,
workerConfiguration: { workerType: glue.WorkerType.G_2X, numberOfWorkers: 2 },
});

workflow.addOnDemandTrigger('OnDemandTrigger', {
Expand Down
Loading
Loading