✨NEW
- Add a composite transform (
WriteToSqs) that sends messages to a SQS queue in batch, using thesend_message_batchmethod of the boto3 package. - Provide options that handle failed records.
- max_trials - The maximum number of trials when there is one or more failed records.
- append_error - Whether to append error details to failed records.
- Return failed elements by a tagged output, which allows users to determine how to handle them subsequently.
- Create a dedicated pipeline option (
SqsOptions) that reads AWS related values (e.g.aws_access_key_id) from pipeline arguments. - Implement metric objects that record the total, succeeded and failed elements counts.
- Add unit and integration testing cases. The moto and localstack-utils are used for unit and integration testing respectively. Also, a custom test client is created for testing retry behavior, which is not supported by the moto package.
- Integrate with GitHub Actions by adding workflows for testing, documentation and release management.