Skip to content

[Merge 2nd]Generate AbortMultipartUpload, CreateBucketMetadataTableConfiguration, DeleteBucket from the model #3847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: development
Choose a base branch
from
Open

[Merge 2nd]Generate AbortMultipartUpload, CreateBucketMetadataTableConfiguration, DeleteBucket from the model #3847

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3b35c81
generator updates to run generator successfully for s3
peterrsongg May 22, 2025
1db9208
commit generator changes
peterrsongg May 27, 2025
d2df562
add custom tt files update other tt files
peterrsongg May 28, 2025
1314f40
updates to restxmlresponseunmarshaller.tt
peterrsongg May 28, 2025
fd46027
Add custom logic for s3
peterrsongg May 29, 2025
9ba0e7c
Generate AbortMultipartUpload and CreateBucketMetadataTableConfigurat…
peterrsongg May 29, 2025
118a58f
Remove greedy trim start for s3
peterrsongg May 29, 2025
29f5ec2
Add DeleteBucket to generated operation and other fixes
peterrsongg May 29, 2025
d1e5605
minor tweaks for pr readiness
peterrsongg May 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions generator/.DevConfigs/7064092c-4180-4624-a402-c6d354a1dfeb.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"services": [
{
"serviceName": "S3",
"type": "patch",
"changeLogMessages": [
"Generate AbortMultipartUpload, CreateBucketMetadataTableConfiguration, DeleteBucket from the model. Delete custom files."
]
}
]
}
4 changes: 4 additions & 0 deletions generator/ServiceClientGeneratorLib/Example.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,8 @@ public IList<string> GetRequestAssignments(int currentIndent)
var last = InputParameters.Last().Key;
foreach (var param in InputParameters)
{
if (Operation == null)
continue;
var member = Operation.RequestStructure.Members.GetMemberByName(param.Key);

if (null == member)
Expand Down Expand Up @@ -186,6 +188,8 @@ public IList<string> GetResponseAssignments()

foreach (var param in OutputParameters)
{
if (Operation == null)
continue;
var member = Operation.ResponseStructure.Members.GetMemberByName(param.Key);

if (null == member)
Expand Down
92 changes: 67 additions & 25 deletions generator/ServiceClientGeneratorLib/GeneratorDriver.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,10 +195,8 @@ public void Execute()

if (Configuration.Namespace == "Amazon.S3")
{
ExecuteProjectFileGenerators();
// The AmazonS3RetryPolicy simply populates the static list of requests to retry for a status code of 200 which returns an exception.
ExecuteGenerator(new AmazonS3RetryPolicy(), "AmazonS3RetryPolicy.cs");
return;
}
// The top level request that all operation requests are children of
ExecuteGenerator(new BaseRequest(), "Amazon" + Configuration.ClassName + "Request.cs", "Model");
Expand All @@ -207,10 +205,16 @@ public void Execute()
string.Format("ServiceEnumerations.{0}.cs", Configuration.ClassName) : "ServiceEnumerations.cs";

// Any enumerations for the service
this.ExecuteGenerator(new ServiceEnumerations(), enumFileName);
// skip s3 until we're at the end of s3 client generation
if (this.Configuration.ServiceId != "S3")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the comment says, I'm skipping enumerations for now. None of these operations touch the enumerations and there is some shape renaming that must happen that we won't deal with in phase 1.

{
this.ExecuteGenerator(new ServiceEnumerations(), enumFileName);
}


// Any paginators for the service
if (Configuration.ServiceModel.HasPaginators)
// skip paginators for s3 until we're at the end of s3 client generation
if (Configuration.ServiceModel.HasPaginators && Configuration.ServiceId != "S3")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as the comment says, paginators will not be included in phase 1

{
foreach (var operation in Configuration.ServiceModel.Operations)
{
Expand All @@ -227,24 +231,43 @@ public void Execute()

// Do not generate base exception if this is a child model.
// We use the base exceptions generated for the parent model.
if (!this.Configuration.IsChildConfig)
if (!this.Configuration.IsChildConfig && this.Configuration.ServiceId != "S3")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the base AmazonS3Exception class and its custom code should be left as is. There is no need to create a custom .tt file just for this one class.

{
this.ExecuteGenerator(new BaseServiceException(), "Amazon" + this.Configuration.ClassName + "Exception.cs");
}

// Generates the Request, Response, Marshaller, Unmarshaller, and Exception objects for a given client operation
foreach (var operation in Configuration.ServiceModel.Operations)
if (Configuration.Namespace == "Amazon.S3")
{
GenerateRequest(operation);
GenerateResponse(operation);
GenerateRequestMarshaller(operation);
GenerateResponseUnmarshaller(operation);
GenerateEndpointDiscoveryMarshaller(operation);
GenerateExceptions(operation);
GenerateStructures(operation);
GenerateEventStreamPublisher(operation);
foreach (var operation in Configuration.ServiceModel.S3AllowListOperations)
{
GenerateRequest(operation);
GenerateResponse(operation);
GenerateRequestMarshaller(operation);
GenerateResponseUnmarshaller(operation);
GenerateEndpointDiscoveryMarshaller(operation);
GenerateExceptions(operation);
GenerateStructures(operation);
GenerateEventStreamPublisher(operation);
}
}

else
{
foreach (var operation in Configuration.ServiceModel.Operations)
{
GenerateRequest(operation);
GenerateResponse(operation);
GenerateRequestMarshaller(operation);
GenerateResponseUnmarshaller(operation);
GenerateEndpointDiscoveryMarshaller(operation);
GenerateExceptions(operation);
GenerateStructures(operation);
GenerateEventStreamPublisher(operation);
}
}
// Generates the Request, Response, Marshaller, Unmarshaller, and Exception objects for a given client operation


if (Configuration.ServiceModel.Customizations.GenerateCustomUnmarshaller)
{
GenerateCustomUnmarshallers(Configuration.ServiceModel.Customizations.CustomUnmarshaller);
Expand All @@ -259,16 +282,19 @@ public void Execute()
var fileName = string.Format("{0}EndpointDiscoveryMarshallingTests.cs", Configuration.ClassName);
ExecuteTestGenerator(new EndpointDiscoveryMarshallingTests(), fileName, "Marshalling");
}

// Test that simple customizations were generated correctly
GenerateCustomizationTests();
ExecuteProjectFileGenerators();
if (this.Configuration.ServiceModel.Customizations.HasExamples)
// Test that simple customizations were generated correctly
if (this.Configuration.ServiceId != "S3")
{
var servicename = Configuration.Namespace.Split('.').Last();
ExecuteExampleGenerator(new ExampleCode(), servicename + ".GeneratedSamples.cs", servicename);
ExecuteExampleGenerator(new ExampleMetadata(), servicename + ".GeneratedSamples.extra.xml");
GenerateCustomizationTests();
if (this.Configuration.ServiceModel.Customizations.HasExamples)
{
var servicename = Configuration.Namespace.Split('.').Last();
ExecuteExampleGenerator(new ExampleCode(), servicename + ".GeneratedSamples.cs", servicename);
ExecuteExampleGenerator(new ExampleMetadata(), servicename + ".GeneratedSamples.extra.xml");
}
}

}

/// <summary>
Expand All @@ -280,7 +306,7 @@ void GenerateRequest(Operation operation)
var requestGenerator = new StructureGenerator
{
ClassName = operation.Name + "Request",
BaseClass = string.Format("Amazon{0}Request", Configuration.ClassName),
BaseClass = this.Configuration.ServiceId != "S3" ? string.Format("Amazon{0}Request", Configuration.ClassName) : "AmazonWebServiceRequest",
StructureType = StructureType.Request,
Operation = operation
};
Expand Down Expand Up @@ -671,6 +697,8 @@ void GenerateResponseUnmarshaller(Operation operation)
// instead we generated a layer over the structure. That layer is EventStreamGenerator.tt.
if (nestedStructure.IsEventStream)
continue;
if (this.Configuration.ServiceId == "S3" && nestedStructure.IsEvent || nestedStructure.IsEventStream)
continue;
// Skip already processed unmarshallers. This handles the case of structures being returned in mulitiple requests.
if (!this._processedUnmarshallers.Contains(nestedStructure.Name))
{
Expand Down Expand Up @@ -700,7 +728,14 @@ private void GenerateUnmarshaller(Shape shape)

if (this.Configuration.ServiceModel.Customizations.IsSubstitutedShape(nestedStructure.Name))
continue;

if (this.Configuration.ServiceId == "S3")
{
if (shape.IsEvent || shape.IsEventStream)
{
this._processedUnmarshallers.Add(nestedStructure.Name);
continue;
}
}
// Document structure don't need a custom unmarshaller, they use
// the 'simple' DocumentMarshaller in AWSSDK.
if (nestedStructure.IsDocument)
Expand Down Expand Up @@ -781,7 +816,6 @@ private void GenerateExceptions(Operation operation)
// Skip exceptions that have already been generated for the parent model
if (IsExceptionPresentInParentModel(this.Configuration, exceptionShape.Name) || this._processedStructures.Contains(exceptionShape.Name))
continue;

var generator = new StructureGenerator()
{
ClassName = exceptionShape.Name,
Expand Down Expand Up @@ -1002,6 +1036,14 @@ void GenerateStructures(Operation operation)
if (definition.IsEventStream && !Configuration.ServiceModel.Operations.Any(x => string.Equals(x.ResponseEventStreamingMember?.Shape.Name, definition.Name)))
continue;

if (this.Configuration.ServiceId == "S3")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in the design doc, events and all eventstream based classes will be kept as custom code since SelectObjectContentEventStream has been deprecated.

{
if (definition.IsEvent || definition.IsEventStream)
{
this._processedStructures.Add(definition.Name);
continue;
}
}
if (!this._processedStructures.Contains(definition.Name))
{
// if the shape had a substitution, we can skip generation
Expand Down
Loading