Migrate type of Keys, NewImage and OldImage in DynamoDBStreamRecord to dynamodb.AttributeValue

[morcmarc]

Keys, NewImage and OldImage in DynamoDBStreamRecord currently have an internal DynamoDBAttributeValue type, however this is virtually the same type as AttributeValue in "github.com/aws/aws-sdk-go/service/dynamodb" but "less usable" since it doesn't have the convenience method, like Marshal and Unmarshal.

As @cmello pointed out in #54, aws-lambda-go doesn't depend on said package at the moment but I think we should explore the idea of either migrating the type and importing the SDK as a dependency, or copying the sub-packages over to this repo (don't like this idea), so we can have the same level of functionality.

For those wondering (@moedeveloper) what to do in the meantime, I found the following solution to be working quite well: https://gist.github.com/morcmarc/35194b39184977c2c44eb9fe063889a3

[canni]

Just today I was doing an upgrade from NodeJS wrapper (Apex) lambdas to native go and was hit by this issue pretty hard on around ~100 lambdas, work-around works for now, but feels wrong on many levels

[morcmarc]

I'm happy to make the change and submit a pull request if the AWS team is happy with that?

[morcmarc]

I implemented the change, and added aws-sdk-go as a dependency. I did a quick test and noticed an increase of ~1MB in the final binary size on average, min: 0.89MB, max: 1.3MB, median: 0.95MB.

Disclaimer: my number are probably not representative. I only tested with 6 different lambdas and numbers were rounded to 1 decimal.

For us, this is acceptable, but I can see this being an issue for others, considering the 50MB package limit.

[bnegrao]

I implemented the change, and added aws-sdk-go as a dependency. I did a quick test and noticed an increase of ~1MB in the final binary size on average, min: 0.89MB, max: 1.3MB, median: 0.95MB.
@morcmarc , will your modification be added to the distribution? how can I be informed ?

[anton-pfc]

Hey guys, any update on this? It's 2019 already and it's still not clear how do we consume DynamoDb stream in Lambda function written in golang... The example from the very first link does not work out of the box with this example https://github.com/aws/aws-lambda-go/blob/master/events/testdata/dynamodb-event.json

[anton-pfc]

I've found this solution on stackoverflow - it works for me:

// UnmarshalStreamImage converts events.DynamoDBAttributeValue to struct
func UnmarshalStreamImage(attribute map[string]events.DynamoDBAttributeValue, out interface{}) error {

	dbAttrMap := make(map[string]*dynamodb.AttributeValue)

	for k, v := range attribute {

		var dbAttr dynamodb.AttributeValue

		bytes, marshalErr := v.MarshalJSON()
		if marshalErr != nil {
			return marshalErr
		}

		err := json.Unmarshal(bytes, &dbAttr)
		if err != nil {
			return err
		}
		dbAttrMap[k] = &dbAttr
	}

	return dynamodbattribute.UnmarshalMap(dbAttrMap, out)
}

[RandomSeeded]

Workarounds do work, but it does seem like this should have improved out-of-the-box support. DynamoDB streams -> lambda is a pretty normal use-case.

[Luke-Davies]

For anyone looking for an aws-sdk-go-v2 solution:
https://gist.github.com/Luke-Davies/eb67ddd90f73676a64ce5b0c7cf2ba80

There's also some related discussion here:
aws/aws-sdk-go-v2#1124

I also think the fact that the sdk had a major upgrade supports aws-lambda-go's decision not to depend on the sdk - which version should they support out of v1, v2 or possible future versions. I think some other solution is required, as discussed on the linked issue.

[MarkCaro]

The previous solutions seem to be outdated because they don't work for me.

Thankfully, I found this neat package that does the trick:
https://github.com/aereal/go-dynamodb-attribute-conversions

I hope AWS adds proper marshal/unmarshal helper functions to SDK V2 because they are so commonly used.

[bmoffatt]

In the Java runtime project, we eventually solved a similar problem by introducing com.amazonaws:aws-lambda-java-events-sdk-transformer - https://github.com/aws/aws-lambda-java-libs/tree/master/aws-lambda-java-events-sdk-transformer

A similar approach could be considered here, and may work out better than it would have in early 2018 before go modules had been released and caught on

A key thing I'd want to understand before anyone puts in the effort to write the code, would be to know if the aws-lambda-go module can declare dependencies on the aws sdk v1 and v2, without also triggering dependabot spam for this project, and any consumer, every several days when the aws sdk does a release