[BUG] Update Training and Serving Merlin on AWS SageMaker using latest merlin image

[rnyak]

Bug description

One of our customers are having an issue of reproducing the training and serving Merlin on AWS SM example and they get an error (will be provided eventually).

The documentation also should be improved/clarified since it is not clear how one can generate the dataset in Generating Dataset without installing Merlin libs, and using Merlin image.

Steps/Code to reproduce bug

The notebooks should be tested with the latest stable merlin-tensorflow image, and updated if required. Currently, in the example merlin-tensorflow:22.10 image is used.

Expected behavior

Environment details

• Merlin version:
• Platform:
• Python version:
• PyTorch version (GPU?):
• Tensorflow version (GPU?):

Additional context

[edknv]

I'm working on updating the merlin-tensorflow image to 23.06 here: #1040.

After bumping the image version to 23.06 and updating the processing workflow in train.py to reflect recent changes, and running the updated example on AWS, we are getting an error:

Failed to transform operator <merlin.systems.dag.runtimes.triton.ops.workflow.TransformWorkflowTriton object at 0x7fe7df82a160>

RuntimeError: Failed for execute the inference request. Model '0_transformworkflowtriton' is not ready.

which doesn't tell us much what is going wrong. I'll try to run the container locally to debug.

[rnyak]

I'm working on updating the merlin-tensorflow image to 23.06 here: #1040.

After bumping the image version to 23.06 and updating the processing workflow in train.py to reflect recent changes, and running the updated example on AWS, we are getting an error:

Failed to transform operator <merlin.systems.dag.runtimes.triton.ops.workflow.TransformWorkflowTriton object at 0x7fe7df82a160>

RuntimeError: Failed for execute the inference request. Model '0_transformworkflowtriton' is not ready.

which doesn't tell us much what is going wrong. I'll try to run the container locally to debug.

thanks @edknv !

[wei-m-teh]

@edknv are there any update on this issue? I am trying to deploy a Merlin model to Sagemaker following the example given. I am running into the same issue.

[edknv]

@wei-m-teh Apologies for the delay. It's in review at the moment, but I updated #1040 with a workaround I found for making the notebook work with the latest 23.08 image.

[rnyak]

@wei-m-teh can you please test this PR at your end? thanks.