RuntimeError: The size of tensor a (6) must match the size of tensor b (5) at non-singleton dimension 3

[push44]

I am using trying to run following code in AWS Sagemaker Studio Jupyterlab Notebook, but getting RuntimeError: The size of tensor a (6) must match the size of tensor b (5) at non-singleton dimension 3.

Key Details:

1. train_df.shape[0] == train_df[['item_id','timestamp']].drop_duplicates().shape[0] So there are no duplicates that cause this error.
2. When I use this same code on local MacBook Pro code runs without any issue.

Hence, I am not able to debug what is causing this issue and leaning more towards it has nothing to do with train_df. Any help help be greatly appreciated. Thanks!

ts_predictor = TimeSeriesPredictor(
    target='target',
    prediction_length='12',
    eval_metric='WAPE'
)

predictor = ts_predictor.fit(
    train_df,
    hyperparameters={
        "Chronos": [
            # Zero-shot model WITHOUT covariates
            {
                "model_path": "bolt_small",
                "ag_args": {"name_suffix": "ZeroShot"},
            }
        ],
    },
    enable_ensemble=False,
    time_limit=600,
)

[abdulfatir]

@push44 Is this related to this package or AutoGluon? Please provide a complete example to reproduce the issue.

[push44]

@abdulfatir not facing issue while using TabularPredictor. Only facing it while using TimeSeriesPredictor.

Instances:

• Also, want to mention that I tried running on ml.t3.medium, ml.g4dn.xlarge, and ml.g5.2xlarge instances. But facing same error message on all three instances.

Package version:

• Autogluon version is same in both local macbook pro laptop and on Sagemaker 1.4.0

[abdulfatir]

@push44 I won't be able to help without a reproducible example.

[push44]

Hello @abdulfatir please take a look at the following reproducible example. Below example gives me error: RuntimeError: The size of tensor a (6) must match the size of tensor b (5) at non-singleton dimension 3

Thanks in advance for taking time. Really appreciate it.

######################## Import Libraries ########################
import numpy as np
import pandas as pd
from autogluon.timeseries import TimeSeriesDataFrame, TimeSeriesPredictor

######################## Create Temp Dataset ########################
train_df_temp = pd.DataFrame({
    'item_id':['prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1',
       'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1',
       'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1',
       'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1',
       'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1',
       'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1',
       'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1',
       'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-1',
       'prod-1', 'prod-1', 'prod-1', 'prod-1', 'prod-123', 'prod-123',
       'prod-123', 'prod-123', 'prod-123', 'prod-123', 'prod-123',
       'prod-123', 'prod-123', 'prod-123', 'prod-123', 'prod-123',
       'prod-123', 'prod-123', 'prod-123', 'prod-123', 'prod-123',
       'prod-123', 'prod-123', 'prod-123', 'prod-123', 'prod-123',
       'prod-123', 'prod-123', 'prod-123', 'prod-123', 'prod-123',
       'prod-123', 'prod-123', 'prod-123', 'prod-123', 'prod-123',
       'prod-123', 'prod-123', 'prod-123', 'prod-123', 'prod-123',
       'prod-123', 'prod-123', 'prod-123', 'prod-123', 'prod-123',
       'prod-123', 'prod-123', 'prod-123'],
    'timestamp':['2020-09', '2020-10', '2020-11', '2020-12', '2021-01', '2021-02',
       '2021-03', '2021-04', '2021-05', '2021-06', '2021-07', '2021-08',
       '2021-09', '2021-10', '2021-11', '2021-12', '2022-01', '2022-02',
       '2022-03', '2022-04', '2022-05', '2022-06', '2022-07', '2022-08',
       '2022-09', '2022-10', '2022-11', '2022-12', '2023-01', '2023-02',
       '2023-03', '2023-04', '2023-05', '2023-06', '2023-07', '2023-08',
       '2023-09', '2023-10', '2023-11', '2023-12', '2024-01', '2024-02',
       '2024-03', '2024-04', '2024-05', '2024-06', '2024-07', '2024-08',
       '2024-09', '2024-10', '2024-11', '2024-12', '2021-04', '2021-05',
       '2021-06', '2021-07', '2021-08', '2021-09', '2021-10', '2021-11',
       '2021-12', '2022-01', '2022-02', '2022-03', '2022-04', '2022-05',
       '2022-06', '2022-07', '2022-08', '2022-09', '2022-10', '2022-11',
       '2022-12', '2023-01', '2023-02', '2023-03', '2023-04', '2023-05',
       '2023-06', '2023-07', '2023-08', '2023-09', '2023-10', '2023-11',
       '2023-12', '2024-01', '2024-02', '2024-03', '2024-04', '2024-05',
       '2024-06', '2024-07', '2024-08', '2024-09', '2024-10', '2024-11',
       '2024-12'],
    'target':[ 14, 126, 187, 200,  14,  14,  14,  14,  14,  14,  14,  14,  14,
        14,  14,  14,  14,  14,  14,  14,  14,  14,  14,  14,  14,  34,
        42,  18,  20,  15,  16,  14,  14,  14,  14,  14,  14,  14,  15,
        14,  14,  14,  14,  14,  14,  15,  23,  14,  14,  14,  14,  14,
        14, 233, 332, 339, 351, 412, 431, 413, 369, 392, 409, 443, 458,
       484, 481, 524, 513, 490, 520, 505, 440, 484, 469, 485, 489, 518,
       515, 505, 514, 512, 507, 479, 441, 466, 456, 479, 478, 505, 498,
       422, 458, 471, 460, 428, 386]
})

######################## Fit AG TS Predictor ########################

ts_predictor = TimeSeriesPredictor(
    target='target',
    prediction_length='3',
    eval_metric='WAPE',
    freq="MS" # Specify Month Start frequency
)

ts_predictor.fit(
    train_df_temp, presets="bolt_small", 
    hyperparameters={
        "Chronos": [
            {"model_path": "bolt_small", "ag_args": {"name_suffix": "ZeroShot"}},
            {"model_path": "bolt_small", "fine_tune": True, "ag_args": {"name_suffix": "FineTuned"}},
        ]
    },
    skip_model_selection=False,
    time_limit=60,  # time limit in seconds
    enable_ensemble=False,
)

[abdulfatir]

I just tested your snippet and it works for me on Ubuntu and MacOS. Did you check which version of AutoGluon you're using? In general, please open issues related to AutoGluon in the AutoGluon repo.

[push44]

@abdulfatir Yes, I have mentioned this in the description. Code works in local machine but fails on AWS infrastructure ie., EC2 Sagemaker. This does not happen with other packages of Autolguon; I tested on TabularPredictor. Do you still think I should move the issue to Autolguon repo?

[push44]

Reported bug on Autogluon repo. Closing this discussion.
Bug Link: autogluon/autogluon#5395