OOM kill during data prep and training

[leifdenby]

I tried running some training jobs last night, but unfortunately my training jobs got OOM killed by slurm :(

Looking at the SLURM "efficiency output":

denlef@login02:~/logs$ seff 33747
Job ID: 33747
Cluster: slurm
User/Group: denlef/cu_0003
State: FAILED (exit code 1)
Nodes: 2
Cores per node: 224
CPU Utilized: 18:51:17
CPU Efficiency: 4.55% of 17-06:01:36 core-walltime
Job Wall-clock time: 00:55:27
Memory Utilized: 568.25 GB (estimated maximum)
Memory Efficiency: 117.32% of 484.38 GB (242.19 GB/node)

I can see that my training job was allocated ~484GB but tried to use 568GB by the time it was killed.

I think we need to increase the amount of memory requested since it appears even with --exclusive to SLURM that not the full 2000GB of the compute node is allocated. I will investigate further and update my issue.

during data prep

I had a similar issue during data prep. There I was able to complete the data-prep for an 8yr dataset with 65GB (not sure if the memory usage scaled with dataset duration, I don't think it should because we're using dask, but I will check)

My experiment setup is on https://github.com/leifdenby/mllam-exps/tree/exp/dataset-duration at commit https://github.com/leifdenby/mllam-exps/tree/ec0439403e6a23dc4e72d81f178ab63d8e1809f0