BERT input checkpoint gives errors while execution

[sudhir-mcw]

I am trying to run large language model benchmark (BERT) on a single GPU setup.
I followed the readme and downloaded the preprocessed results, checkpoints from the gdrive using gdown as mentioned in the readme. But during execution i am encountering this error.

I tried with both tf1_ckpt, tf2_ckpt but was not helpful, here is the error message i got

Use tf.data.Dataset.map(map_func, num_parallel_calls) followed by tf.data.Dataset.batch(batch_size, drop_remainder). Static tf.data optimizations will take care of using the fused implementation.
WARNING:tensorflow:From run_pretraining.py:446: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0522 09:01:04.718369 139677768279872 deprecation.py:339] From run_pretraining.py:446: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
INFO:tensorflow:Calling model_fn.
I0522 09:01:04.736635 139677768279872 estimator.py:1162] Calling model_fn.
INFO:tensorflow:*** Features ***
I0522 09:01:04.736754 139677768279872 run_pretraining.py:129] *** Features ***
INFO:tensorflow: name = input_ids, shape = (8, 512)
I0522 09:01:04.736838 139677768279872 run_pretraining.py:131] name = input_ids, shape = (8, 512)
INFO:tensorflow: name = input_mask, shape = (8, 512)
I0522 09:01:04.736907 139677768279872 run_pretraining.py:131] name = input_mask, shape = (8, 512)
INFO:tensorflow: name = masked_lm_ids, shape = (8, 25)
I0522 09:01:04.736973 139677768279872 run_pretraining.py:131] name = masked_lm_ids, shape = (8, 25)
INFO:tensorflow: name = masked_lm_positions, shape = (8, 25)
I0522 09:01:04.737042 139677768279872 run_pretraining.py:131] name = masked_lm_positions, shape = (8, 25)
INFO:tensorflow: name = masked_lm_weights, shape = (8, 25)
I0522 09:01:04.737108 139677768279872 run_pretraining.py:131] name = masked_lm_weights, shape = (8, 25)
INFO:tensorflow: name = next_sentence_labels, shape = (8, 1)
I0522 09:01:04.737170 139677768279872 run_pretraining.py:131] name = next_sentence_labels, shape = (8, 1)
INFO:tensorflow: name = segment_ids, shape = (8, 512)
I0522 09:01:04.737232 139677768279872 run_pretraining.py:131] name = segment_ids, shape = (8, 512)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/legacy_tf_layers/core.py:171: UserWarning: tf.layers.dense is deprecated and will be removed in a future version. Please use tf.keras.layers.Dense instead.
warnings.warn('tf.layers.dense is deprecated and '
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py:1719: UserWarning: layer.apply is deprecated and will be removed in a future version. Please use layer.__call__ method instead.
warnings.warn('layer.apply is deprecated and '
2025-05-22 09:01:06.782143: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open /workspace/chkpt/model.ckpt-28252.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/py_checkpoint_reader.py", line 95, in NewCheckpointReader
return CheckpointReader(compat.as_bytes(filepattern))
RuntimeError: Unable to open table file /workspace/chkpt/model.ckpt-28252.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run_pretraining.py", line 650, in
absl.app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "run_pretraining.py", line 627, in main
steps=FLAGS.max_eval_steps)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 467, in evaluate
name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 510, in _actual_eval
return _evaluate()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 492, in _evaluate
self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1531, in _evaluate_build_graph
self._call_model_fn_eval(input_fn, self.config))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1567, in _call_model_fn_eval
config)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1163, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "run_pretraining.py", line 168, in model_fn
) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint)
File "/workspace/modeling.py", line 329, in get_assignment_map_from_checkpoint
init_vars = tf.train.list_variables(init_checkpoint)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/checkpoint_utils.py", line 112, in list_variables
reader = load_checkpoint(ckpt_dir_or_file)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/checkpoint_utils.py", line 67, in load_checkpoint
return py_checkpoint_reader.NewCheckpointReader(filename)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/py_checkpoint_reader.py", line 99, in NewCheckpointReader
error_translator(e)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/py_checkpoint_reader.py", line 44, in error_translator
raise errors_impl.DataLossError(None, None, error_message)
tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file /workspace/chkpt/model.ckpt-28252.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?

Here is the command i used for testing
python3 run_pretraining.py --bert_config_file=wiki/bert_config.json --output_dir=/tmp/output/ --input_file="tfrecord/part-00010-of-00500" --do_eval --nodo_train --eval_batch_size=8 --init_checkpoint=/workspace/wiki-dataset/wiki_bert/tf2_ckpt/model.ckpt-28252.data-00000-of-00001 --iterations_per_loop=10 --learning_rate=0.0001 --max_eval_steps=100 --max_predictions_per_seq=25 --max_seq_length=512 --num_gpus=1 --num_train_steps=10 --num_warmup_steps=10 --optimizer=lamb --save_checkpoints_steps=5 --start_warmup_step=0 --train_batch_size=8 --nouse_tpu

Queries

1. Is there any automation setup which can do the setup and run training, like mlcr or cmx4mlops for inference
2. Or can you direct me to any resource/setup instructions for executing training benchmarks

[ShriyaRishab]

Sorry for issues you might have faced with bert, unfortunately this is a very old benchmark and we retired it from Training as well. I recommend trying one of the newer benchmarks and following instructions in the README. We are actively working on llama3.1 8b which is a small LLM pretraining benchmark that should run on a single node, here's the PR for it #799. If you have issues reproducing this PR, just leave comments on the issue and we can help you out.

Also for training, we do not have automation setup so we recommend following the README to reproduce references.