Integrate MS-AMP Support for FP8 Precision #3617

Job	Run time
run-tests (latest, test_prod)	1m 1s
run-tests (latest, test_core)	1m 14s
run-tests (latest, test_cli)	2m 9s
run-tests (latest, test_big_modeling)	1m 1s
run-tests (latest, test_deepspeed)	54s
run-tests (latest, test_fsdp)	54s
run-tests (latest, test_example_differences)	1m 20s
run-tests (latest, test_checkpoint_step)	5m 12s
run-tests (latest, test_checkpoint_epoch)	4m 44s
run-tests (latest, test_rest)	8m 52s
run-tests (minimum, test_prod)	52s
run-tests (minimum, test_core)	3m 36s
run-tests (minimum, test_cli)	2m 7s
run-tests (minimum, test_big_modeling)	1m 7s
run-tests (minimum, test_deepspeed)	1m 10s
run-tests (minimum, test_fsdp)	1m 6s
run-tests (minimum, test_example_differences)	53s
run-tests (minimum, test_checkpoint_step)	4m 51s
run-tests (minimum, test_checkpoint_epoch)	4m 29s
run-tests (minimum, test_rest)	8m 45s
	56m 17s

Provide feedback