Better guards for slow imports #3263

Job	Run time
run-tests (latest, test_prod)	3m 14s
run-tests (latest, test_core)	5m 12s
run-tests (latest, test_cli)	6m 6s
run-tests (latest, test_big_modeling)	4m 19s
run-tests (latest, test_deepspeed)	4m 35s
run-tests (latest, test_fsdp)	5m 3s
run-tests (latest, test_example_differences)	4m 6s
run-tests (latest, test_checkpoint_step)	6m 11s
run-tests (latest, test_checkpoint_epoch)	5m 56s
run-tests (latest, test_rest)	10m 3s
run-tests (minimum, test_prod)	2m 58s
run-tests (minimum, test_core)	5m 13s
run-tests (minimum, test_cli)	7m 1s
run-tests (minimum, test_big_modeling)	5m 52s
run-tests (minimum, test_deepspeed)	4m 52s
run-tests (minimum, test_fsdp)	4m 48s
run-tests (minimum, test_example_differences)	4m 6s
run-tests (minimum, test_checkpoint_step)	6m 3s
run-tests (minimum, test_checkpoint_epoch)	6m 3s
run-tests (minimum, test_rest)	8m 39s
	1h 50m 20s

Provide feedback