Also replace Adam with SGD.
You can run the recipe with CPU.
The above Colab notebook finishes the training using CPU within two minutes (50 epochs in total).
The WER is
[test_set] %WER 0.42% [1 / 240, 0 ins, 1 del, 0 sub ]