* fix a bug about the model_avg during finetuning by exchanging the order of loading pre-trained model and initializing avg model
* only match the exact module prefix
* fixes for `diagnostics`
Replace `2 ** 22` with `512` as the default value of `diagnostics.TensorDiagnosticOptions`
also black formatted some scripts
* fixed formatting issues