Chapter 9 has some cool tools on interpreting LLMs, I had used SHAP before but note BertViz https://github.com/jessevig/bertviz

I tried Bertviz once for a machine translation project and it was great to use though in the end it wasn't terribly useful for interpretability in terms of text alignment because of the many attention layers and heads in most models