Let $$A = \begin{bmatrix} \frac{1}{\sqrt{10}} & \frac{3}{\sqrt{10}} \\ \frac{-3}{\sqrt{10}} & \frac{1}{\sqrt{10}} \end{bmatrix}$$ and $$B = \begin{bmatrix} 1 & -i \\ 0 & 1 \end{bmatrix}$$, where $$i = \sqrt{-1}$$. If $$M = A^T BA$$, then the inverse of the matrix $$AM^{2023}A^T$$ is

We are given: $$A = \begin{bmatrix} \frac{1}{\sqrt{10}} & \frac{3}{\sqrt{10}} \\ \frac{-3}{\sqrt{10}} & \frac{1}{\sqrt{10}} \end{bmatrix}$$ and $$B = \begin{bmatrix} 1 & -i \\ 0 & 1 \end{bmatrix}$$, where $$i = \sqrt{-1}$$. Also $$M = A^T B A$$ and we need to find the inverse of $$AM^{2023}A^T$$.

First we verify that $$A$$ is an orthogonal matrix by computing its transpose and multiplying: $$A^T = \begin{bmatrix} \frac{1}{\sqrt{10}} & \frac{-3}{\sqrt{10}} \\ \frac{3}{\sqrt{10}} & \frac{1}{\sqrt{10}} \end{bmatrix}$$ and then $$A^T A = \begin{bmatrix} \frac{1}{10} + \frac{9}{10} & \frac{3}{10} - \frac{3}{10} \\ \frac{3}{10} - \frac{3}{10} & \frac{9}{10} + \frac{1}{10} \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = I$$. Hence $$A^T = A^{-1}$$.

Next, since $$M = A^T B A$$, we have $$M^2 = (A^T B A)(A^T B A) = A^T B (A A^T) B A = A^T B^2 A$$. By induction it follows that $$M^n = A^T B^n A$$ for any positive integer $$n$$, and in particular $$M^{2023} = A^T B^{2023} A$$.

Multiplying by $$A$$ on the left and by $$A^T$$ on the right gives $$AM^{2023}A^T = A (A^T B^{2023} A) A^T = (A A^T) B^{2023} (A A^T) = I \, B^{2023} \, I = B^{2023}$$.

To find $$B^{2023}$$, note that $$B = \begin{bmatrix} 1 & -i \\ 0 & 1 \end{bmatrix}$$ is upper triangular with 1s on the diagonal. The power of such a matrix is $$B^n = \begin{bmatrix} 1 & -ni \\ 0 & 1 \end{bmatrix}$$, which can be checked by computing $$B^2 = \begin{bmatrix} 1 & -i \\ 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & -i \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 & -2i \\ 0 & 1 \end{bmatrix}$$ and proceeding by induction. Thus $$B^{2023} = \begin{bmatrix} 1 & -2023i \\ 0 & 1 \end{bmatrix}$$.

Finally, the inverse of a $$2\times2$$ upper triangular matrix $$\begin{bmatrix} 1 & a \\ 0 & 1 \end{bmatrix}$$ is $$\begin{bmatrix} 1 & -a \\ 0 & 1 \end{bmatrix}$$ because $$\begin{bmatrix} 1 & a \\ 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & -a \\ 0 & 1 \end{bmatrix} = I$$. Here $$a = -2023i$$, so the inverse of $$B^{2023}$$ is $$\begin{bmatrix} 1 & 2023i \\ 0 & 1 \end{bmatrix}$$. Therefore, $$(AM^{2023}A^T)^{-1} = \begin{bmatrix} 1 & 2023i \\ 0 & 1 \end{bmatrix}$$.