Cfrgtkky

I am looking for general solution of the form $B = XAX^{-1}$
with the following constraints

$X^{-1}=X'$

where $X$ is unknown matrix and $A,,B,, X$, are $3times 3$ matrices.

The problem appears peculiar and thus I wish to simplify it further to more practical problem
X is rotation matrix ($R$), and $A$ is plane equation matrix, B is resulted homography in the sense of
$B =XAX^{-1} = XAX^{T}= R(I-tn^T/d)R^{T}$ ,

where $t$ is translation vector, $n$ is plane normal vector, and $d$ is signed distance, all three are known

My dirty solution so far is as follows :
Finding the Jordan Canonical form:

$MBM^{-1}=J=NAN^{-1}$,

and then $X$ is obviously

$X = N^{-1}M$

but this works only if the canonical forms are the same, otherwise cannot be utilized for solution.

The second dirty trick is to solve a system of Linear equations
in terms of

$AX = BX$

but i cannot figure out how to program this in the software (e.g. Matlab linsolve).
Could you help me to clarify how to write system of linear equations
in the shape of 9 rows?

The trick is how to notate them on the "right side" to use in ready to use functions for solving SLE.

Thank, you

Case 1. $A,B$ are not orthogonally similar; no solutions.

Case 2. $A,B$ are orthogonally similar; $A=PBP^T$, where $Pin O(n)$. Let $C(A)={Xin M_n(mathbb{R});AX=XA}$. Then, it is not difficult to show

$textbf{Proposition}$ ${Xin O(n);A=XBX^T}=(C(A)cap O(n))P$.

It remains to obtain $C(A)cap O(n)$. When $n=3$, $C(A)$ is a vector space of dimension $3,5$ or $9$ (when $A$ is a scalar matrix).

Generically (randomly choose $A$), $A$ is cyclic

https://en.wikipedia.org/wiki/Cyclic_subspace

and $C(A)={aI+bA+cA^2;a,b,cin mathbb{R}}$. Generically, the eigenspaces of $A$ are not orthogonal and, consequently, $C(A)cap O(n)=pm I_n$ and $X=pm P$.

EDIT. $A,B$ are orthogonally similar iff the couples $(A,A^T)$ and $(B,B^T)$ are similar. Then, practically, one studies the linear system

$(S)$ $AX=XB,A^TX=XB^T$.

If $A,B$ are not orthog. similar, then the sole solution is $X=0$.

If $A,B$ are orthog. similar, then, for a generic $A$, $(S)$ admits a vector space of solutions of dimension $1$: $X=uX_0$. It remains to calculate $u$ so that $||uX_0||_2=1$.