Derivative of the trace of a Kronecker product
$begingroup$
I am trying to compute the derivative
$frac{partial}{partial W} text{Tr}(W^top A (Iotimes W)B),$
where $Winmathbb{R}^{Dtimes d}, Iinmathbb{R}^{Ttimes T}$ is an identity matrix, $Ainmathbb{R}^{Dtimes DT}$, and $Binmathbb{R}^{dTtimes d}$.
I have found a similar post:
Derivative involving the trace of a Kronecker product
but it seems that the method is not applicable to my problem.
Thank you!
derivatives trace kronecker-product
asked Feb 10 '18 at 3:07
user3138073user3138073
1758
$endgroup$
add a comment |
$begingroup$
I am trying to compute the derivative
$frac{partial}{partial W} text{Tr}(W^top A (Iotimes W)B),$
where $Winmathbb{R}^{Dtimes d}, Iinmathbb{R}^{Ttimes T}$ is an identity matrix, $Ainmathbb{R}^{Dtimes DT}$, and $Binmathbb{R}^{dTtimes d}$.
I have found a similar post:
Derivative involving the trace of a Kronecker product
but it seems that the method is not applicable to my problem.
Thank you!
derivatives trace kronecker-product
asked Feb 10 '18 at 3:07
user3138073user3138073
1758
$endgroup$
add a comment |
$begingroup$
I am trying to compute the derivative
$frac{partial}{partial W} text{Tr}(W^top A (Iotimes W)B),$
where $Winmathbb{R}^{Dtimes d}, Iinmathbb{R}^{Ttimes T}$ is an identity matrix, $Ainmathbb{R}^{Dtimes DT}$, and $Binmathbb{R}^{dTtimes d}$.
I have found a similar post:
Derivative involving the trace of a Kronecker product
but it seems that the method is not applicable to my problem.
Thank you!
derivatives trace kronecker-product
asked Feb 10 '18 at 3:07
user3138073user3138073
1758
$endgroup$
I am trying to compute the derivative
$frac{partial}{partial W} text{Tr}(W^top A (Iotimes W)B),$
where $Winmathbb{R}^{Dtimes d}, Iinmathbb{R}^{Ttimes T}$ is an identity matrix, $Ainmathbb{R}^{Dtimes DT}$, and $Binmathbb{R}^{dTtimes d}$.
I have found a similar post:
Derivative involving the trace of a Kronecker product
but it seems that the method is not applicable to my problem.
Thank you!
derivatives trace kronecker-product
derivatives trace kronecker-product
asked Feb 10 '18 at 3:07
user3138073user3138073
1758
asked Feb 10 '18 at 3:07
user3138073user3138073
1758
asked Feb 10 '18 at 3:07
user3138073user3138073
1758
asked Feb 10 '18 at 3:07
user3138073user3138073
1758
asked Feb 10 '18 at 3:07
user3138073user3138073
1758
1758
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
The technique from the linked post can be applied to the current problem.
Write the function in terms of the trace/Frobenius product, and find its differential
$$eqalign{
phi &= W:A(Iotimes W)B = A^TWB^T:(Iotimes W) cr
dphi &= A(Iotimes W)B:dW + A^TWB^T:(Iotimes dW)
}$$
At this point, we need use the Pitsianis decomposition on that last term.
$$eqalign{
A^TWB^T &= sum_k Y_kotimes Z_k cr
}$$
The matrices $(Y_k,Z_k)$ are shaped like $(I,W)$ respectively.
Finish calculating the differential, then on to the gradient.
$$eqalign{
dphi
&= A(Iotimes W)B:dW + sum_kY_kotimes Z_k:(Iotimes dW) cr
&= Big(A(Iotimes W)B + sum_k(I:Y_k)Z_kBig):dW cr
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k {rm tr}(Y_k),Z_k crcr
}$$
Another technique uses the SVD of
$$B=sum_ksigma_ku_kv_k^T$$
to handle the second term of $dphi$ as follows.
$$eqalign{
A^TW:(Iotimes dW)B
&= sum_k,A^TW:(Iotimes dW)sigma_ku_kv_k^T cr
&= sum_k,(A^TWsigma_kv_k):(Iotimes dW)u_k cr
&= sum_k,q_k:{rm vec}(dW,U_k) cr
&= sum_k,Q_k:dW,U_k cr
&= sum_k,Q_kU_k^T:dW cr
}$$
where
$$eqalign{
{rm vec}(Q_k) &= q_k = A^TWsigma_kv_k cr
{rm vec}(U_k) &= u_k cr
}$$
Yielding the gradient as
$$eqalign{
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k Q_kU_k^T cr
}$$
answered Feb 11 '18 at 16:26
greggreg
8,5931823
$endgroup$
$begingroup$
Thanks a lot for the reply! But one question is: $A^top W B^top$ is also a function of $W$, so we cannot do the decomposition, am I right?
$endgroup$
– user3138073
Feb 12 '18 at 21:30
$begingroup$
The dependence of the function on the $A^TWB^T$ term was accounted for by the first term in the differential expression. Perform the Pitsianis decomposition using the current value of $W$.
$endgroup$
– greg
Feb 13 '18 at 1:15
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2644106%2fderivative-of-the-trace-of-a-kronecker-product%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The technique from the linked post can be applied to the current problem.
Write the function in terms of the trace/Frobenius product, and find its differential
$$eqalign{
phi &= W:A(Iotimes W)B = A^TWB^T:(Iotimes W) cr
dphi &= A(Iotimes W)B:dW + A^TWB^T:(Iotimes dW)
}$$
At this point, we need use the Pitsianis decomposition on that last term.
$$eqalign{
A^TWB^T &= sum_k Y_kotimes Z_k cr
}$$
The matrices $(Y_k,Z_k)$ are shaped like $(I,W)$ respectively.
Finish calculating the differential, then on to the gradient.
$$eqalign{
dphi
&= A(Iotimes W)B:dW + sum_kY_kotimes Z_k:(Iotimes dW) cr
&= Big(A(Iotimes W)B + sum_k(I:Y_k)Z_kBig):dW cr
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k {rm tr}(Y_k),Z_k crcr
}$$
Another technique uses the SVD of
$$B=sum_ksigma_ku_kv_k^T$$
to handle the second term of $dphi$ as follows.
$$eqalign{
A^TW:(Iotimes dW)B
&= sum_k,A^TW:(Iotimes dW)sigma_ku_kv_k^T cr
&= sum_k,(A^TWsigma_kv_k):(Iotimes dW)u_k cr
&= sum_k,q_k:{rm vec}(dW,U_k) cr
&= sum_k,Q_k:dW,U_k cr
&= sum_k,Q_kU_k^T:dW cr
}$$
where
$$eqalign{
{rm vec}(Q_k) &= q_k = A^TWsigma_kv_k cr
{rm vec}(U_k) &= u_k cr
}$$
Yielding the gradient as
$$eqalign{
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k Q_kU_k^T cr
}$$
answered Feb 11 '18 at 16:26
greggreg
8,5931823
$endgroup$
$begingroup$
Thanks a lot for the reply! But one question is: $A^top W B^top$ is also a function of $W$, so we cannot do the decomposition, am I right?
$endgroup$
– user3138073
Feb 12 '18 at 21:30
$begingroup$
The dependence of the function on the $A^TWB^T$ term was accounted for by the first term in the differential expression. Perform the Pitsianis decomposition using the current value of $W$.
$endgroup$
– greg
Feb 13 '18 at 1:15
add a comment |
$begingroup$
The technique from the linked post can be applied to the current problem.
Write the function in terms of the trace/Frobenius product, and find its differential
$$eqalign{
phi &= W:A(Iotimes W)B = A^TWB^T:(Iotimes W) cr
dphi &= A(Iotimes W)B:dW + A^TWB^T:(Iotimes dW)
}$$
At this point, we need use the Pitsianis decomposition on that last term.
$$eqalign{
A^TWB^T &= sum_k Y_kotimes Z_k cr
}$$
The matrices $(Y_k,Z_k)$ are shaped like $(I,W)$ respectively.
Finish calculating the differential, then on to the gradient.
$$eqalign{
dphi
&= A(Iotimes W)B:dW + sum_kY_kotimes Z_k:(Iotimes dW) cr
&= Big(A(Iotimes W)B + sum_k(I:Y_k)Z_kBig):dW cr
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k {rm tr}(Y_k),Z_k crcr
}$$
Another technique uses the SVD of
$$B=sum_ksigma_ku_kv_k^T$$
to handle the second term of $dphi$ as follows.
$$eqalign{
A^TW:(Iotimes dW)B
&= sum_k,A^TW:(Iotimes dW)sigma_ku_kv_k^T cr
&= sum_k,(A^TWsigma_kv_k):(Iotimes dW)u_k cr
&= sum_k,q_k:{rm vec}(dW,U_k) cr
&= sum_k,Q_k:dW,U_k cr
&= sum_k,Q_kU_k^T:dW cr
}$$
where
$$eqalign{
{rm vec}(Q_k) &= q_k = A^TWsigma_kv_k cr
{rm vec}(U_k) &= u_k cr
}$$
Yielding the gradient as
$$eqalign{
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k Q_kU_k^T cr
}$$
answered Feb 11 '18 at 16:26
greggreg
8,5931823
$endgroup$
$begingroup$
Thanks a lot for the reply! But one question is: $A^top W B^top$ is also a function of $W$, so we cannot do the decomposition, am I right?
$endgroup$
– user3138073
Feb 12 '18 at 21:30
$begingroup$
The dependence of the function on the $A^TWB^T$ term was accounted for by the first term in the differential expression. Perform the Pitsianis decomposition using the current value of $W$.
$endgroup$
– greg
Feb 13 '18 at 1:15
add a comment |
$begingroup$
The technique from the linked post can be applied to the current problem.
Write the function in terms of the trace/Frobenius product, and find its differential
$$eqalign{
phi &= W:A(Iotimes W)B = A^TWB^T:(Iotimes W) cr
dphi &= A(Iotimes W)B:dW + A^TWB^T:(Iotimes dW)
}$$
At this point, we need use the Pitsianis decomposition on that last term.
$$eqalign{
A^TWB^T &= sum_k Y_kotimes Z_k cr
}$$
The matrices $(Y_k,Z_k)$ are shaped like $(I,W)$ respectively.
Finish calculating the differential, then on to the gradient.
$$eqalign{
dphi
&= A(Iotimes W)B:dW + sum_kY_kotimes Z_k:(Iotimes dW) cr
&= Big(A(Iotimes W)B + sum_k(I:Y_k)Z_kBig):dW cr
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k {rm tr}(Y_k),Z_k crcr
}$$
Another technique uses the SVD of
$$B=sum_ksigma_ku_kv_k^T$$
to handle the second term of $dphi$ as follows.
$$eqalign{
A^TW:(Iotimes dW)B
&= sum_k,A^TW:(Iotimes dW)sigma_ku_kv_k^T cr
&= sum_k,(A^TWsigma_kv_k):(Iotimes dW)u_k cr
&= sum_k,q_k:{rm vec}(dW,U_k) cr
&= sum_k,Q_k:dW,U_k cr
&= sum_k,Q_kU_k^T:dW cr
}$$
where
$$eqalign{
{rm vec}(Q_k) &= q_k = A^TWsigma_kv_k cr
{rm vec}(U_k) &= u_k cr
}$$
Yielding the gradient as
$$eqalign{
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k Q_kU_k^T cr
}$$
answered Feb 11 '18 at 16:26
greggreg
8,5931823
$endgroup$
The technique from the linked post can be applied to the current problem.
Write the function in terms of the trace/Frobenius product, and find its differential
$$eqalign{
phi &= W:A(Iotimes W)B = A^TWB^T:(Iotimes W) cr
dphi &= A(Iotimes W)B:dW + A^TWB^T:(Iotimes dW)
}$$
At this point, we need use the Pitsianis decomposition on that last term.
$$eqalign{
A^TWB^T &= sum_k Y_kotimes Z_k cr
}$$
The matrices $(Y_k,Z_k)$ are shaped like $(I,W)$ respectively.
Finish calculating the differential, then on to the gradient.
$$eqalign{
dphi
&= A(Iotimes W)B:dW + sum_kY_kotimes Z_k:(Iotimes dW) cr
&= Big(A(Iotimes W)B + sum_k(I:Y_k)Z_kBig):dW cr
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k {rm tr}(Y_k),Z_k crcr
}$$
Another technique uses the SVD of
$$B=sum_ksigma_ku_kv_k^T$$
to handle the second term of $dphi$ as follows.
$$eqalign{
A^TW:(Iotimes dW)B
&= sum_k,A^TW:(Iotimes dW)sigma_ku_kv_k^T cr
&= sum_k,(A^TWsigma_kv_k):(Iotimes dW)u_k cr
&= sum_k,q_k:{rm vec}(dW,U_k) cr
&= sum_k,Q_k:dW,U_k cr
&= sum_k,Q_kU_k^T:dW cr
}$$
where
$$eqalign{
{rm vec}(Q_k) &= q_k = A^TWsigma_kv_k cr
{rm vec}(U_k) &= u_k cr
}$$
Yielding the gradient as
$$eqalign{
frac{partialphi}{partial W} &= A(Iotimes W)B + sum_k Q_kU_k^T cr
}$$
answered Feb 11 '18 at 16:26
greggreg
8,5931823
edited Dec 4 '18 at 0:20
answered Feb 11 '18 at 16:26
greggreg
8,5931823
answered Feb 11 '18 at 16:26
greggreg
8,5931823
answered Feb 11 '18 at 16:26
greggreg
8,5931823
8,5931823
$begingroup$
Thanks a lot for the reply! But one question is: $A^top W B^top$ is also a function of $W$, so we cannot do the decomposition, am I right?
$endgroup$
– user3138073
Feb 12 '18 at 21:30
$begingroup$
The dependence of the function on the $A^TWB^T$ term was accounted for by the first term in the differential expression. Perform the Pitsianis decomposition using the current value of $W$.
$endgroup$
– greg
Feb 13 '18 at 1:15
add a comment |
$begingroup$
Thanks a lot for the reply! But one question is: $A^top W B^top$ is also a function of $W$, so we cannot do the decomposition, am I right?
$endgroup$
– user3138073
Feb 12 '18 at 21:30
$begingroup$
The dependence of the function on the $A^TWB^T$ term was accounted for by the first term in the differential expression. Perform the Pitsianis decomposition using the current value of $W$.
$endgroup$
– greg
Feb 13 '18 at 1:15
$begingroup$
Thanks a lot for the reply! But one question is: $A^top W B^top$ is also a function of $W$, so we cannot do the decomposition, am I right?
$endgroup$
– user3138073
Feb 12 '18 at 21:30
$begingroup$
Thanks a lot for the reply! But one question is: $A^top W B^top$ is also a function of $W$, so we cannot do the decomposition, am I right?
$endgroup$
– user3138073
Feb 12 '18 at 21:30
$begingroup$
The dependence of the function on the $A^TWB^T$ term was accounted for by the first term in the differential expression. Perform the Pitsianis decomposition using the current value of $W$.
$endgroup$
– greg
Feb 13 '18 at 1:15
$begingroup$
The dependence of the function on the $A^TWB^T$ term was accounted for by the first term in the differential expression. Perform the Pitsianis decomposition using the current value of $W$.
$endgroup$
– greg
Feb 13 '18 at 1:15
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2644106%2fderivative-of-the-trace-of-a-kronecker-product%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
