# Back Prop Algorithm - What remains constant in derivatives

The following are a set of notes on the back-prop algorithm. I happen to work in dynamical systems where one needs to compute the stability of periodic orbits. One could argue a dynamical system is essentially a rule for iteration such as the one below.

$x_{n+1} = f(x_n)$

Interestingly, there are sytems which come back to the same point after some iterations. Lets say the following is true,

$x_{n+4} = x_{n} = f(f(f(f(x_n)))) = f^4(x_n)$

One computes the stability of a discrete system as above by computing its derivative. If the derivative is less than 1, then the system is stable else unstable.

So to compute the stability of the $$f^4(.)$$ function one needs to find the following:

$\frac{d}{dx_n}(f^4(x_n)) = \frac{d}{dx_n}(x_{n+4}) = \frac{dx_{n+1}}{dx_n}.\frac{dx_{n+2}}{dx_{n+1}}.\frac{dx_{n+3}}{dx_{n+2}}.\frac{dx_{n+4}}{dx_{n+3}}$

This is essentially the backpropagation update for the function $$f^4(.)$$ with respect to the previous layer outputs $$x_n,x_{n+1}..$$. While this is obvious for many, how this helps me is to understand what is kept constant when one takes the layer’s derivative.

That is as follows,

$\frac{d}{dx_n}(x_{n+4}) = \frac{dx_{n+1}}{dx_n}.\frac{dx_{n+2}}{dx_{n+1}}.\frac{dx_{n+3}}{dx_{n+2}}.\frac{dx_{n+4}}{dx_{n+3}} = f'(x_n).f'(x_{n+1}).f'(x_{n+2}).f'(x_{n+3})$

Therefore, instead of taking the derivative of the overall function $$f^4(.)$$, we could take the derivative of the function $$f(.)$$ and just multiply.

Consequently, in general, one can just use $$f'(.)$$ and apply the function on the $$n$$ iterates of the function and get the derivative $$n^{th}$$ application of the function. The interesting aspect is what is kept constant when one applies the function is clear here. Cheers.

Reference: Ott, E. (2002). Chaos in Dynamical Systems (2nd ed.). Cambridge: Cambridge University Press. doi:10.1017/CBO9780511803260