3.10 Linearization and Multivariate Taylor Series

The gradient $\triangledown f$ of a function $f$ is often used for a locally linear approximation of $f$ around $x_0$.

$f(x) \approx f(x_0)+(\bigtriangledown_xf)(x_0)(x-x_0)$

Here $(\bigtriangledown_xf)(x_0)$ is the gradient of $f$ with respect to $x$, evaluated at $x_0$.

The following Figure illustrates the linear approximation of a function $f$ at an input $x_0$. The original function is approximated by a straight line.This approximation is locally accurate, but the farther we move away from $x_0$, the worse the approximation gets.This is a special case of multivariate Taylor series expansion of $f$ at $x_0$, where we consider only the first two terms.

Multivariate Taylor Series: Definition

We consider a function

$f:\mathbb{R}^D \to \mathbb{R}$

$x \to f(x), \quad x \in \mathbb{R}^D$

that is smooth at $x_0$. When we define the difference vector $\delta= x- x_0$, the multivariate Taylor series of $f$ at $x_0$ is defined as

$f(x)=\sum_{k=0}^{\infty } \frac{D_x^kf(x_0)}{k!} \delta^k$

where $D^k_xf(x_0)$ is the $k-th$ derivative of $f$ with respect to $x$, evaluated at $x_0$.

Taylor Polynomial: Definition

The Taylor polynomial of degree $n$ of $f$ at $x_0$ contains the first $n+1$ components of the Taylor series and is defined as

$T_n(x)=\sum_{k=0}^{n } \frac{D_x^kf(x_0)}{k!} \delta^k$

Note that both $D_x^kf$ and $\delta^k$ are $k-th$ order tensors.i.e., $k$ dimensional arrays. The $k$th order tensor $\delta^k \in \mathbb{R}^{D \times D \times \ldots \times D}$ is obtained as a $k$ fold outer product denoted by $\bigotimes$, of the vector $\delta \in \mathbb{R}^D$. For example

$\delta^2=\delta \bigotimes \delta=\delta \delta^T,\quad \delta^2[i,j]=\delta[i][\delta[j]$

In general we obtain the terms

$D_x^kf(x_0)\delta^k=\sum_{i_1=1}^D\cdots\sum_{i_k=1}^DD_x^kf(x_0)[i_1,\ldots,i_k]\delta[i_1]\cdots\delta[i_k]$

Lets write down the first few terms of $D_x^kf(x_0)\delta^k$ of the Taylor Series expansion for $k=0,\ldots,3$ and $\delta=x-x_0$.

Example: ( university question)

Consider the function $f(x,y)=x^2+2xy+y^3$. Find the Taylor series expansion of  $f$ at $(x_0,y_0)=(1,2)$

Note that we are looking for a Taylor series expansion, which itself is a linear combination of Polynomials.To express the polynomial of degree 3 , the Taylor series expansion must contain terms of 4th or higher order.

$f(1,2)=13$

$\frac{\partial f}{\partial x}=2x+2y=\frac{\partial f}{\partial x}(1,2)=6$

$\frac{\partial f}{\partial y}=2x+3y^2=\frac{\partial f}{\partial y}(1,2)=14$

Therefore we obtain

$D_{x,y}^1f(1,2)=\bigtriangledown_{x,y}f(1,2)=\left[ \frac{\partial f}{\partial x} \quad \frac{\partial f}{\partial y}\right]=[6 \quad 14] \in \mathbb{R}^{1 \times 2}$

$\frac{D_{x,y}^1f(1,2)}{1!}\delta=\left[6 \quad 14 \right]\begin{bmatrix}
x-1\\
y-2\end{bmatrix}=6(x-1)+14(y-2)$

Note that this contains only linear terms.i.e., first-order polynomials.

The second order partial derivatives are given by

 $\frac{\partial^2 f}{\partial x^2}=2 \Rightarrow \frac{\partial^2 f}{\partial x^2}(1,2)=2$

 $\frac{\partial^2 f}{\partial y^2}=6y \Rightarrow \frac{\partial^2 f}{\partial y^2}(1,2)=12$

 $\frac{\partial^2 f}{\partial y \partial x}=2 \Rightarrow \frac{\partial^2 f}{\partial y \partial x}(1,2)=2$

$\frac{\partial^2 f}{\partial x \partial y}=2 \Rightarrow \frac{\partial^2 f}{\partial x \partial y}(1,2)=2$

$H=\begin{bmatrix}
2 & 2\\
2& 6y
\end{bmatrix} \in \mathbb{R}^{2 \times 2}$

So the Hessian at $(1,2)$ is

$H(1,2)=\begin{bmatrix}
2 & 2\\
2& 12
\end{bmatrix} \in \mathbb{R}^{2 \times 2}$

Therefore the next term of the Taylor series expansion is given by

$\frac{D_{x,y}^2f(1,2)}{2!}\delta^2=\frac{1}{2}\delta^TH(1,2)\delta$

$\quad=\frac{1}{2}\begin{bmatrix}
x-1 & y-2
\end{bmatrix}\begin{bmatrix}
2 & 2\\
2& 12
\end{bmatrix} \begin{bmatrix}
x-1\\
y-2
\end{bmatrix}$

$=(x-1)^2+2(x-1)(y-2)+6(y-2)^2$

This is a second order polynomial

The third order derivatives are obtained as

$D^3_{x,y}f=\left[ \frac{\partial H}{\partial x}\quad \frac{\partial H}{\partial y}\right] \in \mathbb{R}^{2 \times 2 \times 2}$

$D^3_{x,y}f[:,:,1]=\frac{\partial H}{\partial x}=\begin{bmatrix}
\frac{\partial ^3f}{\partial x^3} & \frac{\partial ^3f}{\partial x^2 \partial y}\\
\frac{\partial ^3f}{\partial x \partial y \partial x}& \frac{\partial ^3f}{\partial x\partial y^2}
\end{bmatrix}$


$D^3_{x,y}f[:,:,2]=\frac{\partial H}{\partial y}=\begin{bmatrix}
\frac{\partial ^3f}{\partial y \partial x^2} & \frac{\partial ^3f}{\partial y \partial x \partial y}\\
\frac{\partial ^3f}{\partial y^2 \partial x}& \frac{\partial ^3f}{\partial y^3}
\end{bmatrix}$

Since most of the terms are constant in second order Hessian, the only non zero third order partial derivative  is

$\frac{\partial ^3f}{\partial y^3} =6 \Rightarrow \frac{\partial ^3f}{\partial y^3}(1,2)=6$

All other derivatives will vanish

$D^3_{x,y}f[:,:,1]=\begin{bmatrix}0 & 0\\
0& 0
\end{bmatrix}$
$D^3_{x,y}f[:,:,2]=\begin{bmatrix}0 & 0\\
0& 6
\end{bmatrix}$

$\frac{D^3_{x,y}f(1,2)}{3!}\delta^3=(y-2)^3$

Overall, the exact Taylor series expansion of $f$ at $(x_0,y_0)=(1,2)$ is

$f(x,y)=f(1,2)+D^1_{x,y}f(1,2)\delta+\frac{D^2_{x,y}f(1,2)}{2!}\delta^2+\frac{D^3_{x,y}f(1,2)}{3!}\delta^3$

$=13+6(x-1)+14(y-2)+(x-1)^2+2(x-1)(y-2)+6(y-2)^2+(y-2)^3 $

$=13+6x-6+14y-28+x^2+1-2x+2xy-4x-2y+4+6y^2+24-24y+y^3-8+12y-6y^2 $

$=x^2+2xy+y^3+(6x-6x)+(14y-2y-24y+12y)+(13-6-28+1+4+24-8)$

$=x^2+2xy+y^3$

Find the second order Taylor series expansion for $f(x,y) = e^{-(x^2+y^2)} cos(xy)$ about $(0 , 0)$.

$f(0,0)=1$

First order partial derivatives

$\frac{\partial f}{\partial x}=-2xe^{-(x^2+y^2)}cos(xy)-ye^{-(x^2+y^2)}sin(xy)$

$\frac{\partial f}{\partial x}=-e^{-(x^2+y^2)}(2x.cos(xy)+y.sin(xy))$

$\frac{\partial f}{\partial x}(0,0)=0$

$\frac{\partial f}{\partial y}=-2ye^{-(x^2+y^2)}cos(xy)-xe^{-(x^2+y^2)}sin(xy)$

$\frac{\partial f}{\partial y}=-e^{-(x^2+y^2)}(2y.cos(xy)+x.sin(xy))$

$\frac{\partial f}{\partial y}(0,0)=0$

$D_{x,y}^1f(0,0)=\bigtriangledown_{x,y}f(0,0)=\left[ \frac{\partial f}{\partial x} \quad \frac{\partial f}{\partial y}\right]=[0 \quad 0] \in \mathbb{R}^{1 \times 2}$

$\frac{D_{x,y}^1f(0,0)}{1!}\delta=\left[0 \quad 0 \right]\begin{bmatrix}
x\\
y\end{bmatrix}=0$

The second order partial derivatives are given by
$\frac{\partial^2 f}{\partial x^2}=e^{-(x^2+y^2)}.2x(2x.cos(xy)+y.sin(xy))-e^{(x^2+y^2)}(2.cos(xy)-2xy.sin(xy)+y^2.cos(xy))$

$\frac{\partial^2 f}{\partial x^2}=e^{-(x^2+y^2)}((4x^2-y^2-2)cos(xy)+4xy.sin(xy))$

$\frac{\partial^2 f}{\partial x^2}(0,0)=-2$

similarly

$\frac{\partial^2 f}{\partial y^2}=e^{-(x^2+y^2)}((4xy.sin(xy)-(x^2-4y^2+2)cos(xy))$

$\frac{\partial^2 f}{\partial y^2}(0,0)=-2$

$\frac{\partial^2 f}{\partial x \partial y}=\frac{\partial^2 f}{\partial y \partial x}=e^{-(x^2+y^2)}((2x^2+2y^2-1)sin(xy)+3xy.cos(xy))$

$\frac{\partial^2 f}{\partial x \partial y}=\frac{\partial^2 f}{\partial y \partial x}(0,0)=0$

$\frac{D_{x,y}^2f(0,0)}{2!}\delta^2=\frac{1}{2}\left[x \quad y \right]\begin{bmatrix}
-2 & 0\\
0 & -2
\end{bmatrix}\begin{bmatrix}x\\
y\end{bmatrix}=-x^2-y^2$

the second degree Taylor polynomial at point (0,0) is

$f(0,0)+\frac{D_{x,y}^1f(0,0)}{1!}\delta+\frac{D_{x,y}^2f(0,0)}{2!}\delta^2$

$=1+0-x^2-y^2$

$=1-x^2-y^2$

Find the linear approximation to the function $f(x,y) = 2 - sin(-x -3y)$ at the point (0 , π), and then use your answer to estimate $f(0.001 , π)$.

$f(0,\pi)=2-sin(0-3\pi)=2+sin(3\pi)=2$

First order partial derivatives

$\frac{\partial f}{\partial x}=cos(-x-3y)$

$\frac{\partial f}{\partial x}(0,\pi)=cos(-3\pi)=-1$

$\frac{\partial f}{\partial y}=3cos(-x-3y)$

$\frac{\partial f}{\partial y}(0,\pi)=3.cos(-3\pi)=-3$

$\frac{D_{x,y}^1f(0,0)}{1!}\delta=\left[-1\quad -3 \right]\begin{bmatrix}
x\\
y-\pi\end{bmatrix}=-x-3y+3\pi$

Linear approximation at $(0,\pi)$ is
$f(0,\pi)+\frac{D_{x,y}^1f(0,0)}{1!}\delta$

$=2-x-3y+3\pi$

value at $f(0.001 , π)$ is

$2-0.001-3\pi+3\pi$

$=1.999$

Find the second order Taylor series expansion for $f(x, y) = (x + y)^2$ about $(0 , 0)$

$f(0,0)=0$

First order partial derivatives

$\frac{\partial f}{\partial x}=2(x+y)$

$\frac{\partial f}{\partial x}(0,0)=0$

$\frac{\partial f}{\partial y}=2(x+y)$

$\frac{\partial f}{\partial y}(0,0)=0$

$\frac{D_{x,y}^1f(0,0)}{1!}\delta=\left[0 \quad 0 \right]
\begin{bmatrix}
x\\
y
\end{bmatrix}=0$

Second order partial derivatives

$\frac{\partial^2 f}{\partial x^2}=2$

$\frac{\partial^2 f}{\partial x^2}(0,0)=2$

$\frac{\partial^2 f}{\partial y^2}=2$

$\frac{\partial^2 f}{\partial y^2}(0,0)=2$

$\frac{\partial^2 f}{\partial x \partial y}=2$

$\frac{\partial^2 f}{\partial x \partial y}(0,0)=2$

$\frac{\partial^2 f}{\partial y \partial x}=2$

$\frac{\partial^2 f}{\partial y\partial x}(0,0)=2$

$\frac{D_{x,y}^2f(0,0)}{2!}\delta^2=\frac{1}{2}\left[x \quad y \right]\begin{bmatrix}
2 & 2\\
2 & 2
\end{bmatrix}\begin{bmatrix}x\\
y\end{bmatrix}=x^2+2xy+y^2$

the second order Taylor series expansion about(0,0) is

$f(0,0)+\frac{D_{x,y}^1f(0,0)}{1!}\delta+\frac{D_{x,y}^2f(0,0)}{2!}\delta^2$

$=0+0+x^2+2xy+y^2$

$=x^2+2xy+y^2$

Compute the Taylor series expansion of $f(x,y)=x^2+2xy+y^3$ at $(x_0,y_0)=(1,1)$

$f(1,1)=4$

First order partial derivatives

$\frac{\partial f}{\partial x}=2x+2y$

$\frac{\partial f}{\partial x}(1,1)=4$

$\frac{\partial f}{\partial y}=2x+3y^2$

$\frac{\partial f}{\partial y}(1,1)=5$

$\frac{D_{x,y}^1f(1,1)}{1!}\delta=\left[4 \quad 5 \right]
\begin{bmatrix}
x-1\\
y-1
\end{bmatrix}=4x+5y-9$

Second order partial derivatives

$\frac{\partial^2 f}{\partial x^2}=2$

$\frac{\partial^2 f}{\partial x^2}(1,1)=2$

$\frac{\partial^2 f}{\partial y^2}=6y$

$\frac{\partial^2 f}{\partial y^2}(1,1)=6$

$\frac{\partial^2 f}{\partial x \partial y}=2$

$\frac{\partial^2 f}{\partial x \partial y}(1,1)=2$

$\frac{\partial^2 f}{\partial y \partial x}=2$

$\frac{\partial^2 f}{\partial y\partial x}(1,1)=2$

$\frac{D_{x,y}^2f(1,1)}{2!}\delta^2=\frac{1}{2}\left[x-1 \quad y-1 \right]\begin{bmatrix}
2 & 2\\
2 & 6
\end{bmatrix}
\begin{bmatrix}
x-1\\
y-1\end{bmatrix}=x^2+3y^2-4x-8y+2xy+6$

the second order Taylor series expansion about(1,1) is

$f(1,1)+\frac{D_{x,y}^1f(1,1)}{1!}\delta+\frac{D_{x,y}^2f(1,1)}{2!}\delta^2$

$=4+(4x+5y-9)+x^2+3y^2-4x-8y+2xy+6$

$=x^2+3y^2-3y+2xy+1$

Compute the Taylor series expansion of $f(x,y)=x^2y+3y-2$ at $(x_0,y_0)=(1,-2)$

$f(x,y)=x^2y+3y-2$

$f(1,-2)=-10$

First order partial derivatives

$\frac{\partial f}{\partial x}=2xy$

$\frac{\partial f}{\partial x}(1,-2)=-4$

$\frac{\partial f}{\partial y}=x^2+3$

$\frac{\partial f}{\partial y}(1,-2)=4$

$\frac{D_{x,y}^1f(1,1)}{1!}\delta=\left[-4 \quad 4 \right]
\begin{bmatrix}
x-1\\
y+2
\end{bmatrix}=-4x+4y+12$

Second order partial derivatives

$\frac{\partial^2 f}{\partial x^2}=2y$

$\frac{\partial^2 f}{\partial x^2}(1,-2)=-4$

$\frac{\partial^2 f}{\partial y^2}=0$

$\frac{\partial^2 f}{\partial y^2}(1,-2)=0$

$\frac{\partial^2 f}{\partial x \partial y}=2x$

$\frac{\partial^2 f}{\partial x \partial y}(1,-2)=2$

$\frac{\partial^2 f}{\partial y \partial x}=2x$

$\frac{\partial^2 f}{\partial y\partial x}(1,-2)=2$

$\frac{D_{x,y}^2f(1,-2)}{2!}\delta^2=\frac{1}{2}\left[x-1 \quad y+2 \right]\begin{bmatrix}
-4 & 2\\
2 & 0
\end{bmatrix}
\begin{bmatrix}
x-1\\
y+2\end{bmatrix}=-2x^2+12x-3y+4xy-8$

the second order Taylor series expansion about(1,-2) is

$f(1,-2)+\frac{D_{x,y}^1f(1,-2)}{1!}\delta+\frac{D_{x,y}^2f(1,-2)}{2!}\delta^2$

$=-10+(-4x+4y+12)-2x^2+12x-3y+4xy-8$

$=-2x^2+8x+y-6$