Optimization and optimal control

These notes were originally written by T. Bretl and were transcribed by S. Bout.

• Optimization
  • Minimum vs. Minimizer
• Optimal Control
  • Statement of the problem
  • The HJB equation (our new “first-derivative test”)
  • Solution approach
• LQR
  • Statement of the problem
  • Do the matrices $Q$, $R$, and $M$ really have to be symmetric?
  • Solution to the problem
  • LQR Code

Optimization

The following thing is called an optimization problem:

$$\begin{array}{r}\underset{u}{\mathrm{minimize}}{u}^2-2u+3.\end{array}$$

The solution to this problem is the value of $u$ that makes $u^2-2u+3$ as small as possible.

• We know that we are supposed to choose a value of $u$ because “$u$” appears underneath the “minimize” statement. We call $u$ the decision variable.

• We know that we are supposed to minimize $u^2-2u+3$ because “$u^2-2u+3$” appears to the right of the “minimize” statement. We call $u^2-2u+3$ the cost function.

In particular, the solution to this problem is $u=1$. There are at least two different ways to arrive at this result:

• We could plot the cost function. It is clear from the plot that the minimum is at $u=1$.

  

• We could apply the first derivative test. We compute the first derivative:

  $$\begin{array}{r}\frac{d}{du}(u^2-2u+3)=2u-2.\end{array}$$

  Then, we set the first derivative equal to zero and solve for $u$:

  $$\begin{array}{r}2u-2=0\Rightarrow u=1.\end{array}$$

  Values of $u$ that satisfy the first derivative test are only “candidates” for optimality — they could be maxima instead of minima, or could be only one of many minima. We’ll ignore this distinction for now. Here’s a plot of the cost function and of it’s derivative. Note that, clearly, the derivative is equal to zero when the cost function is minimized:

  

In general, we write optimization problems like this:

$$\begin{array}{r}\underset{u}{\mathrm{minimize}}g(u).\end{array}$$

Again, $u$ is the decision variable and $g(u)$ is the cost function. In the previous example:

$$\begin{array}{r}g(u)=u^2-2u+3.\end{array}$$

Here is another example:

$$\begin{array}{r}\underset{u_1,u_2}{\mathrm{minimize}}{u}_1^2+3u_2^2-2u_1{u}_2+2u_1+2u_2+6.\end{array}$$

The solution to this problem is the value of both $u_1$ and $u_2$ that, together, make

$$u_1^2+3u_2^2-2u_1{u}_2+2u_1+2u_2+6$$

as small as possible. There are two differences between this optimization problem and the previous one. First, there is a different cost function:

$$\begin{array}{r}g(u_1,u_2)=u_1^2+3u_2^2-2u_1{u}_2+2u_1+2u_2+6.\end{array}$$

Second, there are two decision variables instead of one. But again, there are at least two ways of finding the solution to this problem:

• We could plot the cost function. The plot is now 3D — the “x” and “y” axes are $u_1$ and $u_2$, and the “z” axis is $g(u_1,u_2)$. The shape of the plot is a bowl. It’s hard to tell where the minimum is from looking at the bowl, so I’ve also plotted contours of the cost function underneath. “Contours” are like the lines on a topographic map. From the contours, it looks like the minimum is at $(u_1,u_2)=(-2,-1)$.

  

• We could apply the first derivative test. We compute the partial derivative of $g(u_1,u_2)$ with respect to both $u_1$ and $u_2$:

  $$\begin{array}{rl}\frac{\partial }{\partial u_1}g(u_1,u_2)& =2u_1-2u_2+2\\ \frac{\partial }{\partial u_2}g(u_1,u_2)& =6u_2-2u_1+2.\end{array}$$

  Then, we set both partial derivatives equal to zero and solve for $u_1$ and $u_2$:

  $$\begin{array}{r}\begin{array}{rl}2u_1-2u_2+2& =0\\ 6u_2-2u_1+2& =0\end{array}\Rightarrow (u_1,u_2)=(-2,-1).\end{array}$$

  As before, we would have to apply a further test in order to verify that this choice of $(u_1,u_2)$ is actually a minimum. But it is certainly consistent with what we observed above. Here is a plot of each partial derivative as a function of $u_1$ and $u_2$. The shape of each plot is a plane (i.e., a flat surface). Both planes are zero at $(-2,-1)$:

  

An equivalent way of stating this same optimization problem would have been as follows:

$$\begin{array}{r}\underset{u_1,u_2}{\mathrm{minimize}}{\left[\begin{array}{c}{u}_1\\ u_2\end{array}\right]}^T\left[\begin{array}{cc}1& -1\\ -1& 3\end{array}\right]\left[\begin{array}{c}{u}_1\\ u_2\end{array}\right]+{\left[\begin{array}{c}2\\ 2\end{array}\right]}^T\left[\begin{array}{c}{u}_1\\ u_2\end{array}\right]+6.\end{array}$$

You can check that the cost function shown above is the same as the cost function we saw before (e.g., by multiplying it out). We could have gone farther and stated the problem as follows:

$$\begin{array}{r}\underset{u}{\mathrm{minimize}}{u}^T\left[\begin{array}{cc}1& -1\\ -1& 3\end{array}\right]u+{\left[\begin{array}{c}2\\ 2\end{array}\right]}^{T}u+6.\end{array}$$

We have returned to having just one decision variable $u$, as in the first example, but this variable is now a $2\times 1$ matrix — i.e., it has two elements, which we would normally write as $u_1$ and $u_2$. The point here is that the “decision variable” in an optimization problem can be a variety of different things: a scalar, a vector (i.e., an $n\times 1$ matrix), and — as we will see — even a function of time. Before proceeding, however, let’s look at one more example of an optimization problem:

$$\begin{array}{rl}\underset{u,x}{\mathrm{minimize}}& u^2+3x^2-2ux+2u+2x+6\\ \text{subject to}& u+x=3.\end{array}$$

This example is exactly the same as the previous example, except that the two decision variables (now renamed $u$ and $x$) are subject to a constraint:

$$u+x=3.$$

We are no longer free to choose $u$ and $x$ arbitrarily. We are restricted to choices that satisfy the constraint. The solution to this optimization problem is the value $(u,x)$ that minimizes the cost function, chosen from among all values $(u,x)$ that satisfy the constraint. Again, there are a variety of ways to solve this problem. One way is to eliminate the constraint. First, we solve the constraint equation:

$$\begin{array}{r}u+x=3\Rightarrow x=3-u.\end{array}$$

Then, we plug this result into the cost function:

$$\begin{array}{rl}{u}^2+3x^2-2ux+2u+2x+6& =u^2+3(3-u{)}^2-2u(3-u)+2u+2(3-u)+6\\ & =6u^2-24u+39.\end{array}$$

By doing so, we have shown that solving the constrained optimization problem

$$\begin{array}{rl}\underset{u,x}{\mathrm{minimize}}& u^2+3x^2-2ux+2u+2x+6\\ \text{subject to}& u+x=3.\end{array}$$

is equivalent to solving the unconstrained optimization problem

$$\begin{array}{r}\underset{u}{\mathrm{minimize}}6u^2-24u+39.\end{array}$$

and then taking $x=3-u$. We can do so easily by taking the first derivative and setting it equal to zero, as we did in the first example:

$$\begin{array}{r}0=\frac{d}{du}(6u^2-24u+39)=12u-24\Rightarrow u=2\Rightarrow x=3-u=1.\end{array}$$

The point here was not to show how to solve constrained optimization problems in general, but rather to identify the different parts of a problem of this type. As a quick note, you will sometimes see the example optimization problem we’ve been considering written as

$$\begin{array}{rl}\underset{u}{\mathrm{minimize}}& u^2+3x^2-2ux+2u+2x+6\\ \text{subject to}& u+x=3.\end{array}$$

The meaning is exactly the same, but $x$ isn’t listed as one of the decision variables under “minimize.” The idea here is that $x$ is an “extra variable” that we don’t really care about. This optimization problem is trying to say the following:

“Among all choices of $u$ for which there exists an $x$ satisfying $u+x=3$, find the one that minimizes $u^2+3x^2-2ux+2u+2x+6$.”

Minimum vs. Minimizer

We have seen three example problems. In each case, we were looking for the minimizer, i.e., the choice of decision variable that made the cost function as small as possible:

• The solution to

  $$\begin{array}{r}\underset{u}{\mathrm{minimize}}{u}^2-2u+3\end{array}$$

  was $u=1$.

• The solution to

  $$\begin{array}{r}\underset{u_1,u_2}{\mathrm{minimize}}{u}_1^2+3u_2^2-2u_1{u}_2+2u_1+2u_2+6\end{array}$$

  was $(u_1,u_2)=(-2,-1)$.

• The solution to

  $$\begin{array}{rl}\underset{u}{\mathrm{minimize}}& u^2+3x^2-2ux+2u+2x+6\\ \text{subject to}& u+x=3\end{array}$$

  was $(u,x)=(2,1)$.

It is sometimes useful to focus on the minimum instead of on the minimizer, i.e., what the “smallest value” was that we were able to achieve. When focusing on the minimum, we often use the following “set notation” instead:

• The problem

  $$\begin{array}{r}\underset{u}{\mathrm{minimize}}{u}^2-2u+3\end{array}$$

  is rewritten

  $$\begin{array}{r}\underset{u}{\mathrm{minimum}}\{u^2-2u+3\}.\end{array}$$

  The meaning is to “find the minimum value of $u^2-2u+3$ over all choices of $u$.” The solution to this problem can be found by plugging in what we already know is the minimizer, $u=1$. In particular, we find that the solution is $2$.

• The problem

  $$\begin{array}{r}\underset{u_1,u_2}{\mathrm{minimize}}{u}_1^2+3u_2^2-2u_1{u}_2+2u_1+2u_2+6\end{array}$$

  is rewritten

  $$\begin{array}{r}\underset{u_1,u_2}{\mathrm{minimum}}\{u_1^2+3u_2^2-2u_1{u}_2+2u_1+2u_2+6\}.\end{array}$$

  Again, the meaning is to “find the minimum value of $u_1^2+3u_2^2-2u_1{u}_2+2u_1+2u_2+6$ over all choices of $u_1$ and $u_2$.” We plug in what we already know is the minimizer $(u_1,u_2)=(-2,-1)$ to find the solution — it is $3$.

• The problem

  $$\begin{array}{rl}\underset{u}{\mathrm{minimize}}& u^2+3x^2-2ux+2u+2x+6\\ \text{subject to}& u+x=3\end{array}$$

  is rewritten

  $$\begin{array}{r}\underset{u}{\mathrm{minimum}}\{u^2+3x^2-2ux+2u+2x+6:u+x=3\}.\end{array}$$

  And again, the meaning is to “find the minimum value of $u^2+3x^2-2ux+2u+2x+6$ over all choices of $u$ for which there exists $x$ satisfying $u+x=3$.” Plug in the known minimizer, $(u,x)=(2,1)$, and we find that the solution is 15.

The important thing here is to understand the notation and to understand the difference between a “minimum” and a “minimizer.”

Optimal Control

Statement of the problem

The following thing is called an optimal control problem:

$$\begin{array}{rl}\text{(1)}& \underset{u_{[t_0,t_1]}}{\mathrm{minimize}}& h(x(t_1))+{\int }_{t_0}^{t_1}g(x(t),u(t))dt\text{subject to}& \frac{dx(t)}{dt}=f(x(t),u(t)),x(t_0)=x_0.\end{array}$$

Let’s try to understand what it means.

• The statement

  $$\begin{array}{r}\underset{u_{[t_0,t_1]}}{\mathrm{minimize}}\end{array}$$

  says that we are being asked to choose an input trajectory $u$ that minimizes something. Unlike in the optimization problems we saw before, the decision variable $u$ in this problem is a function of time. The notation $u_{[t_0,t_1]}$ is one way of indicating this. Given an initial time $t_0$ and a final time $t_1$, we are being asked to choose the value of $u(t)$ at all times in between, i.e., for all $t\in [t_0,t_1]$.

• The statement

  $$\begin{array}{r}\frac{dx(t)}{dt}=f(x(t),u(t)),x(t_0)=x_0\end{array}$$

  is a constraint. It implies that we are restricted to choices of $u$ for which there exists an $x$ satisfying a given initial condition

  $$\begin{array}{r}x(t_0)=x_0\end{array}$$

  and satisfying the ordinary differential equation

  $$\begin{array}{r}\frac{dx(t)}{dt}=f(x(t),u(t)).\end{array}$$

  One example of an ordinary differential equation that looks like this is our usual description of a system in state-space form:

  $$\begin{array}{r}\dot{x}=Ax+Bu,\end{array}$$

• The statement

  $$\begin{array}{r}h(x(t_1))+{\int }_{t_0}^{t_1}g(x(t),u(t))dt\end{array}$$

  says what we are trying to minimize — it is the cost function in this problem. Notice that the cost function depends on both $x$ and $u$. Part of it, $g(\cdot )$, is integrated (i.e., “added up”) over time. Part of it, $h(\cdot )$, is applied only at the final time. One example of a cost function that looks like this is

  $$\begin{array}{r}x(t_1{)}^{T}Mx(t_1)+{\int }_{t_0}^{t_1}(x(t{)}^{T}Qx(t)+u(t{)}^{T}Ru(t))dt.\end{array}$$

The HJB equation (our new “first-derivative test”)

As usual, there are a variety of ways to solve an optimal control problem. One way is by application of what is called the Hamilton-Jacobi-Bellman Equation, or “HJB.” This equation is to optimal control what the first-derivative test is to optimization. To derive it, we will first rewrite the optimal control problem in “minimum form” (see “Minimum vs Minimizer”):

$$\begin{array}{r}\underset{u_{[t_0,t_1]}}{\mathrm{minimum}}\{h(x(t_1))+{\int }_{t_0}^{t_1}g(x(t),u(t))dt:\frac{dx(t)}{dt}=f(x(t),u(t)),x(t_0)=x_0\}.\end{array}$$

Nothing has changed here, we’re just asking for the minimum and not the minimizer. Next, rather than solve this problem outright, we will first state a slightly different problem:

$$\begin{array}{}\text{(2)}& \underset{{\overline{u}}_{[t,t_1]}}{\mathrm{minimum}}\{h(\overline{x}(t_1))+{\int }_t^{t_1}g(\overline{x}(s),\overline{u}(s))ds:\frac{d\overline{x}(s)}{ds}=f(\overline{x}(s),\overline{u}(s)),\overline{x}(t)=x\}.\end{array}$$

The two changes that I made to go from the original problem to this one are:

• Make the initial time arbitrary (calling it $t$ instead of $t_0$).

• Make the initial state arbitrary (calling it $x$ instead of $x_0$).

I also made three changes in notation. First, I switched from $x$ to $\overline{x}$ to avoid getting confused between $x$ as initial condition and $\overline{x}$ as state trajectory. Second, I switched from $u$ to $\overline{u}$ to be consistent with the switch from $x$ to $\overline{x}$. Third, I switched from calling time $t$ to calling time $s$ to avoid getting confused with my use of $t$ as a name for the initial time.

You should think of the problem (2) as a function itself. In goes an initial time $t$ and an initial state $x$, and out comes a minimum value. We can make this explicit by writing

$$\begin{array}{rl}\text{(3)}& v(t,x)& =\underset{{\overline{u}}_{[t,t_1]}}{\mathrm{minimum}}\{h(\overline{x}(t_1))+{\int }_t^{t_1}g(\overline{x}(s),\overline{u}(s))ds:& \frac{d\overline{x}(s)}{ds}=f(\overline{x}(s),\overline{u}(s)),\overline{x}(t)=x\}.\end{array}$$

We call $v(t,x)$ the value function. Notice that $v(t_0,x_0)$ is the solution to the original optimal control problem that we wanted to solve — the one where the initial time is $t_0$ and the initial state is $x_0$. More importantly, notice that $v(t,x)$ satisfies the following recursion:

$$\begin{array}{rl}\text{(4)}& v(t,x)& =\underset{{\overline{u}}_{[t,t+\mathrm{\Delta }t]}}{\mathrm{minimum}}\{v(t+\mathrm{\Delta }t,\overline{x}(t+\mathrm{\Delta }t))+{\int }_t^{t+\mathrm{\Delta }t}g(\overline{x}(s),\overline{u}(s))ds:& \frac{d\overline{x}(s)}{ds}=f(\overline{x}(s),\overline{u}(s)),\overline{x}(t)=x\}.\end{array}$$

The reason this equation is called a “recursion” is that it expresses the function $v$ in terms of itself. In particular, it splits the optimal control problem into two parts. The first part is from time $t$ to time $t+\mathrm{\Delta }t$. The second part is from time $t+\mathrm{\Delta }t$ to time $t_1$. The recursion says that the minimum value $v(t,x)$ is the sum of the cost

$$\begin{array}{r}\underset{{\overline{u}}_{[t,t+\mathrm{\Delta }t]}}{\mathrm{minimum}}\{{\int }_t^{t+\mathrm{\Delta }t}g(\overline{x}(s),\overline{u}(s))ds:\frac{d\overline{x}(s)}{ds}=f(\overline{x}(s),\overline{u}(s)),\overline{x}(t)=x\}\end{array}$$

from the first part and the cost

$$\begin{array}{r}\underset{{\overline{u}}_{[t+\mathrm{\Delta }t,t_1]}}{\mathrm{minimum}}\{h(\overline{x}(t_1))+{\int }_{t+\mathrm{\Delta }t}^{t_1}g(\overline{x}(t),\overline{u}(t))dt:\frac{d\overline{x}(s)}{ds}=f(\overline{x}(s),\overline{u}(s)),\overline{x}(t+\mathrm{\Delta }t)=\text{blah}\}\end{array}$$

from the second part (where “$\text{blah}$” is whatever the state turns out to be, starting at time $t$ from start $x$ and applying the input $u_{[t,t+\mathrm{\Delta }t]}$), which we recognize as the definition of

$$\begin{array}{r}v(t+\mathrm{\Delta }t,\overline{x}(t+\mathrm{\Delta }t)).\end{array}$$

We now proceed to approximate the terms in (4) by first-order series expansions. In particular, we have

$$\begin{array}{rl}v(t+\mathrm{\Delta }t,\overline{x}(t+\mathrm{\Delta }t))& \approx v(t+\mathrm{\Delta }t,\overline{x}(t)+\frac{d\overline{x}(t)}{dt}\mathrm{\Delta }t)\\ & =v(t+\mathrm{\Delta }t,x+f(x,\overline{u}(t))\mathrm{\Delta }t)\\ & \approx v(t,x)+\frac{\partial v(t,x)}{\partial t}\mathrm{\Delta }t+\frac{\partial v(t,x)}{\partial x}f(x,\overline{u}(t))\mathrm{\Delta }t\end{array}$$

and we also have

$$\begin{array}{rl}{\int }_t^{t+\mathrm{\Delta }t}g(\overline{x}(s),\overline{u}(s))ds& \approx g(\overline{x}(t),\overline{u}(t))\mathrm{\Delta }t\\ & =g(x,\overline{u}(t))\mathrm{\Delta }t.\end{array}$$

If we plug both of these into (4), we find

$$\begin{array}{rl}v(t,x)& =\underset{{\overline{u}}_{[t,t+\mathrm{\Delta }t]}}{\mathrm{minimum}}\{v(t+\mathrm{\Delta }t,\overline{x}(t+\mathrm{\Delta }t))+{\int }_t^{t+\mathrm{\Delta }t}g(\overline{x}(s),\overline{u}(s))ds:\\ & \frac{d\overline{x}(s)}{ds}=f(\overline{x}(s),\overline{u}(s)),\overline{x}(t)=x\}\\ & =\underset{{\overline{u}}_{[t,t+\mathrm{\Delta }t]}}{\mathrm{minimum}}\{v(t,x)+\frac{\partial v(t,x)}{\partial t}\mathrm{\Delta }t+\frac{\partial v(t,x)}{\partial x}f(x,\overline{u}(t))\mathrm{\Delta }t+g(x,\overline{u}(t))\mathrm{\Delta }t:\\ & \frac{d\overline{x}(s)}{ds}=f(\overline{x}(s),\overline{u}(s)),\overline{x}(t)=x\}.\end{array}$$

Notice that nothing inside the minimum depends on anything other than $t$, $x$, and $\overline{u}(t)$. So we can drop the constraint and make $\overline{u}(t)$ the only decision variable. In fact, we might as well replace $\overline{u}(t)$ simply by “$u$” since we only care about the input at a single instant in time:

$$\begin{array}{r}v(t,x)=\underset{u}{\mathrm{minimum}}\{v(t,x)+\frac{\partial v(t,x)}{\partial t}\mathrm{\Delta }t+\frac{\partial v(t,x)}{\partial x}f(x,u)\mathrm{\Delta }t+g(x,u)\mathrm{\Delta }t\}.\end{array}$$

Also, notice that

$$\begin{array}{r}v(t,x)+\frac{\partial v(t,x)}{\partial t}\mathrm{\Delta }t\end{array}$$

does not depend on $u$, so it can be brought out of the minimum:

$$\begin{array}{r}v(t,x)=v(t,x)+\frac{\partial v(t,x)}{\partial t}\mathrm{\Delta }t+\underset{u}{\mathrm{minimum}}\{\frac{\partial v(t,x)}{\partial x}f(x,u)\mathrm{\Delta }t+g(x,\overline{u}(t))\mathrm{\Delta }t\}.\end{array}$$

To simplify further, we can subtract $v(t,x)$ from both sides, then divide everything by $\mathrm{\Delta }t$:

$$\begin{array}{}\text{(5)}& 0=\frac{\partial v(t,x)}{\partial t}+\underset{u}{\mathrm{minimum}}\{\frac{\partial v(t,x)}{\partial x}f(x,u)+g(x,u)\}.\end{array}$$

The equation is called the Hamilton-Jacobi-Bellman Equation, or simply the HJB equation. As you can see, it is a partial differential equation, so it needs a boundary condition. This is easy to obtain. In particular, going all the way back to the definition (3), we find that

$$\begin{array}{}\text{(6)}& v(t_1,x)=h(x).\end{array}$$

The importance of HJB is that if you can find a solution to (5)-(6) — that is, if you can find a function $v(t,x)$ that satisfies the partial differential equation (5) and the boundary condition (6) — then the minimizer $u$ in (5), evaluated at every time $t\in [t_0,t_1]$, is the solution to the optimal control problem (1). You might object that (5) “came out of nowhere.” First of all, it didn’t. We derived it just now, from scratch. Second of all, where did the first-derivative test come from? Could you derive that from scratch? (Do you, every time you need to use it? Or do you just use it?)

Solution approach

The optimal control problem

$$\begin{array}{rl}\underset{u_{[t_0,t_1]}}{\mathrm{minimize}}& h(x(t_1))+{\int }_{t_0}^{t_1}g(x(t),u(t))dt\\ \text{subject to}& \frac{dx(t)}{dt}=f(x(t),u(t)),x(t_0)=x_0\end{array}$$

can be solved in two steps:

• Find $v$:

  $$\begin{array}{r}0=\frac{\partial v(t,x)}{\partial t}+\underset{u}{\mathrm{minimum}}\{\frac{\partial v(t,x)}{\partial x}f(x,u)+g(x,u)\},v(t_1,x)=h(x).\end{array}$$

• Find $u$:

  $$\begin{array}{r}u(t)=\underset{u}{\mathrm{minimize}}\frac{\partial v(t,x)}{\partial x}f(x,u)+g(x,u)\text{for all }t\in [t_0,t_1].\end{array}$$

LQR

Statement of the problem

Here is the linear quadratic regulator (LQR) problem:

$$\begin{array}{rl}\underset{u_{[t_0,t_1]}}{\mathrm{minimize}}& x(t_1{)}^{T}Mx(t_1)+{\int }_{t_0}^{t_1}(x(t{)}^{T}Qx(t)+u(t{)}^{T}Ru(t))dt\\ \text{subject to}& \dot{x}(t)=Ax(t)+Bu(t),x(t_0)=x_0.\end{array}$$

It is an optimal control problem — if you define

$$\begin{array}{r}f(x,u)=Ax+Bug(x,u)=x^{T}Qx+u^{T}Ruh(x)=x^{T}Mx\end{array}$$

then you see that this problem has the same form as (1). It is called “linear” because the dynamics are those of a linear (state space) system. It is called “quadratic” because the cost is quadratic (i.e., polynomial of order at most two) in $x$ and $u$. It is called a “regulator” because the result of solving it is to keep $x$ close to zero (i.e., to keep errors small).

The matrices $Q$, $R$, and $M$ are parameters that can be used to trade off error (non-zero states) with effort (non-zero inputs). These matrices have to be symmetric ($Q=Q^T$, etc.), have to be the right size, and also have to satisfy the following conditions in order for the LQR problem to have a solution:

$$\begin{array}{r}Q\ge 0R>0M\ge 0.\end{array}$$

What this notation means is that $Q$ and $M$ are positive semidefinite and that $R$ is positive definite (https://en.wikipedia.org/wiki/Positive-definite_matrix). We will ignore these terms for now, noting only that this is similar to requiring (for example) that $r>0$ in order for the function $r{u}^2$ to have a minimum.

Do the matrices $Q$, $R$, and $M$ really have to be symmetric?

Suppose $Q_{\text{silly}}$ is not symmetric. Define

$$Q=\frac{1}{2}(Q_{\text{silly}}+Q_{\text{silly}}^T).$$

First, notice that $Q$ is symmetric:

$$\begin{array}{rl}{Q}^T& =\frac{1}{2}{(Q_{\text{silly}}+Q_{\text{silly}}^T)}^T\\ & =\frac{1}{2}(Q_{\text{silly}}^T+Q_{\text{silly}})& & \text{because }(M+N{)}^T=M^T+N^T\\ & =\frac{1}{2}(Q_{\text{silly}}+Q_{\text{silly}}^T)& & \text{because }M+N=N+M\\ & =Q.\end{array}$$

In fact, $Q$ — when defined in this way — is often referred to as the symmetric part of $Q_{\text{silly}}$.

Second, notice that we can replace $Q_{\text{silly}}$ with $Q$ without changing the cost:

$$\begin{array}{rl}{x}^{T}Qx& =x^T\frac{1}{2}(Q_{\text{silly}}+Q_{\text{silly}}^T)x\\ & =\frac{1}{2}{x}^T{Q}_{\text{silly}}x+\frac{1}{2}{x}^T{Q}_{\text{silly}}^{T}x\\ & =\frac{1}{2}{x}^T{Q}_{\text{silly}}x+\frac{1}{2}{\left(x^T{Q}_{\text{silly}}x\right)}^T& & \text{because }(MN{)}^T=N^T{M}^T\\ & =\frac{1}{2}{x}^T{Q}_{\text{silly}}x+\frac{1}{2}\left(x^T{Q}_{\text{silly}}x\right)& & \text{because the transpose of a scalar is itself}\\ & =\frac{1}{2}(x^T{Q}_{\text{silly}}x+x^T{Q}_{\text{silly}}x)\\ & =x^T{Q}_{\text{silly}}x.\end{array}$$

So, it is not that $Q$, $R$, and $M$ “have to be” symmetric, it’s that they “might as well be” symmetric — if they’re not, we can replace them with matrices that are symmetric without changing the cost.

Solution to the problem

Our “Solution approach” tells us to solve the LQR problem in two steps. First, we find a function $v(t,x)$ that satisfies the HJB equation. Here is that equation, with the functions $f$, $g$, and $h$ filled in from our “Statement of the problem”:

$$\begin{array}{r}0=\frac{\partial v(t,x)}{\partial t}+\underset{u}{\mathrm{minimum}}\{\frac{\partial v(t,x)}{\partial x}(Ax+Bu)+x^{T}Qx+u^{T}Ru\},v(t_1,x)=x^{T}Mx.\end{array}$$

What function $v$ might solve this equation? Look at the boundary condition. At time $t_1$,

$$\begin{array}{r}v(t_1,x)=x^{T}Mx.\end{array}$$

This function has the form

$$\begin{array}{r}v(t,x)=x^{T}P(t)x\end{array}$$

for some symmetric matrix $P(t)$ that satisfies $P(t_1)=M$. So let’s “guess” that this form is the solution we are looking for, and see if it satisfies the HJB equation. Before proceeding, we need to compute the partial derivatives of $v$:

$$\begin{array}{r}\frac{\partial v}{\partial t}=x^T\dot{P}x\frac{\partial v}{\partial x}=2x^{T}P\end{array}$$

This is matrix calculus (e.g., see https://en.wikipedia.org/wiki/Matrix_calculus or Chapter A.4.1 of http://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf). The result on the left should surprise no one. The result on the right is the matrix equivalent of

$$\partial (p{x}^2)/\partial x=2px.$$

You could check that this result is correct by considering an example. Plug these partial derivatives into HJB and we have

$$\begin{array}{rl}\text{(7)}& 0& =x^T\dot{P}x+\underset{u}{\mathrm{minimum}}\{2x^{T}P(Ax+Bu)+x^{T}Qx+u^{T}Ru\}& =x^T\dot{P}x+\underset{u}{\mathrm{minimum}}\{x^T(2PA+Q)x+2x^{T}PBu+u^{T}Ru\}\end{array}$$

To evaluate the minimum, we apply the first-derivative test (more matrix calculus!):

$$\begin{array}{rl}0& =\frac{\partial }{\partial u}(x^T(2PA+Q)x+2x^{T}PBu+u^{T}Ru)\\ & =2x^{T}PB+2u^{T}R\\ & =2{(B^{T}Px+Ru)}^T.\end{array}$$

This equation is easily solved:

$$\begin{array}{}\text{(8)}& u=-R^{-1}{B}^{T}Px.\end{array}$$

Plugging this back into (7), we have

$$\begin{array}{rl}0& =x^T\dot{P}x+\underset{u}{\mathrm{minimum}}\{x^T(2PA+Q)x+2x^{T}PBu+u^{T}Ru\}\\ & =x^T\dot{P}x+(x^T(2PA+Q)x+2x^{T}PB(-R^{-1}{B}^{T}Px)+(-R^{-1}{B}^{T}Px{)}^{T}R(-R^{-1}{B}^{T}Px))\\ & =x^T\dot{P}x+(x^T(2PA+Q)x-2x^{T}PB{R}^{-1}{B}^{T}Px+x^{T}PB{R}^{-1}{B}^{T}Px)\\ & =x^T\dot{P}x+x^T(2PA+Q)x-x^{T}PB{R}^{-1}{B}^{T}Px\\ & =x^T\dot{P}x+x^T(PA+A^{T}P+Q)x-x^{T}PB{R}^{-1}{B}^{T}Px\\ & \dots \text{because }{x}^T(N+N^T)x=2x^{T}Nx\text{ for any }N\text{ and }x\dots \\ & =x^T(\dot{P}+PA+A^{T}P+Q-PB{R}^{-1}{B}^{T}P)x.\end{array}$$

In order for this equation to be true for any $x$, it must be the case that

$$\begin{array}{r}\dot{P}=PB{R}^{-1}{B}^{T}P-PA-A^{T}P-Q.\end{array}$$

In summary, we have found that

$$\begin{array}{r}v(t,x)=x^{T}Px\end{array}$$

solves the HJB equation, where $P$ is found by integrating the matrix differential equation

$$\begin{array}{r}\dot{P}=PB{R}^{-1}{B}^{T}P-PA-A^{T}P-Q\end{array}$$

backward in time, starting from

$$\begin{array}{r}P(t_1)=M.\end{array}$$

Now that we know $v$, we can find $u$. Wait, we already did that! The minimizer in the HJB equation is (8). This choice of input has the form

$$u=-Kx$$

for

$$\begin{array}{r}K=R^{-1}{B}^{T}P.\end{array}$$

LQR Code

Here is how to find the solution to an LQR problem with Python:

import numpy as np
from scipy import linalg

def lqr(A, B, Q, R):
    # Find the cost matrix
    P = linalg.solve_continuous_are(A, B, Q, R)
    # Find the gain matrix
    K = linalg.inv(R) @ B.T @ P
    return K