Set up a basic problem

To set up a basic problem, you need to:

Write a function which defines the system and cost function in an m-file,
Set up the problem using the DynaProg interface.

Define the system and cost function

The system and cost function is a MATLAB function having signature:

[x_next, stage_cost, unfeas] = sysfun(x, u, ~)

Your function should accept three inputs: the first two are the state variables x and control variables u. The third input can be used to pass an exogenous input to the system function. If your system model does not make use of exogenous inputs, you can replace the third input in your function signature with a tilde (~).

Also, the function should return three outputs: the first output contains the updated state variables ( x_(k+1)

, the value of the state variables after advancing the system by one stage) and the second output contains the stage cost incurred. The third output is used to define unfeasibility, which allows to set constraints on the state and control variables . If you do not wish to use this feature, you must still specify a third output in your function signature, but this should return an empty array: you should include anywhere in your function the line unfeas = [];.

Note that any name can be used in place of sysfun, provided that it suits MATLAB syntax for function naming.

Even f the system is described by one or more state variable, treat x and x_next as cell arrays, where each cell contains one of the state variables.. For example, if your system has two states x_1

and

, and the dynamics of x_1

is described by the equation dx_1/dt

, this would be coded in the function as:

function [x_next, stage_cost, unfeas] = sysfun(x, u, ~)
...
x_next{1} = x{1} + (x{1} - x{2}) .* dt;
...

Similarly, treat u as a cell array as well.

Finally, the stage cost can be any function of the state and control variables, and it must be a scalar.

Set up and solve the optimization problem

To set up and solve a problem, you must write a script where you:

specify the problem structure,
create a DynaProg problem,
run the solution.

First, you need to create discrete grids for the state and control variables, as the Dynamic Programming algorithm. Note that this does not mean that the state variables evolution will be discretized in your optimal solution, but it may affect the optimality of the solution. For more information, see Theory.

To define a grid, simply create a numeric or logical vector.

x_grid = 0:0.1:1;
u_grid = -1:0.2:1;

If the state and/or control variables are not scalar, specify them as a cell array of vectors, each cell containing the grid for one of the variables.

x1_grid = 0:0.1:1;
x2_grid = 0:0.1:1;
x_grid = {x1_grid, x2_grid};

Then, you need to specify an initial condition for all state variables. Specify them in a cell array.

x_initial = {0, 0};

Finally, you can specify constraints on the final value of the state variables (at the end of the simulation). Specify them as cell arrays of two-element vectors; each vector should contain the lower and upper bounds of your constraint.

x_final = {[0.3 0.4], [0.65 0.7]};

You must specify one initial condition for each of the state variables.

If you do not wish to specify target values for one or more of the final state variables, specify those target values as empty arrays. For example, for a two-state system, you can specify a target value for x_1

and no target value for x_2

x_final = {[0.3 0.4], []};

If you do not wish to specify target values for any of the final state variables, you can also define the final state as an empty array.

x_final = [];

Set up the problem object

Once you have created the system function, state variables grids, initial conditions and target final values and control variables grids, you must create a DynaProg problem:

prob = DynaProg(StateGrid, StateInitial, StateFinal, ControlGrid, Nstages, SysName);

Where Nstages is the number of stages and SysName is a function handle to the system model you created, e.g.

SysName = @sysfun;

Run the optimization problem and visualize results

Finally, you can run the optimization problems and return the problem structure including results by using the run method.

prob = run(prob);

The dynamic programming optimization algorithm runs and creates three main results:

ControlProfile: the optimal control variable(s) for each stage, stored in a cell array of vectors.
StateProfile: the value of the state variable(s) for each stage when applying the optimal control policy, stored in a cell array of vectors.
CostProfile: the value of the stage cost for each stage when applying the optimal control policy, stored vector.

Once the optimization is complete, the results are saved in the problem object itself, and you can access them via dot indexing, e.g. you can access the optimal control variables profile by querying:

optimal_cv = prob.ControlProfile;

You can also visualize them using the plot method.

plot(prob)

By default, the plot automatically names the state variables, control variables and cumulative cost, and it uses the simulation stages as the x-axis.

To specify names for the state and control variables, set the StateName and ControlName properties of your problem structure to a string array. To specify a name for the cumulative cost, set the CostName property to a string.

prob.StateName = ["sv name 1", "sv name 2"];
prob.ControlName = ["cv name 1", "cv name 2"];
prob.CostName = "cost name";

To specify a time array that replaces the stages in the x-axis, set the Time property to a numeric array.

prob.Time = linspace(0, 1, Nstages+1);

Alternatively, specify the relevant name-value pairs in constructing the problem object:

prob = DynaProg(x_grid, x_init, x_final, u_grid, Nint, @cart, ...
    'StateName', ["sv name 1", "sv name 2"], 'ControlName', ...
    ["cv name 1", "cv name 2"], 'CostName', "Energy", 'Time', time);