Table des matières

• 1 Finite horizon problem

1 Finite horizon problem

Let be a finite Markov chain with transition matrix . We want to recursively compute :

    

Question 1   Write a first routine which returns a random stochastic matrix of size NxN. In scilab there is a function called genmarkov that you can also check if you want.

  
function M=my_gen_markov(n); 
  M=rand(...)
  ... 
endfuncion

    

Question 2   Choose a state size, generate a stochastic matrix , then generate and plot some samples of the homogeneous Markov chain with states in described by (using grand in Scilab).

  
n=10;
M=my_gen_markov(n)
// generate and draw some typical samples 
T=4;
m=5;// number of samples 
Y=grand(...) 
Y=[ones(m,1),Y]; // add the first state to the sampled trajectories 
// draw the first trajcetory
plot(Y(1,:))

    

Question 3   Choosing an instantaneous cost and a final cost recusively compute the value function and draw the result as a surface (i.e ).

  
function y=c(x); y=... endfunction 
function y=f(x); y=... endfunction 

states=... // possible states 
Cv=c(states) // vector of c possible values 
Kv=f(states) // vector of f possible values 
r=0.05; // a discount factor 
V=...  // initialize V to fix dimensions 
V(:,T) = ...  // the final value of V for time T 
for i=T-1:-1:1 
  V(:,i) = ...  
end

    

Question 4   Using samples of the markov chain for a given starting state evaluate by Monte Carlo the cost function and compare the results with previous question.

  
m= ... // number of Monte Carlo
X0=ones(m,1); // initial state for all samples 
X=grand(...
X=[X0,X];
// fix n
// Approximate the cost at time n  by Monte Carlo 
n=1;
Cm= mean(.... f(X(:,T)));
for i=(T-1):-1:n;
  Cm = mean(... 
end
Cm - C(1,n)