Gittins Type Index Theorem for Randomly Evolving Graphs

We consider the problem which informally can be described as follows. Initially a finite set of independent trials is available. If a Decision Maker (DM) chooses to test a specific trial she receives a reward, and with some probability, the process of testing is terminated or the tested trial becomes unavailable but some random finite set (possibly empty) of new independent trials is added to the set of initial trials, and so on. The total number of potential trials is finite. A DM knows the rewards and transition probabilities depending on the trials. On each step she can either quit (i.e. stop the process of testing), or continue. Her goal is to select an order to test trials and an quitting (stopping) time to maximize the expected total reward. We simplify and generalize some results obtained earlier for similar problems, we prove that an index can be assigned to each possible trial and all optimal strategy uses on each step the trial with maximal index between available ones. We present a recursive procedure with a transparent interpretation to calculate the index. We discuss the connection between introduced index and Gittins index.