Informatics and Applications
2026, Volume 20, Issue 3, pp 23-32
OPTIMAL STOCHASTIC CONTROL OF MARKOV JUMP PROCESSES UNDER DELAYED NOISE-FREE OBSERVATIONS I: UNIVERSAL CANONICAL SPACE AND THE MARTINGALE PROBLEM
- A. V. Borisov
- Yu. N. Kurinov
Abstract
The first part of this series presents the theoretical foundations required for the proper formulation and solution of a finite-horizon stochastic control problem under incomplete information. The controlled system is represented by a class of Markov jump processes (MJPs) with a finite state space. The optimality criterion is defined as the expected value of an integral loss functional. The observations available for control synthesis are functions of the controlled state, measured without external noise but subject to a time delay. The class of admissible controls consists of processes that are predictable with respect to the natural filtration generated by the observations and that satisfy a set of geometric and integral constraints. A universal filtered probability space is constructed to provide a solution to the martingale problem associated with this class of controlled MJPs. In addition, conditions ensuring the continuity of both the MJP state trajectory and the objective functional with respect to the applied control are established.
[+] References (28)
- Dynkin, E. B., and A. A. Yushkevich. 1975. Upravlyaemye markovskie protsessy i ikh prilozheniya [Controlled Markov processes and their applications]. Moscow: Nauka. 338 p.
- Davis, M.H.A. 1993. Markov models and optimization. London: Chapman and Hall/CRC. 308 p. doi: 10.1201/ 9780203748039.
- Piunovskiy, A., and Y. Zhang. 2020. Continuous-time Markov decision processes: Borel space models and general control strategies. Cham: Springer. 583p.doi: 10.1007/978- 3-030-54987-9.
- Bosov, A. V. 2024. Stabilizatsiya avtonomnogo lineynogo vykhoda markovskoy tsepi po kvadratichnomu kriteriyu na beskonechnom gorizonte [Autonomous linear output of the Markov chain stabilization by square criterion on an infinite horizon]. Informatika i ee Primeneniya - Inform. Appl. 18(3):38-44. doi: 10.14357/19922264240305. EDN: XVTFLH.
- Low, S., F Paganini, and J. Doyle. 2002. Internet congestion control. IEEEContr. Syst. Mag. 22(1):28-43. doi: 10.1109/37.980245.
- Altman, E. 2002. Applications of Markov decision processes in communication networks. Handbook of Markov decision processes. Eds. E.A. Feinberg and A. Shwartz. International series in operations research and management science. Boston, MA: Springer. 40:489-536. doi: 10.1007/978-1-4615-0805-2-16.
- Miller, B. M., K. E. Avrachenkov, K. V. Stepanyan, and G. B. Miller. 2005. Flow control as a stochastic optimal control problem with incomplete information. Probl. Inf. Transm. 41(2):150-170. doi: 10.1007/s11122-005- 0020-8. EDN: LJFPRR.
- McInnes, D., and B. Miller. 2017. Optimal control of a large dam using time-inhomogeneous Markov chains with an application to flood control. IFAC PapersOn- Line 50(1):3499-3504. doi: 10.1016/j.ifacol.2017.08.936. EDN: XPJIEV.
- Shaldambayeva, S., K. Zhumakhanova, A. Sandygulova, et al. 2025. Partially observable Markov decision processes in shared autonomy applications: A survey. IEEE Access 13:186937-186951. doi: 10.1109/ ACCESS.2025.3625601.
- Fox, A., F. De Pellegrini, E. Altman, etal. 2025. Performing load balancing under constraints. 23rd Symposium (International) on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks Proceedings. IEEE. 103-110. doi: 10.23919/WiOpt66569.2025.11123242.
- Rieder, U., and J. Winter. 2009. Optimal control of Markovian jump processes with partial information and applications to a parallel queueing model. Math. Method. Oper. Res. 70:567-596. doi: 10.1007/s00186-009-0284-7.
- Li, Q., J. Ma, R. Fan, and L. Xia. 2019. An overview for Markov decision processes in queues and networks. Stochastic models in reliability, network security and system safety. Eds. Q. L. Li, J. Wang, and H. B. Yu. Communications in computer and information science ser. Singapore: Springer. 1102:44-71. doi: 10.1007/978-981- 15-0864-6_3.
- White, D. J. 1996. A survey of applications of Markov decision processes. J. Oper. Res. Soc. 44(11):1073-1096. doi: 10.2307/2583870.
- Shi, P., and F. Li. 2015. A survey on Markovian jump systems: Modeling and design. Int. J. ControlAutom. 13(1):1- 16. doi: 10.1007/s12555-014-0576-4.
- Borisov, A. 2023. Rynok s markovskoy skachkoobraznoy volatil'nost'yu I: monitoring tseny riska kak zadacha optimal'noy fil'tratsii [Market with Markov jump volatility I: Price of risk monitoring as an optimal filtering problem]. Informatika iee Primeneniya - Inform. Appl. 17(2):27-33. doi: 10.14357/19922264230204. EDN: GAXCHQ.
- Borisov, A. 2024. Rynok s markovskoy skachkoobraznoy volatil'nost'yu V: popolnenie rynka derivativami [Market with Markov jump volatility V: Market completion with derivatives]. Informatika i ee Primeneniya - Inform. Appl. 18(2):9-16. doi: 10.14357/19922264240202. EDN: DQFSDO.
- Bosov, A. V. 2025. Stabilizatsiya traektorii lineynoy sistemy so skachkoobraznym dreyfom pri tselochislennykh ogranicheniyakh na upravlenie [Stabilization of the trajectory of a linear system with jumping drift under integer control constraints]. Informatika i ee Primeneniya - Inform. Appl. 19(3):10-18. doi: 10.14357/19922264250302. EDN: YZEHCC.
- Krylov, N.V. 1977. Upravlyaemye protsessy diffuzionnogo tipa [Controlled diffusion processes]. Moscow: Nauka. 400 p.
- Liptser, R. Sh., and A. N. Shiryayev. 1986. Teoriya martingalov [Theory of martingales]. Moscow: Fizmatlit. 512 p.
- Miller, B.M., G. B. Miller, and K. V Semenikhin. 2011. Methods to design optimal control of Markov process with finite state set in the presence of constraints. Automat. Rem. Contr. 72(2):323-341. doi: 10.1134/S000511791102010X. EDN: OHUBSJ.
- Miller, B.M., G. B. Miller, and K. V Semenikhin. 2016. Optimal control problem regularization for the Markov process with finite number of states and constraints. Automat. Rem. Contr. 77(9):1589-1611. doi: 10.1134/S0005117916090071. EDN: XFJXFJ.
- Jacod, J., and A. N. Shiryaev. 1987. Limit theorems for stochastic processes. Berlin: Springer. 600 p.
- Ishikawa, Y., and H. Kunita. 2006. Malliavin calculus on the Wiener-Poisson space and its application to canonical SDE with jumps. Stoch. Proc. Appl. 116(12):1743-1769. doi: 10.1016/j.spa.2006.04.013.
- Wonham, W. M. 1968. On the separation theorem of stochastic control. SIAM J. Control 6(2):312-326. doi: 10.1137/0306023.
- Borisov, A. V. 2025. Fil'tratsiya sostoyaniy i parametrov spetsial'nykh markovskikh skachkoobraznykh protsessov po kosvennym nablyudeniyam bez shuma [Filtering of states and parameters of special Markov jump processes via indirect perfect observations]. Informatika i ee primeneniya - Inform. Appl. 19(1):25-32. doi: 10.14357/ 19922264250104. EDN: MZAJVZ.
- Borisov, A. V. 2025. The use of optimal filtering methods for passive monitoring of available bandwidth of a network channel. Automat. Rem. Contr. 86(6):545-563.
- Kern, P., and C. Muuller. 2022. A closed form formula for the stochastic exponential of a matrix-valued semimartingale. J. Stochastic Analysis 3(2):3. 19 p. doi: 10.31390/josa.3.2.03. EDN: UGPTFV.
- Lakshmikantam, V., S. Leela, and A. A. Martynyuk. 1991. Ustoichivost' dvizheniya: metod sravneniya [Motion stability: Comparison method]. Kiev: Naukova Dumka. 248 p.
[+] About this article
Title
OPTIMAL STOCHASTIC CONTROL OF MARKOV JUMP PROCESSES UNDER DELAYED NOISE-FREE OBSERVATIONS I: UNIVERSAL CANONICAL SPACE AND THE MARTINGALE PROBLEM
Journal
Informatics and Applications
2026, Volume 20, Issue 3, pp 23-32
Cover Date
2026-30-09
DOI
10.14357/19922264260302
Print ISSN
1992-2264
Publisher
Institute of Informatics Problems, Russian Academy of Sciences
Additional Links
Key words
controlled Markov jump process; Wiener-Poisson space; martingale problem; Poisson stochastic measure
Authors
A. V. Borisov  ,  and Yu. N. Kurinov
Author Affiliations
Federal Research Center "Computer Science and Control" of the Russian Academy of Sciences, 44-2 Vavilov Str., Moscow 119333, Russian Federation
M. V Lomonosov Moscow State University, 1-52 Leninskie Gory, GSP-1, Moscow 119991, Russian Federation
|