آمار

تعداد نشریات32
تعداد شماره‌ها502
تعداد مقالات4,520
تعداد مشاهده مقاله7,972,792
تعداد دریافت فایل اصل مقاله5,359,613
Shukhrat, Umarov, Matluba, Isaqova, Mukhitdinov, Otabek, Kudrat, Boboxujayev, Dadaxon, Abdullayev, Luqmon N, Samiev, Nozimbek, Nosirov, Valisher, Sapayev. (1404). Decentralized Energy Management in Electrical and Thermal Microgrids Utilizing Reinforcement Learning. سامانه مدیریت نشریات علمی, 13(Special Issue), 45-61. doi: 10.22098/joape.2025.18916.2468
Umarov Shukhrat; Isaqova Matluba; Otabek Mukhitdinov; Boboxujayev Kudrat; Abdullayev Dadaxon; Samiev Luqmon N; Nosirov Nozimbek; Sapayev Valisher. "Decentralized Energy Management in Electrical and Thermal Microgrids Utilizing Reinforcement Learning". سامانه مدیریت نشریات علمی, 13, Special Issue, 1404, 45-61. doi: 10.22098/joape.2025.18916.2468
Shukhrat, Umarov, Matluba, Isaqova, Mukhitdinov, Otabek, Kudrat, Boboxujayev, Dadaxon, Abdullayev, Luqmon N, Samiev, Nozimbek, Nosirov, Valisher, Sapayev. (1404). 'Decentralized Energy Management in Electrical and Thermal Microgrids Utilizing Reinforcement Learning', سامانه مدیریت نشریات علمی, 13(Special Issue), pp. 45-61. doi: 10.22098/joape.2025.18916.2468
Shukhrat, Umarov, Matluba, Isaqova, Mukhitdinov, Otabek, Kudrat, Boboxujayev, Dadaxon, Abdullayev, Luqmon N, Samiev, Nozimbek, Nosirov, Valisher, Sapayev. Decentralized Energy Management in Electrical and Thermal Microgrids Utilizing Reinforcement Learning. سامانه مدیریت نشریات علمی, 1404; 13(Special Issue): 45-61. doi: 10.22098/joape.2025.18916.2468

Decentralized Energy Management in Electrical and Thermal Microgrids Utilizing Reinforcement Learning

Journal of Operation and Automation in Power Engineering
دوره 13، Special Issue، 2025، صفحه 45-61    اصل مقاله (2.65 M)
نوع مقاله: Research paper
شناسه دیجیتال (DOI): 10.22098/joape.2025.18916.2468
نویسندگان
Umarov Shukhrat* 1؛ Isaqova Matluba2؛ Otabek Mukhitdinov3؛ Boboxujayev Kudrat4؛ Abdullayev Dadaxon5؛ Samiev Luqmon N6؛ Nosirov Nozimbek7؛ Sapayev Valisher8
1Department of Engineering of Electrical Machines and Drives, Tashkent State Technical University, University Street No2, Tashkent, Uzbekistan.
2Tashkent Institute of Irrigation and Agricultural Mechanization Engineers Institute" National Research University, Kari Niyazov Street 39, 100000, Tashkent, Uzbekistan.
3Kimyo International University in Tashkent, Shota Rustaveli Street 156, 100121, Tashkent, Uzbekistan.
4PhD, Assistant Professor, Alfraganus University, Uzbekistan.
5Tashkent Institute of Irrigation and Agricultural Mechanization Engineers National Research University, Uzbekistan.
6Urgench State University named after Abu Rayhan Biruni, Urgench, Uzbekistan.
7Research Institute of Environmental and Nature Protection Technologies, 100000, Tashkent, Uzbekistan.
8Department of General Professional Subjects, Mamun University, Khiva, Uzbekistan.
چکیده
This paper proposes a fully decentralized reinforcement learning–based energy management framework for hybrid electrical–thermal microgrids with distributed energy resources. Uncertainties in renewable energy generation, variations in load demand, and the nonlinear nature of battery systems make it difficult to achieve optimal energy management in microgrids. Additionally, using centralized controller techniques in large-scale systems increases computational complexity and makes controller procedure implementation more challenging. This study proposes a fully decentralized multi-agent architecture in which the stochastic performance of agents in the microgrid is modeled using Markov decision processes. This model treats consumers, batteries, and distributed thermal and electrical resources as intelligent, self-governing agents that learn from their surroundings and converge to their best policies through decentralized exploitation. The proposed model-free learning-based approach is designed to not only maximize the profits of producers but also minimize the costs for consumers and reduce the microgrid's reliance on the main grid. Finally, using real-world data from renewable power plants and electricity market data, the performance of the proposed method is evaluated through simulation and accuracy assessment.
کلیدواژه‌ها
Decentralized energy management؛ microgrid؛ distributed resources؛ reinforcement learning؛ markov decision process
مراجع
  1. S. Makaba, U. Mardianto, K. Jumintono, and N. Nugrohowati, “Genetic algorithm optimization with machine learning to check the primary health of hospital visitors,” Procedia Environ. Sci. Eng. Manag., vol. 12, no. 1, pp. 7–15, 2025.
  2. M. Rezaee and V. A. Maleki, “On the complex mode shapes and natural frequencies of clamped–clamped fluid-conveying pipe,” Appl. Ocean Res., vol. 150, p. 104113, 2024.
  3. S. Gulandom, G. Shuhratovich, R. Ramatjanovna, K. Yaqipbay, B. Baltabayevna, J. Olimjon, and G. Aleksandrovna, “Development of optimization process for improving the educational classes scheduling,” Procedia Environ. Sci. Eng. Manag., vol. 12, no. 1, pp. 71–80, 2024.
  4. A. Anuchin, N. Kuraev, L. Rassudov, D. Savkin, and G. Demidova, “Multi-functional test benches for electric drive instructional laboratories,” Int. J. Ind. Eng. Manag., vol. 16, no. 2, pp. 161–175, 2025.
  5. N. Sharafkhani, “An ultra-thin multi-layered metamaterial for power transformer noise absorption,” Build. Acoust., vol. 29, no. 1, pp. 53–62, 2022.
  6. N. Sharafkhani, J. O. Orwa, S. D. Adams, J. M. Long, G. Lissorgues, L. Rousseau, and A. Z. Kouzani, “An intracortical polyimide microprobe with piezoelectric-based stiffness control,” J. Appl. Mech., vol. 89, no. 9, p. 091008, 2022.
  7. N. Sharafkhani, A. Z. Kouzani, S. D. Adams, J. M. Long, and J. O. Orwa, “A pneumatic-based mechanism for inserting a flexible microprobe into the brain,” J. Appl. Mech., vol. 89, no. 3, p. 031010, 2022.
  8. H. Dasari and E. Eisenbraun, “Predicting the effect of silicon electrode design parameters on thermal performance of a lithium-ion battery,” J. Electrochem. Sci. Eng., vol. 13, no. 4, pp. 659–672, 2023.
  9. L. Kruitwagen, J. E. Hinkel, M. C. Lioris, M. Stephan, J. L. Hacke, M. D. Islam, and S. A. Kurtz, “A global inventory of photovoltaic solar energy generating units,” Nature, vol. 598, no. 7882, pp. 604–610, 2021.
  10. M. W. Akram, G. Li, Y. Jin, and X. Chen, “Failures of photovoltaic modules and their detection: A review,” Appl. Energy, vol. 313, p. 118822, 2022.
  11. H. M. Hussein, A. Aghmadi, M. S. Abdelrahman, S. M. S. H. Rafin, and O. Mohammed, “A review of battery state of charge estimation and management systems: Models and future prospective,” WIREs Energy Environ., vol. 13, no. 1, p. e507, 2024.
  12. S. Wang, Q. Tan, X. Ding, and J. Li, “Efficient microgrid energy management with neural-fuzzy optimization,” Int. J. Hydrogen Energy, vol. 64, pp. 269–281, 2024.
  13. C. Álvarez Arroyo, S. Vergine, A. Sánchez de la Nieta, L. Alvarado-Barrios, and G. D’Amico, “Optimising microgrid energy management: Leveraging flexible storage systems and full integration of renewable energy sources,” Renewable Energy, vol. 229, p. 120701, 2024.
  14. M. R. Khan, Z. M. Haider, F. H. Malik, F. M. Almasoudi, K. S. S. Alatawi, and M. S. Bhutta, “A comprehensive review of microgrid energy management strategies considering electric vehicles, energy storage systems, and AI techniques,” Processes, vol. 12, no. 2, p. 270, 2024.
  15. G. Liu, M. F. Ferrari, T. B. Ollis, and K. Tomsovic, “An MILP-based distributed energy management for coordination of networked microgrids,” Energies, vol. 15, no. 19, p. 6971, 2022.
  16. P. Buchibabu and J. Somlal, “Sustainable energy management in microgrids: A multi-objective approach for stochastic load and intermittent renewable energy resources,” Electr. Eng., pp. 1–15, 2024.
  17. J. S. Giraldo, J. A. Castrillon, J. C. López, M. J. Rider, and C. A. Castro, “Microgrids energy management using robust convex programming,” IEEE Trans. Smart Grid, vol. 10, no. 4, pp. 4520–4530, 2019.
  18. Z. Shen, C. Wu, L. Wang, and G.-L. Zhang, “Real-time energy management for microgrid with EV station and CHP generation,” IEEE Trans. Netw. Sci. Eng., vol. 8, no. 2, pp. 1492–1501, 2021.
  19. Q. Duan, W. Sheng, H. Wang, C. Zhao, C. Ma, and S. Cheng, “A two-stage robust optimization method based on the expected scenario for islanded microgrid energy management,” Discrete Dyn. Nat. Soc., vol. 2021, p. 7079296, 2021.
  20. S. Haddadipour, V. Amir, and S. J. Arani, “Simultaneous purchase and sale of electricity in a multi-agent microgrid energy market,” Comput. Intell. Electr. Eng., vol. 11, no. 4, pp. 93–110, 2020.
  21. S. Umetani, Y. Fukushima, and H. Morita, “A linear programming-based heuristic algorithm for charge and discharge scheduling of electric vehicles in a building energy management system,” Omega, vol. 67, pp. 115–122, 2019.
  22. A. Mohammad, M. Zuhaib, and I. Ashraf, “An optimal home energy management system with integration of renewable energy and energy storage with home-to-grid capability,” Int. J. Energy Res., vol. 46, no. 6, pp. 8352–8366, 2022.
  23. A. Seifi, M. H. Moradi, M. Abedini, and A. Jahangiri, “Assessing the impact of load response on microgrids considering uncertainty in renewable generation,” Comput. Intell. Electr. Eng., vol. 12, no. 1, pp. 87–98, 2021.
  24. M. Yadipour, F. Hashemzadeh, and M. Baradarannia, “Controller design to enlarge the domain of attraction for a class of nonlinear systems,” in Proc. Int. Conf. Research and Education in Mechatronics (REM), pp. 1–6, IEEE, 2019.
  25. S. Sourani Yancheshmeh, A. Ebrahimpour, and T. Deemyad, “Optimizing chassis design for autonomous vehicles in challenging environments based on finite element analysis and genetic algorithm,” in Proc. ASME Int. Mech. Eng. Congr. Expo. (IMECE), ASME, 2024.
  26. K. C. Bingham, S. Sourani Yancheshmeh, G. Vaidya, A. Ebrahimpour, and T. Deemyad, “Advanced material selection and design strategies for optimized robotic systems,” in Proc. ASME Int. Mech. Eng. Congr. Expo. (IMECE), ASME, 2024.
  27. N. M. Manousakis, P. S. Karagiannopoulos, G. J. Tsekouras, and F. D. Kanellos, “Integration of renewable energy and electric vehicles in power systems: A review,” Processes, vol. 11, no. 5, p. 1544, 2023.
  28. B. Javanmard, M. Tabrizian, M. Ansarian, and A. Ahmarinejad, “Energy management of multi-microgrids based on a game theory approach,” J. Energy Storage, vol. 42, p. 102971, 2021.
  29. A. R. Jordehi, “Two-stage stochastic programming for risk-aware scheduling of energy hubs participating in electricity markets,” Sustainable Cities Soc., vol. 81, p. 103823, 2022.
  30. S. Mahjoubi and Y. Bao, “Game theory-based metaheuristics for structural design optimization,” Comput.-Aided Civ. Infrastruct. Eng., vol. 36, no. 10, pp. 1337–1353, 2021.
  31. B. Zhang, W. Hu, A. M. Ghias, X. Xu, and Z. Chen, “Multiagent deep reinforcement learning-based distributed control architecture for interconnected multi-energy microgrids,” Energy Convers. Manag., vol. 277, p. 116647, 2023.
  32. A. Churkin, J. Bialek, D. Pozo, E. Sauma, and N. Korgin, “Review of cooperative game theory applications in power system expansion planning,” Renewable Sustain. Energy Rev., vol. 145, p. 111056, 2021.
  33. X. Xu, Y. Jia, Y. Xu, Z. Xu, S. Chai, and C. S. Lai, “A multi-agent reinforcement learning-based data-driven method for home energy management,” IEEE Trans. Smart Grid, vol. 11, no. 4, pp. 3201–3211, 2020.
  34. G. K. Venayagamoorthy, R. K. Sharma, P. K. Gautam, and A. Ahmadi, “Dynamic energy management system for a smart microgrid,” IEEE Trans. Neural Netw. Learn. Syst., vol. 27, no. 8, pp. 1643–1656, 2019.
  35. B. Lami, M. Alsolami, A. Alferidi, and S. B. Slama, “A smart microgrid platform integrating AI and deep reinforcement learning for sustainable energy management,” Energies, vol. 18, no. 5, p. 1157, 2025.
  36. F. D. Li, M. Wu, Y. He, and X. Chen, “Optimal control in microgrid using multi-agent reinforcement learning,” ISA Trans., vol. 51, no. 6, pp. 743–751, 2022.
  37. W. Liu, P. Zhuang, H. Liang, J. Peng, and Z. Huang, “Distributed economic dispatch in microgrids based on cooperative reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 6, pp. 2192–2203, 2019.
  38. R. B. Diddigi, C. Kamanchi, and S. Bhatnagar, “A generalized minimax Q-learning algorithm for two-player zero-sum stochastic games,” IEEE Trans. Autom. Control, vol. 67, no. 9, pp. 4816–4823, 2022.
  39. P. A. Tsividis, J. Loula, J. Burga, N. Foss, A. Campero, T. Pouncy, S. J. Gershman, and J. B. Tenenbaum,
    “Human-level reinforcement learning through theory-based modeling, exploration, and planning,” arXiv preprint, vol. arXiv:2107.12544, 2021.
  40. C. Guo, X. Wang, Y. Zheng, and F. Zhang, “Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning,” Energy, vol. 238, p. 121873, 2022.
  41. K. Deshpande, P. Möhl, A. Hämmerle, G. Weichhart, H. Zörrer, and A. Pichler, “Energy management simulation with multi-agent reinforcement learning: Reliability and resilience,” Energies, vol. 15, no. 19, p. 7381, 2022.
  42. M. Andreasson, D. V. Dimarogonas, H. Sandberg, and K. H. Johansson, “Distributed PI-control with applications to power systems frequency control,” in Proc. American Control Conf., pp. 3184–3189, IEEE, 2024.
  43. A. Al-Shetwi, M. Hannan, H. Al-Masri, and M. Sujod, “Latest advancements in smart grid technologies and their transformative role in shaping the power systems of tomorrow: An overview,” Progress in Energy, vol. 7, no. 1, p. 012004, 2024.
  44. R. Darshi, M. A. Bahreini, and S. A. Ebrahim, “Prediction of short-term electricity consumption using artificial neural networks,” in Proc. 5th Iranian Conf. Signal Process. Intell. Syst. (ICSPIS), IEEE, 2019.
  45. T. Chen, Z. Wang, and M. Zhou, “Diffusion policies creating a trust region for offline reinforcement learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 50098–50125, 2024.
  46. Y. Du and F. Li, “Intelligent multi-microgrid energy management based on deep neural networks and model-free reinforcement learning,” IEEE Trans. Smart Grid, vol. 11, no. 2, pp. 1066–1076, 2019.
  47. E. Foruzan, L. K. Soh, and S. Asgarpoor, “Reinforcement learning approach for optimal distributed energy management in a microgrid,” IEEE Trans. Power Syst., vol. 33, no. 5, pp. 5749–5758, 2020.
آمار
تعداد مشاهده مقاله: 310
تعداد دریافت فایل اصل مقاله: 233


سامانه مدیریت نشریات علمی. قدرت گرفته از سیناوب