<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.2 20190208//EN" "https://jats.nlm.nih.gov/publishing/1.2/JATS-journalpublishing1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:mml="http://www.w3.org/1998/Math/MathML" article-type="research-article" dtd-version="1.2" xml:lang="en">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">IJMABHR</journal-id>
<journal-title-group><journal-title>International Journal of Medical and All Body Health Research</journal-title><abbrev-journal-title>IJMABHR</abbrev-journal-title></journal-title-group>
<issn pub-type="epub">2582-8940</issn>
<publisher><publisher-name>Anfo Publication House</publisher-name></publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="publisher-id">278</article-id>
<article-id pub-id-type="doi">10.54660/IJMBHR.2025.6.1.01-11</article-id>
<title-group><article-title>Deep Learning-Based Real-Time Data Quality Assessment and Anomaly Detection for Large-Scale Distributed Data Streams</article-title></title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes"><name><surname>Zhang</surname><given-names>Hanqing</given-names></name><xref ref-type="aff" rid="aff1"/></contrib>
<contrib contrib-type="author"><name><surname>Jia</surname><given-names>Xuzhong</given-names></name><xref ref-type="aff" rid="aff2"/></contrib>
<contrib contrib-type="author"><name><surname>Chen</surname><given-names>Chen</given-names></name><xref ref-type="aff" rid="aff3"/></contrib>
<aff id="aff1"><label>1</label>Master of Science in Information Studies, Trine University, CA, USA</aff>
<aff id="aff2"><label>2</label>Computer Application Technology, Hunan University of Technology, HuNan, China</aff>
<aff id="aff3"><label>3</label>Communication and Information Systems, Nanjing University of Aeronautics and Astronautics, Nan Jing, China</aff>
</contrib-group>
<pub-date publication-format="electronic" date-type="pub"><year>2025</year></pub-date>
<volume>6</volume>
<issue>1</issue>
<fpage>01</fpage>
<lpage>11</lpage>
<history>
<date date-type="received"><day>10</day><month>11</month><year>2024</year></date>
<date date-type="accepted"><day>12</day><month>12</month><year>2024</year></date>
</history>
<permissions><license license-type="open-access" xlink:href="https://creativecommons.org/licenses/by-nc-sa/4.0/?utm"><license-p>CC BY-NC-SA 4.0</license-p></license></permissions>
<self-uri xlink:href="https://www.allmedicaljournal.com/article/278/deep-learning-based-real-time-data-quality-assessment-and-anomaly-detection-for-large-scale"/>
<self-uri content-type="pdf" xlink:href="https://www.allmedicaljournal.com/article/278/download"/>
<abstract><p>Time delay and data quality degradation pose significant challenges in large-scale distributed data streams processing. This paper proposes a deep learning-based real-time data quality assessment and anomaly detection method for distributed streaming data environments. The proposed approach integrates quality-aware feature extraction with adaptive deep neural networks to enable real-time quality monitoring and anomaly detection. A multi-dimensional quality assessment framework is developed, incorporating temporal-spatial correlations and stream characteristics for comprehensive quality evaluation. The system implements a distributed architecture with parallel processing capabilities, enabling scalable operations across multiple nodes while maintaining low-latency responses. A novel online learning mechanism is introduced to adapt model parameters dynamically, ensuring robust performance under evolving data patterns. Experimental evaluation conducted on three large-scale datasets, including industrial IoT sensors (2.5TB), network traffic (1.8TB), and financial transactions (3.2TB), demonstrates superior performance compared to traditional methods. The system achieves 97.8% detection accuracy while maintaining processing latency below 10ms, with linear scalability up to 128 nodes. Results show consistent performance improvement across different operational scenarios, with 95% precision in anomaly detection and throughput exceeding 1.2 million events per second.</p></abstract>
<kwd-group kwd-group-type="author"><kwd>Deep Learning</kwd><kwd>Distributed Data Streams</kwd><kwd>Quality Assessment</kwd><kwd>Real-time Anomaly Detection</kwd></kwd-group>
</article-meta>
</front>
<back><ref-list><title>References</title>
<ref id="R1"><label>1</label><mixed-citation>Yang Y, Xiao Y. A deep reinforcement learning-based optimal transmission control method for streaming videos. IEEE Access; c2024.</mixed-citation></ref>
<ref id="R2"><label>2</label><mixed-citation>Gao Y, Jin H, Wang B, Yang B, Yu W. An adaptive soft sensor method based on online deep evolving fuzzy system for industrial process data streams. In: 2023 IEEE 12th Data Driven Control and Learning Systems Conference (DDCLS). IEEE; 2023:1799-804.</mixed-citation></ref>
<ref id="R3"><label>3</label><mixed-citation>Chen X. Design of big data scheduling optimization algorithm based on deep reinforcement learning. In: 2024 International Conference on Telecommunications and Power Electronics (TELEPE). IEEE; 2024:88-93.</mixed-citation></ref>
<ref id="R4"><label>4</label><mixed-citation>Anitha M, Kumari VS, Pillai NM, Jayarin PJ, David DB. Exploring cutting-edge machine learning and data mining techniques for enhancing big data management with advanced algorithmic strategies for optimal data processing and analysis. In: 2024 Second International Conference on Advances in Information Technology (ICAIT). Vol. 1. IEEE; 2024:1-5.</mixed-citation></ref>
<ref id="R5"><label>5</label><mixed-citation>Gunturu V, Kumari PR, Chithra SM, Saikia B, Singh R, Singh DP. The emerging role of the knowledge-driven applications of wireless networks for next-generation online stream processing. In: 2022 11th International Conference on System Modeling &amp; Advancement in Research Trends (SMART). IEEE; 2022:567-71.</mixed-citation></ref>
<ref id="R6"><label>6</label><mixed-citation>Guanghe C, Zheng S, Liu Y. Real-time anomaly detection in dark pool trading using enhanced transformer networks. Journal of Knowledge Learning and Science Technology. 2024;3(4):320-9. ISSN: 2959-6386 (online).</mixed-citation></ref>
<ref id="R7"><label>7</label><mixed-citation>Chen J, Yan L, Wang S, Zheng W. Deep reinforcement learning-based automatic test case generation for hardware verification. Journal of Artificial Intelligence General Science (JAIGS). 2024;6(1):409-29. ISSN: 3006-4023.</mixed-citation></ref>
<ref id="R8"><label>8</label><mixed-citation>Zhang H, Pu Y, Zheng S, Li L. Enhancing facial micro-expression recognition in low-light conditions using attention-guided deep learning. Journal of Economic Theory and Business Management. 2024;1(5):12-22.</mixed-citation></ref>
<ref id="R9"><label>9</label><mixed-citation>Wang J, Lu T, Li L, Huang D. Enhancing personalized search with AI: a hybrid approach integrating deep learning and cloud computing. International Journal of Innovative Research in Computer Science &amp; Technology. 2024;12(5):127-38.</mixed-citation></ref>
<ref id="R10"><label>10</label><mixed-citation>Zhou S, Zheng W, Xu Y, Liu Y. Enhancing user experience in VR environments through AI-driven adaptive UI design. Journal of Artificial Intelligence General Science (JAIGS). 2024;6(1):59-82. ISSN: 3006-4023.</mixed-citation></ref>
<ref id="R11"><label>11</label><mixed-citation>Yang M, Huang D, Zhang H, Zheng W. AI-enabled precision medicine: optimizing treatment strategies through genomic data analysis. Journal of Computer Technology and Applied Mathematics. 2024;1(3):73-84.</mixed-citation></ref>
<ref id="R12"><label>12</label><mixed-citation>Wen X, Shen Q, Zheng W, Zhang H. AI-driven solar energy generation and smart grid integration: a holistic approach to enhancing renewable energy efficiency. International Journal of Innovative Research in Engineering and Management. 2024;11(4):55-66.</mixed-citation></ref>
<ref id="R13"><label>13</label><mixed-citation>Zhang Y, Bi W, Song R. Research on deep learning-based authentication methods for e-signature verification in financial documents. Academic Journal of Sociology and Management. 2024;2(6):35-43.</mixed-citation></ref>
<ref id="R14"><label>14</label><mixed-citation>Zhang Y, Liu Y, Zheng S. A graph neural network-based approach for detecting fraudulent small-value high-frequency accounting transactions. Academic Journal of Sociology and Management. 2024;2(6):25-34.</mixed-citation></ref>
<ref id="R15"><label>15</label><mixed-citation>Yu K, Shen Q, Lou Q, Zhang Y, Ni X. A deep reinforcement learning approach to enhancing liquidity in the US municipal bond market: an intelligent agent-based trading system. International Journal of Engineering and Management Research. 2024;14(5):113-26.</mixed-citation></ref>
<ref id="R16"><label>16</label><mixed-citation>Wang Y, Zhou Y, Ji H, He Z, Shen X. Construction and International Journal of Medical and All Body Health Research www.allmedicaljournal.com application of artificial intelligence crowdsourcing map based on multi-track GPS data. In: 2024 7th International Conference on Advanced Algorithms and Control Engineering (ICAACE). IEEE; 2024:1425-9.</mixed-citation></ref>
<ref id="R17"><label>17</label><mixed-citation>Lu T, Jin M, Yang M, Huang D. Deep learning-based prediction of critical parameters in CHO cell culture process and its application in monoclonal antibody production. International Journal of Advance in Applied Science Research. 2024;3:108-23.</mixed-citation></ref>
<ref id="R18"><label>18</label><mixed-citation>Zheng W, Yang M, Huang D, Jin M. A deep learning approach for optimizing monoclonal antibody production process parameters. International Journal of Innovative Research in Computer Science &amp; Technology. 2024;12(6):18-29.</mixed-citation></ref>
<ref id="R19"><label>19</label><mixed-citation>Bi W, et al. A dual ensemble learning framework for real-time credit card transaction risk scoring and anomaly detection. Journal of Knowledge Learning and Science Technology. 2024;3(4):330-9. ISSN: 2959-6386 (online).</mixed-citation></ref>
<ref id="R20"><label>20</label><mixed-citation>Ju C, Liu Y, Shu M. Performance evaluation of supply chain disruption risk prediction models in healthcare: a multi-source data analysis.</mixed-citation></ref>
<ref id="R21"><label>21</label><mixed-citation>Zheng H, Xu K, Zhang M, Tan H, Li H. Efficient resource allocation in cloud computing environments using AI-driven predictive analytics. Applied and Computational Engineering. 2024;82:6-12.</mixed-citation></ref>
<ref id="R22"><label>22</label><mixed-citation>Wang B, Zheng H, Qian K, Zhan X, Wang J. Edge computing and AI-driven intelligent traffic monitoring and optimization. Applied and Computational Engineering. 2024;77:225-30.</mixed-citation></ref>
<ref id="R23"><label>23</label><mixed-citation>Ma X, Lu T, Jin G. AI-driven optimization of rare disease drug supply chains: enhancing efficiency and accessibility in the US healthcare system.</mixed-citation></ref>
<ref id="R24"><label>24</label><mixed-citation>Ma D, Jin M, Zhou Z, Wu J. Deep learning-based ADL assessment and personalized care planning optimization in adult day health centers.</mixed-citation></ref>
<ref id="R25"><label>25</label><mixed-citation>Ju C, Liu Y, Shu M. Performance evaluation of supply chain disruption risk prediction models in healthcare: a multi-source data analysis.</mixed-citation></ref>
<ref id="R26"><label>26</label><mixed-citation>Lu T, Zhou Z, Wang J, Wang Y. A large language model-based approach for personalized search results re-ranking in professional domains. The International Journal of Language Studies. 2024;1(2):1-6. ISSN: 3078-2244.</mixed-citation></ref>
<ref id="R27"><label>27</label><mixed-citation>Ni X, Yan L, Ke X, Liu Y. A hierarchical Bayesian market mix model with causal inference for personalized marketing optimization. Journal of Artificial Intelligence General Science (JAIGS). 2024;6(1):378-96. ISSN: 3006-4023.</mixed-citation></ref>
<ref id="R28"><label>28</label><mixed-citation>Zhang H, Pu Y, Zheng S, Li L. AI-driven M&amp;A target selection and synergy prediction: a machine learning-based approach.</mixed-citation></ref>
<ref id="R29"><label>29</label><mixed-citation>Xie H, Zhang Y, Zhongwen Z, Zhou H. Privacy-preserving medical data collaborative modeling: a differential privacy enhanced federated learning framework. Journal of Knowledge Learning and Science Technology. 2024;3(4):340-50. ISSN: 2959-6386 (online).</mixed-citation></ref>
<ref id="R30"><label>30</label><mixed-citation>Zhou Z, Xia S, Shu M, Zhou H. Fine-grained abnormality detection and natural language description of medical CT images using large language models. International Journal of Innovative Research in Computer Science &amp; Technology. 2024;12(6):52-62.</mixed-citation></ref>
</ref-list></back>
</article>