Deep Learning-Based Real-Time Data Quality Assessment and Anomaly Detection for Large-Scale Distributed Data Streams
Article Information
Publication Timeline
- Received10 November 2024
- Accepted12 December 2024
Abstract
Time delay and data quality degradation pose significant challenges in large-scale distributed data streams processing. This paper proposes a deep learning-based real-time data quality assessment and anomaly detection method for distributed streaming data environments. The proposed approach integrates quality-aware feature extraction with adaptive deep neural networks to enable real-time quality monitoring and anomaly detection. A multi-dimensional quality assessment framework is developed, incorporating temporal-spatial correlations and stream characteristics for comprehensive quality evaluation. The system implements a distributed architecture with parallel processing capabilities, enabling scalable operations across multiple nodes while maintaining low-latency responses. A novel online learning mechanism is introduced to adapt model parameters dynamically, ensuring robust performance under evolving data patterns. Experimental evaluation conducted on three large-scale datasets, including industrial IoT sensors (2.5TB), network traffic (1.8TB), and financial transactions (3.2TB), demonstrates superior performance compared to traditional methods. The system achieves 97.8% detection accuracy while maintaining processing latency below 10ms, with linear scalability up to 128 nodes. Results show consistent performance improvement across different operational scenarios, with 95% precision in anomaly detection and throughput exceeding 1.2 million events per second.
References
- Yang Y, Xiao Y. A deep reinforcement learning-based optimal transmission control method for streaming videos. IEEE Access; c2024. [Google Scholar][Crossref]
- Gao Y, Jin H, Wang B, Yang B, Yu W. An adaptive soft sensor method based on online deep evolving fuzzy system for industrial process data streams. In: 2023 IEEE 12th Data Driven Control and Learning Systems Conference (DDCLS). IEEE; 2023:1799-804. [Google Scholar][Crossref]
- Chen X. Design of big data scheduling optimization algorithm based on deep reinforcement learning. In: 2024 International Conference on Telecommunications and Power Electronics (TELEPE). IEEE; 2024:88-93. [Google Scholar][Crossref]
- Anitha M, Kumari VS, Pillai NM, Jayarin PJ, David DB. Exploring cutting-edge machine learning and data mining techniques for enhancing big data management with advanced algorithmic strategies for optimal data processing and analysis. In: 2024 Second International Conference on Advances in Information Technology (ICAIT). Vol. 1. IEEE; 2024:1-5. [Google Scholar][Crossref]
- Gunturu V, Kumari PR, Chithra SM, Saikia B, Singh R, Singh DP. The emerging role of the knowledge-driven applications of wireless networks for next-generation online stream processing. In: 2022 11th International Conference on System Modeling & Advancement in Research Trends (SMART). IEEE; 2022:567-71. [Google Scholar][Crossref]
- Guanghe C, Zheng S, Liu Y. Real-time anomaly detection in dark pool trading using enhanced transformer networks. Journal of Knowledge Learning and Science Technology. 2024;3(4):320-9. ISSN: 2959-6386 (online). [Google Scholar][Crossref]
- Chen J, Yan L, Wang S, Zheng W. Deep reinforcement learning-based automatic test case generation for hardware verification. Journal of Artificial Intelligence General Science (JAIGS). 2024;6(1):409-29. ISSN: 3006-4023. [Google Scholar][Crossref]
- Zhang H, Pu Y, Zheng S, Li L. Enhancing facial micro-expression recognition in low-light conditions using attention-guided deep learning. Journal of Economic Theory and Business Management. 2024;1(5):12-22. [Google Scholar][Crossref]
- Wang J, Lu T, Li L, Huang D. Enhancing personalized search with AI: a hybrid approach integrating deep learning and cloud computing. International Journal of Innovative Research in Computer Science & Technology. 2024;12(5):127-38. [Google Scholar][Crossref]
- Zhou S, Zheng W, Xu Y, Liu Y. Enhancing user experience in VR environments through AI-driven adaptive UI design. Journal of Artificial Intelligence General Science (JAIGS). 2024;6(1):59-82. ISSN: 3006-4023. [Google Scholar][Crossref]
- Yang M, Huang D, Zhang H, Zheng W. AI-enabled precision medicine: optimizing treatment strategies through genomic data analysis. Journal of Computer Technology and Applied Mathematics. 2024;1(3):73-84. [Google Scholar][Crossref]
- Wen X, Shen Q, Zheng W, Zhang H. AI-driven solar energy generation and smart grid integration: a holistic approach to enhancing renewable energy efficiency. International Journal of Innovative Research in Engineering and Management. 2024;11(4):55-66. [Google Scholar][Crossref]
- Zhang Y, Bi W, Song R. Research on deep learning-based authentication methods for e-signature verification in financial documents. Academic Journal of Sociology and Management. 2024;2(6):35-43. [Google Scholar][Crossref]
- Zhang Y, Liu Y, Zheng S. A graph neural network-based approach for detecting fraudulent small-value high-frequency accounting transactions. Academic Journal of Sociology and Management. 2024;2(6):25-34. [Google Scholar][Crossref]
- Yu K, Shen Q, Lou Q, Zhang Y, Ni X. A deep reinforcement learning approach to enhancing liquidity in the US municipal bond market: an intelligent agent-based trading system. International Journal of Engineering and Management Research. 2024;14(5):113-26. [Google Scholar][Crossref]
- Wang Y, Zhou Y, Ji H, He Z, Shen X. Construction and International Journal of Medical and All Body Health Research www.allmedicaljournal.com application of artificial intelligence crowdsourcing map based on multi-track GPS data. In: 2024 7th International Conference on Advanced Algorithms and Control Engineering (ICAACE). IEEE; 2024:1425-9. [Google Scholar][Crossref]
- Lu T, Jin M, Yang M, Huang D. Deep learning-based prediction of critical parameters in CHO cell culture process and its application in monoclonal antibody production. International Journal of Advance in Applied Science Research. 2024;3:108-23. [Google Scholar][Crossref]
- Zheng W, Yang M, Huang D, Jin M. A deep learning approach for optimizing monoclonal antibody production process parameters. International Journal of Innovative Research in Computer Science & Technology. 2024;12(6):18-29. [Google Scholar][Crossref]
- Bi W, et al. A dual ensemble learning framework for real-time credit card transaction risk scoring and anomaly detection. Journal of Knowledge Learning and Science Technology. 2024;3(4):330-9. ISSN: 2959-6386 (online). [Google Scholar][Crossref]
- Ju C, Liu Y, Shu M. Performance evaluation of supply chain disruption risk prediction models in healthcare: a multi-source data analysis. [Google Scholar][Crossref]
- Zheng H, Xu K, Zhang M, Tan H, Li H. Efficient resource allocation in cloud computing environments using AI-driven predictive analytics. Applied and Computational Engineering. 2024;82:6-12. [Google Scholar][Crossref]
- Wang B, Zheng H, Qian K, Zhan X, Wang J. Edge computing and AI-driven intelligent traffic monitoring and optimization. Applied and Computational Engineering. 2024;77:225-30. [Google Scholar][Crossref]
- Ma X, Lu T, Jin G. AI-driven optimization of rare disease drug supply chains: enhancing efficiency and accessibility in the US healthcare system. [Google Scholar][Crossref]
- Ma D, Jin M, Zhou Z, Wu J. Deep learning-based ADL assessment and personalized care planning optimization in adult day health centers. [Google Scholar][Crossref]
- Ju C, Liu Y, Shu M. Performance evaluation of supply chain disruption risk prediction models in healthcare: a multi-source data analysis. [Google Scholar][Crossref]
- Lu T, Zhou Z, Wang J, Wang Y. A large language model-based approach for personalized search results re-ranking in professional domains. The International Journal of Language Studies. 2024;1(2):1-6. ISSN: 3078-2244. [Google Scholar][Crossref]
- Ni X, Yan L, Ke X, Liu Y. A hierarchical Bayesian market mix model with causal inference for personalized marketing optimization. Journal of Artificial Intelligence General Science (JAIGS). 2024;6(1):378-96. ISSN: 3006-4023. [Google Scholar][Crossref]
- Zhang H, Pu Y, Zheng S, Li L. AI-driven M&A target selection and synergy prediction: a machine learning-based approach. [Google Scholar][Crossref]
- Xie H, Zhang Y, Zhongwen Z, Zhou H. Privacy-preserving medical data collaborative modeling: a differential privacy enhanced federated learning framework. Journal of Knowledge Learning and Science Technology. 2024;3(4):340-50. ISSN: 2959-6386 (online). [Google Scholar][Crossref]
- Zhou Z, Xia S, Shu M, Zhou H. Fine-grained abnormality detection and natural language description of medical CT images using large language models. International Journal of Innovative Research in Computer Science & Technology. 2024;12(6):52-62. [Google Scholar][Crossref]
How to Cite
Zhang, H., Jia, X., & Chen, C. (2025). Deep Learning-Based Real-Time Data Quality Assessment and Anomaly Detection for Large-Scale Distributed Data Streams. International Journal of Medical and All Body Health Research, 6(1), 01-11. https://doi.org/10.54660/IJMBHR.2025.6.1.01-11
Zhang, Hanqing, et al. "Deep Learning-Based Real-Time Data Quality Assessment and Anomaly Detection for Large-Scale Distributed Data Streams." International Journal of Medical and All Body Health Research, vol. 6, no. 1, 2025, pp. 01-11. https://doi.org/10.54660/IJMBHR.2025.6.1.01-11.
Zhang, H., Jia, X. and Chen, C. (2025) 'Deep Learning-Based Real-Time Data Quality Assessment and Anomaly Detection for Large-Scale Distributed Data Streams', International Journal of Medical and All Body Health Research, 6(1), pp. 01-11. doi: 10.54660/IJMBHR.2025.6.1.01-11.
Zhang H, Jia X, Chen C. Deep Learning-Based Real-Time Data Quality Assessment and Anomaly Detection for Large-Scale Distributed Data Streams. International Journal of Medical and All Body Health Research. 2025;6(1):01-11. doi:10.54660/IJMBHR.2025.6.1.01-11
Similar Articles
- Multimodal Deep Learning for Intraoperative Hemodynamic Instability Prediction Vol 7, Issue 2 (2026)
- Machine Learning–Driven Clinical Decision Support Systems for Improving Patient Outcomes in US Healthcare Vol 4, Issue 4 (2023)
- Comparison of Elective Cesarean Section and Vaginal Delivery Outcomes: A Study of 100 Patients at Saraswathi Institute of Medical Sciences Vol 6, Issue 1 (2025)
- Classification, Importance, and Pathogenicity of Facklamia Species and Related Bacteria Vol 6, Issue 1 (2025)
- Knowledge and Attitudes towards Sexually Transmitted Infections among Medical Groups and Non-Medical Colleges at Baghdad University 2024 Vol 6, Issue 1 (2025)
Find this Article on
Google Scholar Semantic Scholar OpenAlex Scilit Crossref ResearchGate BASEA newly published article may take a few days to appear on these services.