A Customized and efficient management of compute, storage, and communication in Big Data Processing and its optimization
Abstract
There are specific challenges like handling of variety datasets, velocity, and largest, high dimensional datasets to be handled efficiently in big data processing and its optimization. In this, three crucial aspects considered such as computing, computation storage, and communication in big data optimization. For achieving accurate and timely data assessment and analysis, these three aspects are focused on insights for rest to be assured for future endeavors. The strategies used on computing aspect enable to provide fast data access, effective allocation of resources, and parallel processing. The approaches considered are distributed frameworks, in-memory systems, lazy evaluations, partitioning, and selective algorithms. The second aspect is computation storage which requires hardware, software tools, Orchestration tools, Analytics tools, and data management tools for faster access, efficient data movements, enhanced scalability, and effective security. The third aspect of big data optimization is communication in which the approaches considered related to infrastructure, serialization, protocols, and notification/alerting systems to experience less overhead, proactive issue detection, and effective data exchange. This integrated aspects leads to efficient informed decision making based on insights and significant analysis. The customized framework on compute, communication services in big data optimization results accuracy, scalability, processing efficiency, and visualization analysis in better manner against existing approaches.
References
[1] Chandrima Roy et al (2018), Big Data Optimization Techniques: A Survey, July 2018, International Journal of Information Engineering and Electronic Business 10(4):41-48, DOI: 10.5815/ijieeb.2018.04.06
[2] M. Wu et al (2021), Research on the Optimization Algorithm of Big Data Computing System, 2021 International Wireless Communications and Mobile Computing (IWCMC), Harbin City, China, 2021, pp. 1783-1787, doi: 10.1109/IWCMC51323.2021.9498813.
[3] Hira Zahid et al (2019), Big data analytics in telecommunications: literature review and architecture recommendations, December 2019, PP(99):1-22, DOI: 10.1109/JAS.2019.1911795.
[4] Agnieszka Smalec, Big Data as a tool helpful in communication management, Procedia Computer Science, Volume 192, 2021, Pages 5156-5165,https://doi.org/10.1016/ j.procs.2021.09.293.
[5] Naghib, A. et al (2023), A comprehensive and systematic literature review on the big data management techniques in the internet of things, Wireless Netw 29, 1085–1144, 2023, https://doi.org/10.1007/s11276-022-03177-5.
[6] Madhavi Vaidya et al(2014), Design and Analysis of Large Data Processing Techniques, International Journal of Computer Applications , Volume 100 - No.8, DOI: 100:975-8887.
[7] Sandeep Dasari, Rajesh Kaluri (2023), "Big Data Analytics, Processing Models, Taxonomy of Tools, V’s, and Challenges: State-of-Art Review and Future Implications", Wireless Communications and Mobile Computing, vol. 2023, Article ID 3976302, 14 pages, 2023. https://doi.org/10.1155/2023/3976302.
[8] Rahat Iqbal et al (2021), Big Data analytics and Computational Intelligence for Cyber–Physical Systems: Recent trends and state of the art applications, Future Generation Computer Systems, Volume 105, April 2020, Pages 766-778, https://doi.org/10.1016/j.future.2017.10.021.
[9] Ahmed Hadi Ali AL-Jumaili et al (2023), Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations, Sensors 2023, 23(6), 2952; https://doi.org/10.3390/s23062952.
[10] Qingqing Chang et al (2022), Decision-Making and Computational Modeling of Big Data for Sustaining Influential Usage, Scientific Programming, vol. 2022, Article ID 2099710, 15 pages, 2022. https://doi.org/10.1155/2022/2099710.
[11] Agnieszka Smalec et al (2021), Big Data as a tool helpful in communication management, Procedia Computer Science, Volume 192, 2021, Pages 5156-5165, https://doi.org/10.1016/j.procs.2021.09.293.
[12] Shubham Upadhyay et al (2021), ANALYTICS AND STORAGE OF BIG DATA, International Semantic Intelligence Conference (ISIC 2021), https://ceur-ws.org/Vol-2786/Paper27.pdf.
[13] Cheng Lu (2021), Computer Data Storage and Management Platform Based on Big Data, J. Phys.: Conf. Ser. 2066 012022, https://iopscience.iop.org/article/10.1088/1742-6596/2066/1/012022/pdf.
[14] Berisha, B., Mëziu, E. & Shabani, I. (2022), Big data analytics in Cloud computing: an overview, J Cloud Comp 11, 24 (2022), https://doi.org/10.1186/s13677-022-00301-w.
[15] Amanpreet Kaur Sandhu (2022), Big Data with Cloud Computing: Discussions and Challenges, BIG DATA MINING AND ANALYTICS, Volume 5 , Number 1, DOI: 10.26599/BDMA.2021.9020016.
[16] Mohan Naik Ramachandra (2022), An Efficient and Secure Big Data Storage in Cloud Environment by Using Triple Data Encryption Standard, Big Data Cogn. Comput., 6(4), 101; https://doi.org/10.3390/bdcc6040101.
[17] Ahmed Hadi Ali AL-Jumaili et al (2023), Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations, Sensors 2023, 23(6), 2952; https://doi.org/10.3390/s23062952.
[18] Neelay Jagani et al(2021), BIG DATA IN CLOUD COMPUTING: A LITERATURE REVIEW, International Journal of Engineering Applied Sciences and Technology, March 2021, DOI: 10.33564/IJEAST.2021.v05i11.029.
[19] C. Stergiou and K. E. Psannis (2017), Algorithms for Big Data in Advanced Communication Systems and Cloud Computing, 2017 IEEE 19th Conference on Business Informatics (CBI), Thessaloniki, Greece, 2017, pp. 196-201, doi: 10.1109/CBI.2017.28.
[20] Akansha Gautam et al (2020), Big Data and Cloud Computing: A Critical Review, International Journal of Operations Research and Information Systems, 11(3):19-38, DOI: 10.4018/IJORIS.2020070102.
[21] Lingqi Xue (2021), Financial Big Data Based on Internet of Things and Wireless Network Communication, Wireless Communications and Mobile Computing, vol. 2021, Article ID 8944618, 12 pages, 2021, https://doi.org/10.1155/2021/8944618.
[22] Mahdavisharif, M., Jamali, S. & Fotohi, R. (2021), Big Data-Aware Intrusion Detection System in Communication Networks: a Deep Learning Approach, J Grid Computing 19, 46 (2021), https://doi.org/10.1007/s10723-021-09581-z.
[23] Dr. S. Hrushikesava Raju, Dr. L.R. Burra, Dr. A. Koujalagi, S.F. Waris (2020), Tourism enhancer app: user-friendliness of a map with relevant features, IOP Conf. Ser. Mater. Sci. Eng. 981, 2. https://doi.org/10.1088/1757-899X/981/2/022067.
[24] S. S. R. Jasti, V. Revanth, K. D. N. Rammohan Chowdary, K. C. S. V. Charan, S. H. Raju and S. Kavitha (2023), "Crop Intelligent: Weather based Crop Selection using Machine Learning," 2023 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS), Erode, India, 2023, pp. 1594-1600, doi: 10.1109/ICSCDS56580.2023.10104898.
[25] Hrushikesava Raju, S., Thrilok, S.S., Reddy, K.P.S.K., Karthikeya, G., Kumar, M.T. (2022), An IoT Vision for Dietary Monitoring System and for Health Recommendations. In: Ranganathan, G., Fernando, X., Shi, F. (eds) Inventive Communication and Computational Technologies, Lecture Notes in Networks and Systems, vol 311. Springer, Singapore. https://doi.org/10.1007/978-981-16-5529-6_65.
[26] S. Hrushikesava Raju, V. Lakshmi Lalitha, Praveen Tumuluru, N. Sunanda, S. Kavitha, Saiyed Faiayaz Waris (2024), Output-Oriented Multi-Pane Mail Booster, Smart Computing and Self-Adaptive Systems, CRC Press, 2021, 10.1201/9781003156123-4.
[27] S. Hrushikesava Raju, Lakshmi Ramani Burra, Saiyed Faiayaz Waris, V. Lakshmi Lalitha, S. Dorababu, S. Kavitha (2021), Eyesight Test through Remote Virtual Doctor Using IoT, Smart Computing and Self-Adaptive Systems, CRC Press, 2021, 10.1201/9781003156123-5.
[28] Dey N.S., Sangaraju H.K.R.(2024), A particle swarm optimization inspired global and local stability driven predictive load balancing strategy, Indonesian Journal of Electrical Engineering and Computer Science, Volume 35, Issue 3,10.11591/ijeecs.v35.i3.pp1688-1701.
[29] Dey N.S., Sangaraju H.K.R.(2023), Hybrid Load Balancing Strategy for Cloud Data Centers with Novel Performance Evaluation Strategy, International Journal of Intelligent Systems and Applications in EngineeringVolume 11, Issue 3, Pages 883 - 908, https://ijisae.org/index.php/IJISAE/article/view/3345.
[2] M. Wu et al (2021), Research on the Optimization Algorithm of Big Data Computing System, 2021 International Wireless Communications and Mobile Computing (IWCMC), Harbin City, China, 2021, pp. 1783-1787, doi: 10.1109/IWCMC51323.2021.9498813.
[3] Hira Zahid et al (2019), Big data analytics in telecommunications: literature review and architecture recommendations, December 2019, PP(99):1-22, DOI: 10.1109/JAS.2019.1911795.
[4] Agnieszka Smalec, Big Data as a tool helpful in communication management, Procedia Computer Science, Volume 192, 2021, Pages 5156-5165,https://doi.org/10.1016/ j.procs.2021.09.293.
[5] Naghib, A. et al (2023), A comprehensive and systematic literature review on the big data management techniques in the internet of things, Wireless Netw 29, 1085–1144, 2023, https://doi.org/10.1007/s11276-022-03177-5.
[6] Madhavi Vaidya et al(2014), Design and Analysis of Large Data Processing Techniques, International Journal of Computer Applications , Volume 100 - No.8, DOI: 100:975-8887.
[7] Sandeep Dasari, Rajesh Kaluri (2023), "Big Data Analytics, Processing Models, Taxonomy of Tools, V’s, and Challenges: State-of-Art Review and Future Implications", Wireless Communications and Mobile Computing, vol. 2023, Article ID 3976302, 14 pages, 2023. https://doi.org/10.1155/2023/3976302.
[8] Rahat Iqbal et al (2021), Big Data analytics and Computational Intelligence for Cyber–Physical Systems: Recent trends and state of the art applications, Future Generation Computer Systems, Volume 105, April 2020, Pages 766-778, https://doi.org/10.1016/j.future.2017.10.021.
[9] Ahmed Hadi Ali AL-Jumaili et al (2023), Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations, Sensors 2023, 23(6), 2952; https://doi.org/10.3390/s23062952.
[10] Qingqing Chang et al (2022), Decision-Making and Computational Modeling of Big Data for Sustaining Influential Usage, Scientific Programming, vol. 2022, Article ID 2099710, 15 pages, 2022. https://doi.org/10.1155/2022/2099710.
[11] Agnieszka Smalec et al (2021), Big Data as a tool helpful in communication management, Procedia Computer Science, Volume 192, 2021, Pages 5156-5165, https://doi.org/10.1016/j.procs.2021.09.293.
[12] Shubham Upadhyay et al (2021), ANALYTICS AND STORAGE OF BIG DATA, International Semantic Intelligence Conference (ISIC 2021), https://ceur-ws.org/Vol-2786/Paper27.pdf.
[13] Cheng Lu (2021), Computer Data Storage and Management Platform Based on Big Data, J. Phys.: Conf. Ser. 2066 012022, https://iopscience.iop.org/article/10.1088/1742-6596/2066/1/012022/pdf.
[14] Berisha, B., Mëziu, E. & Shabani, I. (2022), Big data analytics in Cloud computing: an overview, J Cloud Comp 11, 24 (2022), https://doi.org/10.1186/s13677-022-00301-w.
[15] Amanpreet Kaur Sandhu (2022), Big Data with Cloud Computing: Discussions and Challenges, BIG DATA MINING AND ANALYTICS, Volume 5 , Number 1, DOI: 10.26599/BDMA.2021.9020016.
[16] Mohan Naik Ramachandra (2022), An Efficient and Secure Big Data Storage in Cloud Environment by Using Triple Data Encryption Standard, Big Data Cogn. Comput., 6(4), 101; https://doi.org/10.3390/bdcc6040101.
[17] Ahmed Hadi Ali AL-Jumaili et al (2023), Big Data Analytics Using Cloud Computing Based Frameworks for Power Management Systems: Status, Constraints, and Future Recommendations, Sensors 2023, 23(6), 2952; https://doi.org/10.3390/s23062952.
[18] Neelay Jagani et al(2021), BIG DATA IN CLOUD COMPUTING: A LITERATURE REVIEW, International Journal of Engineering Applied Sciences and Technology, March 2021, DOI: 10.33564/IJEAST.2021.v05i11.029.
[19] C. Stergiou and K. E. Psannis (2017), Algorithms for Big Data in Advanced Communication Systems and Cloud Computing, 2017 IEEE 19th Conference on Business Informatics (CBI), Thessaloniki, Greece, 2017, pp. 196-201, doi: 10.1109/CBI.2017.28.
[20] Akansha Gautam et al (2020), Big Data and Cloud Computing: A Critical Review, International Journal of Operations Research and Information Systems, 11(3):19-38, DOI: 10.4018/IJORIS.2020070102.
[21] Lingqi Xue (2021), Financial Big Data Based on Internet of Things and Wireless Network Communication, Wireless Communications and Mobile Computing, vol. 2021, Article ID 8944618, 12 pages, 2021, https://doi.org/10.1155/2021/8944618.
[22] Mahdavisharif, M., Jamali, S. & Fotohi, R. (2021), Big Data-Aware Intrusion Detection System in Communication Networks: a Deep Learning Approach, J Grid Computing 19, 46 (2021), https://doi.org/10.1007/s10723-021-09581-z.
[23] Dr. S. Hrushikesava Raju, Dr. L.R. Burra, Dr. A. Koujalagi, S.F. Waris (2020), Tourism enhancer app: user-friendliness of a map with relevant features, IOP Conf. Ser. Mater. Sci. Eng. 981, 2. https://doi.org/10.1088/1757-899X/981/2/022067.
[24] S. S. R. Jasti, V. Revanth, K. D. N. Rammohan Chowdary, K. C. S. V. Charan, S. H. Raju and S. Kavitha (2023), "Crop Intelligent: Weather based Crop Selection using Machine Learning," 2023 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS), Erode, India, 2023, pp. 1594-1600, doi: 10.1109/ICSCDS56580.2023.10104898.
[25] Hrushikesava Raju, S., Thrilok, S.S., Reddy, K.P.S.K., Karthikeya, G., Kumar, M.T. (2022), An IoT Vision for Dietary Monitoring System and for Health Recommendations. In: Ranganathan, G., Fernando, X., Shi, F. (eds) Inventive Communication and Computational Technologies, Lecture Notes in Networks and Systems, vol 311. Springer, Singapore. https://doi.org/10.1007/978-981-16-5529-6_65.
[26] S. Hrushikesava Raju, V. Lakshmi Lalitha, Praveen Tumuluru, N. Sunanda, S. Kavitha, Saiyed Faiayaz Waris (2024), Output-Oriented Multi-Pane Mail Booster, Smart Computing and Self-Adaptive Systems, CRC Press, 2021, 10.1201/9781003156123-4.
[27] S. Hrushikesava Raju, Lakshmi Ramani Burra, Saiyed Faiayaz Waris, V. Lakshmi Lalitha, S. Dorababu, S. Kavitha (2021), Eyesight Test through Remote Virtual Doctor Using IoT, Smart Computing and Self-Adaptive Systems, CRC Press, 2021, 10.1201/9781003156123-5.
[28] Dey N.S., Sangaraju H.K.R.(2024), A particle swarm optimization inspired global and local stability driven predictive load balancing strategy, Indonesian Journal of Electrical Engineering and Computer Science, Volume 35, Issue 3,10.11591/ijeecs.v35.i3.pp1688-1701.
[29] Dey N.S., Sangaraju H.K.R.(2023), Hybrid Load Balancing Strategy for Cloud Data Centers with Novel Performance Evaluation Strategy, International Journal of Intelligent Systems and Applications in EngineeringVolume 11, Issue 3, Pages 883 - 908, https://ijisae.org/index.php/IJISAE/article/view/3345.
Authors
Vasantham, V. K., & Haritha, D. . (2024). A Customized and efficient management of compute, storage, and communication in Big Data Processing and its optimization. International Journal of Advanced Science and Computer Applications, 3(2). https://doi.org/10.47679/ijasca.v3i2.102
This work is licensed under a Creative Commons Attribution 4.0 International License.