Real Time Mobility Intelligence: Evaluating Kafka Based Pipelines in Global Smart Transit Systems
DOI:
https://doi.org/10.61978/digitus.v3i4.959Keywords:
Real Time Analytics, Transit ETA, Kafka, GTFS Realtime, Anomaly Detection, Smart Cities, Streaming ArchitectureAbstract
Real-time streaming architectures are redefining the landscape of urban transit analytics by enabling low latency, data driven decision making. This study evaluates and compares the real time data processing capabilities of public transit systems in London, New York, and Singapore. The objective is to determine how architectural choices, data freshness, and machine learning integration influence key performance indicators such as latency, ETA accuracy, and anomaly detection. The methodology involves a multi city case study, where Kafka based pipelines integrated with Apache Flink and Spark were assessed for ingestion, processing, and service delivery. Datasets included GTFS Realtime, SIRI feeds, and contextual APIs (e.g., speed bands and crowd density). Metrics for evaluation included feed latency, mean absolute error (MAE) and root mean square error (RMSE) for ETA, and response times for anomaly detection. The results demonstrate that Singapore’s transit system outperformed its counterparts with the lowest latency (~12s), highest ETA accuracy (MAE = 18s; RMSE = 25s), and superior anomaly detection via multi sensor fusion. London and New York, while technologically robust, faced constraints due to longer feed update intervals and integration complexities. Kafka ML's online learning enhanced model adaptability, significantly reducing ETA prediction errors across dynamic conditions. Furthermore, stress testing revealed Singapore’s architecture as the most resilient under peak load. The study concludes that the effectiveness of real-time urban transit systems depends on harmonizing streaming infrastructure... Singapore’s architecture may serve as a potential reference model for other cities, while recognizing contextual differences in implementation. Singapore’s architecture offers a scalable template for other cities. Ethical considerations, including data governance and passenger privacy, are essential for sustainable implementation.
References
Alamri, H. (2023). Instructors’ Self-Efficacy, Perceived Benefits, and Challenges in Transitioning to Online Learning. Education and Information Technologies, 28(11), 15031–15066. https://doi.org/10.1007/s10639-023-11677-w DOI: https://doi.org/10.1007/s10639-023-11677-w
Ali, M., Anjum, A., Rana, O., Zamani, A., Balouek‐Thomert, D., & Parashar, M. (2022). RES: Real-Time Video Stream Analytics Using Edge Enhanced Clouds. Ieee Transactions on Cloud Computing, 10(2), 792–804. https://doi.org/10.1109/tcc.2020.2991748 DOI: https://doi.org/10.1109/TCC.2020.2991748
Becker, P. H. E., Arnau, J.-M., & González, A. (2020). Demystifying Power and Performance Bottlenecks in Autonomous Driving Systems. 205–215. https://doi.org/10.1109/iiswc50251.2020.00028 DOI: https://doi.org/10.1109/IISWC50251.2020.00028
Bibri, S. E. (2021). Data-Driven Smart Sustainable Cities of the Future: Urban Computing and Intelligence for Strategic, Short-Term, and Joined-Up Planning. Computational Urban Science, 1(1). https://doi.org/10.1007/s43762-021-00008-9 DOI: https://doi.org/10.1007/s43762-021-00008-9
Carnero, A., Martín, C., Torres, D., Garrido, D., Dı́az, M., & Rubio, B. (2021). Managing and Deploying Distributed and Deep Neural Models Through Kafka-Ml in the Cloud-to-Things Continuum. Ieee Access, 9, 125478–125495. https://doi.org/10.1109/access.2021.3110291 DOI: https://doi.org/10.1109/ACCESS.2021.3110291
Chaves, A. J., Martín, C., & Dı́az, M. (2023). The Orchestration of Machine Learning Frameworks With Data Streams and GPU Acceleration in Kafka‐ML: A Deep‐learning Performance Comparative. Expert Systems, 41(2). https://doi.org/10.1111/exsy.13287 DOI: https://doi.org/10.1111/exsy.13287
Choi, S., Shin, K., & Kim, H. (2020). End-to-End Latency Prediction for General-Topology Cut-Through Switching Networks. Ieee Access, 8, 13806–13820. https://doi.org/10.1109/access.2020.2966139 DOI: https://doi.org/10.1109/ACCESS.2020.2966139
Ding, W., Zhao, Z., Wang, J., & Li, H. (2020). Task Allocation in Hybrid Big Data Analytics for Urban IoT Applications. Acm/Ims Transactions on Data Science, 1(3), 1–22. https://doi.org/10.1145/3374751 DOI: https://doi.org/10.1145/3374751
Folorunsho, S. O., Adenekan, O. A., Ezeigweneme, C., Somadina, I. C., & Okeleke, P. A. (2024). Developing Smart Cities With Telecommunications: Building Connected and Sustainable Urban Environments. Engineering Science & Technology Journal, 5(8), 2492–2519. https://doi.org/10.51594/estj.v5i8.1441 DOI: https://doi.org/10.51594/estj.v5i8.1441
Hassan, N. R., Kim, S., & Lee, J. (2022). Hedonic motivation and continuance intention in mobile health: The moderating effect of social influence. International Journal of Medical Informatics, 159, 104688. https://doi.org/10.1016/j.ijmedinf.2021.104688
Kang, K., Chen, L., Yi, H., Wang, B., & Sha, M. (2017). Real-Time Information Derivation From Big Sensor Data via Edge Computing. Big Data and Cognitive Computing, 1(1), 5. https://doi.org/10.3390/bdcc1010005 DOI: https://doi.org/10.3390/bdcc1010005
Kitchin, R. (2016). The Ethics of Smart Cities and Urban Science. Philosophical Transactions of the Royal Society a Mathematical Physical and Engineering Sciences, 374(2083), 20160115. https://doi.org/10.1098/rsta.2016.0115 DOI: https://doi.org/10.1098/rsta.2016.0115
Kuiper, G., & Bekooij, M. J. (2017). Latency Analysis of Homogeneous Synchronous Dataflow Graphs Using Timed Automata. 902–905. https://doi.org/10.23919/date.2017.7927116 DOI: https://doi.org/10.23919/DATE.2017.7927116
Kyriakakis, E., Sparsø, J., Puschner, P., & Schoeberl, M. (2021). Synchronizing Real-Time Tasks in Time-Triggered Networks. 11–19. https://doi.org/10.1109/isorc52013.2021.00013 DOI: https://doi.org/10.1109/ISORC52013.2021.00013
Munshi, A. A., Alhindi, A., Qadah, T. M., & Alqurashi, A. (2023). An Electronic Commerce Big Data Analytics Architecture and Platform. Applied Sciences, 13(19), 10962. https://doi.org/10.3390/app131910962 DOI: https://doi.org/10.3390/app131910962
Patil, N. V., Krishna, C. R., & Kumar, K. (2022). KS-DDoS: Kafka Streams-Based Classification Approach for DDoS Attacks. The Journal of Supercomputing, 78(6), 8946–8976. https://doi.org/10.1007/s11227-021-04241-1 DOI: https://doi.org/10.1007/s11227-021-04241-1
Pourmoradnasseri, M., Khoshkhah, K., & Hadachi, A. (2023). Leveraging IoT Data Stream for Near‐real‐time Calibration of City‐scale Microscopic Traffic Simulation. Iet Smart Cities, 5(4), 269–290. https://doi.org/10.1049/smc2.12071 DOI: https://doi.org/10.1049/smc2.12071
Sallapalli, N. (2024). Enhancing Data Governance With Real-Time Analytics in Oil Production Forecasting. International Journal of Scientific Research in Computer Science Engineering and Information Technology, 10(6), 453–459. https://doi.org/10.32628/cseit24106187 DOI: https://doi.org/10.32628/CSEIT24106187
Severino, R., Simão, J., Datia, N., & Serrador, A. (2023). Protecting Hybrid ITS Networks: A Comprehensive Security Approach. Future Internet, 15(12), 388. https://doi.org/10.3390/fi15120388 DOI: https://doi.org/10.3390/fi15120388
Silva, B. N., Khan, M., Jung, C., Seo, J., Diyan, M., Han, J., Yoon, Y., & Han, K. (2018). Urban Planning and Smart City Decision Management Empowered by Real-Time Data Processing Using Big Data Analytics. Sensors, 18(9), 2994. https://doi.org/10.3390/s18092994 DOI: https://doi.org/10.3390/s18092994
Sreekanti, V., Subbaraj, H., Wu, C., Gonzalez, J. E., & Hellerstein, J. M. (2020). Optimizing Prediction Serving on Low-Latency Serverless Dataflow. https://doi.org/10.48550/arxiv.2007.05832
Thanasas, G. L., & Kampiotis, G. (2024). The Role of Big Data Analytics in Financial Decision-Making and Strategic Accounting. Technium Business and Management, 10, 17–33. https://doi.org/10.47577/business.v10i.11877 DOI: https://doi.org/10.47577/business.v10i.11877
Waltemate, T., Senna, I., Hülsmann, F., Rohde, M., Kopp, S., Ernst, M. O., & Botsch, M. (2016). The Impact of Latency on Perceptual Judgments and Motor Performance in Closed-Loop Interaction in Virtual Reality. 27–35. https://doi.org/10.1145/2993369.2993381 DOI: https://doi.org/10.1145/2993369.2993381
Wilbur, M., Pugliese, P., Lászka, Á., & Dubey, A. (2021). Efficient Data Management for Intelligent Urban Mobility Systems. 22–26. https://doi.org/10.1145/3459609.3460527 DOI: https://doi.org/10.1145/3459609.3460527
