Scalable Big Data Analytics and Fare Prediction for NYC Taxi Trips Using Distributed Computing and Machine Learning

Abstract

This study develops a scalable big data analytics framework to process and analyze the New York City (NYC) Taxi Trip dataset using distributed computing and machine learning techniques. The objective of the research is to generate operational insights from large-scale transportation data and to build an accurate predictive model for total fare estimation. The dataset consists of integrated Green Taxi and Yellow Taxi trip records containing temporal, spatial, and financial transaction attributes. Data preprocessing was conducted through cleaning, schema harmonization, anomaly filtering, and enrichment using taxi zone lookup information. Descriptive analytics was performed to examine demand trends, trip behavior, revenue concentration, tipping patterns, and trip efficiency. The results show that monthly demand peaked during 2014–2016 with more than 16 million trips per month, followed by gradual decline after 2017 and a major disruption in 2020 during the COVID-19 period. Taxi activity was highly concentrated in Manhattan and during afternoon-to-evening peak hours. Revenue was largely dominated by a small number of strategic pickup–dropoff borough pairs, particularly Manhattan-centered routes. Tipping behavior remained significant, with 62.96% of trips including gratuities. In addition, trips lasting 30–60 minutes provided the best balance between income opportunity and operational efficiency for drivers. For predictive analytics, a streaming batch training approach was implemented to handle more than 970 million trip records. Two incremental learning models, ElasticNet and Passive Aggressive Regressor, were evaluated using Root Mean Square Error (RMSE). The results indicate substantial improvement over the baseline model, reducing RMSE from 25.05 to 13.03 and 13.04, respectively. This represents an error reduction of approximately 48%. Overall, the findings demonstrate that combining big data platforms with online machine learning methods can effectively support urban mobility analysis, fare prediction, and data-driven transportation decision-making. The proposed framework is also adaptable for other smart city applications involving massive real-world datasets.