Sarcouncil Journal of Engineering and Computer Sciences

Abstract: The widespread adoption of data lakehouse architectures has increased the demand for efficient ACID-compliant operations at scale. Traditional copy-on-write techniques—commonly employed in systems like Apache Hudi and Delta Lake—incur significant performance overhead by requiring entire files to be rewritten even for minimal row-level changes. This article presents an optimized partial copy-on-write mechanism integrated directly into the Apache Parquet file format, specifically engineered to support selective data updates. The solution introduces a fine-grained secondary row index that enables efficient localization of modified records to individual data pages, significantly minimizing I/O and computational costs during upserts. Through empirical benchmarking on production-scale datasets, the partial copy-on-write implementation demonstrates substantial improvement in write throughput compared to conventional strategies. The presented architecture maintains full ACID compliance while addressing performance bottlenecks in update-intensive workloads, enhancing the practical viability of data lakehouses for high-velocity, real-time analytics across distributed environments.