Sarcouncil Journal of Multidisciplinary
Sarcouncil Journal of Multidisciplinary
An Open access peer reviewed international Journal
Publication Frequency- Monthly
Publisher Name-SARC Publisher
ISSN Online- 2945-3445
Country of origin- PHILIPPINES
Frequency- 3.6
Language- English
Keywords
- Social sciences, Medical sciences, Engineering, Biology
Editors

Dr Hazim Abdul-Rahman
Associate Editor
Sarcouncil Journal of Applied Sciences

Entessar Al Jbawi
Associate Editor
Sarcouncil Journal of Multidisciplinary

Rishabh Rajesh Shanbhag
Associate Editor
Sarcouncil Journal of Engineering and Computer Sciences

Dr Md. Rezowan ur Rahman
Associate Editor
Sarcouncil Journal of Biomedical Sciences

Dr Ifeoma Christy
Associate Editor
Sarcouncil Journal of Entrepreneurship And Business Management
Next-Gen ETL: Integrating Large Language Models with Data Orchestration Tools
Keywords: Large Language Models, ETL orchestration, adaptive data pipelines, AI-driven automation, cloud-native architecture.
Abstract: The increased size and complexity of enterprise data pipelines have revealed the shortcomings of old Extract-Transform-Load (ETL) systems, which are static, brittle, and too sluggish to keep pace with evolving data environments. Large Language Models (LLMs) such as GPT and Claude offer new opportunities to embed adaptive intelligence into orchestration platforms such as Azure Data Factory, Apache Airflow, and Prefect. This article examines the integration of LLMs in data orchestration platforms to dynamically build, optimize, and calibrate ETL logic as a function of evolving schemas, disparate data sources, and runtime anomalies. The article takes a design science perspective, pairing prototyping with case study assessments in various sectors. Results show that LLM-enhanced orchestration can decrease manual handling, speed up pipeline creation, enhance resilience against schema drift, and increase operational efficiency. Concurrently, trustworthiness, reproducibility, compliance, and scalability challenges are still pressing and call for research. By introducing a conceptual framework and architectural patterns for LLM embedding in ETL flows, this article puts LLM-driven orchestration at the center of next-generation data management for enterprises.
Author
- Abhinav Reddy Bobba
- Independent Researcher USA