Sarcouncil Journal of Multidisciplinary

Sarcouncil Journal of Multidisciplinary

An Open access peer reviewed international Journal
Publication Frequency- Monthly
Publisher Name-SARC Publisher

ISSN Online- 2945-3445
Country of origin- PHILIPPINES
Frequency- 3.6
Language- English

Keywords

Editors

Next-Gen ETL: Integrating Large Language Models with Data Orchestration Tools

Keywords: Large Language Models, ETL orchestration, adaptive data pipelines, AI-driven automation, cloud-native architecture.

Abstract: The increased size and complexity of enterprise data pipelines have revealed the shortcomings of old Extract-Transform-Load (ETL) systems, which are static, brittle, and too sluggish to keep pace with evolving data environments. Large Language Models (LLMs) such as GPT and Claude offer new opportunities to embed adaptive intelligence into orchestration platforms such as Azure Data Factory, Apache Airflow, and Prefect. This article examines the integration of LLMs in data orchestration platforms to dynamically build, optimize, and calibrate ETL logic as a function of evolving schemas, disparate data sources, and runtime anomalies. The article takes a design science perspective, pairing prototyping with case study assessments in various sectors. Results show that LLM-enhanced orchestration can decrease manual handling, speed up pipeline creation, enhance resilience against schema drift, and increase operational efficiency. Concurrently, trustworthiness, reproducibility, compliance, and scalability challenges are still pressing and call for research. By introducing a conceptual framework and architectural patterns for LLM embedding in ETL flows, this article puts LLM-driven orchestration at the center of next-generation data management for enterprises.

Home

Journals

Policy

About Us

Conference

Contact Us

EduVid
Shop
Wishlist
0 items Cart
My account