Sarcouncil Journal of Engineering and Computer Sciences

Abstract: Transformer-based architectures introduced the field of natural language processing to revolution and ushered in not only large language models but also smaller language models. Smaller language models with one to five billion parameters have become viable to deploy in the enterprise with substantially better computational efficiency, cost-effectiveness, and privacy protection, and can still compete effectively in specialized tasks. Examples of such a paradigm include TinyLlama and Phi-2; these models have shown that quality training data curation, careful architecture, and optimization methods can be used to create capable language models that can operate within resource-constrained settings. The small models are used in a wide variety of applications in healthcare, educational, fitness, and software development, and are commonly implemented in a hybrid environment that uses small and large models together to trade off efficiency and capability. Nevertheless, the difficulties continue to occur in such aspects as complex thinking, factuality, and adherence to human preferences, which require strong validation structures and ongoing monitoring agents. Small language models have been demonstrated to be a vital step in making artificial intelligence more accessible to democracies, which can be deployed on devices, offline, and have less impact on the environment, at the cost of adequate performance to use in business-level applications.