Small language models (SLMs): A cheaper, greener route into AI

The proliferation of generative AI, exemplified by large language models (LLMs) such as GPT, LLama, PaLM and Claude, has sparked an ‘arms race’ among tech companies as they vie to develop ever-larger, more powerful systems.

Despite initial scepticism and caution, higher education systems and institutions have embraced the advantages of generative AI. However, it is crucial to acknowledge the flip side of this technological advancement. Leveraging LLMs demands substantial energy and infrastructure resources, presenting obstacles to both the environmental sustainability and adaptability of AI integration in higher education.

The introduction of small language models (SLMs), whether integrated into a larger ecosystem alongside LLMs or operating independently, offers immense potential in tackling these challenges.

Characterized by streamlined system architectures featuring reduced parameters and training datasets, recently rolled-out SLMs like Microsoft Phi-2 and Google Gemini Nano stand as capable language models which are notably smaller in size compared to their LLM counterparts.

The compact design of SLMs enables them to operate efficiently on less powerful hardware platforms, using a fraction of the power typically required. These SLMs are also easy to deploy locally. This improves accessibility and reduces the need for extensive infrastructure support.

SLMs may yet need to reach the raw performance levels of LLMs in handling highly complex tasks; however, initial reports suggest they can perform comparably well on narrower tasks if adequately trained and fine-tuned. This characteristic makes the technology particularly well-suited for industry-specific applications within well-defined knowledge domains.

As a result, pilot programmes to integrate SLMs have been launched in various industries, including legal, medical and financial services.

Higher education emerges as another domain where SLMs, serving as subject-specific generative AI tools, could wield substantial influence. Once trained on subject- or task-specific datasets and armed with customized algorithms, these models can be invaluable tools for supporting teaching, learning and research activities.

The journey towards carbon neutrality for higher education institutions involves three main steps: 1) establishing comprehensive carbon inventories; 2) monitoring and managing emissions; and 3) mitigating their impact through offsetting measures.

A comprehensive carbon footprint assessment should cover all facets of university operations, such as teaching, research, social engagement, institutional governance and digitization initiatives. Given generative AI’s significant environmental impact, special attention should be given to its sustainable integration within this matrix.

Since the introduction of LLMs, there has been a notable trend where users increasingly resort to these advanced technologies for solutions to given tasks regardless of the scale or specificity of their queries.

In many cases, such usage patterns can be compared to blasting a cannon to swat a mosquito, resulting in unnecessary internet traffic and heightened computing power and energy consumption, as each prompt may entail substantial processing behind the scenes.

The energy intensity of LLMs stems from their immersive ability to sift through vast data sources in pursuit of delivering versatile and accurate results. This capability involves the exhaustive examination and processing of large volumes of unfiltered data, which is essential to the effectiveness of the models, but demands considerable computational resources and leads to significant energy consumption.

While SLM technology is still in its infancy, it offers promising potential for higher education institutions seeking greener alternatives for their generative AI integrations. SLMs are well recognized for their lower energy consumption, providing a low-emission option for institutions looking to minimize their carbon footprint and reinforce their commitment to sustainability.

In regions and nations with limited digital infrastructure and resources, as in the Global South, the accessibility of SLMs is essential. The modest resource requirements of SLMs can indeed extend their availability to a wide range of users. This promising accessibility can effectively bridge the digital divide and facilitate equitable access to AI technology in higher education.

Although the Global North continues to dominate upstream technologies and platforms, Global South countries can use SLMs to cultivate home-grown AI tools and foster supportive local ecosystems. By leveraging local datasets and tailored algorithms, Global South countries could avoid falling behind in technological development and establish themselves as constructive forces in specific AI domains.

Embracing the development of SLMs could inspire Global South countries to create an enabling environment for nurturing indigenous AI talent and post-graduation employment opportunities. By prioritizing indigenous SLMs adapted to linguistic and cultural relevance, AI literacy and capacity-building programmes can equip individuals with the knowledge to apply AI solutions in their communities effectively.

SLMs present a strategic avenue for circumventing constraints in digital infrastructure development currently faced by Global South countries. By facilitating the development of AI applications rooted in local expertise, SLMs can foster the creation of technologies customized to the unique needs and challenges of these countries.

This localized approach to AI tool development is critical to promoting inclusivity in higher education, ensuring that educational resources and opportunities reach a wider population.

Efforts to create such SLM tools can, moreover, stimulate North-South, South-South and triangular cooperation. This will contribute to the emergence of a new global generative AI landscape that serves the common good of all nations.

Big data, machine learning algorithms and computing power are the cornerstones of generative AI capabilities. As the saying goes: ‘Whoever owns the data owns the future.’ LLMs such as GPT rely on aggregating data from multiple sources and interacting with user-provided information. However, this reliance can pose significant challenges for higher education institutions in terms of protecting privacy and data security.

In response to these concerns, it is essential to establish robust data governance frameworks that incorporate diverse dataset ownership models and stringent privacy and data security protocols.

The adaptability of SLMs for on-site deployment makes them particularly well-suited to this strategy, enabling institutions to maintain greater control over data usage. This is particularly important for AI applications involving student data and other sensitive information.

When properly trained and optimized with relevant datasets, SLMs become powerful tools from which higher education institutions can derive significant benefits. To unlock this potential, higher education institutions must establish comprehensive data management frameworks spanning institutional, faculty, subject and programme levels. Such an infrastructure is essential to enable the effective training of SLMs tailored to specific subjects or tasks.

Higher education institutions need to invest in robust data systems and prioritise their ethical management, particularly in developing and utilizing SLMs. This includes obtaining informed consent from campus constituents prior to any such data collection and transparent communication of its purposes, uses and privacy policies for informed decision-making. Transparency builds trust, and ethical standards are essential for the responsible use of SLMs.

In contrast to more generic and versatile LLMs, SLMs can be trained on datasets tailored to specific fields of study or teaching modalities. This customization results in outputs that are more relevant to learners’ needs and can directly improve alignment with learning objectives.

SLMs can effectively fulfil various academic needs for students seeking generative AI support. These include personalized learning, proofreading, research assistance and content generation. Students can view these AI tools as a rich repository of essential publications in their fields and beyond. Learners can also come to regard them as knowledgeable, supplementary mentors capable of efficiently delivering tailored and expert assistance.

For university faculty members, SLMs have the potential to streamline many labour-intensive tasks, allowing faculty to refine the models for customized purposes. In addition, SLMs can also serve as key partners in pedagogical enhancement and innovation.

Mentoring and supervision could reach more students using subject-based or task-specific SLM tools, especially in institutions with limited staff. As SLMs evolve, they have significant potential to inform, support and promote open educational resources, thereby facilitating wider dissemination of knowledge among institutions and students.

SLMs should not be seen as substitutes for LLMs but as complementary tools. Each has unique strengths and is well suited to specific purposes. As a result, institutions can optimise resource allocation by investing in a range of specialized SLMs tailored to specific needs and objectives rather than relying on a single, unwieldy LLM for all purposes.

Governments have a critical role to play in helping users to navigate AI models. They can establish regulatory frameworks that require the AI industry to prioritize sustainability, including energy efficiency certifications and transparent reporting of environmental impacts. With clear standards and data, higher education institution users can effectively assess trade-offs in deciding upon sustainable AI models.

Higher education institutions should adopt a balanced approach, integrating both LLMs and SLMs while prioritizing sustainability and purposeful AI integration. For example, LLMs may be essential for intricate or interdisciplinary research, while SLMs might excel at domain-specific tasks and everyday applications.

By defining and incorporating these various scenarios, higher education institutions can harness the transformative potential of AI while reducing their carbon footprint. As model capabilities evolve, regular reassessment of appropriate use cases is critical to ensure that integration strategies are adjusted as needed.

As highlighted in the outcome statement of the recent UNESCO Asia-Pacific roundtable on generative AI and education (2023), higher education institutions need to cultivate a culture of responsible and ethical use of AI. This includes raising awareness that all generative content prompts, queries and outputs have an environmental impact that is influenced by factors such as model scale, efficiency and aggregate usage levels.

Considering these factors, we can advance towards a more sustainable and customized AI integration in higher education.

This is a lightly adapted version of an article that first appeared at University World News on 16 March 2024.

#HigherEducation #FutureOfEducation #FutureOfLearning