Fine-Tuning Smaller Models with Use Llama 3.1 405B: A Game-Changer for Synthetic Data Generation and Distillation

Introduction

Generative artificial intelligence (AI) models have revolutionized the way we interact with machines. With the advent of large language models (LLMs), we can now generate human-like text, respond to queries, and even create synthetic data. In this article, we will explore the power of LLMs and how they can be used to fine-tune smaller models for improved performance.

Overview of Llama 3.1 405B

The Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pre-trained and instruction-tuned generative models in 8B, 70B, and 405B sizes (text in/text out). All models support long context length (128,000) and are optimized for inference with support for grouped query attention (GQA). The Llama 3.1 instruction-tuned text-only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the publicly available chat models on common industry benchmarks.

Prerequisites

The following prerequisites are needed to implement the steps outlined in this post:

Responses from the Llama 3 8B Instruct model

Firstly, we perform inference with the Llama 3 8B model either directly through Amazon Bedrock or a deployed endpoint using SageMaker JumpStart. With Llama 3 Instruct models, which are optimized for dialogue use cases, the input to the model endpoints is the previous history between the chat assistant and the user. We can ask context-aware questions to conversations that have happened so far, using specific formatting for the input text (described in our earlier Llama 3B release posts, Meta Llama 3 models are now available in Amazon Bedrock and Meta Llama 3 models are now available in Amazon SageMaker JumpStart).

In the following example, the user has a conversation with the assistant about tourist sites in Paris. The assistant generated four different recommendation options, and then the user inquires about the first option:

Input:
A board 7ft. 9 inches long is divided into 3 equal parts.
What is the length of each part??

This answer looks almost correct but not quite. The correct answer is 31 inches long. Similar logical questions are not answered correctly by the Llama 3 8B model.

In order for the Llama 3 8B model to improve its logical question answering capability, we want to fine-tune the model with data from the AQUA-RAT dataset. As we already mentioned, the AQUA-RAT dataset contains multiple choice options for the LLM to choose from. Because we don’t have the full answers for this dataset, we use the Llama 3.1 405B model to generate the verbal answer to the questions, and use that dataset to fine-tune the Llama 3 8B model.

Generate label data using Llama 3.1 405B

Because Llama 3.1 405B is the most capable of the Llama 3.1 collection of models, and because of its state-of-the-art math and general knowledge capabilities, we run direct inference of the questions in the AQUA-RAT dataset on Llama 3.1 405B using either SageMaker JumpStart or Amazon Bedrock. This helps us generate the answers we want to use to fine-tune the smaller Llama 3 8B models. In essence, we’re using Llama 3.1 405B as an alternative to human annotation to generate labels for the dataset. The following are example inference outputs from the 405B model:

Input:
A board 7ft. 9 inches long is divided into 3 equal parts.
What is the length of each part??

This output looks almost correct, but not quite. The correct answer is 31 inches long. Similar logical questions are not answered correctly by the Llama 3 8B model.

As a next step, we encourage you to use this idea along with the Llama-3.1 405B model in your use case to generate labels or even unlabeled data that can then be used by a smaller model downstream to help solve your use case.

Conclusion

In this article, we explored the power of large language models (LLMs) and how they can be used to fine-tune smaller models for improved performance. We demonstrated how to use the Llama 3.1 405B model to generate synthetic data for distillation, and how to fine-tune a smaller model like Llama 3 8B to generate better responses. We also provided a code notebook that you can use to run and test the solution.

Frequently Asked Questions

Q1: What is the Llama 3.1 405B model?

The Llama 3.1 405B model is a collection of pre-trained and instruction-tuned generative models in 8B, 70B, and 405B sizes (text in/text out). It is optimized for inference with support for grouped query attention (GQA) and is designed for multilingual dialogue use cases.

Q2: How can I use the Llama 3.1 405B model to fine-tune a smaller model?

You can use the Llama 3.1 405B model to generate synthetic data for distillation, and then fine-tune a smaller model like Llama 3 8B to generate better responses. This can be done using SageMaker JumpStart or Amazon Bedrock.

Q3: What is the difference between the Llama 3 8B and Llama 3.1 405B models?

The Llama 3 8B model is a smaller model that is optimized for dialogue use cases, while the Llama 3.1 405B model is a larger model that is optimized for multilingual dialogue use cases and has a wider range of capabilities.

Q4: How can I access the Llama 3.1 405B model?

The Llama 3.1 405B model is available on Amazon SageMaker JumpStart and Amazon Bedrock. You can access it through these platforms and use it to fine-tune smaller models or generate synthetic data.

Q5: What are the benefits of using the Llama 3.1 405B model?

The Llama 3.1 405B model has several benefits, including its ability to generate high-quality synthetic data, its ability to fine-tune smaller models for improved performance, and its wide range of capabilities. It is also optimized for multilingual dialogue use cases, making it a versatile tool for a wide range of applications.