Accelerate Generative AI Inference with NVIDIA NIM Microservices on Amazon SageMaker

Introduction

With the increasing adoption of artificial intelligence (AI) and machine learning (ML) in various industries, the need for scalable and efficient AI deployment solutions has grown. NVIDIA’s NIM (NVIDIA Inference Microservices) is a game-changer in this regard, enabling the deployment of industry-leading large language models (LLMs) on Amazon SageMaker Inference. In this article, we will explore the capabilities of NIM and its integration with SageMaker.

Announcing NIM Integration with SageMaker Inference

At the 2024 NVIDIA GTC conference, we announced support for NIM Inference Microservices in Amazon SageMaker Inference. This integration allows developers to deploy state-of-the-art LLMs on SageMaker and optimize their performance and cost. The optimized prebuilt containers enable the deployment of these models in minutes instead of days, facilitating their seamless integration into enterprise-grade AI applications.

NIM is built on technologies like NVIDIA TensorRT, NVIDIA TensorRT-LLM, and vLLM. NIM is engineered to enable straightforward, secure, and performant AI inferencing on NVIDIA GPU-accelerated instances hosted by SageMaker. This allows developers to take advantage of the power of these advanced models using SageMaker APIs and just a few lines of code, accelerating the deployment of cutting-edge AI capabilities within their applications.

Solution Overview

Getting started with NIM is straightforward. Within the NVIDIA API catalog, developers have access to a wide range of NIM optimized AI models that can be used to build and deploy their own AI applications. You can get started with prototyping directly in the catalog using the GUI or interact directly with the API for free.

Prerequisites

As a prerequisite, set up an Amazon SageMaker Studio environment:

1. Make sure the existing SageMaker domain has Docker access enabled. If not, run the following command to update the domain:

# update domain
aws --region region 
    sagemaker update-domain --domain-id domain-id 
    --domain-settings-for-update '{"DockerSettings": {"EnableDockerAccess": "ENABLED"}}'

2. After Docker access is enabled for the domain, create a user profile by running the following command:

aws --region region sagemaker create-user-profile 
    --domain-id domain-id 
    --user-profile-name user-profile-name

3. Create a JupyterLab space for the user profile you created.
4. After you create the JupyterLab space, run the following bash script to install the Docker CLI.

Set up your Jupyter notebook environment

For this series of steps, we use a SageMaker Studio JupyterLab notebook. You also need to attach an Amazon Elastic Block Store (Amazon EBS) volume of at least 300 MB in size, which you can do in the domain settings for SageMaker Studio. In this example, we use an ml.g5.4xlarge instance, powered by a NVIDIA A10G GPU.

We start by opening the example notebook provided on our JupyterLab instance, import the corresponding packages, and set up the SageMaker session, role, and account information:

Conclusion

In this post, we showed you how to get started with NIM on SageMaker for pre-built models. Feel free to try it out following the example notebook.

Frequently Asked Questions

Q: What is NIM?

NIM (NVIDIA Inference Microservices) is a set of inference microservices that brings the power of state-of-the-art LLMs to your applications, providing natural language processing (NLP) and understanding capabilities.

Q: What is LLM?

LLM (Large Language Model) is a type of artificial intelligence (AI) model that is trained on vast amounts of text data to generate human-like language outputs.

Q: How does NIM integrate with SageMaker?

NIM integrates with SageMaker to enable the deployment of industry-leading LLMs on SageMaker and optimize their performance and cost.

Q: What are the prerequisites for using NIM with SageMaker?

The prerequisites for using NIM with SageMaker include setting up an Amazon SageMaker Studio environment, enabling Docker access for the domain, creating a user profile, creating a JupyterLab space, and attaching an Amazon EBS volume of at least 300 MB in size.

Q: How do I get started with NIM on SageMaker?

You can get started with NIM on SageMaker by accessing the NVIDIA API catalog, where you can find a wide range of NIM optimized AI models that can be used to build and deploy your own AI applications.