Package Exports

aws-sagemaker-huggingface-llm
aws-sagemaker-huggingface-llm/lib/index.js

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (aws-sagemaker-huggingface-llm) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

Hugging Face LLM CDK Construct Library

The Hugging Face LLM CDK Construct Library provides constructs to easily deploy a Hugging Face LLM model to Amazon SageMaker.

Getting Started

install the library

npm install aws-sagemaker-huggingface-llm

Add construct

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { HuggingFaceLlm } from 'aws-sagemaker-huggingface-llm';

export class HuggingfaceCdkExampleStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // create new LLM SageMaker Endpoint
    new HuggingFaceLlm(this, 'Llama2Llm', {
      name: 'llama2-chat',
      instanceType: 'ml.g5.2xlarge',
      environmentVariables: {
        HF_MODEL_ID: 'NousResearch/Llama-2-7b-chat-hf',
        SM_NUM_GPUS: '1',
        MAX_INPUT_LENGTH: '2048',
        MAX_TOTAL_TOKENS: '4096',
        MAX_BATCH_TOTAL_TOKENS: '8192'
      }
    })
  }
}

Local test

npm run build

then test with

cdk synth --app='npx ts-node --prefer-ts-exts src/integ.default.ts' --profile xxx

deploy

cdk deploy --app='npx ts-node --prefer-ts-exts src/integ.default.ts' --profile xxxx

Acknowledgements

Big thank you to hayao-k for his blog post