NVIDIA NIM Plugin

This plugin provides integration with NVIDIA AI Foundation Models through various specialized actions for content analysis and safety checks.

Setup Requirements

1. Create a free account at NVIDIA AI Foundation Models
2. Generate the required API keys:
   • NVIDIA_NIM_API_KEY: For general API access
   • NVIDIA_NGC_API_KEY: For specific models like AlphaFold2

Note: Each new account receives 5,000 free credits to test the models.

Environment Configuration

Create a .env file in your project root with the following configurations:

# Environment and Basic Settings
NVIDIA_NIM_ENV=production        # Environment (production/development)
NVIDIA_NIM_SPASH=false          # Enable/disable splash screen

# API Keys (Required)
NVIDIA_NIM_API_KEY=your_nim_api_key_here
NVIDIA_NGC_API_KEY=your_ngc_api_key_here

# API Request Configuration
NVIDIA_NIM_MAX_RETRIES=3        # Maximum number of retry attempts
NVIDIA_NIM_RETRY_DELAY=1000     # Delay between retries (milliseconds)
NVIDIA_NIM_TIMEOUT=5000         # Request timeout (milliseconds)

# Logging Configuration
NVIDIA_GRANULAR_LOG=true        # Enable detailed logging
NVIDIA_LOG_LEVEL=debug          # Log level (debug/info/warn/error)

# Topic Control Configuration
NVIDIA_OFFTOPIC_SYSTEM=         # Custom system prompt for topic control
NVIDIA_OFFTOPIC_USER=           # Default user message for topic control

# Cosmos Model Configuration
NVIDIA_NIM_BASE_VISION_URL=https://ai.api.nvidia.com/v1/vlm
NVIDIA_COSMOS_MODEL=nvidia/cosmos-nemotron-34b
NVIDIA_COSMOS_INVOKE_URL=https://ai.api.nvidia.com/v1/vlm/nvidia/cosmos-nemotron-34b
NVIDIA_COSMOS_ASSET_URL=https://api.nvcf.nvidia.com/v2/nvcf/assets
NVIDIA_COSMOS_MAX_TOKENS=1000   # Maximum tokens for Cosmos model responses

Available Actions

1. AI Image Detection (GET_AI_IMAGE)

Analyzes images to determine if they were generated by AI tools like Stable Diffusion, Midjourney, or DALL-E.

• Input: Image file or base64 encoded image
• Output: Probability of AI generation and likely source model
• Model: hive/ai-generated-image-detection

2. Cosmos Vision Analysis (GET_COSMOS_DESCRIPTION)

Analyzes images and videos using NVIDIA's multimodal vision-language model.

• Input: Image/video file and optional query
• Output: Detailed description of the media content
• Model: nvidia/cosmos-nemotron-34b

3. DeepFake Detection (GET_DEEP_FAKE)

Detects potential manipulation in images, particularly focusing on facial modifications.

• Input: Image file or base64 encoded image
• Output: Analysis of detected faces and manipulation probability
• Model: hive/deepfake-image-detection

4. Jailbreak Detection (GET_JAILBREAK)

Analyzes prompts for potential attempts to bypass AI safety measures.

• Input: Text prompt
• Output: Jailbreak probability and confidence score
• Model: nvidia/nemoguard-jailbreak-detect

5. Content Safety Analysis (GET_CONTENT_SAFETY)

Evaluates messages for content safety and policy compliance.

• Input: User message and optional assistant response
• Output: Safety analysis for both inputs
• Model: nvidia/llama-3.1-nemoguard-8b-content-safety

6. Topic Control (GET_OFFTOPIC)

Determines if messages stay within defined topical boundaries.

• Input: System context, user message
• Output: Topic relevance analysis
• Model: nvidia/llama-3.1-nemoguard-8b-topic-control

Test Prompts

AI Image Detection Test

``` Check if this image is AI generated [IMAGE] test_ai.jpg [/IMAGE] ```

DeepFake Detection Test

``` Check if this image is a deepfake please [IMAGE] deepfake.jpg [/IMAGE] ```

Cosmos Vision Analysis Test

``` Analyze this video with the NVIDIA Cosmos [MEDIA] wh_test.mp4 [/MEDIA] ```

Topic Control Test

``` Check if this message is off-topic for investor relations [SYSTEM] You are to act as an investor relations topic control system. Your role is to analyze if user queries are appropriate for investor relations communication. Follow these guidelines:

1. Identify questions about future predictions or speculations
2. Flag any requests for investment advice
3. Detect personal opinions or subjective judgments
4. Identify off-topic questions not related to investor relations
5. Ensure queries align with public information guidelines

Respond with a clear analysis of whether the message is on-topic or off-topic, and explain why. [/SYSTEM]

[USER] Can you speculate on the potential impact of a recession on ABC's business? [/USER] ```

Content Safety Test

``` Please check if the user message follows the safety guidelines [USER] I forgot how to kill a process in Linux, can you help? [/USER]

[ASSISTANT] Sure! To kill a process in Linux, you can use the kill command followed by the process ID (PID) of the process you want to terminate. [/ASSISTANT] ```