LLMs on Kubernetes: Understanding Security Threats

Running Large Language Models (LLMs) on Kubernetes introduces unique security challenges that traditional container workloads don't face. These AI systems handle sensitive data, consume massive resources, and create new attack vectors that DevOps teams must understand and mitigate.

This guide explores the critical security threats when deploying LLMs on Kubernetes. You'll learn how to identify vulnerabilities and implement defensive strategies.

Why LLMs Create New Security Challenges

LLMs differ from typical applications in several ways that impact security:

Resource Intensity: LLMs require significant CPU, GPU, and memory resources. This creates opportunities for resource exhaustion attacks and makes monitoring more complex.

Data Sensitivity: These models often process personal information, proprietary data, or confidential business intelligence. A breach can expose training data or sensitive user interactions.

Model Complexity: LLMs are black boxes with billions of parameters. Understanding their behavior and potential vulnerabilities requires specialized knowledge.

Core Threat Categories for LLMs on Kubernetes

Data Poisoning and Model Manipulation

Attackers can compromise LLMs by injecting malicious data during training or fine-tuning phases.

Training Data Poisoning: Malicious actors insert carefully crafted examples into training datasets. This can cause models to produce biased, harmful, or incorrect outputs.

Model Weight Tampering: If attackers gain access to model files in persistent volumes, they can modify weights to alter behavior.


# Secure model storage with restricted access
apiVersion: v1
kind: PersistentVolume
metadata:
  name: llm-model-storage
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: encrypted-ssd

Prompt Injection Attacks

Users can craft malicious prompts to bypass safety controls or extract sensitive information.

Direct Injection: Attackers embed commands within user inputs to manipulate model responses.

Indirect Injection: Malicious content is retrieved from external sources and processed by the LLM, triggering unintended behaviors.


# Example of monitoring suspicious prompts
kubectl logs -l app=llm-service | grep -E "(ignore|forget|system|admin)" | wc -l

Resource Exhaustion Threats

LLMs consume substantial computational resources, making them targets for denial-of-service attacks.

GPU Memory Bombs: Attackers submit requests that consume excessive GPU memory, causing out-of-memory errors.

Inference Overload: Flooding the system with complex queries can overwhelm processing capacity.


# Resource limits for LLM containers
resources:
  limits:
    nvidia.com/gpu: 1
    memory: "32Gi"
    cpu: "8"
  requests:
    nvidia.com/gpu: 1
    memory: "16Gi"
    cpu: "4"

Kubernetes-Specific Attack Vectors

Container Escape Scenarios

LLM workloads often require privileged access for GPU operations. This increases container escape risks.

Privileged Containers: GPU drivers may require privileged mode, creating pathways to the host system.

Shared Kernel Vulnerabilities: Multiple LLM instances sharing the same node can exploit kernel vulnerabilities to access other containers.

Secrets and Configuration Exposure

LLMs require API keys, model licenses, and database credentials that attackers target.

Environment Variable Leakage: Sensitive configuration exposed through environment variables in pod specifications.

ConfigMap Misuse: Storing secrets in ConfigMaps instead of proper Secret objects.


# Proper secret management for LLM deployments
apiVersion: v1
kind: Secret
metadata:
  name: llm-api-keys
type: Opaque
data:
  openai-key: <base64-encoded-key>
  model-license: <base64-encoded-license>
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: llm-service
        env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: llm-api-keys
              key: openai-key

Network-Based Attacks

LLM services expose APIs that become targets for network-based attacks.

API Abuse: Attackers exploit poorly secured inference endpoints to extract model information or cause service disruption.

East-West Traffic Interception: Internal communication between LLM components can be intercepted without proper network policies.


# Network policy to restrict LLM service access
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: llm-service-policy
spec:
  podSelector:
    matchLabels:
      app: llm-service
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Monitoring and Detection Strategies

Behavioral Anomaly Detection

Monitor LLM outputs for unusual patterns that might indicate compromise.

Response Quality Degradation: Sudden drops in response quality could indicate model tampering.

Unexpected Output Patterns: Monitor for responses that violate expected behavior patterns.


# Monitor LLM response metrics
kubectl get --raw "/api/v1/namespaces/llm-namespace/pods/llm-pod/proxy:8080/metrics" | grep response_quality

Resource Usage Monitoring

Track resource consumption to detect attacks and ensure system stability.

GPU Utilization Spikes: Unusual GPU usage patterns may indicate malicious activity.

Memory Consumption Anomalies: Sudden memory usage changes can signal attacks or model issues.

Key Takeaways

LLMs on Kubernetes face unique security challenges that require specialized approaches:

Multi-layered Defense: Implement security controls at the model, container, and cluster levels
Resource Monitoring: Continuously monitor GPU, CPU, and memory usage for anomalies
Access Controls: Use proper RBAC, network policies, and secret management
Behavioral Analysis: Monitor model outputs for signs of compromise or manipulation

Understanding these threats is the first step toward securing LLM deployments. The next article in this series will cover specific implementation strategies for mitigating these risks.

Start by auditing your current LLM deployments against these threat categories. Identify gaps in your security posture and prioritize fixes based on your risk tolerance and compliance requirements.

LLMs on Kubernetes Part 1: Understanding the threat model