ShipKit Build Memory Optimization Guide

This guide addresses out-of-memory (OOM) issues during builds, particularly with large dependencies like @huggingface/transformers and other memory-intensive libraries.

🚨 Common Build Issues

Symptoms

• Error: Command "pnpm run build" exited with SIGKILL
• Out of Memory errors during build
• Large WASM files (21.6 MB) won't be precached
• Build timeouts on Vercel/CI platforms

Root Causes

1. Large Dependencies: Libraries like @huggingface/transformers, three.js, remotion
2. WASM Files: Heavy WebAssembly files (especially from AI/ML libraries)
3. Insufficient Memory Allocation: Default Node.js memory limits
4. Build Configuration: Unoptimized webpack/Next.js settings

📊 Current Dependency Analysis

Run the bundle analyzer to understand your current situation:

npm run analyze:bundle

Memory-Intensive Dependencies

• @huggingface/transformers (21.6 MB WASM file)
• three.js and @react-three/* (3D graphics)
• remotion (video processing)
• @opentelemetry/* (monitoring)
• monaco-editor (code editor)
• playwright (browser automation)

🔧 Optimization Solutions

1. Node.js Memory Configuration

Local Development

# Use memory-optimized build
npm run build:memory-optimized

# Or with custom memory settings
NODE_OPTIONS="--max-old-space-size=6144 --max-semi-space-size=1024" npm run build

Vercel/Production

# Use Vercel-optimized build
npm run build:vercel

# Environment variables (set in Vercel dashboard)
NODE_OPTIONS="--max-old-space-size=8192 --max-semi-space-size=512"

2. Next.js Configuration Optimizations

The next.config.ts has been optimized with:

experimental: {
  // Memory optimization settings
  workerThreads: false,
  esmExternals: true,
  cpus: 1, // Limit to single CPU core
  isrMemoryCacheSize: 0,
  largePageDataBytes: 128 * 1000, // 128KB limit
}

3. Webpack Bundle Splitting

Optimized chunk splitting prevents large bundles:

splitChunks: {
  maxSize: 200000, // 200KB max chunk size
  cacheGroups: {
    // Split transformers into smaller chunks
    transformers: {
      test: /[\\/]node_modules[\\/]@huggingface[\\/]transformers[\\/]/,
      name: "transformers",
      chunks: "async", // Only load when needed
      priority: 15,
      maxSize: 50000, // Very small chunks
    },
    // Other optimizations...
  }
}

4. PWA Configuration for Large Files

Updated PWA config to handle large WASM files:

// src/config/nextjs/with-pwa.ts
const pwaConfig = {
  maximumFileSizeToCacheInBytes: 25 * 1024 * 1024, // 25MB
  exclude: [/\.wasm$/, /\.bin$/, /onnx/, /transformers/],
  runtimeCaching: [
    {
      urlPattern: /\.wasm$/,
      handler: "CacheFirst",
      options: {
        cacheName: "wasm-cache",
        expiration: {
          maxEntries: 10,
          maxAgeSeconds: 60 * 60 * 24 * 30, // 30 days
        },
      },
    },
  ],
};

5. Dependency Externalization

Heavy libraries are externalized on the server:

// next.config.ts
config.externals = [
  {
    "@huggingface/transformers": "@huggingface/transformers",
    three: "three",
    "@react-three/fiber": "@react-three/fiber",
    remotion: "remotion",
    "better-sqlite3": "better-sqlite3",
  },
];

⚠️ Important: Do NOT externalize sharp as it's required at runtime by:

• Payload CMS for image processing
• Image optimization services
• Server-side image transformations

🚀 Build Scripts

Available Scripts

{
  "build": "NODE_OPTIONS=\"--max-old-space-size=6144\" next build",
  "build:memory-optimized": "NODE_OPTIONS=\"--max-old-space-size=6144 --max-semi-space-size=1024\" next build",
  "build:vercel": "NODE_OPTIONS=\"--max-old-space-size=8192 --max-semi-space-size=512 --optimize-for-size\" next build",
  "analyze:bundle": "node scripts/analyze-bundle.js"
}

Usage

# Local development
npm run build:memory-optimized

# Production/Vercel
npm run build:vercel

# Analyze dependencies
npm run analyze:bundle

🔍 Monitoring and Troubleshooting

Memory Usage Monitoring

# Check memory before build
npm run prebuild

# Monitor Node.js memory
node --trace-gc --max-old-space-size=8192 node_modules/.bin/next build

# Analyze bundle size
npm run analyze:both

Debugging OOM Issues

1. Check Current Memory Usage:

   npm run analyze:bundle
   

2. Identify Large Dependencies:

   npx webpack-bundle-analyzer .next/static/chunks/*.js
   

3. Monitor Build Process:

   # Enable verbose logging
   DEBUG=* npm run build:memory-optimized
   

Common Solutions

Issue	Solution
WASM files too large	Increase maximumFileSizeToCacheInBytes in PWA config
Build OOM on Vercel	Use build:vercel script with higher memory limits
Large bundle size	Enable chunk splitting and externalization
Slow builds	Disable source maps in production
Memory leaks	Use --optimize-for-size flag

🎯 Best Practices

1. Dynamic Imports for Heavy Libraries

// Instead of static imports
import { AutoTokenizer } from "@huggingface/transformers";

// Use dynamic imports
const loadTransformers = async () => {
  const { AutoTokenizer } = await import("@huggingface/transformers");
  return AutoTokenizer;
};

2. Conditional Loading

// Only load AI features when needed
const AIComponent = lazy(() =>
  import("./ai-component").then((module) => ({
    default: module.AIComponent,
  }))
);

3. Environment-Based Optimization

// next.config.ts
const config = {
  experimental: {
    workerThreads: process.env.NODE_ENV === "development" ? true : false,
    cpus: process.env.NODE_ENV === "development" ? undefined : 1,
  },
};

4. Build Environment Configuration

Create .env.build with optimized settings:

NODE_OPTIONS="--max-old-space-size=8192 --max-semi-space-size=512 --optimize-for-size"
NEXT_TELEMETRY_DISABLED=1
SKIP_ENV_VALIDATION=1
NEXT_DISABLE_SOURCEMAPS=true

📋 Quick Fixes Checklist

• Use build:vercel script for production builds
• Increase Node.js memory limits (--max-old-space-size=8192)
• Enable chunk splitting for large libraries
• Externalize heavy dependencies on server
• Configure PWA for large WASM files
• Disable source maps in production
• Use dynamic imports for AI/ML libraries
• Monitor build with bundle analyzer
• Test build locally before deployment

🆘 Emergency Fixes

If you're experiencing immediate build failures:

1. Temporary workaround:

   # Disable problematic features
   DISABLE_HUGGINGFACE_TRANSFORMERS=true npm run build
   

2. Reduce memory usage:

   # Use single-threaded build
   NODE_OPTIONS="--max-old-space-size=8192 --optimize-for-size" npm run build
   

3. Skip non-essential features:

   # Skip linting and type checking
   SKIP_LINT=true SKIP_TYPE_CHECK=true npm run build
   

📚 Additional Resources

• Next.js Memory Optimization
• Webpack Bundle Splitting
• Node.js Memory Management
• Vercel Build Optimization

🤝 Contributing

If you discover new optimization techniques or encounter issues, please:

1. Test the solution locally
2. Update this documentation
3. Share findings with the team
4. Monitor build performance metrics

---

Last Updated: 2024-01-XX Build Environment: Node.js 18+, Next.js 15+, Vercel