Contents

White paper covering the most common issues related to NVIDIA GPUs.

• 1. Overview
• 2. Preface
  • 2.1. Who Should Read This Guide?
  • 2.2. Assess, Parallelize, Optimize, Deploy
    • 2.2.1. Assess
    • 2.2.2. Parallelize
    • 2.2.3. Optimize
    • 2.2.4. Deploy
  • 2.3. Recommendations and Best Practices
  • 2.4. Assessing Your Application
• 3. Heterogeneous Computing
  • 3.1. Differences between Host and Device
  • 3.2. What Runs on a CUDA-Enabled Device?
• 4. Application Profiling
  • 4.1. Profile
    • 4.1.1. Creating the Profile
    • 4.1.2. Identifying Hotspots
    • 4.1.3. Understanding Scaling
      • 4.1.3.1. Strong Scaling and Amdahl’s Law
      • 4.1.3.2. Weak Scaling and Gustafson’s Law
      • 4.1.3.3. Applying Strong and Weak Scaling
• 5. Parallelizing Your Application
• 6. Getting Started
  • 6.1. Parallel Libraries
  • 6.2. Parallelizing Compilers
  • 6.3. Coding to Expose Parallelism
• 7. Getting the Right Answer
  • 7.1. Verification
    • 7.1.1. Reference Comparison
    • 7.1.2. Unit Testing
  • 7.2. Debugging
  • 7.3. Numerical Accuracy and Precision
    • 7.3.1. Single vs. Double Precision
    • 7.3.2. Floating Point Math Is Not Associative
    • 7.3.3. IEEE 754 Compliance
    • 7.3.4. x86 80-bit Computations
• 8. Optimizing CUDA Applications
• 9. Performance Metrics
  • 9.1. Timing
    • 9.1.1. Using CPU Timers
    • 9.1.2. Using CUDA GPU Timers
  • 9.2. Bandwidth
    • 9.2.1. Theoretical Bandwidth Calculation
    • 9.2.2. Effective Bandwidth Calculation
    • 9.2.3. Throughput Reported by Visual Profiler
• 10. Memory Optimizations
  • 10.1. Data Transfer Between Host and Device
    • 10.1.1. Pinned Memory
    • 10.1.2. Asynchronous and Overlapping Transfers with Computation
    • 10.1.3. Zero Copy
    • 10.1.4. Unified Virtual Addressing
  • 10.2. Device Memory Spaces
    • 10.2.1. Coalesced Access to Global Memory
      • 10.2.1.1. A Simple Access Pattern
      • 10.2.1.2. A Sequential but Misaligned Access Pattern
      • 10.2.1.3. Effects of Misaligned Accesses
      • 10.2.1.4. Strided Accesses
    • 10.2.2. L2 Cache
      • 10.2.2.1. L2 Cache Access Window
      • 10.2.2.2. Tuning the Access Window Hit-Ratio
    • 10.2.3. Shared Memory
      • 10.2.3.1. Shared Memory and Memory Banks
      • 10.2.3.2. Shared Memory in Matrix Multiplication (C=AB)
      • 10.2.3.3. Shared Memory in Matrix Multiplication (C=AAT)
      • 10.2.3.4. Asynchronous Copy from Global Memory to Shared Memory
    • 10.2.4. Local Memory
    • 10.2.5. Texture Memory
      • 10.2.5.1. Additional Texture Capabilities
    • 10.2.6. Constant Memory
    • 10.2.7. Registers
      • 10.2.7.1. Register Pressure
  • 10.3. Allocation
  • 10.4. NUMA Best Practices
• 11. Execution Configuration Optimizations
  • 11.1. Occupancy
    • 11.1.1. Calculating Occupancy
  • 11.2. Hiding Register Dependencies
  • 11.3. Thread and Block Heuristics
  • 11.4. Effects of Shared Memory
  • 11.5. Concurrent Kernel Execution
  • 11.6. Multiple contexts
• 12. Instruction Optimization
  • 12.1. Arithmetic Instructions
    • 12.1.1. Throughput of Native Arithmetic Instructions
    • 12.1.2. Control Flow Instructions
    • 12.1.3. Synchronization Instruction
    • 12.1.4. Division Modulo Operations
    • 12.1.5. Loop Counters Signed vs. Unsigned
    • 12.1.6. Reciprocal Square Root
    • 12.1.7. Other Arithmetic Instructions
    • 12.1.8. Exponentiation With Small Fractional Arguments
    • 12.1.9. Math Libraries
    • 12.1.10. Precision-related Compiler Flags
  • 12.2. Memory Instructions
• 13. Control Flow
  • 13.1. Branching and Divergence
  • 13.2. Branch Predication
• 14. Deploying CUDA Applications
• 15. Understanding the Programming Environment
  • 15.1. CUDA Compute Capability
  • 15.2. Additional Hardware Data
  • 15.3. Which Compute Capability Target
  • 15.4. CUDA Runtime
• 16. CUDA Compatibility Developer’s Guide
  • 16.1. CUDA Toolkit Versioning
  • 16.2. Source Compatibility
  • 16.3. Binary Compatibility
    • 16.3.1. CUDA Binary (cubin) Compatibility
  • 16.4. CUDA Compatibility Across Minor Releases
    • 16.4.1. Existing CUDA Applications within Minor Versions of CUDA
      • 16.4.1.1. Handling New CUDA Features and Driver APIs
      • 16.4.1.2. Using PTX
      • 16.4.1.3. Dynamic Code Generation
      • 16.4.1.4. Recommendations for building a minor-version compatible library
      • 16.4.1.5. Recommendations for taking advantage of minor version compatibility in your application
• 17. Preparing for Deployment
  • 17.1. Testing for CUDA Availability
  • 17.2. Error Handling
  • 17.3. Building for Maximum Compatibility
  • 17.4. Distributing the CUDA Runtime and Libraries
    • 17.4.1. CUDA Toolkit Library Redistribution
      • 17.4.1.1. Which Files to Redistribute
      • 17.4.1.2. Where to Install Redistributed CUDA Libraries
• 18. Deployment Infrastructure Tools
  • 18.1. Nvidia-SMI
    • 18.1.1. Queryable state
    • 18.1.2. Modifiable state
  • 18.2. NVML
  • 18.3. Cluster Management Tools
  • 18.4. Compiler JIT Cache Management Tools
  • 18.5. CUDA_VISIBLE_DEVICES
• 19. Recommendations and Best Practices
  • 19.1. Overall Performance Optimization Strategies
• 20. nvcc Compiler Switches
  • 20.1. nvcc
• 21. Notices
  • 21.1. Notice
  • 21.2. OpenCL
  • 21.3. Trademarks