Parallel computing means executing many calculations simultaneously, by dividing a large problem into smaller independent pieces and running them at the same time on multiple processors. A modern GPU like the NVIDIA RTX 5060 Ti contains 4,608 CUDA cores — all of them working in parallel on your data. Parallel computing is not a niche technique; it is the primary reason modern AI, graphics, and scientific computing are possible at the scale they operate today.
What does “parallel” really mean?
Imagine you need to stamp 10,000 envelopes. You could do it yourself, one at a time — that is serial (sequential) computing. Or you could hire 1,000 workers, each stamping 10 envelopes at the same time — that is parallel computing. Both approaches finish the same total work; parallel just finishes in a fraction of the time.
In computing terms, the “envelopes” are data elements — pixels, numbers, records, tokens — and the “workers” are processor cores. A CPU has 8 to 64 cores optimized for fast, smart decision-making on complex tasks. A GPU has thousands of simpler cores optimized for doing the same thing to millions of data elements simultaneously.
The key requirement for parallel computing is independence: the workers must not need each other’s results to do their share of the work. If each envelope can be stamped without knowing what the next one says, you can parallelize freely. In CUDA, we call these independent units of work threads.
Why did serial computing hit a wall?
For decades, chip makers increased performance by making transistors faster and smaller. Between 1970 and 2005, single-core CPU clock speeds grew from a few MHz to ~3–4 GHz. Engineers were effectively getting free speedups every two years just by waiting for a new chip generation — this was “Dennard scaling.”
Around 2005, Dennard scaling collapsed. Transistors became so small that making them faster generated too much heat. Clock speeds stalled. The industry pivoted: instead of one fast core, put many moderate-speed cores on the same die. Today’s top CPUs have 16–64 cores. GPUs took this idea much further: 4,608 cores on the RTX 5060 Ti, 21,760 cores on the H100.
The implication for software: programs that cannot run in parallel do not get faster on modern hardware. Writing parallel code stopped being optional sometime around 2010. CUDA is the most powerful tool for writing parallel code on NVIDIA GPUs.
What is the difference between serial and parallel execution?
Serial execution means one instruction completes before the next begins. A single processor core executes your loop iterations one by one:
Parallel execution means many instructions execute simultaneously. A GPU runs thousands of loop iterations at once:
How does a GPU implement parallelism?
A GPU organizes its work using a three-level hierarchy:
| Level | CUDA term | Typical count |
|---|---|---|
| Individual worker | Thread | millions per kernel |
| Group sharing fast memory | Block (thread block) | up to 1,024 threads |
| All blocks together | Grid | up to billions of threads |
When you launch a CUDA kernel, you specify how many threads to create. The GPU hardware maps those threads onto its physical cores and runs them concurrently.
You’ll learn about threads, blocks, and grids in depth later. For now, the important insight is: the GPU hides individual core latency by keeping thousands of threads ready to execute. Whenever one thread is waiting on memory, another thread runs. This is fundamentally different from how a CPU works.
What limits parallel speedup? (Amdahl’s Law)
Not every program benefits equally from parallelism. Amdahl’s Law quantifies the limit:
Speedup ≤ 1 / (S + P/N)
where:
- S = fraction of the program that must run serially (cannot be parallelized)
- P = fraction that can be parallelized (S + P = 1)
- N = number of parallel processors
Example: If 10% of your program is serial and 90% is parallel, the maximum speedup with 1,000 processors is 1 / (0.10 + 0.90/1000) ≈ 9.9x — not 1,000x. A 5% serial fraction caps you at 20x, forever, regardless of how many GPUs you add.
Common mistakes:
- Confusing total speedup with kernel speedup. Transfer time dominates for small workloads. Always measure with and without transfers to understand the real bottleneck.
- Assuming all workloads parallelize well. Algorithms with data-dependent branching, sequential dependencies, or tiny datasets often gain little from a GPU. Profile first; don’t assume.
- Ignoring Amdahl’s Law. Even a 1% serial fraction caps your speedup at 100x. In practice, overhead from kernel launches, synchronisation, and memory management adds up.
- Thinking “more threads = always faster.” Launching too many threads causes resource contention and can hurt performance. Choosing the right launch configuration matters.
- Skipping correctness checks. Parallel code has subtle bugs: race conditions, off-by-one errors in index calculations, or uninitialized memory. Always validate GPU results against a CPU reference.
- Not warming up the GPU. The first CUDA call on a fresh context incurs driver initialization overhead (up to hundreds of ms). Benchmark after a warm-up run.
When to use parallel computing — and when not to
Use it when:
- Your workload operates on large arrays or batches of independent data (image pixels, neural network weights, simulation cells, financial time series).
- The same operation applies to many elements (element-wise math, matrix operations, reductions).
- Throughput matters more than the latency of a single result.
Skip it (or GPU parallelism in particular) when:
- The dataset is tiny — PCIe transfer overhead will dwarf any compute savings.
- The algorithm is inherently sequential (each step depends on the previous result).
- Your code is already CPU-bound in a serial section that cannot be parallelized.
- Development time is precious and the workload does not justify GPU optimization effort.
FAQs
What is parallel computing in simple terms?
Parallel computing means breaking a large task into smaller pieces and solving them all at the same time using multiple processors. Instead of one worker doing everything sequentially, many workers tackle different parts simultaneously.
What is the difference between serial and parallel computing?
Serial computing executes one instruction at a time on a single processor. Parallel computing executes many instructions simultaneously across multiple processors or cores, reducing total time proportionally to the number of independent tasks.
Why does parallel computing matter for modern software?
Modern workloads — AI training, scientific simulation, video rendering, database queries — involve billions of operations on large datasets. Serial processors hit physical speed limits around 2005; additional performance now comes almost entirely from parallelism.
How much faster is GPU parallel computing than a CPU?
For highly parallel workloads with data already resident on the GPU, a modern GPU kernel can outperform a single-threaded CPU loop by 100x or more.
What is Amdahl’s Law and why does it limit parallel speedup?
Amdahl’s Law states that the maximum speedup of a program is limited by its serial fraction: speedup ≤ 1 / (serial_fraction + parallel_fraction / P). Even a 5% serial section caps speedup at 20x regardless of how many processors you add.
Is CUDA the only way to do GPU parallel computing?
No. CUDA is NVIDIA’s platform and the most widely used for high-performance GPU computing. Alternatives include HIP (AMD), SYCL/oneAPI (Intel and cross-vendor), OpenCL, and Metal (Apple).
Related tutorials in this series
- How do CPUs and GPUs differ architecturally?
- Why are GPUs good at throughput-oriented workloads?
- What is the difference between latency-oriented and throughput-oriented design?
- What is heterogeneous computing?
- How do Amdahl’s law and Gustafson’s law bound speedup?
References
- NVIDIA — CUDA Programming Guide — the authoritative reference for the CUDA programming model, kernel launches, and thread hierarchy. https://docs.nvidia.com/cuda/cuda-programming-guide/
- NVIDIA — CUDA C++ Best Practices Guide — practical performance advice including memory transfers and occupancy. https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/
- Gene Amdahl — “Validity of the Single Processor Approach to Achieving Large-Scale Computing Capabilities” (AFIPS, 1967) — the original formulation of Amdahl’s Law. https://dl.acm.org/doi/10.1145/1465482.1465560
- NVIDIA Developer Blog — “CUDA Refresher: The CUDA Programming Model” — gentle introduction to threads, blocks, and grids. https://developer.nvidia.com/blog/cuda-refresher-cuda-programming-model/
- John Gustafson — “Reevaluating Amdahl’s Law” (CACM, 1988) — explains the scaled-problem view of parallel efficiency. https://dl.acm.org/doi/10.1145/42411.42415
