# 16. The GPU

> Chapter 16: run a counted loop on the graphics card with gpu for, reduce on the device, keep arrays on the GPU with gpu regions, and know the limits.

Canonical: https://alice-lang.org/documentation/learn/gpu/

[Documentation index](https://alice-lang.org/documentation/llms.txt) · [Complete book](https://alice-lang.org/llms-full.txt)

A CPU has a few dozen cores. A graphics card has thousands of small ones. For work that is the same simple operation on millions of numbers, such as image filters, physics steps or linear algebra, a GPU can be many times faster. Usually it needs a separate language, such as CUDA, and code to copy data back and forth.

In Alice, you write `gpu for` instead of `for`. The compiler turns the loop body into a GPU **kernel**, copies the arrays it needs, runs it, and copies the results back.

## 16.1 Setup

GPU support comes from [ILGPU](https://ilgpu.net), a .NET library that targets CUDA, OpenCL, and a CPU fallback. Add it to a project:

```sh
alice new gpu-demo
cd gpu-demo
alice add ILGPU
```

No GPU on your machine? Set `ALICE_GPU=cpu` and the same loops run on ILGPU's CPU accelerator, with the same results. That is also how continuous integration tests GPU code. The book's test suite runs the example below this way.

## 16.2 A first kernel

The classic first GPU program is SAXPY: `y = a * x + y` for every element.

```alice gpu-project="src/main.al" skip="needs the ILGPU package: built and run with ALICE_GPU=cpu by npm run test:projects"
def saxpy(a : Float32, xs : Array(Float32), ys : Array(Float32))
  gpu for i in 0...xs.size
    ys[i] = a * xs[i] + ys[i]
  end
end

def total(xs : Array(Int64)) : Int64
  t = 0_i64
  gpu for i in 0...xs.size
    t += xs[i] * 2_i64
  end
  t
end

xs = Array(Float32).new(8) { |i| i.to_f32 }
ys = Array(Float32).new(8, 1.0_f32)
saxpy(2.0_f32, xs, ys)
puts ys
puts total(Array(Int64).new(1000) { |i| i.to_i64 })
```

```text title="Output"
[1.0, 3.0, 5.0, 7.0, 9.0, 11.0, 13.0, 15.0]
999000
```

Each iteration of a `gpu for` runs as one GPU thread. That only works if iterations are independent, so the compiler applies the same proof as `par for` ([chapter 11](https://alice-lang.org/documentation/learn/parallelism.md)): each iteration writes only its own element.

`total` shows a **reduction**: `t += ...` into a variable outside the loop. The GPU adds in fast registers, combines once per group of threads, and adds each group's result to `t` once. Integer reductions are exact.

## 16.3 What a kernel may do

A GPU core is simple. The loop body may use:

- arrays of numbers, and numbers from the surrounding code;
- arithmetic, indexing, `Math.*` and number conversions;
- a `+=` reduction into an outer number.

Anything else, such as calling a method, allocating, or using strings, is a compile error that names the cause. The loop does not silently stay on the CPU.

Two differences from CPU code matter:

- **Kernel arithmetic is not overflow-checked.** On the GPU, integers wrap. Choose types that are large enough.
- **Floating-point results may differ in the last digits** from the CPU, because the device compiler may combine a multiply and an add into one instruction.

## 16.4 Keeping data on the device

Copying arrays between main memory and the GPU takes time, often more than the computation. A `gpu ... end` **region** keeps the arrays of all loops inside it on the device: each is uploaded once at the start, and the written ones are copied back once at `end`.

```alice skip="needs the ILGPU package"
def smooth(xs : Array(Float32), ys : Array(Float32), steps : Int32)
  gpu
    k = 0
    while k < steps
      gpu for i in 0...xs.size
        ys[i] = ys[i] + xs[i]
      end
      k += 1
    end
    gpu for i in 0...ys.size
      ys[i] = ys[i] * 0.5_f32
    end
  end
end
```

Inside the region, the device holds the current values, so reading `xs` or `ys` on the CPU before `end` is a compile error. The [GPU guide](https://alice-lang.org/documentation/guides/gpu.md) has measurements and the details of how launches are sized.

## 16.5 When to use the GPU

The GPU wins when there is a lot of uniform arithmetic per byte copied: millions of elements, several operations each. It loses on small arrays, on loops dominated by copying, and on code with many branches. Measure both ways: `par for` on the CPU is often fast enough, and it has no copies at all.

:::caution\[Common mistakes]

- **Forgetting the package.** Without `alice add ILGPU`, `gpu for` cannot compile.
- **Calling a method in the kernel.** Inline the arithmetic; kernels may only use numbers, arrays, `Math.*` and conversions.
- **Reading a device array inside a region.** Read it after `end`.
- **Relying on overflow checks in kernels.** GPU integers wrap. Use `Int64` where values can grow.
- **Expecting speed on small inputs.** Copies and launch cost dominate small loops.
  :::

## Exercises

1. Write a `gpu for` loop that squares every element of an `Array(Float32)` in place.
2. Write a kernel that counts how many elements of an `Array(Int32)` are even, using a reduction.
3. Why can `gpu for i in 1...n; a[i] = a[i - 1] + 1; end` not run on the GPU?

<details>
<summary>Answers</summary>

1. Answer:

   ```alice skip="needs the ILGPU package"
   def square_all(xs : Array(Float32))
     gpu for i in 0...xs.size
       xs[i] = xs[i] * xs[i]
     end
   end
   ```

2. Add 1 or 0 per element. A conditional expression is plain arithmetic for the kernel:

   ```alice skip="needs the ILGPU package"
   def count_even(xs : Array(Int32)) : Int64
     n = 0_i64
     gpu for i in 0...xs.size
       n += xs[i] % 2 == 0 ? 1_i64 : 0_i64
     end
     n
   end
   ```

3. Iteration `i` reads what iteration `i - 1` writes. GPU threads run at the same time, so the result would depend on timing. The compiler rejects the loop for the same reason as `par for` in chapter 11.

</details>

## Summary

- `gpu for` turns a loop body into a GPU kernel through ILGPU. Add the package with `alice add ILGPU`.
- `ALICE_GPU=cpu` runs the same loops on the CPU accelerator, for machines and CI without a GPU.
- Kernels use numbers, arrays, arithmetic, `Math.*` and `+=` reductions; anything else is a compile error.
- A `gpu ... end` region keeps arrays on the device across loops.
- Kernel integers wrap; measure before moving work to the GPU.

[Chapter 17](https://alice-lang.org/documentation/learn/native.md) goes the other way: down to C libraries and native code.

## Continue reading

- Previous: [15. Async code and actors](https://alice-lang.org/documentation/learn/async.md)
- Next: [17. Native code](https://alice-lang.org/documentation/learn/native.md)