# Tensors

> Tensor kernels in the Alice runtime: GEMV and GEMM, quantized int8/int4 weights, BFloat16 and Half, softmax, norms and activations, with no allocation per call.

Canonical: https://alice-lang.org/documentation/guides/tensors/

[Documentation index](https://alice-lang.org/documentation/llms.txt) · [Complete book](https://alice-lang.org/llms-full.txt)

Since 0.14 the runtime has tensor kernels for numeric and machine-learning code (ADR 0030). They are plain .NET APIs
in `Alice.Runtime`, so Alice code calls them with `using Alice.Runtime`.

```alice
using Alice.Runtime

w = Array(Float32).new(8)
x = Array(Float32).new(4)
y = Array(Float32).new(2)
Nn.Fill(w, 1)
Nn.Fill(x, 2)
Nn.Gemv(w, x, y, 2, 4)   # y = w (2x4) * x
puts y[0]                # 8.0
Nn.Softmax(y)
puts y[1]                # 0.5
```

## What is there

| API                                                         | What it does                                                                                                                              |
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `Nn.Dot`, `Nn.Gemv`, `Nn.MatMul`, `Nn.MatMulTransposed`     | dot product, matrix-vector (`Float32`, `BFloat16`, `Half`, int8, int4 weights), tiled parallel matrix multiply                            |
| `Nn.Add`, `Multiply`, `Scale`, `Axpy`, `Fill`, `Sum`, `Max` | elementwise operations and reductions                                                                                                     |
| `Nn.Exp`, `Silu`, `Gelu`, `Tanh`, `Relu`, `Softmax`         | activations, with a fast `exp`                                                                                                            |
| `Nn.RmsNorm`, `Nn.LayerNorm`                                | normalization                                                                                                                             |
| `Tensor<T>`                                                 | strided tensor over 64-byte aligned memory. `Slice`, `Narrow`, `Transpose` and `Reshape` are views that share memory; `Contiguous` copies |
| `TensorArena`                                               | one aligned block that tensors are carved from                                                                                            |
| `QuantizedMatrix`                                           | int8/int4 blocks of 32 values with a `Half` scale (the GGML Q8_0/Q4_0 layout), multiplied without dequantizing first                      |
| `MappedFile`                                                | memory-mapped spans and tensors over a file, with no copy                                                                                 |

## Performance

Kernels pick the widest SIMD the CPU has (AVX-512, AVX2, 128-bit vectors, scalar). Large GEMV and GEMM calls are
split across a persistent worker pool. No call allocates. On a 12-thread laptop, a 4096x4096 `Float32` GEMV takes
2.0 ms (a naive loop takes 17.7 ms); int8 takes 0.67 ms and int4 0.59 ms. The repository's `benchmarks/tensors`
measures them.

To run your own loops on the graphics card instead, see [GPU loops](https://alice-lang.org/documentation/guides/gpu.md).

## Continue reading

- Previous: [GPU loops](https://alice-lang.org/documentation/guides/gpu.md)
- Next: [WebAssembly](https://alice-lang.org/documentation/guides/webassembly.md)