8.8 — Hybrid architectures (Jamba, Samba, MoE-Mamba)

1. Why this session exists

§8.7 ended with the diagnosis: pure SSMs run at $O(1)$ per generated token but lose to Transformers at frontier scale on tasks that need exact in-context retrieval. Pure Transformers have $O(T)$ KV-cache memory and bandwidth, which limits long-context throughput. The natural response is a hybrid: interleave SSM layers (cheap per-token, lossy state) with attention layers (expensive per-token, exact recall). The hope is that you keep the cheap-decode property of SSMs at most layers while preserving Transformer-level recall accuracy from the few attention layers you keep.

In 2024 two designs validated this hope at scale: Jamba (AI21, arXiv:2403.19887) and Samba (Microsoft, arXiv:2406.07522). Both match dense Transformer quality at 7B–52B activated-parameter scales while delivering substantially better long-context throughput. A third design, MoE-Mamba (Pióro et al., arXiv:2401.04081), combines Mamba blocks with MoE FFN routing for a stacked sparsity story. This session surveys these, names the empirical ratios that worked, and frames the design choice for future model authors.

This is a short, survey-style session; the structural ideas are §8.7 and §8.6, and this is the integration.

2. The Jamba design (AI21, 2024)

Jamba is a $52$B-total / $12$B-activated MoE hybrid at $256$k context length. Its block structure is

Repeat for L blocks:
  if (block_index % 8) == 4:
      Attention block (Transformer-style: GQA-8 attention + FFN)
  else:
      Mamba block (selective SSM + gated MLP)

  Every 2nd block, the FFN is replaced by MoE (E=16, k=2).

So in a typical Jamba layer cycle of 8 blocks, 7 are Mamba and 1 is attention; half of those 8 use MoE FFN, half use dense FFN. The published configuration: $L = 32$ "Jamba blocks" each containing 8 sub-blocks ($= 256$ effective transformer-equivalent layers); $4$ of the $32$ blocks contain the attention sub-block.

The key ratio: 1 attention layer per 8 layers. The AI21 paper's main empirical finding: this is enough attention to preserve recall on long-context evaluations (Needle-in-a-Haystack at $256$k), while the remaining $7/8$ Mamba layers reduce KV-cache memory by $\sim 8\times$ vs an all-attention baseline at the same activated parameter count.

2.1 What the attention layers do

The Jamba paper attributes the attention layers' role to content retrieval: when the model needs to look up an exact past token (e.g., remember a name introduced 100k tokens ago), the Mamba state's lossy compression is insufficient, but a single attention layer can do the recall. The 7 surrounding Mamba layers do the local sequence processing — short-range syntax, position-relative pattern matching, etc. — that does not require exact retrieval.

This is consistent with mechanistic-interpretability findings on pure Transformers (6.4 §4) that a small number of "induction head" layers are responsible for the in-context-learning behaviour, and that ablating those specific layers degrades performance disproportionately. The hybrid view: a few attention layers are sufficient; the rest can be replaced by a more efficient sequence operator.

2.2 KV cache and long context

A pure-attention model at $T = 256k$ context spends most of its decode-step bandwidth reading the KV cache. With $1/8$ attention layers, Jamba's cache is $\sim 8\times$ smaller. At iso-throughput targets, Jamba serves $\sim 3\times$ more concurrent users per device than an equivalent dense Transformer.

The cache reduction is the operational headline; the quality preservation is the structural headline. Both matter for adoption.[1]

3. The Samba design (Microsoft, 2024)

Samba is a $7$B parameter dense hybrid at $\sim 1$M context. Its block alternates two operators:

Repeat for L blocks:
  if (block_index is even):
      Mamba block
  else:
      Sliding-window attention block (window 2048)

The differences from Jamba:

• Higher attention ratio. Samba uses $1$ attention per $2$ blocks (Jamba: $1$ per $8$). Half the layers are attention.
• Sliding-window attention. The attention layers use a fixed-window mask (size $2048$) rather than full causal attention. This keeps per-attention-layer KV-cache size $O(2048)$ regardless of $T$, so the cache does not grow.
• Pure dense, no MoE.

Samba's empirical claim: with sliding-window attention plus Mamba interleaving, the model achieves length generalisation up to $1$M tokens (the paper reports needle-in-haystack accuracy at $1$M context) while keeping the KV-cache constant in $T$. The all-Mamba state plus the bounded-window attention combine to give fully bounded memory at arbitrary $T$.

This is a stronger property than Jamba's: Jamba's attention layers still have full causal attention, so their cache grows linearly with $T$. Samba's bounded-window attention means the total cache is $O(1)$ in $T$ — true constant-memory streaming.

3.1 Why sliding window plus Mamba

A natural worry: sliding-window attention can only attend to the last $2048$ tokens; how does the model retain information from earlier? Samba's answer: the Mamba layers' state does carry information from arbitrarily far back (the SSM state is updated by every past token, just with exponential decay). So long-range information flows through the Mamba state; short-range exact retrieval flows through the windowed attention. The two are complementary.

A 1M-context Transformer with sliding-window-only attention would lose all information outside the window. A 1M-context pure-Mamba model would have lossy long-range recall. The hybrid recovers both.[2]

4. MoE-Mamba and other variants

MoE-Mamba (Pióro et al., 2024)[3] is the orthogonal combination: replace the gated MLP in a Mamba block with an MoE router-plus-experts. Activated parameters stay small (one or two experts per token); total parameters scale with $E$. The architecture is Mamba-only (no attention) but MoE-sparsified.

The paper reports that MoE-Mamba matches dense Mamba at iso-activated-FLOPs with substantially more total parameters, recovering the same total-vs-activated arithmetic that MoE provides for Transformers. It does not address the exact-retrieval gap of pure Mamba; for that, you would need to combine the MoE-Mamba block with attention layers, giving a triple-hybrid (Mamba + MoE + occasional attention). At least one industrial training run has been reported using this configuration; it is not yet in a publicly-released frontier model.

Other 2024–2026 hybrid designs that have appeared:

• Hymba (NVIDIA, 2024): parallel attention + SSM in the same block — both operators read the same input, their outputs are concatenated and projected. Removes the question of "which layers get which operator" by giving every layer both.
• Zamba (Zyphra, 2024): a shared attention block applied at every layer, alongside Mamba blocks. Reduces attention parameters by sharing weights.
• Griffin (DeepMind, 2024): hybrid of recurrent linear attention (a Mamba-flavoured operator) and local attention.

The space is exploratory; no single hybrid design has emerged as the consensus default the way GQA-$8$ has on the attention side. The general principle — combine attention with an efficient recurrent operator at some ratio between $1$/$8$ (Jamba) and $1$/$2$ (Samba) — is the stable observation.[4, 5, 6]

5. Why hybrids work: the inductive-bias argument

The post-hoc explanation that the field has converged on:

| Operator | Strength | Weakness | |---|---|---| | Attention | Exact, content-addressed retrieval over arbitrary distance | $O(T)$ memory and bandwidth | | SSM (selective) | Constant-memory state, $O(1)$ decode | Lossy compression of past; weak exact retrieval | | Sliding-window attention | $O(W)$ memory; exact retrieval within window | No retrieval beyond window |

The strengths and weaknesses are complementary: each operator's weakness is precisely the other's strength. A model with some of each can plausibly do everything each one does well, at the cost of some operational complexity.

This is reminiscent of the encoder-decoder vs decoder-only debate of §8.1–§8.2: one architecture is asymptotically right when scale is high enough, but at finite scale a mix of complementary inductive biases is empirically the strongest. As of 2026, the consensus is that hybrids are at the frontier; whether they remain the consensus at $10\times$ more compute is open.

6. What we keep going forward

1. Hybrid architectures interleave attention (or sliding-window attention) with SSM blocks. Typical ratios: $1/8$ (Jamba) to $1/2$ (Samba) attention layers, with the rest being Mamba.

2. Complementary inductive biases. Attention provides exact long-range recall; SSM provides constant-memory state. Together they cover both axes.

3. Long-context throughput is the headline win. At $T \geq 100k$, hybrid decode is $\sim 8\times$ cheaper than equivalent dense Transformer decode, with quality essentially preserved.

4. The space is exploratory in 2026. No single hybrid design has won; the principle that "some attention plus some SSM beats either alone" is stable; the exact ratios and arrangements are open.

Chapter 8's main path closes here. We have covered the 2026 frontier decoder-only block (§8.1), its near-neighbour encoder-only line (§8.2), the four refinements that dense Transformers use — GQA / MQA (§8.3), SwiGLU (§8.4), MLA (§8.5), MoE (§8.6) — and the recurrent alternative and its hybrid form (§8.7, §8.8). The remaining content is the encoder-decoder lineage (Appendix 8.A), which is what BERT's structural cousin grew into and where T5 still sees use in 2026.