24.1-dars: Attention mexanizmi

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

ODDIY LUG'AT (Python dict):
  narx = {"olma": 12, "anor": 25, "non": 4, "sut": 11}
  narx["anor"] -> 25
  so'rov kalitga AYNAN teng bo'lishi kerak, javob - BITTA qiymat

YUMSHOQ LUG'AT (attention):
  so'rov (query, q)  - "nima qidiryapman"
  kalitlar (keys, k_i) - "har bir yozuv nima haqida"
  qiymatlar (values, v_i) - "har bir yozuv nimani beradi"

  1. o'xshashlik:  skor_i = q . k_i          (skalyar ko'paytma)
  2. og'irlik:     a_i = softmax(skor)_i     (manfiy emas, yig'indisi 1)
  3. javob:        chiqish = yig'indi_i a_i * v_i

  so'rov biror kalitga juda o'xshash bo'lsa -> a_i ~ 1 -> oddiy lug'at
  bir nechta kalitga o'xshash bo'lsa -> qiymatlarning aralashmasi

1-MISOL: so'rov 'anor' kalitiga yaqin
  og'irliklar: olma 0.023, anor 0.880, non 0.045, sut 0.052
  natija 23.03 (haqiqiy narx 25 - boshqa narxlar bilan biroz aralashgan)

FORMULA:
  Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

SHAKLLAR (batch bilan):
  Q: (B, n_q, d_k)     n_q ta so'rov
  K: (B, n_k, d_k)     n_k ta kalit - so'rov bilan BIR XIL o'lcham d_k
  V: (B, n_k, d_v)     har kalitga bitta qiymat; d_v ixtiyoriy
  skor = Q K^T:        (B, n_q, n_k)
  og'irlik W:          (B, n_q, n_k)   softmax OXIRGI o'q (kalitlar) bo'yicha
  chiqish = W V:       (B, n_q, d_v)

QOIDALAR:
  - softmax kalitlar o'qi bo'yicha: har SO'ROV uchun og'irliklar yig'indisi 1
  - K va V bir xil sonli qator (n_k); Q qator soni boshqacha bo'lishi mumkin
  - d_v != d_k bo'lishi mumkin (2-bo'limdagi misol: d_k = 16, d_v = 4)
  - chiqish qatorlari soni = so'rovlar soni

HISOB NARXI:
  Q K^T:  n_q * n_k * d_k ko'paytma-qo'shish
  W V:    n_q * n_k * d_v
  og'irlik matritsasi xotirada: n_q * n_k son (har batch va bosh uchun)

23.11 (Luong, dot-product):
  s_t  - decoder holati         -> SO'ROV
  e_i  - encoder chiqishlari    -> KALIT va QIYMAT (bir xil tensor)
  skor = s @ enc^T; w = softmax(skor); kontekst = w @ enc
  = scaled_dot_product_attention(s, enc, enc, scale=1.0)

TRANSFORMERDA NIMA O'ZGARADI:
  1. Q, K, V - alohida o'rgatiladigan chiziqli proyeksiyalar:
       Q = X W_q,  K = X W_k,  V = X W_v
     kalit "nima bilan solishtirilaman" va qiymat "nima beraman" -
     endi bir-biridan mustaqil o'rganiladi
  2. masshtab 1/sqrt(d_k) (2.4-bo'lim)
  3. RNN yo'q: Q, K, V bitta ketma-ketlikdan olinishi mumkin
     (self-attention - keyingi dars)
  4. bir nechta attention parallel (multi-head - 24.3-dars)

FARAZ: q va k ning koordinatalari mustaqil, o'rtacha 0, dispersiya 1
  q . k = q_1 k_1 + ... + q_d k_d     - d_k ta mustaqil qo'shiluvchi
  har birining dispersiyasi 1  ->  Var(q . k) = d_k,  std = sqrt(d_k)

2-MISOL (o'lchov):
  d_k:           4      16     64     256    1024
  std(q . k):   1.75   4.02   8.21   15.90  31.83
  sqrt(d_k):    2.00   4.00   8.00   16.00  32.00

OQIBAT - SOFTMAX TO'YINISHI (32 kalit):
  skorlar orasidagi farq o'nlab birlik bo'lsa, exp ularni
  yuzlab-minglab marta farqlaydi -> deyarli one-hot
  d_k = 1024, xom skor:   maks og'irlik 0.953, samarali kalit soni 1.16
  d_k = 1024, sqrt bilan: maks og'irlik 0.161, samarali kalit soni 20.80

NIMA UCHUN BU YOMON:
  softmax Jacobiani J = diag(a) - a a^T
  a one-hot bo'lsa -> J ~ 0 -> skorlarga gradient deyarli o'tmaydi
  2-misol: d_k = 1024 da ||J|| xom 0.0629, sqrt bilan 0.2399 (3.8 marta)
  float32 da og'irliklarning 13.7% i AYNAN 0 ga aylandi
  o'rgatish boshida model "qayerga qarashni" o'zgartira olmaydi

YECHIM:
  skor / sqrt(d_k)  ->  dispersiya yana 1
  to'yinish d_k ga bog'liq bo'lmay qoladi (maks og'irlik 0.14-0.17)

MUAMMO: batchdagi jumlalar har xil uzunlikda -> <pad> bilan to'ldiriladi
  niqobsiz attention <pad> ni ham kalit deb biladi:
    - og'irlikning bir qismi paddingga ketadi
    - natija jumlaning O'ZIGA emas, batchdagi padding soniga bog'liq

YECHIM: skorni softmax DAN OLDIN -inf ga almashtirish
  skor[.., pad] = -inf  ->  exp(-inf) = 0  ->  og'irlik AYNAN 0
  qolgan og'irliklar yana yig'indisi 1 ga normallashadi
  natija = shu jumlani yolg'iz hisoblash natijasi (3-misol: farq ~1e-7)

MASKA SHAKLI:
  padding maskasi: (B, n_k) -> skorga (B, 1, n_k) sifatida qo'llanadi
  (har so'rov uchun bir xil kalitlar yopiladi)
  kauzal maska: (n_q, n_k) uchburchak - decoder darsida

KONVENSIYALAR (ENG XAVFLI JOY!):
  F.scaled_dot_product_attention, bool attn_mask:  True = QATNASHADI
  nn.MultiheadAttention, key_padding_mask:         True = E'TIBORSIZ
  float maska (ikkalasida):  skorga QO'SHILADI (0 yoki -inf)
  konvensiyani adashtirish xato bermaydi - model faqat padga qaraydi

BUTUNLAY YOPIQ QATOR:
  hamma kalit -inf -> softmax(-inf, ..., -inf) = 0/0 = NaN
  -1e9 bilan NaN yo'q, lekin og'irlik bir tekis - bu ham ma'nosiz
  bo'sh ketma-ketlikni batchga qo'ymang yoki chiqishini alohida nolga tenglang

softmax(x)_i = exp(x_i) / yig'indi exp(x_j)
  exp(1000) float64 da ham inf -> inf / inf = NaN

HIYLA: softmax(x) = softmax(x - max(x))
  surish natijani o'zgartirmaydi (surat va maxraj exp(-max) ga ko'payadi)
  endi eng katta daraja 0 -> exp <= 1, to'lib ketish yo'q

torch.softmax va F.scaled_dot_product_attention buni ichida qiladi;
o'zingiz numpy da yozsangiz - albatta qo'shing

23.11-DARSDAGI IKKI SKOR:
  dot-product (Luong):  skor = q . k
  additiv (Bahdanau):   skor = v^T tanh(W_q q + W_k k)

HISOB SHAKLI:
  dot-product: bitta matritsa ko'paytmasi Q K^T -> (n_q, n_k)
               BLAS/GPU uchun eng yaxshi optimallashtirilgan amal
  additiv:     har (so'rov, kalit) jufti uchun d o'lchamli vektor
               oraliq tensor (n_q, n_k, d) - xotirada d marta katta
               23.11: W_s(s)[:, :, None] + W_h(enc)[:, None] -> (B, t, T, h)

SIFAT:
  23.11, 3-misol: sana vazifasida ikkalasi ham 1.0000 aniq moslik
  kichik d da farq kam; katta d da masshtabsiz dot-product
  yomonlashadi - aynan shu sabab 1/sqrt(d_k) qo'shilgan

XULOSA:
  scaled dot-product = additiv sifatiga yaqin + ancha tez va tejamkor
  Transformer barcha qatlamlarda shuni ishlatadi

import torch
import torch.nn.functional as F

# qo'lda
skor = Q @ K.transpose(-2, -1) / Q.shape[-1] ** 0.5     # (B, n_q, n_k)
skor = skor.masked_fill(~maska[:, None, :], float("-inf"))  # maska: True = token
w = torch.softmax(skor, dim=-1)                          # kalitlar o'qi
chiqish = w @ V                                          # (B, n_q, d_v)

# tayyor funksiya (bir xil natija)
chiqish = F.scaled_dot_product_attention(Q, K, V)
chiqish = F.scaled_dot_product_attention(Q, K, V, attn_mask=maska[:, None, :])  # True = qatnashadi
chiqish = F.scaled_dot_product_attention(Q, K, V, scale=1.0)   # 23.11 dagi Luong

# float (qo'shiluvchi) maska
qosh = torch.zeros(maska.shape).masked_fill(~maska, float("-inf"))
chiqish = F.scaled_dot_product_attention(Q, K, V, attn_mask=qosh[:, None, :])

# padding maskasi uzunliklardan
maska = torch.arange(T)[None, :] < uz[:, None]           # (B, T) bool

# numpy barqaror softmax
e = np.exp(x - x.max(axis=-1, keepdims=True)); w = e / e.sum(axis=-1, keepdims=True)

Q - so'rov, K - kalit, V - qiymat; K va V bir xil n_k
skor = Q K^T / sqrt(d_k) -> softmax (kalitlar o'qi) -> W V
sqrt(d_k): skor dispersiyasi 1 da -> softmax to'yinmaydi
niqob: skorga -inf, softmax dan OLDIN
F.sdpa: True = qatnashadi; nn.MultiheadAttention: True = e'tiborsiz
23.11 dagi attention: Q = decoder, K = V = encoder, scale = 1

"""Scaled dot-product attention noldan: numpy, torch va F.scaled_dot_product_attention."""

import numpy as np
import torch
import torch.nn.functional as F


def softmax_np(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)        # barqarorlik uchun
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)


def attention_np(Q, K, V):
    d_k = Q.shape[-1]
    skor = Q @ np.swapaxes(K, -1, -2) / np.sqrt(d_k)   # (..., n_q, n_k)
    w = softmax_np(skor, axis=-1)
    return w @ V, w


def attention_torch(Q, K, V, scale=None):
    d_k = Q.shape[-1]
    scale = 1 / d_k ** 0.5 if scale is None else scale
    w = torch.softmax(Q @ K.transpose(-2, -1) * scale, dim=-1)
    return w @ V, w


def main() -> None:
    rng = np.random.default_rng(0)

    print("=== 1. Oddiy lug'at va 'yumshoq' lug'at ===")
    sozlar = ["olma", "anor", "non", "sut"]
    narx = {"olma": 12.0, "anor": 25.0, "non": 4.0, "sut": 11.0}
    print(f"  qattiq qidiruv: narx['anor'] = {narx['anor']}")
    K = rng.normal(size=(4, 8))                      # har so'z uchun kalit
    V = np.array([[narx[s]] for s in sozlar])        # qiymat - narx
    q = K[1] + 0.3 * rng.normal(size=8)              # 'anor' ga o'xshash so'rov
    chiq, w = attention_np(q[None], K, V)
    for s, wi in zip(sozlar, w[0]):
        print(f"    {s:<5} og'irlik {wi:.3f}")
    print(f"  yumshoq qidiruv natijasi: {chiq[0, 0]:.2f}  (og'irliklar yig'indisi {w.sum():.3f})")
    eng = sozlar[int(w[0].argmax())]
    print(f"  eng katta og'irlik: '{eng}' - so'rov unga eng o'xshash kalit")

    print("\n=== 2. Shakllar: batch, so'rovlar, kalitlar ===")
    B, n_q, n_k, d_k, d_v = 2, 3, 5, 16, 4
    Q = rng.normal(size=(B, n_q, d_k)).astype(np.float32)
    K = rng.normal(size=(B, n_k, d_k)).astype(np.float32)
    V = rng.normal(size=(B, n_k, d_v)).astype(np.float32)
    O, W = attention_np(Q, K, V)
    print(f"  Q {Q.shape}, K {K.shape}, V {V.shape}")
    print(f"  og'irliklar {W.shape}, chiqish {O.shape}")
    print(f"  har qator yig'indisi: min {W.sum(-1).min():.6f}, max {W.sum(-1).max():.6f}")
    print("  n_q va n_k har xil bo'lishi mumkin; d_v - d_k ga bog'liq emas")

    print("\n=== 3. numpy, torch va F.scaled_dot_product_attention mosligi ===")
    Qt, Kt, Vt = map(torch.from_numpy, (Q, K, V))
    O_t, _ = attention_torch(Qt, Kt, Vt)
    O_f = F.scaled_dot_product_attention(Qt, Kt, Vt)
    print(f"  numpy va torch:  maks farq {np.abs(O - O_t.numpy()).max():.2e}")
    print(f"  torch va F.sdpa: maks farq {(O_t - O_f).abs().max().item():.2e}")
    ok = torch.allclose(O_t, O_f, atol=1e-5)
    print(f"  allclose(atol=1e-5): {ok}")

    print("\n=== 4. 23.11-darsdagi attention - xuddi shu formula ===")
    torch.manual_seed(0)
    s = torch.randn(2, 6, 32)       # decoder holatlari (B, t, h)  -> so'rovlar
    enc = torch.randn(2, 10, 32)    # encoder chiqishlari (B, T, h) -> kalit va qiymat
    w_eski = torch.softmax(s @ enc.transpose(1, 2), dim=2)
    kontekst_eski = w_eski @ enc
    kontekst_yangi = F.scaled_dot_product_attention(s, enc, enc, scale=1.0)
    farq = (kontekst_eski - kontekst_yangi).abs().max().item()
    print("  Luong: Q = decoder holati, K = V = encoder chiqishi, masshtab 1")
    print(f"  kontekst shakli {tuple(kontekst_yangi.shape)}, maks farq {farq:.2e}")
    print("  Transformer farqi: Q, K, V - alohida o'rgatiladigan proyeksiyalar, 1/sqrt(d_k)")

    print("\n=== 5. Barqaror softmax: nega maksimum ayiriladi ===")
    skor = np.array([1000.0, 999.0, 990.0])
    with np.errstate(over="ignore", invalid="ignore"):
        sodda = np.exp(skor) / np.exp(skor).sum()
    print(f"  sodda softmax:    {np.round(sodda, 4)}")
    print(f"  barqaror softmax: {np.round(softmax_np(skor), 4)}")
    if np.isnan(sodda).any():
        print("  exp(1000) = inf -> inf/inf = nan; maksimumni ayirish natijani o'zgartirmaydi")
    print("  ⭐ Attention = so'rov-kalit o'xshashligi -> softmax -> qiymatlarning vaznli o'rtachasi")


if __name__ == "__main__":
    main()

=== 1. Oddiy lug'at va 'yumshoq' lug'at ===
  qattiq qidiruv: narx['anor'] = 25.0
    olma  og'irlik 0.023
    anor  og'irlik 0.880
    non   og'irlik 0.045
    sut   og'irlik 0.052
  yumshoq qidiruv natijasi: 23.03  (og'irliklar yig'indisi 1.000)
  eng katta og'irlik: 'anor' - so'rov unga eng o'xshash kalit

=== 2. Shakllar: batch, so'rovlar, kalitlar ===
  Q (2, 3, 16), K (2, 5, 16), V (2, 5, 4)
  og'irliklar (2, 3, 5), chiqish (2, 3, 4)
  har qator yig'indisi: min 1.000000, max 1.000000
  n_q va n_k har xil bo'lishi mumkin; d_v - d_k ga bog'liq emas

=== 3. numpy, torch va F.scaled_dot_product_attention mosligi ===
  numpy va torch:  maks farq 8.21e-08
  torch va F.sdpa: maks farq 0.00e+00
  allclose(atol=1e-5): True

=== 4. 23.11-darsdagi attention - xuddi shu formula ===
  Luong: Q = decoder holati, K = V = encoder chiqishi, masshtab 1
  kontekst shakli (2, 6, 32), maks farq 4.77e-07
  Transformer farqi: Q, K, V - alohida o'rgatiladigan proyeksiyalar, 1/sqrt(d_k)

=== 5. Barqaror softmax: nega maksimum ayiriladi ===
  sodda softmax:    [nan nan nan]
  barqaror softmax: [0.731  0.2689 0.    ]
  exp(1000) = inf -> inf/inf = nan; maksimumni ayirish natijani o'zgartirmaydi
  ⭐ Attention = so'rov-kalit o'xshashligi -> softmax -> qiymatlarning vaznli o'rtachasi

"""Nega sqrt(d_k) ga bo'linadi: skor dispersiyasi, softmax to'yinishi va gradient."""

import numpy as np
import torch


def jacobian_norm(w):
    """Softmax Jacobiani J = diag(w) - w w^T ning Frobenius normasi (har qator uchun)."""
    diag = w.pow(2).sum(-1)                   # ||diag(w)||^2
    ww = w.pow(2).sum(-1).pow(2)              # ||w w^T||^2
    cross = w.pow(3).sum(-1)                  # <diag(w), w w^T>
    return (diag - 2 * cross + ww).clamp_min(0).sqrt()


def statistika(skor):
    w = torch.softmax(skor, dim=-1)
    maks = w.max(-1).values.mean().item()
    ent = -(w * torch.log(w.clamp_min(1e-30))).sum(-1)
    samarali = ent.exp().mean().item()            # "nechta kalitga qaraydi"
    nol = (w == 0).float().mean().item()
    jac = jacobian_norm(w).mean().item()
    return maks, samarali, nol, jac


def main() -> None:
    torch.manual_seed(0)
    n_q, n_k = 2000, 32

    print("=== 1. Skor dispersiyasi d_k bilan o'sadi ===")
    print(f"  {'d_k':>5} {'std(q.k)':>9} {'sqrt(d_k)':>10} {'std(q.k)/sqrt(d_k)':>19}")
    olchamlar = [4, 16, 64, 256, 1024]
    for d in olchamlar:
        q = torch.randn(n_q, d)
        k = torch.randn(n_k, d)
        s = q @ k.T
        print(f"  {d:>5} {s.std().item():>9.2f} {d ** 0.5:>10.2f} "
              f"{(s / d ** 0.5).std().item():>19.3f}")
    print("  q, k ~ N(0, 1): q.k - d_k ta mustaqil ko'paytma yig'indisi, dispersiya d_k")

    print("\n=== 2. Softmax to'yinishi: masshtabsiz va sqrt(d_k) bilan ===")
    print(f"  {'d_k':>5} {'':>6} {'maks ogirlik':>13} {'samarali kalit':>15} "
          f"{'nol ogirlik':>13} {'||J||':>8}")
    natija = {}
    for d in olchamlar:
        q = torch.randn(n_q, d)
        k = torch.randn(n_k, d)
        s = q @ k.T
        for nom, sk in [("xom", s), ("sqrt", s / d ** 0.5)]:
            maks, sam, nol, jac = statistika(sk)
            natija[(d, nom)] = (maks, sam, jac)
            print(f"  {d:>5} {nom:>6} {maks:>13.3f} {sam:>15.2f} {nol:>13.3f} {jac:>8.4f}")
    print(f"  n_k = {n_k}: bir tekis og'irlikda maks 1/{n_k} = {1 / n_k:.3f}, "
          f"samarali kalit {n_k}")
    print("  ||J|| - softmax Jacobiani normasi: gradient skorlardan qanchalik o'tadi")

    print("\n=== 3. Gradient: so'rov vektoriga qaytadigan signal ===")
    print(f"  {'d_k':>5} {'||dL/dq|| xom':>14} {'||dL/dq|| sqrt':>15} {'nisbat':>8}")
    for d in [16, 256, 1024]:
        normalar = {}
        for nom in ("xom", "sqrt"):
            torch.manual_seed(1)
            q = torch.randn(256, d, requires_grad=True)
            k = torch.randn(n_k, d)
            v = torch.randn(n_k, 8)
            s = q @ k.T
            if nom == "sqrt":
                s = s / d ** 0.5
            nishon = torch.randn(256, 8)
            loss = ((torch.softmax(s, -1) @ v - nishon) ** 2).mean()
            loss.backward()
            normalar[nom] = q.grad.norm(dim=1).median().item()
        print(f"  {d:>5} {normalar['xom']:>14.2e} {normalar['sqrt']:>15.2e} "
              f"{normalar['xom'] / normalar['sqrt']:>8.2f}")
    print("  (median - bir nechta to'yinmagan qator o'rtachani buzmasligi uchun)")

    print("\n=== 4. Xulosa ===")
    m_xom, _, j_xom = natija[(1024, "xom")]
    m_sq, _, j_sq = natija[(1024, "sqrt")]
    print(f"  d_k = 1024: maks og'irlik xom {m_xom:.3f}, sqrt bilan {m_sq:.3f}")
    if j_xom < j_sq / 3:
        print(f"  xom skorda ||J|| {j_sq / j_xom:.1f} marta kichik - gradient so'nadi")
    farqlar = [natija[(d, "sqrt")][0] for d in olchamlar]
    if max(farqlar) - min(farqlar) < 0.1:
        print("  sqrt(d_k) bilan to'yinish d_k ga deyarli bog'liq emas")
    print("  ⭐ 1/sqrt(d_k) skor dispersiyasini 1 ga qaytaradi: softmax to'yinmaydi")


if __name__ == "__main__":
    main()

=== 1. Skor dispersiyasi d_k bilan o'sadi ===
    d_k  std(q.k)  sqrt(d_k)  std(q.k)/sqrt(d_k)
      4      1.75       2.00               0.876
     16      4.02       4.00               1.005
     64      8.21       8.00               1.026
    256     15.90      16.00               0.994
   1024     31.83      32.00               0.995
  q, k ~ N(0, 1): q.k - d_k ta mustaqil ko'paytma yig'indisi, dispersiya d_k

=== 2. Softmax to'yinishi: masshtabsiz va sqrt(d_k) bilan ===
    d_k         maks ogirlik  samarali kalit   nol ogirlik    ||J||
      4    xom         0.318           13.19         0.000   0.2808
      4   sqrt         0.144           23.11         0.000   0.2252
     16    xom         0.639            3.57         0.000   0.2884
     16   sqrt         0.168           20.40         0.000   0.2430
     64    xom         0.814            1.85         0.000   0.1968
     64   sqrt         0.164           20.73         0.000   0.2404
    256    xom         0.909            1.34         0.000   0.1130
    256   sqrt         0.163           20.71         0.000   0.2405
   1024    xom         0.953            1.16         0.137   0.0629
   1024   sqrt         0.161           20.80         0.000   0.2399
  n_k = 32: bir tekis og'irlikda maks 1/32 = 0.031, samarali kalit 32
  ||J|| - softmax Jacobiani normasi: gradient skorlardan qanchalik o'tadi

=== 3. Gradient: so'rov vektoriga qaytadigan signal ===
    d_k  ||dL/dq|| xom  ||dL/dq|| sqrt   nisbat
     16       3.25e-03        6.21e-04     5.23
    256       9.13e-04        5.96e-04     1.53
   1024       1.71e-05        5.90e-04     0.03
  (median - bir nechta to'yinmagan qator o'rtachani buzmasligi uchun)

=== 4. Xulosa ===
  d_k = 1024: maks og'irlik xom 0.953, sqrt bilan 0.161
  xom skorda ||J|| 3.8 marta kichik - gradient so'nadi
  sqrt(d_k) bilan to'yinish d_k ga deyarli bog'liq emas
  ⭐ 1/sqrt(d_k) skor dispersiyasini 1 ga qaytaradi: softmax to'yinmaydi

"""Padding niqobi: niqobsiz sizib chiqish, -inf niqob, F.sdpa konvensiyasi va NaN tuzog'i."""

import torch
import torch.nn.functional as F

JUMLALAR = [
    "men bugun kitob oqidim",
    "havo sovuq",
    "biz ertaga bozorga borib meva olamiz",
]


def lugat_va_tensor(jumlalar):
    sozlar = sorted({s for j in jumlalar for s in j.split()})
    s2i = {s: i + 1 for i, s in enumerate(sozlar)}          # 0 - <pad>
    uz = torch.tensor([len(j.split()) for j in jumlalar])
    X = torch.zeros(len(jumlalar), int(uz.max()), dtype=torch.long)
    for b, j in enumerate(jumlalar):
        X[b, :uz[b]] = torch.tensor([s2i[s] for s in j.split()])
    return X, uz, len(sozlar) + 1


def attention(Q, K, V, maska=None):
    """maska: (B, n_k) bool, True - haqiqiy token (qatnashadi)."""
    skor = Q @ K.transpose(-2, -1) / Q.shape[-1] ** 0.5
    if maska is not None:
        skor = skor.masked_fill(~maska[:, None, :], float("-inf"))
    w = torch.softmax(skor, dim=-1)
    return w @ V, w


def main() -> None:
    torch.manual_seed(0)
    X, uz, n_soz = lugat_va_tensor(JUMLALAR)
    emb = torch.randn(n_soz, 16)
    H = emb[X]                                              # (B, T, d)
    maska = torch.arange(X.shape[1])[None, :] < uz[:, None]

    print("=== 1. Batch va padding ===")
    print(f"  X shakli {tuple(X.shape)}, uzunliklar {uz.tolist()}")
    for b in range(len(X)):
        print(f"  {b}: {X[b].tolist()}   maska {maska[b].int().tolist()}")

    print("\n=== 2. Niqobsiz: padding ma'lumotga sizib kiradi ===")
    O_yolgiz = []
    for b in range(len(X)):
        h = H[b:b + 1, :uz[b]]
        O_yolgiz.append(attention(h, h, h)[0][0])
    O_nq, W_nq = attention(H, H, H)
    O_q, W_q = attention(H, H, H, maska)
    print(f"  {'jumla':<6} {'pad soni':>9} {'pad massasi':>12} "
          f"{'niqobsiz farq':>14} {'niqobli farq':>13}")
    for b in range(len(X)):
        n = int(uz[b])
        pad_massa = W_nq[b, :n, n:].sum(-1).mean().item()
        f_nq = (O_nq[b, :n] - O_yolgiz[b]).abs().max().item()
        f_q = (O_q[b, :n] - O_yolgiz[b]).abs().max().item()
        print(f"  {b:<6} {X.shape[1] - n:>9} {pad_massa:>12.3f} {f_nq:>14.4f} {f_q:>13.2e}")
    print("  farq - batchdagi natija va shu jumlani YOLG'IZ hisoblash orasida")
    print(f"  niqobli og'irliklarning pad ustidagi massasi: "
          f"{W_q[~maska[:, None, :].expand_as(W_q)].sum().item():.1f}")

    print("\n=== 3. F.scaled_dot_product_attention bilan ===")
    O_f = F.scaled_dot_product_attention(H, H, H, attn_mask=maska[:, None, :])
    print(f"  bool attn_mask (True = qatnashadi): maks farq {(O_f - O_q).abs().max().item():.2e}")
    qoshiluvchi = torch.zeros(maska.shape).masked_fill(~maska, float("-inf"))
    O_f2 = F.scaled_dot_product_attention(H, H, H, attn_mask=qoshiluvchi[:, None, :])
    print(f"  float attn_mask (0 / -inf, skorga qo'shiladi): maks farq "
          f"{(O_f2 - O_q).abs().max().item():.2e}")
    teskari = F.scaled_dot_product_attention(H[:1], H[:1], H[:1],
                                             attn_mask=~maska[:1, None, :])
    b = 0
    n = int(uz[b])
    print("  TESKARI konvensiya (True = e'tiborsiz, nn.MultiheadAttention uslubi):")
    print(f"    0-jumla, 1-token chiqishi pad embeddingidan farqi: "
          f"{(teskari[0, 0] - emb[0]).abs().max().item():.2e}")
    print("    model faqat PAD ga qaradi - xato jim o'tadi, NaN yo'q")

    print("\n=== 4. Butunlay niqoblangan qator: NaN tuzog'i ===")
    bosh_maska = maska.clone()
    bosh_maska[1] = False                                  # 1-jumla: hamma kalit yopiq
    O_nan, _ = attention(H, H, H, bosh_maska)
    print(f"  -inf bilan: NaN qiymatlar {torch.isnan(O_nan).sum().item()} ta "
          f"(1-jumla: {torch.isnan(O_nan[1]).all().item()})")
    skor = (H @ H.transpose(1, 2) / 4).masked_fill(~bosh_maska[:, None, :], -1e9)
    w = torch.softmax(skor, -1)
    print(f"  -1e9 bilan: NaN {torch.isnan(w).sum().item()} ta, lekin 1-jumlada "
          f"og'irlik bir tekis: {[round(x, 4) for x in w[1, 0].tolist()]}")
    print("  yechim: bo'sh ketma-ketlikni batchga qo'ymang yoki natijani alohida nolga tenglang")
    print("  ⭐ Niqob skorga softmax DAN OLDIN qo'yiladi: -inf -> og'irlik aynan 0")


if __name__ == "__main__":
    main()

=== 1. Batch va padding ===
  X shakli (3, 6), uzunliklar [4, 2, 6]
  0: [8, 4, 7, 11, 0, 0]   maska [1, 1, 1, 1, 0, 0]
  1: [6, 12, 0, 0, 0, 0]   maska [1, 1, 0, 0, 0, 0]
  2: [1, 5, 3, 2, 9, 10]   maska [1, 1, 1, 1, 1, 1]

=== 2. Niqobsiz: padding ma'lumotga sizib kiradi ===
  jumla   pad soni  pad massasi  niqobsiz farq  niqobli farq
  0              2        0.087         0.3571      2.38e-07
  1              4        0.178         1.0320      1.19e-07
  2              0        0.000         0.0000      0.00e+00
  farq - batchdagi natija va shu jumlani YOLG'IZ hisoblash orasida
  niqobli og'irliklarning pad ustidagi massasi: 0.0

=== 3. F.scaled_dot_product_attention bilan ===
  bool attn_mask (True = qatnashadi): maks farq 2.38e-07
  float attn_mask (0 / -inf, skorga qo'shiladi): maks farq 2.38e-07
  TESKARI konvensiya (True = e'tiborsiz, nn.MultiheadAttention uslubi):
    0-jumla, 1-token chiqishi pad embeddingidan farqi: 0.00e+00
    model faqat PAD ga qaradi - xato jim o'tadi, NaN yo'q

=== 4. Butunlay niqoblangan qator: NaN tuzog'i ===
  -inf bilan: NaN qiymatlar 96 ta (1-jumla: True)
  -1e9 bilan: NaN 0 ta, lekin 1-jumlada og'irlik bir tekis: [0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.1667]
  yechim: bo'sh ketma-ketlikni batchga qo'ymang yoki natijani alohida nolga tenglang
  ⭐ Niqob skorga softmax DAN OLDIN qo'yiladi: -inf -> og'irlik aynan 0

"""Attention o'rganadi: kalit-qiymat qidiruvi, o'rtacha bazaviy va sqrt(d_k) ning ta'siri."""

import math

import numpy as np
import torch
import torch.nn as nn

N_KALIT, N_QIYMAT, M = 32, 16, 8          # 32 xil kalit, 16 xil qiymat, 8 juftlik


def malumot(n, g):
    """Har misol: 8 ta (kalit, qiymat) juftligi va bitta so'rov-kalit."""
    kalit = torch.stack([torch.randperm(N_KALIT, generator=g)[:M] for _ in range(n)])
    qiymat = torch.randint(0, N_QIYMAT, (n, M), generator=g)
    j = torch.randint(0, M, (n,), generator=g)
    ar = torch.arange(n)
    return kalit, qiymat, kalit[ar, j], qiymat[ar, j], j


class Qidiruv(nn.Module):
    """rejim: 'sqrt' - 1/sqrt(d), 'xom' - masshtabsiz, 'ortacha' - bir tekis og'irlik."""

    def __init__(self, rejim, d):
        super().__init__()
        self.rejim, self.d = rejim, d
        self.emb_k = nn.Embedding(N_KALIT, d)
        self.emb_v = nn.Embedding(N_QIYMAT, d)
        self.W_q = nn.Linear(d, d, bias=False)
        self.W_k = nn.Linear(d, d, bias=False)
        self.W_v = nn.Linear(d, d, bias=False)
        self.chiqish = nn.Linear(d, N_QIYMAT)

    def forward(self, kalit, qiymat, sorov):
        juft = self.emb_k(kalit) + self.emb_v(qiymat)          # (B, M, d)
        Q = self.W_q(self.emb_k(sorov))[:, None]               # (B, 1, d)
        skor = Q @ self.W_k(juft).transpose(1, 2)              # (B, 1, M)
        if self.rejim == "sqrt":
            skor = skor / math.sqrt(self.d)
        if self.rejim == "ortacha":
            w = torch.full_like(skor, 1 / M)
        else:
            w = torch.softmax(skor, dim=-1)
        return self.chiqish((w @ self.W_v(juft))[:, 0]), w[:, 0]


def orgat(rejim, d, seed, qadamlar):
    torch.manual_seed(seed)
    model = Qidiruv(rejim, d)
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        k, v, q, y, _ = malumot(128, g)
        loss = nn.functional.cross_entropy(model(k, v, q)[0], y)
        opt.zero_grad()
        loss.backward()
        opt.step()
    return model


def baholash(model, test):
    k, v, q, y, j = test
    with torch.no_grad():
        logit, w = model(k, v, q)
    aniq = (logit.argmax(1) == y).float().mean().item()
    togri_massa = w[torch.arange(len(j)), j].mean().item()
    return aniq, togri_massa


def main() -> None:
    torch.set_num_threads(1)
    torch.set_flush_denormal(True)       # to'yingan softmax dagi subnormal sonlar CPU ni sekinlatadi
    test = malumot(2000, torch.Generator().manual_seed(999))

    print("=== 1. Vazifa: kalit-qiymat qidiruvi ===")
    k, v, q, y, j = malumot(1, torch.Generator().manual_seed(5))
    juftlar = ", ".join(f"{a}:{b}" for a, b in zip(k[0].tolist(), v[0].tolist()))
    print(f"  juftliklar: {juftlar}")
    print(f"  so'rov kalit {q.item()} -> javob {y.item()}")
    print(f"  tasodifiy taxmin aniqligi 1/{N_QIYMAT} = {1 / N_QIYMAT:.4f}")

    print("\n=== 2. Attention va bir tekis o'rtacha (d = 32, 120 qadam) ===")
    for rejim in ["ortacha", "sqrt"]:
        natija = [baholash(orgat(rejim, 32, s, 120), test) for s in range(3)]
        aniq = np.mean([a for a, _ in natija])
        massa = np.mean([m for _, m in natija])
        print(f"  {rejim:<8} aniqlik {aniq:.4f}   to'g'ri juftlikdagi og'irlik {massa:.3f}")
    print(f"  o'rtacha: har juftlikka 1/{M} = {1 / M:.3f} - qaysi biri kerakligini bilmaydi")
    moda = torch.mode(test[1], dim=1).values
    print(f"  taqqoslash: 'eng ko'p uchragan qiymat' qoidasi aniqligi "
          f"{(moda == test[3]).float().mean().item():.4f}")

    print("\n=== 3. Boshlang'ich holatda skorlar (o'rgatishdan oldin) ===")
    for d in [32, 128]:
        for rejim in ["xom", "sqrt"]:
            torch.manual_seed(0)
            m = Qidiruv(rejim, d)
            with torch.no_grad():
                _, w = m(*test[:3])
            print(f"  d = {d:>3} {rejim:<5} o'rtacha maks og'irlik {w.max(1).values.mean():.3f}")

    print("\n=== 4. sqrt(d_k) ning ta'siri: 60 qadam, 4 seed, juftlashgan ===")
    print(f"  {'d':>4} {'sqrt':>8} {'xom':>8} {'farq':>8} {'SE':>7} {'sezilarli':>10}")
    farqlar = {}
    for d in [32, 128]:
        a_sqrt = np.array([baholash(orgat("sqrt", d, s, 60), test)[0] for s in range(4)])
        a_xom = np.array([baholash(orgat("xom", d, s, 60), test)[0] for s in range(4)])
        f = a_sqrt - a_xom
        se = f.std(ddof=1) / np.sqrt(len(f))
        farqlar[d] = (f.mean(), se)
        print(f"  {d:>4} {a_sqrt.mean():>8.4f} {a_xom.mean():>8.4f} {f.mean():>+8.4f} "
              f"{se:>7.4f} {str(abs(f.mean()) > 2 * se):>10}")
    for d, (f, se) in farqlar.items():
        if abs(f) <= 2 * se:
            print(f"  d = {d}: farq sezilarli emas")
        elif f > 0:
            print(f"  d = {d}: sqrt(d_k) bilan sezilarli yaxshi")
        else:
            print(f"  d = {d}: masshtabsiz variant sezilarli yaxshi")
    print("  ⭐ Attention 'qayerga qarashni' o'zi o'rganadi - og'irliklar o'rgatiladi")


if __name__ == "__main__":
    main()

=== 1. Vazifa: kalit-qiymat qidiruvi ===
  juftliklar: 3:6, 17:13, 9:2, 10:14, 0:9, 12:9, 24:10, 15:9
  so'rov kalit 17 -> javob 13
  tasodifiy taxmin aniqligi 1/16 = 0.0625

=== 2. Attention va bir tekis o'rtacha (d = 32, 120 qadam) ===
  ortacha  aniqlik 0.2373   to'g'ri juftlikdagi og'irlik 0.125
  sqrt     aniqlik 0.9998   to'g'ri juftlikdagi og'irlik 0.974
  o'rtacha: har juftlikka 1/8 = 0.125 - qaysi biri kerakligini bilmaydi
  taqqoslash: 'eng ko'p uchragan qiymat' qoidasi aniqligi 0.2640

=== 3. Boshlang'ich holatda skorlar (o'rgatishdan oldin) ===
  d =  32 xom   o'rtacha maks og'irlik 0.615
  d =  32 sqrt  o'rtacha maks og'irlik 0.221
  d = 128 xom   o'rtacha maks og'irlik 0.798
  d = 128 sqrt  o'rtacha maks og'irlik 0.220

=== 4. sqrt(d_k) ning ta'siri: 60 qadam, 4 seed, juftlashgan ===
     d     sqrt      xom     farq      SE  sezilarli
    32   0.9813   0.9919  -0.0106  0.0026       True
   128   0.9999   0.9729  +0.0270  0.0100       True
  d = 32: masshtabsiz variant sezilarli yaxshi
  d = 128: sqrt(d_k) bilan sezilarli yaxshi
  ⭐ Attention 'qayerga qarashni' o'zi o'rganadi - og'irliklar o'rgatiladi

w = torch.softmax(Q @ K.transpose(-2, -1), dim=-2)          # so'rovlar o'qi  # ⚠️
w = torch.softmax(Q @ K.transpose(-2, -1), dim=-1)          # kalitlar o'qi   # ✅

skor = Q @ K.T                        # (B, n, d).T -> (d, n, B)          # ⚠️
skor = Q @ K.transpose(-2, -1)                                            # ✅

skor = Q @ K.transpose(-2, -1)                    # d_k katta bo'lsa to'yinadi  # ⚠️
skor = Q @ K.transpose(-2, -1) / d_model ** 0.5   # ko'p boshlida d_k emas      # ⚠️
skor = Q @ K.transpose(-2, -1) / Q.shape[-1] ** 0.5                            # ✅

w = torch.softmax(skor, -1) * maska[:, None, :]                             # ⚠️
w = torch.softmax(skor.masked_fill(~maska[:, None, :], float("-inf")), -1)  # ✅

F.scaled_dot_product_attention(Q, K, V, attn_mask=pad_maska[:, None, :])   # pad=True  # ⚠️
F.scaled_dot_product_attention(Q, K, V, attn_mask=~pad_maska[:, None, :])  # token=True # ✅

batch = [jumla for jumla in jumlalar]                  # bo'sh jumla ham bor -> NaN  # ⚠️
batch = [jumla for jumla in jumlalar if len(jumla) > 0]                             # ✅

w = np.exp(skor) / np.exp(skor).sum(-1, keepdims=True)                      # ⚠️
e = np.exp(skor - skor.max(-1, keepdims=True)); w = e / e.sum(-1, keepdims=True)  # ✅

1.  # Q (2, 5, 16), K (2, 7, 16), V (2, 7, 8): skor, og'irlik va chiqish shakli?
2.  # softmax qaysi o'q bo'yicha va nima uchun?
3.  # q, k ~ N(0, 1), d_k = 256 bo'lsa std(q . k) taxminan nechaga teng?
4.  # skor ni sqrt(d_k) ga bo'lgandan keyin-chi?
5.  # softmax one-hot ga yaqin bo'lsa, skorlarga gradient qanday bo'ladi?
6.  # padding pozitsiyasiga og'irlik 0 bo'lishi uchun skorga nima yoziladi?
7.  # nima uchun niqobni softmax dan keyin qo'yib bo'lmaydi?
8.  # F.sdpa bool maskasida True nimani anglatadi? nn.MultiheadAttention key_padding_mask da-chi?
9.  # hamma kalit -inf bo'lsa softmax nima beradi?
10. # 23.11 dagi attention ni F.sdpa bilan qanday yozasiz?
11. # softmax(x) va softmax(x - 100) farq qiladimi?
12. # nega kichik d_k da masshtab foyda bermasligi mumkin?

1.  skor = Q @ K.T / 8
    w = torch.softmax(skor, dim=1)

2.  w = torch.softmax(Q @ K.transpose(-2, -1) / d ** 0.5, -1)
    w = w * maska[:, None, :]

3.  pad = X == PAD
    O = F.scaled_dot_product_attention(Q, K, V, attn_mask=pad[:, None, :])

4.  def softmax(x):
        return np.exp(x) / np.exp(x).sum(-1, keepdims=True)

5.  # 8 boshli attention, d_model = 512
    skor = Q_h @ K_h.transpose(-2, -1) / 512 ** 0.5