24.6-dars: Transformer encoder

MATN -> TOKENLAR -> VEKTORLAR -> KONTEKSTLI VEKTORLAR -> BITTA VEKTOR -> SINF

  "telefon sifati juda yaxshi"
  [CLS] telefon sifati juda yaxshi <pad> <pad>     X: (B, L)
  E[x_t] + P[t]                                     (B, L, d)
  N ta blok (pre-LN, 24.5)                          (B, L, d)
  yakuniy LayerNorm                                 (B, L, d)
  pooling                                           (B, d)
  Linear                                            (B, sinf)

LUG'AT (23.2 dagi kabi, faqat o'quvda quriladi):
  0 <pad>, 1 <unk>, 2 <cls>, qolganlari 3..V-1

PYTORCH:
  qatlam = nn.TransformerEncoderLayer(d, h, d_ff, p, batch_first=True,
                                      norm_first=True)
  enc = nn.TransformerEncoder(qatlam, n, norm=nn.LayerNorm(d),
                              enable_nested_tensor=False)
  x = enc(x, src_key_padding_mask=(X == PAD))

PARAMETRLAR (1-misol, d = 32, 2 blok, V = 13):
  bloklar 86.7%, pozitsiya 10.4%, token embedding 2.1%
  katta lug'atda (V = 30 000) esa embedding ulushi keskin oshadi

[CLS] POOLING (BERT):
  har matn boshiga maxsus <cls> token qo'shiladi
  h = x[:, 0]  - shu pozitsiyaning oxirgi blokdagi vektori
  <cls> o'zi hech narsani anglatmaydi: u attention orqali
  boshqa tokenlardan ma'lumot "yig'ishni" o'rganadi

O'RTACHA POOLING:
  h = yig'indi_t m_t x_t / yig'indi_t m_t,   m_t = 1 agar x_t != <pad>
  23.7 dagi maskali o'rtacha - endi kontekstli vektorlar ustida

QAYSI BIRI:
  [CLS] - pretraining da maxsus o'rgatilgan bo'lsa (BERT) tabiiy tanlov
  o'rtacha - noldan, kam davr bilan o'rgatishda ko'pincha barqarorroq
  2-misol (3 davr, 3 seed): o'rtacha 0.7117, [CLS] 0.6872,
                            farq +0.0244, SE 0.0072 - sezilarli
  bu natija byudjetga bog'liq - uzoqroq o'rgatishda farq o'zgarishi mumkin

MUAMMO:
  batch: sharhlar turli uzunlikda -> eng uzunigacha <pad> bilan to'ldiriladi
  self-attention da HAR pozitsiya HAR pozitsiyaga qaraydi - <pad> ga ham
  <pad> embeddingi nol bo'lsa ham, unga POZITSIYA qo'shilgan - nol emas

NIQOB:
  src_key_padding_mask: (B, L) bool, True = <pad> (e'tiborsiz qoldiriladi)
  ichida: skor[:, :, pad] = -cheksiz  ->  softmax dan keyin og'irlik 0
  1-misol: niqobsiz [CLS] padding ga 0.332 og'irlik beradi, niqob bilan 0.000

NIQOBSIZ MODEL - NATIJA PADDING UZUNLIGIGA BOG'LIQ:
  1-misol (o'rgatilmagan model, bitta jumla):
    L = 8 dan 16 ga: [CLS] vektori 0.243 ga, 40 ga: 0.538 ga o'zgardi
    niqob bilan: 3.6e-07 (son xatosi)
  2-misol (o'rgatilgan model, bir xil test sharhlari):
    niqobsiz: L = 23 da 0.6311, L = 64 da 0.4644
    niqobli:  ikkala holatda 0.6872

DIQQAT - "True" NIMANI ANGLATADI:
  src_key_padding_mask da True = e'tiborsiz qoldirish (PAD)
  F.scaled_dot_product_attention attn_mask da True = QATNASHADI
  ikki API da ma'no TESKARI - har safar tekshiring

1. korpus -> o'quv/test bo'linishi (yoki StratifiedKFold)
2. lug'at FAQAT o'quvda; kam uchraganlar -> <unk>
3. kodlash: [<cls>] + indekslar
4. batch: eng uzun sharhgacha to'ldirish (dinamik padding)
5. forward(X, src_key_padding_mask=(X == PAD)) -> logit (B, 3)
6. cross_entropy, AdamW, bir necha davr
7. model.eval() + torch.no_grad() bilan baholash

SAMARADORLIK:
  butun o'quv to'plamini bir marta to'ldirib, batchda eng uzunigacha kesish
  X = X_hammasi[b][:, :uzunlik[b].max()]

USUL (23.5, 23.7 dagi kabi):
  bir xil 5 fold, har fold da uchala model
  farq = model - eng_yaxshisi (har fold da), SE = std / sqrt(5)
  |farq| > 2*SE bo'lsa - sezilarli
  QAROR: eng yaxshisidan sezilarli yomon bo'lmagan eng sodda model
  soddalik tartibi: TF-IDF + LogReg < EmbeddingBag < Transformer

3-MISOL (3000 sharh, 5 fold):
  TF-IDF + bigram + LogReg   0.8073
  EmbeddingBag + bigram      0.8113   (eng yuqori)
  Transformer encoder        0.7197   (2 blok, d = 32, 4 davr)
  TF-IDF - EmbBag:   -0.0040, SE 0.0024  -> sezilarli emas
  Encoder - EmbBag:  -0.0917, SE 0.0052  -> sezilarli yomon
  QAROR: TF-IDF

NIMA UCHUN TRANSFORMER YUTMADI:
  2400 o'quv sharhi - attention va FFN ni noldan o'rgatish uchun kam
  vazifa asosan "qaysi baho iboralari bor" - bigram bag-of-words bunga yetarli
  bigram "yaxshi_emas" inkorni to'g'ridan-to'g'ri ushlaydi;
  encoder esa buni kontekstdan o'rganishi kerak
  (3-misol: inkorli sharhlarda encoder 0.5670, TF-IDF 0.7808)
  Transformer kuchi - katta ma'lumot va oldindan o'rgatish (keyingi darslar)

QAYERDAN OLISH:
  nn.TransformerEncoderLayer og'irlikni qaytarmaydi
  oxirgi blokka kirishni hisoblab, self_attn(..., need_weights=True)
  pre-LN da: y = layer.norm1(x) -> layer.self_attn(y, y, y, ...)
  [CLS] pooling da eng qiziq qator - w[:, 0, :] ([CLS] kimga qaraydi)

NISBIY OG'IRLIK:
  w_t * L  - 1.00 bir tekis taqsimotga teng, 2.00 - ikki barobar ko'p
  4-misol: "emas" 6.22, neytral so'zlar 4.09, baho so'zlari 1.80,
           jihatlar 0.45, qolganlari 0.57

TEKSHIRISH - OLIB TASHLASH (occlusion):
  eng ko'p qaralgan so'z -> <unk>: bashorat 0.302 holatda o'zgardi
  tasodifiy so'z -> <unk>: 0.080
  og'irligi 1/L dan KAM baho so'zi -> <unk>: 0.106
  -> yuqori og'irlik ko'pincha muhimlikka mos, lekin past og'irlik
     "muhim emas" degani emas

CHEGARALAR (23.11 dagi kabi):
  oxirgi blokdagi tokenlar allaqachon bir-biri bilan aralashgan
  og'irlik - "qaysi pozitsiyadan olindi", sabab emas
  bir nechta boshni o'rtachalash tafsilotni yashiradi

import torch
import torch.nn as nn

PAD, UNK, CLS = 0, 1, 2

class Encoder(nn.Module):
    def __init__(self, V, d=32, h=2, n=2, d_ff=64, maks=64, p=0.1):
        super().__init__()
        self.emb = nn.Embedding(V, d, padding_idx=PAD)
        self.pos = nn.Embedding(maks, d)
        qatlam = nn.TransformerEncoderLayer(d, h, d_ff, p, batch_first=True,
                                            norm_first=True)
        self.enc = nn.TransformerEncoder(qatlam, n, norm=nn.LayerNorm(d),
                                         enable_nested_tensor=False)
        self.chiq = nn.Linear(d, 3)

    def forward(self, X, pooling="mean"):
        pad = X == PAD                                    # True = e'tiborsiz
        x = self.emb(X) + self.pos(torch.arange(X.shape[1]))
        x = self.enc(x, src_key_padding_mask=pad)
        if pooling == "cls":
            return self.chiq(x[:, 0])
        m = (~pad).unsqueeze(-1).float()
        return self.chiq((x * m).sum(1) / m.sum(1))

# kodlash va dinamik padding
ids = [[CLS] + [lugat.get(s, UNK) for s in m.split()] for m in matnlar]
X_hammasi = toldir(ids)
X = X_hammasi[b][:, :uzunlik[b].max()]

# oxirgi blokda [CLS] og'irliklari (pre-LN)
y = model.enc.layers[-1].norm1(x_oxirgi_kirish)
_, w = model.enc.layers[-1].self_attn(y, y, y, key_padding_mask=pad,
                                     need_weights=True)
w_cls = w[:, 0]                                            # (B, L)

encoder = embedding + pozitsiya + N blok + pooling + Linear
[CLS] yoki maskali o'rtacha; x.mean(1) - xato
src_key_padding_mask=(X == PAD) - to'g'rilik sharti
bazaviy: TF-IDF va EmbeddingBag, bir xil foldlar, farq + SE
attention og'irligi - gipoteza, olib tashlash bilan tekshiriladi

"""To'liq Transformer encoder: shakllar, parametrlar, pooling va padding niqobi."""

import torch
import torch.nn as nn

PAD, UNK, CLS = 0, 1, 2
LUGAT = ["<pad>", "<unk>", "<cls>", "telefon", "sifati", "juda", "yaxshi", "emas",
         "narxi", "qimmat", "lekin", "tavsiya", "qilaman"]
s2i = {s: i for i, s in enumerate(LUGAT)}


class Encoder(nn.Module):
    """embedding + pozitsiya -> N ta pre-LN blok -> pooling ([CLS] yoki o'rtacha) -> Linear."""

    def __init__(self, V, pooling="cls", d=32, h=2, n=2, d_ff=64, maks=64, p=0.1, sinf=3):
        super().__init__()
        self.pooling = pooling
        self.emb = nn.Embedding(V, d, padding_idx=PAD)
        self.pos = nn.Embedding(maks, d)
        qatlam = nn.TransformerEncoderLayer(d, h, d_ff, p, batch_first=True, norm_first=True)
        self.enc = nn.TransformerEncoder(qatlam, n, norm=nn.LayerNorm(d),
                                         enable_nested_tensor=False)
        self.chiq = nn.Linear(d, sinf)

    def kodla(self, X, niqob=True):
        pad = X == PAD
        x = self.emb(X) + self.pos(torch.arange(X.shape[1]))
        return self.enc(x, src_key_padding_mask=pad if niqob else None), pad

    def forward(self, X, niqob=True):
        x, pad = self.kodla(X, niqob)
        if self.pooling == "cls":
            h = x[:, 0]                                    # [CLS] pozitsiyasi
        else:
            m = (~pad).unsqueeze(-1).float()
            h = (x * m).sum(1) / m.sum(1)                  # faqat haqiqiy tokenlar
        return self.chiq(h)


def tokenla(matn, L):
    ids = [CLS] + [s2i.get(s, UNK) for s in matn.split()]
    return torch.tensor([ids + [PAD] * (L - len(ids))])


def main() -> None:
    torch.manual_seed(0)
    model = Encoder(len(LUGAT)).eval()

    print("=== 1. Shakllar bosqichma-bosqich ===")
    X = torch.cat([tokenla("telefon sifati juda yaxshi", 8),
                   tokenla("narxi qimmat", 8)])
    print(f"  X (B, L)                 {tuple(X.shape)}  {X[1].tolist()}")
    e = model.emb(X) + model.pos(torch.arange(X.shape[1]))
    print(f"  embedding + pozitsiya    {tuple(e.shape)}")
    with torch.no_grad():
        x, pad = model.kodla(X)
        print(f"  {len(model.enc.layers)} blokdan keyin          {tuple(x.shape)}")
        print(f"  [CLS] vektori x[:, 0]    {tuple(x[:, 0].shape)}")
        print(f"  logitlar                 {tuple(model(X).shape)}")

    print("\n=== 2. Parametrlar qayerda ===")
    qismlar = {"token embedding": model.emb, "pozitsiya": model.pos,
               "bloklar": model.enc.layers, "yakuniy LN": model.enc.norm,
               "klassifikator": model.chiq}
    jami = sum(p.numel() for p in model.parameters())
    for nom, q in qismlar.items():
        n = sum(p.numel() for p in q.parameters())
        print(f"  {nom:<16} {n:>6}  ({n / jami:.1%})")
    print(f"  {'jami':<16} {jami:>6}")
    d, d_ff = 32, 64
    blok = 4 * d * d + 2 * d * d_ff + 9 * d + d_ff
    print(f"  formula: 2 blok x {blok} = {2 * blok}")

    print("\n=== 3. Padding niqobi: bir jumla, turli padding uzunligi ===")
    matn = "telefon sifati juda yaxshi lekin narxi qimmat"
    print(f"  jumla: '{matn}' ({len(matn.split()) + 1} token, [CLS] bilan)")
    sarlavha = "niqobli o'rtacha"
    print(f"  {'L':>4} {'niqobsiz [CLS]':>15} {'niqobli [CLS]':>14} {sarlavha:>17}")
    with torch.no_grad():
        asos = {}
        for L in (8, 16, 40):
            Xl = tokenla(matn, L)
            a = model.kodla(Xl, niqob=False)[0][0, 0]
            b = model.kodla(Xl, niqob=True)[0][0, 0]
            model.pooling = "mean"
            c = model(Xl)[0]
            model.pooling = "cls"
            if L == 8:
                asos = {"a": a, "b": b, "c": c}
            print(f"  {L:>4} {(a - asos['a']).abs().max().item():>15.3f} "
                  f"{(b - asos['b']).abs().max().item():>14.1e} "
                  f"{(c - asos['c']).abs().max().item():>17.1e}")
    print("  jadvalda: L = 8 dagi natijadan max |farq| (o'rtacha ustunida - logitlar)")

    print("\n=== 4. Niqobsiz o'rtacha: padding ham sanaladi ===")
    with torch.no_grad():
        for L in (8, 16, 40):
            x, pad = model.kodla(tokenla(matn, L))
            togri = x[0, :8].mean(0)
            xato = x[0].mean(0)
            print(f"  L = {L:>2}: x.mean(1) va maskali o'rtacha farqi "
                  f"{(xato - togri).abs().max().item():.3f}")

    print("\n=== 5. Attention og'irliklari padding ga tushadimi (1-blok, [CLS] qatori) ===")
    qatlam = model.enc.layers[0]
    Xl = tokenla(matn, 12)
    with torch.no_grad():
        y = qatlam.norm1(model.emb(Xl) + model.pos(torch.arange(12)))
        for niqob in (False, True):
            _, w = qatlam.self_attn(y, y, y, key_padding_mask=(Xl == PAD) if niqob else None,
                                    need_weights=True)
            print(f"  niqob={str(niqob):<5}: padding ga og'irlik {w[0, 0, 8:].sum().item():.3f}, "
                  f"haqiqiy tokenlarga {w[0, 0, :8].sum().item():.3f}")
    print("  ⭐ src_key_padding_mask: True - e'tiborsiz qoldiriladigan (PAD) pozitsiya")


if __name__ == "__main__":
    main()

=== 1. Shakllar bosqichma-bosqich ===
  X (B, L)                 (2, 8)  [2, 8, 9, 0, 0, 0, 0, 0]
  embedding + pozitsiya    (2, 8, 32)
  2 blokdan keyin          (2, 8, 32)
  [CLS] vektori x[:, 0]    (2, 32)
  logitlar                 (2, 3)

=== 2. Parametrlar qayerda ===
  token embedding     416  (2.1%)
  pozitsiya          2048  (10.4%)
  bloklar           17088  (86.7%)
  yakuniy LN           64  (0.3%)
  klassifikator        99  (0.5%)
  jami              19715
  formula: 2 blok x 8544 = 17088

=== 3. Padding niqobi: bir jumla, turli padding uzunligi ===
  jumla: 'telefon sifati juda yaxshi lekin narxi qimmat' (8 token, [CLS] bilan)
     L  niqobsiz [CLS]  niqobli [CLS]  niqobli o'rtacha
     8           0.000        0.0e+00           0.0e+00
    16           0.243        3.6e-07           3.0e-08
    40           0.538        3.6e-07           3.0e-08
  jadvalda: L = 8 dagi natijadan max |farq| (o'rtacha ustunida - logitlar)

=== 4. Niqobsiz o'rtacha: padding ham sanaladi ===
  L =  8: x.mean(1) va maskali o'rtacha farqi 0.000
  L = 16: x.mean(1) va maskali o'rtacha farqi 0.493
  L = 40: x.mean(1) va maskali o'rtacha farqi 0.829

=== 5. Attention og'irliklari padding ga tushadimi (1-blok, [CLS] qatori) ===
  niqob=False: padding ga og'irlik 0.332, haqiqiy tokenlarga 0.668
  niqob=True : padding ga og'irlik 0.000, haqiqiy tokenlarga 1.000
  ⭐ src_key_padding_mask: True - e'tiborsiz qoldiriladigan (PAD) pozitsiya

"""Encoder bilan sharhlarni klassifikatsiya: [CLS] va o'rtacha pooling, niqobli va niqobsiz."""

from collections import Counter

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

IJOBIY = ["yaxshi", "zo'r", "a'lo", "ajoyib", "sifatli", "qulay",
          "chiroyli", "mustahkam"]
SALBIY = ["yomon", "sifatsiz", "noqulay", "xunuk", "nosoz", "mo'rt",
          "sust", "chatoq"]
NEYTRAL = ["o'rtacha", "oddiy", "normal", "odatiy", "kutilgandek"]
JIHAT = ["sifati", "narxi", "ekrani", "batareyasi", "ovozi", "qadoqlash",
         "yetkazib berish", "sotuvchi", "dizayni", "materiali"]
MAHSULOT = ["telefon", "noutbuk", "quloqchin", "soat", "choynak", "kurtka",
            "krossovka", "sumka", "planshet", "kolonka"]
BOSH = ["kecha", "o'tgan hafta", "bir oy oldin", "do'kondan", "internetdan",
        "sovg'a uchun", "akam uchun", ""]
FEL = ["oldim", "buyurtma qildim", "sotib oldim", "ishlatib ko'rdim"]
YAKUN = {1: ["tavsiya qilaman", "yana olaman", "hammaga maslahat beraman"],
         -1: ["tavsiya qilmayman", "boshqa olmayman", "pulimga achinaman"],
         0: ["hozircha shu", "vaqt ko'rsatadi", "yana ishlatib ko'ramiz"]}
ULUSH = {1: [0.8, 0.1, 0.1], -1: [0.1, 0.8, 0.1], 0: [0.15, 0.15, 0.7]}


def gap(qutb, rng):
    j = rng.choice(JIHAT)
    juda = "juda " if rng.random() < 0.3 else ""
    r = rng.random()
    if qutb == 1:
        if r < 0.2:
            return f"{j} {rng.choice(SALBIY)} emas"
        if r < 0.3:
            return f"{j} bilan muammo yo'q"
        return f"{j} {juda}{rng.choice(IJOBIY)}"
    if qutb == -1:
        if r < 0.25:
            return f"{j} {rng.choice(IJOBIY)} emas"
        if r < 0.35:
            return f"{j} bilan muammo bor"
        return f"{j} {juda}{rng.choice(SALBIY)}"
    return f"{j} {rng.choice(NEYTRAL)}"


def sharh(rng):
    kayfiyat = int(rng.integers(-1, 2))
    qismlar = [f"{rng.choice(BOSH)} {rng.choice(MAHSULOT)} "
               f"{rng.choice(FEL)}".strip()]
    ballar = []
    for t in range(int(rng.integers(1, 5))):
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        bog = rng.choice(["lekin", "va", "ammo", "ham", ""]) if t else ""
        qismlar.append(f"{bog} {gap(q, rng)}".strip())
        ballar.append(q)
    if rng.random() < 0.5:
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        qismlar.append(rng.choice(YAKUN[q]))
        ballar.append(q)
    ball = np.mean(ballar) + rng.normal(0, 0.1)
    return " ".join(qismlar), 2 if ball > 0.35 else (0 if ball < -0.35 else 1)


def imlo_xato(matn, ulush, rng):
    chiqish = []
    for s in matn.split():
        if len(s) >= 4 and rng.random() < ulush:
            i = int(rng.integers(1, len(s) - 1))
            tur = int(rng.integers(0, 3))
            if tur == 0:
                s = s[:i] + s[i + 1:]
            elif tur == 1:
                s = s[:i] + s[i] + s[i:]
            else:
                s = s[:i - 1] + s[i] + s[i - 1] + s[i + 1:]
        chiqish.append(s)
    return " ".join(chiqish)


def korpus(n, seed=0, xato=0.07):
    """23-qismdagi sintetik o'zbekcha sharhlar: 0 salbiy, 1 neytral, 2 ijobiy."""
    rng = np.random.default_rng(seed)
    toza, y = zip(*[sharh(rng) for _ in range(n)])
    rng2 = np.random.default_rng(seed + 1000)
    return [imlo_xato(t, xato, rng2) for t in toza], np.array(y)


PAD, UNK, CLS = 0, 1, 2


def lugat_qur(matnlar):
    c = Counter(s for m in matnlar for s in m.split())
    return {s: i + 3 for i, s in enumerate(sorted(t for t, k in c.items() if k >= 2))}


def kodla(matnlar, lugat):
    return [[CLS] + [lugat.get(s, UNK) for s in m.split()] for m in matnlar]


def toldir(ketmalar, L=None):
    L = L or max(len(k) for k in ketmalar)
    X = torch.zeros(len(ketmalar), L, dtype=torch.long)
    for i, k in enumerate(ketmalar):
        X[i, :len(k)] = torch.tensor(k)
    return X


class Encoder(nn.Module):
    def __init__(self, V, pooling="cls", d=32, h=2, n=2, d_ff=64, maks=64, p=0.1):
        super().__init__()
        self.pooling = pooling
        self.emb = nn.Embedding(V, d, padding_idx=PAD)
        self.pos = nn.Embedding(maks, d)
        qatlam = nn.TransformerEncoderLayer(d, h, d_ff, p, batch_first=True, norm_first=True)
        self.enc = nn.TransformerEncoder(qatlam, n, norm=nn.LayerNorm(d),
                                         enable_nested_tensor=False)
        self.chiq = nn.Linear(d, 3)

    def forward(self, X, niqob=True):
        pad = X == PAD
        x = self.emb(X) + self.pos(torch.arange(X.shape[1]))
        x = self.enc(x, src_key_padding_mask=pad if niqob else None)
        if self.pooling == "cls":
            return self.chiq(x[:, 0])
        m = (~pad).unsqueeze(-1).float()
        return self.chiq((x * m).sum(1) / m.sum(1))


def orgat(k_tr, ytr, V, seed, pooling="cls", niqob=True, davr=3):
    torch.manual_seed(seed)
    m = Encoder(V, pooling)
    opt = torch.optim.AdamW(m.parameters(), lr=0.003, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    yt = torch.tensor(ytr)
    X_hammasi = toldir(k_tr)                       # bir marta to'ldiramiz
    uzunlik = torch.tensor([len(k) for k in k_tr])
    for _ in range(davr):
        m.train()
        tartib = torch.randperm(len(k_tr), generator=g).tolist()
        for i in range(0, len(tartib), 64):
            b = tartib[i:i + 64]
            X = X_hammasi[b][:, :int(uzunlik[b].max())]    # batch ichidagi eng uzunigacha
            loss = F.cross_entropy(m(X, niqob), yt[b])
            opt.zero_grad()
            loss.backward()
            opt.step()
    m.eval()
    return m


def aniqlik(m, ketmalar, y, niqob=True, L=None):
    with torch.no_grad():
        return (m(toldir(ketmalar, L), niqob).argmax(1).numpy() == y).mean()


def juft(a, b):
    f = np.array(a) - np.array(b)
    se = f.std(ddof=1) / np.sqrt(len(f))
    return f.mean(), se, abs(f.mean()) > 2 * se


def main() -> None:
    torch.set_num_threads(1)
    matnlar, y = korpus(3000)
    idx = np.random.default_rng(0).permutation(len(matnlar))
    tr, te = idx[:2400], idx[2400:]
    lugat = lugat_qur([matnlar[i] for i in tr])
    V = len(lugat) + 3
    k_tr = kodla([matnlar[i] for i in tr], lugat)
    k_te = kodla([matnlar[i] for i in te], lugat)

    print("=== 1. Ma'lumot ===")
    for i in tr[:3]:
        print(f"  [{y[i]}] {matnlar[i]}")
    uz = [len(k) for k in k_te]
    print(f"  o'quv {len(tr)}, test {len(te)}, lug'at {V} (0 <pad>, 1 <unk>, 2 <cls>)")
    print(f"  test uzunligi ([CLS] bilan): {min(uz)}..{max(uz)}, o'rtacha {np.mean(uz):.1f}")
    print(f"  sinflar ulushi (test): {np.round(np.bincount(y[te]) / len(te), 3).tolist()}")

    variantlar = [("cls", True), ("mean", True), ("cls", False)]
    nat = {v: {"tabiiy": [], "L64": []} for v in variantlar}
    for s in range(3):
        for pool, niqob in variantlar:
            m = orgat(k_tr, y[tr], V, s, pool, niqob)
            nat[(pool, niqob)]["tabiiy"].append(aniqlik(m, k_te, y[te], niqob))
            nat[(pool, niqob)]["L64"].append(aniqlik(m, k_te, y[te], niqob, L=64))

    print("\n=== 2. Test aniqligi (3 davr, 3 seed) ===")
    print(f"  {'pooling':<8} {'niqob':<6} {'L = maks test':>14} {'L = 64':>8}")
    for (pool, niqob), r in nat.items():
        print(f"  {pool:<8} {str(niqob):<6} {np.mean(r['tabiiy']):>14.4f} "
              f"{np.mean(r['L64']):>8.4f}")
    print(f"  L = maks test: har sharh {max(uz)} gacha to'ldirilgan; L = 64: 64 gacha")

    print("\n=== 3. Juftlashgan farqlar ===")
    f, se, sez = juft(nat[("mean", True)]["tabiiy"], nat[("cls", True)]["tabiiy"])
    print(f"  o'rtacha - [CLS] (niqobli):           {f:+.4f}, SE {se:.4f}, sezilarli: {sez}")
    for pool, niqob in [("cls", True), ("cls", False)]:
        f, se, sez = juft(nat[(pool, niqob)]["L64"], nat[(pool, niqob)]["tabiiy"])
        nom = "niqobli" if niqob else "niqobsiz"
        print(f"  {nom:<8} [CLS]: (L=64) - (L=maks):  {f:+.4f}, SE {se:.4f}, sezilarli: {sez}")
    siz = nat[("cls", False)]
    if np.mean(siz["L64"]) < np.mean(siz["tabiiy"]) - 0.05:
        print("  niqobsiz model: bir xil sharhlar, faqat PAD soni boshqa - aniqlik tushdi")
    print("  niqobli model: padding uzunligi natijaga ta'sir qilmaydi")
    print("  ⭐ Niqob - ixtiyoriy optimizatsiya emas, to'g'rilik sharti")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  [1] internetdan choynak sotiib oldim bataryasi mo'rt emas ekrani sifatsiz va sifati noqulay ammo qadoqlash oddiy tavsiya qilaman
  [1] sovg'a uchun kurtka ishlatib ko'rdim ovozi o'rtacha yana ishlatib ko'ramiz
  [1] akam uchun planshet ishlatib ko'rdim narxi chiroyli emas lekin qadoqalsh oddiy ovozi normal va narxi normal yana olaman
  o'quv 2400, test 600, lug'at 564 (0 <pad>, 1 <unk>, 2 <cls>)
  test uzunligi ([CLS] bilan): 6..23, o'rtacha 13.8
  sinflar ulushi (test): [0.302, 0.34, 0.358]

=== 2. Test aniqligi (3 davr, 3 seed) ===
  pooling  niqob   L = maks test   L = 64
  cls      True           0.6872   0.6872
  mean     True           0.7117   0.7117
  cls      False          0.6311   0.4644
  L = maks test: har sharh 23 gacha to'ldirilgan; L = 64: 64 gacha

=== 3. Juftlashgan farqlar ===
  o'rtacha - [CLS] (niqobli):           +0.0244, SE 0.0072, sezilarli: True
  niqobli  [CLS]: (L=64) - (L=maks):  +0.0000, SE 0.0000, sezilarli: False
  niqobsiz [CLS]: (L=64) - (L=maks):  -0.1667, SE 0.0115, sezilarli: True
  niqobsiz model: bir xil sharhlar, faqat PAD soni boshqa - aniqlik tushdi
  niqobli model: padding uzunligi natijaga ta'sir qilmaydi
  ⭐ Niqob - ixtiyoriy optimizatsiya emas, to'g'rilik sharti

"""Juftlashgan CV: TF-IDF + LogReg, EmbeddingBag va Transformer encoder."""

from collections import Counter

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold
from sklearn.pipeline import make_pipeline

IJOBIY = ["yaxshi", "zo'r", "a'lo", "ajoyib", "sifatli", "qulay",
          "chiroyli", "mustahkam"]
SALBIY = ["yomon", "sifatsiz", "noqulay", "xunuk", "nosoz", "mo'rt",
          "sust", "chatoq"]
NEYTRAL = ["o'rtacha", "oddiy", "normal", "odatiy", "kutilgandek"]
JIHAT = ["sifati", "narxi", "ekrani", "batareyasi", "ovozi", "qadoqlash",
         "yetkazib berish", "sotuvchi", "dizayni", "materiali"]
MAHSULOT = ["telefon", "noutbuk", "quloqchin", "soat", "choynak", "kurtka",
            "krossovka", "sumka", "planshet", "kolonka"]
BOSH = ["kecha", "o'tgan hafta", "bir oy oldin", "do'kondan", "internetdan",
        "sovg'a uchun", "akam uchun", ""]
FEL = ["oldim", "buyurtma qildim", "sotib oldim", "ishlatib ko'rdim"]
YAKUN = {1: ["tavsiya qilaman", "yana olaman", "hammaga maslahat beraman"],
         -1: ["tavsiya qilmayman", "boshqa olmayman", "pulimga achinaman"],
         0: ["hozircha shu", "vaqt ko'rsatadi", "yana ishlatib ko'ramiz"]}
ULUSH = {1: [0.8, 0.1, 0.1], -1: [0.1, 0.8, 0.1], 0: [0.15, 0.15, 0.7]}


def gap(qutb, rng):
    j = rng.choice(JIHAT)
    juda = "juda " if rng.random() < 0.3 else ""
    r = rng.random()
    if qutb == 1:
        if r < 0.2:
            return f"{j} {rng.choice(SALBIY)} emas"
        if r < 0.3:
            return f"{j} bilan muammo yo'q"
        return f"{j} {juda}{rng.choice(IJOBIY)}"
    if qutb == -1:
        if r < 0.25:
            return f"{j} {rng.choice(IJOBIY)} emas"
        if r < 0.35:
            return f"{j} bilan muammo bor"
        return f"{j} {juda}{rng.choice(SALBIY)}"
    return f"{j} {rng.choice(NEYTRAL)}"


def sharh(rng):
    kayfiyat = int(rng.integers(-1, 2))
    qismlar = [f"{rng.choice(BOSH)} {rng.choice(MAHSULOT)} "
               f"{rng.choice(FEL)}".strip()]
    ballar = []
    for t in range(int(rng.integers(1, 5))):
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        bog = rng.choice(["lekin", "va", "ammo", "ham", ""]) if t else ""
        qismlar.append(f"{bog} {gap(q, rng)}".strip())
        ballar.append(q)
    if rng.random() < 0.5:
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        qismlar.append(rng.choice(YAKUN[q]))
        ballar.append(q)
    ball = np.mean(ballar) + rng.normal(0, 0.1)
    return " ".join(qismlar), 2 if ball > 0.35 else (0 if ball < -0.35 else 1)


def imlo_xato(matn, ulush, rng):
    chiqish = []
    for s in matn.split():
        if len(s) >= 4 and rng.random() < ulush:
            i = int(rng.integers(1, len(s) - 1))
            tur = int(rng.integers(0, 3))
            if tur == 0:
                s = s[:i] + s[i + 1:]
            elif tur == 1:
                s = s[:i] + s[i] + s[i:]
            else:
                s = s[:i - 1] + s[i] + s[i - 1] + s[i + 1:]
        chiqish.append(s)
    return " ".join(chiqish)


def korpus(n, seed=0, xato=0.07):
    """23-qismdagi sintetik o'zbekcha sharhlar: 0 salbiy, 1 neytral, 2 ijobiy."""
    rng = np.random.default_rng(seed)
    toza, y = zip(*[sharh(rng) for _ in range(n)])
    rng2 = np.random.default_rng(seed + 1000)
    return [imlo_xato(t, xato, rng2) for t in toza], np.array(y)


PAD, UNK, CLS = 0, 1, 2


def lugat_qur(matnlar):
    c = Counter(s for m in matnlar for s in m.split())
    return {s: i + 3 for i, s in enumerate(sorted(t for t, k in c.items() if k >= 2))}


def kodla(matnlar, lugat):
    return [[CLS] + [lugat.get(s, UNK) for s in m.split()] for m in matnlar]


def toldir(ketmalar, L=None):
    L = L or max(len(k) for k in ketmalar)
    X = torch.zeros(len(ketmalar), L, dtype=torch.long)
    for i, k in enumerate(ketmalar):
        X[i, :len(k)] = torch.tensor(k)
    return X


class Encoder(nn.Module):
    def __init__(self, V, pooling="mean", d=32, h=2, n=2, d_ff=64, maks=64, p=0.1):
        super().__init__()
        self.pooling = pooling
        self.emb = nn.Embedding(V, d, padding_idx=PAD)
        self.pos = nn.Embedding(maks, d)
        qatlam = nn.TransformerEncoderLayer(d, h, d_ff, p, batch_first=True, norm_first=True)
        self.enc = nn.TransformerEncoder(qatlam, n, norm=nn.LayerNorm(d),
                                         enable_nested_tensor=False)
        self.chiq = nn.Linear(d, 3)

    def forward(self, X, niqob=True):
        pad = X == PAD
        x = self.emb(X) + self.pos(torch.arange(X.shape[1]))
        x = self.enc(x, src_key_padding_mask=pad if niqob else None)
        if self.pooling == "cls":
            return self.chiq(x[:, 0])
        m = (~pad).unsqueeze(-1).float()
        return self.chiq((x * m).sum(1) / m.sum(1))


def encoder_bahola(Xtr, ytr, Xte, yte, seed, davr=4):
    lugat = lugat_qur(Xtr)
    k_tr, k_te = kodla(Xtr, lugat), kodla(Xte, lugat)
    torch.manual_seed(seed)
    m = Encoder(len(lugat) + 3)
    opt = torch.optim.AdamW(m.parameters(), lr=0.003, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    yt = torch.tensor(ytr)
    X_hammasi = toldir(k_tr)
    uzunlik = torch.tensor([len(k) for k in k_tr])
    for _ in range(davr):
        m.train()
        tartib = torch.randperm(len(k_tr), generator=g).tolist()
        for i in range(0, len(tartib), 64):
            b = tartib[i:i + 64]
            X = X_hammasi[b][:, :int(uzunlik[b].max())]
            loss = F.cross_entropy(m(X), yt[b])
            opt.zero_grad()
            loss.backward()
            opt.step()
    m.eval()
    with torch.no_grad():
        return m(toldir(k_te)).argmax(1).numpy(), sum(p.numel() for p in m.parameters())


def tokenlar(matn):
    s = matn.split()
    return s + [f"{a}_{b}" for a, b in zip(s, s[1:])]


def paket(ketmalar):
    offsets = torch.tensor([0] + [len(k) for k in ketmalar[:-1]]).cumsum(0)
    return torch.tensor([t for k in ketmalar for t in k]), offsets


def emb_bahola(Xtr, ytr, Xte, seed, davrlar=10):
    """23.7-darsdagi EmbeddingBag + bigram tokenlar."""
    c = Counter(t for m in Xtr for t in tokenlar(m))
    lugat = {t: i + 2 for i, t in enumerate(sorted(t for t, k in c.items() if k >= 2))}
    kod = lambda ms: [[lugat.get(t, 1) for t in tokenlar(m)] for m in ms]
    k_tr, k_te = kod(Xtr), kod(Xte)
    torch.manual_seed(seed)
    bag = nn.EmbeddingBag(len(lugat) + 2, 32, mode="mean")
    chiq = nn.Linear(32, 3)
    opt = torch.optim.Adam(list(bag.parameters()) + list(chiq.parameters()), lr=0.01)
    g = torch.Generator().manual_seed(seed)
    yt = torch.tensor(ytr)
    for _ in range(davrlar):
        tartib = torch.randperm(len(k_tr), generator=g).tolist()
        for i in range(0, len(tartib), 64):
            b = tartib[i:i + 64]
            opt.zero_grad()
            F.cross_entropy(chiq(bag(*paket([k_tr[j] for j in b]))), yt[b]).backward()
            opt.step()
    with torch.no_grad():
        return chiq(bag(*paket(k_te))).argmax(1).numpy()


def tfidf_bahola(Xtr, ytr, Xte):
    m = make_pipeline(TfidfVectorizer(token_pattern=r"[\w']+", ngram_range=(1, 2)),
                      LogisticRegression(C=10, max_iter=3000))
    return m.fit(Xtr, ytr).predict(Xte)


def main() -> None:
    torch.set_num_threads(1)
    matnlar, y = korpus(3000)
    emas = np.array(["emas" in m.split() for m in matnlar])
    cv = StratifiedKFold(5, shuffle=True, random_state=0)
    modellar = ["TF-IDF", "EmbBag", "Encoder"]
    aniq = {k: [] for k in modellar}
    emas_aniq = {k: [] for k in modellar}
    for k, (tr, te) in enumerate(cv.split(matnlar, y)):
        Xtr = [matnlar[i] for i in tr]
        Xte = [matnlar[i] for i in te]
        bashorat = {"TF-IDF": tfidf_bahola(Xtr, y[tr], Xte),
                    "EmbBag": emb_bahola(Xtr, y[tr], Xte, seed=k)}
        bashorat["Encoder"], n_enc = encoder_bahola(Xtr, y[tr], Xte, y[te], seed=k)
        for nom in modellar:
            togri = bashorat[nom] == y[te]
            aniq[nom].append(togri.mean())
            emas_aniq[nom].append(togri[emas[te]].mean())

    print("=== 1. 5 fold, bir xil bo'linishlar ===")
    print(f"  {'fold':>5}" + "".join(f"{n:>9}" for n in modellar))
    for i in range(5):
        print(f"  {i + 1:>5}" + "".join(f"{aniq[n][i]:>9.4f}" for n in modellar))
    print("  o'rt." + "".join(f"{np.mean(aniq[n]):>9.4f}" for n in modellar))
    print(f"  Encoder parametrlari: {n_enc} (2 blok, d = 32, o'rtacha pooling, 4 davr)")

    print("\n=== 2. Juftlashgan farqlar (eng yaxshisiga nisbatan) ===")
    eng = max(modellar, key=lambda n: np.mean(aniq[n]))
    print(f"  eng yuqori o'rtacha: {eng}")
    yomon_emas = []
    for nom in modellar:
        if nom == eng:
            continue
        f = np.array(aniq[nom]) - np.array(aniq[eng])
        se = f.std(ddof=1) / np.sqrt(len(f))
        sez = abs(f.mean()) > 2 * se
        print(f"  {nom} - {eng}: {f.mean():+.4f}, SE {se:.4f}, sezilarli: {sez}")
        if not sez:
            yomon_emas.append(nom)
    soddalik = {"TF-IDF": 0, "EmbBag": 1, "Encoder": 2}
    tanlov = min([eng] + yomon_emas, key=lambda n: soddalik[n])
    print(f"  QAROR (eng yaxshisidan sezilarli yomon bo'lmagan eng sodda): {tanlov}")

    print("\n=== 3. 'emas' bor sharhlar (inkor) ===")
    print(f"  ulushi: {emas.mean():.1%}")
    print("  " + "".join(f"{n:>9}" for n in modellar))
    print("  " + "".join(f"{np.mean(emas_aniq[n]):>9.4f}" for n in modellar))
    f = np.array(emas_aniq["Encoder"]) - np.array(emas_aniq["TF-IDF"])
    se = f.std(ddof=1) / np.sqrt(len(f))
    print(f"  Encoder - TF-IDF (inkorli): {f.mean():+.4f}, SE {se:.4f}, "
          f"sezilarli: {abs(f.mean()) > 2 * se}")
    print("  ⭐ Kichik ma'lumotda Transformer bazaviyni yengishi shart emas - o'lchang")


if __name__ == "__main__":
    main()

=== 1. 5 fold, bir xil bo'linishlar ===
   fold   TF-IDF   EmbBag  Encoder
      1   0.7950   0.8033   0.7033
      2   0.8150   0.8117   0.7300
      3   0.8017   0.8083   0.7067
      4   0.8300   0.8383   0.7400
      5   0.7950   0.7950   0.7183
  o'rt.   0.8073   0.8113   0.7197
  Encoder parametrlari: 37763 (2 blok, d = 32, o'rtacha pooling, 4 davr)

=== 2. Juftlashgan farqlar (eng yaxshisiga nisbatan) ===
  eng yuqori o'rtacha: EmbBag
  TF-IDF - EmbBag: -0.0040, SE 0.0024, sezilarli: False
  Encoder - EmbBag: -0.0917, SE 0.0052, sezilarli: True
  QAROR (eng yaxshisidan sezilarli yomon bo'lmagan eng sodda): TF-IDF

=== 3. 'emas' bor sharhlar (inkor) ===
  ulushi: 32.2%
     TF-IDF   EmbBag  Encoder
     0.7808   0.7849   0.5670
  Encoder - TF-IDF (inkorli): -0.2138, SE 0.0174, sezilarli: True
  ⭐ Kichik ma'lumotda Transformer bazaviyni yengishi shart emas - o'lchang

"""Attention og'irliklarini tahlil qilish: [CLS] qayerga qaraydi va bu tushuntirishmi."""

from collections import Counter

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

IJOBIY = ["yaxshi", "zo'r", "a'lo", "ajoyib", "sifatli", "qulay",
          "chiroyli", "mustahkam"]
SALBIY = ["yomon", "sifatsiz", "noqulay", "xunuk", "nosoz", "mo'rt",
          "sust", "chatoq"]
NEYTRAL = ["o'rtacha", "oddiy", "normal", "odatiy", "kutilgandek"]
JIHAT = ["sifati", "narxi", "ekrani", "batareyasi", "ovozi", "qadoqlash",
         "yetkazib berish", "sotuvchi", "dizayni", "materiali"]
MAHSULOT = ["telefon", "noutbuk", "quloqchin", "soat", "choynak", "kurtka",
            "krossovka", "sumka", "planshet", "kolonka"]
BOSH = ["kecha", "o'tgan hafta", "bir oy oldin", "do'kondan", "internetdan",
        "sovg'a uchun", "akam uchun", ""]
FEL = ["oldim", "buyurtma qildim", "sotib oldim", "ishlatib ko'rdim"]
YAKUN = {1: ["tavsiya qilaman", "yana olaman", "hammaga maslahat beraman"],
         -1: ["tavsiya qilmayman", "boshqa olmayman", "pulimga achinaman"],
         0: ["hozircha shu", "vaqt ko'rsatadi", "yana ishlatib ko'ramiz"]}
ULUSH = {1: [0.8, 0.1, 0.1], -1: [0.1, 0.8, 0.1], 0: [0.15, 0.15, 0.7]}


def gap(qutb, rng):
    j = rng.choice(JIHAT)
    juda = "juda " if rng.random() < 0.3 else ""
    r = rng.random()
    if qutb == 1:
        if r < 0.2:
            return f"{j} {rng.choice(SALBIY)} emas"
        if r < 0.3:
            return f"{j} bilan muammo yo'q"
        return f"{j} {juda}{rng.choice(IJOBIY)}"
    if qutb == -1:
        if r < 0.25:
            return f"{j} {rng.choice(IJOBIY)} emas"
        if r < 0.35:
            return f"{j} bilan muammo bor"
        return f"{j} {juda}{rng.choice(SALBIY)}"
    return f"{j} {rng.choice(NEYTRAL)}"


def sharh(rng):
    kayfiyat = int(rng.integers(-1, 2))
    qismlar = [f"{rng.choice(BOSH)} {rng.choice(MAHSULOT)} "
               f"{rng.choice(FEL)}".strip()]
    ballar = []
    for t in range(int(rng.integers(1, 5))):
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        bog = rng.choice(["lekin", "va", "ammo", "ham", ""]) if t else ""
        qismlar.append(f"{bog} {gap(q, rng)}".strip())
        ballar.append(q)
    if rng.random() < 0.5:
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        qismlar.append(rng.choice(YAKUN[q]))
        ballar.append(q)
    ball = np.mean(ballar) + rng.normal(0, 0.1)
    return " ".join(qismlar), 2 if ball > 0.35 else (0 if ball < -0.35 else 1)


def imlo_xato(matn, ulush, rng):
    chiqish = []
    for s in matn.split():
        if len(s) >= 4 and rng.random() < ulush:
            i = int(rng.integers(1, len(s) - 1))
            tur = int(rng.integers(0, 3))
            if tur == 0:
                s = s[:i] + s[i + 1:]
            elif tur == 1:
                s = s[:i] + s[i] + s[i:]
            else:
                s = s[:i - 1] + s[i] + s[i - 1] + s[i + 1:]
        chiqish.append(s)
    return " ".join(chiqish)


def korpus(n, seed=0, xato=0.07):
    """23-qismdagi sintetik o'zbekcha sharhlar: 0 salbiy, 1 neytral, 2 ijobiy."""
    rng = np.random.default_rng(seed)
    toza, y = zip(*[sharh(rng) for _ in range(n)])
    rng2 = np.random.default_rng(seed + 1000)
    return [imlo_xato(t, xato, rng2) for t in toza], np.array(y)


PAD, UNK, CLS = 0, 1, 2


def lugat_qur(matnlar):
    c = Counter(s for m in matnlar for s in m.split())
    return {s: i + 3 for i, s in enumerate(sorted(t for t, k in c.items() if k >= 2))}


def kodla(matnlar, lugat):
    return [[CLS] + [lugat.get(s, UNK) for s in m.split()] for m in matnlar]


def toldir(ketmalar, L=None):
    L = L or max(len(k) for k in ketmalar)
    X = torch.zeros(len(ketmalar), L, dtype=torch.long)
    for i, k in enumerate(ketmalar):
        X[i, :len(k)] = torch.tensor(k)
    return X


class Encoder(nn.Module):
    def __init__(self, V, pooling="cls", d=32, h=2, n=2, d_ff=64, maks=64, p=0.1):
        super().__init__()
        self.pooling = pooling
        self.emb = nn.Embedding(V, d, padding_idx=PAD)
        self.pos = nn.Embedding(maks, d)
        qatlam = nn.TransformerEncoderLayer(d, h, d_ff, p, batch_first=True, norm_first=True)
        self.enc = nn.TransformerEncoder(qatlam, n, norm=nn.LayerNorm(d),
                                         enable_nested_tensor=False)
        self.chiq = nn.Linear(d, 3)

    def forward(self, X, niqob=True):
        pad = X == PAD
        x = self.emb(X) + self.pos(torch.arange(X.shape[1]))
        x = self.enc(x, src_key_padding_mask=pad if niqob else None)
        if self.pooling == "cls":
            return self.chiq(x[:, 0])
        m = (~pad).unsqueeze(-1).float()
        return self.chiq((x * m).sum(1) / m.sum(1))


def orgat(k_tr, ytr, V, seed, davr=5):
    torch.manual_seed(seed)
    m = Encoder(V, "cls")
    opt = torch.optim.AdamW(m.parameters(), lr=0.003, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    yt = torch.tensor(ytr)
    X_hammasi = toldir(k_tr)
    uzunlik = torch.tensor([len(k) for k in k_tr])
    for _ in range(davr):
        m.train()
        tartib = torch.randperm(len(k_tr), generator=g).tolist()
        for i in range(0, len(tartib), 64):
            b = tartib[i:i + 64]
            X = X_hammasi[b][:, :int(uzunlik[b].max())]
            loss = F.cross_entropy(m(X), yt[b])
            opt.zero_grad()
            loss.backward()
            opt.step()
    m.eval()
    return m


def cls_ogirliklari(m, X):
    """Oxirgi blokda [CLS] qatori: (B, L), boshlar bo'yicha o'rtacha."""
    pad = X == PAD
    with torch.no_grad():
        x = m.emb(X) + m.pos(torch.arange(X.shape[1]))
        for qatlam in m.enc.layers[:-1]:
            x = qatlam(x, src_key_padding_mask=pad)
        oxirgi = m.enc.layers[-1]
        y = oxirgi.norm1(x)                          # pre-LN: attention LN dan keyin
        _, w = oxirgi.self_attn(y, y, y, key_padding_mask=pad, need_weights=True,
                                average_attn_weights=True)
    return w[:, 0]


def turi(soz):
    if soz in IJOBIY or soz in SALBIY:
        return "baho so'zi"
    if soz == "emas":
        return "emas"
    if soz in NEYTRAL:
        return "neytral"
    if soz in " ".join(JIHAT).split():
        return "jihat"
    return "boshqa"


def main() -> None:
    torch.set_num_threads(1)
    matnlar, y = korpus(3000)
    idx = np.random.default_rng(0).permutation(len(matnlar))
    tr, te = idx[:2400], idx[2400:]
    lugat = lugat_qur([matnlar[i] for i in tr])
    i2s = {i: s for s, i in lugat.items()}
    k_tr = kodla([matnlar[i] for i in tr], lugat)
    k_te = kodla([matnlar[i] for i in te], lugat)
    m = orgat(k_tr, y[tr], len(lugat) + 3, 0)
    Xt = toldir(k_te)
    with torch.no_grad():
        p = m(Xt).argmax(1).numpy()
    print("=== 1. Model ([CLS] pooling, 5 davr) ===")
    print(f"  test aniqligi: {(p == y[te]).mean():.4f}")

    W = cls_ogirliklari(m, Xt)
    print("\n=== 2. Bitta sharh: [CLS] qayerga qaraydi (oxirgi blok, x100) ===")
    j = 4
    sozlar = ["[CLS]"] + [i2s.get(t, "<unk>") for t in k_te[j][1:]]
    print(f"  yorliq {y[te][j]}, bashorat {p[j]}")
    for s, w in zip(sozlar, W[j, :len(sozlar)].tolist()):
        print(f"    {s:<12} {w * 100:>5.1f} " + "#" * int(round(w * 60)))

    print("\n=== 3. So'z turlari bo'yicha o'rtacha og'irlik (600 test sharh) ===")
    yigindi, soni = Counter(), Counter()
    for b, k in enumerate(k_te):
        for t, tok in enumerate(k[1:], start=1):
            tur = turi(i2s.get(tok, "<unk>"))
            yigindi[tur] += W[b, t].item() * len(k)     # 1.0 = bir tekis taqsimot
            soni[tur] += 1
    sarlavha = "nisbiy og'irlik"
    print(f"  {'tur':<10} {'tokenlar':>9} {sarlavha:>16}")
    for tur in ["baho so'zi", "emas", "neytral", "jihat", "boshqa"]:
        print(f"  {tur:<10} {soni[tur]:>9} {yigindi[tur] / soni[tur]:>16.2f}")
    print("  nisbiy og'irlik 1.00 - bir tekis (1/L); 2.00 - o'rtachadan ikki barobar")

    print("\n=== 4. Tekshiruv: eng ko'p qaralgan so'zni olib tashlash ===")
    rng = np.random.default_rng(0)
    X_top, X_tas = Xt.clone(), Xt.clone()
    for b, k in enumerate(k_te):
        L = len(k)
        top = int(W[b, 1:L].argmax()) + 1
        X_top[b, top] = UNK
        X_tas[b, int(rng.integers(1, L))] = UNK
    with torch.no_grad():
        p_top = m(X_top).argmax(1).numpy()
        p_tas = m(X_tas).argmax(1).numpy()
    ozg_top, ozg_tas = (p_top != p).mean(), (p_tas != p).mean()
    print(f"  bashorat o'zgardi: eng ko'p qaralgan -> <unk>: {ozg_top:.3f}, "
          f"tasodifiy so'z -> <unk>: {ozg_tas:.3f}")
    X_past, tanlangan = Xt.clone(), []
    for b, k in enumerate(k_te):
        L = len(k)
        past = [t for t in range(1, L) if turi(i2s.get(k[t], "<unk>")) == "baho so'zi"
                and W[b, t].item() < 1 / L]
        if past:
            X_past[b, past[0]] = UNK
            tanlangan.append(b)
    with torch.no_grad():
        p_past = m(X_past).argmax(1).numpy()
    ozg_past = (p_past[tanlangan] != p[tanlangan]).mean()
    print(f"  og'irligi 1/L dan KAM baho so'zi -> <unk> ({len(tanlangan)} sharh): "
          f"{ozg_past:.3f}")
    ustun = W[:, 1:].max(1).values
    print(f"  [CLS] ning eng katta og'irligi (o'rtacha): {ustun.mean().item():.3f}")
    if ozg_top > 2 * ozg_tas:
        print("  eng ko'p qaralgan so'z haqiqatan muhimroq - lekin bu faqat bitta test")
    if ozg_past > ozg_tas:
        print("  kam qaralgan baho so'zi ham tasodifiy so'zdan KAM ta'sir qilmadi:")
        print("  past og'irlik 'muhim emas' degani emas (ma'lumot oldingi bloklarda aralashgan)")
    print("  ⭐ Og'irlik - 'qayerdan olindi', sabab emas; olib tashlash bilan tekshiring")


if __name__ == "__main__":
    main()

=== 1. Model ([CLS] pooling, 5 davr) ===
  test aniqligi: 0.7483

=== 2. Bitta sharh: [CLS] qayerga qaraydi (oxirgi blok, x100) ===
  yorliq 0, bashorat 1
    [CLS]          1.2 #
    bir            1.3 #
    oy             0.1
    oldiin         0.1
    telefon        0.1
    buyurtma       0.6
    qildim         3.4 ##
    dizayni        5.8 ####
    normal        65.8 #######################################
    ovozi         21.6 #############
    sifatsiz       0.1

=== 3. So'z turlari bo'yicha o'rtacha og'irlik (600 test sharh) ===
  tur         tokenlar  nisbiy og'irlik
  baho so'zi       874             1.80
  emas             215             6.22
  neytral          437             4.09
  jihat           1519             0.45
  boshqa          4633             0.57
  nisbiy og'irlik 1.00 - bir tekis (1/L); 2.00 - o'rtachadan ikki barobar

=== 4. Tekshiruv: eng ko'p qaralgan so'zni olib tashlash ===
  bashorat o'zgardi: eng ko'p qaralgan -> <unk>: 0.302, tasodifiy so'z -> <unk>: 0.080
  og'irligi 1/L dan KAM baho so'zi -> <unk> (311 sharh): 0.106
  [CLS] ning eng katta og'irligi (o'rtacha): 0.478
  eng ko'p qaralgan so'z haqiqatan muhimroq - lekin bu faqat bitta test
  kam qaralgan baho so'zi ham tasodifiy so'zdan KAM ta'sir qilmadi:
  past og'irlik 'muhim emas' degani emas (ma'lumot oldingi bloklarda aralashgan)
  ⭐ Og'irlik - 'qayerdan olindi', sabab emas; olib tashlash bilan tekshiring

x = self.enc(x)                                                  # ⚠️
x = self.enc(x, src_key_padding_mask=(X == PAD))                 # ✅

self.enc(x, src_key_padding_mask=(X != PAD))  # haqiqiy so'zlar o'chadi # ⚠️
self.enc(x, src_key_padding_mask=(X == PAD))  # True = e'tiborsiz   # ✅

h = x.mean(1)                                                    # ⚠️
m = (X != PAD).unsqueeze(-1).float(); h = (x * m).sum(1) / m.sum(1)  # ✅

ids = [lugat.get(s, UNK) for s in m.split()]; h = x[:, 0]       # ⚠️
ids = [CLS] + [lugat.get(s, UNK) for s in m.split()]             # ✅

lugat = lugat_qur(matnlar)                    # test ham kirdi   # ⚠️
lugat = lugat_qur([matnlar[i] for i in tr])                      # ✅

nn.TransformerEncoder(qatlam, 2)              # norm_first=True bilan warning # ⚠️
nn.TransformerEncoder(qatlam, 2, enable_nested_tensor=False)                 # ✅

print(f"Transformer aniqligi: {aniq:.3f}")                       # ⚠️
# TF-IDF va EmbeddingBag bilan bir xil foldlarda, farq + SE     # ✅

1.  # encoder bosqichlari tartibi?
2.  # X (B, L) dan keyin bloklardan chiqqan shakl?
3.  # [CLS] pooling formulasi?
4.  # maskali o'rtacha formulasi?
5.  # src_key_padding_mask da True nimani anglatadi?
6.  # niqobsiz modelda padding uzunligi o'zgarsa?
7.  # nima uchun <pad> embeddingi nol bo'lsa ham niqob kerak?
8.  # enable_nested_tensor=False nima uchun?
9.  # qaror qoidasi?
10. # soddalik tartibi (TF-IDF, EmbBag, Transformer)?
11. # nisbiy og'irlik 1.00 nimani bildiradi?
12. # attention og'irligini qanday tekshirish mumkin?

1.  x = self.enc(self.emb(X) + self.pos(torch.arange(X.shape[1])))
    return self.chiq(x.mean(1))

2.  x = self.enc(x, src_key_padding_mask=(X > 0))

3.  lugat = lugat_qur(matnlar)
    k_tr = kodla([matnlar[i] for i in tr], lugat)

4.  ids = [lugat.get(s, UNK) for s in matn.split()]
    h = self.enc(x, src_key_padding_mask=pad)[:, 0]    # [CLS] pooling

5.  aniq = bahola(encoder, X_test, y_test)
    print("Transformer yaxshi" if aniq > 0.7 else "yomon")