23.7-dars: Embedding bilan klassifikatsiya

MATN -> INDEKSLAR -> VEKTORLAR -> BITTA VEKTOR -> SINF
  "sifati juda zo'r" -> [57, 31, 102] -> (3, d) -> (d,) -> Linear -> 3 logit

LUG'AT 23.2-bob:
  0 = <pad>, 1 = <unk>, qolganlari 2..V-1
  faqat O'QUV to'plamida quriladi, kam uchraganlar -> <unk>

O'RTACHA POOLING:
  h = (1 / L) * sum_{t=1..L} E[x_t]
  so'zlar tartibi YO'QOLADI: "yaxshi emas" va "emas yaxshi" - bir xil h

BU NIMAGA TENG:
  Linear(h) = W * (1/L) * sum E[x_t] = (1/L) * sum (W E)[x_t]
  -> har so'zga (W E) orqali sinf "ovozi" beriladi
  -> bag-of-words ustidagi chiziqli model, lekin og'irlik matritsasi
     past rangli: (V, d) @ (d, 3)
  -> kutilgan natija: TF-IDF unigram + LogReg ga yaqin

PARAMETRLAR:
  V * d + d * 3 + 3   (V = 1000, d = 32 -> ~32 000)

BATCH (21.3 dagi collate_fn):
  sharh 1: [57, 31, 102]          -> [57, 31, 102,  0,  0]
  sharh 2: [12, 9, 44, 7, 88]     -> [12,  9, 44,  7, 88]
  mask:                              [1, 1, 1, 0, 0], [1, 1, 1, 1, 1]

NOTO'G'RI O'RTACHA:
  e.mean(dim=1)  -> padding vektorlari ham sanaladi
  padding_idx=0 bo'lsa E[0] = 0, lekin maxraj 5 (3 emas)
  -> qisqa sharh vektori L / L_max marta kichrayadi
  -> natija BATCHDAGI eng uzun sharhga bog'liq bo'lib qoladi!

TO'G'RI O'RTACHA:
  (e * mask.unsqueeze(-1)).sum(1) / mask.sum(1, keepdim=True)

padding_idx=0:
  E[0] nol bilan boshlanadi va gradient olmaydi - o'zgarmaydi
  lekin bu maskaning o'rnini BOSMAYDI (maxraj muammosi qoladi)

G'OYA: padding umuman kerak emas
  barcha sharhlarni bitta uzun 1D tensorga ulaymiz
  offsets - har sharh qayerdan boshlanishi

  sharhlar: [57, 31, 102], [12, 9, 44, 7, 88], [5, 6]
  ids:     [57, 31, 102, 12, 9, 44, 7, 88, 5, 6]
  offsets: [0, 3, 8]

  bag = nn.EmbeddingBag(V, d, mode="mean")
  h = bag(ids, offsets)          # (3, d) - to'g'ridan-to'g'ri o'rtacha

AFZALLIKLAR:
  padding va maska yo'q - xato qilish imkoni kam
  (L, d) oraliq tensor qurilmaydi - tez va tejamkor
  mode: "mean", "sum", "max"

TENGLIK:
  EmbeddingBag(mean) == maskali o'rtacha (1-misolda tekshiramiz)

MUAMMO:
  "sifati yaxshi emas"  va  "emas sifati yaxshi" -> bir xil vektor
  "yaxshi" kuchli ijobiy, "emas" deyarli neytral -> "ijobiy"

YECHIM 1 - bigram tokenlar (fastText g'oyasi):
  tokenlar: sifati, yaxshi, emas, sifati_yaxshi, yaxshi_emas
  "yaxshi_emas" o'z embeddingini oladi -> inkor ko'rinadi
  lug'at kattalashadi (min_soni bilan cheklanadi)

YECHIM 2 - tartibni ko'radigan modellar:
  RNN (keyingi dars), LSTM/GRU, Transformerlar qismida - attention

23.5 BILAN PARALLEL:
  TF-IDF unigram    ~  embedding unigram
  TF-IDF + bigram   ~  embedding + bigram token

QADAMLAR:
  1. katta BELGILANMAGAN korpusda word2vec 23.6-bob -> E (V, d)
  2. klassifikatorda: bag.weight.data.copy_(E)
  3a. MUZLATISH:   bag.weight.requires_grad_(False)
      faqat Linear o'rganadi (d * 3 + 3 parametr)
  3b. FINE-TUNE:   hamma parametr o'rganadi, E - boshlang'ich nuqta

QACHON FOYDALI:
  belgilangan ma'lumot KAM, belgilanmagani KO'P
  testda o'quvda uchramagan so'zlar bor (sinonimlar) - ular
  embedding orqali tanish so'zlarga yaqin

QACHON FOYDASIZ YOKI ZARARLI:
  embedding vazifaga kerakli farqni saqlamagan bo'lsa:
  word2vec "yaxshi" va "yomon" ni YAQIN qo'yadi - kontekstlari
  bir xil ("sifati ___") -> sentiment uchun muhim farq yo'qolgan
  muzlatilgan embedding bu farqni tiklay olmaydi
  fine-tune qila oladi - lekin kam ma'lumotda qiyin

LUG'AT:
  oldindan o'rgatilgan embedding lug'ati ishlatiladi
  (klassifikator lug'ati u bilan bir xil bo'lishi SHART)

BAZAVIY: 23.5 dagi TF-IDF (so'z + bigram) + LogReg
  sozlash deyarli yo'q, soniyalarda o'rganadi

HALOL SHARTLAR:
  bir xil bo'linishlar (juftlashgan CV yoki bir xil qism to'plamlar)
  tarmoq uchun bir necha seed
  ikkala tomonga teng "imkoniyat": bigram faqat bir tomonga berilmaydi
  kam ma'lumot va to'liq ma'lumot - alohida

QAROR QOIDASI 23.5-bob:
  eng yaxshisidan sezilarli yomon bo'lmagan ENG SODDA model
  soddalik: TF-IDF + LogReg < EmbeddingBag < oldindan o'rgatilgan + fine-tune

BAZAVIY YUTSA:
  bu ham natija - uni yozing
  "tarmoq zamonaviyroq" - dalil emas

import torch
import torch.nn as nn
from collections import Counter


def lugat_qur(matnlar, min_soni=2):
    c = Counter(s for m in matnlar for s in m.split())
    sozlar = sorted(s for s, k in c.items() if k >= min_soni)
    return {"<pad>": 0, "<unk>": 1} | {s: i + 2 for i, s in enumerate(sozlar)}


def toldir(batch):                        # collate_fn: padding + maska
    ketmalar, y = zip(*batch)
    L = max(len(k) for k in ketmalar)
    ids = torch.zeros(len(ketmalar), L, dtype=torch.long)
    for i, k in enumerate(ketmalar):
        ids[i, :len(k)] = torch.tensor(k)
    return ids, (ids != 0).float(), torch.tensor(y)


emb = nn.Embedding(V, d, padding_idx=0)
e = emb(ids)                                            # (B, L, d)
h = (e * mask.unsqueeze(-1)).sum(1) / mask.sum(1, keepdim=True)

bag = nn.EmbeddingBag(V, d, mode="mean")
ids1 = torch.tensor([i for k in ketmalar for i in k])
offsets = torch.tensor([0] + [len(k) for k in ketmalar[:-1]]).cumsum(0)
h = bag(ids1, offsets)                                  # (B, d)

bag.weight.data.copy_(E_oldindan)                       # oldindan o'rgatilgan
bag.weight.requires_grad_(False)                        # muzlatish
opt = torch.optim.Adam([p for p in model.parameters() if p.requires_grad])

lug'at o'quvda: 0 = <pad>, 1 = <unk>
o'rtacha: maska bilan (yoki EmbeddingBag + offsets)
o'rtacha pooling tartibni ko'rmaydi -> bigram tokenlar
oldindan o'rgatilgan: muzlatish / fine-tune - o'lchang
TF-IDF + LogReg bilan juftlashgan taqqoslash, bazaviy yutsa - yozing

"""O'rtacha pooling ning texnik qismi: collate_fn, maska, offsets."""

from collections import Counter

import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import DataLoader

IJOBIY = ["yaxshi", "zo'r", "a'lo", "ajoyib", "sifatli", "qulay",
          "chiroyli", "mustahkam"]
SALBIY = ["yomon", "sifatsiz", "noqulay", "xunuk", "nosoz", "mo'rt",
          "sust", "chatoq"]
NEYTRAL = ["o'rtacha", "oddiy", "normal", "odatiy", "kutilgandek"]
JIHAT = ["sifati", "narxi", "ekrani", "batareyasi", "ovozi", "qadoqlash",
         "yetkazib berish", "sotuvchi", "dizayni", "materiali"]
MAHSULOT = ["telefon", "noutbuk", "quloqchin", "soat", "choynak", "kurtka",
            "krossovka", "sumka", "planshet", "kolonka"]
BOSH = ["kecha", "o'tgan hafta", "bir oy oldin", "do'kondan", "internetdan",
        "sovg'a uchun", "akam uchun", ""]
FEL = ["oldim", "buyurtma qildim", "sotib oldim", "ishlatib ko'rdim"]
YAKUN = {1: ["tavsiya qilaman", "yana olaman", "hammaga maslahat beraman"],
         -1: ["tavsiya qilmayman", "boshqa olmayman", "pulimga achinaman"],
         0: ["hozircha shu", "vaqt ko'rsatadi", "yana ishlatib ko'ramiz"]}
ULUSH = {1: [0.8, 0.1, 0.1], -1: [0.1, 0.8, 0.1], 0: [0.15, 0.15, 0.7]}


def gap(qutb, rng):
    j = rng.choice(JIHAT)
    juda = "juda " if rng.random() < 0.3 else ""
    r = rng.random()
    if qutb == 1:
        if r < 0.2:
            return f"{j} {rng.choice(SALBIY)} emas"
        if r < 0.3:
            return f"{j} bilan muammo yo'q"
        return f"{j} {juda}{rng.choice(IJOBIY)}"
    if qutb == -1:
        if r < 0.25:
            return f"{j} {rng.choice(IJOBIY)} emas"
        if r < 0.35:
            return f"{j} bilan muammo bor"
        return f"{j} {juda}{rng.choice(SALBIY)}"
    return f"{j} {rng.choice(NEYTRAL)}"


def sharh(rng):
    kayfiyat = int(rng.integers(-1, 2))
    qismlar = [f"{rng.choice(BOSH)} {rng.choice(MAHSULOT)} "
               f"{rng.choice(FEL)}".strip()]
    ballar = []
    for t in range(int(rng.integers(1, 5))):
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        bog = rng.choice(["lekin", "va", "ammo", "ham", ""]) if t else ""
        qismlar.append(f"{bog} {gap(q, rng)}".strip())
        ballar.append(q)
    if rng.random() < 0.5:
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        qismlar.append(rng.choice(YAKUN[q]))
        ballar.append(q)
    ball = np.mean(ballar) + rng.normal(0, 0.1)
    return " ".join(qismlar), 2 if ball > 0.35 else (0 if ball < -0.35 else 1)


def imlo_xato(matn, ulush, rng):
    chiqish = []
    for s in matn.split():
        if len(s) >= 4 and rng.random() < ulush:
            i = int(rng.integers(1, len(s) - 1))
            tur = int(rng.integers(0, 3))
            if tur == 0:
                s = s[:i] + s[i + 1:]
            elif tur == 1:
                s = s[:i] + s[i] + s[i:]
            else:
                s = s[:i - 1] + s[i] + s[i - 1] + s[i + 1:]
        chiqish.append(s)
    return " ".join(chiqish)


def korpus(n, seed=0, xato=0.07):
    rng = np.random.default_rng(seed)
    toza, y = zip(*[sharh(rng) for _ in range(n)])
    rng2 = np.random.default_rng(seed + 1000)
    return [imlo_xato(t, xato, rng2) for t in toza], np.array(y), list(toza)


def lugat_qur(matnlar, min_soni=2):
    c = Counter(s for m in matnlar for s in m.split())
    sozlar = sorted(s for s, k in c.items() if k >= min_soni)
    return {"<pad>": 0, "<unk>": 1} | {s: i + 2 for i, s in enumerate(sozlar)}


def kodla(matn, lugat):
    return [lugat.get(s, 1) for s in matn.split()]


def toldir(batch):
    """collate_fn 21.3-bob: turli uzunlik -> padding + maska."""
    ketmalar, y = zip(*batch)
    L = max(len(k) for k in ketmalar)
    ids = torch.zeros(len(ketmalar), L, dtype=torch.long)
    for i, k in enumerate(ketmalar):
        ids[i, :len(k)] = torch.tensor(k)
    return ids, (ids != 0).float(), torch.tensor(y)


def main() -> None:
    matnlar, y, _ = korpus(3000)
    oquv, test = matnlar[:2400], matnlar[2400:]

    print("=== 1. Lug'at (faqat o'quvda, min_soni=2) ===")
    lugat = lugat_qur(oquv)
    hammasi = len({s for m in oquv for s in m.split()})
    print(f"  o'quvdagi noyob so'zlar: {hammasi}, lug'at: {len(lugat)} "
          f"(<pad>, <unk> bilan)")
    test_soz = [s for m in test for s in m.split()]
    print(f"  test so'zlarining <unk> ulushi: "
          f"{np.mean([s not in lugat for s in test_soz]):.1%}")
    print(f"  {test[0]}")
    print(f"  -> {kodla(test[0], lugat)}")

    print("\n=== 2. collate_fn: padding va maska ===")
    ds = [(kodla(m, lugat), int(t)) for m, t in zip(oquv, y[:2400])]
    ids, mask, yb = next(iter(DataLoader(ds[:4], batch_size=4,
                                         collate_fn=toldir)))
    print(f"  ids {tuple(ids.shape)}, mask {tuple(mask.shape)}, "
          f"uzunliklar {mask.sum(1).int().tolist()}")
    i = int(mask.sum(1).argmin())
    print(f"  eng qisqa ({i}-sharh) ids: {ids[i].tolist()}")
    print(f"  eng qisqa ({i}-sharh) mask: {mask[i].int().tolist()}")

    print("\n=== 3. Maskali va maskasiz o'rtacha ===")
    torch.manual_seed(0)
    emb = nn.Embedding(len(lugat), 8, padding_idx=0)
    e = emb(ids)
    togri = (e * mask.unsqueeze(-1)).sum(1) / mask.sum(1, keepdim=True)
    notogri = e.mean(1)
    L = ids.shape[1]
    for i in range(4):
        n = int(mask[i].sum())
        nisbat = (notogri[i].norm() / togri[i].norm()).item()
        print(f"  sharh {i}: uzunlik {n:>2}/{L}, |maskasiz| / |maskali| = "
              f"{nisbat:.3f}  (L/L_max = {n / L:.3f})")
    print(f"  pad vektori E[0]: {emb.weight[0].abs().max().item():.1f} "
          f"(padding_idx=0)")
    print("  E[0] = 0 bo'lsa ham maxraj noto'g'ri - vektor kichrayadi")

    print("\n=== 4. EmbeddingBag + offsets ===")
    ketmalar = [ds[i][0] for i in range(4)]
    ids1 = torch.tensor([t for k in ketmalar for t in k])
    offsets = torch.tensor([0] + [len(k) for k in ketmalar[:-1]]).cumsum(0)
    print(f"  ids1 shakli: {tuple(ids1.shape)}, offsets: {offsets.tolist()}")
    bag = nn.EmbeddingBag(len(lugat), 8, mode="mean")
    bag.weight.data.copy_(emb.weight.data)
    h = bag(ids1, offsets)
    farq = (h - togri).abs().max().item()
    print(f"  EmbeddingBag va maskali o'rtacha farqi: {farq:.2e}")
    print(f"  padding siz: {ids1.numel()} ta indeks, padding bilan: "
          f"{ids.numel()} ta")

    print("\n=== 5. padding_idx gradienti ===")
    emb.zero_grad()
    togri.sum().backward()
    print(f"  E[0] gradienti: {emb.weight.grad[0].norm().item():.1f}")
    ishlatilgan = ids[ids > 1][0].item()
    print(f"  E[{ishlatilgan}] gradienti: "
          f"{emb.weight.grad[ishlatilgan].norm().item():.3f}")

    print("\n=== 6. Parametrlar (d = 32) ===")
    model = nn.ModuleDict({"bag": nn.EmbeddingBag(len(lugat), 32),
                           "chiq": nn.Linear(32, 3)})
    for nom, m in model.items():
        print(f"  {nom:<5} {sum(p.numel() for p in m.parameters()):>7}")
    print("  ⭐ Deyarli hamma parametr - embedding jadvalida")


if __name__ == "__main__":
    main()

=== 1. Lug'at (faqat o'quvda, min_soni=2) ===
  o'quvdagi noyob so'zlar: 872, lug'at: 571 (<pad>, <unk> bilan)
  test so'zlarining <unk> ulushi: 2.1%
  akam uchun sumka ishlatib ko'rdim diizayni bilan muammo bor ovozi juda sifatli ham ovozi a'o ovozi sust emas tavsiya qilmayman
  -> [16, 515, 495, 159, 191, 93, 49, 262, 55, 373, 170, 445, 134, 373, 7, 373, 499, 123, 504, 416]

=== 2. collate_fn: padding va maska ===
  ids (4, 20), mask (4, 20), uzunliklar [13, 20, 17, 6]
  eng qisqa (3-sharh) ids: [495, 470, 344, 396, 531, 123, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
  eng qisqa (3-sharh) mask: [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

=== 3. Maskali va maskasiz o'rtacha ===
  sharh 0: uzunlik 13/20, |maskasiz| / |maskali| = 0.650  (L/L_max = 0.650)
  sharh 1: uzunlik 20/20, |maskasiz| / |maskali| = 1.000  (L/L_max = 1.000)
  sharh 2: uzunlik 17/20, |maskasiz| / |maskali| = 0.850  (L/L_max = 0.850)
  sharh 3: uzunlik  6/20, |maskasiz| / |maskali| = 0.300  (L/L_max = 0.300)
  pad vektori E[0]: 0.0 (padding_idx=0)
  E[0] = 0 bo'lsa ham maxraj noto'g'ri - vektor kichrayadi

=== 4. EmbeddingBag + offsets ===
  ids1 shakli: (56,), offsets: [0, 13, 33, 50]
  EmbeddingBag va maskali o'rtacha farqi: 8.94e-08
  padding siz: 56 ta indeks, padding bilan: 80 ta

=== 5. padding_idx gradienti ===
  E[0] gradienti: 0.0
  E[487] gradienti: 0.218

=== 6. Parametrlar (d = 32) ===
  bag     18272
  chiq       99
  ⭐ Deyarli hamma parametr - embedding jadvalida

"""EmbeddingBag klassifikatori: unigram va bigram tokenlar, TF-IDF bilan."""

from collections import Counter

import numpy as np
import torch
import torch.nn as nn
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

IJOBIY = ["yaxshi", "zo'r", "a'lo", "ajoyib", "sifatli", "qulay",
          "chiroyli", "mustahkam"]
SALBIY = ["yomon", "sifatsiz", "noqulay", "xunuk", "nosoz", "mo'rt",
          "sust", "chatoq"]
NEYTRAL = ["o'rtacha", "oddiy", "normal", "odatiy", "kutilgandek"]
JIHAT = ["sifati", "narxi", "ekrani", "batareyasi", "ovozi", "qadoqlash",
         "yetkazib berish", "sotuvchi", "dizayni", "materiali"]
MAHSULOT = ["telefon", "noutbuk", "quloqchin", "soat", "choynak", "kurtka",
            "krossovka", "sumka", "planshet", "kolonka"]
BOSH = ["kecha", "o'tgan hafta", "bir oy oldin", "do'kondan", "internetdan",
        "sovg'a uchun", "akam uchun", ""]
FEL = ["oldim", "buyurtma qildim", "sotib oldim", "ishlatib ko'rdim"]
YAKUN = {1: ["tavsiya qilaman", "yana olaman", "hammaga maslahat beraman"],
         -1: ["tavsiya qilmayman", "boshqa olmayman", "pulimga achinaman"],
         0: ["hozircha shu", "vaqt ko'rsatadi", "yana ishlatib ko'ramiz"]}
ULUSH = {1: [0.8, 0.1, 0.1], -1: [0.1, 0.8, 0.1], 0: [0.15, 0.15, 0.7]}


def gap(qutb, rng):
    j = rng.choice(JIHAT)
    juda = "juda " if rng.random() < 0.3 else ""
    r = rng.random()
    if qutb == 1:
        if r < 0.2:
            return f"{j} {rng.choice(SALBIY)} emas"
        if r < 0.3:
            return f"{j} bilan muammo yo'q"
        return f"{j} {juda}{rng.choice(IJOBIY)}"
    if qutb == -1:
        if r < 0.25:
            return f"{j} {rng.choice(IJOBIY)} emas"
        if r < 0.35:
            return f"{j} bilan muammo bor"
        return f"{j} {juda}{rng.choice(SALBIY)}"
    return f"{j} {rng.choice(NEYTRAL)}"


def sharh(rng):
    kayfiyat = int(rng.integers(-1, 2))
    qismlar = [f"{rng.choice(BOSH)} {rng.choice(MAHSULOT)} "
               f"{rng.choice(FEL)}".strip()]
    ballar = []
    for t in range(int(rng.integers(1, 5))):
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        bog = rng.choice(["lekin", "va", "ammo", "ham", ""]) if t else ""
        qismlar.append(f"{bog} {gap(q, rng)}".strip())
        ballar.append(q)
    if rng.random() < 0.5:
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        qismlar.append(rng.choice(YAKUN[q]))
        ballar.append(q)
    ball = np.mean(ballar) + rng.normal(0, 0.1)
    return " ".join(qismlar), 2 if ball > 0.35 else (0 if ball < -0.35 else 1)


def imlo_xato(matn, ulush, rng):
    chiqish = []
    for s in matn.split():
        if len(s) >= 4 and rng.random() < ulush:
            i = int(rng.integers(1, len(s) - 1))
            tur = int(rng.integers(0, 3))
            if tur == 0:
                s = s[:i] + s[i + 1:]
            elif tur == 1:
                s = s[:i] + s[i] + s[i:]
            else:
                s = s[:i - 1] + s[i] + s[i - 1] + s[i + 1:]
        chiqish.append(s)
    return " ".join(chiqish)


def korpus(n, seed=0, xato=0.07):
    rng = np.random.default_rng(seed)
    toza, y = zip(*[sharh(rng) for _ in range(n)])
    rng2 = np.random.default_rng(seed + 1000)
    return [imlo_xato(t, xato, rng2) for t in toza], np.array(y), list(toza)


def tokenlar(matn, bigram):
    s = matn.split()
    return s + ([f"{a}_{b}" for a, b in zip(s, s[1:])] if bigram else [])


def lugat_qur(matnlar, bigram, min_soni=2):
    c = Counter(t for m in matnlar for t in tokenlar(m, bigram))
    sozlar = sorted(t for t, k in c.items() if k >= min_soni)
    return {"<pad>": 0, "<unk>": 1} | {t: i + 2 for i, t in enumerate(sozlar)}


def paket(ketmalar):
    offsets = torch.tensor([0] + [len(k) for k in ketmalar[:-1]]).cumsum(0)
    return torch.tensor([t for k in ketmalar for t in k]), offsets


class BagModel(nn.Module):
    def __init__(self, V, d=32):
        super().__init__()
        self.bag = nn.EmbeddingBag(V, d, mode="mean")
        self.chiq = nn.Linear(d, 3)

    def forward(self, ids, offsets):
        return self.chiq(self.bag(ids, offsets))


def orgat(ketmalar, y, V, seed, davrlar=15):
    torch.manual_seed(seed)
    model = BagModel(V)
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    g = torch.Generator().manual_seed(seed)
    yt = torch.tensor(y)
    for _ in range(davrlar):
        tartib = torch.randperm(len(ketmalar), generator=g).tolist()
        for i in range(0, len(tartib), 64):
            b = tartib[i:i + 64]
            opt.zero_grad()
            nn.functional.cross_entropy(
                model(*paket([ketmalar[j] for j in b])), yt[b]).backward()
            opt.step()
    model.eval()
    return model


def bashorat(model, ketmalar):
    with torch.no_grad():
        return model(*paket(ketmalar)).argmax(1).numpy()


def main() -> None:
    matnlar, y, _ = korpus(3000)
    Xtr, Xte, ytr, yte = matnlar[:2400], matnlar[2400:], y[:2400], y[2400:]
    TP = r"[\w']+"
    SINF = ["salbiy", "neytral", "ijobiy"]

    print("=== 1. TF-IDF bazaviylari 23.5-bob ===")
    tf = {}
    for nom, ng in [("TF-IDF unigram", (1, 1)), ("TF-IDF + bigram", (1, 2))]:
        m = make_pipeline(TfidfVectorizer(token_pattern=TP, ngram_range=ng),
                          LogisticRegression(C=10, max_iter=3000))
        tf[nom] = m.fit(Xtr, ytr).score(Xte, yte)
        print(f"  {nom:<16} {tf[nom]:.4f}")

    print("\n=== 2. EmbeddingBag (d=32, 15 davr, 3 seed) ===")
    modellar, natija = {}, {}
    for nom, bigram in [("Emb unigram", False), ("Emb + bigram", True)]:
        lugat = lugat_qur(Xtr, bigram)
        k_tr = [[lugat.get(t, 1) for t in tokenlar(m, bigram)] for m in Xtr]
        k_te = [[lugat.get(t, 1) for t in tokenlar(m, bigram)] for m in Xte]
        aniq = []
        for s in range(3):
            model = orgat(k_tr, ytr, len(lugat), seed=s)
            aniq.append((bashorat(model, k_te) == yte).mean())
        modellar[nom] = (model, lugat, bigram)
        natija[nom] = np.array(aniq)
        print(f"  {nom:<14} lug'at {len(lugat):>5}  aniqlik "
              f"{np.mean(aniq):.4f} ± {np.std(aniq, ddof=1):.4f}")

    print("\n=== 3. Juftlashgan farq: bigram - unigram (3 seed) ===")
    d = natija["Emb + bigram"] - natija["Emb unigram"]
    se = d.std(ddof=1) / np.sqrt(len(d))
    print(f"  {d.mean():+.4f}, SE {se:.4f}, "
          f"{'sezilarli' if abs(d.mean()) > 2 * se else 'sezilarli emas'}")
    for nom, t in [("unigram", "TF-IDF unigram"),
                   ("bigram", "TF-IDF + bigram")]:
        e = natija[f"Emb {nom}" if nom == "unigram" else "Emb + bigram"]
        print(f"  Emb {nom:<8} - {t:<16} {e.mean() - tf[t]:+.4f}")

    print("\n=== 4. Tartib va inkor ===")
    sinov = ["telefon oldim sifati yaxshi",
             "telefon oldim sifati yaxshi emas",
             "telefon oldim emas sifati yaxshi",
             "telefon oldim sifati yomon emas"]
    print(f"  {'matn':<34} {'unigram':>9} {'bigram':>9}")
    for m in sinov:
        chiq = []
        for nom in ["Emb unigram", "Emb + bigram"]:
            model, lugat, bigram = modellar[nom]
            k = [[lugat.get(t, 1) for t in tokenlar(m, bigram)]]
            chiq.append(SINF[bashorat(model, k)[0]])
        print(f"  {m:<34} {chiq[0]:>9} {chiq[1]:>9}")
    print("  unigram modeli uchun 2- va 3-matn AYNAN bir xil vektor")
    print("  ⭐ O'rtacha pooling tartibni ko'rmaydi - bigram token yordam beradi")


if __name__ == "__main__":
    main()

=== 1. TF-IDF bazaviylari 23.5-bob ===
  TF-IDF unigram   0.7250
  TF-IDF + bigram  0.8083

=== 2. EmbeddingBag (d=32, 15 davr, 3 seed) ===
  Emb unigram    lug'at   571  aniqlik 0.7189 ± 0.0025
  Emb + bigram   lug'at  1944  aniqlik 0.8044 ± 0.0035

=== 3. Juftlashgan farq: bigram - unigram (3 seed) ===
  +0.0856, SE 0.0024, sezilarli
  Emb unigram  - TF-IDF unigram   -0.0061
  Emb bigram   - TF-IDF + bigram  -0.0039

=== 4. Tartib va inkor ===
  matn                                 unigram    bigram
  telefon oldim sifati yaxshi           ijobiy    ijobiy
  telefon oldim sifati yaxshi emas      ijobiy    salbiy
  telefon oldim emas sifati yaxshi      ijobiy   neytral
  telefon oldim sifati yomon emas       salbiy    ijobiy
  unigram modeli uchun 2- va 3-matn AYNAN bir xil vektor
  ⭐ O'rtacha pooling tartibni ko'rmaydi - bigram token yordam beradi

"""Belgilanmagan sharhlarda SGNS, keyin kam belgilangan ma'lumotda klassifikator."""

from collections import Counter

import numpy as np
import torch
import torch.nn as nn

IJOBIY = ["yaxshi", "zo'r", "a'lo", "ajoyib", "sifatli", "qulay",
          "chiroyli", "mustahkam"]
SALBIY = ["yomon", "sifatsiz", "noqulay", "xunuk", "nosoz", "mo'rt",
          "sust", "chatoq"]
NEYTRAL = ["o'rtacha", "oddiy", "normal", "odatiy", "kutilgandek"]
JIHAT = ["sifati", "narxi", "ekrani", "batareyasi", "ovozi", "qadoqlash",
         "yetkazib berish", "sotuvchi", "dizayni", "materiali"]
MAHSULOT = ["telefon", "noutbuk", "quloqchin", "soat", "choynak", "kurtka",
            "krossovka", "sumka", "planshet", "kolonka"]
BOSH = ["kecha", "o'tgan hafta", "bir oy oldin", "do'kondan", "internetdan",
        "sovg'a uchun", "akam uchun", ""]
FEL = ["oldim", "buyurtma qildim", "sotib oldim", "ishlatib ko'rdim"]
YAKUN = {1: ["tavsiya qilaman", "yana olaman", "hammaga maslahat beraman"],
         -1: ["tavsiya qilmayman", "boshqa olmayman", "pulimga achinaman"],
         0: ["hozircha shu", "vaqt ko'rsatadi", "yana ishlatib ko'ramiz"]}
ULUSH = {1: [0.8, 0.1, 0.1], -1: [0.1, 0.8, 0.1], 0: [0.15, 0.15, 0.7]}


def gap(qutb, rng):
    j = rng.choice(JIHAT)
    juda = "juda " if rng.random() < 0.3 else ""
    r = rng.random()
    if qutb == 1:
        if r < 0.2:
            return f"{j} {rng.choice(SALBIY)} emas"
        if r < 0.3:
            return f"{j} bilan muammo yo'q"
        return f"{j} {juda}{rng.choice(IJOBIY)}"
    if qutb == -1:
        if r < 0.25:
            return f"{j} {rng.choice(IJOBIY)} emas"
        if r < 0.35:
            return f"{j} bilan muammo bor"
        return f"{j} {juda}{rng.choice(SALBIY)}"
    return f"{j} {rng.choice(NEYTRAL)}"


def sharh(rng):
    kayfiyat = int(rng.integers(-1, 2))
    qismlar = [f"{rng.choice(BOSH)} {rng.choice(MAHSULOT)} "
               f"{rng.choice(FEL)}".strip()]
    ballar = []
    for t in range(int(rng.integers(1, 5))):
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        bog = rng.choice(["lekin", "va", "ammo", "ham", ""]) if t else ""
        qismlar.append(f"{bog} {gap(q, rng)}".strip())
        ballar.append(q)
    if rng.random() < 0.5:
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        qismlar.append(rng.choice(YAKUN[q]))
        ballar.append(q)
    ball = np.mean(ballar) + rng.normal(0, 0.1)
    return " ".join(qismlar), 2 if ball > 0.35 else (0 if ball < -0.35 else 1)


def imlo_xato(matn, ulush, rng):
    chiqish = []
    for s in matn.split():
        if len(s) >= 4 and rng.random() < ulush:
            i = int(rng.integers(1, len(s) - 1))
            tur = int(rng.integers(0, 3))
            if tur == 0:
                s = s[:i] + s[i + 1:]
            elif tur == 1:
                s = s[:i] + s[i] + s[i:]
            else:
                s = s[:i - 1] + s[i] + s[i - 1] + s[i + 1:]
        chiqish.append(s)
    return " ".join(chiqish)


def korpus(n, seed=0, xato=0.07):
    rng = np.random.default_rng(seed)
    toza, y = zip(*[sharh(rng) for _ in range(n)])
    rng2 = np.random.default_rng(seed + 1000)
    return [imlo_xato(t, xato, rng2) for t in toza], np.array(y), list(toza)


def lugat_qur(matnlar, min_soni=2):
    c = Counter(s for m in matnlar for s in m.split())
    sozlar = sorted(s for s, k in c.items() if k >= min_soni)
    return {"<pad>": 0, "<unk>": 1} | {s: i + 2 for i, s in enumerate(sozlar)}


def sgns(ketmalar, V, d=32, k=5, qadamlar=400, seed=0, oyna=2):
    """23.6-dars uslubida skip-gram + negative sampling."""
    m, c = [], []
    for ids in ketmalar:
        for i, w in enumerate(ids):
            for j in range(max(0, i - oyna), min(len(ids), i + oyna + 1)):
                if j != i:
                    m.append(w)
                    c.append(ids[j])
    M, C = torch.tensor(m), torch.tensor(c)
    soni = np.bincount(m, minlength=V).astype(float)
    P = torch.tensor(soni ** 0.75 / (soni ** 0.75).sum(), dtype=torch.float)
    torch.manual_seed(seed)
    markaz, kontekst = nn.Embedding(V, d), nn.Embedding(V, d)
    nn.init.uniform_(markaz.weight, -0.5 / d, 0.5 / d)
    nn.init.zeros_(kontekst.weight)
    opt = torch.optim.Adam([markaz.weight, kontekst.weight], lr=0.01)
    g = torch.Generator().manual_seed(seed)
    qadam = 0
    while qadam < qadamlar:
        tartib = torch.randperm(len(M), generator=g)
        for i in range(0, len(M), 2048):
            b = tartib[i:i + 2048]
            neg = torch.multinomial(P, len(b) * k, replacement=True,
                                    generator=g).view(len(b), k)
            vm = markaz(M[b])
            ijobiy = (vm * kontekst(C[b])).sum(1)
            salbiy = torch.bmm(kontekst(neg), vm.unsqueeze(2)).squeeze(2)
            loss = -(nn.functional.logsigmoid(ijobiy).mean()
                     + nn.functional.logsigmoid(-salbiy).sum(1).mean())
            opt.zero_grad()
            loss.backward()
            opt.step()
            qadam += 1
            if qadam >= qadamlar:
                break
    return markaz.weight.detach().clone()


def paket(ketmalar):
    offsets = torch.tensor([0] + [len(k) for k in ketmalar[:-1]]).cumsum(0)
    return torch.tensor([t for k in ketmalar for t in k]), offsets


class BagModel(nn.Module):
    def __init__(self, V, d, E=None, muzlat=False):
        super().__init__()
        self.bag = nn.EmbeddingBag(V, d, mode="mean")
        if E is not None:
            self.bag.weight.data.copy_(E)
        self.bag.weight.requires_grad_(not muzlat)
        self.chiq = nn.Linear(d, 3)

    def forward(self, ids, offsets):
        return self.chiq(self.bag(ids, offsets))


def orgat_bahola(k_tr, y_tr, k_te, y_te, V, E, muzlat, seed, qadamlar=200):
    """Hamma variant va hamma n uchun BIR XIL qadamlar soni."""
    torch.manual_seed(seed)
    model = BagModel(V, 32, E, muzlat)
    opt = torch.optim.Adam([p for p in model.parameters() if p.requires_grad],
                           lr=0.01)
    g = torch.Generator().manual_seed(seed)
    yt = torch.tensor(y_tr)
    qadam = 0
    while qadam < qadamlar:
        tartib = torch.randperm(len(k_tr), generator=g).tolist()
        for i in range(0, len(tartib), 64):
            b = tartib[i:i + 64]
            opt.zero_grad()
            nn.functional.cross_entropy(
                model(*paket([k_tr[j] for j in b])), yt[b]).backward()
            opt.step()
            qadam += 1
            if qadam >= qadamlar:
                break
    with torch.no_grad():
        return (model(*paket(k_te)).argmax(1).numpy() == y_te).mean()


def main() -> None:
    belgisiz, _, _ = korpus(8000, seed=7)        # yorliqsiz katta korpus
    matnlar, y, _ = korpus(3000)
    lugat = lugat_qur(belgisiz)                  # oldindan o'rgatish lug'ati
    V = len(lugat)
    kod = lambda ms: [[lugat.get(s, 1) for s in m.split()] for m in ms]

    print("=== 1. Oldindan o'rgatish (SGNS, 8000 belgisiz sharh) ===")
    E = sgns(kod(belgisiz), V)
    print(f"  lug'at {V}, embedding {tuple(E.shape)}")
    En = E / E.norm(dim=1, keepdim=True)
    ortacha = lambda A, B: np.mean([float(En[lugat[a]] @ En[lugat[b]])
                                    for a in A for b in B if a != b])
    ij, sa = ["yaxshi", "zo'r", "a'lo", "ajoyib"], ["yomon", "sifatsiz",
                                                    "nosoz", "chatoq"]
    print(f"  cos ijobiy-ijobiy: {ortacha(ij, ij):.3f}")
    print(f"  cos salbiy-salbiy: {ortacha(sa, sa):.3f}")
    print(f"  cos ijobiy-salbiy: {ortacha(ij, sa):.3f}")
    print(f"  cos ijobiy-mahsulot: "
          f"{ortacha(ij, ['telefon', 'soat', 'sumka', 'kurtka']):.3f}")
    print("  word2vec 'yaxshi' va 'yomon' ni deyarli bir joyga qo'ydi -")
    print("  ikkalasi ham 'sifati ___' kontekstida keladi")

    print("\n=== 2. Kam belgilangan ma'lumot (test 1000, 3 seed, 200 qadam) ===")
    k_te, y_te = kod(matnlar[2000:]), y[2000:]
    variantlar = [("tasodifiy", None, False), ("muzlatilgan", E, True),
                  ("fine-tune", E, False)]
    natija = {}
    print(f"  {'n':>5} " + " ".join(f"{v[0]:>12}" for v in variantlar))
    for n in [100, 300, 1000]:
        natija[n] = {v[0]: [] for v in variantlar}
        for s in range(3):
            idx = np.random.default_rng(s).choice(2000, n, replace=False)
            k_tr = kod([matnlar[i] for i in idx])
            for nom, emb, muz in variantlar:
                natija[n][nom].append(orgat_bahola(k_tr, y[idx], k_te, y_te,
                                                   V, emb, muz, seed=s))
        print(f"  {n:>5} " + " ".join(f"{np.mean(natija[n][v[0]]):>12.4f}"
                                      for v in variantlar))

    print("\n=== 3. Juftlashgan farq tasodifiy init ga nisbatan ===")
    for n in natija:
        for nom in ["muzlatilgan", "fine-tune"]:
            d = np.array(natija[n][nom]) - np.array(natija[n]["tasodifiy"])
            se = d.std(ddof=1) / np.sqrt(len(d))
            if abs(d.mean()) <= 2 * se:
                x = "sezilarli emas"
            else:
                x = "sezilarli yaxshi" if d.mean() > 0 else "sezilarli yomon"
            print(f"  n={n:>4} {nom:<12} {d.mean():+.4f}  SE {se:.4f}  {x}")
    print("  ⭐ Oldindan o'rgatilgan embedding - faraz; uni o'lchab tekshiring")


if __name__ == "__main__":
    main()

=== 1. Oldindan o'rgatish (SGNS, 8000 belgisiz sharh) ===
  lug'at 993, embedding (993, 32)
  cos ijobiy-ijobiy: 0.982
  cos salbiy-salbiy: 0.983
  cos ijobiy-salbiy: 0.955
  cos ijobiy-mahsulot: 0.272
  word2vec 'yaxshi' va 'yomon' ni deyarli bir joyga qo'ydi -
  ikkalasi ham 'sifati ___' kontekstida keladi

=== 2. Kam belgilangan ma'lumot (test 1000, 3 seed, 200 qadam) ===
      n    tasodifiy  muzlatilgan    fine-tune
    100       0.5710       0.5483       0.5747
    300       0.6500       0.6190       0.6533
   1000       0.7077       0.6313       0.7120

=== 3. Juftlashgan farq tasodifiy init ga nisbatan ===
  n= 100 muzlatilgan  -0.0227  SE 0.0042  sezilarli yomon
  n= 100 fine-tune    +0.0037  SE 0.0034  sezilarli emas
  n= 300 muzlatilgan  -0.0310  SE 0.0105  sezilarli yomon
  n= 300 fine-tune    +0.0033  SE 0.0043  sezilarli emas
  n=1000 muzlatilgan  -0.0763  SE 0.0045  sezilarli yomon
  n=1000 fine-tune    +0.0043  SE 0.0043  sezilarli emas
  ⭐ Oldindan o'rgatilgan embedding - faraz; uni o'lchab tekshiring

"""Juftlashgan CV: TF-IDF + LogReg va EmbeddingBag, to'liq va kam ma'lumot."""

from collections import Counter

import numpy as np
import torch
import torch.nn as nn
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold
from sklearn.pipeline import make_pipeline

IJOBIY = ["yaxshi", "zo'r", "a'lo", "ajoyib", "sifatli", "qulay",
          "chiroyli", "mustahkam"]
SALBIY = ["yomon", "sifatsiz", "noqulay", "xunuk", "nosoz", "mo'rt",
          "sust", "chatoq"]
NEYTRAL = ["o'rtacha", "oddiy", "normal", "odatiy", "kutilgandek"]
JIHAT = ["sifati", "narxi", "ekrani", "batareyasi", "ovozi", "qadoqlash",
         "yetkazib berish", "sotuvchi", "dizayni", "materiali"]
MAHSULOT = ["telefon", "noutbuk", "quloqchin", "soat", "choynak", "kurtka",
            "krossovka", "sumka", "planshet", "kolonka"]
BOSH = ["kecha", "o'tgan hafta", "bir oy oldin", "do'kondan", "internetdan",
        "sovg'a uchun", "akam uchun", ""]
FEL = ["oldim", "buyurtma qildim", "sotib oldim", "ishlatib ko'rdim"]
YAKUN = {1: ["tavsiya qilaman", "yana olaman", "hammaga maslahat beraman"],
         -1: ["tavsiya qilmayman", "boshqa olmayman", "pulimga achinaman"],
         0: ["hozircha shu", "vaqt ko'rsatadi", "yana ishlatib ko'ramiz"]}
ULUSH = {1: [0.8, 0.1, 0.1], -1: [0.1, 0.8, 0.1], 0: [0.15, 0.15, 0.7]}


def gap(qutb, rng):
    j = rng.choice(JIHAT)
    juda = "juda " if rng.random() < 0.3 else ""
    r = rng.random()
    if qutb == 1:
        if r < 0.2:
            return f"{j} {rng.choice(SALBIY)} emas"
        if r < 0.3:
            return f"{j} bilan muammo yo'q"
        return f"{j} {juda}{rng.choice(IJOBIY)}"
    if qutb == -1:
        if r < 0.25:
            return f"{j} {rng.choice(IJOBIY)} emas"
        if r < 0.35:
            return f"{j} bilan muammo bor"
        return f"{j} {juda}{rng.choice(SALBIY)}"
    return f"{j} {rng.choice(NEYTRAL)}"


def sharh(rng):
    kayfiyat = int(rng.integers(-1, 2))
    qismlar = [f"{rng.choice(BOSH)} {rng.choice(MAHSULOT)} "
               f"{rng.choice(FEL)}".strip()]
    ballar = []
    for t in range(int(rng.integers(1, 5))):
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        bog = rng.choice(["lekin", "va", "ammo", "ham", ""]) if t else ""
        qismlar.append(f"{bog} {gap(q, rng)}".strip())
        ballar.append(q)
    if rng.random() < 0.5:
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        qismlar.append(rng.choice(YAKUN[q]))
        ballar.append(q)
    ball = np.mean(ballar) + rng.normal(0, 0.1)
    return " ".join(qismlar), 2 if ball > 0.35 else (0 if ball < -0.35 else 1)


def imlo_xato(matn, ulush, rng):
    chiqish = []
    for s in matn.split():
        if len(s) >= 4 and rng.random() < ulush:
            i = int(rng.integers(1, len(s) - 1))
            tur = int(rng.integers(0, 3))
            if tur == 0:
                s = s[:i] + s[i + 1:]
            elif tur == 1:
                s = s[:i] + s[i] + s[i:]
            else:
                s = s[:i - 1] + s[i] + s[i - 1] + s[i + 1:]
        chiqish.append(s)
    return " ".join(chiqish)


def korpus(n, seed=0, xato=0.07):
    rng = np.random.default_rng(seed)
    toza, y = zip(*[sharh(rng) for _ in range(n)])
    rng2 = np.random.default_rng(seed + 1000)
    return [imlo_xato(t, xato, rng2) for t in toza], np.array(y), list(toza)


def tokenlar(matn):
    s = matn.split()
    return s + [f"{a}_{b}" for a, b in zip(s, s[1:])]


def paket(ketmalar):
    offsets = torch.tensor([0] + [len(k) for k in ketmalar[:-1]]).cumsum(0)
    return torch.tensor([t for k in ketmalar for t in k]), offsets


def emb_bahola(Xtr, ytr, Xte, yte, seed, davrlar=15):
    """EmbeddingBag + bigram tokenlar (2-misoldagi eng yaxshi variant)."""
    c = Counter(t for m in Xtr for t in tokenlar(m))
    sozlar = sorted(t for t, k in c.items() if k >= 2)
    lugat = {t: i + 2 for i, t in enumerate(sozlar)}
    kod = lambda ms: [[lugat.get(t, 1) for t in tokenlar(m)] for m in ms]
    k_tr, k_te = kod(Xtr), kod(Xte)
    torch.manual_seed(seed)
    bag = nn.EmbeddingBag(len(lugat) + 2, 32, mode="mean")
    chiq = nn.Linear(32, 3)
    opt = torch.optim.Adam(list(bag.parameters()) + list(chiq.parameters()),
                           lr=0.01)
    g = torch.Generator().manual_seed(seed)
    yt = torch.tensor(ytr)
    for _ in range(davrlar):
        tartib = torch.randperm(len(k_tr), generator=g).tolist()
        for i in range(0, len(tartib), 64):
            b = tartib[i:i + 64]
            opt.zero_grad()
            nn.functional.cross_entropy(
                chiq(bag(*paket([k_tr[j] for j in b]))), yt[b]).backward()
            opt.step()
    with torch.no_grad():
        return (chiq(bag(*paket(k_te))).argmax(1).numpy() == yte).mean()


def tfidf_bahola(Xtr, ytr, Xte, yte):
    m = make_pipeline(TfidfVectorizer(token_pattern=r"[\w']+",
                                      ngram_range=(1, 2)),
                      LogisticRegression(C=10, max_iter=3000))
    return m.fit(Xtr, ytr).score(Xte, yte)


def xulosa(a, b, nom_a, nom_b):
    d = b - a
    se = d.std(ddof=1) / np.sqrt(len(d))
    print(f"  o'rtacha: {nom_a} {a.mean():.4f}, {nom_b} {b.mean():.4f}")
    print(f"  farq ({nom_b} - {nom_a}): {d.mean():+.4f}, SE {se:.4f}")
    if abs(d.mean()) <= 2 * se:
        q = f"farq 2*SE ichida - soddaroq model ({nom_a}) tanlanadi"
    elif d.mean() > 0:
        q = f"{nom_b} sezilarli yaxshi"
    else:
        q = f"{nom_a} sezilarli yaxshi - bazaviy yutdi"
    print(f"  QAROR: {q}")


def main() -> None:
    matnlar, y, _ = korpus(3000)
    cv = StratifiedKFold(5, shuffle=True, random_state=0)
    toliq = {"TF-IDF": [], "Emb": []}
    kam = {"TF-IDF": [], "Emb": []}
    for k, (tr, te) in enumerate(cv.split(matnlar, y)):
        Xtr = [matnlar[i] for i in tr]
        Xte = [matnlar[i] for i in te]
        toliq["TF-IDF"].append(tfidf_bahola(Xtr, y[tr], Xte, y[te]))
        toliq["Emb"].append(emb_bahola(Xtr, y[tr], Xte, y[te], seed=k))
        kichik = np.random.default_rng(k).choice(len(tr), 300, replace=False)
        Xk = [Xtr[i] for i in kichik]
        kam["TF-IDF"].append(tfidf_bahola(Xk, y[tr][kichik], Xte, y[te]))
        kam["Emb"].append(emb_bahola(Xk, y[tr][kichik], Xte, y[te], seed=k))

    for sarlavha, nat in [("To'liq o'quv (2400 sharh)", toliq),
                          ("Kam o'quv (300 sharh)", kam)]:
        print(f"=== {sarlavha}, 5 fold ===")
        print(f"  {'fold':>5} {'TF-IDF':>8} {'Emb':>8}")
        for i in range(5):
            print(f"  {i + 1:>5} {nat['TF-IDF'][i]:>8.4f} {nat['Emb'][i]:>8.4f}")
        xulosa(np.array(nat["TF-IDF"]), np.array(nat["Emb"]), "TF-IDF", "Emb")
        print()
    print("  ⭐ Qarorni farq va SE belgiladi - bazaviy yutsa, shuni yozamiz")


if __name__ == "__main__":
    main()

=== To'liq o'quv (2400 sharh), 5 fold ===
   fold   TF-IDF      Emb
      1   0.7950   0.7900
      2   0.8150   0.8050
      3   0.8017   0.7950
      4   0.8300   0.8350
      5   0.7950   0.7883
  o'rtacha: TF-IDF 0.8073, Emb 0.8027
  farq (Emb - TF-IDF): -0.0047, SE 0.0025
  QAROR: farq 2*SE ichida - soddaroq model (TF-IDF) tanlanadi

=== Kam o'quv (300 sharh), 5 fold ===
   fold   TF-IDF      Emb
      1   0.7117   0.6900
      2   0.7233   0.7067
      3   0.6767   0.6450
      4   0.7117   0.6883
      5   0.6850   0.6533
  o'rtacha: TF-IDF 0.7017, Emb 0.6767
  farq (Emb - TF-IDF): -0.0250, SE 0.0029
  QAROR: TF-IDF sezilarli yaxshi - bazaviy yutdi

  ⭐ Qarorni farq va SE belgiladi - bazaviy yutsa, shuni yozamiz

h = emb(ids).mean(1)                                              # ⚠️
h = (emb(ids) * mask.unsqueeze(-1)).sum(1) / mask.sum(1, keepdim=True)  # ✅

bag(ids1, torch.tensor([3, 5, 2]))              # uzunliklar      # ⚠️
bag(ids1, torch.tensor([0, 3, 8]))              # boshlanishlar   # ✅

lugat = {s: i for i, s in enumerate(sozlar)}; lugat[s]   # KeyError  # ⚠️
lugat = {"<pad>": 0, "<unk>": 1} | {...}; lugat.get(s, 1)             # ✅

bag.weight.data.copy_(E)   # E boshqa lug'at tartibida qurilgan         # ⚠️
# klassifikator E qurilgan lug'at bilan kodlaydi                         # ✅

bag.weight.data.copy_(E)                        # hammasi o'rganadi    # ⚠️
bag.weight.data.copy_(E); bag.weight.requires_grad_(False)             # ✅

torch.optim.Adam(model.parameters(), weight_decay=0.01)   # E ham siqiladi  # ⚠️
torch.optim.Adam([p for p in model.parameters() if p.requires_grad])        # ✅

print(f"Embedding modeli: {aniq:.3f}")                            # ⚠️
# TF-IDF + LogReg bilan bir xil foldlarda, farq + SE             # ✅

1.  # <pad> va <unk> indekslari?
2.  # e.mean(1) nima uchun noto'g'ri?
3.  # maskali o'rtacha formulasi?
4.  # padding_idx=0 nima qiladi?
5.  # sharhlar [3, 5, 2] so'zli bo'lsa offsets?
6.  # EmbeddingBag(mean) va maskali o'rtacha farqi?
7.  # "yaxshi emas" va "emas yaxshi" o'rtacha pooling da?
8.  # bigram token misoli?
9.  # muzlatish qanday qilinadi?
10. # nega word2vec "yaxshi" va "yomon" ni yaqin qo'yadi?
11. # V = 1000, d = 32 da embedding parametrlari?
12. # qaror qoidasi?

1.  h = emb(ids).mean(dim=1)

2.  offsets = torch.tensor([len(k) for k in ketmalar])

3.  lugat = {s: i for i, s in enumerate(sorted(set(sozlar)))}
    ids = [lugat[s] for s in test_matn.split()]

4.  model.bag.weight.data.copy_(E)
    opt = torch.optim.Adam(model.parameters())     # muzlatish kerak edi

5.  print("Embedding modeli yaxshi:", aniq_emb)