24.12-dars: Amaliyot — sharhlar tonalligini transformer bilan aniqlash loyihasi

Konfig (frozen dataclass): T, d, L, h, pre_qadam, qadam, lr, bs, C
  |
  v
Ma'lumot: matn, tonallik, tur, muallif          (sintetik, seed bilan)
  |
  v
Muallif bo'yicha bo'lish (GroupShuffleSplit) -> TEST QULFLANADI      23.13-bob
  |
  v
Belgisiz korpus -> tokenizator (Lugat) -> MLM pretraining           (24.8, 24.10)
  |
  v
Tez tekshiruvlar: shakllar, ln 2, maskalash ulushi, bitta batch     21.9-bob
  |
  v
Nomzodlar: ko'pchilik < so'z TF-IDF < belgi TF-IDF < transformer < pretrain+FT
  |
  v
GroupKFold(5) juftlashgan farq -> eng sodda munosib model           (18-qism)
  |
  v
Test BIR MARTA -> tur bo'yicha aniqlik -> xato tahlili
  |
  v
Paket (weights_only) -> Bashoratchi -> hisobot

MA'LUMOT (1-misol):
  1420 sharh, 350 muallif, ijobiy 0.494
  oddiy 49.8%, inkor 15.6%, uzoq inkor 15.1%, kontrast 19.5%
  92 ta qayta yuborilgan sharh, ~3% yorliq shovqini

TURLAR:
  oddiy       "kafe qiziqarli chiqdi"                      sifat hal qiladi
  inkor       "noutbuk biroz tez emas"                     sifat + 'emas' -> teskari
  uzoq inkor  "kurs ovozi noqulay deb hisoblayman"         fe'l hal qiladi:
              "... deb o'ylayman" / "... deb o'ylamayman"   tasdiq yoki inkor
  kontrast    "do'kon narxi yomon ekan biroq dizayni qulay" "lekin/ammo/biroq" dan
                                                            keyingi sifat hal qiladi

NEGA SO'Z XALTASI QIYNALADI (1-misol, 2-bo'lim):
  "kafe juda a'lo edi"                 | "kafe juda a'lo emas edi"          4/5 umumiy
  "kafe narxi a'lo deb o'ylayman"      | "... deb o'ylamayman"              4/6 umumiy
  "kafe narxi yomon lekin sifati a'lo" | "kafe narxi a'lo lekin sifati yomon" 6/6 umumiy!
  kontrastda so'zlar TO'PLAMI bir xil - farq faqat TARTIBDA

BO'LISH: GroupShuffleSplit(groups=muallif): ish 1137 / test 283, umumiy muallif 0
TOKENIZATOR: lug'at 131 so'z, FAQAT belgisiz korpusdan; ishda [UNK] 0.0000

NOMZODLAR (soddalik tartibida):
  1) ko'pchilik      doim ko'p uchragan sinf                       (pastki chegara)
  2) so'z TF-IDF     so'z 1-2 gram + LogisticRegression(C=10)
  3) belgi TF-IDF    char_wb 2-5 gram + LogisticRegression(C=10)
  4) transformer     Encoder(d=48, L=2, h=4) + o'rtacha pooling + bosh, NOLDAN
  5) pretrain+FT     xuddi shu, encoder MLM bilan oldindan o'rgatilgan

PRETRAINING: 20000 belgisiz sharh, MLM 15% (80/10/10), 800 qadam, batch 32
FINE-TUNING / NOLDAN: BIR XIL sikl - 150 qadam, AdamW lr 2e-3, batch 32
  bir xil urug' -> bir xil bosh init va bir xil batchlar (juftlashgan)

BOSH INIT: nn.init.normal_(std=0.02), bias = 0  -> boshlang'ich loss ~ ln 2
  standart Linear init bilan 1-misolning tekshiruvi XATO bergan edi (loss 0.989)

  qadam   transformer   pretrain+FT    farq
     50      0.655         0.714      +0.059
    100      0.787         0.843      +0.056
    150      0.875         0.969      +0.094   <- konfigdagi byudjet
    300      0.937         0.955      +0.017

BIR XIL 5 TA FOLD: GroupKFold(5), groups = muallif (ish to'plamida, test TEGILMAYDI)
  har foldda: TF-IDF - fold o'quvidan; transformer - fold o'quvida 150 qadam
  pretrained encoder - bir marta (belgisiz korpus), har foldga copy.deepcopy

JUFTLASHGAN FARQ (eng yaxshisi - nomzod):
  d_f = aniq_eng,f - aniq_nomzod,f;  SE = std(d) / sqrt(5);  d > 2 * SE -> sezilarli yomon

QOIDA: eng yaxshisidan SEZILARLI yomon bo'lmagan ENG SODDA model

  model          aniqlik    oddiy   inkor  uzoq inkor  kontrast
  ko'pchilik      0.5119    0.524   0.487    0.502      0.506
  so'z TF-IDF     0.7520    0.883   0.699    0.621      0.566
  belgi TF-IDF    0.6623    0.851   0.303    0.552      0.547
  transformer     0.8504    0.916   0.683    0.727      0.899
  pretrain+FT     0.9490    0.967   0.943    0.894      0.948

TEST BIR MARTA (tanlangan model butun ish to'plamida: pretraining + 150 qadam):
  aniqlik + sinf bo'yicha recall / precision
  TUR BO'YICHA xato va uning 2*SE si (qolgan turlar bilan solishtirish)

4-MISOL:
  test aniqligi 0.9364 (18 xato / 283);  CV: 0.9490
  salbiy recall 0.898, ijobiy recall 0.973
  tur          n    xato   2*SE   qolganlar xatosi
  oddiy       137   0.007  0.055       0.116
  inkor        47   0.191  0.117       0.038
  uzoq inkor   41   0.195  0.126       0.041
  kontrast     58   0.000  0.036       0.080
  sezilarli xavfli turlar: inkor, uzoq inkor

paket = {
  "format": 1,
  "konfig": {"model": asdict(k), "tanlov": "pretrain+FT", "pooling": "o'rtacha, PAD siz"},
  "lugat": ["[PAD]", "[CLS]", "[MASK]", "[UNK]", ...],   # 131 ta, TARTIB muhim
  "holat": model.state_dict(),                          # 30 tenzor, 64178 parametr
  "sinflar": ["salbiy", "ijobiy"],
  "metrika": {"test_aniqlik": ..., "tur_aniqligi": {...}},
  "versiyalar": {"torch": str(torch.__version__), "sklearn": ..., "python": ...},
  "nazorat": {"matn": 8 ta xom sharh, "logit": saqlangan logitlar},
}

BASHORATCHI:
  torch.load(yol, weights_only=True) -> Konfig(**konfig) -> Encoder + Klassifikator
  -> load_state_dict -> nazorat: qayta hisoblangan logitlar == saqlangan (atol 1e-5)
  kodla: kichik harf -> so'zlar -> [CLS] + indekslar (noma'lum -> [UNK])

k = Konfig(); seed_everything(k.seed)
matn, y, tur, muallif = malumot(seed=0)
ish, te = guruhli_bolish(y, muallif, k.test_ulushi, k.seed)   # test QULF, muallif bo'yicha
kor = korpus(); lug = Lugat(kor)                              # tokenizator - belgisiz korpusdan
enc_pre, _ = pretrain(k, len(lug.itos), lug.kodla(kor, k.T))  # foldlardan tashqarida, bir marta
for f, (tr, va) in enumerate(GroupKFold(5).split(m, yy, mj), 1):
    noldan = orgat(k, yangi_encoder(k, V), X[tr], yy[tr], f)          # bir xil sikl
    pre = orgat(k, copy.deepcopy(enc_pre), X[tr], yy[tr], f)          # bir xil urug'
d = ball[eng] - ball[nomzod]; yomon = d.mean() > 2 * d.std(ddof=1) / np.sqrt(5)
tanlov = next(n for n in SODDALIK if n == eng or not yomon[n])
p = bashorat(model, Xt)                                       # test BIR MARTA
tur_aniqligi(yt, p, tt)                                       # tur bo'yicha
torch.save({"konfig": ..., "lugat": lug.itos, "holat": model.state_dict(),
            "nazorat": {...}}, yol)
Bashoratchi(yol)                                              # weights_only=True

konfig -> ma'lumot (tur ustuni!) -> muallif bo'yicha bo'lish (test qulf)
belgisiz korpus -> tokenizator + MLM pretraining (foldlardan tashqarida)
tez tekshiruvlar: ln 2 (bosh init std 0.02), maskalash 15%, bitta batch
nomzodlar: ko'pchilik < so'z TF-IDF < belgi TF-IDF < transformer < pretrain+FT
GroupKFold juftlashgan farq -> eng sodda munosib model (bu yerda: pretrain+FT)
test bir marta: tur va sinf bo'yicha; zaif joy - inkor turlari
paket: konfig + lug'at + state_dict + xom nazorat -> hisobot

"""1-qadam: konfig, ma'lumot, qiyin turlar, muallif bo'yicha bo'lish va tez tekshiruvlar."""

import math
import random
from dataclasses import asdict, dataclass

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, GroupShuffleSplit
from sklearn.pipeline import make_pipeline

MAVZU = ["telefon", "noutbuk", "kitob", "kafe", "mehmonxona", "avtobus", "dastur",
         "kurs", "film", "do'kon", "restoran", "poyabzal", "kamera", "sumka", "taksi",
         "klinika", "sartaroshxona", "televizor"]
ASPEKT = ["narxi", "sifati", "xizmati", "dizayni", "yetkazib berish", "xodimlari",
          "joylashuvi", "ovozi"]
IJOBIY = ["a'lo", "ajoyib", "zo'r", "yaxshi", "qulay", "chiroyli", "tez", "sifatli",
          "mazali", "toza", "arzon", "ishonchli", "shinam", "samimiy", "foydali",
          "qiziqarli", "mukammal", "puxta", "yoqimli", "maroqli", "chidamli", "oson",
          "zamonaviy", "beg'ubor"]
SALBIY = ["yomon", "sekin", "qimmat", "iflos", "noqulay", "xunuk", "sifatsiz", "bemaza",
          "eski", "zerikarli", "ishonchsiz", "qo'pol", "buzuq", "chalkash", "nosoz", "xira",
          "tor", "shovqinli", "befoyda", "sovuq", "mo'rt", "murakkab", "dag'al", "g'alati"]
DARAJA = ["juda", "ancha", "biroz", "rosa", "", ""]
OXIR = ["ekan", "edi", "", "chiqdi"]
BOSH = ["kecha", "o'tgan hafta", "shahardagi", "yangi", "onlayn", "bu", "", ""]
DUM = {1: ["hammaga tavsiya qilaman", "yana kelaman", "juda mamnunman", "pulimga arzidi",
           "do'stlarimga aytaman", "rahmat sizlarga"],
       0: ["hech kimga tavsiya qilmayman", "pulimni qaytarib oldim", "boshqa kelmayman",
           "vaqtim behuda ketdi", "shikoyat yozdim", "afsuslandim"]}
TURLAR = ["oddiy", "inkor", "uzoq inkor", "kontrast"]
TUR_ULUSH = [0.5, 0.15, 0.15, 0.2]
SINFLAR = ["salbiy", "ijobiy"]
PAD, CLS, MASK, UNK = 0, 1, 2, 3


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def sharh(rng, tur, dum_ehtimol, bosh=None):
    """Bitta sharh: (matn, tonallik). Qiyin turlar so'z tartibi va inkorga bog'liq."""
    q = int(rng.integers(2))

    def sif(p):
        return tanla(rng, IJOBIY if p else SALBIY)

    if tur == "oddiy":
        s, y = [tanla(rng, MAVZU), tanla(rng, DARAJA), sif(q), tanla(rng, OXIR)], q
    elif tur == "inkor":                                   # "a'lo emas" -> salbiy
        s, y = [tanla(rng, MAVZU), tanla(rng, DARAJA), sif(q), "emas", tanla(rng, OXIR)], 1 - q
    elif tur == "uzoq inkor":                              # inkor fe'lda, sifatdan uzoqda
        tasdiq = rng.random() < 0.5
        fel = tanla(rng, ["deb o'ylayman", "deb hisoblayman", "deya olaman"] if tasdiq
                    else ["deb o'ylamayman", "deb hisoblamayman", "deya olmayman"])
        s, y = [tanla(rng, MAVZU), tanla(rng, ASPEKT), sif(q), fel], q if tasdiq else 1 - q
    else:                                                  # "X yomon, lekin Y a'lo" -> Y hal qiladi
        a1, a2 = rng.choice(len(ASPEKT), 2, replace=False)
        s = [tanla(rng, MAVZU), ASPEKT[a1], sif(1 - q), tanla(rng, ["edi", "", "ekan"]),
             tanla(rng, ["lekin", "ammo", "biroq"]), ASPEKT[a2], tanla(rng, DARAJA), sif(q)]
        y = q
    s = [tanla(rng, BOSH) if bosh is None else bosh] + s
    if rng.random() < dum_ehtimol:
        s.append(tanla(rng, DUM[y]))
    return " ".join(" ".join(s).split()), y


def malumot(seed=0, n_muallif=350, p_takror=0.1, p_shovqin=0.03):
    """Belgili sharhlar: matn, tonallik, tur, muallif. Muallifning o'z uslubi bor."""
    rng = np.random.default_rng(seed)
    matn, y, tur, muallif = [], [], [], []
    for m in range(n_muallif):
        uslub = tanla(rng, BOSH)
        for _ in range(int(rng.geometric(1 / 4))):
            if matn and muallif[-1] == m and rng.random() < p_takror:   # qayta yuborish
                matn.append(matn[-1])
                y.append(y[-1])
                tur.append(tur[-1])
            else:
                t = TURLAR[int(rng.choice(4, p=TUR_ULUSH))]
                s, yorliq = sharh(rng, t, 0.15, uslub if rng.random() < 0.6 else None)
                matn.append(s)
                tur.append(t)
                y.append(1 - yorliq if rng.random() < p_shovqin else yorliq)
            muallif.append(m)
    return np.array(matn, dtype=object), np.array(y), np.array(tur), np.array(muallif)


def korpus(seed=1, n=20000):
    """Belgisiz sharhlar (pretraining uchun): yorliq ham, tur ham saqlanmaydi."""
    rng = np.random.default_rng(seed)
    return [sharh(rng, TURLAR[int(rng.choice(4, p=TUR_ULUSH))], 0.6)[0] for _ in range(n)]


@dataclass(frozen=True)
class Konfig:
    seed: int = 42
    test_ulushi: float = 0.2
    T: int = 24
    d: int = 48
    L: int = 2
    h: int = 4
    pre_qadam: int = 800
    qadam: int = 150
    lr: float = 2e-3
    bs: int = 32
    C: float = 10.0


def seed_everything(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)


def guruhli_bolish(y, guruh, ulush, seed):
    """Muallif bo'yicha: bir muallifning hamma sharhi bitta tomonda."""
    gss = GroupShuffleSplit(n_splits=1, test_size=ulush, random_state=seed)
    return next(gss.split(np.zeros(len(y)), y, guruh))


def soz_tfidf(k):
    return make_pipeline(TfidfVectorizer(token_pattern=r"[^ ]+", ngram_range=(1, 2),
                                         sublinear_tf=True),
                         LogisticRegression(C=k.C, max_iter=3000))


def belgi_tfidf(k):
    return make_pipeline(TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5),
                                         sublinear_tf=True),
                         LogisticRegression(C=k.C, max_iter=3000))


class Lugat:
    """So'z tokenizatori; FAQAT belgisiz korpusdan (pretrained model bilan birga keladi)."""

    def __init__(self, matnlar):
        sozlar = sorted({w for m in matnlar for w in m.split()})
        self.itos = ["[PAD]", "[CLS]", "[MASK]", "[UNK]"] + sozlar
        self.stoi = {w: i for i, w in enumerate(self.itos)}

    def kodla(self, matnlar, T):
        X = torch.zeros(len(matnlar), T, dtype=torch.long)
        for i, m in enumerate(matnlar):
            ids = [CLS] + [self.stoi.get(w, UNK) for w in m.split()][:T - 1]
            X[i, :len(ids)] = torch.tensor(ids)
        return X


class Blok(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x, maska):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).view(B, T, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=maska[:, None, None, :])
        x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class Encoder(nn.Module):
    def __init__(self, V, d, L, h, T):
        super().__init__()
        self.emb, self.poz = nn.Embedding(V, d), nn.Embedding(T, d)
        self.bloklar = nn.ModuleList([Blok(d, h) for _ in range(L)])
        self.ln = nn.LayerNorm(d)
        nn.init.normal_(self.emb.weight, std=0.02)
        nn.init.normal_(self.poz.weight, std=0.02)

    def forward(self, X):
        maska = X != PAD
        x = self.emb(X) + self.poz(torch.arange(X.shape[1]))
        for b in self.bloklar:
            x = b(x, maska)
        return self.ln(x)


class Klassifikator(nn.Module):
    """Encoder + o'rtacha pooling (PAD siz) + chiziqli bosh."""

    def __init__(self, enc):
        super().__init__()
        self.enc = enc
        self.bosh = nn.Linear(enc.emb.embedding_dim, len(SINFLAR))
        nn.init.normal_(self.bosh.weight, std=0.02)       # boshlang'ich loss ~ ln 2
        nn.init.zeros_(self.bosh.bias)

    def forward(self, X):
        h = self.enc(X)
        m = (X != PAD).float()[..., None]
        return self.bosh((h * m).sum(1) / m.sum(1))


def yangi_encoder(k, V):
    return Encoder(V, k.d, k.L, k.h, k.T)


def kes(X):
    return X[:, :int((X != PAD).sum(1).max())]


def maskala(X, g, V, p=0.15):
    Y = X.clone()
    tanl = (torch.rand(X.shape, generator=g) < p) & (X > UNK)
    Y[~tanl] = -100
    r = torch.rand(X.shape, generator=g)
    X = X.clone()
    X[tanl & (r < 0.8)] = MASK
    tas = tanl & (r >= 0.8) & (r < 0.9)
    X[tas] = torch.randint(4, V, (int(tas.sum()),), generator=g)
    return X, Y


def pretrain(k, V, X, seed=0):
    """MLM pretraining (24.8, 24.10): bog'langan vaznlar, 15% maskalash."""
    torch.manual_seed(seed)
    enc = yangi_encoder(k, V)
    bias = nn.Parameter(torch.zeros(V))
    opt = torch.optim.AdamW(list(enc.parameters()) + [bias], lr=k.lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(k.pre_qadam):
        xb, yb = maskala(kes(X[torch.randint(0, len(X), (k.bs,), generator=g)]), g, V)
        loss = F.cross_entropy((enc(xb) @ enc.emb.weight.T + bias).reshape(-1, V), yb.reshape(-1))
        opt.zero_grad()
        loss.backward()
        opt.step()
    return enc, loss.item()


def orgat(k, enc, X, y, seed):
    """Fine-tuning yoki noldan o'rgatish: bir xil sikl, bir xil qadamlar."""
    torch.manual_seed(seed)
    model = Klassifikator(enc)
    opt = torch.optim.AdamW(model.parameters(), lr=k.lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    y = torch.as_tensor(y)
    for _ in range(k.qadam):
        idx = torch.randint(0, len(X), (k.bs,), generator=g)
        loss = F.cross_entropy(model(kes(X[idx])), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()
    return model


@torch.no_grad()
def bashorat(model, X):
    return model(kes(X)).argmax(1).numpy()


def tur_aniqligi(y, p, tur):
    return {t: float(np.mean(p[tur == t] == y[tur == t])) for t in TURLAR}


def main() -> None:
    torch.set_num_threads(1)
    k = Konfig()
    seed_everything(k.seed)
    matn, y, tur, muallif = malumot(seed=0)

    print("=== 1. Belgili ma'lumot ===")
    print(f"  sharhlar {len(y)}, mualliflar {len(set(muallif.tolist()))}, "
          f"ijobiy ulushi {y.mean():.3f}")
    print(f"  {'tur':<11} {'soni':>5} {'ulush':>6}   misol")
    for t in TURLAR:
        i = int(np.flatnonzero(tur == t)[0])
        print(f"  {t:<11} {np.sum(tur == t):>5} {np.mean(tur == t):>6.1%}   "
              f"[{SINFLAR[y[i]]}] {matn[i]}")
    takror = sum(a == b and ma == mb for a, b, ma, mb in
                 zip(matn[:-1], matn[1:], muallif[:-1], muallif[1:]))
    print(f"  qayta yuborilgan sharhlar: {takror}; yorliq shovqini ~3%")

    print("\n=== 2. Nega bu turlar qiyin (so'zlar bir xil, ma'no teskari) ===")
    juftlar = [("kafe juda a'lo edi", "kafe juda a'lo emas edi"),
               ("kafe narxi a'lo deb o'ylayman", "kafe narxi a'lo deb o'ylamayman"),
               ("kafe narxi yomon lekin sifati a'lo", "kafe narxi a'lo lekin sifati yomon")]
    for a, b in juftlar:
        umumiy = set(a.split()) & set(b.split())
        print(f"  {a:<36} | {b}")
        print(f"    umumiy so'zlar {len(umumiy)}/{len(set(a.split()) | set(b.split()))}")
    print("  so'z xaltasi 3-juftlikni UMUMAN ajrata olmaydi - so'zlar to'plami bir xil")

    print("\n=== 3. Muallif bo'yicha bo'lish: test QULF ===")
    ish, te = guruhli_bolish(y, muallif, k.test_ulushi, k.seed)
    print(f"  ish {len(ish)}, test {len(te)}; umumiy muallif: "
          f"{len(set(muallif[ish].tolist()) & set(muallif[te].tolist()))}")
    for nom, ii in [("ish", ish), ("test", te)]:
        print(f"  {nom:<5} ijobiy {y[ii].mean():.3f}   " + ", ".join(
            f"{t} {np.mean(tur[ii] == t):.2f}" for t in TURLAR))
    bir_xil = len(set(matn[ish].tolist()) & set(matn[te].tolist()))
    print(f"  ish va testda aynan bir xil matnlar: {bir_xil}")

    print("\n=== 4. Belgisiz korpus va tokenizator ===")
    kor = korpus()
    lug = Lugat(kor)
    print(f"  korpus {len(kor)} sharh, lug'at {len(lug.itos)} (faqat korpusdan)")
    X = lug.kodla(list(matn[ish]), k.T)
    uz = (X != PAD).sum(1)
    unk = (X == UNK).sum().item() / (X > PAD).sum().item()
    print(f"  ish: [UNK] ulushi {unk:.4f}, eng uzun {uz.max().item()} token (T = {k.T})")
    korpusda = len(set(kor) & set(matn[te].tolist()))
    print(f"  test matnlaridan korpusda ham borlari: {korpusda} / {len(te)} "
          "(yorliqsiz - sizish emas, lekin bilib qo'yamiz)")

    print("\n=== 5. Tez tekshiruvlar ===")
    torch.manual_seed(k.seed)
    model = Klassifikator(yangi_encoder(k, len(lug.itos)))
    xb, yb = X[:k.bs], torch.as_tensor(y[ish][:k.bs])
    with torch.no_grad():
        ch = model(kes(xb))
        boshl = F.cross_entropy(ch, yb).item()
    Xm, Ym = maskala(X, torch.Generator().manual_seed(0), len(lug.itos))
    tekshiruvlar = [
        ("kirish shakli", tuple(kes(xb).shape), kes(xb).shape[0] == k.bs),
        ("chiqish shakli", tuple(ch.shape), tuple(ch.shape) == (k.bs, 2)),
        ("boshlang'ich loss", round(boshl, 3), abs(boshl - math.log(2)) < 0.1),
        ("uzunlik <= T", int(uz.max()), int(uz.max()) < k.T),
        ("maskalangan ulush", round((Ym != -100).float().sum().item()
                                    / (X > UNK).sum().item(), 3),
         abs((Ym != -100).float().sum().item() / (X > UNK).sum().item() - 0.15) < 0.02),
    ]
    for nom, q, ok in tekshiruvlar:
        print(f"  {nom:<18} {str(q):>12} {'OK' if ok else 'XATO':>6}")
    print(f"  kutilgan boshlang'ich loss ln(2) = {math.log(2):.3f}")

    print("\n=== 6. Bitta batch testi ===")
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    model.train()
    for _ in range(100):
        loss = F.cross_entropy(model(kes(xb)), yb)
        opt.zero_grad()
        loss.backward()
        opt.step()
    print(f"  {k.bs} sharh, 100 qadam: loss {loss.item():.4f} -> "
          f"{'OTDI' if loss.item() < 0.05 else 'OTMADI'}")
    print(f"  parametrlar: {sum(p.numel() for p in model.parameters())}")


if __name__ == "__main__":
    main()

=== 1. Belgili ma'lumot ===
  sharhlar 1420, mualliflar 350, ijobiy ulushi 0.494
  tur          soni  ulush   misol
  oddiy         707  49.8%   [ijobiy] kafe qiziqarli chiqdi
  inkor         222  15.6%   [salbiy] bu noutbuk biroz tez emas
  uzoq inkor    214  15.1%   [salbiy] onlayn kurs ovozi noqulay deb hisoblayman vaqtim behuda ketdi
  kontrast      277  19.5%   [ijobiy] kecha do'kon narxi yomon ekan biroq dizayni qulay
  qayta yuborilgan sharhlar: 92; yorliq shovqini ~3%

=== 2. Nega bu turlar qiyin (so'zlar bir xil, ma'no teskari) ===
  kafe juda a'lo edi                   | kafe juda a'lo emas edi
    umumiy so'zlar 4/5
  kafe narxi a'lo deb o'ylayman        | kafe narxi a'lo deb o'ylamayman
    umumiy so'zlar 4/6
  kafe narxi yomon lekin sifati a'lo   | kafe narxi a'lo lekin sifati yomon
    umumiy so'zlar 6/6
  so'z xaltasi 3-juftlikni UMUMAN ajrata olmaydi - so'zlar to'plami bir xil

=== 3. Muallif bo'yicha bo'lish: test QULF ===
  ish 1137, test 283; umumiy muallif: 0
  ish   ijobiy 0.488   oddiy 0.50, inkor 0.15, uzoq inkor 0.15, kontrast 0.19
  test  ijobiy 0.516   oddiy 0.48, inkor 0.17, uzoq inkor 0.14, kontrast 0.20
  ish va testda aynan bir xil matnlar: 0

=== 4. Belgisiz korpus va tokenizator ===
  korpus 20000 sharh, lug'at 131 (faqat korpusdan)
  ish: [UNK] ulushi 0.0000, eng uzun 15 token (T = 24)
  test matnlaridan korpusda ham borlari: 5 / 283 (yorliqsiz - sizish emas, lekin bilib qo'yamiz)

=== 5. Tez tekshiruvlar ===
  kirish shakli          (32, 10)     OK
  chiqish shakli          (32, 2)     OK
  boshlang'ich loss         0.684     OK
  uzunlik <= T                 15     OK
  maskalangan ulush         0.143     OK
  kutilgan boshlang'ich loss ln(2) = 0.693

=== 6. Bitta batch testi ===
  32 sharh, 100 qadam: loss 0.0007 -> OTDI
  parametrlar: 64178

"""2-qadam: bazaviylar, noldan transformer va pretraining + fine-tuning validatsiyada."""

import copy
import math
import random
from dataclasses import asdict, dataclass

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, GroupShuffleSplit
from sklearn.pipeline import make_pipeline

MAVZU = ["telefon", "noutbuk", "kitob", "kafe", "mehmonxona", "avtobus", "dastur",
         "kurs", "film", "do'kon", "restoran", "poyabzal", "kamera", "sumka", "taksi",
         "klinika", "sartaroshxona", "televizor"]
ASPEKT = ["narxi", "sifati", "xizmati", "dizayni", "yetkazib berish", "xodimlari",
          "joylashuvi", "ovozi"]
IJOBIY = ["a'lo", "ajoyib", "zo'r", "yaxshi", "qulay", "chiroyli", "tez", "sifatli",
          "mazali", "toza", "arzon", "ishonchli", "shinam", "samimiy", "foydali",
          "qiziqarli", "mukammal", "puxta", "yoqimli", "maroqli", "chidamli", "oson",
          "zamonaviy", "beg'ubor"]
SALBIY = ["yomon", "sekin", "qimmat", "iflos", "noqulay", "xunuk", "sifatsiz", "bemaza",
          "eski", "zerikarli", "ishonchsiz", "qo'pol", "buzuq", "chalkash", "nosoz", "xira",
          "tor", "shovqinli", "befoyda", "sovuq", "mo'rt", "murakkab", "dag'al", "g'alati"]
DARAJA = ["juda", "ancha", "biroz", "rosa", "", ""]
OXIR = ["ekan", "edi", "", "chiqdi"]
BOSH = ["kecha", "o'tgan hafta", "shahardagi", "yangi", "onlayn", "bu", "", ""]
DUM = {1: ["hammaga tavsiya qilaman", "yana kelaman", "juda mamnunman", "pulimga arzidi",
           "do'stlarimga aytaman", "rahmat sizlarga"],
       0: ["hech kimga tavsiya qilmayman", "pulimni qaytarib oldim", "boshqa kelmayman",
           "vaqtim behuda ketdi", "shikoyat yozdim", "afsuslandim"]}
TURLAR = ["oddiy", "inkor", "uzoq inkor", "kontrast"]
TUR_ULUSH = [0.5, 0.15, 0.15, 0.2]
SINFLAR = ["salbiy", "ijobiy"]
PAD, CLS, MASK, UNK = 0, 1, 2, 3


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def sharh(rng, tur, dum_ehtimol, bosh=None):
    """Bitta sharh: (matn, tonallik). Qiyin turlar so'z tartibi va inkorga bog'liq."""
    q = int(rng.integers(2))

    def sif(p):
        return tanla(rng, IJOBIY if p else SALBIY)

    if tur == "oddiy":
        s, y = [tanla(rng, MAVZU), tanla(rng, DARAJA), sif(q), tanla(rng, OXIR)], q
    elif tur == "inkor":                                   # "a'lo emas" -> salbiy
        s, y = [tanla(rng, MAVZU), tanla(rng, DARAJA), sif(q), "emas", tanla(rng, OXIR)], 1 - q
    elif tur == "uzoq inkor":                              # inkor fe'lda, sifatdan uzoqda
        tasdiq = rng.random() < 0.5
        fel = tanla(rng, ["deb o'ylayman", "deb hisoblayman", "deya olaman"] if tasdiq
                    else ["deb o'ylamayman", "deb hisoblamayman", "deya olmayman"])
        s, y = [tanla(rng, MAVZU), tanla(rng, ASPEKT), sif(q), fel], q if tasdiq else 1 - q
    else:                                                  # "X yomon, lekin Y a'lo" -> Y hal qiladi
        a1, a2 = rng.choice(len(ASPEKT), 2, replace=False)
        s = [tanla(rng, MAVZU), ASPEKT[a1], sif(1 - q), tanla(rng, ["edi", "", "ekan"]),
             tanla(rng, ["lekin", "ammo", "biroq"]), ASPEKT[a2], tanla(rng, DARAJA), sif(q)]
        y = q
    s = [tanla(rng, BOSH) if bosh is None else bosh] + s
    if rng.random() < dum_ehtimol:
        s.append(tanla(rng, DUM[y]))
    return " ".join(" ".join(s).split()), y


def malumot(seed=0, n_muallif=350, p_takror=0.1, p_shovqin=0.03):
    """Belgili sharhlar: matn, tonallik, tur, muallif. Muallifning o'z uslubi bor."""
    rng = np.random.default_rng(seed)
    matn, y, tur, muallif = [], [], [], []
    for m in range(n_muallif):
        uslub = tanla(rng, BOSH)
        for _ in range(int(rng.geometric(1 / 4))):
            if matn and muallif[-1] == m and rng.random() < p_takror:   # qayta yuborish
                matn.append(matn[-1])
                y.append(y[-1])
                tur.append(tur[-1])
            else:
                t = TURLAR[int(rng.choice(4, p=TUR_ULUSH))]
                s, yorliq = sharh(rng, t, 0.15, uslub if rng.random() < 0.6 else None)
                matn.append(s)
                tur.append(t)
                y.append(1 - yorliq if rng.random() < p_shovqin else yorliq)
            muallif.append(m)
    return np.array(matn, dtype=object), np.array(y), np.array(tur), np.array(muallif)


def korpus(seed=1, n=20000):
    """Belgisiz sharhlar (pretraining uchun): yorliq ham, tur ham saqlanmaydi."""
    rng = np.random.default_rng(seed)
    return [sharh(rng, TURLAR[int(rng.choice(4, p=TUR_ULUSH))], 0.6)[0] for _ in range(n)]


@dataclass(frozen=True)
class Konfig:
    seed: int = 42
    test_ulushi: float = 0.2
    T: int = 24
    d: int = 48
    L: int = 2
    h: int = 4
    pre_qadam: int = 800
    qadam: int = 150
    lr: float = 2e-3
    bs: int = 32
    C: float = 10.0


def seed_everything(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)


def guruhli_bolish(y, guruh, ulush, seed):
    """Muallif bo'yicha: bir muallifning hamma sharhi bitta tomonda."""
    gss = GroupShuffleSplit(n_splits=1, test_size=ulush, random_state=seed)
    return next(gss.split(np.zeros(len(y)), y, guruh))


def soz_tfidf(k):
    return make_pipeline(TfidfVectorizer(token_pattern=r"[^ ]+", ngram_range=(1, 2),
                                         sublinear_tf=True),
                         LogisticRegression(C=k.C, max_iter=3000))


def belgi_tfidf(k):
    return make_pipeline(TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5),
                                         sublinear_tf=True),
                         LogisticRegression(C=k.C, max_iter=3000))


class Lugat:
    """So'z tokenizatori; FAQAT belgisiz korpusdan (pretrained model bilan birga keladi)."""

    def __init__(self, matnlar):
        sozlar = sorted({w for m in matnlar for w in m.split()})
        self.itos = ["[PAD]", "[CLS]", "[MASK]", "[UNK]"] + sozlar
        self.stoi = {w: i for i, w in enumerate(self.itos)}

    def kodla(self, matnlar, T):
        X = torch.zeros(len(matnlar), T, dtype=torch.long)
        for i, m in enumerate(matnlar):
            ids = [CLS] + [self.stoi.get(w, UNK) for w in m.split()][:T - 1]
            X[i, :len(ids)] = torch.tensor(ids)
        return X


class Blok(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x, maska):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).view(B, T, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=maska[:, None, None, :])
        x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class Encoder(nn.Module):
    def __init__(self, V, d, L, h, T):
        super().__init__()
        self.emb, self.poz = nn.Embedding(V, d), nn.Embedding(T, d)
        self.bloklar = nn.ModuleList([Blok(d, h) for _ in range(L)])
        self.ln = nn.LayerNorm(d)
        nn.init.normal_(self.emb.weight, std=0.02)
        nn.init.normal_(self.poz.weight, std=0.02)

    def forward(self, X):
        maska = X != PAD
        x = self.emb(X) + self.poz(torch.arange(X.shape[1]))
        for b in self.bloklar:
            x = b(x, maska)
        return self.ln(x)


class Klassifikator(nn.Module):
    """Encoder + o'rtacha pooling (PAD siz) + chiziqli bosh."""

    def __init__(self, enc):
        super().__init__()
        self.enc = enc
        self.bosh = nn.Linear(enc.emb.embedding_dim, len(SINFLAR))
        nn.init.normal_(self.bosh.weight, std=0.02)       # boshlang'ich loss ~ ln 2
        nn.init.zeros_(self.bosh.bias)

    def forward(self, X):
        h = self.enc(X)
        m = (X != PAD).float()[..., None]
        return self.bosh((h * m).sum(1) / m.sum(1))


def yangi_encoder(k, V):
    return Encoder(V, k.d, k.L, k.h, k.T)


def kes(X):
    return X[:, :int((X != PAD).sum(1).max())]


def maskala(X, g, V, p=0.15):
    Y = X.clone()
    tanl = (torch.rand(X.shape, generator=g) < p) & (X > UNK)
    Y[~tanl] = -100
    r = torch.rand(X.shape, generator=g)
    X = X.clone()
    X[tanl & (r < 0.8)] = MASK
    tas = tanl & (r >= 0.8) & (r < 0.9)
    X[tas] = torch.randint(4, V, (int(tas.sum()),), generator=g)
    return X, Y


def pretrain(k, V, X, seed=0):
    """MLM pretraining (24.8, 24.10): bog'langan vaznlar, 15% maskalash."""
    torch.manual_seed(seed)
    enc = yangi_encoder(k, V)
    bias = nn.Parameter(torch.zeros(V))
    opt = torch.optim.AdamW(list(enc.parameters()) + [bias], lr=k.lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(k.pre_qadam):
        xb, yb = maskala(kes(X[torch.randint(0, len(X), (k.bs,), generator=g)]), g, V)
        loss = F.cross_entropy((enc(xb) @ enc.emb.weight.T + bias).reshape(-1, V), yb.reshape(-1))
        opt.zero_grad()
        loss.backward()
        opt.step()
    return enc, loss.item()


def orgat(k, enc, X, y, seed):
    """Fine-tuning yoki noldan o'rgatish: bir xil sikl, bir xil qadamlar."""
    torch.manual_seed(seed)
    model = Klassifikator(enc)
    opt = torch.optim.AdamW(model.parameters(), lr=k.lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    y = torch.as_tensor(y)
    for _ in range(k.qadam):
        idx = torch.randint(0, len(X), (k.bs,), generator=g)
        loss = F.cross_entropy(model(kes(X[idx])), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()
    return model


@torch.no_grad()
def bashorat(model, X):
    return model(kes(X)).argmax(1).numpy()


def tur_aniqligi(y, p, tur):
    return {t: float(np.mean(p[tur == t] == y[tur == t])) for t in TURLAR}


def orgat_kuzat(k, enc, X, y, seed, Xv, yv, nuqtalar):
    """orgat() bilan AYNAN bir xil sikl, lekin belgilangan qadamlarda validatsiya."""
    torch.manual_seed(seed)
    model = Klassifikator(enc)
    opt = torch.optim.AdamW(model.parameters(), lr=k.lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    y = torch.as_tensor(y)
    aniq, P = {}, None
    for q in range(1, max(nuqtalar) + 1):
        model.train()
        idx = torch.randint(0, len(X), (k.bs,), generator=g)
        loss = F.cross_entropy(model(kes(X[idx])), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
        if q in nuqtalar:
            model.eval()
            p = bashorat(model, Xv)
            aniq[q] = float(np.mean(p == yv))
            if q == k.qadam:
                P = p
    return aniq, P


def main() -> None:
    torch.set_num_threads(1)
    k = Konfig()
    matn, y, tur, muallif = malumot(seed=0)
    ish, _ = guruhli_bolish(y, muallif, k.test_ulushi, k.seed)       # test tegilmaydi
    a, b = guruhli_bolish(y[ish], muallif[ish], 0.25, k.seed)
    tr, va = ish[a], ish[b]
    print("=== 1. Ish ichida: o'quv / validatsiya (muallif bo'yicha) ===")
    print(f"  o'quv {len(tr)}, validatsiya {len(va)}")
    kor = korpus()
    lug = Lugat(kor)
    V = len(lug.itos)
    X = lug.kodla(list(matn), k.T)
    enc_pre, mlm = pretrain(k, V, lug.kodla(kor, k.T))
    print(f"  pretraining: {k.pre_qadam} qadam, oxirgi MLM loss {mlm:.3f}")

    nuqtalar = [50, 100, k.qadam, 2 * k.qadam]
    egri, P = {}, {}
    for nom, enc in [("transformer", yangi_encoder(k, V)), ("pretrain+FT", copy.deepcopy(enc_pre))]:
        egri[nom], P[nom] = orgat_kuzat(k, enc, X[tr], y[tr], k.seed, X[va], y[va], nuqtalar)

    print(f"\n=== 2. Nomzodlar validatsiyada (transformerlar {k.qadam} qadam) ===")
    P = {"ko'pchilik": np.full(len(va), np.bincount(y[tr]).argmax()),
         "so'z TF-IDF": soz_tfidf(k).fit(matn[tr], y[tr]).predict(matn[va]),
         "belgi TF-IDF": belgi_tfidf(k).fit(matn[tr], y[tr]).predict(matn[va]), **P}
    print(f"  {'model':<13} {'aniqlik':>8}  " + " ".join(f"{t:>10}" for t in TURLAR))
    for nom, p in P.items():
        ta = tur_aniqligi(y[va], p, tur[va])
        print(f"  {nom:<13} {np.mean(p == y[va]):>8.3f}  "
              + " ".join(f"{ta[t]:>10.3f}" for t in TURLAR))
    print("  validatsiyada turlar soni: " + ", ".join(
        f"{t} {np.sum(tur[va] == t)}" for t in TURLAR))

    print("\n=== 3. Belgi TF-IDF va 'inkor' ===")
    s = tur[va] == "inkor"
    ai = np.mean(P["belgi TF-IDF"][s] == y[va][s])
    print(f"  inkor turida aniqlik {ai:.3f}" + (" - tasodifdan PAST" if ai < 0.5 else ""))
    print("  char_wb n-gramlari so'z ichida qoladi: 'a'lo' va 'emas' ning n-gramlari alohida,")
    print("  chiziqli model ularning BIRGALIGINI ko'ra olmaydi va sifatga ishonadi")

    print("\n=== 4. O'rganish tezligi: validatsiya aniqligi qadamlar bo'yicha ===")
    print(f"  {'qadam':>6} {'transformer':>12} {'pretrain+FT':>12} {'farq':>8}")
    for q in nuqtalar:
        t0, t1 = egri["transformer"][q], egri["pretrain+FT"][q]
        print(f"  {q:>6} {t0:>12.3f} {t1:>12.3f} {t1 - t0:>+8.3f}")
    oxir = nuqtalar[-1]
    if egri["pretrain+FT"][k.qadam] > egri["transformer"][k.qadam] and \
            egri["transformer"][oxir] > egri["transformer"][k.qadam]:
        print(f"  noldan model sekinroq o'rganadi: {oxir} qadamda "
              f"{egri['transformer'][oxir]:.3f} ga chiqdi")
    print("  bitta bo'lak - qaror uchun 3-qadam (5 fold, juftlashgan farq) kerak")


if __name__ == "__main__":
    main()

=== 1. Ish ichida: o'quv / validatsiya (muallif bo'yicha) ===
  o'quv 850, validatsiya 287
  pretraining: 800 qadam, oxirgi MLM loss 2.139

=== 2. Nomzodlar validatsiyada (transformerlar 150 qadam) ===
  model          aniqlik       oddiy      inkor uzoq inkor   kontrast
  ko'pchilik       0.519       0.533      0.429      0.465      0.600
  so'z TF-IDF      0.728       0.834      0.743      0.535      0.475
  belgi TF-IDF     0.679       0.834      0.229      0.535      0.575
  transformer      0.875       0.947      0.771      0.674      0.875
  pretrain+FT      0.969       0.964      0.971      0.977      0.975
  validatsiyada turlar soni: oddiy 169, inkor 35, uzoq inkor 43, kontrast 40

=== 3. Belgi TF-IDF va 'inkor' ===
  inkor turida aniqlik 0.229 - tasodifdan PAST
  char_wb n-gramlari so'z ichida qoladi: 'a'lo' va 'emas' ning n-gramlari alohida,
  chiziqli model ularning BIRGALIGINI ko'ra olmaydi va sifatga ishonadi

=== 4. O'rganish tezligi: validatsiya aniqligi qadamlar bo'yicha ===
   qadam  transformer  pretrain+FT     farq
      50        0.655        0.714   +0.059
     100        0.787        0.843   +0.056
     150        0.875        0.969   +0.094
     300        0.937        0.955   +0.017
  noldan model sekinroq o'rganadi: 300 qadamda 0.937 ga chiqdi
  bitta bo'lak - qaror uchun 3-qadam (5 fold, juftlashgan farq) kerak

"""3-qadam: GroupKFold (muallif) da juftlashgan taqqoslash va qaror."""

import copy
import math
import random
from dataclasses import asdict, dataclass

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, GroupShuffleSplit
from sklearn.pipeline import make_pipeline

MAVZU = ["telefon", "noutbuk", "kitob", "kafe", "mehmonxona", "avtobus", "dastur",
         "kurs", "film", "do'kon", "restoran", "poyabzal", "kamera", "sumka", "taksi",
         "klinika", "sartaroshxona", "televizor"]
ASPEKT = ["narxi", "sifati", "xizmati", "dizayni", "yetkazib berish", "xodimlari",
          "joylashuvi", "ovozi"]
IJOBIY = ["a'lo", "ajoyib", "zo'r", "yaxshi", "qulay", "chiroyli", "tez", "sifatli",
          "mazali", "toza", "arzon", "ishonchli", "shinam", "samimiy", "foydali",
          "qiziqarli", "mukammal", "puxta", "yoqimli", "maroqli", "chidamli", "oson",
          "zamonaviy", "beg'ubor"]
SALBIY = ["yomon", "sekin", "qimmat", "iflos", "noqulay", "xunuk", "sifatsiz", "bemaza",
          "eski", "zerikarli", "ishonchsiz", "qo'pol", "buzuq", "chalkash", "nosoz", "xira",
          "tor", "shovqinli", "befoyda", "sovuq", "mo'rt", "murakkab", "dag'al", "g'alati"]
DARAJA = ["juda", "ancha", "biroz", "rosa", "", ""]
OXIR = ["ekan", "edi", "", "chiqdi"]
BOSH = ["kecha", "o'tgan hafta", "shahardagi", "yangi", "onlayn", "bu", "", ""]
DUM = {1: ["hammaga tavsiya qilaman", "yana kelaman", "juda mamnunman", "pulimga arzidi",
           "do'stlarimga aytaman", "rahmat sizlarga"],
       0: ["hech kimga tavsiya qilmayman", "pulimni qaytarib oldim", "boshqa kelmayman",
           "vaqtim behuda ketdi", "shikoyat yozdim", "afsuslandim"]}
TURLAR = ["oddiy", "inkor", "uzoq inkor", "kontrast"]
TUR_ULUSH = [0.5, 0.15, 0.15, 0.2]
SINFLAR = ["salbiy", "ijobiy"]
PAD, CLS, MASK, UNK = 0, 1, 2, 3


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def sharh(rng, tur, dum_ehtimol, bosh=None):
    """Bitta sharh: (matn, tonallik). Qiyin turlar so'z tartibi va inkorga bog'liq."""
    q = int(rng.integers(2))

    def sif(p):
        return tanla(rng, IJOBIY if p else SALBIY)

    if tur == "oddiy":
        s, y = [tanla(rng, MAVZU), tanla(rng, DARAJA), sif(q), tanla(rng, OXIR)], q
    elif tur == "inkor":                                   # "a'lo emas" -> salbiy
        s, y = [tanla(rng, MAVZU), tanla(rng, DARAJA), sif(q), "emas", tanla(rng, OXIR)], 1 - q
    elif tur == "uzoq inkor":                              # inkor fe'lda, sifatdan uzoqda
        tasdiq = rng.random() < 0.5
        fel = tanla(rng, ["deb o'ylayman", "deb hisoblayman", "deya olaman"] if tasdiq
                    else ["deb o'ylamayman", "deb hisoblamayman", "deya olmayman"])
        s, y = [tanla(rng, MAVZU), tanla(rng, ASPEKT), sif(q), fel], q if tasdiq else 1 - q
    else:                                                  # "X yomon, lekin Y a'lo" -> Y hal qiladi
        a1, a2 = rng.choice(len(ASPEKT), 2, replace=False)
        s = [tanla(rng, MAVZU), ASPEKT[a1], sif(1 - q), tanla(rng, ["edi", "", "ekan"]),
             tanla(rng, ["lekin", "ammo", "biroq"]), ASPEKT[a2], tanla(rng, DARAJA), sif(q)]
        y = q
    s = [tanla(rng, BOSH) if bosh is None else bosh] + s
    if rng.random() < dum_ehtimol:
        s.append(tanla(rng, DUM[y]))
    return " ".join(" ".join(s).split()), y


def malumot(seed=0, n_muallif=350, p_takror=0.1, p_shovqin=0.03):
    """Belgili sharhlar: matn, tonallik, tur, muallif. Muallifning o'z uslubi bor."""
    rng = np.random.default_rng(seed)
    matn, y, tur, muallif = [], [], [], []
    for m in range(n_muallif):
        uslub = tanla(rng, BOSH)
        for _ in range(int(rng.geometric(1 / 4))):
            if matn and muallif[-1] == m and rng.random() < p_takror:   # qayta yuborish
                matn.append(matn[-1])
                y.append(y[-1])
                tur.append(tur[-1])
            else:
                t = TURLAR[int(rng.choice(4, p=TUR_ULUSH))]
                s, yorliq = sharh(rng, t, 0.15, uslub if rng.random() < 0.6 else None)
                matn.append(s)
                tur.append(t)
                y.append(1 - yorliq if rng.random() < p_shovqin else yorliq)
            muallif.append(m)
    return np.array(matn, dtype=object), np.array(y), np.array(tur), np.array(muallif)


def korpus(seed=1, n=20000):
    """Belgisiz sharhlar (pretraining uchun): yorliq ham, tur ham saqlanmaydi."""
    rng = np.random.default_rng(seed)
    return [sharh(rng, TURLAR[int(rng.choice(4, p=TUR_ULUSH))], 0.6)[0] for _ in range(n)]


@dataclass(frozen=True)
class Konfig:
    seed: int = 42
    test_ulushi: float = 0.2
    T: int = 24
    d: int = 48
    L: int = 2
    h: int = 4
    pre_qadam: int = 800
    qadam: int = 150
    lr: float = 2e-3
    bs: int = 32
    C: float = 10.0


def seed_everything(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)


def guruhli_bolish(y, guruh, ulush, seed):
    """Muallif bo'yicha: bir muallifning hamma sharhi bitta tomonda."""
    gss = GroupShuffleSplit(n_splits=1, test_size=ulush, random_state=seed)
    return next(gss.split(np.zeros(len(y)), y, guruh))


def soz_tfidf(k):
    return make_pipeline(TfidfVectorizer(token_pattern=r"[^ ]+", ngram_range=(1, 2),
                                         sublinear_tf=True),
                         LogisticRegression(C=k.C, max_iter=3000))


def belgi_tfidf(k):
    return make_pipeline(TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5),
                                         sublinear_tf=True),
                         LogisticRegression(C=k.C, max_iter=3000))


class Lugat:
    """So'z tokenizatori; FAQAT belgisiz korpusdan (pretrained model bilan birga keladi)."""

    def __init__(self, matnlar):
        sozlar = sorted({w for m in matnlar for w in m.split()})
        self.itos = ["[PAD]", "[CLS]", "[MASK]", "[UNK]"] + sozlar
        self.stoi = {w: i for i, w in enumerate(self.itos)}

    def kodla(self, matnlar, T):
        X = torch.zeros(len(matnlar), T, dtype=torch.long)
        for i, m in enumerate(matnlar):
            ids = [CLS] + [self.stoi.get(w, UNK) for w in m.split()][:T - 1]
            X[i, :len(ids)] = torch.tensor(ids)
        return X


class Blok(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x, maska):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).view(B, T, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=maska[:, None, None, :])
        x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class Encoder(nn.Module):
    def __init__(self, V, d, L, h, T):
        super().__init__()
        self.emb, self.poz = nn.Embedding(V, d), nn.Embedding(T, d)
        self.bloklar = nn.ModuleList([Blok(d, h) for _ in range(L)])
        self.ln = nn.LayerNorm(d)
        nn.init.normal_(self.emb.weight, std=0.02)
        nn.init.normal_(self.poz.weight, std=0.02)

    def forward(self, X):
        maska = X != PAD
        x = self.emb(X) + self.poz(torch.arange(X.shape[1]))
        for b in self.bloklar:
            x = b(x, maska)
        return self.ln(x)


class Klassifikator(nn.Module):
    """Encoder + o'rtacha pooling (PAD siz) + chiziqli bosh."""

    def __init__(self, enc):
        super().__init__()
        self.enc = enc
        self.bosh = nn.Linear(enc.emb.embedding_dim, len(SINFLAR))
        nn.init.normal_(self.bosh.weight, std=0.02)       # boshlang'ich loss ~ ln 2
        nn.init.zeros_(self.bosh.bias)

    def forward(self, X):
        h = self.enc(X)
        m = (X != PAD).float()[..., None]
        return self.bosh((h * m).sum(1) / m.sum(1))


def yangi_encoder(k, V):
    return Encoder(V, k.d, k.L, k.h, k.T)


def kes(X):
    return X[:, :int((X != PAD).sum(1).max())]


def maskala(X, g, V, p=0.15):
    Y = X.clone()
    tanl = (torch.rand(X.shape, generator=g) < p) & (X > UNK)
    Y[~tanl] = -100
    r = torch.rand(X.shape, generator=g)
    X = X.clone()
    X[tanl & (r < 0.8)] = MASK
    tas = tanl & (r >= 0.8) & (r < 0.9)
    X[tas] = torch.randint(4, V, (int(tas.sum()),), generator=g)
    return X, Y


def pretrain(k, V, X, seed=0):
    """MLM pretraining (24.8, 24.10): bog'langan vaznlar, 15% maskalash."""
    torch.manual_seed(seed)
    enc = yangi_encoder(k, V)
    bias = nn.Parameter(torch.zeros(V))
    opt = torch.optim.AdamW(list(enc.parameters()) + [bias], lr=k.lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(k.pre_qadam):
        xb, yb = maskala(kes(X[torch.randint(0, len(X), (k.bs,), generator=g)]), g, V)
        loss = F.cross_entropy((enc(xb) @ enc.emb.weight.T + bias).reshape(-1, V), yb.reshape(-1))
        opt.zero_grad()
        loss.backward()
        opt.step()
    return enc, loss.item()


def orgat(k, enc, X, y, seed):
    """Fine-tuning yoki noldan o'rgatish: bir xil sikl, bir xil qadamlar."""
    torch.manual_seed(seed)
    model = Klassifikator(enc)
    opt = torch.optim.AdamW(model.parameters(), lr=k.lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    y = torch.as_tensor(y)
    for _ in range(k.qadam):
        idx = torch.randint(0, len(X), (k.bs,), generator=g)
        loss = F.cross_entropy(model(kes(X[idx])), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()
    return model


@torch.no_grad()
def bashorat(model, X):
    return model(kes(X)).argmax(1).numpy()


def tur_aniqligi(y, p, tur):
    return {t: float(np.mean(p[tur == t] == y[tur == t])) for t in TURLAR}


SODDALIK = ["ko'pchilik", "so'z TF-IDF", "belgi TF-IDF", "transformer", "pretrain+FT"]


def main() -> None:
    torch.set_num_threads(1)
    k = Konfig()
    matn, y, tur, muallif = malumot(seed=0)
    ish, _ = guruhli_bolish(y, muallif, k.test_ulushi, k.seed)       # test tegilmaydi
    m, yy, tt, mj = matn[ish], y[ish], tur[ish], muallif[ish]
    kor = korpus()
    lug = Lugat(kor)
    V = len(lug.itos)
    X = lug.kodla(list(m), k.T)
    enc_pre, _ = pretrain(k, V, lug.kodla(kor, k.T))     # belgisiz korpus - foldlardan tashqarida

    print("=== 1. GroupKFold(5): muallif bo'yicha, bir xil foldlar ===")
    ball = {n: [] for n in SODDALIK}
    turb = {n: {t: [] for t in TURLAR} for n in SODDALIK}
    for f, (tr, va) in enumerate(GroupKFold(5).split(m, yy, mj), 1):
        P = {"ko'pchilik": np.full(len(va), np.bincount(yy[tr]).argmax()),
             "so'z TF-IDF": soz_tfidf(k).fit(m[tr], yy[tr]).predict(m[va]),
             "belgi TF-IDF": belgi_tfidf(k).fit(m[tr], yy[tr]).predict(m[va]),
             "transformer": bashorat(orgat(k, yangi_encoder(k, V), X[tr], yy[tr], f), X[va]),
             "pretrain+FT": bashorat(orgat(k, copy.deepcopy(enc_pre), X[tr], yy[tr], f), X[va])}
        for n, p in P.items():
            ball[n].append(np.mean(p == yy[va]))
            for t, v in tur_aniqligi(yy[va], p, tt[va]).items():
                turb[n][t].append(v)
        print(f"  fold {f}: " + ", ".join(f"{n.split()[0]} {ball[n][-1]:.3f}"
                                         for n in SODDALIK[1:]))
    ball = {n: np.array(v) for n, v in ball.items()}

    print("\n=== 2. Natijalar (5 fold o'rtachasi) ===")
    print(f"  {'model':<13} {'aniqlik':>8} {'+-std':>6}  " + " ".join(f"{t:>10}" for t in TURLAR))
    for n in SODDALIK:
        print(f"  {n:<13} {ball[n].mean():>8.4f} {ball[n].std(ddof=1):>6.3f}  "
              + " ".join(f"{np.mean(turb[n][t]):>10.3f}" for t in TURLAR))
    eng = max(SODDALIK, key=lambda n: ball[n].mean())
    print(f"  eng yuqori o'rtacha: {eng}")

    print(f"\n=== 3. Juftlashgan farq: eng yaxshisi ({eng}) - model ===")
    print(f"  {'model':<13} {'farq':>8} {'2*SE':>7}  sezilarli yomonmi")
    yomon = {}
    for n in SODDALIK:
        if n == eng:
            continue
        d = ball[eng] - ball[n]
        se = d.std(ddof=1) / np.sqrt(len(d))
        yomon[n] = d.mean() > 2 * se
        print(f"  {n:<13} {d.mean():>+8.4f} {2 * se:>7.4f}  {'ha' if yomon[n] else "yo'q"}")

    print("\n=== 4. Qaror ===")
    tanlov = next(n for n in SODDALIK if n == eng or not yomon[n])
    print("  qoida: eng yaxshisidan SEZILARLI yomon bo'lmagan ENG SODDA model")
    print(f"  soddalik tartibi: {' < '.join(SODDALIK)}")
    print(f"  TANLOV: {tanlov}")
    if tanlov != eng:
        print(f"  ({eng} o'rtachada yuqori, lekin farq 2*SE ichida - "
              "murakkablik o'zini oqlamadi)")
    baz = max(SODDALIK[:3], key=lambda n: ball[n].mean())
    print(f"  eng yaxshi bazaviy ({baz}) dan ustunlik: {ball[tanlov].mean() - ball[baz].mean():+.4f}")


if __name__ == "__main__":
    main()

=== 1. GroupKFold(5): muallif bo'yicha, bir xil foldlar ===
  fold 1: so'z 0.759, belgi 0.667, transformer 0.912, pretrain+FT 0.934
  fold 2: so'z 0.772, belgi 0.658, transformer 0.908, pretrain+FT 0.978
  fold 3: so'z 0.762, belgi 0.700, transformer 0.859, pretrain+FT 0.952
  fold 4: so'z 0.784, belgi 0.670, transformer 0.722, pretrain+FT 0.925
  fold 5: so'z 0.683, belgi 0.617, transformer 0.850, pretrain+FT 0.956

=== 2. Natijalar (5 fold o'rtachasi) ===
  model          aniqlik  +-std       oddiy      inkor uzoq inkor   kontrast
  ko'pchilik      0.5119  0.027       0.524      0.487      0.502      0.506
  so'z TF-IDF     0.7520  0.040       0.883      0.699      0.621      0.566
  belgi TF-IDF    0.6623  0.030       0.851      0.303      0.552      0.547
  transformer     0.8504  0.077       0.916      0.683      0.727      0.899
  pretrain+FT     0.9490  0.021       0.967      0.943      0.894      0.948
  eng yuqori o'rtacha: pretrain+FT

=== 3. Juftlashgan farq: eng yaxshisi (pretrain+FT) - model ===
  model             farq    2*SE  sezilarli yomonmi
  ko'pchilik     +0.4371  0.0324  ha
  so'z TF-IDF    +0.1970  0.0437  ha
  belgi TF-IDF   +0.2867  0.0360  ha
  transformer    +0.0986  0.0593  ha

=== 4. Qaror ===
  qoida: eng yaxshisidan SEZILARLI yomon bo'lmagan ENG SODDA model
  soddalik tartibi: ko'pchilik < so'z TF-IDF < belgi TF-IDF < transformer < pretrain+FT
  TANLOV: pretrain+FT
  eng yaxshi bazaviy (so'z TF-IDF) dan ustunlik: +0.1970

"""4-qadam: test bir marta, tur bo'yicha xato tahlili, weights_only paket va hisobot."""

import math
import random
import sys
import tempfile
from dataclasses import asdict, dataclass
from pathlib import Path

import numpy as np
import sklearn
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, GroupShuffleSplit
from sklearn.pipeline import make_pipeline

MAVZU = ["telefon", "noutbuk", "kitob", "kafe", "mehmonxona", "avtobus", "dastur",
         "kurs", "film", "do'kon", "restoran", "poyabzal", "kamera", "sumka", "taksi",
         "klinika", "sartaroshxona", "televizor"]
ASPEKT = ["narxi", "sifati", "xizmati", "dizayni", "yetkazib berish", "xodimlari",
          "joylashuvi", "ovozi"]
IJOBIY = ["a'lo", "ajoyib", "zo'r", "yaxshi", "qulay", "chiroyli", "tez", "sifatli",
          "mazali", "toza", "arzon", "ishonchli", "shinam", "samimiy", "foydali",
          "qiziqarli", "mukammal", "puxta", "yoqimli", "maroqli", "chidamli", "oson",
          "zamonaviy", "beg'ubor"]
SALBIY = ["yomon", "sekin", "qimmat", "iflos", "noqulay", "xunuk", "sifatsiz", "bemaza",
          "eski", "zerikarli", "ishonchsiz", "qo'pol", "buzuq", "chalkash", "nosoz", "xira",
          "tor", "shovqinli", "befoyda", "sovuq", "mo'rt", "murakkab", "dag'al", "g'alati"]
DARAJA = ["juda", "ancha", "biroz", "rosa", "", ""]
OXIR = ["ekan", "edi", "", "chiqdi"]
BOSH = ["kecha", "o'tgan hafta", "shahardagi", "yangi", "onlayn", "bu", "", ""]
DUM = {1: ["hammaga tavsiya qilaman", "yana kelaman", "juda mamnunman", "pulimga arzidi",
           "do'stlarimga aytaman", "rahmat sizlarga"],
       0: ["hech kimga tavsiya qilmayman", "pulimni qaytarib oldim", "boshqa kelmayman",
           "vaqtim behuda ketdi", "shikoyat yozdim", "afsuslandim"]}
TURLAR = ["oddiy", "inkor", "uzoq inkor", "kontrast"]
TUR_ULUSH = [0.5, 0.15, 0.15, 0.2]
SINFLAR = ["salbiy", "ijobiy"]
PAD, CLS, MASK, UNK = 0, 1, 2, 3


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def sharh(rng, tur, dum_ehtimol, bosh=None):
    """Bitta sharh: (matn, tonallik). Qiyin turlar so'z tartibi va inkorga bog'liq."""
    q = int(rng.integers(2))

    def sif(p):
        return tanla(rng, IJOBIY if p else SALBIY)

    if tur == "oddiy":
        s, y = [tanla(rng, MAVZU), tanla(rng, DARAJA), sif(q), tanla(rng, OXIR)], q
    elif tur == "inkor":                                   # "a'lo emas" -> salbiy
        s, y = [tanla(rng, MAVZU), tanla(rng, DARAJA), sif(q), "emas", tanla(rng, OXIR)], 1 - q
    elif tur == "uzoq inkor":                              # inkor fe'lda, sifatdan uzoqda
        tasdiq = rng.random() < 0.5
        fel = tanla(rng, ["deb o'ylayman", "deb hisoblayman", "deya olaman"] if tasdiq
                    else ["deb o'ylamayman", "deb hisoblamayman", "deya olmayman"])
        s, y = [tanla(rng, MAVZU), tanla(rng, ASPEKT), sif(q), fel], q if tasdiq else 1 - q
    else:                                                  # "X yomon, lekin Y a'lo" -> Y hal qiladi
        a1, a2 = rng.choice(len(ASPEKT), 2, replace=False)
        s = [tanla(rng, MAVZU), ASPEKT[a1], sif(1 - q), tanla(rng, ["edi", "", "ekan"]),
             tanla(rng, ["lekin", "ammo", "biroq"]), ASPEKT[a2], tanla(rng, DARAJA), sif(q)]
        y = q
    s = [tanla(rng, BOSH) if bosh is None else bosh] + s
    if rng.random() < dum_ehtimol:
        s.append(tanla(rng, DUM[y]))
    return " ".join(" ".join(s).split()), y


def malumot(seed=0, n_muallif=350, p_takror=0.1, p_shovqin=0.03):
    """Belgili sharhlar: matn, tonallik, tur, muallif. Muallifning o'z uslubi bor."""
    rng = np.random.default_rng(seed)
    matn, y, tur, muallif = [], [], [], []
    for m in range(n_muallif):
        uslub = tanla(rng, BOSH)
        for _ in range(int(rng.geometric(1 / 4))):
            if matn and muallif[-1] == m and rng.random() < p_takror:   # qayta yuborish
                matn.append(matn[-1])
                y.append(y[-1])
                tur.append(tur[-1])
            else:
                t = TURLAR[int(rng.choice(4, p=TUR_ULUSH))]
                s, yorliq = sharh(rng, t, 0.15, uslub if rng.random() < 0.6 else None)
                matn.append(s)
                tur.append(t)
                y.append(1 - yorliq if rng.random() < p_shovqin else yorliq)
            muallif.append(m)
    return np.array(matn, dtype=object), np.array(y), np.array(tur), np.array(muallif)


def korpus(seed=1, n=20000):
    """Belgisiz sharhlar (pretraining uchun): yorliq ham, tur ham saqlanmaydi."""
    rng = np.random.default_rng(seed)
    return [sharh(rng, TURLAR[int(rng.choice(4, p=TUR_ULUSH))], 0.6)[0] for _ in range(n)]


@dataclass(frozen=True)
class Konfig:
    seed: int = 42
    test_ulushi: float = 0.2
    T: int = 24
    d: int = 48
    L: int = 2
    h: int = 4
    pre_qadam: int = 800
    qadam: int = 150
    lr: float = 2e-3
    bs: int = 32
    C: float = 10.0


def seed_everything(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)


def guruhli_bolish(y, guruh, ulush, seed):
    """Muallif bo'yicha: bir muallifning hamma sharhi bitta tomonda."""
    gss = GroupShuffleSplit(n_splits=1, test_size=ulush, random_state=seed)
    return next(gss.split(np.zeros(len(y)), y, guruh))


def soz_tfidf(k):
    return make_pipeline(TfidfVectorizer(token_pattern=r"[^ ]+", ngram_range=(1, 2),
                                         sublinear_tf=True),
                         LogisticRegression(C=k.C, max_iter=3000))


def belgi_tfidf(k):
    return make_pipeline(TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5),
                                         sublinear_tf=True),
                         LogisticRegression(C=k.C, max_iter=3000))


class Lugat:
    """So'z tokenizatori; FAQAT belgisiz korpusdan (pretrained model bilan birga keladi)."""

    def __init__(self, matnlar):
        sozlar = sorted({w for m in matnlar for w in m.split()})
        self.itos = ["[PAD]", "[CLS]", "[MASK]", "[UNK]"] + sozlar
        self.stoi = {w: i for i, w in enumerate(self.itos)}

    def kodla(self, matnlar, T):
        X = torch.zeros(len(matnlar), T, dtype=torch.long)
        for i, m in enumerate(matnlar):
            ids = [CLS] + [self.stoi.get(w, UNK) for w in m.split()][:T - 1]
            X[i, :len(ids)] = torch.tensor(ids)
        return X


class Blok(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x, maska):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).view(B, T, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=maska[:, None, None, :])
        x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class Encoder(nn.Module):
    def __init__(self, V, d, L, h, T):
        super().__init__()
        self.emb, self.poz = nn.Embedding(V, d), nn.Embedding(T, d)
        self.bloklar = nn.ModuleList([Blok(d, h) for _ in range(L)])
        self.ln = nn.LayerNorm(d)
        nn.init.normal_(self.emb.weight, std=0.02)
        nn.init.normal_(self.poz.weight, std=0.02)

    def forward(self, X):
        maska = X != PAD
        x = self.emb(X) + self.poz(torch.arange(X.shape[1]))
        for b in self.bloklar:
            x = b(x, maska)
        return self.ln(x)


class Klassifikator(nn.Module):
    """Encoder + o'rtacha pooling (PAD siz) + chiziqli bosh."""

    def __init__(self, enc):
        super().__init__()
        self.enc = enc
        self.bosh = nn.Linear(enc.emb.embedding_dim, len(SINFLAR))
        nn.init.normal_(self.bosh.weight, std=0.02)       # boshlang'ich loss ~ ln 2
        nn.init.zeros_(self.bosh.bias)

    def forward(self, X):
        h = self.enc(X)
        m = (X != PAD).float()[..., None]
        return self.bosh((h * m).sum(1) / m.sum(1))


def yangi_encoder(k, V):
    return Encoder(V, k.d, k.L, k.h, k.T)


def kes(X):
    return X[:, :int((X != PAD).sum(1).max())]


def maskala(X, g, V, p=0.15):
    Y = X.clone()
    tanl = (torch.rand(X.shape, generator=g) < p) & (X > UNK)
    Y[~tanl] = -100
    r = torch.rand(X.shape, generator=g)
    X = X.clone()
    X[tanl & (r < 0.8)] = MASK
    tas = tanl & (r >= 0.8) & (r < 0.9)
    X[tas] = torch.randint(4, V, (int(tas.sum()),), generator=g)
    return X, Y


def pretrain(k, V, X, seed=0):
    """MLM pretraining (24.8, 24.10): bog'langan vaznlar, 15% maskalash."""
    torch.manual_seed(seed)
    enc = yangi_encoder(k, V)
    bias = nn.Parameter(torch.zeros(V))
    opt = torch.optim.AdamW(list(enc.parameters()) + [bias], lr=k.lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(k.pre_qadam):
        xb, yb = maskala(kes(X[torch.randint(0, len(X), (k.bs,), generator=g)]), g, V)
        loss = F.cross_entropy((enc(xb) @ enc.emb.weight.T + bias).reshape(-1, V), yb.reshape(-1))
        opt.zero_grad()
        loss.backward()
        opt.step()
    return enc, loss.item()


def orgat(k, enc, X, y, seed):
    """Fine-tuning yoki noldan o'rgatish: bir xil sikl, bir xil qadamlar."""
    torch.manual_seed(seed)
    model = Klassifikator(enc)
    opt = torch.optim.AdamW(model.parameters(), lr=k.lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    y = torch.as_tensor(y)
    for _ in range(k.qadam):
        idx = torch.randint(0, len(X), (k.bs,), generator=g)
        loss = F.cross_entropy(model(kes(X[idx])), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()
    return model


@torch.no_grad()
def bashorat(model, X):
    return model(kes(X)).argmax(1).numpy()


def tur_aniqligi(y, p, tur):
    return {t: float(np.mean(p[tur == t] == y[tur == t])) for t in TURLAR}


FORMAT = 1
TANLOV = "pretrain+FT"                   # 3-qadam qarori


class Bashoratchi:
    """Paketdan (weights_only=True) tokenizator va modelni tiklaydi, nazoratni tekshiradi."""

    def __init__(self, yol):
        p = torch.load(yol, weights_only=True)
        if p["format"] != FORMAT:
            raise ValueError("format mos emas")
        self.k = Konfig(**p["konfig"]["model"])
        self.stoi = {w: i for i, w in enumerate(p["lugat"])}
        self.sinflar = p["sinflar"]
        self.model = Klassifikator(yangi_encoder(self.k, len(p["lugat"])))
        self.model.load_state_dict(p["holat"])
        self.model.eval()
        q = self.logit(p["nazorat"]["matn"])
        self.nazorat_ok = bool(torch.allclose(q, p["nazorat"]["logit"], atol=1e-5))

    def kodla(self, matnlar):
        X = torch.zeros(len(matnlar), self.k.T, dtype=torch.long)
        for i, m in enumerate(matnlar):
            ids = [CLS] + [self.stoi.get(w, UNK) for w in m.lower().split()][:self.k.T - 1]
            X[i, :len(ids)] = torch.tensor(ids)
        return X

    @torch.no_grad()
    def logit(self, matnlar):
        return self.model(kes(self.kodla(matnlar)))

    def bashorat(self, matnlar):
        p = torch.softmax(self.logit(matnlar), 1)
        return [(self.sinflar[int(i)], float(v)) for v, i in zip(*p.max(1))]


def main() -> None:
    torch.set_num_threads(1)
    k = Konfig()
    matn, y, tur, muallif = malumot(seed=0)
    ish, te = guruhli_bolish(y, muallif, k.test_ulushi, k.seed)
    kor = korpus()
    lug = Lugat(kor)
    V = len(lug.itos)

    print(f"=== 1. Yakuniy model (3-qadam qarori: {TANLOV}) ish to'plamida ===")
    enc, mlm = pretrain(k, V, lug.kodla(kor, k.T))
    model = orgat(k, enc, lug.kodla(list(matn[ish]), k.T), y[ish], k.seed)
    print(f"  pretraining {k.pre_qadam} qadam (MLM loss {mlm:.3f}), "
          f"fine-tuning {k.qadam} qadam, o'quv {len(ish)} sharh")

    print("\n=== 2. Testni BIR MARTA ochamiz ===")
    Xt = lug.kodla(list(matn[te]), k.T)
    p = bashorat(model, Xt)
    yt, tt = y[te], tur[te]
    aniq = float(np.mean(p == yt))
    print(f"  test aniqligi {aniq:.4f} ({int(np.sum(p != yt))} xato / {len(te)})")
    for s, nom in enumerate(SINFLAR):
        rec = np.mean(p[yt == s] == s)
        prec = np.mean(yt[p == s] == s)
        print(f"  {nom:<7} recall {rec:.3f}  precision {prec:.3f}")

    print("\n=== 3. Xato tahlili: tur bo'yicha ===")
    xato = p != yt
    print(f"  {'tur':<11} {'n':>4} {'xato':>6} {'2*SE':>6}   {'qolganlar xatosi':>16}")
    xavfli = []
    for t in TURLAR:
        b = tt == t
        e1, e0 = xato[b].mean(), xato[~b].mean()
        se = np.sqrt(e1 * (1 - e1) / b.sum() + e0 * (1 - e0) / (~b).sum())
        print(f"  {t:<11} {b.sum():>4} {e1:>6.3f} {2 * se:>6.3f}   {e0:>16.3f}")
        if e1 - e0 > 2 * se:
            xavfli.append(t)
    print(f"  sezilarli xavfli turlar: {', '.join(xavfli) or 'topilmadi'}")
    print("  xato qilingan sharhlar (birinchi 5 ta):")
    for i in np.flatnonzero(xato)[:5]:
        print(f"    [{tt[i]}] {matn[te][i]}  -> {SINFLAR[p[i]]} (to'g'ri: {SINFLAR[yt[i]]})")

    with tempfile.TemporaryDirectory() as yol_papka:
        print("\n=== 4. Topshirish paketi (weights_only) ===")
        nazorat = [str(s) for s in matn[te[:8]]]
        with torch.no_grad():
            nazorat_logit = model(kes(lug.kodla(nazorat, k.T)))
        paket = {
            "format": FORMAT,
            "konfig": {"model": asdict(k), "tanlov": TANLOV, "pooling": "o'rtacha, PAD siz"},
            "lugat": list(lug.itos),
            "holat": model.state_dict(),
            "sinflar": list(SINFLAR),
            "metrika": {"test_aniqlik": round(aniq, 4),
                        "tur_aniqligi": {t: round(v, 4)
                                         for t, v in tur_aniqligi(yt, p, tt).items()}},
            "versiyalar": {"torch": str(torch.__version__), "sklearn": sklearn.__version__,
                           "python": sys.version.split()[0]},
            "nazorat": {"matn": nazorat, "logit": nazorat_logit},
        }
        yol = Path(yol_papka) / "sharh_tonallik.pt"
        torch.save(paket, yol)
        print(f"  kalitlar: {sorted(paket)}")
        print(f"  holat: {len(paket['holat'])} tenzor, "
              f"{sum(v.numel() for v in paket['holat'].values())} parametr; "
              f"lug'at {len(paket['lugat'])}")

        print("\n=== 5. Bashoratchi (torch.load(..., weights_only=True)) ===")
        b = Bashoratchi(yol)
        print(f"  nazorat (saqlangan logitlar bilan): {'OK' if b.nazorat_ok else 'XATO'}")
        yangi = ["Telefon juda a'lo ekan hammaga tavsiya qilaman",
                 "kafe xizmati samimiy emas edi",
                 "mehmonxona narxi qimmat ekan lekin xodimlari juda samimiy",
                 "kitob dizayni chiroyli deb o'ylamayman",
                 "taksi rosa sekin chiqdi"]
        for m, (s, e) in zip(yangi, b.bashorat(yangi)):
            print(f"  {m:<58} -> {s:<7} ({e:.2f})")

    print("\n=== 6. Yakuniy hisobot ===")
    ta = tur_aniqligi(yt, p, tt)
    print("  VAZIFA: o'zbekcha sharhlar tonalligi (ijobiy / salbiy)")
    print("  METRIKA: aniqlik (sinflar muvozanatli) + qiyin turlar bo'yicha aniqlik")
    print("  DIZAYN: muallif bo'yicha bo'lish; tokenizator belgisiz korpusdan; test bir marta")
    print(f"  QAROR: GroupKFold juftlashgan taqqoslash -> {TANLOV}")
    print(f"  TEST: aniqlik {aniq:.4f}; " + ", ".join(f"{t} {v:.3f}" for t, v in ta.items()))
    eng_past = min(ta, key=ta.get)
    print(f"  ZAIF JOY: {eng_past} ({ta[eng_past]:.3f}); xavfli turlar: "
          f"{', '.join(xavfli) or 'yo`q'}".replace("`", "'"))
    print(f"  PAKET: format {FORMAT}, weights_only=True, nazorat "
          f"{'OK' if b.nazorat_ok else 'XATO'}")
    print("  CHEKLOVLAR: sintetik sharhlar, yopiq lug'at (yangi so'z -> [UNK]),")
    print("    sarkazm va aralash tonallik tekshirilmagan")


if __name__ == "__main__":
    main()

=== 1. Yakuniy model (3-qadam qarori: pretrain+FT) ish to'plamida ===
  pretraining 800 qadam (MLM loss 2.139), fine-tuning 150 qadam, o'quv 1137 sharh

=== 2. Testni BIR MARTA ochamiz ===
  test aniqligi 0.9364 (18 xato / 283)
  salbiy  recall 0.898  precision 0.969
  ijobiy  recall 0.973  precision 0.910

=== 3. Xato tahlili: tur bo'yicha ===
  tur            n   xato   2*SE   qolganlar xatosi
  oddiy        137  0.007  0.055              0.116
  inkor         47  0.191  0.117              0.038
  uzoq inkor    41  0.195  0.126              0.041
  kontrast      58  0.000  0.036              0.080
  sezilarli xavfli turlar: inkor, uzoq inkor
  xato qilingan sharhlar (birinchi 5 ta):
    [uzoq inkor] onlayn sumka joylashuvi toza deya olmayman  -> ijobiy (to'g'ri: salbiy)
    [uzoq inkor] kitob narxi foydali deb o'ylamayman  -> ijobiy (to'g'ri: salbiy)
    [uzoq inkor] kitob narxi foydali deb o'ylamayman  -> ijobiy (to'g'ri: salbiy)
    [oddiy] shahardagi poyabzal ancha sifatli pulimga arzidi  -> ijobiy (to'g'ri: salbiy)
    [uzoq inkor] o'tgan hafta taksi dizayni oson deb o'ylamayman  -> ijobiy (to'g'ri: salbiy)

=== 4. Topshirish paketi (weights_only) ===
  kalitlar: ['format', 'holat', 'konfig', 'lugat', 'metrika', 'nazorat', 'sinflar', 'versiyalar']
  holat: 30 tenzor, 64178 parametr; lug'at 131

=== 5. Bashoratchi (torch.load(..., weights_only=True)) ===
  nazorat (saqlangan logitlar bilan): OK
  Telefon juda a'lo ekan hammaga tavsiya qilaman             -> ijobiy  1.00-bob
  kafe xizmati samimiy emas edi                              -> salbiy  0.85-bob
  mehmonxona narxi qimmat ekan lekin xodimlari juda samimiy  -> ijobiy  0.99-bob
  kitob dizayni chiroyli deb o'ylamayman                     -> ijobiy  0.54-bob
  taksi rosa sekin chiqdi                                    -> salbiy  1.00-bob

=== 6. Yakuniy hisobot ===
  VAZIFA: o'zbekcha sharhlar tonalligi (ijobiy / salbiy)
  METRIKA: aniqlik (sinflar muvozanatli) + qiyin turlar bo'yicha aniqlik
  DIZAYN: muallif bo'yicha bo'lish; tokenizator belgisiz korpusdan; test bir marta
  QAROR: GroupKFold juftlashgan taqqoslash -> pretrain+FT
  TEST: aniqlik 0.9364; oddiy 0.993, inkor 0.809, uzoq inkor 0.805, kontrast 1.000
  ZAIF JOY: uzoq inkor 0.805-bob; xavfli turlar: inkor, uzoq inkor
  PAKET: format 1, weights_only=True, nazorat OK
  CHEKLOVLAR: sintetik sharhlar, yopiq lug'at (yangi so'z -> [UNK]),
    sarkazm va aralash tonallik tekshirilmagan

train_test_split(matn, y, test_size=0.2)                    # ⚠️ bir muallif ikki tomonda
guruhli_bolish(y, muallif, 0.2, seed)                       # ✅ GroupShuffleSplit

lug = Lugat(matn[tr]); enc.load_state_dict(pretrained)      # ⚠️ indekslar mos emas
lug = Lugat(kor)             # pretraining bilan BIR XIL lug'at # ✅

orgat(k_300_qadam, yangi_encoder(k, V), ...)                # ⚠️ noldan - 300 qadam,
orgat(k_150_qadam, copy.deepcopy(enc_pre), ...)             #    pretrained - 150
orgat(k, yangi_encoder(k, V), X[tr], yy[tr], f)             # ✅ bir xil k, bir xil urug'
orgat(k, copy.deepcopy(enc_pre), X[tr], yy[tr], f)

model = orgat(k, enc_pre, X[tr], yy[tr], f)                 # ⚠️ enc_pre har foldda o'zgaradi
model = orgat(k, copy.deepcopy(enc_pre), X[tr], yy[tr], f)  # ✅

print(f"aniqlik {np.mean(p == yt):.3f}")                    # ⚠️ inkordagi xato ko'rinmaydi
print(tur_aniqligi(yt, p, tt))                              # ✅ tur bo'yicha

self.bosh = nn.Linear(d, 2)                                 # ⚠️ boshlang'ich loss 0.989
nn.init.normal_(self.bosh.weight, std=0.02)                 # ✅ ~ ln 2
nn.init.zeros_(self.bosh.bias)

torch.save(model.state_dict(), yol)                         # ⚠️ so'z -> indeks yo'qoldi
torch.save({"konfig": asdict(k), "lugat": lug.itos,         # ✅
            "holat": model.state_dict(), "nazorat": ...}, yol)

1.  # ikki sinfda boshlang'ich loss taxminan?
2.  # nega "kafe narxi yomon lekin sifati a'lo" ni so'z xaltasi ajrata olmaydi?
3.  # char_wb n-gramlari "a'lo emas" ni qanday ko'radi?
4.  # nega bo'lish muallif bo'yicha?
5.  # tokenizator qaysi matnlardan quriladi va nega?
6.  # pretraining nega foldlardan tashqarida bajariladi?
7.  # noldan va pretrained modelni solishtirishda nima bir xil bo'lishi kerak?
8.  # juftlashgan taqqoslashda "sezilarli yomon" mezoni?
9.  # eng yaxshi va soddaroq model farqi 2*SE ichida - qaysi biri tanlanadi?
10. # test CV dan 0.013 past - tashvishlanish kerakmi?
11. # paketda qaysi to'rtta narsa bo'lishi shart?
12. # nazorat namunasi qaysi ko'rinishda saqlanadi?

1.  tr, te = train_test_split(np.arange(len(y)), test_size=0.2, stratify=y)

2.  lug = Lugat(list(matn[ish]))
    enc = pretrain(k, len(lug.itos), lug.kodla(kor, k.T))[0]

3.  for f, (tr, va) in enumerate(GroupKFold(5).split(m, yy, mj), 1):
        enc_pre, _ = pretrain(k, V, lug.kodla(kor, k.T))
        model = orgat(k, enc_pre, X[tr], yy[tr], f)

4.  print(f"test aniqligi {np.mean(p == yt):.3f}")   # hisobotdagi yagona qator

5.  torch.save({"holat": model.state_dict(), "torch": torch.__version__}, yol)