24.10-dars: Pretraining va fine-tuning

1-BOSQICH - PRETRAINING (belgisiz matn, ko'p):
  korpus: 20000 sharh, yorliq YO'Q
  vazifa: o'z-o'zini nazorat (self-supervised) - yorliq matnning o'zidan
    MLM (BERT):  "telefon juda [MASK] ekan" -> "a'lo"
    CLM (GPT):   "telefon juda a'lo" -> keyingi so'z
  natija: encoder vaznlari = tilning "umumiy bilimi"

2-BOSQICH - FINE-TUNING (belgili matn, kam):
  50 / 200 / 1000 sharh + yorliq (ijobiy/salbiy)
  encoder = pretrained vaznlar;  ustiga YANGI bosh (Linear(d, 2), tasodifiy)
  hammasi (yoki bir qismi) kichik lr bilan o'rgatiladi

NIMA UCHUN ISHLAYDI:
  belgili 200 misol 48 ta sifatning faqat bir qismini ko'rsatadi
  korpus esa "beg'ubor" va "a'lo" bir xil kontekstda kelishini ko'rsatgan
  -> ularning vakillari yaqin -> bosh bittasida o'rgangan narsa ikkinchisiga o'tadi

TRANSFER O'RGANISH (22-qismda rasmlar bilan ko'rgan g'oya):
  ImageNet da o'rgatilgan CNN -> 200 ta rasm bilan yangi sinflar
  bu darsda:  sharhlar korpusida MLM -> 200 ta sharh bilan tonallik

MASKALASH (BERT retsepti, maskala() funksiyasi):
  har tokenni 15% ehtimol bilan TANLASH ([PAD], [CLS] tanlanmaydi)
  tanlanganlarning:
    80% -> [MASK]                  "telefon juda [MASK] ekan"
    10% -> tasodifiy so'z          "telefon juda kitob ekan"
    10% -> o'zgarmaydi             "telefon juda a'lo ekan"
  nishon Y: tanlangan joyda asl token, qolgan joyda -100 (ignore_index)

NEGA 80/10/10:
  faqat [MASK] bo'lsa - fine-tuning da [MASK] hech qachon uchramaydi,
  model "[MASK] bo'lmagan joyda o'ylamasa ham bo'ladi" deb o'rganadi
  tasodifiy/o'zgarmagan tokenlar: model HAR pozitsiyani tekshirishga majbur

LOSS: faqat tanlangan pozitsiyalarda cross-entropy
  boshlang'ich qiymat ~ ln V (tekis taqsimot):  V = 110 -> ln 110 = 4.700

BOG'LANGAN VAZNLAR (weight tying):
  logit = h @ E^T + b     (E - kirish embedding matritsasi, V x d)
  alohida Linear(d, V) yo'q -> parametr kam, embedding ikki tomondan o'rganadi
  MLMBosh da faqat bias (110 parametr) - 1-misol

1. SO'Z VAKILLARI (embeddinglar):
   bir xil kontekstda keladigan so'zlar yaqinlashadi (23.6-darsdagi word2vec g'oyasi)
   1-misol, kosinus o'xshashlik o'rtachasi:
                          ijobiy-ijobiy   ijobiy-salbiy
     tasodifiy init          -0.016          +0.006
     pretraining dan keyin   +0.815          +0.563
   'a'lo' ga eng yaqin:  shinam, qulay, toza, mazali    (hammasi ijobiy)
   'iflos' ga eng yaqin: xira, shovqinli, qo'pol, mo'rt (hammasi salbiy)

2. KONTEKST MANTIG'I (attention qatlamlari):
   "kafe juda [MASK] edi boshqa kelmayman"       P(ijobiy) 0.40  P(salbiy) 0.59
   "kafe juda [MASK] emas edi boshqa kelmayman"  P(ijobiy) 0.61  P(salbiy) 0.37
   'emas' qo'shilganda taqsimot teskari tomonga siljidi:
   model inkor + salbiy dum -> ijobiy sifat ekanini korpusdan o'rgangan

3. NIMANI O'RGANA OLMAYDI:
   korpusda YO'Q bog'lanishni
   "kecha [MASK] juda mazali edi" -> klinika, sartaroshxona, kafe (har biri 0.08)
   sintetik korpusda mavzu va sifat mustaqil tanlangan - "mazali" ovqatga
   bog'liqligini model bilmaydi. Real korpusda bu bog'lanish bor bo'lardi.

TAJRIBA DIZAYNI (2-misol):
  belgili hovuz: 3000 sharh, faqat 32/48 sifat bilan (16 ijobiy + 16 salbiy)
  test: 2000 sharh, 48 sifatning hammasi -> 33.0% jumlada ko'rilmagan sifat
  n = 50, 200, 1000;  3 urug' (s = 0, 1, 2)
  BIR XIL: urug' -> bir xil belgili misollar, bir xil bosh init, bir xil batchlar
           qadamlar 100, lr 1e-3, batch 16 - ikkala model uchun
  JUFTLASHGAN FARQ: d_s = aniq_pretrained,s - aniq_noldan,s
                    SE = std(d) / sqrt(3);  |mean(d)| > 2*SE -> sezilarli

NATIJA (test aniqligi):
     n    noldan   pretrained    farq     SE
    50     0.536      0.810     +0.274   0.018
   200     0.611      0.915     +0.303   0.031
  1000     0.757      0.997     +0.240   0.023

KO'RILGAN va KO'RILMAGAN sifatlar bo'yicha foyda:
  ko'rilgan:    n=50 +0.252,  n=200 +0.247,  n=1000 +0.086  -> kamayadi
  ko'rilmagan:  n=50 +0.319,  n=200 +0.418,  n=1000 +0.554  -> kamaymaydi
  noldan model ko'rilmagan sifatlarda n=1000 da ham 0.438 (tasodifdan past!)

                         o'rgatiladi             lr      3-misol (n=200)
  noldan                 hammasi (tasodifiy)     1e-3    0.611
  faqat bosh (probe)     Linear(48, 2) = 98      1e-2    0.761   (0.2%)
  yuqori blok + bosh     2-blok + ln + bosh      1e-3    0.822   (45.3%)
  to'liq FT              hammasi (pretrained)    1e-3    0.915   (100%)

FAQAT BOSH (linear probe):
  encoder muzlatilgan (requires_grad=False), faqat bosh o'rganadi
  vakillar "qanchalik tayyor" ekanini o'lchaydi; eng arzon, unutish yo'q
  bosh kichik - lr kattaroq (1e-2)

MUZLATISH (pastki qatlamlar):
  pastki qatlamlar - umumiy xususiyatlar (so'z ma'nosi)
  yuqori qatlamlar - vazifaga yaqin xususiyatlar
  muzlatilgan qism: embedding, pozitsiya, 1-blok

TO'LIQ FINE-TUNING:
  hamma vaznlar, KICHIK lr (pretraining dagidan kichik yoki teng)
  eng kuchli, lekin eng ko'p unutadi (2.6)

HODISA: fine-tuning encoder vaznlarini vazifa uchun o'zgartiradi
  -> pretraining da o'rganilgan boshqa bilim (MLM) buziladi
O'LCHASH: fine-tuning dan keyingi encoder + ASL MLM boshi -> nazorat MLM loss

4-MISOL (n=1000, pretraining dan keyin MLM loss 1.665):
  usul                          test aniqlik   MLM loss   o'sish
  faqat bosh, lr 1e-2               0.743        1.665     +0.000
  to'liq, lr 3e-4                   0.808        1.759     +0.094
  to'liq, lr 1e-3                   0.999        2.367     +0.702
  to'liq, lr 3e-3, 300 qadam        1.000        3.669     +2.004

NIMA QILISH MUMKIN:
  kichik lr, kam qadam            - kam unutadi, lekin vazifani ham kam o'rganadi
  muzlatish / faqat bosh          - unutish yo'q, sifat past 2.5-bob
  LoRA kabi adapterlar            - asl vaznlar o'zgarmaydi (keyingi dars)
  aralash o'rgatish               - fine-tuning loss + kichik vazn bilan MLM loss

# Hugging Face bilan odatiy fine-tuning (KONTSEPTUAL, bu kursda ishga tushirilmaydi)
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import Trainer, TrainingArguments

nom = "tashkilot/uzbek-bert-base"                  # Hub dagi model nomi
tok = AutoTokenizer.from_pretrained(nom)           # lug'at + tokenizatsiya qoidalari
model = AutoModelForSequenceClassification.from_pretrained(nom, num_labels=2)
#   encoder vaznlari yuklanadi, klassifikatsiya boshi TASODIFIY yaratiladi

kirish = tok(["telefon juda a'lo ekan"], padding=True, truncation=True,
             return_tensors="pt")                  # input_ids, attention_mask
args = TrainingArguments(output_dir="natija", learning_rate=2e-5,
                         num_train_epochs=3, per_device_train_batch_size=16)
Trainer(model=model, args=args, train_dataset=oquv, eval_dataset=val).train()
model.save_pretrained("mening_modelim")            # config.json + model.safetensors
tok.save_pretrained("mening_modelim")              # tokenizer fayllari

import torch
import torch.nn.functional as F

# MLM maskalash: 15% tanlash, 80% [MASK], 10% tasodifiy, 10% o'zi
Xm, Y = maskala(X, g, V)                           # Y: tanlanmaganlar -100
logit = enc(Xm) @ enc.emb.weight.T + bias          # bog'langan vaznlar
loss = F.cross_entropy(logit.reshape(-1, V), Y.reshape(-1))   # ignore_index=-100

# fine-tuning: pretrained encoder + yangi bosh
enc = Encoder(V); enc.load_state_dict(pretrained_holat)
model = Klassifikator(enc)                         # bosh - tasodifiy
for p in model.enc.parameters():                   # faqat bosh (linear probe)
    p.requires_grad_(False)
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-2)

# halol taqqoslash: bir urug' - bir xil misollar, bir xil init, bir xil batchlar
d = aniq_pretrained - aniq_noldan                  # urug'lar bo'yicha massiv
sezilarli = abs(d.mean()) > 2 * d.std(ddof=1) / len(d) ** 0.5

# unutishni o'lchash: fine-tuned encoder + ASL MLM boshi
L_keyin = mlm_loss(model.enc, mlm_bosh, Xn, Yn)    # qat'iy maskalar bilan

# saqlash (from_pretrained g'oyasi)
torch.save({"konfig": {...}, "lugat": itos, "encoder": enc.state_dict()}, yol)
p = torch.load(yol, weights_only=True)

pretraining: belgisiz korpus + MLM (15%, 80/10/10) -> til bilimi
fine-tuning: pretrained encoder + yangi bosh + kichik lr -> vazifa
foyda: kam yorliq va yangi so'zlarda eng katta; juftlashgan urug'lar bilan o'lchang
usullar: faqat bosh < muzlatish < to'liq (sifat), teskari tartibda (unutish)
katastrofik unutish: lr va qadamlar bilan o'sadi; pretrained vaznlarni saqlang
HF: AutoTokenizer + AutoModel... + Trainer + Hub + model kartasi

"""MLM pretraining: belgisiz sharhlar korpusida maskalangan so'zlarni tiklash."""

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

MAVZU = ["telefon", "noutbuk", "kitob", "kafe", "mehmonxona", "avtobus", "dastur",
         "kurs", "film", "do'kon", "restoran", "poyabzal", "kamera", "sumka", "taksi",
         "klinika", "sartaroshxona", "televizor"]
IJOBIY = ["a'lo", "ajoyib", "zo'r", "yaxshi", "qulay", "chiroyli", "tez", "sifatli",
          "mazali", "toza", "arzon", "ishonchli", "shinam", "samimiy", "foydali",
          "qiziqarli", "mukammal", "puxta", "yoqimli", "maroqli", "chidamli", "oson",
          "zamonaviy", "beg'ubor"]
SALBIY = ["yomon", "sekin", "qimmat", "iflos", "noqulay", "xunuk", "sifatsiz", "bemaza",
          "eski", "zerikarli", "ishonchsiz", "qo'pol", "buzuq", "chalkash", "nosoz", "xira",
          "tor", "shovqinli", "befoyda", "sovuq", "mo'rt", "murakkab", "dag'al", "g'alati"]
DARAJA = ["juda", "ancha", "biroz", "rosa", "", ""]
OXIR = ["ekan", "edi", "", "chiqdi"]
BOSH = ["kecha", "o'tgan hafta", "shahardagi", "yangi", "onlayn", "", "", ""]
DUM = {1: ["hammaga tavsiya qilaman", "yana kelaman", "juda mamnunman", "pulimga arzidi",
           "do'stlarimga aytaman", "rahmat sizlarga"],
       0: ["hech kimga tavsiya qilmayman", "pulimni qaytarib oldim", "boshqa kelmayman",
           "vaqtim behuda ketdi", "shikoyat yozdim", "afsuslandim"]}
PAD, CLS, MASK, UNK = 0, 1, 2, 3


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def gap(rng, dum_ehtimol, sifatlar=(IJOBIY, SALBIY)):
    """Sharh: (matn, tonallik, sifat). 25% holatda 'emas' tonallikni teskari qiladi."""
    q = int(rng.integers(2))
    sif = tanla(rng, sifatlar[0] if q else sifatlar[1])
    inkor = rng.random() < 0.25
    y = 1 - q if inkor else q
    s = [tanla(rng, BOSH), tanla(rng, MAVZU), tanla(rng, DARAJA), sif,
         "emas" if inkor else "", tanla(rng, OXIR)]
    if rng.random() < dum_ehtimol:
        s.append(tanla(rng, DUM[y]))
    return " ".join(" ".join(s).split()), y, sif


class Lugat:
    """So'z darajasidagi tokenizator: korpusdan quriladi, 4 ta maxsus token."""

    def __init__(self, matnlar):
        sozlar = sorted({w for m in matnlar for w in m.split()})
        self.itos = ["[PAD]", "[CLS]", "[MASK]", "[UNK]"] + sozlar
        self.stoi = {w: i for i, w in enumerate(self.itos)}

    def kodla(self, matnlar, T=16):
        X = torch.zeros(len(matnlar), T, dtype=torch.long)
        for i, m in enumerate(matnlar):
            ids = [CLS] + [self.stoi.get(w, UNK) for w in m.split()][:T - 1]
            X[i, :len(ids)] = torch.tensor(ids)
        return X


class Blok(nn.Module):
    """Pre-LN transformer bloki: self-attention + FFN, qoldiq ulanishlar."""

    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x, maska):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).view(B, T, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=maska[:, None, None, :])
        x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class Encoder(nn.Module):
    def __init__(self, V, d=48, L=2, h=4, T=16):
        super().__init__()
        self.emb, self.poz = nn.Embedding(V, d), nn.Embedding(T, d)
        self.bloklar = nn.ModuleList([Blok(d, h) for _ in range(L)])
        self.ln = nn.LayerNorm(d)
        nn.init.normal_(self.emb.weight, std=0.02)
        nn.init.normal_(self.poz.weight, std=0.02)

    def forward(self, X):
        maska = X != PAD
        x = self.emb(X) + self.poz(torch.arange(X.shape[1]))
        for b in self.bloklar:
            x = b(x, maska)
        return self.ln(x)


class MLMBosh(nn.Module):
    """MLM boshi: vazni embedding bilan bog'langan (weight tying), faqat bias o'rganiladi."""

    def __init__(self, V):
        super().__init__()
        self.bias = nn.Parameter(torch.zeros(V))

    def forward(self, enc, X):
        return enc(X) @ enc.emb.weight.T + self.bias


def kes(X):
    """Batchdagi eng uzun jumlagacha qirqish (ortiqcha PAD ustunlari kerak emas)."""
    return X[:, :int((X != PAD).sum(1).max())]


def maskala(X, g, V, p=0.15):
    """BERT uslubi: 15% token tanlanadi; ulardan 80% [MASK], 10% tasodifiy, 10% o'zi."""
    Y = X.clone()
    tanl = (torch.rand(X.shape, generator=g) < p) & (X > UNK)
    Y[~tanl] = -100
    r = torch.rand(X.shape, generator=g)
    X = X.clone()
    X[tanl & (r < 0.8)] = MASK
    tas = tanl & (r >= 0.8) & (r < 0.9)
    X[tas] = torch.randint(4, V, (int(tas.sum()),), generator=g)
    return X, Y


def mlm_loss(enc, bosh, X, Y):
    V = enc.emb.num_embeddings
    return F.cross_entropy(bosh(enc, X).reshape(-1, V), Y.reshape(-1))


def pretrain(V, X, qadam, seed=0, bs=32, lr=2e-3, kuzat=None):
    """MLM pretraining: belgisiz korpusda maskalangan so'zlarni tiklash."""
    torch.manual_seed(seed)
    enc, bosh = Encoder(V), MLMBosh(V)
    opt = torch.optim.AdamW(list(enc.parameters()) + list(bosh.parameters()),
                            lr=lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for q in range(1, qadam + 1):
        xb, yb = maskala(kes(X[torch.randint(0, len(X), (bs,), generator=g)]), g, V)
        loss = mlm_loss(enc, bosh, xb, yb)
        opt.zero_grad()
        loss.backward()
        opt.step()
        if kuzat is not None:
            kuzat(q, loss.item())
    enc.eval()
    return enc, bosh


class Klassifikator(nn.Module):
    """Encoder + o'rtacha pooling (PAD siz) + chiziqli bosh."""

    def __init__(self, enc, sinflar=2):
        super().__init__()
        self.enc = enc
        self.bosh = nn.Linear(enc.emb.embedding_dim, sinflar)

    def forward(self, X):
        h = self.enc(X)
        m = (X != PAD).float()[..., None]
        return self.bosh((h * m).sum(1) / m.sum(1))


def fine_tune(model, X, y, seed, qadam=100, lr=1e-3, bs=16):
    """Faqat requires_grad=True parametrlar yangilanadi."""
    opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad],
                            lr=lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    model.train()
    for _ in range(qadam):
        idx = torch.randint(0, len(X), (min(bs, len(X)),), generator=g)
        loss = F.cross_entropy(model(kes(X[idx])), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()
    return model


@torch.no_grad()
def bashorat(model, X):
    return model(kes(X)).argmax(1)


def main() -> None:
    torch.set_num_threads(1)
    rng = np.random.default_rng(0)
    korpus = [gap(rng, 0.7)[0] for _ in range(20000)]
    lug = Lugat(korpus)
    V = len(lug.itos)
    X = lug.kodla(korpus)
    X_oquv, X_nazorat = X[:19000], X[19000:]

    print("=== 1. Belgisiz korpus (pretraining uchun) ===")
    for m in korpus[:3]:
        print(f"  {m}")
    uz = (X != PAD).sum(1).float()
    print(f"  jumlalar {len(korpus)}, lug'at {V} (4 maxsus + {V - 4} so'z), "
          f"o'rtacha uzunlik {uz.mean():.2f} token ([CLS] bilan)")
    print("  yorliq YO'Q: model faqat matnning o'zidan o'rganadi")

    print("\n=== 2. Maskalash (80/10/10) ===")
    g = torch.Generator().manual_seed(1)
    Xm, Ym = maskala(X_nazorat, g, V)
    tanl = Ym != -100
    haqiqiy = (X_nazorat > UNK).sum().item()
    print(f"  tanlangan tokenlar: {tanl.sum().item()} / {haqiqiy} = "
          f"{tanl.sum().item() / haqiqiy:.3f}")
    print(f"    [MASK] ga almashdi: {(Xm[tanl] == MASK).float().mean():.3f}")
    print(f"    tasodifiy so'z:     {((Xm[tanl] != MASK) & (Xm[tanl] != Ym[tanl])).float().mean():.3f}")
    print(f"    o'zgarmadi:         {(Xm[tanl] == Ym[tanl]).float().mean():.3f}")
    kir = " ".join(lug.itos[i] for i in Xm[0].tolist() if i != PAD)
    print(f"  misol kirish: {kir}")

    print("\n=== 3. MLM pretraining (2000 qadam, batch 32) ===")
    jurnal = []

    def kuzat(q, loss):
        jurnal.append(loss)
        if q in (1, 500, 1000, 1500, 2000):
            print(f"  qadam {q:>4}: o'quv loss (oxirgi 50 o'rtacha) {np.mean(jurnal[-50:]):.3f}")

    torch.manual_seed(0)
    enc0, bosh0 = Encoder(V), MLMBosh(V)
    with torch.no_grad():
        boshl = mlm_loss(enc0, bosh0, Xm, Ym).item()
    enc, bosh = pretrain(V, X_oquv, 2000, kuzat=kuzat)
    with torch.no_grad():
        oxir = mlm_loss(enc, bosh, Xm, Ym).item()
    print(f"  nazorat MLM loss: boshida {boshl:.3f} (ln V = {np.log(V):.3f}) -> {oxir:.3f}")
    print(f"  parametrlar: encoder {sum(p.numel() for p in enc.parameters())}, "
          f"MLM boshi {sum(p.numel() for p in bosh.parameters())} (faqat bias)")

    print("\n=== 4. Model bo'shliqni to'ldiradi (top-3) ===")
    ij = torch.tensor([lug.stoi[w] for w in IJOBIY])
    sa = torch.tensor([lug.stoi[w] for w in SALBIY])
    jumlalar = ["telefon juda [MASK] ekan hammaga tavsiya qilaman",
                "kafe juda [MASK] edi boshqa kelmayman",
                "kafe juda [MASK] emas edi boshqa kelmayman",
                "kecha [MASK] juda mazali edi"]
    for j in jumlalar:
        Xj = lug.kodla([j])[:, :len(j.split()) + 1]
        with torch.no_grad():
            p = torch.softmax(bosh(enc, Xj)[0, (Xj[0] == MASK).nonzero()[0, 0]], 0)
        top = p.topk(3)
        pij = p[ij].sum().item()
        psa = p[sa].sum().item()
        print(f"  {j}")
        print("     -> " + ", ".join(f"{lug.itos[i]} {v:.2f}"
                                    for v, i in zip(top.values.tolist(), top.indices.tolist()))
              + f"   | P(ijobiy sifat) {pij:.2f}, P(salbiy sifat) {psa:.2f}")

    print("\n=== 5. Sifatlar embeddinglari: o'xshashlik (kosinus) ===")
    for nom, E in [("tasodifiy init", enc0.emb.weight), ("pretraining dan keyin", enc.emb.weight)]:
        E = F.normalize(E.detach(), dim=1)
        ichki = ((E[ij] @ E[ij].T).sum() - len(ij)) / (len(ij) * (len(ij) - 1))
        orasi = (E[ij] @ E[sa].T).mean()
        print(f"  {nom:<22} ijobiy-ijobiy {ichki:+.3f}   ijobiy-salbiy {orasi:+.3f}")
    E = F.normalize(enc.emb.weight.detach(), dim=1)
    for w in ["a'lo", "iflos"]:
        o = E @ E[lug.stoi[w]]
        o[lug.stoi[w]] = -1
        print(f"  '{w}' ga eng yaqin: " + ", ".join(lug.itos[i] for i in o.topk(4).indices.tolist()))
    print("  hech kim 'a'lo' va 'zo'r' sinonim deb aytmadi - model buni kontekstdan topdi")


if __name__ == "__main__":
    main()

=== 1. Belgisiz korpus (pretraining uchun) ===
  shahardagi telefon juda qiziqarli ekan
  sumka rosa oson
  yangi klinika rosa sovuq emas ekan
  jumlalar 20000, lug'at 110 (4 maxsus + 106 so'z), o'rtacha uzunlik 7.06 token ([CLS] bilan)
  yorliq YO'Q: model faqat matnning o'zidan o'rganadi

=== 2. Maskalash (80/10/10) ===
  tanlangan tokenlar: 943 / 6091 = 0.155
    [MASK] ga almashdi: 0.803
    tasodifiy so'z:     0.095
    o'zgarmadi:         0.102
  misol kirish: [CLS] yangi dastur ancha [MASK] ekan pulimga arzidi

=== 3. MLM pretraining (2000 qadam, batch 32) ===
  qadam    1: o'quv loss (oxirgi 50 o'rtacha) 4.654
  qadam  500: o'quv loss (oxirgi 50 o'rtacha) 1.895
  qadam 1000: o'quv loss (oxirgi 50 o'rtacha) 1.786
  qadam 1500: o'quv loss (oxirgi 50 o'rtacha) 1.742
  qadam 2000: o'quv loss (oxirgi 50 o'rtacha) 1.702
  nazorat MLM loss: boshida 4.696 (ln V = 4.700) -> 1.668
  parametrlar: encoder 62688, MLM boshi 110 (faqat bias)

=== 4. Model bo'shliqni to'ldiradi (top-3) ===
  telefon juda [MASK] ekan hammaga tavsiya qilaman
     -> oson 0.08, yoqimli 0.08, toza 0.06   | P(ijobiy sifat) 0.88, P(salbiy sifat) 0.11
  kafe juda [MASK] edi boshqa kelmayman
     -> oson 0.04, tor 0.04, sekin 0.04   | P(ijobiy sifat) 0.40, P(salbiy sifat) 0.59
  kafe juda [MASK] emas edi boshqa kelmayman
     -> oson 0.06, sifatli 0.06, arzon 0.05   | P(ijobiy sifat) 0.61, P(salbiy sifat) 0.37
  kecha [MASK] juda mazali edi
     -> klinika 0.08, sartaroshxona 0.08, kafe 0.08   | P(ijobiy sifat) 0.00, P(salbiy sifat) 0.00

=== 5. Sifatlar embeddinglari: o'xshashlik (kosinus) ===
  tasodifiy init         ijobiy-ijobiy -0.016   ijobiy-salbiy +0.006
  pretraining dan keyin  ijobiy-ijobiy +0.815   ijobiy-salbiy +0.563
  'a'lo' ga eng yaqin: shinam, qulay, toza, mazali
  'iflos' ga eng yaqin: xira, shovqinli, qo'pol, mo'rt
  hech kim 'a'lo' va 'zo'r' sinonim deb aytmadi - model buni kontekstdan topdi

"""O'rganish egri chizig'i: noldan o'rgatish va pretraining + fine-tuning, juftlashgan urug'lar."""

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

MAVZU = ["telefon", "noutbuk", "kitob", "kafe", "mehmonxona", "avtobus", "dastur",
         "kurs", "film", "do'kon", "restoran", "poyabzal", "kamera", "sumka", "taksi",
         "klinika", "sartaroshxona", "televizor"]
IJOBIY = ["a'lo", "ajoyib", "zo'r", "yaxshi", "qulay", "chiroyli", "tez", "sifatli",
          "mazali", "toza", "arzon", "ishonchli", "shinam", "samimiy", "foydali",
          "qiziqarli", "mukammal", "puxta", "yoqimli", "maroqli", "chidamli", "oson",
          "zamonaviy", "beg'ubor"]
SALBIY = ["yomon", "sekin", "qimmat", "iflos", "noqulay", "xunuk", "sifatsiz", "bemaza",
          "eski", "zerikarli", "ishonchsiz", "qo'pol", "buzuq", "chalkash", "nosoz", "xira",
          "tor", "shovqinli", "befoyda", "sovuq", "mo'rt", "murakkab", "dag'al", "g'alati"]
DARAJA = ["juda", "ancha", "biroz", "rosa", "", ""]
OXIR = ["ekan", "edi", "", "chiqdi"]
BOSH = ["kecha", "o'tgan hafta", "shahardagi", "yangi", "onlayn", "", "", ""]
DUM = {1: ["hammaga tavsiya qilaman", "yana kelaman", "juda mamnunman", "pulimga arzidi",
           "do'stlarimga aytaman", "rahmat sizlarga"],
       0: ["hech kimga tavsiya qilmayman", "pulimni qaytarib oldim", "boshqa kelmayman",
           "vaqtim behuda ketdi", "shikoyat yozdim", "afsuslandim"]}
PAD, CLS, MASK, UNK = 0, 1, 2, 3


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def gap(rng, dum_ehtimol, sifatlar=(IJOBIY, SALBIY)):
    """Sharh: (matn, tonallik, sifat). 25% holatda 'emas' tonallikni teskari qiladi."""
    q = int(rng.integers(2))
    sif = tanla(rng, sifatlar[0] if q else sifatlar[1])
    inkor = rng.random() < 0.25
    y = 1 - q if inkor else q
    s = [tanla(rng, BOSH), tanla(rng, MAVZU), tanla(rng, DARAJA), sif,
         "emas" if inkor else "", tanla(rng, OXIR)]
    if rng.random() < dum_ehtimol:
        s.append(tanla(rng, DUM[y]))
    return " ".join(" ".join(s).split()), y, sif


class Lugat:
    """So'z darajasidagi tokenizator: korpusdan quriladi, 4 ta maxsus token."""

    def __init__(self, matnlar):
        sozlar = sorted({w for m in matnlar for w in m.split()})
        self.itos = ["[PAD]", "[CLS]", "[MASK]", "[UNK]"] + sozlar
        self.stoi = {w: i for i, w in enumerate(self.itos)}

    def kodla(self, matnlar, T=16):
        X = torch.zeros(len(matnlar), T, dtype=torch.long)
        for i, m in enumerate(matnlar):
            ids = [CLS] + [self.stoi.get(w, UNK) for w in m.split()][:T - 1]
            X[i, :len(ids)] = torch.tensor(ids)
        return X


class Blok(nn.Module):
    """Pre-LN transformer bloki: self-attention + FFN, qoldiq ulanishlar."""

    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x, maska):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).view(B, T, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=maska[:, None, None, :])
        x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class Encoder(nn.Module):
    def __init__(self, V, d=48, L=2, h=4, T=16):
        super().__init__()
        self.emb, self.poz = nn.Embedding(V, d), nn.Embedding(T, d)
        self.bloklar = nn.ModuleList([Blok(d, h) for _ in range(L)])
        self.ln = nn.LayerNorm(d)
        nn.init.normal_(self.emb.weight, std=0.02)
        nn.init.normal_(self.poz.weight, std=0.02)

    def forward(self, X):
        maska = X != PAD
        x = self.emb(X) + self.poz(torch.arange(X.shape[1]))
        for b in self.bloklar:
            x = b(x, maska)
        return self.ln(x)


class MLMBosh(nn.Module):
    """MLM boshi: vazni embedding bilan bog'langan (weight tying), faqat bias o'rganiladi."""

    def __init__(self, V):
        super().__init__()
        self.bias = nn.Parameter(torch.zeros(V))

    def forward(self, enc, X):
        return enc(X) @ enc.emb.weight.T + self.bias


def kes(X):
    """Batchdagi eng uzun jumlagacha qirqish (ortiqcha PAD ustunlari kerak emas)."""
    return X[:, :int((X != PAD).sum(1).max())]


def maskala(X, g, V, p=0.15):
    """BERT uslubi: 15% token tanlanadi; ulardan 80% [MASK], 10% tasodifiy, 10% o'zi."""
    Y = X.clone()
    tanl = (torch.rand(X.shape, generator=g) < p) & (X > UNK)
    Y[~tanl] = -100
    r = torch.rand(X.shape, generator=g)
    X = X.clone()
    X[tanl & (r < 0.8)] = MASK
    tas = tanl & (r >= 0.8) & (r < 0.9)
    X[tas] = torch.randint(4, V, (int(tas.sum()),), generator=g)
    return X, Y


def mlm_loss(enc, bosh, X, Y):
    V = enc.emb.num_embeddings
    return F.cross_entropy(bosh(enc, X).reshape(-1, V), Y.reshape(-1))


def pretrain(V, X, qadam, seed=0, bs=32, lr=2e-3, kuzat=None):
    """MLM pretraining: belgisiz korpusda maskalangan so'zlarni tiklash."""
    torch.manual_seed(seed)
    enc, bosh = Encoder(V), MLMBosh(V)
    opt = torch.optim.AdamW(list(enc.parameters()) + list(bosh.parameters()),
                            lr=lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for q in range(1, qadam + 1):
        xb, yb = maskala(kes(X[torch.randint(0, len(X), (bs,), generator=g)]), g, V)
        loss = mlm_loss(enc, bosh, xb, yb)
        opt.zero_grad()
        loss.backward()
        opt.step()
        if kuzat is not None:
            kuzat(q, loss.item())
    enc.eval()
    return enc, bosh


class Klassifikator(nn.Module):
    """Encoder + o'rtacha pooling (PAD siz) + chiziqli bosh."""

    def __init__(self, enc, sinflar=2):
        super().__init__()
        self.enc = enc
        self.bosh = nn.Linear(enc.emb.embedding_dim, sinflar)

    def forward(self, X):
        h = self.enc(X)
        m = (X != PAD).float()[..., None]
        return self.bosh((h * m).sum(1) / m.sum(1))


def fine_tune(model, X, y, seed, qadam=100, lr=1e-3, bs=16):
    """Faqat requires_grad=True parametrlar yangilanadi."""
    opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad],
                            lr=lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    model.train()
    for _ in range(qadam):
        idx = torch.randint(0, len(X), (min(bs, len(X)),), generator=g)
        loss = F.cross_entropy(model(kes(X[idx])), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()
    return model


@torch.no_grad()
def bashorat(model, X):
    return model(kes(X)).argmax(1)


def main() -> None:
    torch.set_num_threads(1)
    rng = np.random.default_rng(0)
    korpus = [gap(rng, 0.7)[0] for _ in range(20000)]
    lug = Lugat(korpus)
    V = len(lug.itos)
    enc_pre, _ = pretrain(V, lug.kodla(korpus), 2000)
    holat = enc_pre.state_dict()

    print("=== 1. Belgili ma'lumot: kam va 'tor' ===")
    korilgan = (IJOBIY[:16], SALBIY[:16])          # belgili to'plamda faqat shular
    rng = np.random.default_rng(1)
    hovuz = [gap(rng, 0.0, korilgan) for _ in range(3000)]
    test = [gap(rng, 0.0) for _ in range(2000)]
    Xh, yh = lug.kodla([h[0] for h in hovuz]), torch.tensor([h[1] for h in hovuz])
    Xt, yt = lug.kodla([t[0] for t in test]), torch.tensor([t[1] for t in test])
    yangi = torch.tensor([t[2] not in korilgan[0] + korilgan[1] for t in test])
    print(f"  belgili hovuz {len(hovuz)} (sifatlarning 32/48 tasi), test {len(test)} "
          f"(hammasi 48 ta)")
    print(f"  testda ko'rilmagan sifatli jumlalar: {yangi.sum().item()} ({yangi.float().mean():.1%})")
    print("  belgili jumlalarda dum ('tavsiya qilaman' ...) yo'q - faqat sifat va 'emas'")

    print("\n=== 2. Egri chiziq: bir xil urug'lar, bir xil qadamlar (100, lr 1e-3) ===")
    olchamlar = [50, 200, 1000]
    natija = {}
    for n in olchamlar:
        for s in range(3):
            idx = torch.tensor(np.random.default_rng(100 + s).choice(len(hovuz), n, replace=False))
            for nom in ["noldan", "pretrained"]:
                torch.manual_seed(s)
                enc = Encoder(V)
                if nom == "pretrained":
                    enc.load_state_dict(holat)
                torch.manual_seed(s)
                model = fine_tune(Klassifikator(enc), Xh[idx], yh[idx], seed=s)
                p = bashorat(model, Xt) == yt
                natija[(n, s, nom)] = (p.float().mean().item(), p[yangi].float().mean().item(),
                                       p[~yangi].float().mean().item())
    print(f"  {'n':>5} {'noldan':>8} {'pretrained':>11} {'farq':>8} {'SE':>7} {'sezilarli':>10}")
    for n in olchamlar:
        a0 = np.array([natija[(n, s, "noldan")][0] for s in range(3)])
        a1 = np.array([natija[(n, s, "pretrained")][0] for s in range(3)])
        d = a1 - a0
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  {n:>5} {a0.mean():>8.3f} {a1.mean():>11.3f} {d.mean():>+8.3f} {se:>7.3f} "
              f"{str(abs(d.mean()) > 2 * se):>10}")

    print("\n=== 3. Ko'rilgan va ko'rilmagan sifatlar (test aniqligi) ===")
    print(f"  {'n':>5} {'model':<11} {'korilgan':>9} {'korilmagan':>11}")
    for n in olchamlar:
        for nom in ["noldan", "pretrained"]:
            k = np.mean([natija[(n, s, nom)][2] for s in range(3)])
            y_ = np.mean([natija[(n, s, nom)][1] for s in range(3)])
            print(f"  {n:>5} {nom:<11} {k:>9.3f} {y_:>11.3f}")
    y0 = np.mean([natija[(1000, s, "noldan")][1] for s in range(3)])
    y1 = np.mean([natija[(1000, s, "pretrained")][1] for s in range(3)])
    if y1 - y0 > 0.1:
        print(f"  n=1000 da ham noldan model yangi sifatlarda {y0:.3f}: u ularni hech qachon "
              "ko'rmagan")
        print("  pretrained model ularni korpusdan 'tanigan' - bilim belgisiz matndan keldi")

    print("\n=== 4. Xulosa (natijadan) ===")
    for qism, k in [("korilgan", 2), ("korilmagan", 1)]:
        f = [np.mean([natija[(n, s, "pretrained")][k] - natija[(n, s, "noldan")][k]
                      for s in range(3)]) for n in olchamlar]
        print(f"  {qism:<11} sifatlarda foyda: " + ", ".join(
            f"n={n} {v:+.3f}" for n, v in zip(olchamlar, f)))
        if f[-1] < f[0] / 2:
            print("    -> belgili misollar ko'paygan sari foyda kamaydi")
        elif f[-1] > 0.3:
            print("    -> foyda kamaymadi: bu so'zlarni belgili ma'lumot umuman o'rgatmaydi")


if __name__ == "__main__":
    main()

=== 1. Belgili ma'lumot: kam va 'tor' ===
  belgili hovuz 3000 (sifatlarning 32/48 tasi), test 2000 (hammasi 48 ta)
  testda ko'rilmagan sifatli jumlalar: 660 (33.0%)
  belgili jumlalarda dum ('tavsiya qilaman' ...) yo'q - faqat sifat va 'emas'

=== 2. Egri chiziq: bir xil urug'lar, bir xil qadamlar (100, lr 1e-3) ===
      n   noldan  pretrained     farq      SE  sezilarli
     50    0.536       0.810   +0.274   0.018       True
    200    0.611       0.915   +0.303   0.031       True
   1000    0.757       0.997   +0.240   0.023       True

=== 3. Ko'rilgan va ko'rilmagan sifatlar (test aniqligi) ===
      n model        korilgan  korilmagan
     50 noldan          0.562       0.482
     50 pretrained      0.814       0.802
    200 noldan          0.677       0.478
    200 pretrained      0.924       0.895
   1000 noldan          0.914       0.438
   1000 pretrained      1.000       0.992
  n=1000 da ham noldan model yangi sifatlarda 0.438: u ularni hech qachon ko'rmagan
  pretrained model ularni korpusdan 'tanigan' - bilim belgisiz matndan keldi

=== 4. Xulosa (natijadan) ===
  korilgan    sifatlarda foyda: n=50 +0.252, n=200 +0.247, n=1000 +0.086
    -> belgili misollar ko'paygan sari foyda kamaydi
  korilmagan  sifatlarda foyda: n=50 +0.319, n=200 +0.418, n=1000 +0.554
    -> foyda kamaymadi: bu so'zlarni belgili ma'lumot umuman o'rgatmaydi

"""Fine-tuning usullari: to'liq, faqat bosh (linear probe) va pastki qatlamlarni muzlatish."""

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

MAVZU = ["telefon", "noutbuk", "kitob", "kafe", "mehmonxona", "avtobus", "dastur",
         "kurs", "film", "do'kon", "restoran", "poyabzal", "kamera", "sumka", "taksi",
         "klinika", "sartaroshxona", "televizor"]
IJOBIY = ["a'lo", "ajoyib", "zo'r", "yaxshi", "qulay", "chiroyli", "tez", "sifatli",
          "mazali", "toza", "arzon", "ishonchli", "shinam", "samimiy", "foydali",
          "qiziqarli", "mukammal", "puxta", "yoqimli", "maroqli", "chidamli", "oson",
          "zamonaviy", "beg'ubor"]
SALBIY = ["yomon", "sekin", "qimmat", "iflos", "noqulay", "xunuk", "sifatsiz", "bemaza",
          "eski", "zerikarli", "ishonchsiz", "qo'pol", "buzuq", "chalkash", "nosoz", "xira",
          "tor", "shovqinli", "befoyda", "sovuq", "mo'rt", "murakkab", "dag'al", "g'alati"]
DARAJA = ["juda", "ancha", "biroz", "rosa", "", ""]
OXIR = ["ekan", "edi", "", "chiqdi"]
BOSH = ["kecha", "o'tgan hafta", "shahardagi", "yangi", "onlayn", "", "", ""]
DUM = {1: ["hammaga tavsiya qilaman", "yana kelaman", "juda mamnunman", "pulimga arzidi",
           "do'stlarimga aytaman", "rahmat sizlarga"],
       0: ["hech kimga tavsiya qilmayman", "pulimni qaytarib oldim", "boshqa kelmayman",
           "vaqtim behuda ketdi", "shikoyat yozdim", "afsuslandim"]}
PAD, CLS, MASK, UNK = 0, 1, 2, 3


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def gap(rng, dum_ehtimol, sifatlar=(IJOBIY, SALBIY)):
    """Sharh: (matn, tonallik, sifat). 25% holatda 'emas' tonallikni teskari qiladi."""
    q = int(rng.integers(2))
    sif = tanla(rng, sifatlar[0] if q else sifatlar[1])
    inkor = rng.random() < 0.25
    y = 1 - q if inkor else q
    s = [tanla(rng, BOSH), tanla(rng, MAVZU), tanla(rng, DARAJA), sif,
         "emas" if inkor else "", tanla(rng, OXIR)]
    if rng.random() < dum_ehtimol:
        s.append(tanla(rng, DUM[y]))
    return " ".join(" ".join(s).split()), y, sif


class Lugat:
    """So'z darajasidagi tokenizator: korpusdan quriladi, 4 ta maxsus token."""

    def __init__(self, matnlar):
        sozlar = sorted({w for m in matnlar for w in m.split()})
        self.itos = ["[PAD]", "[CLS]", "[MASK]", "[UNK]"] + sozlar
        self.stoi = {w: i for i, w in enumerate(self.itos)}

    def kodla(self, matnlar, T=16):
        X = torch.zeros(len(matnlar), T, dtype=torch.long)
        for i, m in enumerate(matnlar):
            ids = [CLS] + [self.stoi.get(w, UNK) for w in m.split()][:T - 1]
            X[i, :len(ids)] = torch.tensor(ids)
        return X


class Blok(nn.Module):
    """Pre-LN transformer bloki: self-attention + FFN, qoldiq ulanishlar."""

    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x, maska):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).view(B, T, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=maska[:, None, None, :])
        x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class Encoder(nn.Module):
    def __init__(self, V, d=48, L=2, h=4, T=16):
        super().__init__()
        self.emb, self.poz = nn.Embedding(V, d), nn.Embedding(T, d)
        self.bloklar = nn.ModuleList([Blok(d, h) for _ in range(L)])
        self.ln = nn.LayerNorm(d)
        nn.init.normal_(self.emb.weight, std=0.02)
        nn.init.normal_(self.poz.weight, std=0.02)

    def forward(self, X):
        maska = X != PAD
        x = self.emb(X) + self.poz(torch.arange(X.shape[1]))
        for b in self.bloklar:
            x = b(x, maska)
        return self.ln(x)


class MLMBosh(nn.Module):
    """MLM boshi: vazni embedding bilan bog'langan (weight tying), faqat bias o'rganiladi."""

    def __init__(self, V):
        super().__init__()
        self.bias = nn.Parameter(torch.zeros(V))

    def forward(self, enc, X):
        return enc(X) @ enc.emb.weight.T + self.bias


def kes(X):
    """Batchdagi eng uzun jumlagacha qirqish (ortiqcha PAD ustunlari kerak emas)."""
    return X[:, :int((X != PAD).sum(1).max())]


def maskala(X, g, V, p=0.15):
    """BERT uslubi: 15% token tanlanadi; ulardan 80% [MASK], 10% tasodifiy, 10% o'zi."""
    Y = X.clone()
    tanl = (torch.rand(X.shape, generator=g) < p) & (X > UNK)
    Y[~tanl] = -100
    r = torch.rand(X.shape, generator=g)
    X = X.clone()
    X[tanl & (r < 0.8)] = MASK
    tas = tanl & (r >= 0.8) & (r < 0.9)
    X[tas] = torch.randint(4, V, (int(tas.sum()),), generator=g)
    return X, Y


def mlm_loss(enc, bosh, X, Y):
    V = enc.emb.num_embeddings
    return F.cross_entropy(bosh(enc, X).reshape(-1, V), Y.reshape(-1))


def pretrain(V, X, qadam, seed=0, bs=32, lr=2e-3, kuzat=None):
    """MLM pretraining: belgisiz korpusda maskalangan so'zlarni tiklash."""
    torch.manual_seed(seed)
    enc, bosh = Encoder(V), MLMBosh(V)
    opt = torch.optim.AdamW(list(enc.parameters()) + list(bosh.parameters()),
                            lr=lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for q in range(1, qadam + 1):
        xb, yb = maskala(kes(X[torch.randint(0, len(X), (bs,), generator=g)]), g, V)
        loss = mlm_loss(enc, bosh, xb, yb)
        opt.zero_grad()
        loss.backward()
        opt.step()
        if kuzat is not None:
            kuzat(q, loss.item())
    enc.eval()
    return enc, bosh


class Klassifikator(nn.Module):
    """Encoder + o'rtacha pooling (PAD siz) + chiziqli bosh."""

    def __init__(self, enc, sinflar=2):
        super().__init__()
        self.enc = enc
        self.bosh = nn.Linear(enc.emb.embedding_dim, sinflar)

    def forward(self, X):
        h = self.enc(X)
        m = (X != PAD).float()[..., None]
        return self.bosh((h * m).sum(1) / m.sum(1))


def fine_tune(model, X, y, seed, qadam=100, lr=1e-3, bs=16):
    """Faqat requires_grad=True parametrlar yangilanadi."""
    opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad],
                            lr=lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    model.train()
    for _ in range(qadam):
        idx = torch.randint(0, len(X), (min(bs, len(X)),), generator=g)
        loss = F.cross_entropy(model(kes(X[idx])), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()
    return model


@torch.no_grad()
def bashorat(model, X):
    return model(kes(X)).argmax(1)


def muzlat(model, usul):
    """usul: 'toliq', 'bosh' (encoder muzlatilgan) yoki 'yuqori' (emb + 1-blok muzlatilgan)."""
    for p in model.parameters():
        p.requires_grad_(True)
    if usul == "bosh":
        for p in model.enc.parameters():
            p.requires_grad_(False)
    elif usul == "yuqori":
        for m in [model.enc.emb, model.enc.poz, model.enc.bloklar[0]]:
            for p in m.parameters():
                p.requires_grad_(False)
    return model


def main() -> None:
    torch.set_num_threads(1)
    rng = np.random.default_rng(0)
    korpus = [gap(rng, 0.7)[0] for _ in range(20000)]
    lug = Lugat(korpus)
    V = len(lug.itos)
    holat = pretrain(V, lug.kodla(korpus), 2000)[0].state_dict()
    korilgan = (IJOBIY[:16], SALBIY[:16])
    rng = np.random.default_rng(1)
    hovuz = [gap(rng, 0.0, korilgan) for _ in range(3000)]
    test = [gap(rng, 0.0) for _ in range(2000)]
    Xh, yh = lug.kodla([h[0] for h in hovuz]), torch.tensor([h[1] for h in hovuz])
    Xt, yt = lug.kodla([t[0] for t in test]), torch.tensor([t[1] for t in test])
    yangi = torch.tensor([t[2] not in korilgan[0] + korilgan[1] for t in test])

    print("=== 1. Usullar: nima o'rgatiladi (n=200, 100 qadam) ===")
    usullar = [("noldan", "toliq", False, 1e-3), ("faqat bosh", "bosh", True, 1e-2),
               ("yuqori blok+bosh", "yuqori", True, 1e-3), ("to'liq FT", "toliq", True, 1e-3)]
    torch.manual_seed(0)
    jami = sum(p.numel() for p in Klassifikator(Encoder(V)).parameters())
    print(f"  {'usul':<18} {'pretrained':>10} {'lr':>6} {'orgatiladi':>11} {'ulush':>7}")
    for nom, usul, pre, lr in usullar:
        m = muzlat(Klassifikator(Encoder(V)), usul)
        n_or = sum(p.numel() for p in m.parameters() if p.requires_grad)
        print(f"  {nom:<18} {str(pre):>10} {lr:>6.0e} {n_or:>11} {n_or / jami:>7.1%}")

    print("\n=== 2. Natija (3 urug', test 2000) ===")
    natija = {}
    for s in range(3):
        idx = torch.tensor(np.random.default_rng(100 + s).choice(len(hovuz), 200, replace=False))
        for nom, usul, pre, lr in usullar:
            torch.manual_seed(s)
            enc = Encoder(V)
            if pre:
                enc.load_state_dict(holat)
            torch.manual_seed(s)
            model = muzlat(Klassifikator(enc), usul)
            model = fine_tune(model, Xh[idx], yh[idx], seed=s, lr=lr)
            p = bashorat(model, Xt) == yt
            natija[(nom, s)] = (p.float().mean().item(), p[yangi].float().mean().item())
    print(f"  {'usul':<18} {'aniqlik':>8} {'korilmagan':>11} {'to`liq FT - usul':>17} {'2*SE':>7}"
          .replace("`", "'"))
    toliq = np.array([natija[("to'liq FT", s)][0] for s in range(3)])
    yomon = {}
    for nom, _, _, _ in usullar:
        a = np.array([natija[(nom, s)][0] for s in range(3)])
        k = np.mean([natija[(nom, s)][1] for s in range(3)])
        d = toliq - a
        se = d.std(ddof=1) / np.sqrt(3)
        yomon[nom] = d.mean() > 2 * se
        qator = f"  {nom:<18} {a.mean():>8.3f} {k:>11.3f}"
        if nom != "to'liq FT":
            qator += f" {d.mean():>+17.3f} {2 * se:>7.3f}"
        print(qator)

    print("\n=== 3. Xulosa (natijadan) ===")
    for nom in ["faqat bosh", "yuqori blok+bosh", "noldan"]:
        print(f"  {nom:<18} to'liq FT dan sezilarli yomonmi: {'ha' if yomon[nom] else "yo'q"}")
    if not yomon["yuqori blok+bosh"]:
        print("  pastki qatlamlarni muzlatish sifatni sezilarli tushirmadi - "
              "kamroq parametr o'rgatildi")
    else:
        print("  muzlatish sifatni tushirdi: pastki blok ham vazifaga moslashishi kerak edi")
    if yomon["faqat bosh"]:
        print("  faqat bosh yetmadi: pretraining vakillari bu vazifa uchun 'tayyor' emas")
    else:
        print("  faqat bosh ham yetarli bo'ldi: pretraining vakillari vazifaga tayyor")


if __name__ == "__main__":
    main()

=== 1. Usullar: nima o'rgatiladi (n=200, 100 qadam) ===
  usul               pretrained     lr  orgatiladi   ulush
  noldan                  False  1e-03       62786  100.0%
  faqat bosh               True  1e-02          98    0.2%
  yuqori blok+bosh         True  1e-03       28466   45.3%
  to'liq FT                True  1e-03       62786  100.0%

=== 2. Natija (3 urug', test 2000) ===
  usul                aniqlik  korilmagan  to'liq FT - usul    2*SE
  noldan                0.611       0.478            +0.303   0.062
  faqat bosh            0.761       0.756            +0.154   0.038
  yuqori blok+bosh      0.822       0.818            +0.093   0.029
  to'liq FT             0.915       0.895

=== 3. Xulosa (natijadan) ===
  faqat bosh         to'liq FT dan sezilarli yomonmi: ha
  yuqori blok+bosh   to'liq FT dan sezilarli yomonmi: ha
  noldan             to'liq FT dan sezilarli yomonmi: ha
  muzlatish sifatni tushirdi: pastki blok ham vazifaga moslashishi kerak edi
  faqat bosh yetmadi: pretraining vakillari bu vazifa uchun 'tayyor' emas

"""Katastrofik unutish va pretrained modelni saqlash/yuklash (from_pretrained g'oyasi)."""

import copy
import tempfile
from pathlib import Path

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

MAVZU = ["telefon", "noutbuk", "kitob", "kafe", "mehmonxona", "avtobus", "dastur",
         "kurs", "film", "do'kon", "restoran", "poyabzal", "kamera", "sumka", "taksi",
         "klinika", "sartaroshxona", "televizor"]
IJOBIY = ["a'lo", "ajoyib", "zo'r", "yaxshi", "qulay", "chiroyli", "tez", "sifatli",
          "mazali", "toza", "arzon", "ishonchli", "shinam", "samimiy", "foydali",
          "qiziqarli", "mukammal", "puxta", "yoqimli", "maroqli", "chidamli", "oson",
          "zamonaviy", "beg'ubor"]
SALBIY = ["yomon", "sekin", "qimmat", "iflos", "noqulay", "xunuk", "sifatsiz", "bemaza",
          "eski", "zerikarli", "ishonchsiz", "qo'pol", "buzuq", "chalkash", "nosoz", "xira",
          "tor", "shovqinli", "befoyda", "sovuq", "mo'rt", "murakkab", "dag'al", "g'alati"]
DARAJA = ["juda", "ancha", "biroz", "rosa", "", ""]
OXIR = ["ekan", "edi", "", "chiqdi"]
BOSH = ["kecha", "o'tgan hafta", "shahardagi", "yangi", "onlayn", "", "", ""]
DUM = {1: ["hammaga tavsiya qilaman", "yana kelaman", "juda mamnunman", "pulimga arzidi",
           "do'stlarimga aytaman", "rahmat sizlarga"],
       0: ["hech kimga tavsiya qilmayman", "pulimni qaytarib oldim", "boshqa kelmayman",
           "vaqtim behuda ketdi", "shikoyat yozdim", "afsuslandim"]}
PAD, CLS, MASK, UNK = 0, 1, 2, 3


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def gap(rng, dum_ehtimol, sifatlar=(IJOBIY, SALBIY)):
    """Sharh: (matn, tonallik, sifat). 25% holatda 'emas' tonallikni teskari qiladi."""
    q = int(rng.integers(2))
    sif = tanla(rng, sifatlar[0] if q else sifatlar[1])
    inkor = rng.random() < 0.25
    y = 1 - q if inkor else q
    s = [tanla(rng, BOSH), tanla(rng, MAVZU), tanla(rng, DARAJA), sif,
         "emas" if inkor else "", tanla(rng, OXIR)]
    if rng.random() < dum_ehtimol:
        s.append(tanla(rng, DUM[y]))
    return " ".join(" ".join(s).split()), y, sif


class Lugat:
    """So'z darajasidagi tokenizator: korpusdan quriladi, 4 ta maxsus token."""

    def __init__(self, matnlar):
        sozlar = sorted({w for m in matnlar for w in m.split()})
        self.itos = ["[PAD]", "[CLS]", "[MASK]", "[UNK]"] + sozlar
        self.stoi = {w: i for i, w in enumerate(self.itos)}

    def kodla(self, matnlar, T=16):
        X = torch.zeros(len(matnlar), T, dtype=torch.long)
        for i, m in enumerate(matnlar):
            ids = [CLS] + [self.stoi.get(w, UNK) for w in m.split()][:T - 1]
            X[i, :len(ids)] = torch.tensor(ids)
        return X


class Blok(nn.Module):
    """Pre-LN transformer bloki: self-attention + FFN, qoldiq ulanishlar."""

    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))

    def forward(self, x, maska):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).view(B, T, 3, self.h, d // self.h).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=maska[:, None, None, :])
        x = x + self.proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class Encoder(nn.Module):
    def __init__(self, V, d=48, L=2, h=4, T=16):
        super().__init__()
        self.emb, self.poz = nn.Embedding(V, d), nn.Embedding(T, d)
        self.bloklar = nn.ModuleList([Blok(d, h) for _ in range(L)])
        self.ln = nn.LayerNorm(d)
        nn.init.normal_(self.emb.weight, std=0.02)
        nn.init.normal_(self.poz.weight, std=0.02)

    def forward(self, X):
        maska = X != PAD
        x = self.emb(X) + self.poz(torch.arange(X.shape[1]))
        for b in self.bloklar:
            x = b(x, maska)
        return self.ln(x)


class MLMBosh(nn.Module):
    """MLM boshi: vazni embedding bilan bog'langan (weight tying), faqat bias o'rganiladi."""

    def __init__(self, V):
        super().__init__()
        self.bias = nn.Parameter(torch.zeros(V))

    def forward(self, enc, X):
        return enc(X) @ enc.emb.weight.T + self.bias


def kes(X):
    """Batchdagi eng uzun jumlagacha qirqish (ortiqcha PAD ustunlari kerak emas)."""
    return X[:, :int((X != PAD).sum(1).max())]


def maskala(X, g, V, p=0.15):
    """BERT uslubi: 15% token tanlanadi; ulardan 80% [MASK], 10% tasodifiy, 10% o'zi."""
    Y = X.clone()
    tanl = (torch.rand(X.shape, generator=g) < p) & (X > UNK)
    Y[~tanl] = -100
    r = torch.rand(X.shape, generator=g)
    X = X.clone()
    X[tanl & (r < 0.8)] = MASK
    tas = tanl & (r >= 0.8) & (r < 0.9)
    X[tas] = torch.randint(4, V, (int(tas.sum()),), generator=g)
    return X, Y


def mlm_loss(enc, bosh, X, Y):
    V = enc.emb.num_embeddings
    return F.cross_entropy(bosh(enc, X).reshape(-1, V), Y.reshape(-1))


def pretrain(V, X, qadam, seed=0, bs=32, lr=2e-3, kuzat=None):
    """MLM pretraining: belgisiz korpusda maskalangan so'zlarni tiklash."""
    torch.manual_seed(seed)
    enc, bosh = Encoder(V), MLMBosh(V)
    opt = torch.optim.AdamW(list(enc.parameters()) + list(bosh.parameters()),
                            lr=lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for q in range(1, qadam + 1):
        xb, yb = maskala(kes(X[torch.randint(0, len(X), (bs,), generator=g)]), g, V)
        loss = mlm_loss(enc, bosh, xb, yb)
        opt.zero_grad()
        loss.backward()
        opt.step()
        if kuzat is not None:
            kuzat(q, loss.item())
    enc.eval()
    return enc, bosh


class Klassifikator(nn.Module):
    """Encoder + o'rtacha pooling (PAD siz) + chiziqli bosh."""

    def __init__(self, enc, sinflar=2):
        super().__init__()
        self.enc = enc
        self.bosh = nn.Linear(enc.emb.embedding_dim, sinflar)

    def forward(self, X):
        h = self.enc(X)
        m = (X != PAD).float()[..., None]
        return self.bosh((h * m).sum(1) / m.sum(1))


def fine_tune(model, X, y, seed, qadam=100, lr=1e-3, bs=16):
    """Faqat requires_grad=True parametrlar yangilanadi."""
    opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad],
                            lr=lr, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    model.train()
    for _ in range(qadam):
        idx = torch.randint(0, len(X), (min(bs, len(X)),), generator=g)
        loss = F.cross_entropy(model(kes(X[idx])), y[idx])
        opt.zero_grad()
        loss.backward()
        opt.step()
    model.eval()
    return model


@torch.no_grad()
def bashorat(model, X):
    return model(kes(X)).argmax(1)


def main() -> None:
    torch.set_num_threads(1)
    rng = np.random.default_rng(0)
    korpus = [gap(rng, 0.7)[0] for _ in range(20000)]
    lug = Lugat(korpus)
    V = len(lug.itos)
    enc_pre, mlm_bosh = pretrain(V, lug.kodla(korpus), 2000)
    rng = np.random.default_rng(2)
    nazorat = lug.kodla([gap(rng, 0.7)[0] for _ in range(1000)])
    Xn, Yn = maskala(nazorat, torch.Generator().manual_seed(5), V)   # qat'iy maskalar
    hovuz = [gap(rng, 0.0) for _ in range(1000)]
    test = [gap(rng, 0.0) for _ in range(2000)]
    Xh, yh = lug.kodla([h[0] for h in hovuz]), torch.tensor([h[1] for h in hovuz])
    Xt, yt = lug.kodla([t[0] for t in test]), torch.tensor([t[1] for t in test])

    def mlm(enc):
        with torch.no_grad():
            return mlm_loss(enc, mlm_bosh, Xn, Yn).item()

    L0 = mlm(enc_pre)
    print("=== 1. Katastrofik unutish: fine-tuning dan keyin MLM loss ===")
    print(f"  pretraining dan keyin nazorat MLM loss: {L0:.3f}")
    print(f"  {'usul':<24} {'test aniqlik':>12} {'MLM loss':>9} {'o`sish':>8}".replace("`", "'"))
    usullar = [("faqat bosh, lr 1e-2", True, 1e-2, 100),
               ("to'liq, lr 3e-4", False, 3e-4, 100),
               ("to'liq, lr 1e-3", False, 1e-3, 100),
               ("to'liq, lr 3e-3, 300 qadam", False, 3e-3, 300)]
    natija = {}
    for nom, muzlat, lr, qadam in usullar:
        torch.manual_seed(0)
        model = Klassifikator(copy.deepcopy(enc_pre))
        if muzlat:
            for p in model.enc.parameters():
                p.requires_grad_(False)
        model = fine_tune(model, Xh, yh, seed=0, qadam=qadam, lr=lr)
        aniq = (bashorat(model, Xt) == yt).float().mean().item()
        L = mlm(model.enc)
        natija[nom] = (aniq, L)
        print(f"  {nom:<26} {aniq:>10.3f} {L:>9.3f} {L - L0:>+8.3f}")
    eng = max(natija, key=lambda k: natija[k][1])
    print(f"  eng ko'p unutgan: {eng} (MLM loss {natija[eng][1]:.3f})")
    if natija["faqat bosh, lr 1e-2"][1] == L0:
        print("  faqat bosh: encoder o'zgarmadi -> MLM bilimi aynan saqlandi")
    kich = natija["to'liq, lr 3e-4"]
    if kich[1] - L0 < natija[eng][1] - L0:
        print(f"  kichik lr: aniqlik {kich[0]:.3f}, unutish {kich[1] - L0:+.3f} - "
              "yumshoq moslashish")

    print("\n=== 2. Pretrained modelni saqlash: konfig + lug'at + vaznlar ===")
    with tempfile.TemporaryDirectory() as papka:
        yol = Path(papka) / "uzbert_kichik.pt"
        paket = {
            "konfig": {"V": V, "d": 48, "L": 2, "h": 4, "T": 16},
            "lugat": list(lug.itos),
            "encoder": enc_pre.state_dict(),
            "mlm_bias": mlm_bosh.bias.detach().clone(),
            "karta": {"korpus": "20000 sintetik sharh", "vazifa": "MLM 15%",
                      "qadamlar": 2000, "nazorat_mlm_loss": round(L0, 4)},
        }
        torch.save(paket, yol)
        p = torch.load(yol, weights_only=True)
    print(f"  kalitlar: {sorted(p)}")
    print(f"  encoder tenzorlari: {len(p['encoder'])}, lug'at {len(p['lugat'])} ta so'z")

    print("\n=== 3. Yuklash (from_pretrained kabi) va tekshirish ===")
    k = p["konfig"]
    enc2 = Encoder(k["V"], k["d"], k["L"], k["h"], k["T"])
    enc2.load_state_dict(p["encoder"])
    enc2.eval()
    lug2_stoi = {w: i for i, w in enumerate(p["lugat"])}
    print(f"  lug'at mos: {lug2_stoi == lug.stoi}")
    with torch.no_grad():
        farq = (enc2(nazorat) - enc_pre(nazorat)).abs().max().item()
    print(f"  chiqishlar farqi (max |a-b|): {farq}")
    torch.manual_seed(0)
    yangi = Klassifikator(enc2)
    print(f"  ustiga yangi bosh: {sum(q.numel() for q in yangi.bosh.parameters())} parametr "
          "(tasodifiy) - endi fine-tuning ga tayyor")
    print(f"  model kartasi: {p['karta']}")


if __name__ == "__main__":
    main()

=== 1. Katastrofik unutish: fine-tuning dan keyin MLM loss ===
  pretraining dan keyin nazorat MLM loss: 1.665
  usul                     test aniqlik  MLM loss   o'sish
  faqat bosh, lr 1e-2             0.743     1.665   +0.000
  to'liq, lr 3e-4                 0.808     1.759   +0.094
  to'liq, lr 1e-3                 0.999     2.367   +0.702
  to'liq, lr 3e-3, 300 qadam      1.000     3.669   +2.004
  eng ko'p unutgan: to'liq, lr 3e-3, 300 qadam (MLM loss 3.669)
  faqat bosh: encoder o'zgarmadi -> MLM bilimi aynan saqlandi
  kichik lr: aniqlik 0.808, unutish +0.094 - yumshoq moslashish

=== 2. Pretrained modelni saqlash: konfig + lug'at + vaznlar ===
  kalitlar: ['encoder', 'karta', 'konfig', 'lugat', 'mlm_bias']
  encoder tenzorlari: 28, lug'at 110 ta so'z

=== 3. Yuklash (from_pretrained kabi) va tekshirish ===
  lug'at mos: True
  chiqishlar farqi (max |a-b|): 0.0
  ustiga yangi bosh: 98 parametr (tasodifiy) - endi fine-tuning ga tayyor
  model kartasi: {'korpus': '20000 sintetik sharh', 'vazifa': 'MLM 15%', 'qadamlar': 2000, 'nazorat_mlm_loss': 1.6649}

noldan = orgat(Encoder(V), X[:200], qadam=50)                      # ⚠️
pre = orgat(pretrained, X[200:400], qadam=300)                     # ⚠️ boshqa misol, boshqa qadam
idx = rng_s.choice(len(X), 200); torch.manual_seed(s)              # ✅ bir urug' - bir xil misol
noldan = fine_tune(Klassifikator(Encoder(V)), X[idx], y[idx], seed=s)

lug = Lugat(belgili_matnlar)                                       # ⚠️ indekslar boshqa
enc.load_state_dict(pretrained)                                    #    embeddinglar mos emas
stoi = {w: i for i, w in enumerate(p["lugat"])}                     # ✅ pretraining lug'ati

loss = F.cross_entropy(logit.reshape(-1, V), X.reshape(-1))       # ⚠️ nusxa ko'chirish vazifasi
Y[~tanlangan] = -100                                               # ✅ faqat tanlanganlar

model.enc.eval()                     # "encoder muzlatildi"          # ⚠️ vaznlar baribir o'zgaradi
for p in model.enc.parameters():                                   # ✅ gradient umuman hisoblanmaydi
    p.requires_grad_(False)

fine_tune(model, X, y, lr=3e-3, qadam=300)                        # ⚠️ MLM loss +2.004
fine_tune(model, X, y, lr=1e-3, qadam=100)                        # ✅ va unutishni o'lchang

h.mean(1)                                                          # ⚠️ PAD vakillari aralashadi
m = (X != PAD).float()[..., None]; (h * m).sum(1) / m.sum(1)       # ✅

model_a = Klassifikator(enc); fine_tune(model_a, ...)              # ⚠️ enc o'zgardi
model_b = Klassifikator(enc); fine_tune(model_b, ...)              #    b - "unutgan" enc dan
model_b = Klassifikator(copy.deepcopy(enc_pre))                    # ✅ har vazifaga toza nusxa

1.  # V = 110 bo'lsa tasodifiy modelning MLM loss i taxminan?
2.  # 15% tanlangan tokenlarning qancha qismi [MASK] ga almashadi?
3.  # nega faqat [MASK] ishlatilmaydi?
4.  # bog'langan vaznlarda MLM logit formulasi?
5.  # fine-tuning da klassifikatsiya boshi qayerdan keladi?
6.  # "faqat bosh" usulida encoder parametrlari qanday belgilanadi?
7.  # pretraining foydasi n oshganda qanday o'zgaradi (ko'rilgan so'zlarda)?
8.  # ko'rilmagan so'zlarda-chi?
9.  # katastrofik unutish qanday o'lchanadi?
10. # nega fine-tuning dan keyin MLM loss o'sadi?
11. # HF da `Lugat` va `Klassifikator` ning analoglari?
12. # model kartasida nima yoziladi?

1.  Y = X.clone()
    loss = F.cross_entropy(model(Xm).reshape(-1, V), Y.reshape(-1))

2.  model.enc.eval()                  # faqat boshni o'rgatmoqchimiz
    opt = torch.optim.AdamW(model.parameters(), lr=1e-2, weight_decay=0.01)

3.  enc_a = fine_tune(Klassifikator(enc_pre), Xa, ya, seed=0).enc
    enc_b = fine_tune(Klassifikator(enc_pre), Xb, yb, seed=0).enc

4.  lug = Lugat(belgili_matnlar)
    enc.load_state_dict(torch.load("pretrained.pt", weights_only=True))

5.  pooled = model.enc(X).mean(1)