25.11-dars: LLM fine-tuning

QAROR DARAXTI (yuqoridan pastga):

  1. Yaxshi prompt 25.4-bob + bir necha misol bilan vazifa hal bo'ladimi?
       ha  -> PROMPT. Eng arzon, eng tez o'zgartiriladi. To'xtang.
  2. Muammo - model BILMAYDIGAN yoki tez O'ZGARADIGAN ma'lumotmi
     (ichki hujjatlar, narxlar, qoidalar)?
       ha  -> RAG (25.8, 25.9). Fine-tuning faktlarni ishonchli "yuklamaydi",
              yangilash uchun qayta o'rgatish kerak bo'ladi.
  3. Muammo - XULQ: format, uslub, ohang, tor vazifada barqarorlik,
     uzun promptni qisqartirish, kichikroq modelga o'tish?
       ha  -> FINE-TUNING ni ko'rib chiqing, agar:
              - yuzlab-minglab SIFATLI misol bor (yoki yig'ish mumkin)
              - eval to'plami va regressiya testlari tayyor 25.10-bob
              - model/versiyani saqlash va yangilashga resurs bor
  4. Ko'pincha: RAG (bilim) + fine-tuning (format/xulq) birga

FINE-TUNING NIMA BERMAYDI:
  yangi faktlarni ishonchli eslab qolish - buning uchun RAG
  noto'g'ri yoki shovqinli ma'lumotni "tuzatish" - aksincha, o'rganadi (3-misol)
  bepul yaxshilanish - eski qobiliyat yo'qolishi mumkin (4-misol)

KO'RSATMA-JAVOB (instruction-response):
  {"instruction": "Matndan ismni ajrating", "input": "...", "output": "anvar"}

CHAT SHABLON (zamonaviy LLM lar):
  [system] siz ... yordamchisiz
  [user]   savol
  [assistant] javob
  model har rolni MAXSUS TOKENLAR bilan ajratadi; shablon modelga xos
  O'RGATISHDA va ISHLATISHDA aynan bir xil shablon bo'lishi shart

BIZNING MINI-SHABLON (1-misol):
  prompt: "nigora zavodga bordi. kim? "   javob: "nigora.\n"
  "kim?" -> ism, "qayerga?" -> joy; "\n" - javob tugadi belgisi (EOS o'rnida)

MA'LUMOT QOIDALARI:
  xilma-xil ko'rsatmalar; javoblar tekshirilgan
  test/eval ko'rsatmalari o'rgatishga tushmasin (deduplikatsiya, 25.10 - oqib ketish)
  JSONL: har qator - bitta misol (provayderlar ham shu formatni kutadi)

KETMA-KETLIK: [prompt tokenlari][javob tokenlari]
  niqobsiz: loss = barcha keyingi-token bashoratlari bo'yicha
  niqobli:  prompt pozitsiyalarida maqsad = -100 (ignore_index) -> loss faqat javobda

1-MISOL, SANOQ (200 misol):
  niqobsiz: 7360 maqsad token, niqobli: 1622 (22%)
  ya'ni niqobsiz loss ning ~78% i - promptni (bizda ma'lum jumlani) bashorat qilish

1-MISOL NATIJASI (bir xil 150 qadam, bir xil batchlar):
  niqobli  aniqlik 0.683
  niqobsiz aniqlik 0.137    farq +0.547 (SE 0.031)
  niqobsiz model gradientining asosiy qismini promptni modellashtirishga sarflaydi;
  niqobli esa aynan "savolga javob berish" ni o'rganadi

LEKIN (1-misol): niqobli model umumiy PPL 1.866 -> 161.33, niqobsiz -> 13.55
  niqobsiz - promptdagi umumiy matnni ham o'rganishda davom etadi -> kam unutish

PRETRAINED MODEL (1-misol): umumiy PPL 1.866, lekin vazifada 0.000 -
  u shunchaki matnni davom ettiradi: "malika maktabga bordi. qayerga? " -> "shahlo shaharg"
  ko'rsatmaga amal qilish - alohida o'rgatiladigan xulq (SFT)

NOLDAN SFT (pretrainingsiz) bilan taqqoslash (1-misol, 2000 misol, 150 qadam):
  pretrained + SFT 0.683, noldan + SFT 0.673 - farq +0.010 (SE 0.034), sezilarli emas
  BU TOR VAZIFADA pretrainingning aniqlikka foydasi ko'rinmadi
  lekin noldan model umumiy tilni umuman bilmaydi (PPL 865)

HAQIQIY LLM LARDA:
  pretraining - trillionlab token: til, faktlar, fikrlash; SFT - ming-yuz ming misol
  SFT faqat "qanday javob berish" ni o'rgatadi, bilimning asosiy qismi pretrainingdan
  bu yerdagi kichik model bu nisbatni ko'rsata olmaydi - vazifa juda tor

TO'LIQ FINE-TUNING: barcha vaznlar yangilanadi
  har vazifa uchun modelning to'liq nusxasi; katta modelda xotira va vaqt katta

LoRA (24.11-dars): W muzlatiladi, faqat past rangli qo'shimcha o'rgatiladi
  y = W x + (alfa / r) * B A x,   A: r x kirish,  B: chiqish x r,  B = 0 dan boshlanadi
  bizda r = 4, qkv, proj va FFN qatlamlarida: 4608 parametr (to'liqning 10.7%)
  katta modellarda bu ulush odatda 1% dan ham kam

BIRLASHTIRISH (merge): W' = W + (alfa / r) B A - inference da qo'shimcha xarajat yo'q
  2-misol: birlashtirishdan keyin chiqish farqi 1.4e-06, aniqlik o'zgarmadi
  bitta asosiy model + ko'p kichik adapterlar (har vazifa/mijozga bittadan)

2-MISOL NATIJASI (bir xil 150 qadam):
  usul                aniqlik   umumiy PPL (pretrained 1.866)
  to'liq, lr 1e-2      0.683      161.33  (86.5x)
  to'liq, lr 3e-3      0.463        9.47  (5.1x)
  LoRA r=4, lr 1e-2    0.237        7.08  (3.8x)
  bu kichik modelda LoRA shu byudjetda to'liq fine-tuning dan ancha orqada qoldi,
  lekin eng kam unutdi; LoRA odatda sekinroq yaqinlashadi - qadamlar soni muhim

QLoRA (g'oya): asosiy model 4 bitli kvantlangan, LoRA adapterlar to'liq aniqlikda -
  katta modelni bitta GPU da fine-tuning qilish imkonini beradi

KATASTROFIK UNUTISH: yangi vazifada o'rgatish eski qobiliyatni buzadi
  o'lchov: fine-tuning dan OLDIN va KEYIN umumiy (vazifadan tashqari) to'plamda
  perplexity yoki umumiy eval

4-MISOL (lr 1e-2, qadamlar bo'yicha):
  qadam   faqat SFT: aniqlik / PPL     SFT + 30% replay: aniqlik / PPL
    25        0.087 /  19.15               0.120 /  2.57
   150        0.707 / 161.33               0.433 /  2.12
  namuna matn: faqat SFT -> "rustabtabtabtab. ahlo."
               replay    -> "feruza kasalxonaga bordi" (umumiy til saqlangan)

YUMSHATISH USULLARI:
  replay - har batchga umumiy (pretraining turidagi) matn qo'shish
  kichikroq lr va kamroq qadam (2-misol: lr 3e-3 - PPL 9.47, 161.33 o'rniga)
  LoRA - asosiy vaznlar o'zgarmaydi (2-misol: 7.08)
  niqobsiz loss ham qisman saqlaydi (1-misol: 13.55)
  NARXI: bir xil qadamda vazifa aniqligi pastroq (4-misolda -0.273) -
    ko'proq qadam yoki murosa kerak

AMALDA: fine-tuning dan keyin FAQAT vazifa metrikasini emas, umumiy eval ni ham
  (25.10 dagi regressiya testlari) ishga tushiring

3-MISOL (bir xil 150 qadam, faqat ma'lumot o'zgaradi):
  300 toza                 0.453
  2000 toza                0.683
  2000, 30% noto'g'ri      0.280
  300 toza - 2000 shovqinli: +0.173 (SE 0.040) - sezilarli

NIMA UCHUN SHOVQIN BUNCHA ZARARLI:
  SFT modeli belgilarga TAQLID qiladi - noto'g'ri javob ham "to'g'ri naqsh" deb o'rganiladi
  shovqin gradientni qarama-qarshi yo'nalishlarga tortadi -> sekin va noto'g'ri o'rganish

AMALIY QOIDALAR:
  ko'p shovqinli misoldan ko'ra kam, lekin tekshirilgan misol
  namunani qo'lda tekshiring (masalan, 100 ta) - xato ulushini baholang
  qarama-qarshi misollarni topish: bir xil kirish, turli javob
  LLM bilan sintetik ma'lumot - albatta filtrlash va tekshirish bilan

SFT CHEGARASI: faqat "to'g'ri javob" ni ko'rsatadi; "qaysi javob yaxshiroq" ni emas

AFZALLIK MA'LUMOTI: (prompt x, yaxshi javob y_w, yomon javob y_l) - inson tanlagan

RLHF (klassik): afzallikdan mukofot modeli -> RL (PPO) bilan model optimallashtiriladi
  murakkab: ikki model, beqaror o'rgatish

DPO (Direct Preference Optimization): mukofot modelisiz, to'g'ridan-to'g'ri loss
  loss = -log sigmoid( beta * [ (log pi(y_w|x) - log pi_ref(y_w|x))
                              - (log pi(y_l|x) - log pi_ref(y_l|x)) ] )
  pi     - o'rgatilayotgan model,  pi_ref - muzlatilgan SFT modeli
  beta   - ref modeldan qanchalik uzoqlashishga ruxsat (odatda 0.1 atrofida)
  ma'nosi: yaxshi javob ehtimolini ref ga nisbatan oshir, yomonini pasaytir

TARTIB: pretraining -> SFT -> afzallik bilan moslash (DPO / RLHF)

UMUMIY JARAYON (provayder va modelga qarab farq qiladi):
  1. ma'lumot - JSONL, har qatorda chat xabarlari (system/user/assistant)
  2. faylni yuklash va fine-tuning ishini (job) yaratish
  3. giperparametrlar: epoxalar soni, lr ko'paytuvchisi (ko'pincha avtomatik)
  4. natija - yangi model identifikatori; u bilan oddiy API chaqiruvi
  5. baholash: 25.10 dagi eval to'plami va regressiya testlari - asosiy model bilan juftlashgan

E'TIBOR:
  hamma model va provayder fine-tuning ni qo'llab-quvvatlamaydi - hujjatlarni tekshiring
  narx: o'rgatish tokenlari + ba'zan qimmatroq inference (haqiqiy narxni
    provayderning rasmiy sahifasidan oling)
  ma'lumot maxfiyligi: nima yuklanayotganini va qayerda saqlanishini tekshiring
  ochiq vaznli modellar (o'z serveringizda) - LoRA/QLoRA bilan to'liq nazorat

import json

import torch
import torch.nn as nn
import torch.nn.functional as F

# SFT: faqat javob tokenlarida loss
ids = lug.kod(prompt + javob)
maqsad = ids[1:] + [-100]
maqsad = [-100] * (len(prompt) - 1) + maqsad[len(prompt) - 1:]
loss = F.cross_entropy(logit.reshape(-1, V), Y.reshape(-1), ignore_index=-100)


# LoRA
class LoRALinear(nn.Module):
    def __init__(self, asos, r=4, alfa=8):
        super().__init__()
        self.asos = asos.requires_grad_(False)
        self.A = nn.Parameter(torch.randn(r, asos.in_features) / asos.in_features ** 0.5)
        self.B = nn.Parameter(torch.zeros(asos.out_features, r))
        self.k = alfa / r

    def forward(self, x):
        return self.asos(x) + (x @ self.A.T @ self.B.T) * self.k


W_birlashgan = W + (alfa / r) * B @ A                     # merge

# replay: SFT loss + umumiy matn loss
loss = 0.7 * sft_loss + 0.3 * umumiy_loss


# DPO loss (log p - javob tokenlari log-ehtimollari yig'indisi)
def dpo_loss(lp_w, lp_l, ref_w, ref_l, beta=0.1):
    return -F.logsigmoid(beta * ((lp_w - ref_w) - (lp_l - ref_l))).mean()


# provayder uchun JSONL (chat format)
qator = {"messages": [{"role": "system", "content": "Siz HR yordamchisisiz."},
                      {"role": "user", "content": "Moliya bo'limida ta'til necha kun?"},
                      {"role": "assistant", "content": "Yillik ta'til 28 kun."}]}
satr = json.dumps(qator, ensure_ascii=False)

avval prompt, bilim uchun RAG, xulq/format uchun fine-tuning
SFT: ko'rsatma -> javob, bitta shablon, loss faqat javobda, EOS ni o'rgating
LoRA: W + (alfa/r) BA; kam parametr, merge; sifatni o'lchang
unutish: umumiy eval ni ham kuzating; replay, kichik lr, LoRA
sifat > miqdor: belgilarni tekshiring
DPO: afzallik juftliklari, muzlatilgan ref model

"""Kichik GPT: pretraining -> SFT; faqat javob tokenlarida loss (niqob) va niqobsiz farqi."""

import copy
import math

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab shahar klinika kutubxona choyxona dala bekat "
          "kasalxona stadion muzey teatr vokzal ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch shakar asal soat telefon").split()
SIFATLAR = "katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat".split()
L = 48
SAVOLLAR = {"kim": 0, "qayerga": 1}


def jumla(rng):
    t = int(rng.integers(0, 8))
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = rng.choice(VAQTLAR), rng.choice(JOYLAR)
    narsa, sifat = rng.choice(NARSALAR), rng.choice(SIFATLAR)
    return [f"{ism} {joy}ga {vaqt} bordi.", f"{ism} {joy}dan {vaqt} qaytdi.",
            f"{ism} {sifat} {narsa} sotib oldi.", f"{joy}da {sifat} {narsa} bor.",
            f"{ism} {ism2}ga {sifat} {narsa} berdi.", f"{ism} qayerga bordi?",
            f"{ism} {joy}ga bordi. {joy} {ism}ga yoqdi.",          # takrorlanish: nusxa
            f"{ism} {narsa} oldi. {narsa} {ism}ga kerak edi."][t]  # ko'chirishni o'rgatadi


def umumiy_korpus(n, seed):
    rng = np.random.default_rng(seed)
    return "\n".join(jumla(rng) for _ in range(n)) + "\n"


def sft_misollar(n, seed, shovqin=0.0):
    """Ko'rsatma -> javob: "<jumla> kim? " -> "anvar." (shovqin - noto'g'ri belgilar ulushi)."""
    rng = np.random.default_rng(seed)
    out = []
    for _ in range(n):
        ism, joy = str(rng.choice(ISMLAR)), str(rng.choice(JOYLAR))
        j = f"{ism} {joy}ga bordi."
        q = str(rng.choice(list(SAVOLLAR)))
        togri = [ism, joy][SAVOLLAR[q]]
        if rng.random() < shovqin:                     # boshqa, noto'g'ri qiymat
            togri = str(rng.choice([x for x in (ISMLAR, JOYLAR)[SAVOLLAR[q]] if x != togri]))
        out.append((f"{j} {q}? ", f"{togri}.\n", [ism, joy][SAVOLLAR[q]]))
    return out


class Blok(nn.Module):
    def __init__(self, d, boshlar):
        super().__init__()
        self.boshlar = boshlar
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))

    def forward(self, x):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=2)
        q, k, v = [t.view(B, T, self.boshlar, -1).transpose(1, 2) for t in (q, k, v)]
        o = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        x = x + self.proj(o.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class GPT(nn.Module):
    def __init__(self, V, d=48, boshlar=4, qatlamlar=2):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.poz = nn.Embedding(L, d)
        self.bloklar = nn.Sequential(*[Blok(d, boshlar) for _ in range(qatlamlar)])
        self.ln = nn.LayerNorm(d)
        self.chiqish = nn.Linear(d, V)

    def forward(self, x):
        x = x[:, -L:]
        h = self.emb(x) + self.poz(torch.arange(x.shape[1]))
        return self.chiqish(self.ln(self.bloklar(h)))


class Lugat:
    def __init__(self, matn):
        self.belgilar = sorted(set(matn))
        self.s2i = {c: i for i, c in enumerate(self.belgilar)}

    def kod(self, s):
        return [self.s2i[c] for c in s]

    def matn(self, ids):
        return "".join(self.belgilar[i] for i in ids)


def pretrain(T, V, qadamlar=300, B=24, seed=0, lr=0.02):
    """Kichik GPT ni umumiy korpusda noldan o'rgatish (keyingi belgi)."""
    torch.manual_seed(seed)
    model = GPT(V)
    opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01)
    jadval = torch.optim.lr_scheduler.LambdaLR(
        opt, lambda s: min(1.0, (s + 1) / 30) * 0.5 * (1 + math.cos(math.pi * s / qadamlar)))
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        bosh = torch.randint(0, len(T) - L - 1, (B,), generator=g)
        w = torch.stack([T[i:i + L + 1] for i in bosh.tolist()])
        loss = F.cross_entropy(model(w[:, :-1]).reshape(-1, V), w[:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
        jadval.step()
    return model.eval()


def sft_batch(misollar, lug, niqob=True):
    """Prompt + javob ni bitta ketma-ketlikka qo'shish; niqob=True - prompt tokenlari loss da -100."""
    x, y = [], []
    for p, j, _ in misollar:
        ids = lug.kod(p + j)
        maqsad = ids[1:] + [-100]
        if niqob:
            maqsad = [-100] * (len(p) - 1) + maqsad[len(p) - 1:]
        x.append(ids)
        y.append(maqsad)
    n = max(len(t) for t in x)
    X = torch.tensor([t + [0] * (n - len(t)) for t in x])
    Y = torch.tensor([t + [-100] * (n - len(t)) for t in y])
    return X, Y


def sft(model, misollar, lug, V, qadamlar=200, B=32, lr=3e-3, niqob=True, seed=0,
        parametrlar=None, qayta=None, T_umumiy=None, kuzat=None):
    """SFT: parametrlar=None - hammasi (to'liq); qayta - umumiy matn ulushi (replay);
    kuzat(qadam, model) - har qadam oxirida chaqiriladi (o'lchov uchun)."""
    torch.manual_seed(seed)
    model.train()
    params = [p for p in (parametrlar or model.parameters()) if p.requires_grad]
    opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
    rng = np.random.default_rng(seed)
    g = torch.Generator().manual_seed(seed)
    for q in range(qadamlar):
        idx = rng.integers(0, len(misollar), B)
        X, Y = sft_batch([misollar[i] for i in idx], lug, niqob)
        loss = F.cross_entropy(model(X).reshape(-1, V), Y.reshape(-1), ignore_index=-100)
        if qayta:
            bosh = torch.randint(0, len(T_umumiy) - L - 1, (B // 2,), generator=g)
            w = torch.stack([T_umumiy[i:i + L + 1] for i in bosh.tolist()])
            loss = (1 - qayta) * loss + qayta * F.cross_entropy(
                model(w[:, :-1]).reshape(-1, V), w[:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(params, 1.0)
        opt.step()
        if kuzat is not None:
            kuzat(q + 1, model)
    return model.eval()


@torch.no_grad()
def javob_ber(model, promptlar, lug, maks=14):
    """Greedy generatsiya (batch), '.' yoki '\\n' gacha."""
    x = torch.tensor([lug.kod(p) for p in promptlar])
    chiqdi = [""] * len(promptlar)
    tugadi = [False] * len(promptlar)
    for _ in range(maks):
        k = model(x)[:, -1].argmax(-1)
        x = torch.cat([x, k[:, None]], 1)
        for i, t in enumerate(k.tolist()):
            c = lug.belgilar[t]
            if not tugadi[i]:
                if c in ".\n":
                    tugadi[i] = True
                else:
                    chiqdi[i] += c
    return chiqdi


def aniqlik(model, test, lug):
    """Test promptlari bir xil uzunlikda emas - uzunlik bo'yicha guruhlab batch qilamiz."""
    natija = np.zeros(len(test))
    guruh = {}
    for i, (p, _, togri) in enumerate(test):
        guruh.setdefault(len(p), []).append(i)
    for idx in guruh.values():
        chiq = javob_ber(model, [test[i][0] for i in idx], lug)
        for i, c in zip(idx, chiq):
            natija[i] = float(c == test[i][2])
    return natija


@torch.no_grad()
def perplexity(model, T, V):
    n = (len(T) - 1) // L
    x = T[:n * L].view(n, L)
    y = T[1:n * L + 1].view(n, L)
    return math.exp(F.cross_entropy(model(x).reshape(-1, V), y.reshape(-1)).item())

def main() -> None:
    torch.set_num_threads(1)
    oquv, val = umumiy_korpus(4000, 0), umumiy_korpus(500, 1)
    lug = Lugat(oquv + val)
    V = len(lug.belgilar)
    T, Tv = torch.tensor(lug.kod(oquv)), torch.tensor(lug.kod(val))

    print("=== 1. Pretraining: umumiy matnda keyingi belgi ===")
    base = pretrain(T, V)
    print(f"  parametrlar {sum(p.numel() for p in base.parameters())}, lug'at {V} belgi, "
          f"kontekst {L}")
    print(f"  umumiy val perplexity: {perplexity(base, Tv, V):.3f}")
    print(f"  namuna korpus: {oquv.splitlines()[0]!r}")

    print("\n=== 2. SFT ma'lumoti: ko'rsatma -> javob va niqob ===")
    oquv_sft, test = sft_misollar(2000, 10), sft_misollar(300, 11)
    p, j, _ = oquv_sft[0]
    print(f"  prompt {p!r} -> javob {j!r}")
    X, Y = sft_batch([oquv_sft[0]], lug, niqob=True)
    korinish = "".join("_" if t == -100 else "^" for t in Y[0].tolist())
    print(f"  {lug.matn(X[0].tolist())!r}")
    print(f"  {korinish!r}  (^ - loss hisoblanadigan maqsad, _ - niqoblangan)")
    _, Y0 = sft_batch(oquv_sft[:200], lug, niqob=False)
    _, Y1 = sft_batch(oquv_sft[:200], lug, niqob=True)
    n0, n1 = int((Y0 != -100).sum()), int((Y1 != -100).sum())
    print(f"  200 misolda loss maqsadlari: niqobsiz {n0}, niqobli {n1} ({n1 / n0:.0%})")

    print("\n=== 3. SFT siz: faqat prompt (zero-shot) ===")
    a0 = aniqlik(base, test, lug)
    chiq = javob_ber(base, [test[0][0]], lug)[0]
    print(f"  aniqlik {a0.mean():.3f}; namuna: {test[0][0]!r} -> {chiq!r}")
    print(f"  kontekst {L} belgi - bir nechta misolli (few-shot) prompt sig'maydi")

    print("\n=== 4. SFT: niqobli, niqobsiz va pretrainingsiz (150 qadam, bir xil batchlar) ===")
    natija = {}
    for nom, bosh, niqob in (("niqobli", base, True), ("niqobsiz", base, False),
                             ("noldan, niqobli", None, True)):
        if bosh is None:
            torch.manual_seed(1)
            m = GPT(V)
        else:
            m = copy.deepcopy(bosh)
        m = sft(m, oquv_sft, lug, V, qadamlar=150, B=24, lr=1e-2, niqob=niqob)
        natija[nom] = aniqlik(m, test, lug)
        print(f"  {nom:<16} aniqlik {natija[nom].mean():.3f}   umumiy PPL {perplexity(m, Tv, V):8.2f}")
    for a, b in (("niqobli", "niqobsiz"), ("niqobli", "noldan, niqobli")):
        d = natija[a] - natija[b]
        se = d.std(ddof=1) / math.sqrt(len(d))
        print(f"  {a} - {b}: {d.mean():+.3f} (SE {se:.3f}, savollar bo'yicha juftlashgan) -> "
              f"{'sezilarli' if abs(d.mean()) > 2 * se else 'sezilarli emas'}")
    print("  ⭐ SFT da loss faqat javob tokenlarida: model promptni emas, javobni o'rganadi")


if __name__ == "__main__":
    main()

=== 1. Pretraining: umumiy matnda keyingi belgi ===
  parametrlar 43036, lug'at 28 belgi, kontekst 48
  umumiy val perplexity: 1.866
  namuna korpus: 'rustam bozorga bordi. bozor rustamga yoqdi.'

=== 2. SFT ma'lumoti: ko'rsatma -> javob va niqob ===
  prompt 'nigora zavodga bordi. kim? ' -> javob 'nigora.\n'
  'nigora zavodga bordi. kim? nigora.\n'
  '__________________________^^^^^^^^_'  (^ - loss hisoblanadigan maqsad, _ - niqoblangan)
  200 misolda loss maqsadlari: niqobsiz 7360, niqobli 1622 (22%)

=== 3. SFT siz: faqat prompt (zero-shot) ===
  aniqlik 0.000; namuna: 'malika maktabga bordi. qayerga? ' -> 'shahlo shaharg'
  kontekst 48 belgi - bir nechta misolli (few-shot) prompt sig'maydi

=== 4. SFT: niqobli, niqobsiz va pretrainingsiz (150 qadam, bir xil batchlar) ===
  niqobli          aniqlik 0.683   umumiy PPL   161.33
  niqobsiz         aniqlik 0.137   umumiy PPL    13.55
  noldan, niqobli  aniqlik 0.673   umumiy PPL   865.00
  niqobli - niqobsiz: +0.547 (SE 0.031, savollar bo'yicha juftlashgan) -> sezilarli
  niqobli - noldan, niqobli: +0.010 (SE 0.034, savollar bo'yicha juftlashgan) -> sezilarli emas
  ⭐ SFT da loss faqat javob tokenlarida: model promptni emas, javobni o'rganadi

"""LoRA va to'liq fine-tuning: o'rgatiladigan parametrlar, vazifa aniqligi va unutish."""

import copy
import math

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab shahar klinika kutubxona choyxona dala bekat "
          "kasalxona stadion muzey teatr vokzal ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch shakar asal soat telefon").split()
SIFATLAR = "katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat".split()
L = 48
SAVOLLAR = {"kim": 0, "qayerga": 1}


def jumla(rng):
    t = int(rng.integers(0, 8))
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = rng.choice(VAQTLAR), rng.choice(JOYLAR)
    narsa, sifat = rng.choice(NARSALAR), rng.choice(SIFATLAR)
    return [f"{ism} {joy}ga {vaqt} bordi.", f"{ism} {joy}dan {vaqt} qaytdi.",
            f"{ism} {sifat} {narsa} sotib oldi.", f"{joy}da {sifat} {narsa} bor.",
            f"{ism} {ism2}ga {sifat} {narsa} berdi.", f"{ism} qayerga bordi?",
            f"{ism} {joy}ga bordi. {joy} {ism}ga yoqdi.",          # takrorlanish: nusxa
            f"{ism} {narsa} oldi. {narsa} {ism}ga kerak edi."][t]  # ko'chirishni o'rgatadi


def umumiy_korpus(n, seed):
    rng = np.random.default_rng(seed)
    return "\n".join(jumla(rng) for _ in range(n)) + "\n"


def sft_misollar(n, seed, shovqin=0.0):
    """Ko'rsatma -> javob: "<jumla> kim? " -> "anvar." (shovqin - noto'g'ri belgilar ulushi)."""
    rng = np.random.default_rng(seed)
    out = []
    for _ in range(n):
        ism, joy = str(rng.choice(ISMLAR)), str(rng.choice(JOYLAR))
        j = f"{ism} {joy}ga bordi."
        q = str(rng.choice(list(SAVOLLAR)))
        togri = [ism, joy][SAVOLLAR[q]]
        if rng.random() < shovqin:                     # boshqa, noto'g'ri qiymat
            togri = str(rng.choice([x for x in (ISMLAR, JOYLAR)[SAVOLLAR[q]] if x != togri]))
        out.append((f"{j} {q}? ", f"{togri}.\n", [ism, joy][SAVOLLAR[q]]))
    return out


class Blok(nn.Module):
    def __init__(self, d, boshlar):
        super().__init__()
        self.boshlar = boshlar
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))

    def forward(self, x):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=2)
        q, k, v = [t.view(B, T, self.boshlar, -1).transpose(1, 2) for t in (q, k, v)]
        o = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        x = x + self.proj(o.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class GPT(nn.Module):
    def __init__(self, V, d=48, boshlar=4, qatlamlar=2):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.poz = nn.Embedding(L, d)
        self.bloklar = nn.Sequential(*[Blok(d, boshlar) for _ in range(qatlamlar)])
        self.ln = nn.LayerNorm(d)
        self.chiqish = nn.Linear(d, V)

    def forward(self, x):
        x = x[:, -L:]
        h = self.emb(x) + self.poz(torch.arange(x.shape[1]))
        return self.chiqish(self.ln(self.bloklar(h)))


class Lugat:
    def __init__(self, matn):
        self.belgilar = sorted(set(matn))
        self.s2i = {c: i for i, c in enumerate(self.belgilar)}

    def kod(self, s):
        return [self.s2i[c] for c in s]

    def matn(self, ids):
        return "".join(self.belgilar[i] for i in ids)


def pretrain(T, V, qadamlar=300, B=24, seed=0, lr=0.02):
    """Kichik GPT ni umumiy korpusda noldan o'rgatish (keyingi belgi)."""
    torch.manual_seed(seed)
    model = GPT(V)
    opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01)
    jadval = torch.optim.lr_scheduler.LambdaLR(
        opt, lambda s: min(1.0, (s + 1) / 30) * 0.5 * (1 + math.cos(math.pi * s / qadamlar)))
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        bosh = torch.randint(0, len(T) - L - 1, (B,), generator=g)
        w = torch.stack([T[i:i + L + 1] for i in bosh.tolist()])
        loss = F.cross_entropy(model(w[:, :-1]).reshape(-1, V), w[:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
        jadval.step()
    return model.eval()


def sft_batch(misollar, lug, niqob=True):
    """Prompt + javob ni bitta ketma-ketlikka qo'shish; niqob=True - prompt tokenlari loss da -100."""
    x, y = [], []
    for p, j, _ in misollar:
        ids = lug.kod(p + j)
        maqsad = ids[1:] + [-100]
        if niqob:
            maqsad = [-100] * (len(p) - 1) + maqsad[len(p) - 1:]
        x.append(ids)
        y.append(maqsad)
    n = max(len(t) for t in x)
    X = torch.tensor([t + [0] * (n - len(t)) for t in x])
    Y = torch.tensor([t + [-100] * (n - len(t)) for t in y])
    return X, Y


def sft(model, misollar, lug, V, qadamlar=200, B=32, lr=3e-3, niqob=True, seed=0,
        parametrlar=None, qayta=None, T_umumiy=None, kuzat=None):
    """SFT: parametrlar=None - hammasi (to'liq); qayta - umumiy matn ulushi (replay);
    kuzat(qadam, model) - har qadam oxirida chaqiriladi (o'lchov uchun)."""
    torch.manual_seed(seed)
    model.train()
    params = [p for p in (parametrlar or model.parameters()) if p.requires_grad]
    opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
    rng = np.random.default_rng(seed)
    g = torch.Generator().manual_seed(seed)
    for q in range(qadamlar):
        idx = rng.integers(0, len(misollar), B)
        X, Y = sft_batch([misollar[i] for i in idx], lug, niqob)
        loss = F.cross_entropy(model(X).reshape(-1, V), Y.reshape(-1), ignore_index=-100)
        if qayta:
            bosh = torch.randint(0, len(T_umumiy) - L - 1, (B // 2,), generator=g)
            w = torch.stack([T_umumiy[i:i + L + 1] for i in bosh.tolist()])
            loss = (1 - qayta) * loss + qayta * F.cross_entropy(
                model(w[:, :-1]).reshape(-1, V), w[:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(params, 1.0)
        opt.step()
        if kuzat is not None:
            kuzat(q + 1, model)
    return model.eval()


@torch.no_grad()
def javob_ber(model, promptlar, lug, maks=14):
    """Greedy generatsiya (batch), '.' yoki '\\n' gacha."""
    x = torch.tensor([lug.kod(p) for p in promptlar])
    chiqdi = [""] * len(promptlar)
    tugadi = [False] * len(promptlar)
    for _ in range(maks):
        k = model(x)[:, -1].argmax(-1)
        x = torch.cat([x, k[:, None]], 1)
        for i, t in enumerate(k.tolist()):
            c = lug.belgilar[t]
            if not tugadi[i]:
                if c in ".\n":
                    tugadi[i] = True
                else:
                    chiqdi[i] += c
    return chiqdi


def aniqlik(model, test, lug):
    """Test promptlari bir xil uzunlikda emas - uzunlik bo'yicha guruhlab batch qilamiz."""
    natija = np.zeros(len(test))
    guruh = {}
    for i, (p, _, togri) in enumerate(test):
        guruh.setdefault(len(p), []).append(i)
    for idx in guruh.values():
        chiq = javob_ber(model, [test[i][0] for i in idx], lug)
        for i, c in zip(idx, chiq):
            natija[i] = float(c == test[i][2])
    return natija


@torch.no_grad()
def perplexity(model, T, V):
    n = (len(T) - 1) // L
    x = T[:n * L].view(n, L)
    y = T[1:n * L + 1].view(n, L)
    return math.exp(F.cross_entropy(model(x).reshape(-1, V), y.reshape(-1)).item())

class LoRALinear(nn.Module):
    """y = W x + (alfa / r) * B A x;  W muzlatilgan, faqat A (r x kirish) va B (chiqish x r)."""

    def __init__(self, asos, r=4, alfa=8):
        super().__init__()
        self.asos = asos
        self.asos.requires_grad_(False)
        self.A = nn.Parameter(torch.randn(r, asos.in_features) / math.sqrt(asos.in_features))
        self.B = nn.Parameter(torch.zeros(asos.out_features, r))        # boshida delta = 0
        self.k = alfa / r

    def forward(self, x):
        return self.asos(x) + (x @ self.A.T @ self.B.T) * self.k

    def birlashtir(self):
        """Adapterni asosiy vaznga qo'shib, oddiy nn.Linear qaytaradi (inference tezligi)."""
        yangi = copy.deepcopy(self.asos)
        yangi.weight.data += self.k * self.B.data @ self.A.data
        return yangi


def lora_qosh(model, r=4):
    m = copy.deepcopy(model)
    m.requires_grad_(False)
    for b in m.bloklar:
        b.qkv, b.proj = LoRALinear(b.qkv, r), LoRALinear(b.proj, r)
        b.ffn[0], b.ffn[2] = LoRALinear(b.ffn[0], r), LoRALinear(b.ffn[2], r)
    return m


def main() -> None:
    torch.set_num_threads(1)
    oquv, val = umumiy_korpus(4000, 0), umumiy_korpus(500, 1)
    lug = Lugat(oquv + val)
    V = len(lug.belgilar)
    T, Tv = torch.tensor(lug.kod(oquv)), torch.tensor(lug.kod(val))
    base = pretrain(T, V)
    oquv_sft, test = sft_misollar(2000, 10), sft_misollar(300, 11)
    ppl0 = perplexity(base, Tv, V)
    jami = sum(p.numel() for p in base.parameters())

    print("=== 1. O'rgatiladigan parametrlar ===")
    lm = lora_qosh(base)
    n_lora = sum(p.numel() for p in lm.parameters() if p.requires_grad)
    print(f"  to'liq fine-tuning: {jami} (100%)")
    print(f"  LoRA r=4 (qkv, proj, ffn - 2 blok): {n_lora} ({n_lora / jami:.1%})")
    print(f"  pretrained model: umumiy PPL {ppl0:.3f}")

    print("\n=== 2. Bir xil 150 qadam: aniqlik va unutish ===")
    natija = {}
    for nom, m, lr in (("to'liq, lr 1e-2", copy.deepcopy(base), 1e-2),
                       ("to'liq, lr 3e-3", copy.deepcopy(base), 3e-3),
                       ("LoRA r=4, lr 1e-2", lm, 1e-2)):
        m = sft(m, oquv_sft, lug, V, qadamlar=150, B=24, lr=lr)
        natija[nom] = (aniqlik(m, test, lug), perplexity(m, Tv, V), m)
        print(f"  {nom:<18} aniqlik {natija[nom][0].mean():.3f}   umumiy PPL {natija[nom][1]:8.2f} "
              f"({natija[nom][1] / ppl0:.1f}x)")
    a, b = natija["to'liq, lr 1e-2"][0], natija["LoRA r=4, lr 1e-2"][0]
    d = a - b
    se = d.std(ddof=1) / math.sqrt(len(d))
    print(f"  to'liq (1e-2) - LoRA: {d.mean():+.3f} (SE {se:.3f}) -> "
          f"{'sezilarli' if abs(d.mean()) > 2 * se else 'sezilarli emas'}")

    print("\n=== 3. LoRA ni birlashtirish (merge) ===")
    m = natija["LoRA r=4, lr 1e-2"][2]
    x = torch.randn(5, 48)
    q = m.bloklar[0].qkv
    with torch.no_grad():
        farq = (q(x) - q.birlashtir()(x)).abs().max().item()
    print(f"  W + (alfa/r) B A bilan bitta Linear: maksimal farq {farq:.1e}")
    for b in m.bloklar:
        b.qkv, b.proj = b.qkv.birlashtir(), b.proj.birlashtir()
        b.ffn[0], b.ffn[2] = b.ffn[0].birlashtir(), b.ffn[2].birlashtir()
    print(f"  birlashtirilgan model aniqligi {aniqlik(m, test, lug).mean():.3f} "
          f"(adapter bilan {natija['LoRA r=4, lr 1e-2'][0].mean():.3f})")
    print(f"  saqlanadigan adapter: {n_lora} son; to'liq nusxa: {jami} son")

    print("\n=== 4. Xulosa (natijadan) ===")
    eng = max(natija, key=lambda k: natija[k][0].mean())
    kam = min(natija, key=lambda k: natija[k][1])
    print(f"  eng yuqori aniqlik: {eng}; eng kam unutish: {kam}")
    print("  ⭐ LoRA - kam parametr va kam unutish; aniqlikni esa o'z vazifangizda o'lchang")


if __name__ == "__main__":
    main()

=== 1. O'rgatiladigan parametrlar ===
  to'liq fine-tuning: 43036 (100%)
  LoRA r=4 (qkv, proj, ffn - 2 blok): 4608 (10.7%)
  pretrained model: umumiy PPL 1.866

=== 2. Bir xil 150 qadam: aniqlik va unutish ===
  to'liq, lr 1e-2    aniqlik 0.683   umumiy PPL   161.33 (86.5x)
  to'liq, lr 3e-3    aniqlik 0.463   umumiy PPL     9.47 (5.1x)
  LoRA r=4, lr 1e-2  aniqlik 0.237   umumiy PPL     7.08 (3.8x)
  to'liq (1e-2) - LoRA: +0.447 (SE 0.029) -> sezilarli

=== 3. LoRA ni birlashtirish (merge) ===
  W + (alfa/r) B A bilan bitta Linear: maksimal farq 1.4e-06
  birlashtirilgan model aniqligi 0.237 (adapter bilan 0.237)
  saqlanadigan adapter: 4608 son; to'liq nusxa: 43036 son

=== 4. Xulosa (natijadan) ===
  eng yuqori aniqlik: to'liq, lr 1e-2; eng kam unutish: LoRA r=4, lr 1e-2
  ⭐ LoRA - kam parametr va kam unutish; aniqlikni esa o'z vazifangizda o'lchang

"""Ma'lumot sifati va miqdori: SFT misollari soni va shovqinli (noto'g'ri) belgilar ulushi."""

import copy
import math

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab shahar klinika kutubxona choyxona dala bekat "
          "kasalxona stadion muzey teatr vokzal ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch shakar asal soat telefon").split()
SIFATLAR = "katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat".split()
L = 48
SAVOLLAR = {"kim": 0, "qayerga": 1}


def jumla(rng):
    t = int(rng.integers(0, 8))
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = rng.choice(VAQTLAR), rng.choice(JOYLAR)
    narsa, sifat = rng.choice(NARSALAR), rng.choice(SIFATLAR)
    return [f"{ism} {joy}ga {vaqt} bordi.", f"{ism} {joy}dan {vaqt} qaytdi.",
            f"{ism} {sifat} {narsa} sotib oldi.", f"{joy}da {sifat} {narsa} bor.",
            f"{ism} {ism2}ga {sifat} {narsa} berdi.", f"{ism} qayerga bordi?",
            f"{ism} {joy}ga bordi. {joy} {ism}ga yoqdi.",          # takrorlanish: nusxa
            f"{ism} {narsa} oldi. {narsa} {ism}ga kerak edi."][t]  # ko'chirishni o'rgatadi


def umumiy_korpus(n, seed):
    rng = np.random.default_rng(seed)
    return "\n".join(jumla(rng) for _ in range(n)) + "\n"


def sft_misollar(n, seed, shovqin=0.0):
    """Ko'rsatma -> javob: "<jumla> kim? " -> "anvar." (shovqin - noto'g'ri belgilar ulushi)."""
    rng = np.random.default_rng(seed)
    out = []
    for _ in range(n):
        ism, joy = str(rng.choice(ISMLAR)), str(rng.choice(JOYLAR))
        j = f"{ism} {joy}ga bordi."
        q = str(rng.choice(list(SAVOLLAR)))
        togri = [ism, joy][SAVOLLAR[q]]
        if rng.random() < shovqin:                     # boshqa, noto'g'ri qiymat
            togri = str(rng.choice([x for x in (ISMLAR, JOYLAR)[SAVOLLAR[q]] if x != togri]))
        out.append((f"{j} {q}? ", f"{togri}.\n", [ism, joy][SAVOLLAR[q]]))
    return out


class Blok(nn.Module):
    def __init__(self, d, boshlar):
        super().__init__()
        self.boshlar = boshlar
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))

    def forward(self, x):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=2)
        q, k, v = [t.view(B, T, self.boshlar, -1).transpose(1, 2) for t in (q, k, v)]
        o = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        x = x + self.proj(o.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class GPT(nn.Module):
    def __init__(self, V, d=48, boshlar=4, qatlamlar=2):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.poz = nn.Embedding(L, d)
        self.bloklar = nn.Sequential(*[Blok(d, boshlar) for _ in range(qatlamlar)])
        self.ln = nn.LayerNorm(d)
        self.chiqish = nn.Linear(d, V)

    def forward(self, x):
        x = x[:, -L:]
        h = self.emb(x) + self.poz(torch.arange(x.shape[1]))
        return self.chiqish(self.ln(self.bloklar(h)))


class Lugat:
    def __init__(self, matn):
        self.belgilar = sorted(set(matn))
        self.s2i = {c: i for i, c in enumerate(self.belgilar)}

    def kod(self, s):
        return [self.s2i[c] for c in s]

    def matn(self, ids):
        return "".join(self.belgilar[i] for i in ids)


def pretrain(T, V, qadamlar=300, B=24, seed=0, lr=0.02):
    """Kichik GPT ni umumiy korpusda noldan o'rgatish (keyingi belgi)."""
    torch.manual_seed(seed)
    model = GPT(V)
    opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01)
    jadval = torch.optim.lr_scheduler.LambdaLR(
        opt, lambda s: min(1.0, (s + 1) / 30) * 0.5 * (1 + math.cos(math.pi * s / qadamlar)))
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        bosh = torch.randint(0, len(T) - L - 1, (B,), generator=g)
        w = torch.stack([T[i:i + L + 1] for i in bosh.tolist()])
        loss = F.cross_entropy(model(w[:, :-1]).reshape(-1, V), w[:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
        jadval.step()
    return model.eval()


def sft_batch(misollar, lug, niqob=True):
    """Prompt + javob ni bitta ketma-ketlikka qo'shish; niqob=True - prompt tokenlari loss da -100."""
    x, y = [], []
    for p, j, _ in misollar:
        ids = lug.kod(p + j)
        maqsad = ids[1:] + [-100]
        if niqob:
            maqsad = [-100] * (len(p) - 1) + maqsad[len(p) - 1:]
        x.append(ids)
        y.append(maqsad)
    n = max(len(t) for t in x)
    X = torch.tensor([t + [0] * (n - len(t)) for t in x])
    Y = torch.tensor([t + [-100] * (n - len(t)) for t in y])
    return X, Y


def sft(model, misollar, lug, V, qadamlar=200, B=32, lr=3e-3, niqob=True, seed=0,
        parametrlar=None, qayta=None, T_umumiy=None, kuzat=None):
    """SFT: parametrlar=None - hammasi (to'liq); qayta - umumiy matn ulushi (replay);
    kuzat(qadam, model) - har qadam oxirida chaqiriladi (o'lchov uchun)."""
    torch.manual_seed(seed)
    model.train()
    params = [p for p in (parametrlar or model.parameters()) if p.requires_grad]
    opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
    rng = np.random.default_rng(seed)
    g = torch.Generator().manual_seed(seed)
    for q in range(qadamlar):
        idx = rng.integers(0, len(misollar), B)
        X, Y = sft_batch([misollar[i] for i in idx], lug, niqob)
        loss = F.cross_entropy(model(X).reshape(-1, V), Y.reshape(-1), ignore_index=-100)
        if qayta:
            bosh = torch.randint(0, len(T_umumiy) - L - 1, (B // 2,), generator=g)
            w = torch.stack([T_umumiy[i:i + L + 1] for i in bosh.tolist()])
            loss = (1 - qayta) * loss + qayta * F.cross_entropy(
                model(w[:, :-1]).reshape(-1, V), w[:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(params, 1.0)
        opt.step()
        if kuzat is not None:
            kuzat(q + 1, model)
    return model.eval()


@torch.no_grad()
def javob_ber(model, promptlar, lug, maks=14):
    """Greedy generatsiya (batch), '.' yoki '\\n' gacha."""
    x = torch.tensor([lug.kod(p) for p in promptlar])
    chiqdi = [""] * len(promptlar)
    tugadi = [False] * len(promptlar)
    for _ in range(maks):
        k = model(x)[:, -1].argmax(-1)
        x = torch.cat([x, k[:, None]], 1)
        for i, t in enumerate(k.tolist()):
            c = lug.belgilar[t]
            if not tugadi[i]:
                if c in ".\n":
                    tugadi[i] = True
                else:
                    chiqdi[i] += c
    return chiqdi


def aniqlik(model, test, lug):
    """Test promptlari bir xil uzunlikda emas - uzunlik bo'yicha guruhlab batch qilamiz."""
    natija = np.zeros(len(test))
    guruh = {}
    for i, (p, _, togri) in enumerate(test):
        guruh.setdefault(len(p), []).append(i)
    for idx in guruh.values():
        chiq = javob_ber(model, [test[i][0] for i in idx], lug)
        for i, c in zip(idx, chiq):
            natija[i] = float(c == test[i][2])
    return natija


@torch.no_grad()
def perplexity(model, T, V):
    n = (len(T) - 1) // L
    x = T[:n * L].view(n, L)
    y = T[1:n * L + 1].view(n, L)
    return math.exp(F.cross_entropy(model(x).reshape(-1, V), y.reshape(-1)).item())

def main() -> None:
    torch.set_num_threads(1)
    oquv, val = umumiy_korpus(4000, 0), umumiy_korpus(500, 1)
    lug = Lugat(oquv + val)
    V = len(lug.belgilar)
    base = pretrain(torch.tensor(lug.kod(oquv)), V)
    test = sft_misollar(300, 11)

    print("=== 1. Tajriba: bir xil 150 qadam, faqat ma'lumot o'zgaradi ===")
    sozlamalar = [("300 toza", 300, 0.0), ("2000 toza", 2000, 0.0),
                  ("2000, 30% shovqin", 2000, 0.3)]
    for nom, n, sh in sozlamalar[2:]:
        m = sft_misollar(n, 10, sh)
        xato = [j != f"{t}.\n" for _, j, t in m]
        namuna = next(p + j.strip() for (p, j, _), x in zip(m, xato) if x)
        print(f"  {nom}: haqiqatan noto'g'ri belgilar {np.mean(xato):.1%}; namuna: {namuna!r}")

    print("\n=== 2. Test aniqligi (300 savol) ===")
    natija = {}
    for nom, n, sh in sozlamalar:
        m = sft(copy.deepcopy(base), sft_misollar(n, 10, sh), lug, V, qadamlar=150, B=24, lr=1e-2)
        natija[nom] = aniqlik(m, test, lug)
        print(f"  {nom:<19} aniqlik {natija[nom].mean():.3f}")

    print("\n=== 3. Juftlashgan farqlar (savollar bo'yicha) ===")
    for a, b in (("2000 toza", "300 toza"), ("2000 toza", "2000, 30% shovqin"),
                 ("300 toza", "2000, 30% shovqin")):
        d = natija[a] - natija[b]
        se = d.std(ddof=1) / math.sqrt(len(d))
        x = ("sezilarli yaxshi" if d.mean() > 2 * se else
             "sezilarli yomon" if d.mean() < -2 * se else "sezilarli farq yo'q")
        print(f"  {a:<10} - {b:<18} {d.mean():+.3f} (SE {se:.3f}) -> {x}")

    print("\n=== 4. Xulosa (natijadan) ===")
    t300, s30 = natija["300 toza"].mean(), natija["2000, 30% shovqin"].mean()
    print(f"  300 toza misol: {t300:.3f}; 2000 ta, lekin 30% noto'g'ri: {s30:.3f}")
    print(f"  shovqin 0 -> 30%: {natija['2000 toza'].mean():.3f} -> {s30:.3f}")
    print("  ⭐ Belgilarni tekshirish ko'pincha yangi misol qo'shishdan foydaliroq")


if __name__ == "__main__":
    main()

=== 1. Tajriba: bir xil 150 qadam, faqat ma'lumot o'zgaradi ===
  2000, 30% shovqin: haqiqatan noto'g'ri belgilar 29.3%; namuna: 'jasur choyxonaga bordi. qayerga? kasalxona.'

=== 2. Test aniqligi (300 savol) ===
  300 toza            aniqlik 0.453
  2000 toza           aniqlik 0.683
  2000, 30% shovqin   aniqlik 0.280

=== 3. Juftlashgan farqlar (savollar bo'yicha) ===
  2000 toza  - 300 toza           +0.230 (SE 0.036) -> sezilarli yaxshi
  2000 toza  - 2000, 30% shovqin  +0.403 (SE 0.033) -> sezilarli yaxshi
  300 toza   - 2000, 30% shovqin  +0.173 (SE 0.040) -> sezilarli yaxshi

=== 4. Xulosa (natijadan) ===
  300 toza misol: 0.453; 2000 ta, lekin 30% noto'g'ri: 0.280
  shovqin 0 -> 30%: 0.683 -> 0.280
  ⭐ Belgilarni tekshirish ko'pincha yangi misol qo'shishdan foydaliroq

"""Katastrofik unutish: SFT qadamlari bo'yicha umumiy perplexity va replay (umumiy matn aralashmasi)."""

import copy
import math

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab shahar klinika kutubxona choyxona dala bekat "
          "kasalxona stadion muzey teatr vokzal ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch shakar asal soat telefon").split()
SIFATLAR = "katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat".split()
L = 48
SAVOLLAR = {"kim": 0, "qayerga": 1}


def jumla(rng):
    t = int(rng.integers(0, 8))
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = rng.choice(VAQTLAR), rng.choice(JOYLAR)
    narsa, sifat = rng.choice(NARSALAR), rng.choice(SIFATLAR)
    return [f"{ism} {joy}ga {vaqt} bordi.", f"{ism} {joy}dan {vaqt} qaytdi.",
            f"{ism} {sifat} {narsa} sotib oldi.", f"{joy}da {sifat} {narsa} bor.",
            f"{ism} {ism2}ga {sifat} {narsa} berdi.", f"{ism} qayerga bordi?",
            f"{ism} {joy}ga bordi. {joy} {ism}ga yoqdi.",          # takrorlanish: nusxa
            f"{ism} {narsa} oldi. {narsa} {ism}ga kerak edi."][t]  # ko'chirishni o'rgatadi


def umumiy_korpus(n, seed):
    rng = np.random.default_rng(seed)
    return "\n".join(jumla(rng) for _ in range(n)) + "\n"


def sft_misollar(n, seed, shovqin=0.0):
    """Ko'rsatma -> javob: "<jumla> kim? " -> "anvar." (shovqin - noto'g'ri belgilar ulushi)."""
    rng = np.random.default_rng(seed)
    out = []
    for _ in range(n):
        ism, joy = str(rng.choice(ISMLAR)), str(rng.choice(JOYLAR))
        j = f"{ism} {joy}ga bordi."
        q = str(rng.choice(list(SAVOLLAR)))
        togri = [ism, joy][SAVOLLAR[q]]
        if rng.random() < shovqin:                     # boshqa, noto'g'ri qiymat
            togri = str(rng.choice([x for x in (ISMLAR, JOYLAR)[SAVOLLAR[q]] if x != togri]))
        out.append((f"{j} {q}? ", f"{togri}.\n", [ism, joy][SAVOLLAR[q]]))
    return out


class Blok(nn.Module):
    def __init__(self, d, boshlar):
        super().__init__()
        self.boshlar = boshlar
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))

    def forward(self, x):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=2)
        q, k, v = [t.view(B, T, self.boshlar, -1).transpose(1, 2) for t in (q, k, v)]
        o = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        x = x + self.proj(o.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class GPT(nn.Module):
    def __init__(self, V, d=48, boshlar=4, qatlamlar=2):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.poz = nn.Embedding(L, d)
        self.bloklar = nn.Sequential(*[Blok(d, boshlar) for _ in range(qatlamlar)])
        self.ln = nn.LayerNorm(d)
        self.chiqish = nn.Linear(d, V)

    def forward(self, x):
        x = x[:, -L:]
        h = self.emb(x) + self.poz(torch.arange(x.shape[1]))
        return self.chiqish(self.ln(self.bloklar(h)))


class Lugat:
    def __init__(self, matn):
        self.belgilar = sorted(set(matn))
        self.s2i = {c: i for i, c in enumerate(self.belgilar)}

    def kod(self, s):
        return [self.s2i[c] for c in s]

    def matn(self, ids):
        return "".join(self.belgilar[i] for i in ids)


def pretrain(T, V, qadamlar=300, B=24, seed=0, lr=0.02):
    """Kichik GPT ni umumiy korpusda noldan o'rgatish (keyingi belgi)."""
    torch.manual_seed(seed)
    model = GPT(V)
    opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01)
    jadval = torch.optim.lr_scheduler.LambdaLR(
        opt, lambda s: min(1.0, (s + 1) / 30) * 0.5 * (1 + math.cos(math.pi * s / qadamlar)))
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        bosh = torch.randint(0, len(T) - L - 1, (B,), generator=g)
        w = torch.stack([T[i:i + L + 1] for i in bosh.tolist()])
        loss = F.cross_entropy(model(w[:, :-1]).reshape(-1, V), w[:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
        jadval.step()
    return model.eval()


def sft_batch(misollar, lug, niqob=True):
    """Prompt + javob ni bitta ketma-ketlikka qo'shish; niqob=True - prompt tokenlari loss da -100."""
    x, y = [], []
    for p, j, _ in misollar:
        ids = lug.kod(p + j)
        maqsad = ids[1:] + [-100]
        if niqob:
            maqsad = [-100] * (len(p) - 1) + maqsad[len(p) - 1:]
        x.append(ids)
        y.append(maqsad)
    n = max(len(t) for t in x)
    X = torch.tensor([t + [0] * (n - len(t)) for t in x])
    Y = torch.tensor([t + [-100] * (n - len(t)) for t in y])
    return X, Y


def sft(model, misollar, lug, V, qadamlar=200, B=32, lr=3e-3, niqob=True, seed=0,
        parametrlar=None, qayta=None, T_umumiy=None, kuzat=None):
    """SFT: parametrlar=None - hammasi (to'liq); qayta - umumiy matn ulushi (replay);
    kuzat(qadam, model) - har qadam oxirida chaqiriladi (o'lchov uchun)."""
    torch.manual_seed(seed)
    model.train()
    params = [p for p in (parametrlar or model.parameters()) if p.requires_grad]
    opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
    rng = np.random.default_rng(seed)
    g = torch.Generator().manual_seed(seed)
    for q in range(qadamlar):
        idx = rng.integers(0, len(misollar), B)
        X, Y = sft_batch([misollar[i] for i in idx], lug, niqob)
        loss = F.cross_entropy(model(X).reshape(-1, V), Y.reshape(-1), ignore_index=-100)
        if qayta:
            bosh = torch.randint(0, len(T_umumiy) - L - 1, (B // 2,), generator=g)
            w = torch.stack([T_umumiy[i:i + L + 1] for i in bosh.tolist()])
            loss = (1 - qayta) * loss + qayta * F.cross_entropy(
                model(w[:, :-1]).reshape(-1, V), w[:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(params, 1.0)
        opt.step()
        if kuzat is not None:
            kuzat(q + 1, model)
    return model.eval()


@torch.no_grad()
def javob_ber(model, promptlar, lug, maks=14):
    """Greedy generatsiya (batch), '.' yoki '\\n' gacha."""
    x = torch.tensor([lug.kod(p) for p in promptlar])
    chiqdi = [""] * len(promptlar)
    tugadi = [False] * len(promptlar)
    for _ in range(maks):
        k = model(x)[:, -1].argmax(-1)
        x = torch.cat([x, k[:, None]], 1)
        for i, t in enumerate(k.tolist()):
            c = lug.belgilar[t]
            if not tugadi[i]:
                if c in ".\n":
                    tugadi[i] = True
                else:
                    chiqdi[i] += c
    return chiqdi


def aniqlik(model, test, lug):
    """Test promptlari bir xil uzunlikda emas - uzunlik bo'yicha guruhlab batch qilamiz."""
    natija = np.zeros(len(test))
    guruh = {}
    for i, (p, _, togri) in enumerate(test):
        guruh.setdefault(len(p), []).append(i)
    for idx in guruh.values():
        chiq = javob_ber(model, [test[i][0] for i in idx], lug)
        for i, c in zip(idx, chiq):
            natija[i] = float(c == test[i][2])
    return natija


@torch.no_grad()
def perplexity(model, T, V):
    n = (len(T) - 1) // L
    x = T[:n * L].view(n, L)
    y = T[1:n * L + 1].view(n, L)
    return math.exp(F.cross_entropy(model(x).reshape(-1, V), y.reshape(-1)).item())

NUQTALAR = (25, 50, 100, 150)


def main() -> None:
    torch.set_num_threads(1)
    oquv, val = umumiy_korpus(4000, 0), umumiy_korpus(500, 1)
    lug = Lugat(oquv + val)
    V = len(lug.belgilar)
    T, Tv = torch.tensor(lug.kod(oquv)), torch.tensor(lug.kod(val))
    base = pretrain(T, V)
    oquv_sft, test = sft_misollar(2000, 10), sft_misollar(150, 11)
    ppl0 = perplexity(base, Tv, V)

    print("=== 1. Boshlang'ich holat ===")
    print(f"  pretrained: umumiy PPL {ppl0:.3f}, vazifa aniqligi {aniqlik(base, test, lug).mean():.3f}")

    print("\n=== 2. Qadamlar bo'yicha: faqat SFT va SFT + 30% umumiy matn (replay) ===")
    tarix, oxirgi = {}, {}
    for nom, qayta in (("faqat SFT", None), ("SFT + replay", 0.3)):
        tarix[nom] = {}

        def kuzat(q, m, nom=nom):
            if q in NUQTALAR:
                m.eval()
                tarix[nom][q] = (aniqlik(m, test, lug), perplexity(m, Tv, V))
                m.train()

        oxirgi[nom] = sft(copy.deepcopy(base), oquv_sft, lug, V, qadamlar=max(NUQTALAR), B=24,
                          lr=1e-2, qayta=qayta, T_umumiy=T, kuzat=kuzat)
    print("  qadam   faqat SFT: aniqlik / PPL     SFT + replay: aniqlik / PPL")
    for q in NUQTALAR:
        a, b = tarix["faqat SFT"][q], tarix["SFT + replay"][q]
        print(f"  {q:>5}   {a[0].mean():>13.3f} / {a[1]:>7.2f}   {b[0].mean():>16.3f} / {b[1]:>6.2f}")

    print(f"\n=== 3. Namuna: umumiy matnni davom ettirish (greedy, {max(NUQTALAR)}-qadam) ===")
    boshlanish = "rustam bozorga bordi. "
    for nom, m in (("pretrained", base), *oxirgi.items()):
        x = torch.tensor([lug.kod(boshlanish)])
        with torch.no_grad():
            for _ in range(24):
                x = torch.cat([x, m(x)[:, -1].argmax(-1, keepdim=True)], 1)
        print(f"  {nom:<13} {lug.matn(x[0].tolist())!r}")

    n = max(NUQTALAR)
    print(f"\n=== 4. Xulosa (natijadan, {n}-qadam) ===")
    a, b = tarix["faqat SFT"][n], tarix["SFT + replay"][n]
    d = a[0] - b[0]
    se = d.std(ddof=1) / math.sqrt(len(d))
    print(f"  aniqlik farqi (faqat SFT - replay): {d.mean():+.3f} (SE {se:.3f}) -> "
          f"{'sezilarli' if abs(d.mean()) > 2 * se else 'sezilarli emas'}")
    print(f"  umumiy PPL: {ppl0:.2f} -> faqat SFT {a[1]:.2f} ({a[1] / ppl0:.0f}x), "
          f"replay {b[1]:.2f} ({b[1] / ppl0:.1f}x)")
    print("  ⭐ Fine-tuning eski bilimni buzishi mumkin: umumiy o'lchovni ham kuzating")


if __name__ == "__main__":
    main()

=== 1. Boshlang'ich holat ===
  pretrained: umumiy PPL 1.866, vazifa aniqligi 0.000

=== 2. Qadamlar bo'yicha: faqat SFT va SFT + 30% umumiy matn (replay) ===
  qadam   faqat SFT: aniqlik / PPL     SFT + replay: aniqlik / PPL
     25           0.087 /   19.15              0.120 /   2.57
     50           0.133 /   29.24              0.087 /   2.26
    100           0.380 /   50.45              0.293 /   2.12
    150           0.707 /  161.33              0.433 /   2.12

=== 3. Namuna: umumiy matnni davom ettirish (greedy, 150-qadam) ===
  pretrained    'rustam bozorga bordi. sherzod kichik shakar bo'
  faqat SFT     'rustam bozorga bordi. rustabtabtabtab.\nahlo.\na'
  SFT + replay  'rustam bozorga bordi. feruza kasalxonaga bordi'

=== 4. Xulosa (natijadan, 150-qadam) ===
  aniqlik farqi (faqat SFT - replay): +0.273 (SE 0.045) -> sezilarli
  umumiy PPL: 1.87 -> faqat SFT 161.33 (86x), replay 2.12 (1.1x)
  ⭐ Fine-tuning eski bilimni buzishi mumkin: umumiy o'lchovni ham kuzating

Y = ids[1:] + [-100]                                                # ⚠️ prompt ham loss da
Y = [-100] * (len(prompt) - 1) + (ids[1:] + [-100])[len(prompt) - 1:]   # ✅

Y = [-100] * len(prompt) + ids[len(prompt) + 1:] + [-100]           # ⚠️ javobning 1-belgisi yo'q
Y = [-100] * (len(prompt) - 1) + ids[len(prompt):] + [-100]         # ✅

javob = "anvar"                                                     # ⚠️ model to'xtamaydi
javob = "anvar.\n"                                                  # ✅ EOS / tugash belgisi

print(aniqlik(model, test))                                          # ⚠️
print(aniqlik(model, test), perplexity(model, umumiy_val))           # ✅ unutishni ham

opt = torch.optim.AdamW(model.parameters())                          # ⚠️ hammasi o'rganadi
model.requires_grad_(False); ...; opt = torch.optim.AdamW(
    [p for p in model.parameters() if p.requires_grad])              # ✅ faqat A, B

sft(model, hamma_misollar)                                           # ⚠️
namuna = random.sample(hamma_misollar, 100)   # qo'lda tekshirish, xato ulushi   # ✅

oquv = barcha_misollar                                               # ⚠️ oqib ketish
oquv = [m for m in barcha_misollar if normal(m["prompt"]) not in eval_promptlar]   # ✅

1.  # kompaniya narxlari har hafta o'zgaradi - fine-tuning mi, RAG mi?
2.  # model JSON formatini 5% holatda buzadi, prompt yordam bermadi - nima?
3.  # prompt 27 belgi, javob 8 belgi - niqobli loss nechta maqsad?
4.  # ignore_index=-100 nima qiladi?
5.  # LoRA da B nega noldan boshlanadi?
6.  # r = 4, qatlam 48 -> 144: LoRA parametrlari?
7.  # merge dan keyin inference tezligi?
8.  # katastrofik unutishni qanday o'lchaysiz?
9.  # replay ning narxi?
10. # 30% xatoli belgilar aniqlikni necha foiz tushiradi deb kutasiz?
11. # DPO da ref model nima uchun kerak?
12. # nega eval ko'rsatmalari SFT ma'lumotida bo'lmasligi kerak?

1.  maqsad = ids[1:] + [-100]
    loss = F.cross_entropy(logit.reshape(-1, V), torch.tensor(maqsad))

2.  lora = lora_qosh(model)
    opt = torch.optim.AdamW(model.parameters(), lr=1e-2)

3.  sft(model, misollar)
    print("tayyor:", aniqlik(model, test))

4.  misollar = [(p, j) for p, j in xom_misollar]       # j = "anvar"
    sft(model, misollar)

5.  natija_lora = sft(lora_model, qadamlar=50)
    natija_toliq = sft(toliq_model, qadamlar=50)
    print("LoRA yomon" if natija_lora < natija_toliq else "LoRA yaxshi")