23.10-dars: Til modeli

MAQSAD: matnga ehtimol berish
  P("bozorga") = P(b) * P(o | b) * P(z | bo) * P(o | boz) * ... * P(a | bozorg)

ZANJIR QOIDASI (9-qism, ehtimollik):
  P(x_1, ..., x_N) = P(x_1) * P(x_2 | x_1) * ... * P(x_N | x_1 ... x_{N-1})
  bu TAXMINSIZ tenglik - istalgan taqsimot uchun to'g'ri

DEMAK TIL MODELI = KEYINGI BELGI KLASSIFIKATORI:
  kirish: oldingi belgilar (kontekst)
  chiqish: V ta belgi bo'yicha ehtimollar (softmax)
  yo'qotish: cross-entropy (14-qism, klassifikatsiya)

MODELLAR FARQI - KONTEKSTNI QANCHA "KO'RADI":
  unigramm:  P(x_t)                   - kontekst yo'q
  bigramm:   P(x_t | x_{t-1})         - 1 belgi
  n-gramm:   P(x_t | oxirgi n-1 belgi)
  RNN/LSTM:  P(x_t | h_{t-1})         - h butun o'tmishni siqib saqlaydi

BELGI DARAJASI:
  lug'at kichik (bu darsda V = 29), noma'lum so'z muammosi yo'q
  lekin ketma-ketlik uzun: so'z o'rniga 5-10 qadam
  (23.2-23.3: so'z va subword tokenizatsiyasi - boshqa murosa)

BIGRAMM - SANASH:
  P(b | a) = son(a, b) / son(a)
  "q" dan keyin: 'a' 0.300, ' ' 0.268, 'i' 0.241 ...  (1-misol)

NOL MUAMMOSI:
  o'quvda uchramagan juftlik -> P = 0 -> log 0 = -cheksiz
  bitta ko'rilmagan juftlik butun val perplexity ni cheksiz qiladi

SILLIQLASH (add-k, Laplace k = 1):
  P(b | a) = (son(a, b) + k) / (son(a) + k * V)
  har hodisaga "k ta soxta kuzatish" qo'shiladi
  k katta -> ko'rilgan hodisalardan ehtimol "o'g'irlanadi"
  k kichik -> ko'rilmagan hodisa juda kichik ehtimol oladi
  k ni VAL da tanlanadi (2-misol: k = 0.01 eng yaxshi)

n ORTISHI BILAN:
  kontekst uzayadi -> o'quvda aniqroq (o'quv PPL pasayadi)
  kontekstlar soni eksponent o'sadi -> ko'pi 1-2 marta uchraydi
  val da ko'rilmagan kontekstlar ko'payadi -> val PPL O'SADI
  bu - 12-qismdagi ortiqcha moslashish, sanash ko'rinishida

ASOSIY CHEKLOV:
  n-gramm "bozor" va "bozorga" ni bog'liq deb bilmaydi -
  har kontekst alohida hisob; umumlashtirish yo'q

CROSS-ENTROPY (val matnida, N ta bashorat):
  CE = -(1/N) * yig'indi_t log P(x_t | kontekst)      (natural log - nat)
  bit da: CE / ln 2

PERPLEXITY:
  PPL = exp(CE) = (P(x_1) * P(x_2 | ...) * ... * P(x_N | ...))^(-1/N)
  ya'ni o'rtacha ehtimolning geometrik o'rtachasiga teskari son

TALQIN - "SAMARALI TANLOV SONI":
  tekis model: har belgi 1/V -> PPL = V (bu darsda 29)
  PPL = 7.8 -> model go'yo har qadamda 7.8 ta teng variant orasidan tanlaydi
  PPL = 1   -> mukammal, har belgi aniq bilinadi
  pastki chegara - matnning o'z noaniqligi (bu korpusda ism, joy tanlovi)

QO'LDA (1-misol, bigramm, " bozorga"):
  7 ta ehtimol -> -log p lar o'rtachasi = CE
  exp(CE) va (p1 * ... * p7)^(-1/7) aynan bir xil

DIQQAT:
  PPL ni faqat BIR XIL lug'at va tokenizatsiyada taqqoslash mumkin
  belgi PPL va so'z PPL - turli o'lchov birliklari
  o'rgatilmagan model: CE ~ ln V (logitlar ~ 0 -> deyarli tekis)

ARXITEKTURA:
  Embedding(V, 32) -> LSTM(32, 64) -> Linear(64, V)
  har pozitsiyada logit: (B, L, V)

O'QUV JUFTLIGI - BIR BELGIGA SURILGAN OYNA:
  matn:   "oldi.\ntil qil"
  kirish: "oldi.\ntil qi"     (w[:-1])
  nishon: "ldi.\ntil qil"     (w[1:])
  bitta oynadan L ta bashorat - hammasi parallel

TEACHER FORCING:
  o'rgatishda t-qadamda modelga HAQIQIY x_{t-1} beriladi
  (modelning o'z bashorati emas)
  + barqaror va tez: xato to'planmaydi, butun oyna bitta forward
  - o'rgatish va generatsiya farqli: generatsiyada model O'Z xatolari
    ustiga quradi (exposure bias - seq2seq darsida batafsil)

YO'QOTISH:
  cross_entropy(logit.reshape(-1, V), nishon.reshape(-1))
  bu aynan 2.3 dagi CE -> exp(loss) = o'quv PPL

NIMA UCHUN n-GRAMMDAN YAXSHI:
  embedding va yashirin holat - o'xshash kontekstlar o'xshash vektor
  h uzoq kontekstni saqlaydi: "-ga ... bordi" / "-dan ... qaytdi"
  (3-misol: 10 belgidan uzoq bog'liqlik)

GENERATSIYA SIKLI:
  x = "\n"  (jumla boshi), holat = None
  takrorla:
    logit, holat = model(x, holat)       # faqat oxirgi belgi va holat
    keyingi = TANLA(logit)
    x = keyingi; "\n" kelsa - to'xta

GREEDY:  keyingi = argmax(logit)
  deterministik, eng ehtimolli yo'l
  har safar AYNAN bir xil jumla (4-misol: 300 tadan 1 noyob)

NAMUNA OLISH (sampling):
  p = softmax(logit / T);  keyingi ~ p
  T < 1: taqsimot o'tkirlashadi -> ehtimolli belgilar ustun, xato kam
  T = 1: modelning o'z taqsimoti
  T > 1: taqsimot tekislashadi -> kam ehtimolli "dum" ko'p tanlanadi

TOP-k:
  faqat eng ehtimolli k ta belgi qoladi, qolganlari -cheksiz
  keyin qayta normallash va namuna olish
  kamchiligi - k QAT'IY: taqsimot tekis bo'lganda (so'z boshida 28 ism)
  to'g'ri variantlarni ham kesadi; moslashuvchan varianti - top-p (nucleus)

DETERMINIZM:
  g = torch.Generator().manual_seed(0)
  torch.multinomial(p, 1, generator=g)
  global seed ga bog'liq emas - natija takrorlanadi

PERPLEXITY YETMAYDI:
  PPL - bir qadamlik bashorat sifati (teacher forcing sharoitida)
  generatsiya - uzun zanjir: kichik xatolar to'planadi

SINTETIK KORPUSDA ANIQ O'LCHOVLAR:
  lug'atda   - yaratilgan so'zlarning o'quv lug'atidagi ulushi (imlo)
  shablon    - jumla korpus grammatikasiga to'liq mosmi (regex)
  noyob      - noyob jumlalar ulushi (xilma-xillik)
  o'quvda bor - aynan o'quv jumlasini takrorlash (yodlash)

REAL MATNDA:
  to'g'rilik uchun: lug'at ulushi, grammatika tekshiruvi, odam bahosi
  xilma-xillik: distinct-n (noyob n-grammlar ulushi)
  yodlash: o'quv to'plami bilan ustma-ust tushish

MUROSA (4-misol):
  greedy: 0.3% noyob; so'zlari 100% lug'atda, lekin jumla shablonga mos EMAS
  T=0.5: 79.7% mos, 85.3% noyob
  T=1.5: 100% noyob, shablonga mos 3.0%
  o'rtada - T = 0.5..1.0 yoki top-k/top-p

import math
import torch
import torch.nn as nn

# lug'at va kodlash
lugat = sorted(set(matn))
s2i = {c: i for i, c in enumerate(lugat)}
T = torch.tensor([s2i[c] for c in matn])

# teacher forcing oynasi
w = T[i:i + L + 1]
x, y = w[:-1], w[1:]

class TilModeli(nn.Module):
    def __init__(self, V, d=32, h=128):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.lstm = nn.LSTM(d, h, batch_first=True)
        self.chiqish = nn.Linear(h, V)

    def forward(self, x, holat=None):
        o, holat = self.lstm(self.emb(x), holat)
        return self.chiqish(o), holat

logit, _ = model(x)                                   # (B, L, V)
loss = nn.functional.cross_entropy(logit.reshape(-1, V), y.reshape(-1))
ppl = math.exp(loss.item())

# generatsiya
g = torch.Generator().manual_seed(0)
logit, holat = model(x_oxirgi, holat)                 # x_oxirgi: (B, 1)
logit = logit[:, -1] / temp
chegara = logit.topk(k, dim=1).values[:, -1:]         # top-k
logit = logit.masked_fill(logit < chegara, float("-inf"))
keyingi = torch.multinomial(torch.softmax(logit, 1), 1, generator=g)

til modeli = har qadamdagi keyingi belgi klassifikatori
PPL = exp(CE); tekis model PPL = V
n-gramm: sanash + silliqlash; katta n - siyraklik
LSTM: teacher forcing, butun o'tmish holatda
generatsiya: greedy (bir xil) <-> yuqori T (xatoli); o'rtasi - T < 1, top-k
to'g'rilik va xilma-xillikni birga o'lchang

"""Belgi darajasidagi korpus, bigramm modeli va perplexity ni qo'lda hisoblash."""

import math
from collections import Counter

import numpy as np

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab bog' shahar qishloq kutubxona do'kon universitet "
          "choyxona dala bekat kasalxona stadion muzey teatr vokzal "
          "ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch go'sht shakar asal pishloq ko'ylak "
            "do'ppi soat telefon").split()
SIFATLAR = ("katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat "
            "yashil sariq issiq sovuq mazali").split()
MAQOLLAR = ["sabr tagi sariq oltin.", "til qilichdan o'tkir.",
            "vaqt oltindan qimmat.", "ko'p o'qigan ko'p biladi.",
            "mehnat qilsang rohat ko'rasan.", "yaxshi so'z jon ozig'i.",
            "daraxt mevasidan odam mehnatidan.",
            "ilm olish igna bilan quduq qazish."]


def jumla(rng):
    t = rng.integers(0, 7)
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = rng.choice(VAQTLAR), rng.choice(JOYLAR)
    narsa, sifat = rng.choice(NARSALAR), rng.choice(SIFATLAR)
    if t == 0:
        return f"{ism} {joy}ga {vaqt} bordi."
    if t == 1:
        return f"{ism} {joy}dan {vaqt} qaytdi."
    if t == 2:
        return f"{ism} {sifat} {narsa} sotib oldi."
    if t == 3:
        return f"{joy}da {sifat} {narsa} bor."
    if t == 4:
        return f"{ism} {ism2}ga {sifat} {narsa} berdi."
    if t == 5:
        return f"{ism} {joy}dan {narsa} olib keldi."
    return str(rng.choice(MAQOLLAR))


def korpus(n, seed):
    rng = np.random.default_rng(seed)
    return "\n".join(jumla(rng) for _ in range(n)) + "\n"


def main() -> None:
    oquv, val = korpus(2000, 0), korpus(400, 1)
    lugat = sorted(set(oquv + val))
    V = len(lugat)

    print("=== 1. Korpus ===")
    for j in oquv.split("\n")[:5]:
        print(f"  {j}")
    jumlalar_o = set(oquv.strip().split("\n"))
    jumlalar_v = val.strip().split("\n")
    print(f"  o'quv: {len(jumlalar_o)} noyob jumla, {len(oquv)} belgi")
    print(f"  val:   {len(jumlalar_v)} jumla, {len(val)} belgi")
    print(f"  belgilar lug'ati V = {V}: {''.join(lugat[1:])!r} + '\\n'")
    sozlar = set(oquv.replace(".", " ").split())
    print(f"  so'zlar lug'ati: {len(sozlar)} ta so'z")
    bor = np.mean([j in jumlalar_o for j in jumlalar_v])
    print(f"  val jumlalaridan o'quvda aynan bori: {bor:.1%}")

    print("\n=== 2. Bigramm: sanash ===")
    k = 0.01
    juft = Counter(zip(oquv[:-1], oquv[1:]))
    oldin = Counter(oquv[:-1])

    def p_bigramm(a, b):
        return (juft[(a, b)] + k) / (oldin[a] + k * V)

    for a in ["q", "'", "."]:
        top = sorted(lugat, key=lambda b: -p_bigramm(a, b))[:4]
        print(f"  P(. | {a!r}): " + ", ".join(f"{b!r} {p_bigramm(a, b):.3f}"
                                             for b in top))
    print(f"  qator yig'indisi P(. | 'q'): "
          f"{sum(p_bigramm('q', b) for b in lugat):.6f}")

    print("\n=== 3. Perplexity qo'lda: 'bozorga' so'zi ===")
    soz = " bozorga"
    loglar = []
    for a, b in zip(soz[:-1], soz[1:]):
        p = p_bigramm(a, b)
        loglar.append(math.log(p))
        print(f"  P({b!r} | {a!r}) = {p:.4f}   -log p = {-math.log(p):.4f}")
    ce = -np.mean(loglar)
    print(f"  cross-entropy = o'rtacha(-log p) = {ce:.4f} nat "
          f"= {ce / math.log(2):.4f} bit")
    print(f"  perplexity = exp(CE) = {math.exp(ce):.4f}")
    kopaytma = math.prod(math.exp(v) for v in loglar)
    print(f"  (p1 * ... * pN)^(-1/N) = {kopaytma ** (-1 / len(loglar)):.4f}")

    print("\n=== 4. Val perplexity: uch bazaviy model ===")
    birlik = Counter(oquv)
    n = len(val) - 1

    def ppl(p_fn):
        return math.exp(-sum(math.log(p_fn(a, b))
                             for a, b in zip(val[:-1], val[1:])) / n)

    natija = {
        "tekis (1/V)": ppl(lambda a, b: 1 / V),
        "unigramm": ppl(lambda a, b: (birlik[b] + k) / (len(oquv) + k * V)),
        "bigramm": ppl(p_bigramm),
    }
    for nom, v in natija.items():
        print(f"  {nom:<12} PPL {v:>8.3f}   CE {math.log(v):.4f}")
    print(f"  tekis model PPL = V = {V}: {abs(natija['tekis (1/V)'] - V) < 1e-9}")
    print(f"  bigramm 'tanlov' ni {V} tadan {natija['bigramm']:.1f} taga "
          f"qisqartirdi")
    print("  ⭐ Perplexity - model har qadamda nechta belgi orasida 'ikkilanadi'")


if __name__ == "__main__":
    main()

=== 1. Korpus ===
  rustam bozordan olma olib keldi.
  nigora bekatdan tushda qaytdi.
  rustam sevaraga katta go'sht berdi.
  sevara yangi ko'ylak sotib oldi.
  feruza kutubxonaga ertalab bordi.
  o'quv: 1694 noyob jumla, 62827 belgi
  val:   400 jumla, 12546 belgi
  belgilar lug'ati V = 29: " '.abcdefghijklmnopqrstuvxyz" + '\n'
  so'zlar lug'ati: 194 ta so'z
  val jumlalaridan o'quvda aynan bori: 19.2%

=== 2. Bigramm: sanash ===
  P(. | 'q'): 'a' 0.300, ' ' 0.268, 'i' 0.241, 'u' 0.085
  P(. | "'"): 'p' 0.232, 'k' 0.114, 'y' 0.108, 's' 0.095
  P(. | '.'): '\n' 1.000, ' ' 0.000, "'" 0.000, '.' 0.000
  qator yig'indisi P(. | 'q'): 1.000000

=== 3. Perplexity qo'lda: 'bozorga' so'zi ===
  P('b' | ' ') = 0.1656   -log p = 1.7984
  P('o' | 'b') = 0.3006   -log p = 1.2018
  P('z' | 'o') = 0.0381   -log p = 3.2665
  P('o' | 'z') = 0.1610   -log p = 1.8263
  P('r' | 'o') = 0.1836   -log p = 1.6948
  P('g' | 'r') = 0.0357   -log p = 3.3332
  P('a' | 'g') = 0.5553   -log p = 0.5883
  cross-entropy = o'rtacha(-log p) = 1.9585 nat = 2.8255 bit
  perplexity = exp(CE) = 7.0884
  (p1 * ... * pN)^(-1/N) = 7.0884

=== 4. Val perplexity: uch bazaviy model ===
  tekis (1/V)  PPL   29.000   CE 3.3673
  unigramm     PPL   21.090   CE 3.0488
  bigramm      PPL    7.767   CE 2.0499
  tekis model PPL = V = 29: True
  bigramm 'tanlov' ni 29 tadan 7.8 taga qisqartirdi
  ⭐ Perplexity - model har qadamda nechta belgi orasida 'ikkilanadi'

"""n-gramm tartibi: o'quv va val perplexity, siyraklik va silliqlash."""

import math
from collections import Counter, defaultdict

import numpy as np

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab bog' shahar qishloq kutubxona do'kon universitet "
          "choyxona dala bekat kasalxona stadion muzey teatr vokzal "
          "ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch go'sht shakar asal pishloq ko'ylak "
            "do'ppi soat telefon").split()
SIFATLAR = ("katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat "
            "yashil sariq issiq sovuq mazali").split()
MAQOLLAR = ["sabr tagi sariq oltin.", "til qilichdan o'tkir.",
            "vaqt oltindan qimmat.", "ko'p o'qigan ko'p biladi.",
            "mehnat qilsang rohat ko'rasan.", "yaxshi so'z jon ozig'i.",
            "daraxt mevasidan odam mehnatidan.",
            "ilm olish igna bilan quduq qazish."]


def jumla(rng):
    t = rng.integers(0, 7)
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = rng.choice(VAQTLAR), rng.choice(JOYLAR)
    narsa, sifat = rng.choice(NARSALAR), rng.choice(SIFATLAR)
    if t == 0:
        return f"{ism} {joy}ga {vaqt} bordi."
    if t == 1:
        return f"{ism} {joy}dan {vaqt} qaytdi."
    if t == 2:
        return f"{ism} {sifat} {narsa} sotib oldi."
    if t == 3:
        return f"{joy}da {sifat} {narsa} bor."
    if t == 4:
        return f"{ism} {ism2}ga {sifat} {narsa} berdi."
    if t == 5:
        return f"{ism} {joy}dan {narsa} olib keldi."
    return str(rng.choice(MAQOLLAR))


def korpus(n, seed):
    rng = np.random.default_rng(seed)
    return "\n".join(jumla(rng) for _ in range(n)) + "\n"


def ngramm_orgat(matn, n):
    """Kontekst (n-1 belgi) -> keyingi belgi sanog'i."""
    sanoq = defaultdict(Counter)
    t = "\n" * (n - 1) + matn
    for i in range(n - 1, len(t)):
        sanoq[t[i - n + 1:i]][t[i]] += 1
    return sanoq


def ngramm_ce(sanoq, matn, n, k, V):
    """O'rtacha -log p (nat) va ko'rilmagan kontekstlar ulushi."""
    t = "\n" * (n - 1) + matn
    jami, korilmagan = 0.0, 0
    for i in range(n - 1, len(t)):
        c = sanoq.get(t[i - n + 1:i])
        if c is None:
            korilmagan += 1
            p = 1 / V
        else:
            p = (c[t[i]] + k) / (sum(c.values()) + k * V)
        jami -= math.log(p)
    N = len(t) - (n - 1)
    return jami / N, korilmagan / N


def main() -> None:
    oquv, val = korpus(2000, 0), korpus(400, 1)
    V = len(set(oquv + val))

    print("=== 1. n-gramm tartibi (k = 0.01) ===")
    print(f"  {'n':>2} {'kontekstlar':>12} {'o_quv PPL':>10} {'val PPL':>9} "
          f"{'ko_rilmagan':>12}")
    val_ppl = {}
    for n in range(1, 10):
        sanoq = ngramm_orgat(oquv, n)
        ce_o, _ = ngramm_ce(sanoq, oquv, n, 0.01, V)
        ce_v, yangi = ngramm_ce(sanoq, val, n, 0.01, V)
        val_ppl[n] = math.exp(ce_v)
        print(f"  {n:>2} {len(sanoq):>12} {math.exp(ce_o):>10.3f} "
              f"{val_ppl[n]:>9.3f} {yangi:>12.2%}")
    eng = min(val_ppl, key=val_ppl.get)
    print(f"  eng yaxshi val: n = {eng} (PPL {val_ppl[eng]:.3f})")
    if val_ppl[9] > val_ppl[eng]:
        print(f"  n = 9 da val PPL {val_ppl[9]:.3f} - o'quv PPL pasaysa ham "
              f"val yomonlashdi: siyraklik")

    print("\n=== 2. Silliqlash koeffitsiyenti k (n = 6) ===")
    sanoq6 = ngramm_orgat(oquv, 6)
    for k in [1.0, 0.1, 0.01, 0.001]:
        ce_o, _ = ngramm_ce(sanoq6, oquv, 6, k, V)
        ce_v, _ = ngramm_ce(sanoq6, val, 6, k, V)
        print(f"  k = {k:<6} o'quv PPL {math.exp(ce_o):>7.3f}   "
              f"val PPL {math.exp(ce_v):>7.3f}")
    print("  katta k - ko'rilgan hodisalarga ham ortiqcha ehtimol 'yoyiladi'")

    print("\n=== 3. Siyraklik: nima uchun katta n yomonlashadi ===")
    for n in [3, 6, 9]:
        sanoq = ngramm_orgat(oquv, n)
        bir_marta = sum(1 for c in sanoq.values() if sum(c.values()) == 1)
        print(f"  n = {n}: {len(sanoq)} kontekst, ulardan {bir_marta} tasi "
              f"({bir_marta / len(sanoq):.0%}) faqat bir marta uchragan")

    print("\n=== 4. Uzoq kontekst: 'bordi' yoki 'qaytdi' ===")
    print("  shablon: '<ism> <joy>ga <vaqt> bordi.' / '<ism> <joy>dan <vaqt> qaytdi.'")
    oxirgi = []
    sanoq6 = ngramm_orgat(oquv, 6)
    for j in val.split("\n"):
        if j.endswith(" bordi.") or j.endswith(" qaytdi."):
            oxirgi.append(j)
    togri = []
    for j in oxirgi:
        fel = j.rsplit(" ", 1)[1]
        kontekst = j[:len(j) - len(fel)][-5:]
        c = sanoq6.get(kontekst)
        togri.append((c[fel[0]] + 0.01) / (sum(c.values()) + 0.01 * V))
    print(f"  val da {len(oxirgi)} ta shunday jumla")
    print(f"  6-gramm fe'lning birinchi harfiga bergan o'rtacha ehtimol: "
          f"{np.mean(togri):.3f}")
    masofa = np.mean([len(j) - j.index("ga " if j.endswith("bordi.") else "dan ")
                      - len(j.rsplit(" ", 1)[1]) for j in oxirgi])
    print(f"  hal qiluvchi qo'shimcha (-ga/-dan) fe'ldan o'rtacha {masofa:.1f} "
          f"belgi oldin - 6-gramm konteksti (5 belgi) unga yetmaydi")
    print("  ⭐ n-gramm: kichik n - kontekst qisqa, katta n - ma'lumot siyrak")


if __name__ == "__main__":
    main()

=== 1. n-gramm tartibi (k = 0.01) ===
   n  kontekstlar  o_quv PPL   val PPL  ko_rilmagan
   1            1     21.014    21.091        0.00%
   2           29      7.728     7.767        0.00%
   3          289      3.058     3.057        0.00%
   4          867      1.806     1.869        0.09%
   5         1823      1.547     1.658        0.42%
   6         3359      1.478     1.673        1.18%
   7         5404      1.456     1.757        2.47%
   8         7772      1.434     1.869        4.16%
   9        10377      1.421     2.032        6.30%
  eng yaxshi val: n = 5 (PPL 1.658)
  n = 9 da val PPL 2.032 - o'quv PPL pasaysa ham val yomonlashdi: siyraklik

=== 2. Silliqlash koeffitsiyenti k (n = 6) ===
  k = 1.0    o'quv PPL   2.584   val PPL   2.750
  k = 0.1    o'quv PPL   1.629   val PPL   1.795
  k = 0.01   o'quv PPL   1.478   val PPL   1.673
  k = 0.001  o'quv PPL   1.460   val PPL   1.701
  katta k - ko'rilgan hodisalarga ham ortiqcha ehtimol 'yoyiladi'

=== 3. Siyraklik: nima uchun katta n yomonlashadi ===
  n = 3: 289 kontekst, ulardan 1 tasi (0%) faqat bir marta uchragan
  n = 6: 3359 kontekst, ulardan 519 tasi (15%) faqat bir marta uchragan
  n = 9: 10377 kontekst, ulardan 3544 tasi (34%) faqat bir marta uchragan

=== 4. Uzoq kontekst: 'bordi' yoki 'qaytdi' ===
  shablon: '<ism> <joy>ga <vaqt> bordi.' / '<ism> <joy>dan <vaqt> qaytdi.'
  val da 117 ta shunday jumla
  6-gramm fe'lning birinchi harfiga bergan o'rtacha ehtimol: 0.496
  hal qiluvchi qo'shimcha (-ga/-dan) fe'ldan o'rtacha 10.9 belgi oldin - 6-gramm konteksti (5 belgi) unga yetmaydi
  ⭐ n-gramm: kichik n - kontekst qisqa, katta n - ma'lumot siyrak

"""Belgi darajasidagi LSTM til modeli: teacher forcing, perplexity, n-gramm bilan taqqoslash."""

import math
from collections import Counter, defaultdict

import numpy as np
import torch
import torch.nn as nn

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab bog' shahar qishloq kutubxona do'kon universitet "
          "choyxona dala bekat kasalxona stadion muzey teatr vokzal "
          "ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch go'sht shakar asal pishloq ko'ylak "
            "do'ppi soat telefon").split()
SIFATLAR = ("katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat "
            "yashil sariq issiq sovuq mazali").split()
MAQOLLAR = ["sabr tagi sariq oltin.", "til qilichdan o'tkir.",
            "vaqt oltindan qimmat.", "ko'p o'qigan ko'p biladi.",
            "mehnat qilsang rohat ko'rasan.", "yaxshi so'z jon ozig'i.",
            "daraxt mevasidan odam mehnatidan.",
            "ilm olish igna bilan quduq qazish."]


def jumla(rng):
    t = rng.integers(0, 7)
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = rng.choice(VAQTLAR), rng.choice(JOYLAR)
    narsa, sifat = rng.choice(NARSALAR), rng.choice(SIFATLAR)
    if t == 0:
        return f"{ism} {joy}ga {vaqt} bordi."
    if t == 1:
        return f"{ism} {joy}dan {vaqt} qaytdi."
    if t == 2:
        return f"{ism} {sifat} {narsa} sotib oldi."
    if t == 3:
        return f"{joy}da {sifat} {narsa} bor."
    if t == 4:
        return f"{ism} {ism2}ga {sifat} {narsa} berdi."
    if t == 5:
        return f"{ism} {joy}dan {narsa} olib keldi."
    return str(rng.choice(MAQOLLAR))


def korpus(n, seed):
    rng = np.random.default_rng(seed)
    return "\n".join(jumla(rng) for _ in range(n)) + "\n"


def ngramm_ppl(oquv, val, n, V, k=0.01):
    sanoq = defaultdict(Counter)
    t = "\n" * (n - 1) + oquv
    for i in range(n - 1, len(t)):
        sanoq[t[i - n + 1:i]][t[i]] += 1
    t = "\n" * (n - 1) + val
    p = []
    for i in range(n - 1, len(t)):
        c = sanoq.get(t[i - n + 1:i])
        p.append(1 / V if c is None
                 else (c[t[i]] + k) / (sum(c.values()) + k * V))
    return np.array(p)                    # val[j] ning ehtimoli, j = 0..


class TilModeli(nn.Module):
    def __init__(self, V, d=32, h=64):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.lstm = nn.LSTM(d, h, batch_first=True)
        self.chiqish = nn.Linear(h, V)

    def forward(self, x, holat=None):
        o, holat = self.lstm(self.emb(x), holat)
        return self.chiqish(o), holat


def oyna_batch(T, L, B, g):
    """Tasodifiy L+1 uzunlikdagi oynalar: kirish [:-1], nishon [1:]."""
    bosh = torch.randint(0, len(T) - L - 1, (B,), generator=g)
    w = torch.stack([T[i:i + L + 1] for i in bosh.tolist()])
    return w[:, :-1], w[:, 1:]


def orgat(T_oquv, V, seed, qadamlar=400, L=48, B=32):
    torch.manual_seed(seed)
    model = TilModeli(V)
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        x, y = oyna_batch(T_oquv, L, B, g)
        logit, _ = model(x)
        loss = nn.functional.cross_entropy(logit.reshape(-1, V), y.reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    model.eval()
    return model


def main() -> None:
    torch.set_num_threads(1)
    oquv, val = korpus(2000, 0), korpus(400, 1)
    lugat = sorted(set(oquv + val))
    V = len(lugat)
    s2i = {c: i for i, c in enumerate(lugat)}
    T_oquv = torch.tensor([s2i[c] for c in oquv])
    T_val = torch.tensor([s2i[c] for c in val])

    print("=== 1. Teacher forcing: kirish va nishon ===")
    x, y = oyna_batch(T_oquv, 12, 1, torch.Generator().manual_seed(5))
    print(f"  kirish: {''.join(lugat[i] for i in x[0].tolist())!r}")
    print(f"  nishon: {''.join(lugat[i] for i in y[0].tolist())!r}")
    print("  har qadamda modelga HAQIQIY oldingi belgi beriladi")
    torch.manual_seed(0)
    model = TilModeli(V)
    x, y = oyna_batch(T_oquv, 48, 32, torch.Generator().manual_seed(0))
    with torch.no_grad():
        logit, (h, c) = model(x)
        ce0 = nn.functional.cross_entropy(logit.reshape(-1, V), y.reshape(-1))
    print(f"  x {tuple(x.shape)} -> logit {tuple(logit.shape)}, h {tuple(h.shape)}")
    print(f"  o'rgatilmagan model: CE {ce0.item():.4f}, PPL {ce0.exp().item():.2f}"
          f"  (tekis: ln V = {math.log(V):.4f}, PPL {V})")
    print(f"  parametrlar: {sum(p.numel() for p in model.parameters())}")

    print("\n=== 2. LSTM va n-gramm: val perplexity ===")
    p2 = ngramm_ppl(oquv, val, 2, V)[1:]
    p5 = ngramm_ppl(oquv, val, 5, V)[1:]
    ppl2 = math.exp(-np.log(p2).mean())
    ppl5 = math.exp(-np.log(p5).mean())
    lstm_ppl, lstm_p = [], []
    for s in range(3):
        m = orgat(T_oquv, V, s)
        with torch.no_grad():
            logit, _ = m(T_val[:-1].unsqueeze(0))
            logp = torch.log_softmax(logit[0], 1)
            lp = logp[torch.arange(len(T_val) - 1), T_val[1:]]
        lstm_ppl.append(math.exp(-lp.mean().item()))
        lstm_p.append(lp.exp().numpy())
    print(f"  {'bigramm':<14} PPL {ppl2:.4f}")
    print(f"  {'5-gramm':<14} PPL {ppl5:.4f}  (2-misoldagi eng yaxshi n)")
    print(f"  {'LSTM (3 seed)':<14} PPL " + ", ".join(f"{v:.4f}" for v in lstm_ppl)
          + f"  o'rtacha {np.mean(lstm_ppl):.4f}")
    farq = ppl5 - np.array(lstm_ppl)
    se = farq.std(ddof=1) / np.sqrt(len(farq))
    xulosa = ("LSTM sezilarli yaxshi" if farq.mean() > 2 * se else
              "5-gramm sezilarli yaxshi" if farq.mean() < -2 * se else
              "sezilarli farq yo'q")
    print(f"  5-gramm - LSTM: {farq.mean():+.4f}, SE {se:.4f} -> {xulosa}")

    print("\n=== 3. Qayerda qiyin: so'z boshi va so'z ichi ===")
    oldingi = val[:-1]
    kelgan = val[1:]
    bosh = np.array([o in " \n" for o in oldingi])
    lp = np.log(lstm_p[0])
    for nom, maska in [("so'z boshi (bo'shliqdan keyin)", bosh),
                       ("so'z ichi va oxiri", ~bosh)]:
        print(f"  {nom:<31} ulush {maska.mean():>6.1%}  "
              f"PPL {math.exp(-lp[maska].mean()):>7.3f}")
    ulush = (-lp[bosh]).sum() / (-lp).sum()
    print(f"  jami noaniqlikning {ulush:.0%} i so'z boshidagi belgilarda")
    nuqta = np.array([k == "\n" for k in kelgan])
    print(f"  '.' dan keyingi '\\n' ehtimoli (LSTM): "
          f"{lstm_p[0][nuqta & (np.array(list(oldingi)) == '.')].mean():.4f}")

    print("\n=== 4. Uzoq kontekst: 'bordi' / 'qaytdi' ning birinchi harfi ===")
    joylar = []
    ofset = 0
    for j in val.split("\n"):
        if j.endswith(" bordi.") or j.endswith(" qaytdi."):
            joylar.append(ofset + j.rindex(" ") + 1 - 1)   # p[] indeksi
        ofset += len(j) + 1
    joylar = np.array(joylar)
    print(f"  {len(joylar)} ta jumla; to'g'ri harf ehtimoli:")
    print(f"    5-gramm: {p5[joylar].mean():.3f}")
    for s in range(3):
        print(f"    LSTM seed {s}: {lstm_p[s][joylar].mean():.3f}")
    lstm_ort = np.mean([lstm_p[s][joylar].mean() for s in range(3)])
    if lstm_ort > p5[joylar].mean() + 0.2:
        print(f"  LSTM (o'rtacha {lstm_ort:.3f}) 10+ belgi oldingi -ga/-dan ni "
              f"ko'p hollarda eslab qoldi, 5-gramm - tanga tashlash darajasida")
    print("  ⭐ Teacher forcing + cross-entropy = keyingi belgi bashorati")


if __name__ == "__main__":
    main()

=== 1. Teacher forcing: kirish va nishon ===
  kirish: 'oldi.\ntil qi'
  nishon: 'ldi.\ntil qil'
  har qadamda modelga HAQIQIY oldingi belgi beriladi
  x (32, 48) -> logit (32, 48, 29), h (1, 32, 64)
  o'rgatilmagan model: CE 3.3904, PPL 29.68  (tekis: ln V = 3.3673, PPL 29)
  parametrlar: 27901

=== 2. LSTM va n-gramm: val perplexity ===
  bigramm        PPL 7.7670
  5-gramm        PPL 1.6578  (2-misoldagi eng yaxshi n)
  LSTM (3 seed)  PPL 1.4780, 1.4963, 1.4844  o'rtacha 1.4863
  5-gramm - LSTM: +0.1716, SE 0.0053 -> LSTM sezilarli yaxshi

=== 3. Qayerda qiyin: so'z boshi va so'z ichi ===
  so'z boshi (bo'shliqdan keyin)  ulush  14.3%  PPL   6.429
  so'z ichi va oxiri              ulush  85.7%  PPL   1.157
  jami noaniqlikning 68% i so'z boshidagi belgilarda
  '.' dan keyingi '\n' ehtimoli (LSTM): 0.9987

=== 4. Uzoq kontekst: 'bordi' / 'qaytdi' ning birinchi harfi ===
  117 ta jumla; to'g'ri harf ehtimoli:
    5-gramm: 0.484
    LSTM seed 0: 0.892
    LSTM seed 1: 0.800
    LSTM seed 2: 0.866
  LSTM (o'rtacha 0.853) 10+ belgi oldingi -ga/-dan ni ko'p hollarda eslab qoldi, 5-gramm - tanga tashlash darajasida
  ⭐ Teacher forcing + cross-entropy = keyingi belgi bashorati

"""Generatsiya: greedy, temperature va top-k - to'g'rilik va xilma-xillik."""

import re

import numpy as np
import torch
import torch.nn as nn

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab bog' shahar qishloq kutubxona do'kon universitet "
          "choyxona dala bekat kasalxona stadion muzey teatr vokzal "
          "ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch go'sht shakar asal pishloq ko'ylak "
            "do'ppi soat telefon").split()
SIFATLAR = ("katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat "
            "yashil sariq issiq sovuq mazali").split()
MAQOLLAR = ["sabr tagi sariq oltin.", "til qilichdan o'tkir.",
            "vaqt oltindan qimmat.", "ko'p o'qigan ko'p biladi.",
            "mehnat qilsang rohat ko'rasan.", "yaxshi so'z jon ozig'i.",
            "daraxt mevasidan odam mehnatidan.",
            "ilm olish igna bilan quduq qazish."]


def jumla(rng):
    t = rng.integers(0, 7)
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = rng.choice(VAQTLAR), rng.choice(JOYLAR)
    narsa, sifat = rng.choice(NARSALAR), rng.choice(SIFATLAR)
    if t == 0:
        return f"{ism} {joy}ga {vaqt} bordi."
    if t == 1:
        return f"{ism} {joy}dan {vaqt} qaytdi."
    if t == 2:
        return f"{ism} {sifat} {narsa} sotib oldi."
    if t == 3:
        return f"{joy}da {sifat} {narsa} bor."
    if t == 4:
        return f"{ism} {ism2}ga {sifat} {narsa} berdi."
    if t == 5:
        return f"{ism} {joy}dan {narsa} olib keldi."
    return str(rng.choice(MAQOLLAR))


def korpus(n, seed):
    rng = np.random.default_rng(seed)
    return "\n".join(jumla(rng) for _ in range(n)) + "\n"


def shablon_tekshiruvchi():
    """Jumla korpus grammatikasiga (shablonlarga) mosmi."""
    def y(lst):
        return "(?:" + "|".join(re.escape(s) for s in lst) + ")"
    i, v, j = y(ISMLAR), y(VAQTLAR), y(JOYLAR)
    n, s = y(NARSALAR), y(SIFATLAR)
    naqshlar = [f"{i} {j}ga {v} bordi\\.", f"{i} {j}dan {v} qaytdi\\.",
                f"{i} {s} {n} sotib oldi\\.", f"{j}da {s} {n} bor\\.",
                f"{i} {i}ga {s} {n} berdi\\.", f"{i} {j}dan {n} olib keldi\\.",
                y(MAQOLLAR)]
    umumiy = re.compile("|".join(f"(?:{p})" for p in naqshlar))
    return lambda jumla_: umumiy.fullmatch(jumla_) is not None


class TilModeli(nn.Module):
    def __init__(self, V, d=32, h=64):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.lstm = nn.LSTM(d, h, batch_first=True)
        self.chiqish = nn.Linear(h, V)

    def forward(self, x, holat=None):
        o, holat = self.lstm(self.emb(x), holat)
        return self.chiqish(o), holat


def orgat(T_oquv, V, seed=0, qadamlar=400, L=48, B=32):
    torch.manual_seed(seed)
    model = TilModeli(V)
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        bosh = torch.randint(0, len(T_oquv) - L - 1, (B,), generator=g)
        w = torch.stack([T_oquv[i:i + L + 1] for i in bosh.tolist()])
        logit, _ = model(w[:, :-1])
        loss = nn.functional.cross_entropy(logit.reshape(-1, V),
                                           w[:, 1:].reshape(-1))
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    model.eval()
    return model


def generatsiya(model, lugat, n, usul, temp=1.0, k=None, seed=0, maks=60):
    """n ta jumlani parallel yaratadi; '\n' dan boshlab '\n' gacha."""
    s2i = {c: i for i, c in enumerate(lugat)}
    g = torch.Generator().manual_seed(seed)
    x = torch.full((n, 1), s2i["\n"])
    holat, chiqdi = None, []
    tugadi = torch.zeros(n, dtype=torch.bool)
    with torch.no_grad():
        for _ in range(maks):
            logit, holat = model(x, holat)
            logit = logit[:, -1]
            if usul == "greedy":
                keyingi = logit.argmax(1)
            else:
                logit = logit / temp
                if k is not None:
                    chegara = logit.topk(k, dim=1).values[:, -1:]
                    logit = logit.masked_fill(logit < chegara, float("-inf"))
                p = torch.softmax(logit, 1)
                keyingi = torch.multinomial(p, 1, generator=g)[:, 0]
            chiqdi.append(keyingi)
            tugadi |= keyingi == s2i["\n"]
            x = keyingi.unsqueeze(1)
            if tugadi.all():
                break
    belgilar = torch.stack(chiqdi, 1).tolist()
    natija = []
    for qator in belgilar:
        matn = "".join(lugat[i] for i in qator)
        natija.append(matn.split("\n")[0])
    return natija


def main() -> None:
    torch.set_num_threads(1)
    oquv = korpus(2000, 0)
    lugat = sorted(set(oquv + korpus(400, 1)))
    V = len(lugat)
    s2i = {c: i for i, c in enumerate(lugat)}
    model = orgat(torch.tensor([s2i[c] for c in oquv]), V)
    sozlar = set(oquv.replace(".", " ").split())
    jumlalar = set(oquv.strip().split("\n"))
    togrimi = shablon_tekshiruvchi()
    print("=== 1. Tekshiruvchi ===")
    print(f"  o'quv jumlalarining shablonga mosligi: "
          f"{np.mean([togrimi(j) for j in jumlalar]):.1%}")
    print(f"  buzilgan misol: {togrimi('anvar bozorga ertalab qaytdi.')}  "
          f"('-ga' bilan 'qaytdi' mos emas)")

    usullar = [("greedy", "greedy", 1.0, None), ("T=0.5", "namuna", 0.5, None),
               ("T=1.0", "namuna", 1.0, None), ("T=1.5", "namuna", 1.5, None),
               ("top-k=3", "namuna", 1.0, 3)]
    print("\n=== 2. Namunalar (har usuldan 3 ta) ===")
    natijalar = {}
    for nom, usul, temp, k in usullar:
        natijalar[nom] = generatsiya(model, lugat, 300, usul, temp, k)
        print(f"  {nom}:")
        for j in natijalar[nom][:3]:
            print(f"    {j}")

    print("\n=== 3. To'g'rilik va xilma-xillik (300 jumla) ===")
    print(f"  {'usul':<8} {'lug_atda':>9} {'shablon':>8} {'noyob':>7} "
          f"{'o_quvda bor':>12}")
    olchov = {}
    for nom, _, _, _ in usullar:
        js = natijalar[nom]
        barcha = [w for j in js for w in j.replace(".", " ").split()]
        lug = np.mean([w in sozlar for w in barcha])
        sh = np.mean([togrimi(j) for j in js])
        noyob = len(set(js)) / len(js)
        bor = np.mean([j in jumlalar for j in js])
        olchov[nom] = (lug, sh, noyob)
        print(f"  {nom:<8} {lug:>9.1%} {sh:>8.1%} {noyob:>7.1%} {bor:>12.1%}")

    print("\n=== 4. Xulosa ===")
    if olchov["greedy"][2] < 0.01:
        print("  greedy: har safar AYNAN bitta jumla - xilma-xillik yo'q")
    if olchov["greedy"][1] == 0:
        print("  va u shablonga mos emas: har qadamda eng ehtimolli belgi -")
        print("  butun jumla eng ehtimolli degani emas")
    t05, t15 = olchov["T=0.5"], olchov["T=1.5"]
    print(f"  T 0.5 -> 1.5: shablon {t05[1]:.1%} -> {t15[1]:.1%}, "
          f"noyob {t05[2]:.1%} -> {t15[2]:.1%}")
    if t15[1] < t05[1]:
        print("  yuqori temperatura - xilma-xil, lekin xatoli")
    tk = olchov["top-k=3"]
    t10 = olchov["T=1.0"]
    print(f"  top-k=3 (T=1.0 ga nisbatan): shablon {t10[1]:.1%} -> {tk[1]:.1%}, "
          f"noyob {t10[2]:.1%} -> {tk[2]:.1%}")
    if tk[1] > t10[1] and tk[2] > t05[2]:
        print("  top-k dumdagi kam ehtimolli belgilarni kesadi: T=1.0 dan to'g'riroq,"
              " T=0.5 dan xilma-xilroq")
    elif t05[1] >= tk[1] and t05[2] >= tk[2]:
        print("  bu modelda T=0.5 top-k=3 dan IKKALA o'lchovda ham yaxshi:")
        print("  k=3 so'z boshidagi o'nlab to'g'ri variantdan faqat 3 tasini qoldiradi")
    print("  ⭐ Generatsiya usuli - to'g'rilik va xilma-xillik orasidagi murosa")


if __name__ == "__main__":
    main()

=== 1. Tekshiruvchi ===
  o'quv jumlalarining shablonga mosligi: 100.0%
  buzilgan misol: False  ('-ga' bilan 'qaytdi' mos emas)

=== 2. Namunalar (har usuldan 3 ta) ===
  greedy:
    shahlo sovuq anor berdi.
    shahlo sovuq anor berdi.
    shahlo sovuq anor berdi.
  T=0.5:
    sherzod kasalxonadan bugun qaytdi.
    otabek universitetdan kecha qaytdi.
    muzeyda sovuq guruch sotib oldi.
  T=1.0:
    stadionda shirin anor bor.
    otabek universitetdan choy olib keldi.
    muzeyda qizilaga sariq qalam sotib oldi.
  T=1.5:
    stadiondan cheozerda bog'dan erta qama seisha dalan ikoch bo
    oalsi dalsang rohat sotib oldi.
    muzeyda qishlaq oldi.
  top-k=3:
    sherzod kasalxonadan bugun qaytdi.
    sardor otabekga katta bordi.
    muzeyda sovazardan kechqa qaytdi.

=== 3. To'g'rilik va xilma-xillik (300 jumla) ===
  usul      lug_atda  shablon   noyob  o_quvda bor
  greedy      100.0%     0.0%    0.3%         0.0%
  T=0.5        99.6%    79.7%   85.3%        15.3%
  T=1.0        89.7%    34.3%   96.7%         8.3%
  T=1.5        58.9%     3.0%  100.0%         0.7%
  top-k=3      95.1%    50.3%   88.7%         6.0%

=== 4. Xulosa ===
  greedy: har safar AYNAN bitta jumla - xilma-xillik yo'q
  va u shablonga mos emas: har qadamda eng ehtimolli belgi -
  butun jumla eng ehtimolli degani emas
  T 0.5 -> 1.5: shablon 79.7% -> 3.0%, noyob 85.3% -> 100.0%
  yuqori temperatura - xilma-xil, lekin xatoli
  top-k=3 (T=1.0 ga nisbatan): shablon 34.3% -> 50.3%, noyob 96.7% -> 88.7%
  top-k dumdagi kam ehtimolli belgilarni kesadi: T=1.0 dan to'g'riroq, T=0.5 dan xilma-xilroq
  ⭐ Generatsiya usuli - to'g'rilik va xilma-xillik orasidagi murosa

logit, _ = model(w); loss = ce(logit, w)      # o'zini ko'chiradi   # ⚠️
logit, _ = model(w[:, :-1]); loss = ce(logit, w[:, 1:])            # ✅

p = son[(a, b)] / son[a]                      # 0 -> log 0         # ⚠️
p = (son[(a, b)] + k) / (son[a] + k * V)                           # ✅

ppl = 2 ** loss.item()                        # loss nat da        # ⚠️
ppl = math.exp(loss.item())                                        # ✅

print("PPL", math.exp(oquv_loss))                                  # ⚠️
print("val PPL", math.exp(val_loss))          # alohida matnda     # ✅

nn.functional.cross_entropy(logit, y)         # (B, L, V) va (B, L) # ⚠️
nn.functional.cross_entropy(logit.reshape(-1, V), y.reshape(-1))   # ✅

logit, _ = model(x_oxirgi)                    # har safar noldan   # ⚠️
logit, holat = model(x_oxirgi, holat)                              # ✅

keyingi = torch.multinomial(p, 1)                                  # ⚠️
keyingi = torch.multinomial(p, 1, generator=g)                     # ✅

1.  # zanjir qoidasi P(x1, x2, x3) ni qanday yoyadi?
2.  # tekis modelning perplexity si (V = 29)?
3.  # PPL formulasi CE orqali?
4.  # CE = 2.0 nat bo'lsa PPL?
5.  # add-k silliqlash formulasi?
6.  # n oshganda o'quv PPL va val PPL qanday o'zgaradi?
7.  # "salom" oynasi uchun kirish va nishon?
8.  # o'rgatilmagan LSTM ning CE si taxminan?
9.  # greedy 300 marta ishlatilsa nechta noyob jumla?
10. # T -> 0 da namuna olish nimaga aylanadi?
11. # top-k=1 nimaga teng?
12. # nima uchun 6-gramm 'bordi'/'qaytdi' ni ajrata olmaydi?

1.  logit, _ = model(w)
    loss = nn.functional.cross_entropy(logit.reshape(-1, V), w.reshape(-1))

2.  p = juft[(a, b)] / oldin[a]
    ce = -sum(math.log(p) for ...)

3.  ppl = 2 ** nn.functional.cross_entropy(logit, y).item()

4.  for _ in range(60):
        logit, _ = model(x)
        x = logit[:, -1].argmax(1, keepdim=True)

5.  p = torch.softmax(logit / 0.0, 1)