26.1-dars: Generativ modellar

DISKRIMINATIV:  p(y | x)          "rasm berilsa - qaysi raqam?"
  logistik regressiya, daraxtlar, CNN, BERT-klassifikator
  x ning o'zi qanday taqsimlanganini BILMAYDI
  deyarli butunlay shovqindan iborat namunalarga ham ishonch bilan javob beradi
  (3-misol: ularning 0.539 qismiga ishonch > 0.9)

GENERATIV:      p(x)  yoki  p(x | y)
  "bu rasm raqamlar dunyosida qanchalik odatiy?"  -> p(x) qiymati
  "menga yangi raqam chiz"                       -> x ~ p(x)
  "menga yangi 7 chiz"                           -> x ~ p(x | y = 7)
  GMM (16-qism), Naive Bayes (14-qism), GPT 24.9-bob, VAE, GAN, diffusion

BAYES KO'PRIGI:
  p(y | x) = p(x | y) * p(y) / p(x)
  generativ modeldan klassifikator yasash mumkin (Naive Bayes shunday)
  teskarisi - yo'q: p(y | x) dan p(x) ni tiklab bo'lmaydi

NIMA UCHUN QIYIN:
  8x8 raqam = 64 o'lchov; 8x8 binar rasmlar soni 2^64 ~ 1.8 * 10^19
  p(x) - shu ulkan fazoning HAR nuqtasiga ehtimol berishi kerak,
  yig'indisi (integrali) esa aniq 1 bo'lishi shart
  klassifikatorga 10 ta son kifoya; generativ modelga - butun taqsimot

IKKI ALOHIDA QOBILIYAT:
  (a) BAHOLASH:   berilgan x uchun p(x) ni hisoblash
      -> anomaliya aniqlash 16.10-bob, modellarni solishtirish (LL)
  (b) NAMUNA OLISH: x ~ p(x) ni chiqarish
      -> sintetik ma'lumot, rasm/matn yaratish, augmentatsiya

HAR MODELDA IKKALASI HAM BO'LMAYDI:
  model            p(x) aniq?        namuna olish
  gistogramma      ha                oson (katak + tekis shovqin)
  KDE              ha                oson (o'quv nuqtasi + shovqin)
  GMM              ha                oson (komponent, keyin Gauss)
  avtoregressiv    ha                sekin: D ta ketma-ket qadam
  VAE              pastki chegara    oson: z ~ N(0, I) -> decoder
  GAN              YO'Q              juda oson: z -> generator
  diffusion        chegara/taxmin    sekin: ko'p qadam
  flow             ha                oson (teskari akslantirish)

SHARTLI MODEL:
  p(x | y) - sinf (26.2 da shartli VAE), matn (Matn-rasm modellari darsida)
  yoki prompt (24.9 dagi GPT - prompt = shart)

O'RGATISH MAQSADI (deyarli hamma likelihood-modellarda):
  theta* = argmax  (1/N) * sum_i log p_theta(x_i)
  bu KL(p_data || p_theta) ni minimallashtirishga teng:
    KL = E_data[log p_data(x)] - E_data[log p_theta(x)]
         ^ modelga bog'liq emas   ^ biz maksimallashtiramiz
  GPT ning cross-entropy loss i - aynan shu 24.9-bob

HELD-OUT LL - ASOSIY O'LCHOV:
  o'quv LL doim murakkablik bilan o'sadi (yodlash ham "yaxshi" ko'rinadi)
  val/test LL - modelning haqiqiy umumlashishi
  1-misol, GMM:  K = 16: o'quv -2.01, val -2.12
                 K = 32: o'quv -1.96, val -2.17  <- o'quv o'sdi, val tushdi
  2-misol, digits PCA-20 + GMM:
                 K = 40: o'quv -37.46, val -96.88 (ortiqcha moslashish)

BIRLIKLAR:
  nat: ln bilan;  bit: log2 bilan;  1 nat = 1.443 bit
  diskret rasm: bit/rasm yoki bit/piksel (3-misol: 25.26 bit/rasm)
  uzluksiz ma'lumot: log-ZICHLIK, musbat ham bo'lishi mumkin
    (zichlik 1 dan katta bo'la oladi); faqat bir xil fazoda solishtiriladi

-inf TUZOG'I:
  bitta test nuqtasiga p = 0 berilsa - o'rtacha LL = -inf
  1-misol: silliqlashsiz gistogramma, 4 ta nuqta bo'sh katakda -> -inf
  yechim: silliqlash (Laplace, alfa = 0.5), har yerda musbat zichlik

GISTOGRAMMA:
  fazoni b^d katakka bo'lamiz; p = katakdagi ulush / katak hajmi
  d = 2, b = 20:  400 katak - ishlaydi
  d = 64, b = 2:  2^64 katak - hech qachon to'lmaydi (o'lchamlar la'nati)

KDE (yadroli zichlik baholash):
  p(x) = (1/N) * sum_i N(x; x_i, h^2 * I)
  h - kenglik: kichik h -> har o'quv nuqtasi atrofida "igna" (yodlash)
                katta h -> hamma narsa surtilgan
  butun o'quv to'plamini saqlaydi (1-misol: 2000 son)
  namuna = tasodifiy o'quv nuqtasi + N(0, h^2) shovqin -> kichik h da NUSXA

GMM (16.6-dars):
  p(x) = sum_k pi_k * N(x; mu_k, Sigma_k), EM bilan o'rgatiladi
  1-misolda K = 16: atigi 95 son, test LL eng yaxshi (-2.108)
  halqani 16 ta "cho'zilgan Gauss" bilan yopadi

YUQORI O'LCHOVDA:
  xom piksellarda (64 o'lchov) to'liq kovariatsiyali GMM - juda ko'p parametr
  amaliy yo'l: avval PCA (16.8-dars) bilan 20 o'lchovga, keyin GMM
  2-misol: PCA-20 dispersiyaning 0.897 qismini saqlaydi

1. AVTOREGRESSIV (24.9 - GPT; bu darsning 3-misoli):
   p(x) = p(x_1) * p(x_2 | x_1) * ... * p(x_D | x_1..x_{D-1})
   + LL aniq, o'rgatish oddiy (cross-entropy)
   - namuna olish KETMA-KET: D qadam (8x8 da 64, 256x256 rangli rasmda ~200 ming)
   matnda - asosiy usul; rasmda - PixelCNN, rasm tokenlari ustidagi transformerlar

2. LATENT O'ZGARUVCHILI (VAE - 26.2):
   z ~ N(0, I),  x ~ p(x | z)       p(x) = integral p(x | z) p(z) dz
   integral hisoblab bo'lmaydi -> pastki chegara (ELBO) maksimallashtiriladi
   + tez namuna, ma'noli latent fazo
   - namunalar ko'pincha xira (o'rtachalashgan)

3. ADVERSARIAL (GAN - 26.3, 26.4):
   generator G(z) va diskriminator D o'yini; p(x) umuman yo'q
   + o'tkir, realistik namunalar
   - o'rgatish beqaror, mode collapse, LL bilan baholab bo'lmaydi

4. DIFFUSION (26.5, 26.6):
   ma'lumotga asta-sekin shovqin qo'shiladi; model shovqinni olib tashlashni o'rganadi
   + yuqori sifat va xilma-xillik, barqaror o'rgatish
   - namuna olish ko'p qadamli (sekin)
   zamonaviy matn-rasm modellarining asosi

5. FLOW (normallashtiruvchi oqimlar, qisqa):
   x = f(z), f - teskarilanadigan; o'zgaruvchini almashtirish formulasi:
   log p(x) = log p(z) - log |det(df/dz)|
   + LL aniq, namuna tez
   - arxitektura cheklangan (teskarilanish, det hisoblash)

KLASSIK (bu dars): gistogramma, KDE, GMM - past o'lchovda yaxshi etalon

ZANJIR QOIDASI (hech qanday taxminsiz - ANIQ):
  p(x_1, ..., x_D) = prod_i p(x_i | x_1, ..., x_{i-1})
  har omil - oddiy klassifikator: "oldingi piksellarga qarab keyingisi 1 mi?"

8x8 BINAR RASM, TARTIB - QATOR BO'YICHA:
  x_1 x_2 ... x_8
  x_9 ...         <- x_12 ni bashorat qilishda x_1..x_11 ko'rinadi
  ...

UCH MODEL (3-misol):
  mustaqil:  p(x_i) - oldingi piksellarga qaramaydi        64 parametr
  chiziqli:  logit_i = b_i + sum_{j<i} W_ij * x_j          4 160 parametr
  MADE:      niqobli MLP: yashirin neyronlar ham "darajali" 20 736 parametr
             i-chiqish faqat j < i kirishlarga bog'liq bo'lishi niqob bilan kafolatlanadi

NIQOB G'OYASI (24.7 dagi kauzal niqob bilan bir xil):
  W_ij * niqob_ij, niqob_ij = 1 faqat j < i bo'lsa
  bitta forward - barcha 64 shartli ehtimol birdaniga (o'rgatishda tez)
  namuna olishda esa 64 ta ketma-ket forward (sekin)

BIT/RASM:
  tekis tasodif: 64 bit (har piksel 1 bit)
  3-misol test: mustaqil 37.98, chiziqli 26.57, MADE 25.26

1. YAXSHI LL, YOMON NAMUNALAR (3-misol):
   p_aralash = 0.01 * p_MADE + 0.99 * p_shovqin
   log2 p_aralash >= log2 p_MADE + log2(0.01) = log2 p_MADE - 6.64
   test NLL: 31.89 bit - mustaqil modeldan 37.98-bob YAXSHI
   lekin namunalarning ~99% i - tasodifiy shovqin
   xulosa: LL 6.6 bitga yomonlashdi, namunalar esa deyarli butunlay buzildi

2. YAXSHI NAMUNALAR, YOMON LL (4-misol):
   yodlovchi KDE (h = 0.5): o'quv LL -11.42, test LL -323.93
   namunalari hakamga eng "ishonchli" 0.954-bob - chunki ular o'quv rasmlarining nusxasi

3. TURLI FAZODAGI LL LAR SOLISHTIRILMAYDI:
   PCA-20 dagi log-zichlik va 64 pikseldagi log-zichlik - boshqa birliklar
   uzluksiz zichlik va diskret ehtimol - boshqa narsalar
   (diskret piksellarga uzluksiz model: "dequantization" - shovqin qo'shib)

4. YUQORI O'LCHOVDA LL MAYDA DETALLARGA E'TIBOR BERADI:
   LL ning katta qismi piksel darajasidagi shovqinni to'g'ri modellashdan keladi,
   inson esa shakl va ma'noga qaraydi

XULOSA:
  LL - zarur, lekin yetarli emas; namuna-asosli o'lchovlar bilan birga:
    hakam-klassifikator bilan "tanib olinish" (2, 3-misollar)
    sinf taqsimoti - mode coverage (qaysi sinflar yaratilmayapti?)
    eng yaqin o'quv namunasigacha masofa - yodlash (4-misol)
    Frechet masofa (FID uslubi) - Generativ modellarni baholash darsida

HAKAM-KLASSIFIKATOR:
  generativ modeldan MUSTAQIL, haqiqiy ma'lumotda o'rgatilgan (MLP 64-128-10)
  2-misol: test aniqligi 0.971; haqiqiy test rasmlarida ishonch > 0.9 - 0.947
  o'lchovlar:
    ishonch > 0.9 ulushi   - namuna "aniq bir raqamga" o'xshaydimi
    sinflar soni (0..9)    - hamma raqamlar yaratilyaptimi (mode coverage)
  CHEKLOV: hakam ham aldanadi - 3-misolda 99% shovqindan iborat namunalarning
  0.539 qismiga ishonch > 0.9 berdi (diskriminativ model p(x) ni bilmaydi!)
  shuning uchun doim haqiqiy test rasmlaridagi qiymat bilan solishtiring

YODLAB OLISH TEKSHIRUVI:
  har namuna uchun eng yaqin O'QUV rasmigacha masofa d_nn
  ma'lumotnoma: haqiqiy TEST rasmlarining o'quvgacha masofasi
    (test ham o'quvni ko'rmagan - "halol yangi" rasmlar shunchalik uzoq)
  4-misol: test median 17.38; 5-foizil 12.02 -> "nusxa" chegarasi
    yodlovchi: namunalarning 0.784 qismi chegaradan yaqin (haqiqiyda 0.051)
    GMM K=10:  0.017 - nusxa ko'chirmayapti

NEGA MUHIM:
  maxfiylik (tibbiy, moliyaviy yozuvlar), mualliflik huquqi -
  Generativ AI etikasi va xavfsizligi darsida batafsil

import numpy as np
from sklearn.mixture import GaussianMixture
from sklearn.neighbors import KernelDensity

# zichlikni baholash va held-out LL
gm = GaussianMixture(16, random_state=0).fit(x_oquv)
val_ll = gm.score(x_val)                        # o'rtacha log p(x)
kd = KernelDensity(bandwidth=0.1).fit(x_oquv)
lp = kd.score_samples(x_test)                   # har nuqta uchun log p(x)

# namuna olish
x_new, komp = gm.sample(1000)                   # komponentlar bo'yicha TARTIBLANGAN
x_new = x_new[np.random.default_rng(1).permutation(len(x_new))]
x_kde = kd.sample(1000, random_state=1)

# PCA + GMM (yuqori o'lchov)
z = pca.transform(x); x_gen = np.clip(pca.inverse_transform(gm.sample(1000)[0]), 0, 16)

# avtoregressiv: bit/rasm va piksel-piksel namuna
nll_bit = F.binary_cross_entropy_with_logits(model(x), x, reduction="none").sum(1) / math.log(2)
for i in range(64):
    x[:, i] = torch.bernoulli(torch.sigmoid(model(x)[:, i]), generator=g)

# yodlab olish tekshiruvi
d_nn = torch.cdist(namuna, oquv).min(1).values
chegara = np.quantile(d_test_nn, 0.05)          # haqiqiy test rasmlari asosida
nusxa_ulushi = (d_nn < chegara).float().mean()

diskriminativ p(y|x); generativ p(x) yoki p(x|y)
ikki qobiliyat: p(x) ni baholash va namuna olish - har modelda ikkalasi emas
maqsad: max E[log p(x)] = min KL(p_data || p_theta); tanlov - held-out LL bilan
oilalar: avtoregressiv, VAE, GAN, diffusion, flow - har biri murosa
LL yetarli emas: + hakam-klassifikator, sinf taqsimoti, eng yaqin qo'shni masofasi

"""2D zichlikni baholash: gistogramma, KDE va GMM - held-out log-likelihood bilan tanlash."""

import math
import warnings

import numpy as np
from sklearn.mixture import GaussianMixture
from sklearn.neighbors import KernelDensity

R, SIGMA_R, SIGMA_M, ULUSH_HALQA = 2.0, 0.15, 0.3, 0.6
QUTI = 3.5                                    # gistogramma uchun [-3.5, 3.5]^2


def namuna(n, rng):
    """Halqa (60%) + markazdagi Gauss (40%) aralashmasi."""
    halqa = rng.random(n) < ULUSH_HALQA
    burchak = rng.uniform(0, 2 * math.pi, n)
    r = R + SIGMA_R * rng.standard_normal(n)
    x_h = np.c_[r * np.cos(burchak), r * np.sin(burchak)]
    x_m = SIGMA_M * rng.standard_normal((n, 2))
    return np.where(halqa[:, None], x_h, x_m)


def haqiqiy_logp(x):
    """Haqiqiy zichlik (biz bilamiz!) - 'shift' sifatida."""
    r = np.linalg.norm(x, axis=1)
    p_h = np.exp(-0.5 * ((r - R) / SIGMA_R) ** 2) / (SIGMA_R * math.sqrt(2 * math.pi)) / (2 * math.pi * r)
    p_m = np.exp(-0.5 * (r / SIGMA_M) ** 2) / (2 * math.pi * SIGMA_M ** 2)
    return np.log(ULUSH_HALQA * p_h + (1 - ULUSH_HALQA) * p_m)


class Gistogramma:
    def __init__(self, b, alfa=0.0):
        self.b, self.alfa = b, alfa
        self.chet = np.linspace(-QUTI, QUTI, b + 1)
        self.yuza = (2 * QUTI / b) ** 2

    def fit(self, x):
        c, _, _ = np.histogram2d(x[:, 0], x[:, 1], bins=[self.chet, self.chet])
        c = c + self.alfa                                       # Laplace silliqlash
        self.p = c / c.sum()
        return self

    def logp(self, x):
        i = np.clip(np.digitize(x[:, 0], self.chet) - 1, 0, self.b - 1)
        j = np.clip(np.digitize(x[:, 1], self.chet) - 1, 0, self.b - 1)
        with np.errstate(divide="ignore"):
            return np.log(self.p[i, j] / self.yuza)

    def sample(self, n, rng):
        k = rng.choice(self.b * self.b, n, p=self.p.ravel())
        i, j = np.divmod(k, self.b)
        w = 2 * QUTI / self.b
        return np.c_[self.chet[i] + w * rng.random(n), self.chet[j] + w * rng.random(n)]


def oila_modellari():
    return {
        "gistogramma": [(f"b={b}", b * b - 1, lambda b=b: Gistogramma(b, alfa=0.5)) for b in (5, 10, 20, 40)],
        "KDE": [(f"h={h}", None, lambda h=h: KernelDensity(bandwidth=h)) for h in (0.05, 0.1, 0.2, 0.4)],
        "GMM": [(f"K={k}", 6 * k - 1, lambda k=k: GaussianMixture(k, random_state=0, n_init=1))
                for k in (1, 4, 8, 16, 32)],
    }


def logp(model, x):
    if isinstance(model, Gistogramma):
        return model.logp(x)
    return model.score_samples(x)


def main() -> None:
    rng = np.random.default_rng(0)
    oquv, val, test = namuna(1000, rng), namuna(1000, rng), namuna(2000, rng)

    print("=== 1. Haqiqiy taqsimot: halqa (60%) + markaz (40%) ===")
    print(f"  o'quv 1000, val 1000, test 2000 nuqta")
    print(f"  haqiqiy zichlikning test log-likelihoodi: {haqiqiy_logp(test).mean():.3f} (shift)")

    print("\n=== 2. Nol hisob tuzog'i: silliqlashsiz gistogramma ===")
    for b in (10, 40):
        g0 = Gistogramma(b).fit(oquv)
        lp = g0.logp(val)
        print(f"  b={b:<3} bo'sh katakdagi val nuqtalari {np.isinf(lp).sum():>3} ta -> "
              f"val LL = {lp.mean():.3f}")

    print("\n=== 3. Har oilada val LL bo'yicha tanlash ===")
    tanlangan = {}
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        for oila, variantlar in oila_modellari().items():
            qator = []
            eng = None
            for nom, _, yasa in variantlar:
                m = yasa().fit(oquv)
                tr, va = logp(m, oquv).mean(), logp(m, val).mean()
                qator.append(f"{nom} {tr:.2f}/{va:.2f}")
                if eng is None or va > eng[1]:
                    eng = (nom, va, yasa)
            tanlangan[oila] = eng
            print(f"  {oila:<11} (o'quv/val): " + ", ".join(qator))
            print(f"  {'':<11} -> tanlandi {eng[0]}")

    print("\n=== 4. Tanlangan modellar: 5 ta ma'lumot urug'ida test LL (juftlashgan) ===")
    natija = {o: [] for o in tanlangan}
    for s in range(5):
        r = np.random.default_rng(100 + s)
        o_, t_ = namuna(1000, r), namuna(2000, r)
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            for oila, (nom, _, yasa) in tanlangan.items():
                natija[oila].append(logp(yasa().fit(o_), t_).mean())
    parametr = {"gistogramma": lambda n: int(n[2:]) ** 2 - 1, "KDE": lambda n: 2 * 1000,
                "GMM": lambda n: 6 * int(n[2:]) - 1}
    for oila, v in natija.items():
        nom = tanlangan[oila][0]
        print(f"  {oila:<11} {nom:<6} test LL {np.mean(v):.3f}   saqlanadigan sonlar: {parametr[oila](nom)}")
    eng_oila = max(natija, key=lambda o: np.mean(natija[o]))
    for oila in natija:
        if oila == eng_oila:
            continue
        f = np.array(natija[oila]) - np.array(natija[eng_oila])
        se = f.std(ddof=1) / math.sqrt(len(f))
        print(f"  {oila} - {eng_oila}: {f.mean():+.3f}, SE {se:.3f} -> "
              f"{'sezilarli yomon' if f.mean() < -2 * se else 'sezilarli emas'}")

    print("\n=== 5. Namuna olish: namunalar qanchalik 'haqiqiy'? ===")
    chegara = np.quantile(haqiqiy_logp(test), 0.05)      # haqiqiy nuqtalarning 95% shundan yuqori
    print(f"  o'lchov: haqiqiy log-zichlik {chegara:.2f} dan yuqori ulush (haqiqiy ma'lumotda 0.95)")
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        for oila, (nom, _, yasa) in tanlangan.items():
            m = yasa().fit(oquv)
            if oila == "gistogramma":
                x = m.sample(2000, np.random.default_rng(1))
            elif oila == "KDE":
                x = m.sample(2000, random_state=1)
            else:
                x = m.sample(2000)[0]
            lp = haqiqiy_logp(x)
            ichida = (lp > chegara).mean()
            halqa = (np.abs(np.linalg.norm(x, axis=1) - R) < 3 * SIGMA_R).mean()
            print(f"  {oila:<11} ichida {ichida:.3f}, halqada {halqa:.3f} (haqiqiy ~{ULUSH_HALQA * 0.997:.3f})")
            if oila == "GMM":
                gmm_x = x

    print("\n=== 6. Haqiqiy ma'lumot va GMM namunalari (ASCII zichlik) ===")
    chet = np.linspace(-2.75, 2.75, 12)
    rasmlar = []
    for x in (test, gmm_x):
        c, _, _ = np.histogram2d(x[:, 1], x[:, 0], bins=[chet, np.linspace(-2.75, 2.75, 23)])
        c = np.sqrt(c[::-1] / c.max())                # ildiz: halqa ham ko'rinsin
        rasmlar.append(["".join(" .:-=+*#%@"[min(9, int(v * 10))] for v in q) for q in c])
    print(f"  {'haqiqiy (test)':<22}    GMM namunalari")
    for a, b in zip(*rasmlar):
        print(f"  {a}    {b}")


if __name__ == "__main__":
    main()

=== 1. Haqiqiy taqsimot: halqa (60%) + markaz (40%) ===
  o'quv 1000, val 1000, test 2000 nuqta
  haqiqiy zichlikning test log-likelihoodi: -2.062 (shift)

=== 2. Nol hisob tuzog'i: silliqlashsiz gistogramma ===
  b=10  bo'sh katakdagi val nuqtalari   4 ta -> val LL = -inf
  b=40  bo'sh katakdagi val nuqtalari  76 ta -> val LL = -inf

=== 3. Har oilada val LL bo'yicha tanlash ===
  gistogramma (o'quv/val): b=5 -2.74/-2.80, b=10 -2.51/-2.59, b=20 -2.27/-2.39, b=40 -2.33/-2.63
              -> tanlandi b=20
  KDE         (o'quv/val): h=0.05 -1.62/-2.39, h=0.1 -1.95/-2.13, h=0.2 -2.16/-2.22, h=0.4 -2.55/-2.60
              -> tanlandi h=0.1
  GMM         (o'quv/val): K=1 -3.02/-3.07, K=4 -2.49/-2.58, K=8 -2.13/-2.17, K=16 -2.01/-2.12, K=32 -1.96/-2.17
              -> tanlandi K=16

=== 4. Tanlangan modellar: 5 ta ma'lumot urug'ida test LL (juftlashgan) ===
  gistogramma b=20   test LL -2.361   saqlanadigan sonlar: 399
  KDE         h=0.1  test LL -2.125   saqlanadigan sonlar: 2000
  GMM         K=16   test LL -2.108   saqlanadigan sonlar: 95
  gistogramma - GMM: -0.253, SE 0.009 -> sezilarli yomon
  KDE - GMM: -0.018, SE 0.005 -> sezilarli yomon

=== 5. Namuna olish: namunalar qanchalik 'haqiqiy'? ===
  o'lchov: haqiqiy log-zichlik -3.80 dan yuqori ulush (haqiqiy ma'lumotda 0.95)
  gistogramma ichida 0.758, halqada 0.503 (haqiqiy ~0.598)
  KDE         ichida 0.902, halqada 0.582 (haqiqiy ~0.598)
  GMM         ichida 0.959, halqada 0.589 (haqiqiy ~0.598)

=== 6. Haqiqiy ma'lumot va GMM namunalari (ASCII zichlik) ===
  haqiqiy (test)            GMM namunalari
                                       .
        :----=---.               .:--=-=-::.
     .:=-:..   ::---.          .-==-:.   .:-=:
    .==.          :=-.        .--:     .  . :=-.
    -=:   .=++=.   :--        -=-   .:#+-:   :=-
   .==.  :-#@@*-   :-=:       :-.  .=*@%*=.  .--
    :-.   .-*+=:   .=:       .:=.   :=++=:   :=:
    .--.    .     :=-:        .--.          .=:
     .:=-::. ..::=-:          ..:--::    ::--:.
       .:---:-=-::.               :--:--==-:.
            :

"""Digits da GMM (PCA dan keyin) bilan yangi raqamlar yaratish va ularni klassifikator bilan tekshirish."""

import warnings

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.mixture import GaussianMixture
from sklearn.model_selection import train_test_split

BELGI = " .:-=+*#%@"


def ascii_qator(rasmlar, izohlar):
    """8x8 rasmlarni (0..16) yonma-yon ASCII qatorlarga aylantiradi."""
    satrlar = ["   ".join(f"{s:<8}" for s in izohlar)]
    for r in range(8):
        satrlar.append("   ".join(
            "".join(BELGI[min(9, int(v / 16 * 10))] for v in np.clip(im.reshape(8, 8)[r], 0, 16))
            for im in rasmlar))
    return satrlar


class Klassifikator(nn.Module):
    """O'zimiz o'rgatadigan 'hakam': 64 -> 128 -> 10."""

    def __init__(self):
        super().__init__()
        self.h = nn.Linear(64, 128)
        self.chiq = nn.Linear(128, 10)

    def forward(self, x):
        return self.chiq(F.relu(self.h(x / 16)))


def klassifikator_orgat(x, y, seed=0):
    torch.manual_seed(seed)
    m = Klassifikator()
    opt = torch.optim.Adam(m.parameters(), lr=1e-2, weight_decay=1e-4)
    xt, yt = torch.tensor(x, dtype=torch.float32), torch.tensor(y)
    g = torch.Generator().manual_seed(seed)
    for _ in range(60):
        for idx in torch.randperm(len(xt), generator=g).split(64):
            loss = F.cross_entropy(m(xt[idx]), yt[idx])
            opt.zero_grad()
            loss.backward()
            opt.step()
    return m.eval()


@torch.no_grad()
def baholash(m, x):
    p = torch.softmax(m(torch.tensor(x, dtype=torch.float32)), -1).numpy()
    return p.argmax(1), p.max(1)


def main() -> None:
    torch.set_num_threads(1)
    d = load_digits()
    x_oquv, x_test, y_oquv, y_test = train_test_split(
        d.data, d.target, test_size=0.25, stratify=d.target, random_state=0)
    x_o, x_v = x_oquv[:1000], x_oquv[1000:]

    print("=== 1. Hakam-klassifikator (generativ modeldan mustaqil) ===")
    hakam = klassifikator_orgat(x_oquv, y_oquv)
    pred, ish = baholash(hakam, x_test)
    print(f"  test aniqligi {np.mean(pred == y_test):.3f}; haqiqiy test rasmlarida "
          f"ishonch > 0.9 ulushi {np.mean(ish > 0.9):.3f}")

    print("\n=== 2. PCA(20) + GMM: komponentlar sonini val LL bilan tanlash ===")
    pca = PCA(20, random_state=0).fit(x_o)
    print(f"  PCA 20 komponent dispersiyaning {pca.explained_variance_ratio_.sum():.3f} qismini saqlaydi")
    z_o, z_v = pca.transform(x_o), pca.transform(x_v)
    natija = []
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        for k in (1, 5, 10, 20, 40):
            gm = GaussianMixture(k, covariance_type="full", random_state=0).fit(z_o)
            natija.append((k, gm.score(z_o), gm.score(z_v)))
            print(f"  K={k:<3} o'quv LL {natija[-1][1]:8.2f}, val LL {natija[-1][2]:8.2f}")
    eng_k = max(natija, key=lambda t: t[2])[0]
    print(f"  -> tanlandi K={eng_k} (o'quv LL K bilan doim o'sadi, val LL - yo'q)")

    print("\n=== 3. 1000 ta yangi raqam: tanib olinish va sinf taqsimoti ===")
    z_all = pca.transform(x_oquv)
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        modellar = {f"K={k}": GaussianMixture(k, covariance_type="full", random_state=0).fit(z_all)
                    for k in sorted({1, eng_k})}
    namunalar, tanish = {}, {}
    for nom, gm in modellar.items():
        z, komp = gm.sample(1000)                   # DIQQAT: komponentlar bo'yicha tartiblangan!
        if nom != "K=1":
            print(f"  {nom}: gm.sample dagi dastlabki 6 komponent: {komp[:6].tolist()} -> aralashtiramiz")
        z = z[np.random.default_rng(1).permutation(len(z))]
        x = np.clip(pca.inverse_transform(z), 0, 16)
        namunalar[nom] = x
        p, i = baholash(hakam, x)
        soni = np.bincount(p, minlength=10)
        ulush = soni / soni.sum()
        tanish[nom] = np.mean(i > 0.9)
        entr = -(ulush[ulush > 0] * np.log(ulush[ulush > 0])).sum() / np.log(10)
        print(f"  {nom:<5} ishonch > 0.9: {np.mean(i > 0.9):.3f}; sinflar: {' '.join(map(str, soni))}; "
              f"entropiya {entr:.3f}")
    print(f"  haqiqiy test: ishonch > 0.9: {np.mean(ish > 0.9):.3f}")
    a, b = tanish["K=1"], tanish[f"K={eng_k}"]
    se = np.sqrt(a * (1 - a) / 1000 + b * (1 - b) / 1000)
    print(f"  K={eng_k} - K=1: {b - a:+.3f}, SE {se:.3f} -> "
          f"{'sezilarli' if abs(b - a) > 2 * se else 'sezilarli emas'}")

    print("\n=== 4. Namunalar (ASCII) ===")
    for nom in modellar:
        x = namunalar[nom][:6]
        p, i = baholash(hakam, x)
        for s in ascii_qator(x, [f"{nom}:{a}" for a in p]):
            print("  " + s)


if __name__ == "__main__":
    main()

=== 1. Hakam-klassifikator (generativ modeldan mustaqil) ===
  test aniqligi 0.971; haqiqiy test rasmlarida ishonch > 0.9 ulushi 0.947

=== 2. PCA(20) + GMM: komponentlar sonini val LL bilan tanlash ===
  PCA 20 komponent dispersiyaning 0.897 qismini saqlaydi
  K=1   o'quv LL   -64.37, val LL   -64.49
  K=5   o'quv LL   -58.14, val LL   -59.18
  K=10  o'quv LL   -54.57, val LL   -57.69
  K=20  o'quv LL   -50.07, val LL   -61.56
  K=40  o'quv LL   -37.46, val LL   -96.88
  -> tanlandi K=10 (o'quv LL K bilan doim o'sadi, val LL - yo'q)

=== 3. 1000 ta yangi raqam: tanib olinish va sinf taqsimoti ===
  K=1   ishonch > 0.9: 0.478; sinflar: 86 85 113 138 104 69 97 106 113 89; entropiya 0.993
  K=10: gm.sample dagi dastlabki 6 komponent: [0, 0, 0, 0, 0, 0] -> aralashtiramiz
  K=10  ishonch > 0.9: 0.865; sinflar: 84 90 90 104 96 105 102 97 116 116; entropiya 0.998
  haqiqiy test: ishonch > 0.9: 0.947
  K=10 - K=1: +0.387, SE 0.019 -> sezilarli

=== 4. Namunalar (ASCII) ===
  K=1:6      K=1:7      K=1:5      K=1:0      K=1:7      K=1:8
    -*#=       -#%+       :%@*        #@+-        @@=      #@*
   .*+-=      .*=+:      :@*=#       +@@*-       #@@+     :@@@=
   :@+:=       --@=      -# :+       #* -:      :+-@=     .+%@=
   .@@#=      -=+@@-     :=:+=.     :@= =-     :*==@+      *@@=.
    =-:*:     =%+*@=     .. .+-     =@@=@+     -@%@@-      +@+=.
   .*-.#*     :-=:-      .== ==     :@=:%+     ..:@%       ---#:
   .@*-#+       ..        +*.+:      #@@*         @-       #+@@
    -*@*       =*:        -@#=.      .%*         .+        #@@.
  K=10:2     K=10:4     K=10:7     K=10:9     K=10:7     K=10:5
    *@-         -%:.      =##*-      -+#=       *%#        -*@%:
   -%#+        :#% .     .%%@#:     .#+%@.     :@+@=       @*--:
   .-=@       .=#::=      *=*+      :%*#@.     .: @-      :@-+.
     *@       :@@=@*      . -+.     .+*##.     .-+%+:     -@@@#.
     #%       -@@*@=     .=@@@-        :*.     .#@#=.     .  -@:
    -*+-.     .:++%.     .#@++:       .:%-      =%        ::  @:
    #%%%=       :#*       .# ..      =.=@:      =*        .%=-*
    *@@+:      .:@-       +*         -+@#.      #+ .       -#*.

"""Avtoregressiv piksel modeli: binar 8x8 raqamlar, p(x) = p(x1) p(x2|x1) ... p(x64|x1..x63)."""

import math

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

D = 64


class Mustaqil(nn.Module):
    """Bazaviy: har piksel alohida Bernoulli - piksellar orasida bog'liqlik yo'q."""

    def __init__(self):
        super().__init__()
        self.b = nn.Parameter(torch.zeros(D))

    def forward(self, x):
        return self.b.expand(len(x), D)


class NiqobliChiziqli(nn.Linear):
    def __init__(self, kirish, chiqish, niqob):
        super().__init__(kirish, chiqish)
        self.register_buffer("niqob", niqob.float())

    def forward(self, x):
        return F.linear(x, self.weight * self.niqob, self.bias)


class MADE(nn.Module):
    """Avtoregressiv MLP: i-chiqish faqat 0..i-1 piksellarga bog'liq (niqoblar bilan)."""

    def __init__(self, h=128, seed=0):
        super().__init__()
        g = torch.Generator().manual_seed(seed)
        tartib = torch.arange(D)                                   # piksel tartibi: qator bo'yicha
        daraja_h = torch.randint(0, D - 1, (h,), generator=g)      # yashirin neyron "darajasi"
        self.l1 = NiqobliChiziqli(D, h, daraja_h[:, None] >= tartib[None, :])
        self.l2 = NiqobliChiziqli(h, D, tartib[:, None] > daraja_h[None, :])
        self.to_g = NiqobliChiziqli(D, D, tartib[:, None] > tartib[None, :])   # to'g'ridan-to'g'ri

    def forward(self, x):
        return self.l2(torch.relu(self.l1(x))) + self.to_g(x)


class ChiziqliAR(nn.Module):
    """Eng sodda avtoregressiv model: i-piksel logiti = oldingi piksellarning chiziqli funksiyasi."""

    def __init__(self):
        super().__init__()
        tartib = torch.arange(D)
        self.l = NiqobliChiziqli(D, D, tartib[:, None] > tartib[None, :])

    def forward(self, x):
        return self.l(x)


class Klassifikator(nn.Module):
    def __init__(self):
        super().__init__()
        self.h = nn.Linear(D, 128)
        self.chiq = nn.Linear(128, 10)

    def forward(self, x):
        return self.chiq(F.relu(self.h(x)))


def klassifikator_orgat(x, y, seed=0):
    torch.manual_seed(seed)
    m = Klassifikator()
    opt = torch.optim.Adam(m.parameters(), lr=1e-2, weight_decay=1e-4)
    g = torch.Generator().manual_seed(seed)
    for _ in range(60):
        for idx in torch.randperm(len(x), generator=g).split(64):
            loss = F.cross_entropy(m(x[idx]), y[idx])
            opt.zero_grad()
            loss.backward()
            opt.step()
    return m.eval()


def nll_bit(model, x):
    """Bitta rasm uchun o'rtacha -log2 p(x) (bit)."""
    with torch.no_grad():
        return F.binary_cross_entropy_with_logits(model(x), x, reduction="none").sum(1).mean().item() / math.log(2)


def orgat(klass, x, qadamlar=1500, seed=0):
    torch.manual_seed(seed)
    model = klass()
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        idx = torch.randint(0, len(x), (128,), generator=g)
        loss = F.binary_cross_entropy_with_logits(model(x[idx]), x[idx], reduction="none").sum(1).mean()
        opt.zero_grad()
        loss.backward()
        opt.step()
    return model.eval()


@torch.no_grad()
def namuna_ol(model, n, g):
    """Piksel-piksel: 64 qadam, har qadamda bitta pikselni Bernoulli dan olamiz."""
    x = torch.zeros(n, D)
    for i in range(D):
        p = torch.sigmoid(model(x)[:, i])
        x[:, i] = torch.bernoulli(p, generator=g)
    return x


def main() -> None:
    torch.set_num_threads(1)
    d = load_digits()
    xb = (d.data > 7).astype(np.float32)
    x_o, x_t, y_o, y_t = train_test_split(xb, d.target, test_size=0.25, stratify=d.target, random_state=0)
    xo, xt = torch.tensor(x_o), torch.tensor(x_t)

    print("=== 1. Test NLL (bit/rasm; kam - yaxshi) ===")
    print(f"  tekis tasodif (har piksel 0.5): {D:.1f} bit")
    modellar = {}
    for nom, m in [("mustaqil", Mustaqil), ("chiziqli", ChiziqliAR), ("MADE", MADE)]:
        m = orgat(m, xo)
        modellar[nom] = m
        print(f"  {nom:<9} parametr {sum(p.numel() for p in m.parameters()):>6}: "
              f"o'quv {nll_bit(m, xo):6.2f}, test {nll_bit(m, xt):6.2f} bit")
    with torch.no_grad():
        har = {n: F.binary_cross_entropy_with_logits(m(xt), xt, reduction="none").sum(1).numpy() / math.log(2)
               for n, m in modellar.items()}
    f = har["MADE"] - har["chiziqli"]
    se = f.std(ddof=1) / math.sqrt(len(f))
    print(f"  MADE - chiziqli (juftlashgan, {len(f)} test rasm): {f.mean():+.2f} bit, SE {se:.2f} -> "
          f"{'MADE sezilarli yaxshi' if f.mean() < -2 * se else "sezilarli farq yo'q"}")

    hakam = klassifikator_orgat(xo, torch.tensor(y_o))
    with torch.no_grad():
        pt = torch.softmax(hakam(xt), -1)
    print(f"\n=== 2. Piksel-piksel namunalar (hakam: binar test aniqligi "
          f"{(pt.argmax(1).numpy() == y_t).mean():.3f}) ===")
    print(f"  haqiqiy test:  ishonch > 0.9 ulushi {(pt.max(1).values > 0.9).float().mean().item():.3f}")
    namunalar = {}
    for nom, m in modellar.items():
        x = namuna_ol(m, 1000, torch.Generator().manual_seed(1))
        namunalar[nom] = x
        with torch.no_grad():
            p = torch.softmax(hakam(x), -1)
        soni = np.bincount(p.argmax(1).numpy(), minlength=10)
        print(f"  {nom:<9}  ishonch > 0.9 ulushi {(p.max(1).values > 0.9).float().mean().item():.3f}; "
              f"sinflar: {' '.join(map(str, soni))}")
        if nom != "chiziqli":
            for r in range(8):
                print("      " + "   ".join("".join("#" if v else "." for v in im.view(8, 8)[r].tolist())
                                           for im in x[:6]))

    print("\n=== 3. LL ning cheklovi: 1% MADE + 99% tasodifiy shovqin ===")
    nll_aralash = -np.log2(0.01 * 2.0 ** (-har["MADE"]) + 0.99 * 2.0 ** (-D))
    print(f"  aralash model test NLL: {nll_aralash.mean():.2f} bit "
          f"(MADE {har['MADE'].mean():.2f} + ~{math.log2(100):.2f})")
    print(f"  mustaqil model test NLL: {har['mustaqil'].mean():.2f} bit")
    g = torch.Generator().manual_seed(2)
    tanlov = torch.rand(1000, generator=g) < 0.01
    x = torch.where(tanlov[:, None], namuna_ol(modellar["MADE"], 1000, g),
                    torch.bernoulli(torch.full((1000, D), 0.5), generator=g))
    print(f"  aralash modeldan 1000 namuna: MADE dan {int(tanlov.sum())} ta, qolgani shovqin")
    print("  manba              ishonch > 0.9   eng yaqin o'quv rasmigacha (Hamming)")
    for nom, v in [("haqiqiy test", xt), ("MADE", namunalar["MADE"]),
                   ("mustaqil", namunalar["mustaqil"]), ("aralash", x)]:
        with torch.no_grad():
            ish = torch.softmax(hakam(v), -1).max(1).values
        masofa = torch.cdist(v, xo, p=1).min(1).values
        print(f"  {nom:<18} {(ish > 0.9).float().mean().item():>12.3f}   {masofa.mean().item():>10.2f}")
    if nll_aralash.mean() < har["mustaqil"].mean():
        print("  -> LL bo'yicha aralash 'mustaqil' dan yaxshi, lekin namunalarning ~99% i shovqin")


if __name__ == "__main__":
    main()

=== 1. Test NLL (bit/rasm; kam - yaxshi) ===
  tekis tasodif (har piksel 0.5): 64.0 bit
  mustaqil  parametr     64: o'quv  37.83, test  37.98 bit
  chiziqli  parametr   4160: o'quv  26.08, test  26.57 bit
  MADE      parametr  20736: o'quv  23.19, test  25.26 bit
  MADE - chiziqli (juftlashgan, 450 test rasm): -1.31 bit, SE 0.13 -> MADE sezilarli yaxshi

=== 2. Piksel-piksel namunalar (hakam: binar test aniqligi 0.933) ===
  haqiqiy test:  ishonch > 0.9 ulushi 0.920
  mustaqil   ishonch > 0.9 ulushi 0.520; sinflar: 48 113 117 114 82 52 55 82 245 92
      ...##...   ...##...   ....#...   ...##...   #..###..   ...##...
      ...###..   ....###.   ..#..#..   ...#.#..   #.##.#..   .#.##...
      ...#..#.   ...##...   .##..#..   #.###...   .##.##..   ..#...#.
      ##..#...   ..####.#   .####.#.   ...#....   ...##.#.   ..###...
      ..#.##..   ...##...   ..#.###.   ...##...   ...##..#   ###.....
      .....#..   ....#.#.   .....#..   ......#.   ..#...#.   ..#.#...
      ..#.###.   ..###...   .#.##...   ...###.#   ...##.#.   ...###..
      ...#.#..   ..###...   ..##.#..   ...#.#..   ...#..#.   ...##...
  chiziqli   ishonch > 0.9 ulushi 0.689; sinflar: 80 126 107 130 87 111 75 83 86 115
  MADE       ishonch > 0.9 ulushi 0.727; sinflar: 71 120 96 97 104 101 98 88 96 129
      ...##...   ...##...   ....#...   ....#...   ...###..   ...##...
      ...##...   ...##...   ........   ...#....   ..##.#..   ...##...
      ...#....   ...##...   ..##....   ..##....   .##..#..   ...#..#.
      ....#...   ..###...   .####.#.   ..##....   ..###...   ..##..#.
      ..####..   ..###...   ..####..   ..###...   ...##...   .###..#.
      ..##.#..   ...##...   .....#..   ..###.#.   ...##...   ..###...
      ..#.###.   ...##...   ....#...   ...##.#.   ...##...   ...###..
      ...###..   ...##...   ........   ...###..   ...#....   ...##...

=== 3. LL ning cheklovi: 1% MADE + 99% tasodifiy shovqin ===
  aralash model test NLL: 31.89 bit (MADE 25.26 + ~6.64)
  mustaqil model test NLL: 37.98 bit
  aralash modeldan 1000 namuna: MADE dan 13 ta, qolgani shovqin
  manba              ishonch > 0.9   eng yaqin o'quv rasmigacha (Hamming)
  haqiqiy test              0.920         3.27
  MADE                      0.727         4.63
  mustaqil                  0.520         9.30
  aralash                   0.539        22.76
  -> LL bo'yicha aralash 'mustaqil' dan yaxshi, lekin namunalarning ~99% i shovqin

"""Yodlab olish tekshiruvi: namunalar o'quv rasmlarining nusxasimi?"""

import warnings

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.mixture import GaussianMixture
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KernelDensity

BELGI = " .:-=+*#%@"


class Klassifikator(nn.Module):
    def __init__(self):
        super().__init__()
        self.h = nn.Linear(64, 128)
        self.chiq = nn.Linear(128, 10)

    def forward(self, x):
        return self.chiq(F.relu(self.h(x / 16)))


def klassifikator_orgat(x, y, seed=0):
    torch.manual_seed(seed)
    m = Klassifikator()
    opt = torch.optim.Adam(m.parameters(), lr=1e-2, weight_decay=1e-4)
    xt, yt = torch.tensor(x, dtype=torch.float32), torch.tensor(y)
    g = torch.Generator().manual_seed(seed)
    for _ in range(60):
        for idx in torch.randperm(len(xt), generator=g).split(64):
            loss = F.cross_entropy(m(xt[idx]), yt[idx])
            opt.zero_grad()
            loss.backward()
            opt.step()
    return m.eval()


@torch.no_grad()
def ishonch(m, x):
    return torch.softmax(m(torch.tensor(x, dtype=torch.float32)), -1).max(1).values.numpy()


def eng_yaqin(x, baza):
    """Har x uchun eng yaqin baza rasmigacha Evklid masofa va uning indeksi."""
    d = torch.cdist(torch.tensor(x, dtype=torch.float32), torch.tensor(baza, dtype=torch.float32))
    m = d.min(1)
    return m.values.numpy(), m.indices.numpy()


def ascii_juftlar(chap, ong, izoh):
    satrlar = [izoh]
    for r in range(8):
        satrlar.append("   ".join(
            "".join(BELGI[min(9, int(v / 16 * 10))] for v in np.clip(a.reshape(8, 8)[r], 0, 16)) + " | " +
            "".join(BELGI[min(9, int(v / 16 * 10))] for v in np.clip(b.reshape(8, 8)[r], 0, 16))
            for a, b in zip(chap, ong)))
    return satrlar


def main() -> None:
    torch.set_num_threads(1)
    d = load_digits()
    x_oquv, x_test, y_oquv, y_test = train_test_split(
        d.data, d.target, test_size=0.25, stratify=d.target, random_state=0)
    x_o, x_v = x_oquv[:1000], x_oquv[1000:]
    hakam = klassifikator_orgat(x_oquv, y_oquv)
    pca = PCA(20, random_state=0).fit(x_o)
    z_o, z_v, z_t = pca.transform(x_o), pca.transform(x_v), pca.transform(x_test)

    print("=== 1. Ma'lumotnoma: haqiqiy test rasmlari o'quvdan qanchalik uzoq? ===")
    d_test, _ = eng_yaqin(x_test, x_o)
    chegara = np.quantile(d_test, 0.05)
    print(f"  test -> eng yaqin o'quv rasmi: median {np.median(d_test):.2f}, 5-foizil {chegara:.2f}")
    print(f"  'nusxa' deymiz: eng yaqin o'quvgacha masofa < {chegara:.2f} "
          f"(haqiqiy test rasmlarining atigi 5% i)")

    print("\n=== 2. KDE kengligini val LL bilan tanlash (PCA-20 fazosida) ===")
    qator = []
    for h in (0.5, 1.0, 2.0, 3.0, 4.0, 6.0):
        kd = KernelDensity(bandwidth=h).fit(z_o)
        qator.append((h, kd.score_samples(z_o).mean(), kd.score_samples(z_v).mean()))
    for h, tr, va in qator:
        print(f"  h={h:<4} o'quv LL {tr:8.2f}, val LL {va:8.2f}")
    eng_h = max(qator, key=lambda t: t[2])[0]
    print(f"  -> tanlandi h={eng_h}")

    print("\n=== 3. Uch model: LL va namunalar ===")
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        modellar = {
            "yodlovchi (KDE h=0.5)": KernelDensity(bandwidth=0.5).fit(z_o),
            f"KDE h={eng_h}": KernelDensity(bandwidth=eng_h).fit(z_o),
            "GMM K=10": GaussianMixture(10, random_state=0).fit(z_o),
        }
    print("  model                  o'quv LL   test LL   ishonch>0.9   masofa(med)   nusxa")
    namunalar = {}
    for nom, m in modellar.items():
        if isinstance(m, KernelDensity):
            z = m.sample(1000, random_state=1)
        else:
            z = m.sample(1000)[0][np.random.default_rng(1).permutation(1000)]
        x = np.clip(pca.inverse_transform(z), 0, 16)
        namunalar[nom] = x
        dd, _ = eng_yaqin(x, x_o)
        print(f"  {nom:<22} {m.score_samples(z_o).mean():8.2f} {m.score_samples(z_t).mean():9.2f} "
              f"{np.mean(ishonch(hakam, x) > 0.9):12.3f} {np.median(dd):12.2f} {np.mean(dd < chegara):8.3f}")
    print(f"  {'haqiqiy test':<22} {'':>8} {'':>9} {np.mean(ishonch(hakam, x_test) > 0.9):12.3f} "
          f"{np.median(d_test):12.2f} {np.mean(d_test < chegara):8.3f}")

    print("\n=== 4. Namuna | eng yaqin o'quv rasmi ===")
    for nom in ("yodlovchi (KDE h=0.5)", "GMM K=10"):
        x = namunalar[nom][:4]
        _, idx = eng_yaqin(x, x_o)
        for s in ascii_juftlar(x, x_o[idx], f"  {nom}:"):
            print("  " + s)


if __name__ == "__main__":
    main()

=== 1. Ma'lumotnoma: haqiqiy test rasmlari o'quvdan qanchalik uzoq? ===
  test -> eng yaqin o'quv rasmi: median 17.38, 5-foizil 12.02
  'nusxa' deymiz: eng yaqin o'quvgacha masofa < 12.02 (haqiqiy test rasmlarining atigi 5% i)

=== 2. KDE kengligini val LL bilan tanlash (PCA-20 fazosida) ===
  h=0.5  o'quv LL   -11.42, val LL  -273.35
  h=1.0  o'quv LL   -25.29, val LL   -88.58
  h=2.0  o'quv LL   -39.15, val LL   -62.01
  h=3.0  o'quv LL   -47.25, val LL   -57.76
  h=4.0  o'quv LL   -52.98, val LL   -58.60
  h=6.0  o'quv LL   -60.82, val LL   -62.74
  -> tanlandi h=3.0

=== 3. Uch model: LL va namunalar ===
  model                  o'quv LL   test LL   ishonch>0.9   masofa(med)   nusxa
  yodlovchi (KDE h=0.5)    -11.42   -323.93        0.954        10.20    0.784
  KDE h=3.0                -47.25    -58.66        0.879        15.14    0.081
  GMM K=10                 -54.57    -57.97        0.902        18.20    0.017
  haqiqiy test                                     0.947        17.38    0.051

=== 4. Namuna | eng yaqin o'quv rasmi ===
    yodlovchi (KDE h=0.5):
     *+#=  |    *%#-      %%@%:  |  :%@@@.       -*@#. |    *#@@%     .%@-   |   :#%-
    *%.*=  |   %%::#     =%-+@-  |  *#=+@+      -@:-%. |   -@-:%+    .@+=#   |  .@+*@.
   .@=.+:  |  .@=:#-     .  *@.  |     #@.      #= :*  |   *- .%.    :%.:%.  |  .%..@.
   :@@@-   |  .@@@-        -@+   |    :@*      .+..*=  |   %  #-      *%@*   |   *@@=
    .+@:   |    +@         +@-   |    +@:       . -*:  |   . -%        +##.  |    =%@.
     -%.   |    .@         .@*   |     @*         *:   |     %.       :+.%=  |   .* #-
     #=    |    *-        -.@%   |  -=:@%        +%    |    -%        ++ #:  |   =+ %.
     %:    |    @         @@%=.  |  -@@@=        *-    |    %-        :@%=   |   :@@=
    GMM K=10:
    +@#*.  |   %@@:       -%@=   |   -@%.         %-   |    .%        -#@#-  |   :%@%
   :@**@:  |  :@+*@.      +#*@   |   #@#%        +@    |    %#        +##%-  |   %**@
   -@-+@.  |  -@ +@.      -@@%   |   ##%%       .%: :  |   =@-.         -#:  |     +*
   :**%*   |  .%*@@-      :@@-   |   .@%.      :@%.+=  |  .@*-@.       -*%=  |    -%@+
    :==#-  |   .*+@-     .#@%    |   +%@.      -@**#.  |  -@**@-      +@@#:  |   *@@+-
    . .#=  |      @-     .@.+*   |   @-+#      -*+%@   |   -#@%.      :#+    |   .**
    ::+@-  |   ..*@      .@-#%   |   @::@:     .=-@%   |     @=       .#-    |    %-
    +@@=   |   %@%:       =@@#:  |   -@@#.        @*.  |    .@=       =#.    |   -@.

eng = max(modellar, key=lambda m: m.score(x_oquv))                  # ⚠️
eng = max(modellar, key=lambda m: m.score(x_val))                   # ✅

p = soni / soni.sum()                   # bo'sh katak -> log 0 = -inf  # ⚠️
p = (soni + 0.5) / (soni + 0.5).sum()                               # ✅

x, _ = gm.sample(1000); korsat(x[:6])      # hammasi 0-komponentdan   # ⚠️
x, _ = gm.sample(1000)
x = x[np.random.default_rng(1).permutation(len(x))]; korsat(x[:6])  # ✅

print(gmm_pca20.score(z_test), made_bit_rasm)                       # ⚠️
# bir xil fazo va birlikda: hammasi PCA-20 da yoki hammasi bit/rasm da  # ✅

niqob = tartib[:, None] >= tartib[None, :]   # piksel o'zini ko'radi  # ⚠️
niqob = tartib[:, None] > tartib[None, :]                           # ✅

modellar = [MADE(), ChiziqliAR()]; torch.manual_seed(0)             # ⚠️
torch.manual_seed(0); model = MADE()                                # ✅

print("namunalar juda realistik!")                                  # ⚠️
d_nn = torch.cdist(namuna, oquv).min(1).values
print((d_nn < np.quantile(d_test_nn, 0.05)).float().mean())         # ✅

1.  # diskriminativ va generativ model nimani o'rganadi?
2.  # p(x|y) va p(y) dan p(y|x) ni qanday olamiz?
3.  # max E[log p_theta(x)] qaysi KL ni minimallashtiradi?
4.  # silliqlashsiz gistogrammada bitta val nuqtasi bo'sh katakda - o'rtacha LL?
5.  # KDE da h -> 0 bo'lsa o'quv LL va test LL nima bo'ladi?
6.  # 8x8 binar rasm uchun tekis tasodif necha bit?
7.  # avtoregressiv modelda namuna olish uchun nechta forward kerak (64 piksel)?
8.  # 1% model + 99% shovqin aralashmasi LL ni necha bitga yomonlashtiradi (ko'pi bilan)?
9.  # GAN p(x) ni beradimi?
10. # nega hakam-klassifikator ishonchi yetarli o'lchov emas?
11. # "nusxa" chegarasini nimaga qarab tanlaymiz?
12. # PCA-20 dagi GMM LL ini bit/rasm dagi MADE NLL i bilan solishtirsa bo'ladimi?

1.  natija = {k: GaussianMixture(k).fit(x_oquv).score(x_oquv) for k in (1, 5, 10, 40)}
    eng_k = max(natija, key=natija.get)

2.  x, _ = gm.sample(500)
    print("birinchi 10 namuna sinflari:", hakam(x[:10]).argmax(1))

3.  niqob = torch.tril(torch.ones(64, 64))
    logit = F.linear(x, W * niqob, b)

4.  models = [MADE() for _ in range(3)]
    for s, m in enumerate(models):
        torch.manual_seed(s); orgat(m)

5.  print("yangi raqamlar sifatli:", (hakam_ishonch(namunalar) > 0.9).mean())