24.8-dars: BERT — maskali til modeli

UCH OILA (bitta Transformer blokidan):
  encoder-only  (BERT)        - ikki tomonlama, MLM bilan o'rgatiladi
                                tushunish: klassifikatsiya, teglash, qidiruv
  decoder-only  (GPT)         - kauzal, keyingi token bilan o'rgatiladi
                                generatsiya
  encoder-decoder (asl, T5)   - kirish -> chiqish (tarjima, qisqartirish)

NEGA ODDIY TIL MODELI IKKI TOMONLAMA BO'LA OLMAYDI:
  kauzal LM: nishon y_t = x_{t+1}
  ikki tomonlama model x_{t+1} ni kirishda ko'radi -> shunchaki ko'chiradi
  (24.7, 2-misol: loss 0.045, halol PPL 453)
  ELMo (2018) ikki alohida LSTM (chap-o'ng va o'ng-chap) ni birlashtirgan -
  lekin har biri bir tomonlama, "chuqur" ikki tomonlama emas

MLM YECHIMI:
  nishonni kirishdan OLIB TASHLASH: x_t -> [MASK]
  model x_t ni ikkala tomondagi kontekstdan tiklaydi
  P(x_t | x_1 ... x_{t-1}, [MASK], x_{t+1} ... x_T)

ARXITEKTURA:
  24.6-darsdagi encoder + MLM boshi (Linear d -> V)
  BERT-base: 12 qatlam, d = 768, 12 bosh, 110M parametr
  bizniki:    2 qatlam, d = 64,   4 bosh, ~82 ming parametr

1. TANLASH: har oddiy token 15% ehtimol bilan tanlanadi
   maxsus tokenlar ([CLS], [SEP], [PAD]) HECH QACHON tanlanmaydi

2. TANLANGANLARNI ALMASHTIRISH:
   80% -> [MASK]
   10% -> lug'atdan tasodifiy token
   10% -> o'zgarishsiz qoladi

3. LOSS: faqat tanlangan pozitsiyalarda
   nishon = asl token, qolgan joylarda -100 (ignore_index)

NEGA 80/10/10, 100% [MASK] EMAS:
  fine-tuning va real ishlatishda [MASK] tokeni UCHRAMAYDI
  100% [MASK] bo'lsa: model faqat [MASK] ko'rgan joyda "o'ylaydi",
    qolgan tokenlar vakilligiga e'tibor bermasligi mumkin
  10% tasodifiy: model ko'rinib turgan tokenga ham to'liq ishonmaydi
  10% o'zgarmagan: to'g'ri token ham baholanadi - vakillik real kirishga moslashadi

1-MISOL STATISTIKASI (20000 jumla):
  tanlangan 0.1491; ulardan [MASK] 0.8009, tasodifiy 0.1012, o'zgarmagan 0.0979
  tasodifiy token ba'zan asl tokenga teng chiqadi: 0.0084 (~1/105)

DINAMIK MASKALASH (RoBERTa):
  har epoxada yangi niqob: 10 epoxada tokenlarning 0.8029 qismi
  kamida bir marta nishon bo'ladi (1 - 0.85^10 = 0.8031)
  statik niqob (asl BERT - ma'lumotni oldindan maskalash): doim o'sha 15%

NARXI - SIGNAL ZICHLIGI:
  kauzal LM: har token nishon (bizda jumlaga 6.34 ta bashorat)
  MLM: faqat 15% (0.95 ta) - bir xil hisobda kamroq signal
  evaziga: har bashorat ikki tomonlama kontekst bilan
  (ELECTRA kabi keyingi modellar aynan shu kamchilikni tuzatadi)

BIR JUMLA:     [CLS] anvar bozor ##ga ertalab bordi . [SEP]
JUFTLIK:       [CLS] A jumla [SEP] B jumla [SEP]

EMBEDDING = token + pozitsiya + segment    (uchalasi QO'SHILADI)
  token:     WordPiece lug'ati (23.3-dars): "##ga" - so'z davomi
  pozitsiya: O'RGATILADIGAN embedding (24.4-darsdagi sinusoidal emas),
             BERT da maksimal uzunlik 512
  segment:   A -> 0, B -> 1 (ikki vektorli jadval)

3-MISOLDAGI KO'RINISH:
    [CLS] dilshod kutubxona ##ga bugun bordi . [SEP]  u  kutubxona ...
      0      0       0       0    0     0    0   0    1      1       <- segment
      0      1       2       3    4     5    6   7    8      9       <- pozitsiya

[CLS]:
  ma'nosiz token, lekin self-attention orqali butun ketma-ketlikni ko'radi
  uning oxirgi vektori - "butun kirish vektori" (NSP va klassifikatsiya uchun)
  diqqat: o'rgatilmagan [CLS] vektori hali yaxshi "jumla vektori" EMAS

[SEP]: jumla chegarasi; [PAD]: attention da maskalanadi (kalit sifatida)

SEGMENT NIMA BERADI (3-misol, 4-bo'lim):
  hamma segmentni 0 qilsak: MLM 0.6113 -> 0.5718, B qismida 0.6679 -> 0.6163
  model segmentdan "qaysi jumlada turibman" ni biladi

NSP (Next Sentence Prediction, asl BERT):
  50%: B - A dan keyin haqiqatan kelgan jumla (IsNext = 1)
  50%: B - korpusdan tasodifiy jumla (IsNext = 0)
  [CLS] vektoridan ikki sinfli klassifikator; loss = MLM + NSP

MAQSAD: jumlalar orasidagi munosabatni o'rgatish (savol-javob, NLI uchun)

MUAMMO - VAZIFA JUDA OSON VA "YORLIQLI":
  tasodifiy B odatda BOSHQA MAVZUDA -> mavzu/so'z o'xshashligi yetarli
  3-misol: "A va B da umumiy so'z bormi" qoidasi - 0.9647 aniqlik
           BERT ning NSP boshi - 0.83 atrofida (qoidadan ham past)
  MLM aniqligiga ta'siri: +0.0033, SE 0.0024 - sezilarli emas

KEYINGI TADQIQOTLAR:
  RoBERTa (2019): NSP ni olib tashladi, uzun ketma-ket matn bilan
                  natija yomonlashmadi, ba'zi vazifalarda yaxshilandi
  ALBERT: NSP o'rniga SOP (jumlalar tartibi to'g'rimi) - mavzu yorlig'i
          ishlamaydi, chunki ikkala jumla bir matndan
  xulosa: jumlalararo munosabat MLM ning o'zidan ham o'rganiladi

O'LCHOV: [MASK] qo'yilgan pozitsiyalarda top-1 aniqlik
  "o'zgarmagan" 10% ni aralashtirmang - token ko'rinib turibdi, aniqlik oshadi
  (2-misol: [MASK] da 0.5224, o'zgarmaganda 0.7021)

BAZAVIY MODELLAR (2-misol):
  eng ko'p token ('.')             0.1713
  unigram namuna olish             0.0412
  pozitsiya bo'yicha eng ko'p      0.2823   <- eng kuchlisi
  BERT (3 seed o'rtachasi)         0.5261

KAMAYTIRIB BO'LMAYDIGAN NOANIQLIK:
  sintetik korpusda ism, joy, narsa TASODIFIY tanlanadi
  "[MASK] bozor ##ga bordi" - qaysi ism? 28 tadan istalgani
  ism aniqligi 0.0291 ~ tasodif 1/28 = 0.0357
  demak umumiy aniqlikning "shipi" 1 dan ancha past

TOKEN TURI BO'YICHA (BERT va faqat chap kontekst):
  qo'shimcha  0.9836 vs 0.7676   (+0.2160) - o'ngdagi fe'l hal qiladi
  fe'l        0.9698 vs 0.9561   (+0.0137)
  ism         0.0291 vs 0.0213   (sezilarli emas - ikkalasi tasodif)
  umumiy      0.5261 vs 0.4859   (+0.0402, SE 0.0032)

XULOSA:
  bitta umumiy raqam emas - token turlari bo'yicha va bazaviy bilan
  ikki tomonlama kontekst foydasi aynan o'ng kontekst hal qiladigan joyda

STATIK EMBEDDING (23.6-dars, word2vec):
  bitta so'z - bitta vektor
  "yuz" (100) va "yuz" (bet) - AYNAN bir xil vektor

KONTEKSTLI (BERT):
  h_t = f(butun jumla) - bir so'z har jumlada boshqa vektor
  qaysi qatlamdan olish: odatda oxirgi yoki bir necha oxirgi qatlam

4-MISOL (ikki ma'noli so'z doim 2-o'rinda - pozitsiya yordam bermaydi):
  token embedding va (token + pozitsiya): ma'nolar farqi 0.000
  2-blok: 'yuz' bir ma'no ichida 0.853, ma'nolar orasida 0.759
  markazga eng yaqin (o'quvdagi shablonlar): 2-blokda 1.000 va 1.000

HALOL CHEKLOV (4-misol, B ustun):
  markazlar faqat 'a' shablonlaridan, test boshqa shablonlarda:
  'yuz' 0.018, 'kun' 0.667 - YANGI kontekst turiga umumlashmadi
  kichik korpusda "ma'no" = ko'rilgan kontekstlar to'plami
  real BERT milliardlab so'zda o'rganadi - kontekstlar xilma-xil

ANIZOTROPIYA:
  kontekstli vektorlar odatda bir-biriga o'xshash (kosinus 0.76-0.93)
  farq kichik, lekin izchil; o'rtachani ayirish ko'pincha yordam beradi

FINE-TUNING (Pretraining va fine-tuning darsida batafsil):
  jumla klassifikatsiyasi: [CLS] vektori -> Linear -> sinf
  token teglash (NER):     har token vektori -> Linear -> teg
  savol-javob:             har token uchun "javob boshi/oxiri" ehtimoli
  juftlik (NLI):           [CLS] A [SEP] B [SEP] -> [CLS] -> sinf

FEATURE EXTRACTION:
  BERT ni muzlatib, vektorlarni oddiy klassifikatorga berish

GENERATSIYAGA YARAMAYDI:
  MLM chapdan o'ngga yozishni o'rgatmaydi - buning uchun GPT (keyingi dars)

AVLODLAR (qisqa):
  RoBERTa  - NSP siz, dinamik niqob, ko'p ma'lumot
  ALBERT   - parametrlarni ulashish, SOP
  DistilBERT - distillyatsiya bilan kichraytirilgan
  ELECTRA  - "almashtirilgan tokenni aniqlash": har token signal beradi
  ko'p tilli (mBERT, XLM-R) - o'zbek tili ham bor

import torch
import torch.nn.functional as F

PAD, CLS, SEP, MASK = 0, 1, 2, 3

# MLM maskalash: 15%, keyin 80/10/10
oddiy = X > MASK                                          # maxsus tokenlar emas
tanlandi = (torch.rand(X.shape, generator=g) < 0.15) & oddiy
u = torch.rand(X.shape, generator=g)
kirish = X.clone()
kirish[tanlandi & (u < 0.8)] = MASK
tasodifiy = tanlandi & (u >= 0.8) & (u < 0.9)
kirish[tasodifiy] = torch.randint(MASK + 1, V, X.shape, generator=g)[tasodifiy]
nishon = torch.where(tanlandi, X, torch.full_like(X, -100))

# loss faqat tanlanganlarda
loss = F.cross_entropy(model(kirish).reshape(-1, V), nishon.reshape(-1),
                       ignore_index=-100)

# kirish: token + pozitsiya + segment
h = emb(x) + poz(torch.arange(T)) + seg(s)                 # s: 0 - A, 1 - B
ruxsat = (x != PAD)[:, None, None, :]                      # sdpa: True - ruxsat
o = F.scaled_dot_product_attention(q, k, v, attn_mask=ruxsat)

# NSP (ixtiyoriy, tarixiy)
nsp_logit = nsp_bosh(h[:, 0])                              # [CLS] vektori

# baholash: faqat [MASK] qo'yilgan joylar
aniq = (model(kirish).argmax(-1) == X)[kirish == MASK].float().mean()

encoder-only, ikki tomonlama; kauzal niqob yo'q
MLM: 15% tanlash, 80% [MASK] / 10% tasodifiy / 10% o'zgarmagan
loss faqat tanlanganlarda (-100); maxsus tokenlar maskalanmaydi
kirish: token + pozitsiya + segment; [CLS] ... [SEP] ... [SEP]
NSP - oson "yorliqli" vazifa, keyingi modellarda olib tashlangan
baholash: [MASK] joylarida, bazaviy va token turlari bilan
kontekstli vektor: bir so'z - turli kontekstda turli vektor

"""MLM maskalash noldan: 15% tanlash, 80/10/10 qoidasi, maxsus tokenlar va dinamik niqob."""

import numpy as np
import torch

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab bog' shahar qishloq kutubxona do'kon universitet "
          "choyxona dala bekat kasalxona stadion muzey teatr vokzal "
          "ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch go'sht shakar asal pishloq ko'ylak "
            "do'ppi soat telefon").split()
SIFATLAR = ("katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat "
            "yashil sariq issiq sovuq mazali").split()
QOSHIMCHALAR = ["##ga", "##dan", "##da"]
FELLAR = "bordi qaytdi sotib oldi bor berdi olib keldi".split()
MAXSUS = ["[PAD]", "[CLS]", "[SEP]", "[MASK]"]
PAD, CLS, SEP, MASK = 0, 1, 2, 3
LUGAT = MAXSUS + sorted(set(ISMLAR + VAQTLAR + JOYLAR + NARSALAR + SIFATLAR
                            + QOSHIMCHALAR + FELLAR + ["."]))
s2i = {s: i for i, s in enumerate(LUGAT)}
V = len(LUGAT)
L = 12


def jumla(rng):
    t = rng.integers(0, 6)
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = str(rng.choice(VAQTLAR)), str(rng.choice(JOYLAR))
    narsa, sifat = str(rng.choice(NARSALAR)), str(rng.choice(SIFATLAR))
    if t == 0:
        return [ism, joy, "##ga", vaqt, "bordi", "."]
    if t == 1:
        return [ism, joy, "##dan", vaqt, "qaytdi", "."]
    if t == 2:
        return [ism, sifat, narsa, "sotib", "oldi", "."]
    if t == 3:
        return [joy, "##da", sifat, narsa, "bor", "."]
    if t == 4:
        return [ism, ism2, "##ga", sifat, narsa, "berdi", "."]
    return [ism, joy, "##dan", narsa, "olib", "keldi", "."]


def tensorla(jumlalar):
    X = torch.full((len(jumlalar), L), PAD)
    for i, j in enumerate(jumlalar):
        ids = [CLS] + [s2i[str(w)] for w in j] + [SEP]
        X[i, :len(ids)] = torch.tensor(ids)
    return X


def maskala(X, g, p=0.15):
    """Qaytaradi: model kirishi, nishon (-100 - loss yo'q) va tanlov turi.

    tur: 0 - tanlanmagan, 1 - [MASK], 2 - tasodifiy token, 3 - o'zgarmagan
    """
    oddiy = X > MASK                                   # maxsus tokenlar tanlanmaydi
    tanlandi = (torch.rand(X.shape, generator=g) < p) & oddiy
    u = torch.rand(X.shape, generator=g)
    tur = torch.zeros_like(X)
    tur[tanlandi & (u < 0.8)] = 1
    tur[tanlandi & (u >= 0.8) & (u < 0.9)] = 2
    tur[tanlandi & (u >= 0.9)] = 3
    kirish = X.clone()
    kirish[tur == 1] = MASK
    tasodifiy = torch.randint(MASK + 1, V, X.shape, generator=g)
    kirish[tur == 2] = tasodifiy[tur == 2]
    nishon = torch.where(tanlandi, X, torch.full_like(X, -100))
    return kirish, nishon, tur


def korsat(qator):
    return " ".join(LUGAT[i] for i in qator if i != PAD)


def main() -> None:
    rng = np.random.default_rng(0)
    X = tensorla([jumla(rng) for _ in range(20000)])
    g = torch.Generator().manual_seed(0)

    print("=== 1. Maxsus tokenlar va kirish ===")
    print(f"  lug'at: {V} token, shundan maxsus: {MAXSUS}")
    print(f"  jumla:  {korsat(X[0].tolist())}")
    print(f"  id lar: {X[0].tolist()}")
    print("  [CLS] - boshida (butun ketma-ketlik vektori), [SEP] - jumla oxiri,"
          " [PAD] - to'ldirish")

    print("\n=== 2. Bitta jumlani maskalash (bir necha urinish) ===")
    kichik = X[:1].repeat(4, 1)
    kir, nish, tur = maskala(kichik, torch.Generator().manual_seed(3), p=0.3)
    belgi = {0: "", 1: "  <- [MASK]", 2: "  <- tasodifiy", 3: "  <- o'zgarmagan"}
    for r in range(4):
        print(f"  {r + 1}) {korsat(kir[r].tolist())}")
        for j in torch.where(tur[r] > 0)[0].tolist():
            print(f"       poz {j}: nishon {LUGAT[nish[r, j]]!r}{belgi[int(tur[r, j])]}")
    print("  (bu yerda ko'rinishi uchun p = 0.3; BERT da p = 0.15)")

    print("\n=== 3. Statistika: 20000 jumla, p = 0.15 ===")
    kir, nish, tur = maskala(X, g)
    oddiy = X > MASK
    tanlandi = tur > 0
    print(f"  oddiy tokenlar: {int(oddiy.sum())}, tanlangan: {int(tanlandi.sum())} "
          f"({tanlandi.sum().item() / oddiy.sum().item():.4f})")
    for k, nom in [(1, "[MASK]"), (2, "tasodifiy"), (3, "o'zgarmagan")]:
        ulush = (tur == k).sum().item() / tanlandi.sum().item()
        print(f"    {nom:<12} {ulush:.4f}")
    print(f"  maxsus tokenlardan tanlangani: {int((tanlandi & ~oddiy).sum())}")
    teng = ((tur == 2) & (kir == X)).sum().item() / (tur == 2).sum().item()
    print(f"  tasodifiy token asl token bilan tasodifan teng: {teng:.4f} "
          f"(kutilgan 1/{V - 4} = {1 / (V - 4):.4f})")
    print(f"  loss hisoblanadigan pozitsiyalar (nishon != -100): "
          f"{(nish != -100).float().mean().item():.4f} barcha {L} pozitsiyadan (PAD bilan)")
    print(f"  kirishda asl tokeni ko'rinmaydigan (o'zgartirilgan) pozitsiyalar: "
          f"{((tur == 1) | ((tur == 2) & (kir != X))).sum().item() / oddiy.sum().item():.4f}")

    print("\n=== 4. Dinamik maskalash: har epoxada yangi niqob ===")
    _, n1, _ = maskala(X, torch.Generator().manual_seed(10))
    _, n2, _ = maskala(X, torch.Generator().manual_seed(11))
    a, b = n1 != -100, n2 != -100
    print(f"  ikki epoxada ikkalasida ham tanlangan: {(a & b).sum().item() / a.sum().item():.4f}"
          f" (kutilgan 0.15)")
    epoxa = 10
    korilgan = torch.zeros_like(a)
    for e in range(epoxa):
        _, n, _ = maskala(X, torch.Generator().manual_seed(100 + e))
        korilgan |= n != -100
    print(f"  {epoxa} epoxada kamida bir marta nishon bo'lgan tokenlar: "
          f"{korilgan.sum().item() / oddiy.sum().item():.4f} "
          f"(1 - 0.85^{epoxa} = {1 - 0.85 ** epoxa:.4f})")
    print("  statik niqob: har epoxada AYNAN 15% - qolgan 85% hech qachon nishon emas")

    print("\n=== 5. Signal zichligi: MLM va kauzal til modeli ===")
    uz = oddiy.sum(1).float().mean().item()
    print(f"  o'rtacha jumla: {uz:.2f} ta oddiy token")
    print(f"  kauzal LM: {uz:.2f} ta bashorat (har token keyingisini)")
    print(f"  MLM (15%): o'rtacha {0.15 * uz:.2f} ta bashorat, lekin har biri IKKI "
          f"tomonlama kontekst bilan")
    print("  ⭐ MLM qoidasi: 15% tanlash, 80% [MASK], 10% tasodifiy, 10% o'zgarmagan")


if __name__ == "__main__":
    main()

=== 1. Maxsus tokenlar va kirish ===
  lug'at: 109 token, shundan maxsus: ['[PAD]', '[CLS]', '[SEP]', '[MASK]']
  jumla:  [CLS] rustam bozor ##dan olma olib keldi . [SEP]
  id lar: [1, 79, 20, 5, 68, 67, 50, 7, 2, 0, 0, 0]
  [CLS] - boshida (butun ketma-ketlik vektori), [SEP] - jumla oxiri, [PAD] - to'ldirish

=== 2. Bitta jumlani maskalash (bir necha urinish) ===
  1) [CLS] rustam [MASK] [MASK] olma olib keldi . [SEP]
       poz 1: nishon 'rustam'  <- o'zgarmagan
       poz 2: nishon 'bozor'  <- [MASK]
       poz 3: nishon '##dan'  <- [MASK]
       poz 5: nishon 'olib'  <- o'zgarmagan
  2) [CLS] rustam [MASK] ##dan [MASK] olib keldi . [SEP]
       poz 2: nishon 'bozor'  <- [MASK]
       poz 4: nishon 'olma'  <- [MASK]
  3) [CLS] [MASK] ##ga ##dan [MASK] olib [MASK] . [SEP]
       poz 1: nishon 'rustam'  <- [MASK]
       poz 2: nishon 'bozor'  <- tasodifiy
       poz 4: nishon 'olma'  <- [MASK]
       poz 5: nishon 'olib'  <- o'zgarmagan
       poz 6: nishon 'keldi'  <- [MASK]
  4) [CLS] [MASK] [MASK] ##dan olma olib keldi [MASK] [SEP]
       poz 1: nishon 'rustam'  <- [MASK]
       poz 2: nishon 'bozor'  <- [MASK]
       poz 7: nishon '.'  <- [MASK]
  (bu yerda ko'rinishi uchun p = 0.3; BERT da p = 0.15)

=== 3. Statistika: 20000 jumla, p = 0.15 ===
  oddiy tokenlar: 126702, tanlangan: 18893 0.1491-bob
    [MASK]       0.8009
    tasodifiy    0.1012
    o'zgarmagan  0.0979
  maxsus tokenlardan tanlangani: 0
  tasodifiy token asl token bilan tasodifan teng: 0.0084 (kutilgan 1/105 = 0.0095)
  loss hisoblanadigan pozitsiyalar (nishon != -100): 0.0787 barcha 12 pozitsiyadan (PAD bilan)
  kirishda asl tokeni ko'rinmaydigan (o'zgartirilgan) pozitsiyalar: 0.1344

=== 4. Dinamik maskalash: har epoxada yangi niqob ===
  ikki epoxada ikkalasida ham tanlangan: 0.1494 (kutilgan 0.15)
  10 epoxada kamida bir marta nishon bo'lgan tokenlar: 0.8029 (1 - 0.85^10 = 0.8031)
  statik niqob: har epoxada AYNAN 15% - qolgan 85% hech qachon nishon emas

=== 5. Signal zichligi: MLM va kauzal til modeli ===
  o'rtacha jumla: 6.34 ta oddiy token
  kauzal LM: 6.34 ta bashorat (har token keyingisini)
  MLM (15%): o'rtacha 0.95 ta bashorat, lekin har biri IKKI tomonlama kontekst bilan
  ⭐ MLM qoidasi: 15% tanlash, 80% [MASK], 10% tasodifiy, 10% o'zgarmagan

"""Kichik BERT: MLM pretraining, bazaviy modellar va faqat chap kontekstli variant."""

import math
from collections import Counter

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab bog' shahar qishloq kutubxona do'kon universitet "
          "choyxona dala bekat kasalxona stadion muzey teatr vokzal "
          "ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch go'sht shakar asal pishloq ko'ylak "
            "do'ppi soat telefon").split()
SIFATLAR = ("katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat "
            "yashil sariq issiq sovuq mazali").split()
QOSHIMCHALAR = ["##ga", "##dan", "##da"]
FELLAR = "bordi qaytdi sotib oldi bor berdi olib keldi".split()
MAXSUS = ["[PAD]", "[CLS]", "[SEP]", "[MASK]"]
PAD, CLS, SEP, MASK = 0, 1, 2, 3
LUGAT = MAXSUS + sorted(set(ISMLAR + VAQTLAR + JOYLAR + NARSALAR + SIFATLAR
                            + QOSHIMCHALAR + FELLAR + ["."]))
s2i = {s: i for i, s in enumerate(LUGAT)}
V = len(LUGAT)
L = 12


def jumla(rng):
    """So'z + WordPiece uslubidagi qo'shimcha (##ga, ##dan, ##da) tokenlari."""
    t = rng.integers(0, 6)
    ism, ism2 = rng.choice(ISMLAR, 2, replace=False)
    vaqt, joy = str(rng.choice(VAQTLAR)), str(rng.choice(JOYLAR))
    narsa, sifat = str(rng.choice(NARSALAR)), str(rng.choice(SIFATLAR))
    if t == 0:
        return [ism, joy, "##ga", vaqt, "bordi", "."]
    if t == 1:
        return [ism, joy, "##dan", vaqt, "qaytdi", "."]
    if t == 2:
        return [ism, sifat, narsa, "sotib", "oldi", "."]
    if t == 3:
        return [joy, "##da", sifat, narsa, "bor", "."]
    if t == 4:
        return [ism, ism2, "##ga", sifat, narsa, "berdi", "."]
    return [ism, joy, "##dan", narsa, "olib", "keldi", "."]


def tensorla(jumlalar):
    X = torch.full((len(jumlalar), L), PAD)
    for i, j in enumerate(jumlalar):
        ids = [CLS] + [s2i[str(w)] for w in j] + [SEP]
        X[i, :len(ids)] = torch.tensor(ids)
    return X


def maskala(X, g, p=0.15):
    """BERT qoidasi: 15% tanlanadi; ulardan 80% [MASK], 10% tasodifiy, 10% o'zgarmaydi."""
    oddiy = X > MASK
    tanlandi = (torch.rand(X.shape, generator=g) < p) & oddiy
    u = torch.rand(X.shape, generator=g)
    kirish = X.clone()
    kirish[tanlandi & (u < 0.8)] = MASK
    tasodifiy = tanlandi & (u >= 0.8) & (u < 0.9)
    kirish[tasodifiy] = torch.randint(MASK + 1, V, X.shape, generator=g)[tasodifiy]
    nishon = torch.where(tanlandi, X, torch.full_like(X, -100))
    return kirish, nishon, tanlandi & (u < 0.8)


class Blok(nn.Module):
    def __init__(self, d, boshlar):
        super().__init__()
        self.boshlar = boshlar
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))

    def forward(self, x, ruxsat):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=2)
        q, k, v = [t.view(B, T, self.boshlar, -1).transpose(1, 2) for t in (q, k, v)]
        o = F.scaled_dot_product_attention(q, k, v, attn_mask=ruxsat)   # True - ruxsat
        x = x + self.proj(o.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class KichikBERT(nn.Module):
    """faqat_chap=True - xuddi shu model, lekin kauzal niqob bilan (faqat chap kontekst)."""

    def __init__(self, faqat_chap=False, d=64, boshlar=4, qatlamlar=2):
        super().__init__()
        self.faqat_chap = faqat_chap
        self.emb = nn.Embedding(V, d)
        self.poz = nn.Embedding(L, d)
        self.bloklar = nn.ModuleList(Blok(d, boshlar) for _ in range(qatlamlar))
        self.ln = nn.LayerNorm(d)
        self.mlm = nn.Linear(d, V)

    def forward(self, x):
        ruxsat = (x != PAD)[:, None, None, :]                  # pad kalitlarga qaralmaydi
        if self.faqat_chap:
            ruxsat = ruxsat & torch.ones(L, L, dtype=torch.bool).tril()
        h = self.emb(x) + self.poz(torch.arange(x.shape[1]))
        for b in self.bloklar:
            h = b(h, ruxsat)
        return self.mlm(self.ln(h))


def orgat(X, faqat_chap, seed, qadamlar=300, B=64):
    torch.manual_seed(seed)
    model = KichikBERT(faqat_chap)
    opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        idx = torch.randint(0, len(X), (B,), generator=g)
        kirish, nishon, _ = maskala(X[idx], g)
        loss = F.cross_entropy(model(kirish).reshape(-1, V), nishon.reshape(-1),
                               ignore_index=-100)
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    model.eval()
    return model


def turi(i):
    s = LUGAT[i]
    if s in QOSHIMCHALAR:
        return "qo'shimcha"
    if s in ISMLAR:
        return "ism"
    if s in FELLAR:
        return "fe'l"
    return "boshqa"


def main() -> None:
    torch.set_num_threads(1)
    rng = np.random.default_rng(0)
    X = tensorla([jumla(rng) for _ in range(6000)])
    Xv = tensorla([jumla(rng) for _ in range(1500)])
    kirish_v, nishon_v, mask_v = maskala(Xv, torch.Generator().manual_seed(123))
    tanlangan = nishon_v != -100
    y = Xv[mask_v]                           # baholash: faqat [MASK] qo'yilgan joylar
    print("=== 1. Ma'lumot ===")
    print(f"  lug'at V = {V} (4 maxsus token), o'quv 6000, val 1500 jumla")
    print(f"  misol: {' '.join(LUGAT[i] for i in X[0].tolist() if i != PAD)}")
    print(f"  val: tanlangan {int(tanlangan.sum())} pozitsiya, ulardan [MASK] - "
          f"{int(mask_v.sum())} (baholash shularda)")

    print("\n=== 2. Bazaviy modellar (val MLM aniqligi) ===")
    sanoq = Counter(X[X > MASK].tolist())
    eng_kop = max(sanoq, key=sanoq.get)
    jami = sum(sanoq.values())
    unigram = sum(sanoq[t] / jami for t in y.tolist()) / len(y)
    poz = {}
    for p in range(L):
        c = Counter(X[:, p][X[:, p] > MASK].tolist())
        poz[p] = max(c, key=c.get) if c else PAD
    poz_bash = torch.tensor([poz[p] for p in torch.where(mask_v)[1].tolist()])
    bazaviy = {
        f"eng ko'p token ({LUGAT[eng_kop]!r})": (y == eng_kop).float().mean().item(),
        "unigram (namuna olish)": unigram,
        "pozitsiya bo'yicha eng ko'p": (poz_bash == y).float().mean().item(),
    }
    for nom, a in bazaviy.items():
        print(f"  {nom:<30} {a:.4f}")

    print("\n=== 3. BERT (ikki tomonlama) va faqat chap kontekst, 3 seed ===")
    natija = {False: [], True: []}
    tur = np.array([turi(i) for i in y.tolist()])
    for s in range(3):
        for faqat_chap in (False, True):
            m = orgat(X, faqat_chap, s)
            with torch.no_grad():
                bash = m(kirish_v)[mask_v].argmax(1)
            togri = (bash == y).numpy()
            natija[faqat_chap].append({t: togri[tur == t].mean() for t in
                                       ["qo'shimcha", "ism", "fe'l", "boshqa"]}
                                      | {"hammasi": togri.mean()})
            if s == 0 and not faqat_chap:
                bert0 = m
    n_par = sum(p.numel() for p in bert0.parameters())
    print(f"  parametrlar: {n_par}, 300 qadam, batch 64")
    print(f"  {'token turi':<11} {'ulush':>6} {'BERT':>7} {'chap':>7} {'farq':>8} "
          f"{'SE':>7} {'sezilarli':>10}")
    for t in ["hammasi", "qo'shimcha", "fe'l", "ism", "boshqa"]:
        a = np.array([r[t] for r in natija[False]])
        b = np.array([r[t] for r in natija[True]])
        farq = a - b
        se = farq.std(ddof=1) / math.sqrt(len(farq))
        ulush = 1.0 if t == "hammasi" else (tur == t).mean()
        print(f"  {t:<11} {ulush:>6.1%} {a.mean():>7.4f} {b.mean():>7.4f} "
              f"{farq.mean():>+8.4f} {se:>7.4f} {str(abs(farq.mean()) > 2 * se):>10}")
    print(f"  ism uchun tasodif darajasi: 1/{len(ISMLAR)} = {1 / len(ISMLAR):.4f}")

    print("\n=== 4. 80/10/10: tanlangan pozitsiya turi bo'yicha (BERT, seed 0) ===")
    with torch.no_grad():
        bash_hammasi = bert0(kirish_v).argmax(2)
    for nom, joy in [("[MASK] qo'yilgan", mask_v),
                     ("tasodifiy token", tanlangan & ~mask_v & (kirish_v != Xv)),
                     ("o'zgarmagan", tanlangan & ~mask_v & (kirish_v == Xv))]:
        a = (bash_hammasi[joy] == Xv[joy]).float().mean().item()
        print(f"  {nom:<17} {int(joy.sum()):>5} ta   aniqlik {a:.4f}")
    oddiy = (Xv > MASK) & ~tanlangan
    a = (bash_hammasi[oddiy] == Xv[oddiy]).float().mean().item()
    print(f"  {'tanlanmagan':<17} {int(oddiy.sum()):>5} ta   aniqlik {a:.4f} (loss yo'q)")

    print("\n=== 5. [MASK] ni to'ldirish (BERT, seed 0) ===")
    for gap in [["anvar", "bozor", "[MASK]", "ertalab", "bordi", "."],
                ["anvar", "bozor", "[MASK]", "ertalab", "qaytdi", "."],
                ["bog'", "[MASK]", "katta", "olma", "bor", "."]]:
        x = torch.full((1, L), PAD)
        ids = [CLS] + [s2i[w] for w in gap] + [SEP]
        x[0, :len(ids)] = torch.tensor(ids)
        j = ids.index(MASK)
        with torch.no_grad():
            p = torch.softmax(bert0(x)[0, j], 0)
        top = p.topk(3)
        print(f"  {' '.join(gap):<36} -> " + ", ".join(
            f"{LUGAT[i]} {v:.3f}" for v, i in zip(top.values.tolist(), top.indices.tolist())))
    print("  ⭐ MLM: ikki tomonlama kontekst o'ngdagi fe'lga qarab qo'shimchani tanlaydi")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  lug'at V = 109 (4 maxsus token), o'quv 6000, val 1500 jumla
  misol: [CLS] rustam bozor ##dan olma olib keldi . [SEP]
  val: tanlangan 1444 pozitsiya, ulardan [MASK] - 1162 (baholash shularda)

=== 2. Bazaviy modellar (val MLM aniqligi) ===
  eng ko'p token ('.')           0.1713
  unigram (namuna olish)         0.0412
  pozitsiya bo'yicha eng ko'p    0.2823

=== 3. BERT (ikki tomonlama) va faqat chap kontekst, 3 seed ===
  parametrlar: 81901, 300 qadam, batch 64
  token turi   ulush    BERT    chap     farq      SE  sezilarli
  hammasi     100.0%  0.5261  0.4859  +0.0402  0.0032       True
  qo'shimcha   12.2%  0.9836  0.7676  +0.2160  0.0300       True
  fe'l         20.9%  0.9698  0.9561  +0.0137  0.0036       True
  ism          14.8%  0.0291  0.0213  +0.0078  0.0084      False
  boshqa       52.1%  0.3818  0.3631  +0.0187  0.0024       True
  ism uchun tasodif darajasi: 1/28 = 0.0357

=== 4. 80/10/10: tanlangan pozitsiya turi bo'yicha (BERT, seed 0) ===
  [MASK] qo'yilgan   1162 ta   aniqlik 0.5224
  tasodifiy token     141 ta   aniqlik 0.5461
  o'zgarmagan         141 ta   aniqlik 0.7021
  tanlanmagan        8095 ta   aniqlik 0.6566 (loss yo'q)

=== 5. [MASK] ni to'ldirish (BERT, seed 0) ===
  anvar bozor [MASK] ertalab bordi .   -> ##ga 0.999, ##dan 0.001, . 0.000
  anvar bozor [MASK] ertalab qaytdi .  -> ##dan 0.996, ##ga 0.001, sotib 0.000
  bog' [MASK] katta olma bor .         -> ##da 0.992, . 0.002, mazali 0.001
  ⭐ MLM: ikki tomonlama kontekst o'ngdagi fe'lga qarab qo'shimchani tanlaydi

"""Jumla juftliklari: [CLS] A [SEP] B [SEP], segment embedding va NSP."""

import math

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab bog' shahar qishloq kutubxona do'kon universitet "
          "choyxona dala bekat kasalxona stadion muzey teatr vokzal "
          "ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch go'sht shakar asal pishloq ko'ylak "
            "do'ppi soat telefon").split()
SIFATLAR = ("katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat "
            "yashil sariq issiq sovuq mazali").split()
BOSHQA = "##ga ##dan ##da ##ni u bordi qaytdi sotib oldi berdi olib keldi yedi .".split()
MAXSUS = ["[PAD]", "[CLS]", "[SEP]", "[MASK]"]
PAD, CLS, SEP, MASK = 0, 1, 2, 3
LUGAT = MAXSUS + sorted(set(ISMLAR + VAQTLAR + JOYLAR + NARSALAR + SIFATLAR + BOSHQA))
s2i = {s: i for i, s in enumerate(LUGAT)}
V = len(LUGAT)
L = 20


def hikoya(rng):
    """Ikki bog'liq jumla: B jumla A dagi joy yoki narsani takrorlaydi."""
    ism, ism2 = (str(s) for s in rng.choice(ISMLAR, 2, replace=False))
    vaqt, joy = str(rng.choice(VAQTLAR)), str(rng.choice(JOYLAR))
    narsa, sifat = str(rng.choice(NARSALAR)), str(rng.choice(SIFATLAR))
    t = rng.integers(0, 3)
    if t == 0:
        return ([ism, joy, "##ga", vaqt, "bordi", "."],
                ["u", joy, "##da", sifat, narsa, "sotib", "oldi", "."])
    if t == 1:
        return ([ism, joy, "##dan", narsa, "olib", "keldi", "."],
                ["u", narsa, "##ni", ism2, "##ga", "berdi", "."])
    return ([ism, sifat, narsa, "sotib", "oldi", "."],
            ["u", vaqt, narsa, "##ni", "yedi", "."])


def juftliklar(n, rng):
    """50% - haqiqiy davom (IsNext=1), 50% - boshqa hikoyadan tasodifiy B (0)."""
    hik = [hikoya(rng) for _ in range(n)]
    A, B, y = [], [], []
    for i, (a, b) in enumerate(hik):
        A.append(a)
        if rng.random() < 0.5:
            B.append(b)
            y.append(1)
        else:
            j = int(rng.integers(0, n - 1))
            j += j >= i                                  # o'zidan boshqa hikoya
            B.append(hik[j][1])
            y.append(0)
    return A, B, torch.tensor(y)


def tensorla(A, B):
    X = torch.full((len(A), L), PAD)
    S = torch.zeros((len(A), L), dtype=torch.long)
    for i, (a, b) in enumerate(zip(A, B)):
        ids = [CLS] + [s2i[w] for w in a] + [SEP] + [s2i[w] for w in b] + [SEP]
        X[i, :len(ids)] = torch.tensor(ids)
        S[i, len(a) + 2:len(ids)] = 1                 # B jumla va uning [SEP] i
    return X, S


def maskala(X, g, p=0.15):
    oddiy = X > MASK
    tanlandi = (torch.rand(X.shape, generator=g) < p) & oddiy
    u = torch.rand(X.shape, generator=g)
    kirish = X.clone()
    kirish[tanlandi & (u < 0.8)] = MASK
    tasodifiy = tanlandi & (u >= 0.8) & (u < 0.9)
    kirish[tasodifiy] = torch.randint(MASK + 1, V, X.shape, generator=g)[tasodifiy]
    return kirish, torch.where(tanlandi, X, torch.full_like(X, -100)), tanlandi & (u < 0.8)


class Blok(nn.Module):
    def __init__(self, d, boshlar):
        super().__init__()
        self.boshlar = boshlar
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))

    def forward(self, x, ruxsat):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=2)
        q, k, v = [t.view(B, T, self.boshlar, -1).transpose(1, 2) for t in (q, k, v)]
        o = F.scaled_dot_product_attention(q, k, v, attn_mask=ruxsat)
        x = x + self.proj(o.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class BERT(nn.Module):
    def __init__(self, d=64, boshlar=4, qatlamlar=2):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.poz = nn.Embedding(L, d)
        self.seg = nn.Embedding(2, d)                     # A = 0, B = 1
        self.bloklar = nn.ModuleList(Blok(d, boshlar) for _ in range(qatlamlar))
        self.ln = nn.LayerNorm(d)
        self.mlm = nn.Linear(d, V)
        self.nsp = nn.Linear(d, 2)                        # [CLS] vektoridan

    def forward(self, x, s):
        ruxsat = (x != PAD)[:, None, None, :]
        h = self.emb(x) + self.poz(torch.arange(x.shape[1])) + self.seg(s)
        for b in self.bloklar:
            h = b(h, ruxsat)
        h = self.ln(h)
        return self.mlm(h), self.nsp(h[:, 0])


def orgat(X, S, Y, nsp_bilan, seed, qadamlar=200, B=64):
    torch.manual_seed(seed)
    model = BERT()
    opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        idx = torch.randint(0, len(X), (B,), generator=g)
        kirish, nishon, _ = maskala(X[idx], g)
        mlm, nsp = model(kirish, S[idx])
        loss = F.cross_entropy(mlm.reshape(-1, V), nishon.reshape(-1), ignore_index=-100)
        if nsp_bilan:
            loss = loss + F.cross_entropy(nsp, Y[idx])
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    model.eval()
    return model


def main() -> None:
    torch.set_num_threads(1)
    rng = np.random.default_rng(0)
    A, B, Y = juftliklar(6000, rng)
    Av, Bv, Yv = juftliklar(1500, rng)
    X, S = tensorla(A, B)
    Xv, Sv = tensorla(Av, Bv)

    print("=== 1. Kirish formati: tokenlar, segmentlar, pozitsiyalar ===")
    ids = [i for i in X[0].tolist() if i != PAD]
    print("  " + " ".join(f"{LUGAT[i]:>7}" for i in ids[:10]))
    print("  " + " ".join(f"{s:>7}" for s in S[0, :10].tolist()) + "   <- segment")
    print("  " + " ".join(f"{p:>7}" for p in range(10)) + "   <- pozitsiya")
    print(f"  IsNext = {int(Y[0])};  o'quvda IsNext ulushi {Y.float().mean().item():.3f}")
    print(f"  embedding = token + pozitsiya + segment;  V = {V}, L = {L}")

    print("\n=== 2. NSP uchun 'yorliq' qoida: A va B da umumiy so'z bormi ===")
    xizmat = {"u", ".", "##ga", "##dan", "##da", "##ni", "sotib", "oldi"}

    def umumiy(a, b):
        return int(len((set(a) - xizmat) & (set(b) - xizmat)) > 0)

    qoida = torch.tensor([umumiy(a, b) for a, b in zip(Av, Bv)])
    print(f"  qoida aniqligi (val): {(qoida == Yv).float().mean().item():.4f}")
    print(f"  tasodifiy B da umumiy so'z: {qoida[Yv == 0].float().mean().item():.4f}")

    print("\n=== 3. MLM + NSP va faqat MLM (200 qadam, 3 seed) ===")
    kirish_v, _, mask_v = maskala(Xv, torch.Generator().manual_seed(7))
    natija = {True: [], False: []}
    nsp_aniq = []
    for s in range(3):
        for nsp_bilan in (True, False):
            m = orgat(X, S, Y, nsp_bilan, s)
            with torch.no_grad():
                mlm, nsp = m(kirish_v, Sv)
            natija[nsp_bilan].append((mlm.argmax(2) == Xv)[mask_v].float().mean().item())
            if nsp_bilan:
                nsp_aniq.append((nsp.argmax(1) == Yv).float().mean().item())
                if s == 0:
                    model0 = m
    print("  NSP aniqligi ([CLS] boshi): " + ", ".join(f"{a:.4f}" for a in nsp_aniq))
    for nsp_bilan, nom in [(True, "MLM + NSP"), (False, "faqat MLM")]:
        print(f"  {nom:<10} MLM aniqligi: "
              + ", ".join(f"{a:.4f}" for a in natija[nsp_bilan])
              + f"  o'rtacha {np.mean(natija[nsp_bilan]):.4f}")
    farq = np.array(natija[True]) - np.array(natija[False])
    se = farq.std(ddof=1) / math.sqrt(len(farq))
    xulosa = ("NSP MLM ni sezilarli yaxshiladi" if farq.mean() > 2 * se else
              "NSP MLM ni sezilarli yomonlashtirdi" if farq.mean() < -2 * se else
              "MLM aniqligida sezilarli farq yo'q")
    print(f"  MLM farqi (NSP bilan - NSP siz): {farq.mean():+.4f}, SE {se:.4f} -> {xulosa}")
    if np.mean(nsp_aniq) < (qoida == Yv).float().mean().item():
        print("  NSP boshi oddiy 'umumiy so'z' qoidasidan ham past")

    print("\n=== 4. Segment embedding nima beradi (MLM + NSP, seed 0) ===")
    with torch.no_grad():
        mlm0, nsp0 = model0(kirish_v, Sv)
        mlm1, nsp1 = model0(kirish_v, torch.zeros_like(Sv))
    for nom, mlm, nsp in [("to'g'ri segmentlar", mlm0, nsp0),
                          ("hammasi segment 0", mlm1, nsp1)]:
        a = (mlm.argmax(2) == Xv)[mask_v]
        b_qism = a[(Sv == 1)[mask_v]].float().mean().item()
        print(f"  {nom:<19} MLM {a.float().mean().item():.4f} (B qismida {b_qism:.4f}),"
              f" NSP {(nsp.argmax(1) == Yv).float().mean().item():.4f}")
    print(f"  segment vektorlari orasidagi kosinus: "
          f"{F.cosine_similarity(model0.seg.weight[0], model0.seg.weight[1], 0).item():.3f}")
    print("  ⭐ [CLS] - butun juftlik vektori; segment - qaysi jumlaga tegishlilik belgisi")


if __name__ == "__main__":
    main()

=== 1. Kirish formati: tokenlar, segmentlar, pozitsiyalar ===
    [CLS] dilshod kutubxona    ##ga   bugun   bordi       .   [SEP]       u kutubxona
        0       0       0       0       0       0       0       0       1       1   <- segment
        0       1       2       3       4       5       6       7       8       9   <- pozitsiya
  IsNext = 1;  o'quvda IsNext ulushi 0.502
  embedding = token + pozitsiya + segment;  V = 111, L = 20

=== 2. NSP uchun 'yorliq' qoida: A va B da umumiy so'z bormi ===
  qoida aniqligi (val): 0.9647
  tasodifiy B da umumiy so'z: 0.0707

=== 3. MLM + NSP va faqat MLM (200 qadam, 3 seed) ===
  NSP aniqligi ([CLS] boshi): 0.8367, 0.8287, 0.8307
  MLM + NSP  MLM aniqligi: 0.6113, 0.6093, 0.6117  o'rtacha 0.6108
  faqat MLM  MLM aniqligi: 0.6034, 0.6089, 0.6101  o'rtacha 0.6075
  MLM farqi (NSP bilan - NSP siz): +0.0033, SE 0.0024 -> MLM aniqligida sezilarli farq yo'q
  NSP boshi oddiy 'umumiy so'z' qoidasidan ham past

=== 4. Segment embedding nima beradi (MLM + NSP, seed 0) ===
  to'g'ri segmentlar  MLM 0.6113 (B qismida 0.6679), NSP 0.8367
  hammasi segment 0   MLM 0.5718 (B qismida 0.6163), NSP 0.8353
  segment vektorlari orasidagi kosinus: 0.106
  ⭐ [CLS] - butun juftlik vektori; segment - qaysi jumlaga tegishlilik belgisi

"""Kontekstli vektorlar: bir xil so'z ('yuz', 'kun') turli ma'noda - BERT qatlamlarida."""

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

ISMLAR = ("anvar dilnoza bobur malika sardor nodira jasur zarina otabek madina "
          "sherzod gulnora farhod laylo rustam sevara aziz kamola bahrom "
          "shahlo temur nigora dilshod feruza jamshid mohira sanjar yulduz").split()
VAQTLAR = "ertalab kechqurun bugun kecha tushda yakshanba erta indinga".split()
JOYLAR = ("bozor maktab bog' shahar qishloq kutubxona do'kon universitet "
          "choyxona dala bekat kasalxona stadion muzey teatr vokzal "
          "ofis zavod").split()
NARSALAR = ("kitob olma non qalam daftar gul choy sut anor uzum ruchka gazeta "
            "qovun tarvuz sabzi guruch go'sht shakar asal pishloq ko'ylak "
            "do'ppi soat telefon").split()
SIFATLAR = ("katta kichik yangi eski chiroyli shirin qizil oq arzon qimmat "
            "yashil sariq issiq sovuq mazali").split()
BOSHQA = ("##ga ##dan ##da ##ni ##ini ##i bordi qaytdi sotib oldi bor berdi olib "
          "keldi . yuz ta so'm turadi qo'l yuvdi qizardi kun har bir chiqdi "
          "tongda nuri yoritdi osmonda porladi").split()
MAXSUS = ["[PAD]", "[CLS]", "[SEP]", "[MASK]"]
PAD, CLS, SEP, MASK = 0, 1, 2, 3
LUGAT = MAXSUS + sorted(set(ISMLAR + VAQTLAR + JOYLAR + NARSALAR + SIFATLAR + BOSHQA))
s2i = {s: i for i, s in enumerate(LUGAT)}
V = len(LUGAT)
L = 12


def jumla(rng):
    """(tokenlar, so'z, ma'no, shablon). Ikki ma'noli so'z doim 2-o'rinda turadi -
    pozitsiya ma'no haqida hech narsa aytmaydi, farqni faqat kontekst beradi."""
    ism, ism2 = (str(s) for s in rng.choice(ISMLAR, 2, replace=False))
    vaqt, joy = str(rng.choice(VAQTLAR)), str(rng.choice(JOYLAR))
    narsa, sifat = str(rng.choice(NARSALAR)), str(rng.choice(SIFATLAR))
    t = rng.integers(0, 14)
    if t == 0:
        return [ism, "yuz", "ta", narsa, "sotib", "oldi", "."], "yuz", 0, "a"
    if t == 1:
        return [narsa, "yuz", "so'm", "turadi", "."], "yuz", 0, "b"
    if t == 2:
        return [ism, "yuz", "qo'l", "##ini", "yuvdi", "."], "yuz", 1, "a"
    if t == 3:
        return [ism, "yuz", "##i", "qizardi", "."], "yuz", 1, "b"
    if t == 4:
        return ["har", "kun", ism, joy, "##ga", "bordi", "."], "kun", 0, "a"
    if t == 5:
        return ["bir", "kun", ism, joy, "##dan", "qaytdi", "."], "kun", 0, "b"
    if t == 6:
        return [vaqt, "kun", "chiqdi", "."], "kun", 1, "a"
    if t == 7:
        return ["tongda", "kun", "nuri", joy, "##ni", "yoritdi", "."], "kun", 1, "b"
    if t == 8:
        return [ism, joy, "##ga", vaqt, "bordi", "."], None, -1, ""
    if t == 9:
        return [ism, joy, "##dan", vaqt, "qaytdi", "."], None, -1, ""
    if t == 10:
        return [ism, sifat, narsa, "sotib", "oldi", "."], None, -1, ""
    if t == 11:
        return [joy, "##da", sifat, narsa, "bor", "."], None, -1, ""
    if t == 12:
        return [ism, ism2, "##ga", sifat, narsa, "berdi", "."], None, -1, ""
    return ["osmonda", "kun", "porladi", "."], "kun", 1, "c"


def tensorla(jumlalar):
    X = torch.full((len(jumlalar), L), PAD)
    for i, (j, _, _, _) in enumerate(jumlalar):
        ids = [CLS] + [s2i[w] for w in j] + [SEP]
        X[i, :len(ids)] = torch.tensor(ids)
    return X


def maskala(X, g, p=0.15):
    oddiy = X > MASK
    tanlandi = (torch.rand(X.shape, generator=g) < p) & oddiy
    u = torch.rand(X.shape, generator=g)
    kirish = X.clone()
    kirish[tanlandi & (u < 0.8)] = MASK
    tasodifiy = tanlandi & (u >= 0.8) & (u < 0.9)
    kirish[tasodifiy] = torch.randint(MASK + 1, V, X.shape, generator=g)[tasodifiy]
    return kirish, torch.where(tanlandi, X, torch.full_like(X, -100))


class Blok(nn.Module):
    def __init__(self, d, boshlar):
        super().__init__()
        self.boshlar = boshlar
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.ffn = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))

    def forward(self, x, ruxsat):
        B, T, d = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=2)
        q, k, v = [t.view(B, T, self.boshlar, -1).transpose(1, 2) for t in (q, k, v)]
        o = F.scaled_dot_product_attention(q, k, v, attn_mask=ruxsat)
        x = x + self.proj(o.transpose(1, 2).reshape(B, T, d))
        return x + self.ffn(self.ln2(x))


class KichikBERT(nn.Module):
    def __init__(self, d=64, boshlar=4, qatlamlar=2):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.poz = nn.Embedding(L, d)
        self.bloklar = nn.ModuleList(Blok(d, boshlar) for _ in range(qatlamlar))
        self.ln = nn.LayerNorm(d)
        self.mlm = nn.Linear(d, V)

    def forward(self, x, holatlar=False):
        ruxsat = (x != PAD)[:, None, None, :]
        h = self.emb(x) + self.poz(torch.arange(x.shape[1]))
        qatlam = {"token embedding": self.emb(x), "kirish (token + poz)": h}
        for i, b in enumerate(self.bloklar):
            h = b(h, ruxsat)
            qatlam[f"{i + 1}-blok"] = h
        return qatlam if holatlar else self.mlm(self.ln(h))


def orgat(X, seed=0, qadamlar=400, B=64):
    torch.manual_seed(seed)
    model = KichikBERT()
    opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        idx = torch.randint(0, len(X), (B,), generator=g)
        kirish, nishon = maskala(X[idx], g)
        loss = F.cross_entropy(model(kirish).reshape(-1, V), nishon.reshape(-1),
                               ignore_index=-100)
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    model.eval()
    return model


@torch.no_grad()
def vektorlar(model, jumlalar, soz):
    """Har jumlada 'soz' pozitsiyasidagi vektor - har qatlam uchun."""
    tanlov = [j for j in jumlalar if j[1] == soz]
    X = tensorla(tanlov)
    joy = torch.tensor([j[0].index(soz) + 1 for j in tanlov])        # +1: [CLS]
    qatlam = model(X, holatlar=True)
    return ({k: v[torch.arange(len(X)), joy] for k, v in qatlam.items()},
            np.array([j[2] for j in tanlov]), np.array([j[3] for j in tanlov]))


def main() -> None:
    torch.set_num_threads(1)
    rng = np.random.default_rng(0)
    oquv = [jumla(rng) for _ in range(8000)]
    val = [jumla(rng) for _ in range(2000)]
    model = orgat(tensorla(oquv))

    print("=== 1. Ikki ma'noli so'zlar (val dan namunalar) ===")
    for soz in ("yuz", "kun"):
        for m in (0, 1):
            j = next(j for j in val if j[1] == soz and j[2] == m)
            print(f"  {soz!r} ma'no {m}: {' '.join(j[0])}")
    print("  token embedding (statik) ikkala ma'noda AYNAN bir xil vektor")

    print("\n=== 2. Kosinus o'xshashlik: bir ma'no ichida va ma'nolar orasida ===")
    natija = {}
    for soz in ("yuz", "kun"):
        v_o, m_o, sh_o = vektorlar(model, oquv, soz)
        v_v, m_v, sh_v = vektorlar(model, val, soz)
        natija[soz] = (v_o, m_o, sh_o, v_v, m_v, sh_v)
        print(f"  {soz!r} (val: {len(m_v)} ta)")
        for k, v in v_v.items():
            n = F.normalize(v, dim=1)
            s = (n @ n.T).numpy()
            bir = s[m_v[:, None] == m_v[None, :]].mean()
            boshqa = s[m_v[:, None] != m_v[None, :]].mean()
            print(f"    {k:<21} bir ma'no {bir:.3f}   boshqa ma'no {boshqa:.3f}   "
                  f"farq {bir - boshqa:.3f}")

    print("\n=== 3. Ma'noni aniqlash: markazga eng yaqin (val) ===")
    print("  A: markazlar o'quvdagi barcha shablonlardan")
    print("  B: markazlar faqat 'a' shablonlardan, test - boshqa shablonlarda")
    print(f"  {'qatlam':<21} {'yuz A':>7} {'yuz B':>7} {'kun A':>7} {'kun B':>7}")
    qatorlar = {}
    for soz in ("yuz", "kun"):
        v_o, m_o, sh_o, v_v, m_v, sh_v = natija[soz]
        for k in v_o:
            for rejim in ("A", "B"):
                tanla = np.ones(len(m_o), bool) if rejim == "A" else sh_o == "a"
                test = np.ones(len(m_v), bool) if rejim == "A" else sh_v != "a"
                markaz = torch.stack([v_o[k][torch.tensor(tanla & (m_o == c))].mean(0)
                                      for c in (0, 1)])
                o = F.cosine_similarity(v_v[k][torch.tensor(test)][:, None],
                                        markaz[None], dim=2)
                a = (o.argmax(1).numpy() == m_v[test]).mean()
                qatorlar.setdefault(k, []).append(a)
    for k, r in qatorlar.items():
        print(f"  {k:<21} " + " ".join(f"{a:>7.3f}" for a in r))
    print("  tasodif ~0.5; token embedding - bir xil vektor, argmax doim bir sinf")

    print("\n=== 4. Bitta so'z, to'rt kontekst (oxirgi blok, kosinus) ===")
    misollar = [["anvar", "yuz", "ta", "olma", "sotib", "oldi", "."],
                ["non", "yuz", "so'm", "turadi", "."],
                ["anvar", "yuz", "qo'l", "##ini", "yuvdi", "."],
                ["malika", "yuz", "##i", "qizardi", "."]]
    X = tensorla([(j, None, 0, "") for j in misollar])
    joy = torch.tensor([j.index("yuz") + 1 for j in misollar])
    with torch.no_grad():
        h = model(X, holatlar=True)["2-blok"][torch.arange(4), joy]
    n = F.normalize(h, dim=1)
    s = n @ n.T
    for i, j in enumerate(misollar):
        print(f"  {' '.join(j):<34} " + " ".join(f"{x:6.3f}" for x in s[i].tolist()))
    print("  1-2: 'yuz' = 100 (son), 3-4: 'yuz' = a'zo (bet)")
    print("  ⭐ BERT vektori - so'zning O'ZI emas, so'z + uning konteksti")


if __name__ == "__main__":
    main()

=== 1. Ikki ma'noli so'zlar (val dan namunalar) ===
  'yuz' ma'no 0: shahlo yuz ta ruchka sotib oldi .
  'yuz' ma'no 1: laylo yuz ##i qizardi .
  'kun' ma'no 0: har kun sardor muzey ##ga bordi .
  'kun' ma'no 1: osmonda kun porladi .
  token embedding (statik) ikkala ma'noda AYNAN bir xil vektor

=== 2. Kosinus o'xshashlik: bir ma'no ichida va ma'nolar orasida ===
  'yuz' (val: 549 ta)
    token embedding       bir ma'no 1.000   boshqa ma'no 1.000   farq 0.000
    kirish (token + poz)  bir ma'no 1.000   boshqa ma'no 1.000   farq 0.000
    1-blok                bir ma'no 0.904   boshqa ma'no 0.870   farq 0.034
    2-blok                bir ma'no 0.853   boshqa ma'no 0.759   farq 0.094
  'kun' (val: 704 ta)
    token embedding       bir ma'no 1.000   boshqa ma'no 1.000   farq -0.000
    kirish (token + poz)  bir ma'no 1.000   boshqa ma'no 1.000   farq -0.000
    1-blok                bir ma'no 0.964   boshqa ma'no 0.920   farq 0.044
    2-blok                bir ma'no 0.934   boshqa ma'no 0.886   farq 0.048

=== 3. Ma'noni aniqlash: markazga eng yaqin (val) ===
  A: markazlar o'quvdagi barcha shablonlardan
  B: markazlar faqat 'a' shablonlardan, test - boshqa shablonlarda
  qatlam                  yuz A   yuz B   kun A   kun B
  token embedding         0.554   0.536   0.385   0.328
  kirish (token + poz)    0.554   0.464   0.385   0.328
  1-blok                  0.767   0.536   1.000   0.667
  2-blok                  1.000   0.018   1.000   0.667
  tasodif ~0.5; token embedding - bir xil vektor, argmax doim bir sinf

=== 4. Bitta so'z, to'rt kontekst (oxirgi blok, kosinus) ===
  anvar yuz ta olma sotib oldi .      1.000  0.558  0.753  0.878
  non yuz so'm turadi .               0.558  1.000  0.605  0.717
  anvar yuz qo'l ##ini yuvdi .        0.753  0.605  1.000  0.881
  malika yuz ##i qizardi .            0.878  0.717  0.881  1.000
  1-2: 'yuz' = 100 (son), 3-4: 'yuz' = a'zo (bet)
  ⭐ BERT vektori - so'zning O'ZI emas, so'z + uning konteksti

tanlandi = torch.rand(X.shape) < 0.15                              # ⚠️
tanlandi = (torch.rand(X.shape, generator=g) < 0.15) & (X > MASK)  # ✅

loss = F.cross_entropy(logit.reshape(-1, V), X.reshape(-1))        # ⚠️
nishon = torch.where(tanlandi, X, torch.full_like(X, -100))
loss = F.cross_entropy(logit.reshape(-1, V), nishon.reshape(-1),
                       ignore_index=-100)                          # ✅

kirish[tanlandi] = MASK                                            # ⚠️
# 80% [MASK], 10% tasodifiy, 10% o'zgarmagan                       # ✅

torch.randint(0, V, X.shape)                  # [PAD], [CLS] ham    # ⚠️
torch.randint(MASK + 1, V, X.shape, generator=g)                   # ✅

kirish, nishon = maskala(X, g)                # bir marta, sikldan oldin  # ⚠️
for _ in range(qadamlar):
    kirish, nishon = maskala(X[idx], g)       # har batchda yangi  # ✅

aniq = (bash == X)[nishon != -100].mean()     # o'zgarmaganlar ham # ⚠️
aniq = (bash == X)[kirish == MASK].float().mean()                  # ✅

F.scaled_dot_product_attention(q, k, v)                            # ⚠️
F.scaled_dot_product_attention(q, k, v,
                               attn_mask=(x != PAD)[:, None, None, :])  # ✅

1.  # 1000 ta oddiy tokendan taxminan nechtasi tanlanadi?
2.  # tanlanganlardan nechtasi [MASK] bo'ladi?
3.  # tanlanmagan pozitsiyaning nishoni qanday?
4.  # [CLS] tokeni maskalanishi mumkinmi?
5.  # nega 10% tokenni o'zgarishsiz qoldiramiz?
6.  # BERT kirish embeddingi qaysi uch qismdan iborat?
7.  # [CLS] A [SEP] B [SEP] da ikkinchi [SEP] ning segmenti?
8.  # NSP da IsNext ulushi qancha?
9.  # nega RoBERTa NSP ni olib tashladi?
10. # 10 epoxa dinamik maskalashda token kamida bir marta nishon bo'lish ehtimoli?
11. # MLM aniqligini qaysi pozitsiyalarda o'lchash kerak?
12. # statik embeddingda "yuz" (100) va "yuz" (bet) vektorlari?

1.  tanlandi = torch.rand(X.shape) < 0.15
    kirish = X.masked_fill(tanlandi, MASK)

2.  loss = F.cross_entropy(model(kirish).reshape(-1, V), X.reshape(-1))

3.  kirish[tasodifiy] = torch.randint(0, V, X.shape)[tasodifiy]

4.  kirish, nishon = maskala(X, g)
    for _ in range(1000):
        loss = ce(model(kirish), nishon)

5.  aniq = (model(kirish).argmax(-1) == X)[nishon != -100].float().mean()
    print("MLM aniqligi:", aniq)