23.3-dars: Subword tokenizatsiya

UCH DARAJA 23.2-bob:
  so'z     - lug'at katta, OOV bor, ketma-ketlik qisqa
  belgi    - lug'at ~30-50, OOV yo'q, ketma-ketlik ~6 barobar uzun
  subword  - lug'at o'zimiz tanlaymiz, OOV yo'q, ketma-ketlik o'rtacha

ASOSIY G'OYA:
  tez-tez so'z       -> butun token         "bank"
  kam uchragan so'z  -> ma'lum bo'laklar    "jamoa | larimiz | ning"
  hech ko'rilmagan   -> kichikroq bo'laklar  "f | u | t | bo | l | chi ..."
  eng yomon holat    -> alohida belgilar (OOV emas!)

NEGA AGGLUTINATIV TILGA MOS:
  qo'shimchalar MINGLAB so'zda takrorlanadi -> chastotasi yuqori
  -> BPE ularni erta birlashtiradi -> qo'shimcha bitta token
  asos + qo'shimcha tokenlari kombinatsiyasi ko'rilmagan shakllarni ham qoplaydi

TAYYORGARLIK:
  so'z chastotalari: {"kitob": 6, "kitoblar": 5, ...}
  har so'z belgilarga: "kitob" -> k i t o b</w>
  </w> - so'z oxiri belgisi (oxirgi belgiga qo'shiladi)
    "lar</w>" (so'z oxiridagi -lar) va "lar" (so'z o'rtasidagi) - TURLI tokenlar

TSIKL (n marta):
  1. barcha yonma-yon juftlarni chastota bilan sanash
       ("l", "a"): 26   ("t", "a"): 17 ...
  2. eng ko'p juftni tanlash (teng bo'lsa - deterministik qoida)
  3. shu juftni hamma so'zda yangi tokenga birlashtirish  l + a -> la
  4. qoidani ro'yxatga yozish

NATIJA:
  qoidalar ro'yxati [("l","a"), ("t","a"), ..., ("nin","g</w>")]
  lug'at = boshlang'ich belgilar + har qoida bitta yangi token
  lug'at hajmi = belgilar soni + qoidalar soni (n - GIPERPARAMETR)

SAMARADORLIK:
  sodda usul har qadamda hamma juftlarni qayta sanaydi
  amaliy usul faqat o'zgargan so'zlarni yangilaydi (juft -> so'zlar indeksi)

KODLASH (encode):
  1. so'zni belgilarga bo'lish: j a m o a l a r i m i z n i n g</w>
  2. qoidalarni O'RGANILGAN TARTIBDA qo'llash:
     har qadamda yonma-yon juftlardan eng kichik RANGLISINI birlashtirish
  3. birlashtiriladigan juft qolmaganda to'xtash

NEGA TARTIB MUHIM:
  ("l","a") 1-qoida, ("la","r</w>") 6-qoida
  teskari tartibda qo'llasak - boshqa bo'linish, o'quvdagi bilan mos emas
  -> o'quv va ishlatishda bo'linish bir xil bo'lishi SHART

OOV YO'Q:
  har so'z oxir-oqibat boshlang'ich belgilarga bo'linadi
  faqat o'quvda UMUMAN ko'rilmagan BELGI <unk> bo'ladi
  (masalan turkcha U+015F yoki e-akut)
  yechim: bayt darajasidagi BPE - boshlang'ich lug'at 256 bayt

DEKODLASH:
  tokenlarni yopishtirish, </w> ni bo'sh joyga almashtirish
  "jamoa" + "larimiz" + "ning</w>" -> "jamoalarimizning "

BIRLASHMALAR SONI k:
  k = 0      -> faqat belgilar: lug'at kichik, token/so'z = so'z uzunligi
  k kichik   -> tez-tez juftlar (la, ar, ni) - katta foyda
  k katta    -> butun so'zlar token bo'ladi - foyda kamayadi
  k -> cheksiz -> so'z darajasidagi lug'atga yaqinlashadi

KAMAYIB BORUVCHI FOYDA:
  birinchi 50 birlashma token/so'zni keskin kamaytiradi
  keyingi yuzlab birlashmalar - tobora kamroq
  (Zipf ning natijasi, 23.2: kam juftlar juda ko'p)

NIMA ORQALI TANLANADI:
  embedding jadvali = lug'at * d   (katta lug'at - ko'p parametr)
  ketma-ketlik uzunligi - keyingi qismlardagi modellarda hisob narxi
    uzunlik bilan o'sadi (Transformerlar qismida - kvadratik)
  odatiy qiymatlar: 8 000 - 50 000 (bitta til), 100 000+ (ko'p til)

NIMA KUTAMIZ:
  -lar, -ning, -dagi, -dan, -da, -ni, -ga minglab so'zda
  -> ularning belgi juftlari eng ko'p sanaladi
  -> BPE ularni ERTA birlashtiradi (birinchi o'nlab qadamlarda)

NATIJA:
  "jamoalarimizning" -> jamoa | larimizning</w>
  "stadionlaringizdan" -> stadion | laringiz | dan</w>
  asos va qo'shimcha orasidagi chegara ko'pincha BPE chegarasiga to'g'ri keladi

LEKIN:
  BPE morfologiyani BILMAYDI - faqat chastotani
  tez-tez uchraydigan birikma bitta token bo'lib qoladi ("larimiz")
  tez-tez so'z to'liq token bo'ladi ("bankdagi</w>") - chegara yo'q
  kam asos bo'laklarga bo'linadi ("f | u | t | bo | l | chi")

TEKSHIRISH:
  asos chegarasida BPE chegarasi bormi - ulush
  taqqoslash: xuddi shuncha TASODIFIY chegara bilan kutilgan ulush
  juftlashgan farq va SE (18-qism)

WORDPIECE (BERT oilasi, Transformerlar qismida):
  birlashtirish mezoni - chastota emas, "foydalilik":
    ball(a, b) = son(ab) / (son(a) * son(b))
    alohida kam, birga ko'p uchraydigan juft afzal
  so'z ichidagi bo'laklar "##" bilan belgilanadi: jamoa ##lar ##imiz
  kodlash: chapdan ENG UZUN mos bo'lakni ochko'z tanlash
  lug'atda yo'q bo'lsa - butun so'z [UNK]

SENTENCEPIECE:
  matnni oldindan so'zlarga bo'lmaydi - bo'sh joy ham oddiy belgi
    (maxsus U+2581 belgisiga almashtiriladi)
  -> tilga bog'liq bo'lmagan: xitoy, yapon, o'zbek - bir xil
  ikki rejim: BPE va Unigram
    Unigram: katta lug'atdan boshlab, ehtimolni eng kam kamaytiradigan
    tokenlarni olib tashlaydi; bir so'zning bir necha bo'linishi mumkin
  normallashtirishni (NFKC) o'zi bajaradi

BAYT DARAJASIDAGI BPE (GPT oilasi, Katta til modellari qismida):
  boshlang'ich lug'at = 256 bayt -> <unk> UMUMAN yo'q
  diqqat: U+2019 = 3 bayt, U+02BB = 2 bayt
    normallashtirilmagan o'zbekcha matnda apostrof ko'proq token oladi
    va "o'" ning har varianti uchun alohida birlashmalar kerak bo'ladi

from collections import Counter

OXIR = "</w>"


def bolakla(soz):
    return tuple(soz[:-1]) + (soz[-1] + OXIR,)


def juftlarni_sana(lugat):                    # {bo'laklar: chastota}
    sanoq = Counter()
    for b, n in lugat.items():
        for juft in zip(b, b[1:]):
            sanoq[juft] += n
    return sanoq


lugat = {bolakla(s): n for s, n in chastota.items()}
qoidalar = []
for _ in range(n_birlashma):
    sanoq = juftlarni_sana(lugat)
    juft = max(sanoq, key=lambda j: (sanoq[j], j))   # deterministik
    lugat = birlashtir(lugat, juft)
    qoidalar.append(juft)

rang = {j: i for i, j in enumerate(qoidalar)}         # qo'llash uchun
tokenlar = bpe_qolla("jamoalarimizning", rang)       # eng kichik rang birinchi

BPE: belgilardan boshlab eng ko'p juftni birlashtirish, n marta
qoidalar ro'yxati = model; tartibda qo'llanadi
</w> so'z oxirini belgilaydi: lar va lar</w> turli tokenlar
OOV so'z yo'q; faqat noma'lum belgi (bayt BPE da u ham yo'q)
lug'at hajmi - giperparametr: token/so'z egri chizig'i bilan tanlang
o'zbekcha qo'shimchalar erta birlashadi, lekin bu morfologiya emas

"""BPE noldan: juftlarni sanash, eng ko'p juftni birlashtirish, qoidalar ro'yxati."""

from collections import Counter

OXIR = "</w>"

CHASTOTA = {"kitob": 6, "kitoblar": 5, "kitobning": 3, "kitoblarning": 2,
            "daftar": 4, "daftarlar": 4, "daftarning": 3, "qalam": 5,
            "qalamlar": 3, "qalamning": 2, "maktab": 4, "maktablar": 2}


def bolakla(soz):
    return tuple(soz[:-1]) + (soz[-1] + OXIR,)


def juftlarni_sana(lugat):
    """lugat: {bo'laklar kortegi: chastota} -> {(a, b): jami chastota}."""
    sanoq = Counter()
    for bolaklar, n in lugat.items():
        for juft in zip(bolaklar, bolaklar[1:]):
            sanoq[juft] += n
    return sanoq


def birlashtir(lugat, juft):
    a, b = juft
    yangi_lugat = {}
    for bolaklar, n in lugat.items():
        yangi, i = [], 0
        while i < len(bolaklar):
            if i < len(bolaklar) - 1 and (bolaklar[i], bolaklar[i + 1]) == juft:
                yangi.append(a + b)
                i += 2
            else:
                yangi.append(bolaklar[i])
                i += 1
        yangi_lugat[tuple(yangi)] = n
    return yangi_lugat


def main() -> None:
    lugat = {bolakla(s): n for s, n in CHASTOTA.items()}
    print("=== 1. Boshlang'ich holat: har so'z - belgilar ===")
    for b, n in list(lugat.items())[:3]:
        print(f"  {n} x {' '.join(b)}")
    belgilar = sorted({x for b in lugat for x in b})
    print(f"  boshlang'ich lug'at ({len(belgilar)} belgi): {belgilar}")
    jami_tok = sum(len(b) * n for b, n in lugat.items())
    print(f"  jami tokenlar (chastota bilan): {jami_tok}")

    print("\n=== 2. Birlashtirish qadamlari ===")
    qoidalar = []
    print(f"  {'qadam':>5} {'juft':<22} {'soni':>5} {'yangi token':<12} "
          f"{'tokenlar':>8}")
    for qadam in range(1, 15):
        sanoq = juftlarni_sana(lugat)
        juft = max(sanoq, key=lambda j: (sanoq[j], j))
        lugat = birlashtir(lugat, juft)
        qoidalar.append(juft)
        jami_tok = sum(len(b) * n for b, n in lugat.items())
        print(f"  {qadam:>5} {str(juft):<22} {sanoq[juft]:>5} "
              f"{juft[0] + juft[1]:<12} {jami_tok:>8}")

    print("\n=== 3. 14 qadamdan keyingi bo'linish ===")
    for b, n in lugat.items():
        print(f"  {n} x {' | '.join(b)}")

    print("\n=== 4. Qoidalar ro'yxati - BPE modelining o'zi ===")
    print(f"  {len(qoidalar)} ta qoida, tartib MUHIM:")
    print("  " + ", ".join(f"{a}+{b}" for a, b in qoidalar[:7]))
    print("  " + ", ".join(f"{a}+{b}" for a, b in qoidalar[7:]))
    qoshimcha = [a + b for a, b in qoidalar if (a + b) in ("lar" + OXIR, "ning" + OXIR)]
    print(f"  qo'shimchalar alohida token bo'ldi: {qoshimcha}")
    print("  ⭐ BPE - chastotali juftlarni ochko'z birlashtirish; qoidalar tartibi saqlanadi")


if __name__ == "__main__":
    main()

=== 1. Boshlang'ich holat: har so'z - belgilar ===
  6 x k i t o b</w>
  5 x k i t o b l a r</w>
  3 x k i t o b n i n g</w>
  boshlang'ich lug'at (17 belgi): ['a', 'b', 'b</w>', 'd', 'f', 'g</w>', 'i', 'k', 'l', 'm', 'm</w>', 'n', 'o', 'q', 'r', 'r</w>', 't']
  jami tokenlar (chastota bilan): 320

=== 2. Birlashtirish qadamlari ===
  qadam juft                    soni yangi token  tokenlar
      1 ('l', 'a')                26 la                294
      2 ('t', 'a')                17 ta                277
      3 ('t', 'o')                16 to                261
      4 ('k', 'i')                16 ki                245
      5 ('ki', 'to')              16 kito              229
      6 ('la', 'r</w>')           14 lar</w>           215
      7 ('f', 'ta')               11 fta               204
      8 ('d', 'a')                11 da                193
      9 ('da', 'fta')             11 dafta             182
     10 ('q', 'a')                10 qa                172
     11 ('qa', 'la')              10 qala              162
     12 ('n', 'i')                10 ni                152
     13 ('ni', 'n')               10 nin               142
     14 ('nin', 'g</w>')          10 ning</w>          132

=== 3. 14 qadamdan keyingi bo'linish ===
  6 x kito | b</w>
  5 x kito | b | lar</w>
  3 x kito | b | ning</w>
  2 x kito | b | la | r | ning</w>
  4 x dafta | r</w>
  4 x dafta | r | lar</w>
  3 x dafta | r | ning</w>
  5 x qala | m</w>
  3 x qala | m | lar</w>
  2 x qala | m | ning</w>
  4 x m | a | k | ta | b</w>
  2 x m | a | k | ta | b | lar</w>

=== 4. Qoidalar ro'yxati - BPE modelining o'zi ===
  14 ta qoida, tartib MUHIM:
  l+a, t+a, t+o, k+i, ki+to, la+r</w>, f+ta
  d+a, da+fta, q+a, qa+la, n+i, ni+n, nin+g</w>
  qo'shimchalar alohida token bo'ldi: ['lar</w>', 'ning</w>']
  ⭐ BPE - chastotali juftlarni ochko'z birlashtirish; qoidalar tartibi saqlanadi

"""BPE ni korpusda o'rgatish va yangi so'zlarga qo'llash: OOV qayerga ketdi."""

import re
import unicodedata
from collections import Counter, defaultdict

import numpy as np

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    yoz = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yoz and "CYRILLIC" in yoz:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    """23.1-darsdagi quvur: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam."""
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    return re.sub(r"[0-9]+", "0", matn.lower())


OXIR = "</w>"


def bolakla(soz):
    """'narx' -> ('n', 'a', 'r', 'x</w>'): so'z oxiri belgisi oxirgi harfga qo'shiladi."""
    return tuple(soz[:-1]) + (soz[-1] + OXIR,)


def juftlar(b):
    return list(zip(b, b[1:]))


def birlashtir_bolak(b, juft):
    a, c = juft
    yangi, i = [], 0
    while i < len(b):
        if i < len(b) - 1 and b[i] == a and b[i + 1] == c:
            yangi.append(a + c)
            i += 2
        else:
            yangi.append(b[i])
            i += 1
    return tuple(yangi)


def bpe_orgat(chastota, n_birlashma):
    """Juftlar sanog'ini qayta hisoblamay, faqat o'zgargan so'zlarni yangilaydi."""
    bolaklar = {s: bolakla(s) for s in sorted(chastota)}
    sanoq = Counter()
    joy = defaultdict(set)
    for s, b in bolaklar.items():
        for j in juftlar(b):
            sanoq[j] += chastota[s]
            joy[j].add(s)
    qoidalar = []
    for _ in range(n_birlashma):
        if not sanoq:
            break
        eng = max(sanoq, key=lambda j: (sanoq[j], j))
        qoidalar.append(eng)
        for s in sorted(joy.pop(eng)):
            eski = bolaklar[s]
            for j in juftlar(eski):
                sanoq[j] -= chastota[s]
                if sanoq[j] <= 0:
                    del sanoq[j]
            yangi = birlashtir_bolak(eski, eng)
            bolaklar[s] = yangi
            for j in juftlar(yangi):
                sanoq[j] += chastota[s]
                joy[j].add(s)
    return qoidalar


def bpe_qolla(soz, rang):
    """Qoidalarni o'rganilgan tartibda: har qadamda eng kichik rangli juft."""
    b = bolakla(soz)
    while len(b) > 1:
        nomzod = [(rang[j], j) for j in juftlar(b) if j in rang]
        if not nomzod:
            break
        b = birlashtir_bolak(b, min(nomzod)[1])
    return b


SOZ_RE = re.compile(r"[a-z]+(?:'[a-z]+)*")


def main() -> None:
    rng = np.random.default_rng(0)
    korpus = [SOZ_RE.findall(normalla(hujjat(rng)[1])) for _ in range(3000)]
    tartib = np.random.default_rng(1).permutation(len(korpus))
    oquv = [korpus[i] for i in tartib[:2400]]
    test = [korpus[i] for i in tartib[2400:]]
    chastota = Counter(s for h in oquv for s in h)

    print("=== 1. O'rgatish ===")
    qoidalar = bpe_orgat(chastota, 500)
    rang = {j: i for i, j in enumerate(qoidalar)}
    belgilar = sorted({x for s in chastota for x in bolakla(s)})
    print(f"  o'quv so'zlari: {sum(chastota.values())}, noyob: {len(chastota)}")
    print(f"  boshlang'ich belgilar: {len(belgilar)}, qoidalar: {len(qoidalar)}")
    print(f"  BPE lug'ati: {len(belgilar) + len(qoidalar)} token")
    print(f"  birinchi 10 qoida: {[a + '+' + b for a, b in qoidalar[:10]]}")

    print("\n=== 2. Yangi so'zlarga qo'llash ===")
    yangi = ["jamoalarimizning", "o'yinchilarimizdagi", "futbolchilarimizning",
             "jamaolarimiz", "stadionlaringizdan", "kompyuterlashtirish"]
    for s in yangi:
        b = bpe_qolla(s, rang)
        print(f"  {s:<22} o'quvda {chastota[s]:>2} marta -> "
              f"{' | '.join(b)}")

    print("\n=== 3. OOV: so'z lug'ati va BPE ===")
    soz_lug = {s for s, n in chastota.items() if n >= 2}
    test_sozlar = [s for h in test for s in h]
    oov_soz = np.mean([s not in soz_lug for s in test_sozlar])
    bpe_lug = set(belgilar) | {a + b for a, b in qoidalar}
    kesh = {}
    unk = 0
    for s in test_sozlar:
        if s not in kesh:
            kesh[s] = bpe_qolla(s, rang)
        unk += sum(t not in bpe_lug for t in kesh[s])
    print(f"  so'z lug'ati (min_chastota=2): {len(soz_lug)} so'z, "
          f"test OOV {oov_soz:.2%}")
    print(f"  BPE lug'ati: {len(bpe_lug)} token, <unk> tokenlar: {unk}")
    print(f"  test so'zi o'rtacha {np.mean([len(kesh[s]) for s in test_sozlar]):.2f} "
          f"BPE tokenga bo'lindi")

    print("\n=== 4. Noma'lum BELGI - BPE ham ojiz ===")
    for s in ["qo'shiq", "\u015fahar", "caf\u00e9"]:
        b = bpe_qolla(s, rang)
        yoq = [t for t in b if t not in bpe_lug]
        print(f"  {ascii(s):<12} -> {ascii(' | '.join(b)):<34} lug'atda yo'q: "
              f"{ascii(yoq)}")
    print("  yechim: bayt darajasidagi BPE (256 bayt - hamma narsa qoplanadi)")
    print("  ⭐ BPE da OOV so'z yo'q: noma'lum so'z ma'lum bo'laklarga bo'linadi")


if __name__ == "__main__":
    main()

=== 1. O'rgatish ===
  o'quv so'zlari: 16440, noyob: 2947
  boshlang'ich belgilar: 46, qoidalar: 500
  BPE lug'ati: 546 token
  birinchi 10 qoida: ['l+a', 'd+i</w>', 'm+a', 'd+a', 'i+n', 'la+r', 't+a', 's+h', "o+'", 'o+r']

=== 2. Yangi so'zlarga qo'llash ===
  jamoalarimizning       o'quvda  0 marta -> jamoa | larimiz | ning</w>
  o'yinchilarimizdagi    o'quvda  1 marta -> o'yinchi | larimizdagi</w>
  futbolchilarimizning   o'quvda  0 marta -> f | u | t | bo | l | chi | larimiz | ning</w>
  jamaolarimiz           o'quvda  0 marta -> ja | ma | o | larimiz</w>
  stadionlaringizdan     o'quvda  0 marta -> stadion | laringiz | dan</w>
  kompyuterlashtirish    o'quvda  0 marta -> kompyuter | la | sh | ti | ri | s | h</w>

=== 3. OOV: so'z lug'ati va BPE ===
  so'z lug'ati (min_chastota=2): 1500 so'z, test OOV 16.55%
  BPE lug'ati: 546 token, <unk> tokenlar: 0
  test so'zi o'rtacha 1.69 BPE tokenga bo'lindi

=== 4. Noma'lum BELGI - BPE ham ojiz ===
  "qo'shiq"    -> "q | o' | shi | q</w>"             lug'atda yo'q: []
  '\u015fahar' -> '\u015f | a | ha | r</w>'          lug'atda yo'q: ['\u015f']
  'caf\xe9'    -> 'c | a | f | \xe9</w>'             lug'atda yo'q: ['\xe9</w>']
  yechim: bayt darajasidagi BPE (256 bayt - hamma narsa qoplanadi)
  ⭐ BPE da OOV so'z yo'q: noma'lum so'z ma'lum bo'laklarga bo'linadi

"""Lug'at hajmi va ketma-ketlik uzunligi savdosi: nechta birlashma kerak?"""

import re
import unicodedata
from collections import Counter, defaultdict

import numpy as np

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    yoz = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yoz and "CYRILLIC" in yoz:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    """23.1-darsdagi quvur: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam."""
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    return re.sub(r"[0-9]+", "0", matn.lower())


OXIR = "</w>"


def bolakla(soz):
    """'narx' -> ('n', 'a', 'r', 'x</w>'): so'z oxiri belgisi oxirgi harfga qo'shiladi."""
    return tuple(soz[:-1]) + (soz[-1] + OXIR,)


def juftlar(b):
    return list(zip(b, b[1:]))


def birlashtir_bolak(b, juft):
    a, c = juft
    yangi, i = [], 0
    while i < len(b):
        if i < len(b) - 1 and b[i] == a and b[i + 1] == c:
            yangi.append(a + c)
            i += 2
        else:
            yangi.append(b[i])
            i += 1
    return tuple(yangi)


def bpe_orgat(chastota, n_birlashma):
    """Juftlar sanog'ini qayta hisoblamay, faqat o'zgargan so'zlarni yangilaydi."""
    bolaklar = {s: bolakla(s) for s in sorted(chastota)}
    sanoq = Counter()
    joy = defaultdict(set)
    for s, b in bolaklar.items():
        for j in juftlar(b):
            sanoq[j] += chastota[s]
            joy[j].add(s)
    qoidalar = []
    for _ in range(n_birlashma):
        if not sanoq:
            break
        eng = max(sanoq, key=lambda j: (sanoq[j], j))
        qoidalar.append(eng)
        for s in sorted(joy.pop(eng)):
            eski = bolaklar[s]
            for j in juftlar(eski):
                sanoq[j] -= chastota[s]
                if sanoq[j] <= 0:
                    del sanoq[j]
            yangi = birlashtir_bolak(eski, eng)
            bolaklar[s] = yangi
            for j in juftlar(yangi):
                sanoq[j] += chastota[s]
                joy[j].add(s)
    return qoidalar


def bpe_qolla(soz, rang):
    """Qoidalarni o'rganilgan tartibda: har qadamda eng kichik rangli juft."""
    b = bolakla(soz)
    while len(b) > 1:
        nomzod = [(rang[j], j) for j in juftlar(b) if j in rang]
        if not nomzod:
            break
        b = birlashtir_bolak(b, min(nomzod)[1])
    return b


SOZ_RE = re.compile(r"[a-z]+(?:'[a-z]+)*")


def main() -> None:
    rng = np.random.default_rng(0)
    korpus = [SOZ_RE.findall(normalla(hujjat(rng)[1])) for _ in range(9000)]
    oquv, test = korpus[:8000], korpus[8000:]
    chastota = Counter(s for h in oquv for s in h)
    test_sozlar = [s for h in test for s in h]
    test_noyob = sorted(set(test_sozlar))
    belgilar = sorted({x for s in chastota for x in bolakla(s)})
    qoidalar = bpe_orgat(chastota, 2000)

    print("=== 1. Ma'lumot ===")
    print(f"  o'quv: {len(oquv)} hujjat, {len(chastota)} noyob so'z")
    print(f"  test: {len(test)} hujjat, {len(test_sozlar)} so'z")
    print(f"  o'rgatilgan qoidalar: {len(qoidalar)}")

    print("\n=== 2. Birlashmalar soni bo'yicha savdo ===")
    print(f"  {'birlashma':>9} {'lug_at':>7} {'token/so_z':>11} "
          f"{'1 tokenli so_z':>15} {'token/hujjat':>13}")
    natija, unk_soni = {}, {}
    for k in [0, 50, 100, 200, 500, 1000, 2000]:
        rang = {j: i for i, j in enumerate(qoidalar[:k])}
        bolak = {s: bpe_qolla(s, rang) for s in test_noyob}
        lug_k = set(belgilar) | {a + b for a, b in qoidalar[:k]}
        unk_soni[k] = sum(any(t not in lug_k for t in bolak[s]) for s in test_sozlar)
        uzun = {s: len(bolak[s]) for s in test_noyob}
        tok = np.array([uzun[s] for s in test_sozlar])
        hujjat_uz = np.mean([sum(uzun[s] for s in h) for h in test])
        natija[k] = tok.mean()
        print(f"  {k:>9} {len(belgilar) + k:>7} {tok.mean():>11.2f} "
              f"{(tok == 1).mean():>15.1%} {hujjat_uz:>13.1f}")

    print("\n=== 3. Taqqoslash uchun: so'z lug'ati ===")
    for m in [1, 2, 5]:
        lug = {s for s, n in chastota.items() if n >= m}
        oov = np.mean([s not in lug for s in test_sozlar])
        print(f"  min_chastota={m}: lug'at {len(lug):>5}, 1 token/so'z, "
              f"test OOV {oov:.2%}")
    print(f"  BPE: <unk> tokenli test so'zlari - eng ko'pi {max(unk_soni.values())} ta "
          f"(barcha k da)")

    print("\n=== 4. Kamayib boruvchi foyda ===")
    kk = sorted(natija)
    for a, b in zip(kk, kk[1:]):
        foyda = (natija[a] - natija[b]) / (b - a) * 100
        print(f"  {a:>4} -> {b:<4}: har 100 birlashma token/so'zni "
              f"{foyda:.3f} ga kamaytiradi")
    print("  ⭐ Lug'at hajmi - giperparametr: kichik lug'at uzun ketma-ketlik beradi")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  o'quv: 8000 hujjat, 5149 noyob so'z
  test: 1000 hujjat, 6701 so'z
  o'rgatilgan qoidalar: 2000

=== 2. Birlashmalar soni bo'yicha savdo ===
  birlashma  lug_at  token/so_z  1 tokenli so_z  token/hujjat
          0      46        8.39            0.0%          56.2
         50      96        5.14            3.3%          34.4
        100     146        4.20           12.3%          28.2
        200     246        3.17           12.3%          21.3
        500     546        1.66           46.6%          11.1
       1000    1046        1.34           71.5%           8.9
       2000    2046        1.17           86.5%           7.8

=== 3. Taqqoslash uchun: so'z lug'ati ===
  min_chastota=1: lug'at  5149, 1 token/so'z, test OOV 3.70%
  min_chastota=2: lug'at  3107, 1 token/so'z, test OOV 6.67%
  min_chastota=5: lug'at  1466, 1 token/so'z, test OOV 13.21%
  BPE: <unk> tokenli test so'zlari - eng ko'pi 0 ta (barcha k da)

=== 4. Kamayib boruvchi foyda ===
     0 -> 50  : har 100 birlashma token/so'zni 6.508 ga kamaytiradi
    50 -> 100 : har 100 birlashma token/so'zni 1.864 ga kamaytiradi
   100 -> 200 : har 100 birlashma token/so'zni 1.029 ga kamaytiradi
   200 -> 500 : har 100 birlashma token/so'zni 0.506 ga kamaytiradi
   500 -> 1000: har 100 birlashma token/so'zni 0.064 ga kamaytiradi
  1000 -> 2000: har 100 birlashma token/so'zni 0.016 ga kamaytiradi
  ⭐ Lug'at hajmi - giperparametr: kichik lug'at uzun ketma-ketlik beradi

"""O'zbekcha qo'shimchalar BPE da o'z-o'zidan ajralib chiqadimi?"""

import re
import unicodedata
from collections import Counter, defaultdict

import numpy as np

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    yoz = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yoz and "CYRILLIC" in yoz:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    """23.1-darsdagi quvur: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam."""
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    return re.sub(r"[0-9]+", "0", matn.lower())


OXIR = "</w>"


def bolakla(soz):
    """'narx' -> ('n', 'a', 'r', 'x</w>'): so'z oxiri belgisi oxirgi harfga qo'shiladi."""
    return tuple(soz[:-1]) + (soz[-1] + OXIR,)


def juftlar(b):
    return list(zip(b, b[1:]))


def birlashtir_bolak(b, juft):
    a, c = juft
    yangi, i = [], 0
    while i < len(b):
        if i < len(b) - 1 and b[i] == a and b[i + 1] == c:
            yangi.append(a + c)
            i += 2
        else:
            yangi.append(b[i])
            i += 1
    return tuple(yangi)


def bpe_orgat(chastota, n_birlashma):
    """Juftlar sanog'ini qayta hisoblamay, faqat o'zgargan so'zlarni yangilaydi."""
    bolaklar = {s: bolakla(s) for s in sorted(chastota)}
    sanoq = Counter()
    joy = defaultdict(set)
    for s, b in bolaklar.items():
        for j in juftlar(b):
            sanoq[j] += chastota[s]
            joy[j].add(s)
    qoidalar = []
    for _ in range(n_birlashma):
        if not sanoq:
            break
        eng = max(sanoq, key=lambda j: (sanoq[j], j))
        qoidalar.append(eng)
        for s in sorted(joy.pop(eng)):
            eski = bolaklar[s]
            for j in juftlar(eski):
                sanoq[j] -= chastota[s]
                if sanoq[j] <= 0:
                    del sanoq[j]
            yangi = birlashtir_bolak(eski, eng)
            bolaklar[s] = yangi
            for j in juftlar(yangi):
                sanoq[j] += chastota[s]
                joy[j].add(s)
    return qoidalar


def bpe_qolla(soz, rang):
    """Qoidalarni o'rganilgan tartibda: har qadamda eng kichik rangli juft."""
    b = bolakla(soz)
    while len(b) > 1:
        nomzod = [(rang[j], j) for j in juftlar(b) if j in rang]
        if not nomzod:
            break
        b = birlashtir_bolak(b, min(nomzod)[1])
    return b


SOZ_RE = re.compile(r"[a-z]+(?:'[a-z]+)*")


def main() -> None:
    rng = np.random.default_rng(0)
    korpus = [SOZ_RE.findall(normalla(hujjat(rng)[1])) for _ in range(9000)]
    oquv, test = korpus[:8000], korpus[8000:]
    chastota = Counter(s for h in oquv for s in h)
    qoidalar = bpe_orgat(chastota, 1000)
    rang = {j: i for i, j in enumerate(qoidalar)}
    hosil = {a + b: i + 1 for i, (a, b) in enumerate(qoidalar)}

    print("=== 1. Qo'shimcha tokenlari qaysi qadamda paydo bo'ldi ===")
    nomzodlar = ["lar", "lar</w>", "ning</w>", "dagi</w>", "dan</w>", "da</w>",
                 "ni</w>", "ga</w>", "imiz", "ingiz", "larimiz", "di</w>",
                 "madi</w>", "maydi</w>", "adi</w>"]
    for i in range(0, len(nomzodlar), 5):
        print("  " + "  ".join(f"{t:<10}{hosil.get(t, '-'):>4}"
                               for t in nomzodlar[i:i + 5]))
    topildi = sum(t in hosil for t in nomzodlar)
    print(f"  {len(nomzodlar)} ta qo'shimchadan {topildi} tasi 1000 qadamda token bo'ldi")

    print("\n=== 2. Bo'linish namunalari ===")
    for s in ["jamoalarimizning", "bankdagi", "bozorlaringizdan", "shifokorimni",
              "oshmadi", "tayyorlaydi", "kompaniyamizning", "sog'lig'imiz"]:
        print(f"  {s:<18} -> {' | '.join(bpe_qolla(s, rang))}")

    print("\n=== 3. BPE chegarasi asos chegarasiga to'g'ri keladimi? ===")
    otlar = sorted({a for o in OTLAR.values() for a in o}, key=len, reverse=True)
    namuna = []
    for s in sorted({s for h in test for s in h}):
        asos = next((a for a in otlar if s.startswith(a)), None)
        if asos and len(s) > len(asos):
            namuna.append((s, asos))
    bpe_hit, tasodif, asos_butun = [], [], []
    for s, asos in namuna:
        b = bpe_qolla(s, rang)
        chegaralar, poz = set(), 0
        for t in b[:-1]:
            poz += len(t)
            chegaralar.add(poz)
        bpe_hit.append(len(asos) in chegaralar)
        tasodif.append(len(chegaralar) / (len(s) - 1))
        asos_butun.append(b[0] == asos)
    farq = np.array(bpe_hit, dtype=float) - np.array(tasodif)
    se = farq.std(ddof=1) / np.sqrt(len(farq))
    print(f"  test dagi qo'shimchali ot shakllari: {len(namuna)}")
    print(f"  asos chegarasida BPE chegarasi bor: {np.mean(bpe_hit):.1%}")
    print(f"  shuncha tasodifiy chegara bilan kutilgan: {np.mean(tasodif):.1%}")
    print(f"  farq: {farq.mean():+.3f}, SE {se:.3f}")
    if farq.mean() > 2 * se:
        print("  BPE chegaralari asos chegarasiga tasodifdan sezilarli ko'p tushadi")
    else:
        print("  sezilarli farq yo'q")
    print(f"  asos BITTA birinchi token bo'lgan: {np.mean(asos_butun):.1%}")
    print("  ⭐ BPE morfologiyani bilmaydi, lekin chastota orqali qo'shimchalarni topadi")


if __name__ == "__main__":
    main()

=== 1. Qo'shimcha tokenlari qaysi qadamda paydo bo'ldi ===
  lar          6  lar</w>     29  ning</w>    26  dagi</w>    20  dan</w>     19
  da</w>      12  ni</w>      22  ga</w>      13  imiz        79  ingiz       77
  larimiz    358  di</w>       2  madi</w>    50  maydi</w>   42  adi</w>     43
  15 ta qo'shimchadan 15 tasi 1000 qadamda token bo'ldi

=== 2. Bo'linish namunalari ===
  jamoalarimizning   -> jamoa | larimizning</w>
  bankdagi           -> bankdagi</w>
  bozorlaringizdan   -> bozor | laringizdan</w>
  shifokorimni       -> shifokor | imni</w>
  oshmadi            -> oshmadi</w>
  tayyorlaydi        -> tayyorlaydi</w>
  kompaniyamizning   -> kompaniya | mizning</w>
  sog'lig'imiz       -> sog' | li | g' | imiz</w>

=== 3. BPE chegarasi asos chegarasiga to'g'ri keladimi? ===
  test dagi qo'shimchali ot shakllari: 1581
  asos chegarasida BPE chegarasi bor: 65.8%
  shuncha tasodifiy chegara bilan kutilgan: 8.2%
  farq: +0.576, SE 0.011
  BPE chegaralari asos chegarasiga tasodifdan sezilarli ko'p tushadi
  asos BITTA birinchi token bo'lgan: 65.8%
  ⭐ BPE morfologiyani bilmaydi, lekin chastota orqali qo'shimchalarni topadi

bolaklar = tuple(soz)                            # lar = lar</w>          # ⚠️
bolaklar = tuple(soz[:-1]) + (soz[-1] + "</w>",)                         # ✅

juft = max(set(sanoq), key=sanoq.get)            # teng holatda tasodifiy # ⚠️
juft = max(sanoq, key=lambda j: (sanoq[j], j))                           # ✅

for juft in sorted(qoidalar): b = birlashtir_bolak(b, juft)              # ⚠️
b = bpe_qolla(soz, {j: i for i, j in enumerate(qoidalar)})               # ✅

qoidalar = bpe_orgat(Counter(oquv_sozlar + test_sozlar), 1000)           # ⚠️
qoidalar = bpe_orgat(Counter(oquv_sozlar), 1000)                         # ✅

chastota = Counter(xom_matn.split())             # o', o U+2019 ... alohida # ⚠️
chastota = Counter(SOZ_RE.findall(normalla(xom_matn)))                   # ✅

qoidalar = bpe_orgat(chastota, 50000)            # kichik korpusda        # ⚠️
# k bo'yicha token/so'z egri chizig'ini o'lchab, foyda kamaygan joyda   # ✅

for _ in range(n): sanoq = juftlarni_sana(lugat)  # har qadam hamma so'z  # ⚠️
# juft -> so'zlar indeksi, faqat o'zgargan so'zlarni yangilash         # ✅

1.  # bolakla("narx") ?
2.  # BPE ning har qadamida nima birlashtiriladi?
3.  # 46 belgi va 500 qoida - lug'at hajmi?
4.  # "lar" va "lar</w>" bir xil tokenmi?
5.  # qoidalar qaysi tartibda qo'llanadi?
6.  # o'quvda ko'rilmagan so'z BPE da nima bo'ladi?
7.  # o'quvda ko'rilmagan BELGI nima bo'ladi?
8.  # birlashmalar soni 0 bo'lsa token/so'z nimaga teng?
9.  # birlashmalar soni oshsa token/so'z qanday o'zgaradi?
10. # WordPiece birlashtirish bali?
11. # SentencePiece bo'sh joyni qanday ko'radi?
12. # bayt BPE da boshlang'ich lug'at hajmi?

1.  bolaklar = tuple(soz)

2.  juft = max(set(sanoq), key=sanoq.get)

3.  qoidalar = bpe_orgat(Counter(barcha_sozlar), 1000)   # o'quv + test

4.  for juft in reversed(qoidalar):
        b = birlashtir_bolak(b, juft)

5.  chastota = Counter(matn.split())                     # xom matn