23.2-dars: Tokenizatsiya

TOKEN = modelning eng kichik birligi
  so'z tokenlari   "narx oshdi"  -> ["narx", "oshdi"]
  belgi tokenlari  "narx"        -> ["n", "a", "r", "x"]
  subword tokenlar "narxlari"    -> ["narx", "lari"]   (23.3-dars)

SAVDO (trade-off):
                    lug'at hajmi     ketma-ketlik uzunligi   OOV
  so'z              katta (minglab)  qisqa                   ko'p
  belgi             kichik (~30-100) uzun (x6)               deyarli yo'q
  subword           o'rtacha         o'rtacha                deyarli yo'q

SO'Z TOKENIZATORLARI:
  s.split()          - faqat bo'sh joy; tinish so'zga yopishadi ("bor!")
  re.findall(r"\w+") - harf/raqam ketma-ketligi; apostrofda bo'linadi
  o'z regex          - til qoidalarini hisobga oladi

MUAMMO (\w+):
  "o'qituvchi"   -> ["o", "qituvchi"]         U+0027 harf emas
  "ma'lumot"     -> ["ma", "lumot"]
  U+02BB bilan   -> bitta token               U+02BB harf (Lm)
  -> bir so'z yozilishiga qarab 1 yoki 2 token (23.1 ning davomi)
  -> lug'atda "o" va "qituvchi" kabi ma'nosiz bo'laklar

YECHIM: avval normallashtirish 23.1-bob, keyin o'z regex:
  TOKEN_RE = r"[a-z0-9]+(?:['-][a-z0-9]+)*|[^\w\s]"
    [a-z0-9]+             - harf/raqam bo'lagi
    (?:['-][a-z0-9]+)*    - ICHIDA apostrof yoki chiziqcha bo'lsa davomi
    [^\w\s]               - har tinish belgisi alohida token

  "o'qituvchi"   -> ["o'qituvchi"]
  "'yaxshi'"     -> ["'", "yaxshi", "'"]      qo'shtirnoq sifatida
  "0-yilda"      -> ["0-yilda"]
  "bor!"         -> ["bor", "!"]

TINISH BELGILARI:
  alohida token qiling - "!" va "?" sentiment uchun foydali
  so'zga yopishtirib qoldirmang - "bor", "bor!", "bor." uch xil so'z bo'ladi

LUG'AT = token <-> butun son jadvali
  itos: ["<pad>", "<unk>", ".", "!", "?", "0", "emas", ...]   indeks -> token
  stoi: {"<pad>": 0, "<unk>": 1, ".": 2, ...}                token -> indeks

MAXSUS TOKENLAR:
  <pad> = 0  - batchdagi qisqa ketma-ketliklarni to'ldirish
               niqob: X != 0; Embedding(padding_idx=0) (21-qism)
  <unk> = 1  - lug'atda yo'q har qanday token

QURISH TARTIBI:
  1. tokenlarni O'QUV qismidan sanash
  2. min_chastota dan kamlarini tashlash
  3. chastota kamayishi, teng bo'lsa alifbo bo'yicha tartiblash (deterministik)
  4. maxsus tokenlar oldinda

NIMA UCHUN FAQAT O'QUVDAN:
  test/ishlab chiqarishdagi so'zlar oldindan ma'lum emas
  lug'at hammasidan qurilsa -> test OOV = 0 (yolg'on optimizm)
  bu 18-qismdagi sizib ketish (leakage) ning matndagi ko'rinishi

OOV ULUSHI = lug'atda yo'q tokenlar / barcha tokenlar
  o'quvda (min_chastota=1) - har doim 0
  testda                   - haqiqiy ko'rsatkich

MIN_CHASTOTA:
  kattalashtirsak -> lug'at kichrayadi, OOV ortadi
  1 marta uchragan so'z uchun embedding o'rganib bo'lmaydi
    (bitta misoldan vektor - shovqin)
  min_chastota=2..5 - odatiy tanlov
  o'quvda ham OOV paydo bo'ladi -> model <unk> ni o'rganadi (FOYDALI)

OOV NIMADAN IBORAT:
  imlo xatolari           - normallashtirish tuzatmaydi
  kam uchraydigan shakllar - o'zbekchada ASOSIY manba (qo'shimchalar)
  yangi so'zlar           - nomlar, atamalar

ZIPF QONUNI:
  so'zlarni chastota bo'yicha tartiblang: rang r = 1, 2, 3, ...
  chastota f(r) ~ C / r        ->   f(r) * r ~ C (taxminan o'zgarmas)
  log-log grafikda: log f = log C - 1 * log r   (qiyalik ~ -1)

OQIBATLARI:
  bir necha so'z (the, a, is / emas, bor) juda ko'p
  so'zlarning katta qismi bir-ikki marta uchraydi (uzun dum)
  top-100 so'z tokenlarning ~yarmini qoplaydi
  lekin qolgan yarmi uchun minglab so'z kerak

TEKSHIRISH:
  qiyalikning o'zi yetarli emas - f * r ham barqaror bo'lishi kerak
  kombinatsiyadan yasalgan sun'iy matn ham -1 ga yaqin qiyalik berishi
  mumkin, lekin f * r unda keskin tebranadi

AMALIY MA'NOSI:
  lug'atni kichraytirish tokenlarning kichik ulushini <unk> ga aylantiradi
  lekin OOV ni to'liq yo'qotish mumkin emas - dum cheksiz

AGGLUTINATIV TIL: asosga qo'shimchalar ketma-ket yopishadi
  jamoa + lar + imiz + ning  = jamoalarimizning
  asos   ko'plik egalik kelishik

BITTA OT ASOSI UCHUN (soddalashtirilgan):
  ko'plik:  2 (yo'q / -lar)
  egalik:   5 (yo'q / -im / -ingiz / -imiz / -i)
  kelishik: 7 (yo'q / -ning / -ni / -da / -dan / -ga / -dagi)
  -> 2 * 5 * 7 = 70 shakl, haqiqiy tilda yuzlab

OQIBAT:
  so'z lug'ati korpus bilan birga UZOQ o'sadi (Heaps qonuni)
  asoslar lug'ati tez to'yinadi
  test OOV ning katta qismi - imlo xatosi emas, TO'G'RI lekin kam shakl

YECHIMLAR:
  stemming/lemmatizatsiya - qo'shimchani kesish (qo'lda qoidalar, xatoli)
  subword tokenizatsiya   - BPE qo'shimchalarni O'ZI topadi (23.3-dars)
  belgi darajasi          - OOV yo'q, lekin ketma-ketlik uzun

import re
from collections import Counter

TOKEN_RE = re.compile(r"[a-z0-9]+(?:['-][a-z0-9]+)*|[^\w\s]")


def tokenla(matn):
    return TOKEN_RE.findall(normalla(matn))            # 23.1 dagi normalla


class Lugat:
    PAD, UNK = 0, 1

    def __init__(self, hujjatlar, min_chastota=2):
        sanoq = Counter(t for h in hujjatlar for t in h)
        sozlar = sorted((s for s, n in sanoq.items() if n >= min_chastota),
                        key=lambda s: (-sanoq[s], s))
        self.itos = ["<pad>", "<unk>"] + sozlar
        self.stoi = {s: i for i, s in enumerate(self.itos)}

    def kodla(self, tokenlar):
        return [self.stoi.get(t, self.UNK) for t in tokenlar]


lugat = Lugat(oquv_tokenlar)                           # faqat o'quvdan!
oov = sum(t not in lugat.stoi for h in test for t in h) / sum(map(len, test))

avval normallashtirish, keyin tokenlash
apostrof ichkarida - so'z qismi; tinish belgilari - alohida token
lug'at: <pad>=0, <unk>=1, faqat o'quvdan, deterministik tartib
min_chastota - lug'at hajmi va OOV savdosi
test OOV ni har doim o'lchang
o'zbekchada OOV ning asosiy manbai - qo'shimchalar birikmasi

"""So'z, belgi va regex tokenizatsiyasi: apostrofli so'zlar va tinish belgilari."""

import re
import unicodedata
from collections import Counter

import numpy as np

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    yoz = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yoz and "CYRILLIC" in yoz:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    """23.1-darsdagi quvur: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam."""
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    return re.sub(r"[0-9]+", "0", matn.lower())


SOZ_RE = re.compile(r"\w+")
TOKEN_RE = re.compile(r"[a-z0-9]+(?:['-][a-z0-9]+)*|[^\w\s]")


def tokenizatorlar():
    return {
        "split()": lambda t: t.split(),
        "regex \\w+": lambda t: SOZ_RE.findall(t),
        "normalla+\\w+": lambda t: SOZ_RE.findall(normalla(t)),
        "normalla+TOKEN_RE": lambda t: TOKEN_RE.findall(normalla(t)),
    }


def main() -> None:
    gap = "O\u2018qituvchi: \u00abBugun 2024-yilda 3 ta dars bor!\u00bb — dedi."
    print("=== 1. Bitta gap - to'rt xil tokenizator ===")
    print(f"  gap: {ascii(gap)}")
    for nom, f in tokenizatorlar().items():
        tok = f(gap)
        korinish = " | ".join(x if x.isascii() else ascii(x)[1:-1] for x in tok)
        print(f"  {nom:<18} {len(tok):>2} ta: {korinish}")
    belgilar = list(normalla(gap))
    print(f"  {'belgilar':<18} {len(belgilar):>2} ta: {ascii(belgilar[:14])[:70]} ...")

    print("\n=== 2. Apostrof qanday bo'linadi ===")
    for s in ["o'qituvchi", "o\u02bbqituvchi", "o\u2019qituvchi", "'yaxshi'",
              "ma'lumotlarimizning"]:
        print(f"  {ascii(s):<24} \\w+: {ascii(SOZ_RE.findall(s)):<28} "
              f"TOKEN_RE: {TOKEN_RE.findall(normalla(s))}")

    print("\n=== 3. Korpusda ===")
    rng = np.random.default_rng(0)
    korpus = [hujjat(rng)[1] for _ in range(3000)]
    print(f"  {'tokenizator':<18} {'tokenlar':>9} {'lug_at':>7} "
          f"{'1-harfli tok':>13}")
    for nom, f in tokenizatorlar().items():
        tok = [x for t in korpus for x in f(t)]
        bir = sum(len(x) == 1 and x.isalpha() for x in tok)
        print(f"  {nom:<18} {len(tok):>9} {len(set(tok)):>7} {bir:>13}")
    sanoq = Counter(x for t in korpus for x in SOZ_RE.findall(normalla(t)))
    print(f"  normalla+\\w+ dagi eng ko'p tokenlar: "
          f"{[s for s, _ in sanoq.most_common(6)]}")

    print("\n=== 4. Belgi va so'z tokenizatsiyasi ===")
    toza = [normalla(t) for t in korpus]
    soz_tok = [TOKEN_RE.findall(t) for t in toza]
    belgi_tok = [list(t) for t in toza]
    uz_soz = np.array([len(x) for x in soz_tok])
    uz_belgi = np.array([len(x) for x in belgi_tok])
    print(f"  lug'at: so'z {len({x for d in soz_tok for x in d})}, "
          f"belgi {len({x for d in belgi_tok for x in d})}")
    print(f"  hujjat uzunligi (o'rtacha): so'z {uz_soz.mean():.1f}, "
          f"belgi {uz_belgi.mean():.1f} ({uz_belgi.mean() / uz_soz.mean():.1f}x)")
    print(f"  eng uzun hujjat: so'z {uz_soz.max()}, belgi {uz_belgi.max()}")
    print("  ⭐ So'z tokenlari - katta lug'at, qisqa ketma-ketlik; belgilar - aksincha")


if __name__ == "__main__":
    main()

=== 1. Bitta gap - to'rt xil tokenizator ===
  gap: 'O\u2018qituvchi: \xabBugun 2024-yilda 3 ta dars bor!\xbb \u2014 dedi.'
  split()             9 ta: O\u2018qituvchi: | \xabBugun | 2024-yilda | 3 | ta | dars | bor!\xbb | \u2014 | dedi.
  regex \w+          10 ta: O | qituvchi | Bugun | 2024 | yilda | 3 | ta | dars | bor | dedi
  normalla+\w+       10 ta: o | qituvchi | bugun | 0 | yilda | 0 | ta | dars | bor | dedi
  normalla+TOKEN_RE  14 ta: o'qituvchi | : | \xab | bugun | 0-yilda | 0 | ta | dars | bor | ! | \xbb | \u2014 | dedi | .
  belgilar           50 ta: ['o', "'", 'q', 'i', 't', 'u', 'v', 'c', 'h', 'i', ':', ' ', '\xab', ' ...

=== 2. Apostrof qanday bo'linadi ===
  "o'qituvchi"             \w+: ['o', 'qituvchi']            TOKEN_RE: ["o'qituvchi"]
  'o\u02bbqituvchi'        \w+: ['o\u02bbqituvchi']          TOKEN_RE: ["o'qituvchi"]
  'o\u2019qituvchi'        \w+: ['o', 'qituvchi']            TOKEN_RE: ["o'qituvchi"]
  "'yaxshi'"               \w+: ['yaxshi']                   TOKEN_RE: ["'", 'yaxshi', "'"]
  "ma'lumotlarimizning"    \w+: ['ma', 'lumotlarimizning']   TOKEN_RE: ["ma'lumotlarimizning"]

=== 3. Korpusda ===
  tokenizator         tokenlar  lug_at  1-harfli tok
  split()                21458    6910             0
  regex \w+              25450    5584          1276
  normalla+\w+           26295    3352          1636
  normalla+TOKEN_RE      32269    3352             3
  normalla+\w+ dagi eng ko'p tokenlar: ['0', 'o', 'emas', 'ta', 'yilda', 'mln']

=== 4. Belgi va so'z tokenizatsiyasi ===
  lug'at: so'z 3352, belgi 32
  hujjat uzunligi (o'rtacha): so'z 10.8, belgi 68.0 (6.3x)
  eng uzun hujjat: so'z 27, belgi 172
  ⭐ So'z tokenlari - katta lug'at, qisqa ketma-ketlik; belgilar - aksincha

"""Lug'at qurish: faqat o'quvda, <pad>=0, <unk>=1, min chastota va OOV ulushi."""

import re
import unicodedata
from collections import Counter

import numpy as np
import torch

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    yoz = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yoz and "CYRILLIC" in yoz:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    """23.1-darsdagi quvur: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam."""
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    return re.sub(r"[0-9]+", "0", matn.lower())


def togri_shakllar():
    """Generator bera oladigan barcha to'g'ri (xatosiz) so'z shakllari."""
    shakllar = set(SIFATLAR) | {"emas", "bor", "yo'q", "foizga", "mln",
                                "0", "0-yilda", "0.0"}
    for asos in [a for otlar in OTLAR.values() for a in otlar]:
        for kop in ["", "lar"]:
            s1 = asos + kop
            unli = s1[-1] in UNLILAR
            for eg in [""] + (["si", "miz", "ngiz", "m"] if unli
                              else ["i", "imiz", "ingiz", "im"]):
                s2 = s1 + eg
                for k in ["", "ning", "ni", "da", "dan", "ga", "dagi"]:
                    if k == "ga" and s2[-1] in "kq":
                        k = "ka" if s2[-1] == "k" else "qa"
                    shakllar.add(s2 + k)
    for asos in FELLAR:
        shakllar |= {asos + "di", asos + "madi", asos + "maydi",
                     asos + ("ydi" if asos[-1] in UNLILAR else "adi")}
    return shakllar


TOKEN_RE = re.compile(r"[a-z0-9]+(?:['-][a-z0-9]+)*|[^\w\s]")


def tokenla(matn):
    return TOKEN_RE.findall(normalla(matn))


class Lugat:
    PAD, UNK = 0, 1

    def __init__(self, hujjatlar, min_chastota=1):
        sanoq = Counter(t for h in hujjatlar for t in h)
        # tartib: chastota kamayishi, teng bo'lsa alifbo - deterministik
        sozlar = sorted((s for s, n in sanoq.items() if n >= min_chastota),
                        key=lambda s: (-sanoq[s], s))
        self.itos = ["<pad>", "<unk>"] + sozlar
        self.stoi = {s: i for i, s in enumerate(self.itos)}

    def __len__(self):
        return len(self.itos)

    def kodla(self, tokenlar):
        return [self.stoi.get(t, self.UNK) for t in tokenlar]

    def dekodla(self, idlar):
        return [self.itos[i] for i in idlar if i != self.PAD]


def oov(lugat, hujjatlar):
    jami = sum(len(h) for h in hujjatlar)
    return sum(t not in lugat.stoi for h in hujjatlar for t in h) / jami


def main() -> None:
    rng = np.random.default_rng(0)
    korpus = [tokenla(hujjat(rng)[1]) for _ in range(3000)]
    tartib = np.random.default_rng(1).permutation(len(korpus))
    oquv = [korpus[i] for i in tartib[:2400]]
    test = [korpus[i] for i in tartib[2400:]]

    print("=== 1. Lug'at faqat o'quv qismidan ===")
    lugat = Lugat(oquv, min_chastota=2)
    print(f"  o'quv: {len(oquv)} hujjat, test: {len(test)} hujjat")
    print(f"  lug'at hajmi (min_chastota=2): {len(lugat)}")
    print(f"  birinchi 8 ta: {lugat.itos[:8]}")
    print(f"  stoi['<pad>']={lugat.stoi['<pad>']}, stoi['<unk>']={lugat.stoi['<unk>']}")

    print("\n=== 2. Kodlash va dekodlash ===")
    gap = tokenla("Jamoamizning O\u2018YINCHILARIMIZDAGI kuchi stadionda yutdi!")
    idlar = lugat.kodla(gap)
    print(f"  tokenlar: {gap}")
    print(f"  idlar:    {idlar}")
    print(f"  qaytarish: {lugat.dekodla(idlar)}")
    print(f"  <unk> ga tushgan: {[t for t, i in zip(gap, idlar) if i == Lugat.UNK]}")

    print("\n=== 3. Padding: bir xil uzunlikdagi batch ===")
    batch = [lugat.kodla(h) for h in test[:4]]
    L = max(len(b) for b in batch)
    X = torch.tensor([b + [Lugat.PAD] * (L - len(b)) for b in batch])
    niqob = X != Lugat.PAD
    print(f"  uzunliklar: {[len(b) for b in batch]} -> tensor {tuple(X.shape)}")
    print(f"  1-qator: {X[0].tolist()}")
    print(f"  niqob yig'indisi: {niqob.sum(1).tolist()}")

    print("\n=== 4. min_chastota: lug'at hajmi va OOV ===")
    print(f"  {'min_ch':>6} {'lug_at':>7} {'OOV o_quv':>10} {'OOV test':>9}")
    natija = {}
    for m in [1, 2, 3, 5, 10, 20]:
        lg = Lugat(oquv, min_chastota=m)
        natija[m] = oov(lg, test)
        print(f"  {m:>6} {len(lg):>7} {oov(lg, oquv):>10.2%} {natija[m]:>9.2%}")
    hamma = Lugat(oquv + test, min_chastota=1)
    print(f"  SIZIB KETISH: lug'at o'quv+testdan -> OOV test {oov(hamma, test):.2%}")
    print(f"  halol baho (min_ch=1): OOV test {natija[1]:.2%}")

    print("\n=== 5. Testdagi OOV tokenlar qanday? ===")
    lg = Lugat(oquv, min_chastota=1)
    yoq = Counter(t for h in test for t in h if t not in lg.stoi)
    print(f"  noyob OOV tokenlar: {len(yoq)}, jami: {sum(yoq.values())}")
    print(f"  namunalar: {sorted(yoq)[::max(1, len(yoq) // 6)][:6]}")
    togri = togri_shakllar()
    t_soni = sum(n for t, n in yoq.items() if t in togri)
    print(f"  ulardan to'g'ri so'z shakli: {t_soni / sum(yoq.values()):.0%}, "
          f"qolgani imlo xatosi va boshqalar")
    print("  ⭐ Lug'at faqat o'quvdan; test OOV - model haqiqatda ko'radigan holat")


if __name__ == "__main__":
    main()

=== 1. Lug'at faqat o'quv qismidan ===
  o'quv: 2400 hujjat, test: 600 hujjat
  lug'at hajmi (min_chastota=2): 1507
  birinchi 8 ta: ['<pad>', '<unk>', '.', '!', '?', '0', 'emas', 'mln']
  stoi['<pad>']=0, stoi['<unk>']=1

=== 2. Kodlash va dekodlash ===
  tokenlar: ['jamoamizning', "o'yinchilarimizdagi", 'kuchi', 'stadionda', 'yutdi', '!']
  idlar:    [1, 1, 1, 1368, 25, 3]
  qaytarish: ['<unk>', '<unk>', '<unk>', 'stadionda', 'yutdi', '!']
  <unk> ga tushgan: ['jamoamizning', "o'yinchilarimizdagi", 'kuchi']

=== 3. Padding: bir xil uzunlikdagi batch ===
  uzunliklar: [9, 9, 6, 3] -> tensor (4, 9)
  1-qator: [59, 657, 11, 3, 1, 1, 8, 36, 2]
  niqob yig'indisi: [9, 9, 6, 3]

=== 4. min_chastota: lug'at hajmi va OOV ===
  min_ch  lug_at  OOV o_quv  OOV test
       1    2953      0.00%     6.59%
       2    1507      5.63%    10.67%
       3    1011      9.50%    13.69%
       5     624     14.57%    18.37%
      10     248     24.16%    26.89%
      20     146     29.04%    30.60%
  SIZIB KETISH: lug'at o'quv+testdan -> OOV test 0.00%
  halol baho (min_ch=1): OOV test 6.59%

=== 5. Testdagi OOV tokenlar qanday? ===
  noyob OOV tokenlar: 401, jami: 435
  namunalar: ['0-yila', 'dasturchingizni', 'jamamdan', 'musobaqamda', 'serverimizga', 'tbbiyot']
  ulardan to'g'ri so'z shakli: 73%, qolgani imlo xatosi va boshqalar
  ⭐ Lug'at faqat o'quvdan; test OOV - model haqiqatda ko'radigan holat

"""Zipf qonuni: haqiqiy matnda (stdlib docstringlari) va sintetik korpusda."""

import ast
import pathlib
import re
import sysconfig
import unicodedata
import warnings
from collections import Counter

import numpy as np

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    yoz = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yoz and "CYRILLIC" in yoz:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    """23.1-darsdagi quvur: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam."""
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    return re.sub(r"[0-9]+", "0", matn.lower())


TOKEN_RE = re.compile(r"[a-z0-9]+(?:['-][a-z0-9]+)*")


def docstring_matni():
    """Python standart kutubxonasining yuqori darajali modullari docstringlari."""
    papka = pathlib.Path(sysconfig.get_paths()["stdlib"])
    qismlar = []
    for f in sorted(papka.glob("*.py")):
        try:
            with warnings.catch_warnings():
                warnings.simplefilter("ignore")
                daraxt = ast.parse(f.read_text(encoding="utf-8"))
        except (SyntaxError, UnicodeDecodeError):
            continue
        for tugun in ast.walk(daraxt):
            if isinstance(tugun, (ast.Module, ast.ClassDef, ast.FunctionDef,
                                  ast.AsyncFunctionDef)):
                d = ast.get_docstring(tugun)
                if d:
                    qismlar.append(d)
    return "\n".join(qismlar)


def zipf_tahlil(nom, tokenlar):
    sanoq = Counter(tokenlar)
    chastota = np.array(sorted(sanoq.values(), reverse=True), dtype=float)
    rang = np.arange(1, len(chastota) + 1)
    print(f"\n=== {nom} ===")
    print(f"  tokenlar: {len(tokenlar)}, lug'at: {len(sanoq)}")
    eng = sorted(sanoq.items(), key=lambda kv: (-kv[1], kv[0]))[:8]
    print(f"  eng ko'p: {[s for s, _ in eng]}")
    print(f"  {'rang':>6} {'chastota':>9} {'chastota x rang':>16}")
    for r in [1, 2, 5, 10, 50, 100, 500, 1000]:
        if r <= len(chastota):
            print(f"  {r:>6} {chastota[r - 1]:>9.0f} {chastota[r - 1] * r:>16.0f}")
    oraliq = (rang >= 10) & (rang <= min(1000, len(rang)))
    qiyalik = np.polyfit(np.log(rang[oraliq]), np.log(chastota[oraliq]), 1)[0]
    top10 = chastota[:10].sum() / chastota.sum()
    yakka = (chastota == 1).sum() / len(chastota)
    fr = chastota[oraliq] * rang[oraliq]
    tebranish = fr.max() / fr.min()
    print(f"  log-log qiyalik (rang 10..1000): {qiyalik:.2f}")
    print(f"  chastota x rang (rang 10..1000): min {fr.min():.0f}, "
          f"max {fr.max():.0f}, nisbat {tebranish:.1f}")
    print(f"  top-10 so'z ulushi: {top10:.1%}, bir marta uchraganlar: {yakka:.1%}")
    qamrov = [chastota[:n].sum() / chastota.sum() for n in (100, 1000)]
    print(f"  qamrov: top-100 so'z {qamrov[0]:.1%}, top-1000 so'z {qamrov[1]:.1%} "
          f"tokenlarni qoplaydi")
    return qiyalik, tebranish


def main() -> None:
    matn = docstring_matni()
    haqiqiy = TOKEN_RE.findall(matn.lower())
    q1 = zipf_tahlil("1. Haqiqiy matn: stdlib docstringlari (inglizcha)", haqiqiy)

    rng = np.random.default_rng(0)
    sintetik = [t for _ in range(12000)
                for t in TOKEN_RE.findall(normalla(hujjat(rng)[1]))]
    q2 = zipf_tahlil("2. Sintetik o'zbekcha korpus (12000 hujjat)", sintetik)

    print("\n=== 3. Xulosa ===")
    for nom, (q, t) in [("haqiqiy", q1), ("sintetik", q2)]:
        qiya = "qiyalik -1 ga yaqin" if abs(q + 1) < 0.25 else "qiyalik -1 dan uzoq"
        barqaror = "f x r barqaror" if t < 2.5 else "f x r barqaror EMAS"
        print(f"  {nom:<9} qiyalik {q:+.2f}, nisbat {t:.1f}: {qiya}, {barqaror}")
    print("  ⭐ Bir necha so'z juda ko'p, ko'pchilik so'z juda kam - lug'at dumi uzun")


if __name__ == "__main__":
    main()

=== 1. Haqiqiy matn: stdlib docstringlari (inglizcha) ===
  tokenlar: 136289, lug'at: 7513
  eng ko'p: ['the', 'a', 'is', 'to', 'of', 'and', 'if', 'for']
    rang  chastota  chastota x rang
       1      8166             8166
       2      4423             8846
       5      2734            13670
      10      1418            14180
      50       397            19850
     100       198            19800
     500        40            20000
    1000        18            18000
  log-log qiyalik (rang 10..1000): -0.99
  chastota x rang (rang 10..1000): min 13079, max 20340, nisbat 1.6
  top-10 so'z ulushi: 22.2%, bir marta uchraganlar: 36.1%
  qamrov: top-100 so'z 50.6%, top-1000 so'z 83.1% tokenlarni qoplaydi

=== 2. Sintetik o'zbekcha korpus (12000 hujjat) ===
  tokenlar: 86626, lug'at: 5979
  eng ko'p: ['0', 'emas', 'mln', 'bor', '0-yilda', "yo'q", 'foizga', 'zaif']
    rang  chastota  chastota x rang
       1      4518             4518
       2      2630             5260
       5      1482             7410
      10       572             5720
      50       399            19950
     100       140            14000
     500        29            14500
    1000        13            13000
  log-log qiyalik (rang 10..1000): -0.96
  chastota x rang (rang 10..1000): min 5720, max 20043, nisbat 3.5
  top-10 so'z ulushi: 18.8%, bir marta uchraganlar: 36.2%
  qamrov: top-100 so'z 50.1%, top-1000 so'z 83.6% tokenlarni qoplaydi

=== 3. Xulosa ===
  haqiqiy   qiyalik -0.99, nisbat 1.6: qiyalik -1 ga yaqin, f x r barqaror
  sintetik  qiyalik -0.96, nisbat 3.5: qiyalik -1 ga yaqin, f x r barqaror EMAS
  ⭐ Bir necha so'z juda ko'p, ko'pchilik so'z juda kam - lug'at dumi uzun

"""Agglutinativlik: o'zbekcha qo'shimchalar lug'atni qanday portlatadi."""

import re
import unicodedata
from collections import Counter, defaultdict

import numpy as np

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    yoz = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yoz and "CYRILLIC" in yoz:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    """23.1-darsdagi quvur: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam."""
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    return re.sub(r"[0-9]+", "0", matn.lower())


SOZ_RE = re.compile(r"[a-z]+(?:'[a-z]+)*")
QOSHIMCHALAR = sorted(["lar", "imiz", "ingiz", "miz", "ngiz", "im", "si", "i",
                       "ning", "ni", "da", "dan", "ga", "ka", "qa", "dagi"],
                      key=len, reverse=True)


def asoslar():
    return sorted({a for o in OTLAR.values() for a in o} | set(FELLAR)
                  | set(SIFATLAR), key=len, reverse=True)


def haqiqiy_asos(soz, asos_royxat):
    """Generator bilganidan: eng uzun mos keladigan asos (topilmasa None)."""
    for a in asos_royxat:
        if soz.startswith(a):
            return a
    return None


def sodda_stemmer(soz):
    """Oxirdan ma'lum qo'shimchalarni ochko'z tarzda kesadi."""
    for _ in range(4):
        for q in QOSHIMCHALAR:
            if soz.endswith(q) and len(soz) - len(q) >= 3:
                soz = soz[:-len(q)]
                break
        else:
            break
    return soz


def main() -> None:
    rng = np.random.default_rng(0)
    korpus = [SOZ_RE.findall(normalla(hujjat(rng)[1])) for _ in range(9000)]
    test, oquv = korpus[:1000], korpus[1000:]
    royxat = asoslar()
    asos_of = {}
    for h in korpus:
        for s in h:
            if s not in asos_of:
                asos_of[s] = haqiqiy_asos(s, royxat)

    print("=== 1. Bitta asos - ko'p shakl ===")
    print("  jamoa + lar + imiz + ning = jamoalarimizning")
    shakl = defaultdict(set)
    for h in oquv:
        for s in h:
            if asos_of[s]:
                shakl[asos_of[s]].add(s)
    for a in ["jamoa", "o'yinchi", "bank", "sog'liq"]:
        print(f"  {a:<9} korpusda {len(shakl[a]):>3} xil shakl, masalan: "
              f"{sorted(shakl[a], key=lambda s: (len(s), s))[-2:]}")
    otlar = [a for o in OTLAR.values() for a in o]
    print(f"  ot asoslari: o'rtacha {np.mean([len(shakl[a]) for a in otlar]):.1f} "
          f"shakl (nazariy maksimum 70)")

    print("\n=== 2. Lug'at o'sishi (Heaps qonuni) ===")
    test_sozlar = [s for h in test for s in h]
    print(f"  {'hujjat':>7} {'so_z shakllari':>15} {'asoslar':>8} "
          f"{'asossiz':>8} {'OOV shakl':>10} {'OOV asos':>9}")
    for n in [250, 500, 1000, 2000, 4000, 8000]:
        sozlar = {s for h in oquv[:n] for s in h}
        asos_set = {asos_of[s] or s for s in sozlar}
        haqiqiy = sum(asos_of[s] is None for s in sozlar)
        oov_s = np.mean([s not in sozlar for s in test_sozlar])
        oov_a = np.mean([(asos_of[s] or s) not in asos_set for s in test_sozlar])
        print(f"  {n:>7} {len(sozlar):>15} {len(asos_set) - haqiqiy:>8} "
              f"{haqiqiy:>8} {oov_s:>10.2%} {oov_a:>9.2%}")
    print("  asossiz - asosi tanilmagan so'zlar (asos ichidagi imlo xatosi,")
    print("  shuningdek emas, bor, yo'q kabi bir nechta yordamchi so'z)")

    print("\n=== 3. Sodda qo'shimcha kesuvchi (stemmer) ===")
    for s in ["jamoalarimizning", "o'yinchilarda", "banklarimizdagi", "o'yinchi"]:
        print(f"  {s:<18} -> {sodda_stemmer(s):<12} (haqiqiy: {asos_of.get(s) or haqiqiy_asos(s, royxat)})")
    ot_sozlar = sorted({s for h in oquv for s in h if asos_of[s] in otlar})
    togri = np.mean([sodda_stemmer(s) == asos_of[s] for s in ot_sozlar])
    print(f"  ot shakllarida to'g'ri asos: {togri:.1%} ({len(ot_sozlar)} shakl)")
    xato = Counter(asos_of[s] for s in ot_sozlar if sodda_stemmer(s) != asos_of[s])
    print(f"  eng ko'p adashgan asoslar: {[a for a, _ in xato.most_common(4)]}")
    stem_lug = {sodda_stemmer(s) for h in oquv for s in h}
    oov_stem = np.mean([sodda_stemmer(s) not in stem_lug for s in test_sozlar])
    print(f"  lug'at: shakllar {len({s for h in oquv for s in h})} -> "
          f"stem {len(stem_lug)}; test OOV {oov_stem:.2%}")
    print("  ⭐ Agglutinativ tilda so'z darajasidagi lug'at portlaydi - asos kichik")


if __name__ == "__main__":
    main()

=== 1. Bitta asos - ko'p shakl ===
  jamoa + lar + imiz + ning = jamoalarimizning
  jamoa     korpusda  63 xil shakl, masalan: ['jamoalaringizdagi', 'jamoalaringizning']
  o'yinchi  korpusda  62 xil shakl, masalan: ["o'yinchilaringizdagi", "o'yinchilaringizning"]
  bank      korpusda  67 xil shakl, masalan: ['banklaringizdagi', 'banklaringizning']
  sog'liq   korpusda  64 xil shakl, masalan: ["sog'liqlaringizdan", "sog'liqlaringzdagi"]
  ot asoslari: o'rtacha 63.6 shakl (nazariy maksimum 70)

=== 2. Lug'at o'sishi (Heaps qonuni) ===
   hujjat  so_z shakllari  asoslar  asossiz  OOV shakl  OOV asos
      250             820       95       45     31.68%     1.68%
      500            1274       95       74     22.53%     1.65%
     1000            1930       95      140     15.63%     1.58%
     2000            2768       95      275      9.91%     1.49%
     4000            3861       95      510      6.03%     1.37%
     8000            5151       95      907      3.65%     1.27%
  asossiz - asosi tanilmagan so'zlar (asos ichidagi imlo xatosi,
  shuningdek emas, bor, yo'q kabi bir nechta yordamchi so'z)

=== 3. Sodda qo'shimcha kesuvchi (stemmer) ===
  jamoalarimizning   -> jamoa        (haqiqiy: jamoa)
  o'yinchilarda      -> o'yinch      (haqiqiy: o'yinchi)
  banklarimizdagi    -> bank         (haqiqiy: bank)
  o'yinchi           -> o'yinch      (haqiqiy: o'yinchi)
  ot shakllarida to'g'ri asos: 82.2% (4068 shakl)
  eng ko'p adashgan asoslar: ['dori', "o'yinchi", 'sportchi', 'dasturchi']
  lug'at: shakllar 5151 -> stem 1250; test OOV 1.57%
  ⭐ Agglutinativ tilda so'z darajasidagi lug'at portlaydi - asos kichik

tokenlar = TOKEN_RE.findall(matn)                # U+2019, katta harf    # ⚠️
tokenlar = TOKEN_RE.findall(normalla(matn))                              # ✅

re.findall(r"\w+", "o'qituvchi")                 # ['o', 'qituvchi']     # ⚠️
re.findall(r"[a-z0-9]+(?:['-][a-z0-9]+)*", "o'qituvchi")                 # ✅

lugat = Lugat(oquv + test)                       # test OOV = 0          # ⚠️
lugat = Lugat(oquv)                                                      # ✅

itos = ["<pad>", "<unk>"] + list(set(tokenlar))  # har safar boshqa      # ⚠️
itos = ["<pad>", "<unk>"] + sorted(sanoq, key=lambda s: (-sanoq[s], s))  # ✅

itos = sozlar + ["<pad>"]                        # pad = oxirgi indeks   # ⚠️
itos = ["<pad>", "<unk>"] + sozlar               # pad = 0, unk = 1      # ✅

[lugat.stoi[t] for t in tokenlar]                # KeyError              # ⚠️
[lugat.stoi.get(t, Lugat.UNK) for t in tokenlar]                         # ✅

oov(lugat, oquv)                                 # min_ch=1 da doim 0    # ⚠️
oov(lugat, test)                                                         # ✅

1.  # "bor! bor. bor".split() - nechta noyob token?
2.  # re.findall(r"\w+", "ma'lumot") ?
3.  # TOKEN_RE.findall("'yaxshi'") ?
4.  # TOKEN_RE.findall("0-yilda") ?
5.  # Lugat da <pad> va <unk> indekslari?
6.  # min_chastota=1 da o'quv OOV ulushi?
7.  # lug'at o'quv+testdan qurilsa test OOV?
8.  # min_chastota oshsa lug'at va OOV qanday o'zgaradi?
9.  # Zipf qonunida f(r) * r ?
10. # log-log grafikda Zipf qiyaligi?
11. # jamoa + lar + imiz + ning = ?
12. # bitta ot asosidan (2 * 5 * 7) nechta shakl?

1.  tokenlar = re.findall(r"\w+", matn)

2.  lugat = Lugat(oquv + test, min_chastota=2)

3.  itos = ["<pad>", "<unk>"] + list(set(barcha_tokenlar))

4.  idlar = [lugat.stoi[t] for t in tokenlar]

5.  X = torch.tensor([lugat.kodla(h) for h in batch])   # uzunliklar har xil