23.1-dars: Matn ma'lumoti

KOMPYUTER UCHUN "HARF" YO'Q:
  Python str  = Unicode KOD NUQTALARI ketma-ketligi
  kod nuqtasi = butun son, odatda U+XXXX ko'rinishida yoziladi
    'o'  -> U+006F (111)
    'g'  -> U+0067 (103)
    "'"  -> U+0027 (39)

ASOSIY FUNKSIYALAR:
  ord('o')               -> 111        belgi -> son
  chr(111)               -> 'o'        son -> belgi
  len(s)                 -> kod nuqtalari soni (baytlar emas!)
  unicodedata.name(ch)   -> 'LATIN SMALL LETTER O'
  unicodedata.category   -> 'Ll' (kichik harf), 'Po' (tinish), 'Lm' ...

NIMA UCHUN BU MUHIM:
  s1 == s2  - kod nuqtalari bir xilmi? (ko'rinish EMAS)
  ekranda bir xil ko'ringan ikki satr teng bo'lmasligi mumkin
  set(), dict, Counter, lug'at - hammasi == ga tayanadi

KODLASH (encoding) = kod nuqtasini BAYTLARGA aylantirish qoidasi
  UTF-8 - bugungi standart, o'zgaruvchan uzunlik:
    U+0000..U+007F  (ASCII)          -> 1 bayt   'o' = 6f
    U+0080..U+07FF  (kirill, U+02BB) -> 2 bayt   kirill 'o' = d0 be
    U+0800..U+FFFF  (U+2019 va h.k.) -> 3 bayt   U+2019 = e2 80 99

  len("o'zbek")               = 6 belgi, 6 bayt
  len("o" + U+2019 + "zbek")  = 6 belgi, 8 bayt

MOJIBAKE (buzilgan matn):
  UTF-8 baytlarni boshqa kodlash (cp1252, latin-1) bilan o'qish
    e2 80 98 -> uchta "g'alati" belgi
  belgilar soni o'zgaradi, so'z tanib bo'lmas holga keladi
  tuzatish: noto'g'ri o'qilgan satrni qayta kodlab, to'g'ri o'qish

QOIDA:
  fayl o'qishda kodlashni ANIQ ko'rsating: open(f, encoding="utf-8")
  Windows da sukut kodlash boshqa bo'lishi mumkin (cp1251, cp1252)

RASMIY IMLO:
  o' va g' harflari uchun   - U+02BB (MODIFIER LETTER TURNED COMMA)
  tutuq belgisi (ta'lim)    - U+02BC (MODIFIER LETTER APOSTROPHE)

AMALDA UCHRAYDIGANLAR:
  U+0027  '   oddiy apostrof (klaviatura)      toifa Po (tinish)
  U+2018  ‘   chap bittalik qo'shtirnoq        toifa Pi
  U+2019  ’   o'ng bittalik qo'shtirnoq        toifa Pf (Word avto-almashtiradi)
  U+02BB      teskari vergul (rasmiy o')       toifa Lm (HARF!)
  U+02BC      apostrof-harf (rasmiy tutuq)     toifa Lm (HARF!)
  U+0060  `   gravis (grave accent)            toifa Sk

NATIJA:
  "o'zbek" so'zining 6 xil yozilishi = 6 xil satr = lug'atda 6 xil so'z
  regex \w U+02BB va U+02BC ni harf deb biladi, qolganlarini - yo'q
    -> bir xil so'z ba'zan 1 token, ba'zan 2 token (keyingi dars)

YECHIM:
  hamma variantni BITTA kanonik belgiga keltirish (masalan U+0027)
  qaysi biri kanonik - muhim emas, IZCHIL bo'lishi muhim

HOMOGLIF = ko'rinishi bir xil, kod nuqtasi boshqa belgi
  lotin  o  U+006F        kirill  o  U+043E
  lotin  a  U+0061        kirill  a  U+0430
  lotin  e  U+0065        kirill  e  U+0435
  lotin  c, p, x          kirill  s, r, h (ko'rinishi c, p, x)

QANDAY PAYDO BO'LADI:
  klaviatura tartibini almashtirishni unutish
  kirill matndan ko'chirib, qo'lda "tuzatish"
  OCR (skaner) xatosi

OQIBAT:
  "o'zbek" (lotin o) != "o'zbek" (kirill o) - baytlar ham farqli: 6 va 7
  qidiruv topmaydi, lug'atda dublikat, model uchun yangi so'z

TUZATISH QOIDASI:
  so'zda LOTIN va KIRILL harflari ARALASH bo'lsa -> homoglifni lotinga
  to'liq kirill so'zga TEGMANG - bu haqiqiy kirill matn bo'lishi mumkin
    sodda almashtirish: kirill "soha" -> "coxa" (noto'g'ri lotin)
    kirill matnni lotinga o'girish - alohida vazifa (transliteratsiya)

ANIQLASH:
  unicodedata.name(ch).split()[0] -> 'LATIN' yoki 'CYRILLIC'

BIR BELGI - IKKI XIL KODLASH:
  e-akut: U+00E9 (tayyor)  yoki  'e' + U+0301 (qo'shiluvchi urg'u)
  ko'rinish bir xil, satrlar TENG EMAS

TO'RT SHAKL:
  NFC  - tayyor (kompozit) shaklga yig'ish           caf+U+00E9 (4 belgi)
  NFD  - bo'laklarga ajratish                        cafe+U+0301 (5 belgi)
  NFKC - NFC + "moslik" belgilarini oddiylashtirish
  NFKD - NFD + "moslik" belgilarini oddiylashtirish

NFKC NIMA QILADI:
  keng (fullwidth) harf/raqam -> oddiy    U+FF4F -> o,  U+FF12 -> 2
  ligatura                    -> harflar  U+FB01 -> fi
  yuqori indeks               -> raqam    U+00B2 -> 2  (m2 - ma'no yo'qoldi!)
  bo'linmas bo'sh joy         -> bo'sh joy U+00A0 -> ' '
  doiradagi raqam             -> raqam    U+2460 -> 1

NFKC NIMA QILMAYDI:
  U+2018, U+2019, U+02BB, U+02BC, U+0060 - hammasi O'ZGARMAYDI
  kirill homogliflar - O'ZGARMAYDI
  -> o'zbekcha uchun o'z qoidalarimiz kerak

QAYSINI TANLASH:
  qidiruv, klassifikatsiya, lug'at uchun - odatda NFKC
  matnni asl ko'rinishda saqlash kerak bo'lsa - NFC

KICHIK HARF:
  s.lower()    - oddiy holatlar uchun
  s.casefold() - kuchliroq (nemischa ss -> ss)
  diqqat: turkcha I-nuqtali (U+0130).lower() -> 2 belgi (i + U+0307)
  lug'atni sezilarli kichraytiradi (gap boshidagi bosh harf, KATTA matn)

BO'SH JOY:
  s.split(" ")  - faqat U+0020 bo'yicha, bo'sh bo'laklar qoladi
  s.split()     - barcha Unicode bo'sh joylar (tab, U+00A0, \n)
  re.sub(r"\s+", " ", s) - ketma-ket bo'sh joylarni bittaga

RAQAMLAR:
  "2024-yilda", "1995-yilda" -> "0-yilda"  (lug'at kichrayadi)
  [0-9] faqat ASCII raqam; \d Unicode raqamlarini ham ushlaydi
  vazifa raqamga bog'liq bo'lsa (narx, sana) - almashtirmang

QUVUR (TARTIB MUHIM):
  1. bo'sh joylar       re.sub(r"\s+", " ")
  2. NFKC               keng raqamlar ASCII ga aylanadi
  3. apostrof -> '      o'z jadvalimiz
  4. kirill -> lotin    faqat aralash so'zlarda
  5. kichik harf
  6. raqam -> 0         NFKC dan KEYIN (aks holda keng raqamlar qoladi)

TEKSHIRUV:
  idempotentlik: normalla(normalla(x)) == normalla(x)
  noyob so'zlar sonini har qadamdan keyin o'lchang

import re
import unicodedata

ord("o"), chr(111), unicodedata.name("\u02bb")      # kod nuqtasi, nomi
len("o\u2019zbek"), len("o\u2019zbek".encode("utf-8"))  # 6 belgi, 8 bayt
open("korpus.txt", encoding="utf-8")                  # kodlashni aniq bering

APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")


def normalla(matn):
    matn = re.sub(r"\s+", " ", matn).strip()          # 1. bo'sh joy
    matn = unicodedata.normalize("NFKC", matn)        # 2. NFKC
    matn = APOSTROF_RE.sub("'", matn)                 # 3. apostrof
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))  # 4. kirill
    matn = matn.lower()                               # 5. kichik harf
    return re.sub(r"[0-9]+", "0", matn)               # 6. raqamlar


assert normalla(normalla(x)) == normalla(x)           # idempotentlik

str - kod nuqtalari; bytes - ularning UTF-8 ko'rinishi
"o'zbek" 6 xil apostrof bilan yoziladi -> bitta kanonik belgiga
homoglif: faqat aralash yozuvli so'zni tuzating
NFKC kerak, lekin apostrof va kirillni tuzatmaydi
quvur tartibi: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam
bitta funksiya - o'quv korpusi uchun ham, so'rov uchun ham

"""Matn - belgilar ketma-ketligi: kod nuqtalari, UTF-8 baytlari, o'xshash belgilar."""

import unicodedata

APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]


def belgi_jadvali(matn):
    for ch in matn:
        bayt = ch.encode("utf-8")
        print(f"  {ascii(ch):>10}  U+{ord(ch):04X}  {len(bayt)} bayt "
              f"{bayt.hex(' '):<9} {unicodedata.name(ch, '?')}")


def main() -> None:
    print("=== 1. Satr - kod nuqtalari ketma-ketligi ===")
    matn = "Bog' 2024"
    print(f"  matn: {matn!r}")
    print(f"  belgilar soni len(str): {len(matn)}")
    print(f"  UTF-8 baytlar soni:     {len(matn.encode('utf-8'))}")
    belgi_jadvali("Bog'")

    print("\n=== 2. Bitta so'z - olti xil yozilish ===")
    variantlar = ["o" + a + "zbek" for a in APOSTROFLAR]
    print(f"  {'yozilishi':<16} {'2-belgi':>8} {'belgi':>6} {'bayt':>5}  nomi")
    for v in variantlar:
        ch = v[1]
        print(f"  {ascii(v):<16} U+{ord(ch):04X} {len(v):>6} "
              f"{len(v.encode('utf-8')):>5}  {unicodedata.name(ch)}")
    print(f"  noyob satrlar soni: {len(set(variantlar))} / {len(variantlar)}")
    print(f"  'o' + U+0027 == 'o' + U+2019 ? "
          f"{variantlar[0] == variantlar[2]}")
    harf = [unicodedata.category(a) for a in APOSTROFLAR]
    print(f"  Unicode toifasi: {harf}")
    print("  (Lm - harf, Po/Pi/Pf - tinish, Sk - belgi)")

    print("\n=== 3. Kirill 'o' va lotin 'o' ===")
    lotin = "o'zbek"
    aralash = "\u043e'zbek"
    print(f"  ko'rinishda bir xil, lekin: {ascii(lotin)} va {ascii(aralash)}")
    print(f"  tengmi: {lotin == aralash}")
    belgi_jadvali(lotin[0] + aralash[0])
    for s in (lotin, aralash):
        yozuvlar = sorted({unicodedata.name(c).split()[0]
                           for c in s if c.isalpha()})
        print(f"  {ascii(s):<16} bayt={len(s.encode('utf-8'))}  "
              f"yozuvlar={yozuvlar}")
    gap = "men " + aralash + " tilini o'rganaman"
    print(f"  gapda {ascii(lotin)} qidirilsa: {lotin in gap}, "
          f"'zbek' qidirilsa: {'zbek' in gap}")

    print("\n=== 4. Kodlash xatosi (mojibake) ===")
    asl = "o\u2018zbek tili"
    bayt = asl.encode("utf-8")
    print(f"  UTF-8 baytlar: {bayt.hex(' ')}")
    for kodlash in ["utf-8", "cp1252", "latin-1"]:
        try:
            print(f"  {kodlash:<8} bilan o'qish -> {ascii(bayt.decode(kodlash))}")
        except UnicodeDecodeError as e:
            print(f"  {kodlash:<8} bilan o'qish -> {type(e).__name__}")
    buzuq = bayt.decode("cp1252")
    print(f"  buzilgan satr uzunligi: {len(buzuq)} (asli {len(asl)})")
    tiklangan = buzuq.encode("cp1252").decode("utf-8")
    print(f"  qayta tiklash: {tiklangan == asl}")
    try:
        bytes([0x6F, 0xE2, 0x80]).decode("utf-8")
    except UnicodeDecodeError as e:
        print(f"  kesilgan bayt: {type(e).__name__} ({e.reason})")
    print(f"  errors='replace': "
          f"{ascii(bytes([0x6F, 0xE2, 0x80]).decode('utf-8', errors='replace'))}")
    print("  ⭐ Belgi - kod nuqtasi, bayt esa uning kodlangan ko'rinishi")


if __name__ == "__main__":
    main()

=== 1. Satr - kod nuqtalari ketma-ketligi ===
  matn: "Bog' 2024"
  belgilar soni len(str): 9
  UTF-8 baytlar soni:     9
         'B'  U+0042  1 bayt 42        LATIN CAPITAL LETTER B
         'o'  U+006F  1 bayt 6f        LATIN SMALL LETTER O
         'g'  U+0067  1 bayt 67        LATIN SMALL LETTER G
         "'"  U+0027  1 bayt 27        APOSTROPHE

=== 2. Bitta so'z - olti xil yozilish ===
  yozilishi         2-belgi  belgi  bayt  nomi
  "o'zbek"         U+0027      6     6  APOSTROPHE
  'o\u2018zbek'    U+2018      6     8  LEFT SINGLE QUOTATION MARK
  'o\u2019zbek'    U+2019      6     8  RIGHT SINGLE QUOTATION MARK
  'o\u02bbzbek'    U+02BB      6     7  MODIFIER LETTER TURNED COMMA
  'o\u02bczbek'    U+02BC      6     7  MODIFIER LETTER APOSTROPHE
  'o`zbek'         U+0060      6     6  GRAVE ACCENT
  noyob satrlar soni: 6 / 6
  'o' + U+0027 == 'o' + U+2019 ? False
  Unicode toifasi: ['Po', 'Pi', 'Pf', 'Lm', 'Lm', 'Sk']
  (Lm - harf, Po/Pi/Pf - tinish, Sk - belgi)

=== 3. Kirill 'o' va lotin 'o' ===
  ko'rinishda bir xil, lekin: "o'zbek" va "\u043e'zbek"
  tengmi: False
         'o'  U+006F  1 bayt 6f        LATIN SMALL LETTER O
    '\u043e'  U+043E  2 bayt d0 be     CYRILLIC SMALL LETTER O
  "o'zbek"         bayt=6  yozuvlar=['LATIN']
  "\u043e'zbek"    bayt=7  yozuvlar=['CYRILLIC', 'LATIN']
  gapda "o'zbek" qidirilsa: False, 'zbek' qidirilsa: True

=== 4. Kodlash xatosi (mojibake) ===
  UTF-8 baytlar: 6f e2 80 98 7a 62 65 6b 20 74 69 6c 69
  utf-8    bilan o'qish -> 'o\u2018zbek tili'
  cp1252   bilan o'qish -> 'o\xe2\u20ac\u02dczbek tili'
  latin-1  bilan o'qish -> 'o\xe2\x80\x98zbek tili'
  buzilgan satr uzunligi: 13 (asli 11)
  qayta tiklash: True
  kesilgan bayt: UnicodeDecodeError (unexpected end of data)
  errors='replace': 'o\ufffd'
  ⭐ Belgi - kod nuqtasi, bayt esa uning kodlangan ko'rinishi

"""Unicode normallashtirish: NFC, NFKC, kichik harf va bo'sh joylar."""

import re
import unicodedata

APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]


def main() -> None:
    print("=== 1. Bir belgi - ikki xil kodlash (NFC va NFD) ===")
    tayyor = "caf\u00e9"
    qoshma = "cafe\u0301"
    print(f"  {ascii(tayyor)} len={len(tayyor)}   {ascii(qoshma)} len={len(qoshma)}")
    print(f"  xom holda tengmi:  {tayyor == qoshma}")
    for shakl in ["NFC", "NFD"]:
        a = unicodedata.normalize(shakl, tayyor)
        b = unicodedata.normalize(shakl, qoshma)
        print(f"  {shakl}: tengmi {a == b}, uzunlik {len(a)}")

    print("\n=== 2. NFKC: 'moslik' belgilarini oddiy shaklga keltiradi ===")
    misollar = ["\uff4f'zbek", "\uff12\uff10\uff12\uff14-yil", "\ufb01le",
                "10 m\u00b2", "narx\u00a0oshdi", "\u2460 bob"]
    print(f"  {'xom':<26} {'NFC':<26} {'NFKC':<16}")
    for m in misollar:
        print(f"  {ascii(m):<26} {ascii(unicodedata.normalize('NFC', m)):<26} "
              f"{ascii(unicodedata.normalize('NFKC', m)):<16}")
    print("  diqqat: m + U+00B2 -> m2 - daraja ma'nosi yo'qoldi")

    print("\n=== 3. NFKC apostrofni tuzatmaydi ===")
    variantlar = ["o" + a + "zbek" for a in APOSTROFLAR]
    nfkc = {unicodedata.normalize("NFKC", v) for v in variantlar}
    print(f"  NFKC dan keyin noyob: {len(nfkc)} / {len(variantlar)}")
    yagona = re.compile("[\u2018\u2019\u02bb\u02bc`]")
    ozimiz = {yagona.sub("'", v) for v in variantlar}
    print(f"  o'z jadvalimiz bilan noyob: {len(ozimiz)} -> {sorted(ozimiz)}")

    print("\n=== 4. Kichik harf: lower va casefold ===")
    for s in ["O'ZBEKISTON", "Stra\u00dfe", "\u0130stanbul", "O\u02bbZBEK"]:
        past = s.lower()
        print(f"  {ascii(s):<18} lower={ascii(past):<18} "
              f"casefold={ascii(s.casefold()):<18} len {len(s)}->{len(past)}")
    print("  I nuqtali (U+0130) lower() dan keyin 2 belgi bo'ladi")

    print("\n=== 5. Bo'sh joylar ===")
    matn = "narx  oshdi.\tBank\u00a0krediti\n yangi"
    print(f"  matn: {ascii(matn)}")
    print(f"  split(' '): {len(matn.split(' '))} bo'lak -> "
          f"{ascii(matn.split(' '))}")
    print(f"  split():    {len(matn.split())} bo'lak -> {matn.split()}")
    print(f"  re.sub(r'\\s+', ' '): {ascii(re.sub(r'\s+', ' ', matn))}")
    print(f"  '\\u00a0'.isspace() = {chr(0xA0).isspace()}")

    print("\n=== 6. Raqamlar ===")
    matn = "2024-yilda narx 15 foizga, 2023-yilda esa 7 foizga oshdi"
    print(f"  asl:      {matn}")
    print(f"  \\d+ -> 0: {re.sub(r'[0-9]+', '0', matn)}")
    print(f"  noyob so'zlar: {len(set(matn.split()))} -> "
          f"{len(set(re.sub(r'[0-9]+', '0', matn).split()))}")
    keng = chr(0xFF15)
    print(f"  U+FF15 (keng 5): isdigit={keng.isdigit()}, "
          f"[0-9] ushlaydi={bool(re.match('[0-9]', keng))}, "
          f"\\d ushlaydi={bool(re.match(r'\d', keng))}")
    print("  ⭐ NFKC - boshlanish, lekin o'zbekcha apostrof uchun o'z qoidamiz kerak")


if __name__ == "__main__":
    main()

=== 1. Bir belgi - ikki xil kodlash (NFC va NFD) ===
  'caf\xe9' len=4   'cafe\u0301' len=5
  xom holda tengmi:  False
  NFC: tengmi True, uzunlik 4
  NFD: tengmi True, uzunlik 5

=== 2. NFKC: 'moslik' belgilarini oddiy shaklga keltiradi ===
  xom                        NFC                        NFKC
  "\uff4f'zbek"              "\uff4f'zbek"              "o'zbek"
  '\uff12\uff10\uff12\uff14-yil' '\uff12\uff10\uff12\uff14-yil' '2024-yil'
  '\ufb01le'                 '\ufb01le'                 'file'
  '10 m\xb2'                 '10 m\xb2'                 '10 m2'
  'narx\xa0oshdi'            'narx\xa0oshdi'            'narx oshdi'
  '\u2460 bob'               '\u2460 bob'               '1 bob'
  diqqat: m + U+00B2 -> m2 - daraja ma'nosi yo'qoldi

=== 3. NFKC apostrofni tuzatmaydi ===
  NFKC dan keyin noyob: 6 / 6
  o'z jadvalimiz bilan noyob: 1 -> ["o'zbek"]

=== 4. Kichik harf: lower va casefold ===
  "O'ZBEKISTON"      lower="o'zbekiston"      casefold="o'zbekiston"      len 11->11
  'Stra\xdfe'        lower='stra\xdfe'        casefold='strasse'          len 6->6
  '\u0130stanbul'    lower='i\u0307stanbul'   casefold='i\u0307stanbul'   len 8->9
  'O\u02bbZBEK'      lower='o\u02bbzbek'      casefold='o\u02bbzbek'      len 6->6
  I nuqtali (U+0130) lower() dan keyin 2 belgi bo'ladi

=== 5. Bo'sh joylar ===
  matn: 'narx  oshdi.\tBank\xa0krediti\n yangi'
  split(' '): 4 bo'lak -> ['narx', '', 'oshdi.\tBank\xa0krediti\n', 'yangi']
  split():    5 bo'lak -> ['narx', 'oshdi.', 'Bank', 'krediti', 'yangi']
  re.sub(r'\s+', ' '): 'narx oshdi. Bank krediti yangi'
  '\u00a0'.isspace() = True

=== 6. Raqamlar ===
  asl:      2024-yilda narx 15 foizga, 2023-yilda esa 7 foizga oshdi
  \d+ -> 0: 0-yilda narx 0 foizga, 0-yilda esa 0 foizga oshdi
  noyob so'zlar: 9 -> 7
  U+FF15 (keng 5): isdigit=True, [0-9] ushlaydi=False, \d ushlaydi=True
  ⭐ NFKC - boshlanish, lekin o'zbekcha apostrof uchun o'z qoidamiz kerak

"""Sintetik o'zbekcha korpusni bosqichma-bosqich normallashtirish."""

import re
import unicodedata
from collections import Counter, defaultdict

import numpy as np

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


def togri_shakllar():
    """Generator bera oladigan barcha to'g'ri (xatosiz) so'z shakllari."""
    shakllar = set(SIFATLAR) | {"emas", "bor", "yo'q", "foizga", "mln",
                                "0", "0-yilda", "0.0"}
    for asos in [a for otlar in OTLAR.values() for a in otlar]:
        for kop in ["", "lar"]:
            s1 = asos + kop
            unli = s1[-1] in UNLILAR
            for eg in [""] + (["si", "miz", "ngiz", "m"] if unli
                              else ["i", "imiz", "ingiz", "im"]):
                s2 = s1 + eg
                for k in ["", "ning", "ni", "da", "dan", "ga", "dagi"]:
                    if k == "ga" and s2[-1] in "kq":
                        k = "ka" if s2[-1] == "k" else "qa"
                    shakllar.add(s2 + k)
    for asos in FELLAR:
        shakllar |= {asos + "di", asos + "madi", asos + "maydi",
                     asos + ("ydi" if asos[-1] in UNLILAR else "adi")}
    return shakllar


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    """Lotin harfi bor so'z ichidagi kirill 'egizaklarini' lotinga almashtiradi."""
    yozuvlar = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yozuvlar and "CYRILLIC" in yozuvlar:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


BOSQICHLAR = [
    ("0 xom: split(' ')", lambda t: t),
    ("1 bo'sh joy: split()", lambda t: re.sub(r"\s+", " ", t)),
    ("2 NFKC", lambda t: unicodedata.normalize("NFKC", t)),
    ("3 apostrof -> '", lambda t: APOSTROF_RE.sub("'", t)),
    ("4 kirill -> lotin", lambda t: " ".join(kirill_tuzat(s) for s in t.split(" "))),
    ("5 kichik harf", lambda t: t.lower()),
    ("6 raqam -> 0", lambda t: re.sub(r"[0-9]+", "0", t)),
]


def sozlar(matn):
    return [s.strip(".,!?") for s in matn.split(" ") if s.strip(".,!?")]


def main() -> None:
    rng = np.random.default_rng(0)
    korpus = [hujjat(rng)[1] for _ in range(3000)]
    print("=== 1. Korpus ===")
    print(f"  hujjatlar: {len(korpus)}, belgilar: {sum(map(len, korpus))}")
    for t in korpus[:3]:
        print(f"  {ascii(t)[:74]}")
    ascii_emas = Counter(c for t in korpus for c in t if ord(c) > 127)
    print(f"  ASCII bo'lmagan belgilar: "
          f"{', '.join(f'U+{ord(c):04X}:{n}' for c, n in ascii_emas.most_common(8))}")

    print("\n=== 2. Har bosqichdan keyin noyob so'zlar ===")
    matnlar = list(korpus)
    oldingi = None
    print(f"  {'bosqich':<22} {'tokenlar':>9} {'noyob':>7} {'kamaydi':>8} "
          f"{'o_zgargan hujjat':>17}")
    for nom, f in BOSQICHLAR:
        yangi = [f(t) for t in matnlar]
        ozgargan = sum(a != b for a, b in zip(matnlar, yangi))
        matnlar = yangi
        barcha = [s for t in matnlar for s in sozlar(t)]
        noyob = len(set(barcha))
        farq = "" if oldingi is None else f"{oldingi - noyob:>8}"
        print(f"  {nom:<22} {len(barcha):>9} {noyob:>7} {farq:>8} {ozgargan:>17}")
        if oldingi is None:
            boshlangich = noyob
        oldingi = noyob
    print(f"  jami: {boshlangich} -> {oldingi} "
          f"({1 - oldingi / boshlangich:.1%} kamaydi)")

    print("\n=== 3. Eng ko'p yozilish shakliga ega so'zlar ===")
    shakllar = defaultdict(set)
    for xom, toza in zip(korpus, matnlar):
        a = sozlar(re.sub(r"\s+", " ", xom))
        b = sozlar(toza)
        if len(a) == len(b):
            for x, y in zip(a, b):
                shakllar[y].add(x)
    harfli = {k: v for k, v in shakllar.items() if not re.search("[0-9]", k)}
    eng = sorted(harfli.items(), key=lambda kv: (-len(kv[1]), kv[0]))[:5]
    for toza, xomlar in eng:
        namuna = ", ".join(ascii(x) for x in sorted(xomlar)[:3])
        print(f"  {toza:<14} {len(xomlar):>2} shakl: {namuna}")

    print("\n=== 4. Normallashtirish tuzatmaydigan narsa ===")
    sanoq = Counter(s for t in matnlar for s in sozlar(t))
    yakka = [s for s, n in sanoq.items() if n == 1]
    print(f"  bir marta uchraydigan so'zlar: {len(yakka)} "
          f"({len(yakka) / len(sanoq):.1%} lug'at)")
    harfli = [s for s in sorted(yakka) if not re.search("[0-9]", s)]
    print(f"  misollar: {harfli[::150][:6]}")
    togri = togri_shakllar()
    xato = [s for s in yakka if s not in togri]
    print(f"  ulardan generator bera oladigan TO'G'RI shakl: "
          f"{len(yakka) - len(xato)}, imlo xatosi: {len(xato)} "
          f"({len(xato) / len(yakka):.1%})")
    if len(xato) / len(yakka) > 0.5:
        print("  yakka so'zlarning asosiy qismi - imlo xatolari")
    else:
        print("  yakka so'zlarning ko'pi - noyob qo'shimcha birikmalari")
    print("  ⭐ Normallashtirish bir so'zning turli YOZILISHINI birlashtiradi")


if __name__ == "__main__":
    main()

=== 1. Korpus ===
  hujjatlar: 3000, belgilar: 206501
  "Serverlarimiz qo'llaydi!\xa0Serverlarga platformada zaif emas! Telefonga
  "O\u2019yinchiga mag'lubiyatim arzon emas..."
  "Yuqudagi qo'lladi?\tParhez 53 foizga tekshiradi."
  ASCII bo'lmagan belgilar: U+2019:815, U+00A0:648, U+02BB:551, U+2018:531, U+0430:436, U+02BC:298, U+043E:183, U+0435:52

=== 2. Har bosqichdan keyin noyob so'zlar ===
  bosqich                 tokenlar   noyob  kamaydi  o_zgargan hujjat
  0 xom: split(' ')          20146    7149                          0
  1 bo'sh joy: split()       21458    6278      871              1459
  2 NFKC                     21458    6220       58                52
  3 apostrof -> '            21458    5490      730              1405
  4 kirill -> lotin          21458    5107      383               167
  5 kichik harf              21458    3580     1527              3000
  6 raqam -> 0               21458    3345      235              1008
  jami: 7149 -> 3345 (53.2% kamaydi)

=== 3. Eng ko'p yozilish shakliga ega so'zlar ===
  ko'rsatadi     17 shakl: "KO'RSATADI", 'KO\u02bcRSATADI', 'KO\u2018RSATADI'
  yo'q           15 shakl: "YO'Q", 'YO\u02bbQ', 'YO\u02bcQ'
  da'vo          14 shakl: "DA'VO", 'DA\u02bbVO', "Da'vo"
  ta'minladi     14 shakl: "TA'MINLADI", 'TA`MINLADI', 'TA\u02bbMINLADI'
  a'lo           13 shakl: "A'LO", "A'L\u041e", 'A`LO'

=== 4. Normallashtirish tuzatmaydigan narsa ===
  bir marta uchraydigan so'zlar: 1589 (47.5% lug'at)
  misollar: ['agloritmining', "da'vomizga", 'doringizdagi', 'interneltarimiz', 'kompaniymizdan', 'musobaqamizdagi']
  ulardan generator bera oladigan TO'G'RI shakl: 1097, imlo xatosi: 492 (31.0%)
  yakka so'zlarning ko'pi - noyob qo'shimcha birikmalari
  ⭐ Normallashtirish bir so'zning turli YOZILISHINI birlashtiradi

"""Normallashtirish chegaralari: ortiqcha birlashtirish, kirill, tartib, qidiruv."""

import re
import unicodedata
from collections import Counter, defaultdict

import numpy as np

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def yozuvlar(soz):
    return sorted({unicodedata.name(c).split()[0] for c in soz if c.isalpha()
                   and unicodedata.name(c).split()[0] in ("LATIN", "CYRILLIC")})


def kirill_tuzat(soz):
    if yozuvlar(soz) == ["CYRILLIC", "LATIN"]:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    matn = matn.lower()
    return re.sub(r"[0-9]+", "0", matn)


def sozlar(matn):
    return [s.strip(".,!?") for s in matn.split(" ") if s.strip(".,!?")]


def main() -> None:
    rng = np.random.default_rng(0)
    juftlar = [hujjat(rng) for _ in range(3000)]
    mavzular = [m for m, _ in juftlar]
    toza = [normalla(t) for _, t in juftlar]

    print("=== 1. Apostrofni birlashtirish va o'chirish ===")
    lugat = sorted({s for t in toza for s in sozlar(t)})
    guruh = defaultdict(set)
    for s in lugat:
        guruh[s.replace("'", "")].add(s)
    toqnash = {k: v for k, v in guruh.items() if len(v) > 1}
    print(f"  apostrof -> ' : lug'at {len(lugat)}")
    print(f"  apostrof o'chirilsa: lug'at {len(guruh)}, "
          f"{len(toqnash)} guruhda turli so'zlar qo'shilib ketdi")
    mavzu_soz = defaultdict(Counter)
    for m, t in zip(mavzular, toza):
        for s in sozlar(t):
            mavzu_soz[s][m] += 1
    for a, b in [("surat", "sur'at"), ("tarif", "ta'rif"), ("davo", "da'vo")]:
        qatorlar = []
        for s in (a, b):
            m, n = mavzu_soz[s].most_common(1)[0]
            qatorlar.append(f"{s!s:<7} {sum(mavzu_soz[s].values()):>3} marta, "
                            f"{m} {n / sum(mavzu_soz[s].values()):.0%}")
        birga = mavzu_soz[a] + mavzu_soz[b]
        m, n = birga.most_common(1)[0]
        print(f"  {qatorlar[0]:<34} | {qatorlar[1]}")
        print(f"    qo'shilsa: {sum(birga.values())} marta, eng ko'p mavzu "
              f"{m} atigi {n / sum(birga.values()):.0%}")

    print("\n=== 2. Kirill: faqat ARALASH so'zni tuzatish ===")
    sinov = {"aralash": "\u043e'yinchi",
             "kirill (ota)": "\u043e\u0442\u0430",
             "kirill (soha)": "\u0441\u043e\u0445\u0430"}
    for nom, s in sinov.items():
        sodda = "".join(LOTINGA.get(c, c) for c in s)
        ehtiyot = kirill_tuzat(s)
        print(f"  {nom}: xom {ascii(s)} {yozuvlar(s)}")
        print(f"    hammasini almashtirish -> {ascii(sodda):<10} {yozuvlar(sodda)}")
        print(f"    faqat aralash so'zni   -> {ascii(ehtiyot):<10} {yozuvlar(ehtiyot)}")
    print("  'soha' so'zi kirillda sodda almashtirishdan keyin lotincha")
    print("  ko'rinadi, lekin noto'g'ri: c-o-x-a (s-o-h-a emas)")
    aralash = sum(yozuvlar(s) == ["CYRILLIC", "LATIN"]
                  for _, t in juftlar for s in t.split())
    toliq = sum(yozuvlar(s) == ["CYRILLIC"] for _, t in juftlar for s in t.split())
    print(f"  korpusda: aralash so'zlar {aralash}, to'liq kirill so'zlar {toliq}")

    print("\n=== 3. Tartib va idempotentlik ===")
    ikki_marta = sum(normalla(t) != t for t in toza)
    print(f"  normalla(normalla(x)) != normalla(x): {ikki_marta} hujjat")

    def boshqa_tartib(matn):
        """Xuddi shu qadamlar, lekin raqam NFKC dan OLDIN almashtiriladi."""
        matn = re.sub(r"\s+", " ", matn).strip()
        matn = re.sub(r"[0-9]+", "0", matn)
        matn = unicodedata.normalize("NFKC", matn)
        matn = APOSTROF_RE.sub("'", matn)
        matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
        return matn.lower()

    farq = [i for i, (_, t) in enumerate(juftlar) if boshqa_tartib(t) != toza[i]]
    print(f"  raqamni NFKC dan OLDIN almashtirsak farq qiladi: {len(farq)} hujjat")
    if farq:
        i = farq[0]
        a = [s for s in sozlar(toza[i]) if re.search("[0-9]", s)]
        b = [s for s in sozlar(boshqa_tartib(juftlar[i][1]))
             if re.search("[0-9]", s)]
        print(f"    to'g'ri tartib: {a}  boshqa tartib: {b}")
    lug_a = {s for t in toza for s in sozlar(t)}
    lug_b = {s for _, t in juftlar for s in sozlar(boshqa_tartib(t))}
    print(f"  lug'at: to'g'ri tartibda {len(lug_a)}, boshqa tartibda {len(lug_b)}")

    print("\n=== 4. Qidiruv: so'rov ham xuddi shunday normallanadi ===")
    sorovlar = ["o'yinchi", "O\u2018YINCHI", "o\u02bbyinchi", "\u043e'yinchi"]
    print(f"  {'so_rov':<18} {'xom korpusda':>13} {'toza korpusda':>14}")
    for q in sorovlar:
        xom_n = sum(q in t for _, t in juftlar)
        toza_n = sum(normalla(q) in t for t in toza)
        print(f"  {ascii(q):<18} {xom_n:>13} {toza_n:>14}")
    print("  ⭐ Normallashtirish - bitta funksiya: o'quvda ham, so'rovda ham")


if __name__ == "__main__":
    main()

=== 1. Apostrofni birlashtirish va o'chirish ===
  apostrof -> ' : lug'at 3345
  apostrof o'chirilsa: lug'at 3237, 101 guruhda turli so'zlar qo'shilib ketdi
  surat    33 marta, texnologiya 100% | sur'at   32 marta, iqtisod 100%
    qo'shilsa: 65 marta, eng ko'p mavzu texnologiya atigi 51%
  tarif    41 marta, iqtisod 100%    | ta'rif   35 marta, texnologiya 100%
    qo'shilsa: 76 marta, eng ko'p mavzu iqtisod atigi 54%
  davo     36 marta, salomatlik 100% | da'vo    38 marta, iqtisod 100%
    qo'shilsa: 74 marta, eng ko'p mavzu iqtisod atigi 51%

=== 2. Kirill: faqat ARALASH so'zni tuzatish ===
  aralash: xom "\u043e'yinchi" ['CYRILLIC', 'LATIN']
    hammasini almashtirish -> "o'yinchi" ['LATIN']
    faqat aralash so'zni   -> "o'yinchi" ['LATIN']
  kirill (ota): xom '\u043e\u0442\u0430' ['CYRILLIC']
    hammasini almashtirish -> 'o\u0442a' ['CYRILLIC', 'LATIN']
    faqat aralash so'zni   -> '\u043e\u0442\u0430' ['CYRILLIC']
  kirill (soha): xom '\u0441\u043e\u0445\u0430' ['CYRILLIC']
    hammasini almashtirish -> 'coxa'     ['LATIN']
    faqat aralash so'zni   -> '\u0441\u043e\u0445\u0430' ['CYRILLIC']
  'soha' so'zi kirillda sodda almashtirishdan keyin lotincha
  ko'rinadi, lekin noto'g'ri: c-o-x-a (s-o-h-a emas)
  korpusda: aralash so'zlar 602, to'liq kirill so'zlar 0

=== 3. Tartib va idempotentlik ===
  normalla(normalla(x)) != normalla(x): 0 hujjat
  raqamni NFKC dan OLDIN almashtirsak farq qiladi: 52 hujjat
    to'g'ri tartib: ['0.0']  boshqa tartib: ['8.8']
  lug'at: to'g'ri tartibda 3345, boshqa tartibda 3404

=== 4. Qidiruv: so'rov ham xuddi shunday normallanadi ===
  so_rov              xom korpusda  toza korpusda
  "o'yinchi"                    15            155
  'O\u2018YINCHI'                0            155
  'o\u02bbyinchi'                7            155
  "\u043e'yinchi"                1            155
  ⭐ Normallashtirish - bitta funksiya: o'quvda ham, so'rovda ham

open("korpus.txt").read()                        # Windows: cp1251   # ⚠️
open("korpus.txt", encoding="utf-8").read()                          # ✅

matn.replace("\u2019", "'")                      # qolgan 4 tasi-chi? # ⚠️
re.sub("[\u2018\u2019\u02bb\u02bc`]", "'", matn)                     # ✅

re.sub("['\u2018\u2019\u02bb\u02bc`]", "", matn)  # surat == sur'at  # ⚠️
re.sub("[\u2018\u2019\u02bb\u02bc`]", "'", matn)  # bitta shaklga     # ✅

"".join(LOTINGA.get(c, c) for c in soz)          # kirill so'z buziladi # ⚠️
kirill_tuzat(soz)                                # faqat aralash so'z   # ✅

matn.split(" ")                                  # tab, U+00A0, "" qoladi # ⚠️
matn.split()                                                          # ✅

re.sub(r"[0-9]+", "0", matn); unicodedata.normalize("NFKC", matn)    # ⚠️
unicodedata.normalize("NFKC", matn); re.sub(r"[0-9]+", "0", matn)    # ✅

natija = [t for t in toza_korpus if sorov in t]                      # ⚠️
natija = [t for t in toza_korpus if normalla(sorov) in t]            # ✅

1.  # len("o\u2019zbek") va len("o\u2019zbek".encode("utf-8"))?
2.  # "o'zbek" == "o\u2018zbek" ?
3.  # kirill "o" UTF-8 da necha bayt?
4.  # unicodedata.category("\u02bb") ?
5.  # re.findall(r"\w+", "o\u02bbzbek o'zbek") ?
6.  # unicodedata.normalize("NFKC", "o\u2019zbek") ?
7.  # unicodedata.normalize("NFKC", "10 m\u00b2") ?
8.  # "narx\u00a0oshdi".split(" ") necha bo'lak?
9.  # "\u0130".lower() uzunligi?
10. # bool(re.match(r"\d", "\uff15")) va bool(re.match("[0-9]", "\uff15"))?
11. # apostrofni o'chirsak "surat" va "sur'at" nima bo'ladi?
12. # kirill "soha" ni sodda jadval bilan lotinga almashtirsak?

1.  matn = open("sharhlar.txt").read()

2.  matn = matn.replace("\u2019", "'")   # apostrofni normallashtirish

3.  matn = re.sub("['\u2018\u2019\u02bb\u02bc`]", "", matn)

4.  matn = matn.translate(str.maketrans("\u043e\u0430\u0435", "oae"))

5.  sozlar = matn.lower().split(" ")