23.4-dars: Bag-of-words va TF-IDF

G'OYA:
  lug'at: [bank, emas, kredit, narx, oshdi, ...]   (V ta so'z)
  hujjat -> V uzunlikdagi vektor, har ustun = so'z sanog'i

  "bank krediti arzon emas"  -> [1, 1, 1, 0, 0, ...]
  "narx oshdi narx"          -> [0, 0, 0, 2, 1, ...]

NIMA YO'QOLADI:
  so'z tartibi: "narx oshdi" == "oshdi narx"
  kontekst:     "yaxshi emas" -> {yaxshi: 1, emas: 1}
  -> n-gramlar tartibni QISMAN qaytaradi 2.3-bob

NIMA QOLADI:
  qaysi so'zlar va necha marta - mavzu uchun ko'pincha yetarli
  oddiy, tez, talqin qilinadigan (har ustun - aniq so'z)

VARIANTLAR:
  sanoq      CountVectorizer()
  binar      CountVectorizer(binary=True)   - bor/yo'q
  TF-IDF     TfidfVectorizer()              - og'irlangan (2.5)

STANDART:
  token_pattern = r"(?u)\b\w\w+\b"    - kamida 2 ta \w belgi
  lowercase = True

O'ZBEKCHADA NIMA BUZILADI:
  "o'yinchi" -> "yinchi"          'o' bitta harf, tashlab yuboriladi!
  "g'alaba"  -> "alaba"
  U+02BB bilan -> "o" + U+02BB + "yinchi" (harf, butun qoladi)
  -> bitta so'z uch xil ustunga tushadi, ma'nosiz bo'laklar paydo bo'ladi

YECHIM:
  CountVectorizer(
      preprocessor=normalla,                        # 23.1 quvuri
      token_pattern=r"[a-z0-9]+(?:['-][a-z0-9]+)*", # 23.2 naqshi
  )
  diqqat: preprocessor berilsa, lowercase u orqali bajarilmaydi
          -> kichik harfga o'tkazishni normalla o'zi qilishi kerak

MUQOBIL:
  tokenizer=o'z_funksiya  - to'liq nazorat (masalan BPE, 23.3)
  analyzer="char_wb", ngram_range=(2, 4) - belgi n-gramlari

N-GRAM:
  ngram_range=(1, 1)  - faqat so'zlar (unigram)
  ngram_range=(1, 2)  - so'zlar + ketma-ket juftlar (bigram)
    "sifati yaxshi emas" -> sifati, yaxshi, emas,
                            "sifati yaxshi", "yaxshi emas"
  lug'at keskin o'sadi (3000 hujjatda: 3348 -> 18650)

MIN_DF:
  min_df=2    - kamida 2 hujjatda uchragan (butun son - hujjatlar soni)
  min_df=0.01 - kamida 1% hujjatda (kasr - ulush)
  noyob n-gramlarni tozalaydi - lug'at bir necha barobar kichrayadi

MAX_DF:
  max_df=0.05 - 5% dan ko'p hujjatda uchraganlarni tashlash
  "avtomatik stop so'zlar": emas, bor, raqamlar
  diqqat: inkor so'zini ("emas") tashlash sentimentni buzadi

TARTIB:
  vektorizator ham model qismi - fit faqat o'quvda (23.2 bilan bir xil)

BAG-OF-WORDS MATRITSASI:
  shakl (n_hujjat, V), har qatorda atigi o'nlab nolmas qiymat
  zichlik odatda < 1%

CSR (Compressed Sparse Row) FORMATI:
  data    - nolmas qiymatlar
  indices - ularning ustun raqamlari
  indptr  - har qator qayerdan boshlanadi (n_hujjat + 1)
  xotira ~ nnz * (8 + 4) + n_hujjat * 4 bayt

ZICH VA SIYRAK:
  zich float64: n_hujjat * V * 8 bayt
  3000 x 18650 bigram matritsa: zich ~448 MB, CSR ~0.5 MB

QOIDALAR:
  .toarray() ni faqat kichik matritsada chaqiring
  sklearn modellari (LogisticRegression, LinearSVC, NB) CSR ni to'g'ridan-to'g'ri qabul qiladi
  StandardScaler(with_mean=False) - aks holda siyraklik yo'qoladi

TF (term frequency) - so'z hujjatda necha marta:
  tf(t, d) = sanoq(t, d)                   sublinear_tf=False (sukut)
  tf(t, d) = 1 + ln(sanoq(t, d))           sublinear_tf=True

IDF (inverse document frequency) - so'z qanchalik noyob:
  n  - hujjatlar soni, df(t) - t uchragan hujjatlar soni
  idf(t) = ln((1 + n) / (1 + df(t))) + 1   smooth_idf=True (sukut)
  idf(t) = ln(n / df(t)) + 1               smooth_idf=False
  "+1" lar: nolga bo'lish yo'q; hamma hujjatdagi so'z ham 0 emas

TF-IDF VA NORMA:
  w(t, d) = tf(t, d) * idf(t)
  v_d = w_d / ||w_d||_2                    norm="l2" (sukut)
  -> har hujjat vektori uzunligi 1

MA'NOSI:
  hujjatda ko'p, korpusda kam  -> katta og'irlik (mavzu so'zi)
  deyarli hamma hujjatda       -> kichik og'irlik (emas, bor, 0)
  sublinear: 2 marta uchragan so'z 2 emas, 1.69 barobar og'ir

KOSINUS O'XSHASHLIK:
  cos(a, b) = (a . b) / (||a|| * ||b||)
  1 - bir xil yo'nalish, 0 - umumiy so'z yo'q (manfiy bo'lmaydi: sanoqlar >= 0)
  L2 normallangan TF-IDF da: cos(a, b) = a . b

NEGA EVKLID EMAS:
  uzun hujjat - katta vektor; Evklid masofasi uzunlikka sezgir
  kosinus faqat YO'NALISHNI (so'zlar nisbatini) taqqoslaydi

EN YAQIN HUJJATLAR:
  S = cosine_similarity(X)             (n, n) matritsa
  o'zini chiqarib tashlash: diagonal = -inf
  har qator uchun eng katta k ta

BAHOLASH:
  precision@k = eng yaqin k qo'shnidan necha ulushi o'sha mavzuda
  vektorlash usullarini bir xil so'rovlarda juftlashgan farq + SE (18-qism)

SO'ROV:
  so'rov matni ham XUDDI SHU vektorizatordan o'tadi (transform, fit emas)

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

TOKEN_NAQSH = r"[a-z0-9]+(?:['-][a-z0-9]+)*"

tv = TfidfVectorizer(preprocessor=normalla,        # 23.1 quvuri
                     token_pattern=TOKEN_NAQSH,     # apostrof saqlanadi
                     ngram_range=(1, 2),            # inkor uchun bigram
                     min_df=2, max_df=0.5,          # shovqin va umumiy so'zlar
                     sublinear_tf=True)             # 1 + ln(tf)
X_oquv = tv.fit_transform(oquv_matnlar)            # fit faqat o'quvda
X_test = tv.transform(test_matnlar)

tv.get_feature_names_out(), tv.idf_               # lug'at va idf
X_oquv.shape, X_oquv.nnz                          # siyrak CSR

S = cosine_similarity(tv.transform([sorov]), X_oquv).ravel()
eng_yaqin = S.argsort()[::-1][:5]

hujjat -> lug'at uzunligidagi siyrak vektor (tartib yo'qoladi)
o'zbekcha: preprocessor=normalla, o'z token_pattern
n-gram (1, 2) - inkor va iboralar; lug'at bir necha barobar o'sadi
tf-idf = tf * (ln((1+n)/(1+df)) + 1), keyin L2 norma
kosinus o'xshashlik - qidiruv va eng yaqin hujjatlar
vektorizator ham model qismi: fit faqat o'quvda

"""CountVectorizer: apostrofni saqlovchi token_pattern, n-gram, min_df/max_df, siyrak matritsa."""

import re
import unicodedata

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    yoz = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yoz and "CYRILLIC" in yoz:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    """23.1-darsdagi quvur: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam."""
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    return re.sub(r"[0-9]+", "0", matn.lower())


TOKEN_NAQSH = r"[a-z0-9]+(?:['-][a-z0-9]+)*"


def xotira(X):
    return X.data.nbytes + X.indices.nbytes + X.indptr.nbytes


def main() -> None:
    rng = np.random.default_rng(0)
    korpus = [hujjat(rng)[1] for _ in range(3000)]

    print("=== 1. Standart token_pattern apostrofni buzadi ===")
    gaplar = ["O'yinchi g'alaba qozondi", "O\u2018yinchi g\u2018alaba qozondi",
              "O\u02bbyinchi g\u02bbalaba qozondi"]
    standart = CountVectorizer().fit(gaplar)
    print(f"  standart (\\b\\w\\w+\\b): {ascii(list(standart.get_feature_names_out()))}")
    ozimiz = CountVectorizer(preprocessor=normalla, token_pattern=TOKEN_NAQSH).fit(gaplar)
    print(f"  normalla + o'z naqsh:  {list(ozimiz.get_feature_names_out())}")

    print("\n=== 2. Korpusda: lug'at hajmi ===")
    variantlar = {
        "standart": CountVectorizer(),
        "normalla + naqsh": CountVectorizer(preprocessor=normalla,
                                            token_pattern=TOKEN_NAQSH),
        "+ bigram (1,2)": CountVectorizer(preprocessor=normalla,
                                          token_pattern=TOKEN_NAQSH,
                                          ngram_range=(1, 2)),
        "+ min_df=2": CountVectorizer(preprocessor=normalla, token_pattern=TOKEN_NAQSH,
                                      ngram_range=(1, 2), min_df=2),
        "+ max_df=0.05": CountVectorizer(preprocessor=normalla,
                                         token_pattern=TOKEN_NAQSH,
                                         ngram_range=(1, 2), min_df=2, max_df=0.05),
    }
    print(f"  {'variant':<18} {'lug_at':>7} {'nnz':>7} {'zichlik':>8}")
    matritsalar = {}
    for nom, v in variantlar.items():
        X = v.fit_transform(korpus)
        matritsalar[nom] = (v, X)
        print(f"  {nom:<18} {X.shape[1]:>7} {X.nnz:>7} "
              f"{X.nnz / (X.shape[0] * X.shape[1]):>8.3%}")
    v, X = matritsalar["standart"]
    buzuq = [s for s in v.get_feature_names_out() if s.startswith("yinchi")]
    print(f"  standartda 'yinchi' bilan boshlanuvchi bo'laklar: {len(buzuq)}")

    print("\n=== 3. min_df va max_df nimani olib tashladi ===")
    v, _ = matritsalar["+ max_df=0.05"]
    v_min, _ = matritsalar["+ min_df=2"]
    kop = sorted(set(v_min.get_feature_names_out()) - set(v.get_feature_names_out()))
    print(f"  max_df=0.05 olib tashlagan (5% dan ko'p hujjatda): {len(kop)} ta")
    print(f"  ulardan: {kop[:10]}")

    print("\n=== 4. Siyrak matritsa xotirasi ===")
    for nom in ["normalla + naqsh", "+ bigram (1,2)"]:
        _, X = matritsalar[nom]
        zich = X.shape[0] * X.shape[1] * 8
        print(f"  {nom:<18} shakl {X.shape}, CSR {xotira(X) / 1e6:.2f} MB, "
              f"zich float64 {zich / 1e6:.1f} MB ({zich / xotira(X):.0f}x)")
    _, X = matritsalar["normalla + naqsh"]
    qator = X[0]
    v, _ = matritsalar["normalla + naqsh"]
    nomlar = v.get_feature_names_out()
    print(f"  1-hujjat: {qator.nnz} ta nolmas ustun: "
          f"{[(str(nomlar[j]), int(c)) for j, c in zip(qator.indices, qator.data)][:6]}")
    print("  ⭐ Bag-of-words - so'z tartibini unutadi, faqat sanaydi; matritsa siyrak")


if __name__ == "__main__":
    main()

=== 1. Standart token_pattern apostrofni buzadi ===
  standart (\b\w\w+\b): ['alaba', 'g\u02bbalaba', 'o\u02bbyinchi', 'qozondi', 'yinchi']
  normalla + o'z naqsh:  ["g'alaba", "o'yinchi", 'qozondi']

=== 2. Korpusda: lug'at hajmi ===
  variant             lug_at     nnz  zichlik
  standart              4203   23216   0.184%
  normalla + naqsh      3348   21039   0.209%
  + bigram (1,2)       18650   39818   0.071%
  + min_df=2            2602   23770   0.305%
  + max_df=0.05         2590   19268   0.248%
  standartda 'yinchi' bilan boshlanuvchi bo'laklar: 43

=== 3. min_df va max_df nimani olib tashladi ===
  max_df=0.05 olib tashlagan (5% dan ko'p hujjatda): 12 ta
  ulardan: ['0', '0 0', '0 foizga', '0 mln', '0-yilda', 'bor', 'emas', 'foizga', 'mln', 'qimmat']

=== 4. Siyrak matritsa xotirasi ===
  normalla + naqsh   shakl (3000, 3348), CSR 0.26 MB, zich float64 80.4 MB (304x)
  + bigram (1,2)     shakl (3000, 18650), CSR 0.49 MB, zich float64 447.6 MB (914x)
  1-hujjat: 8 ta nolmas ustun: [('serverlarimiz', 1), ("qo'llaydi", 1), ('serverlarga', 1), ('platformada', 1), ('zaif', 1), ('emas', 1)]
  ⭐ Bag-of-words - so'z tartibini unutadi, faqat sanaydi; matritsa siyrak

"""TF-IDF qo'lda va sklearn bilan: smooth_idf, sublinear_tf, L2 norma."""

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

TOKEN_NAQSH = r"[a-z0-9]+(?:['-][a-z0-9]+)*"
HUJJATLAR = [
    "bank krediti arzon emas",
    "bank krediti qimmat , narx oshdi",
    "jamoa g'alaba qozondi , o'yinchi gol urdi gol",
    "o'yinchi jarohat oldi , jamoa yutqazdi",
    "yangi telefon arzon emas , lekin qulay",
    "telefon narx oshdi , bank emas",
]


def qolda_tfidf(C, smooth=True, sublinear=False):
    """C: (n_hujjat, n_soz) sanoqlar matritsasi (zich)."""
    n = C.shape[0]
    df = (C > 0).sum(axis=0)
    if smooth:
        idf = np.log((1 + n) / (1 + df)) + 1
    else:
        idf = np.log(n / df) + 1
    tf = C.astype(float)
    if sublinear:
        tf = np.where(C > 0, 1 + np.log(np.maximum(C, 1)), 0.0)
    W = tf * idf
    norma = np.sqrt((W ** 2).sum(axis=1, keepdims=True))
    return W / norma, idf, df


def main() -> None:
    cv = CountVectorizer(token_pattern=TOKEN_NAQSH)
    C = cv.fit_transform(HUJJATLAR).toarray()
    sozlar = cv.get_feature_names_out()
    n = len(HUJJATLAR)

    print("=== 1. Hujjat chastotasi (df) va IDF ===")
    W, idf, df = qolda_tfidf(C)
    tartib = np.argsort(-idf, kind="stable")
    print(f"  n = {n} hujjat")
    print(f"  {'so_z':<10} {'df':>3} {'idf (smooth)':>13}")
    for j in list(tartib[:3]) + list(tartib[-4:]):
        print(f"  {sozlar[j]:<10} {df[j]:>3} {idf[j]:>13.4f}")
    print("  idf = ln((1 + n) / (1 + df)) + 1 : kam hujjatdagi so'z - katta og'irlik")

    print("\n=== 2. Qo'lda va sklearn: bir xilmi? ===")
    print(f"  {'sozlama':<34} {'max |farq|':>11}")
    for smooth in (True, False):
        for sub in (False, True):
            W_q, _, _ = qolda_tfidf(C, smooth=smooth, sublinear=sub)
            tv = TfidfVectorizer(token_pattern=TOKEN_NAQSH, smooth_idf=smooth,
                                 sublinear_tf=sub)
            W_s = tv.fit_transform(HUJJATLAR).toarray()
            farq = np.abs(W_q - W_s).max()
            print(f"  smooth_idf={str(smooth):<5} sublinear_tf={str(sub):<5}"
                  f"{'':>6} {farq:>11.2e}")

    print("\n=== 3. Bitta hujjat ichida: sanoq -> tf-idf ===")
    i = 2
    W_s = TfidfVectorizer(token_pattern=TOKEN_NAQSH).fit_transform(HUJJATLAR).toarray()
    print(f"  hujjat: {HUJJATLAR[i]!r}")
    print(f"  {'so_z':<10} {'sanoq':>6} {'idf':>7} {'tf*idf':>8} {'L2 dan keyin':>13}")
    for j in np.nonzero(C[i])[0]:
        print(f"  {sozlar[j]:<10} {C[i, j]:>6} {idf[j]:>7.4f} "
              f"{C[i, j] * idf[j]:>8.4f} {W_s[i, j]:>13.4f}")
    print(f"  vektor uzunligi (L2): {np.linalg.norm(W_s[i]):.6f}")

    print("\n=== 4. sublinear_tf: takrorlangan so'z ta'siri ===")
    j = list(sozlar).index("gol")
    for sub in (False, True):
        W = TfidfVectorizer(token_pattern=TOKEN_NAQSH, sublinear_tf=sub
                            ).fit_transform(HUJJATLAR).toarray()
        k = list(sozlar).index("jamoa")
        print(f"  sublinear_tf={str(sub):<5}: 'gol' (2 marta) {W[i, j]:.4f}, "
              f"'jamoa' (1 marta) {W[i, k]:.4f}, nisbat {W[i, j] / W[i, k]:.3f}")
    print(f"  tf: 2 -> 1 + ln 2 = {1 + np.log(2):.4f}")
    print("  ⭐ TF-IDF: ko'p hujjatda uchraydigan so'z pasayadi, noyob so'z ko'tariladi")


if __name__ == "__main__":
    main()

=== 1. Hujjat chastotasi (df) va IDF ===
  n = 6 hujjat
  so_z        df  idf (smooth)
  g'alaba      1        2.2528
  gol          1        2.2528
  jarohat      1        2.2528
  oshdi        2        1.8473
  telefon      2        1.8473
  bank         3        1.5596
  emas         3        1.5596
  idf = ln((1 + n) / (1 + df)) + 1 : kam hujjatdagi so'z - katta og'irlik

=== 2. Qo'lda va sklearn: bir xilmi? ===
  sozlama                             max |farq|
  smooth_idf=True  sublinear_tf=False          1.11e-16
  smooth_idf=True  sublinear_tf=True           1.11e-16
  smooth_idf=False sublinear_tf=False          0.00e+00
  smooth_idf=False sublinear_tf=True           0.00e+00

=== 3. Bitta hujjat ichida: sanoq -> tf-idf ===
  hujjat: "jamoa g'alaba qozondi , o'yinchi gol urdi gol"
  so_z        sanoq     idf   tf*idf  L2 dan keyin
  g'alaba         1  2.2528   2.2528        0.3462
  gol             2  2.2528   4.5055        0.6923
  jamoa           1  1.8473   1.8473        0.2839
  o'yinchi        1  1.8473   1.8473        0.2839
  qozondi         1  2.2528   2.2528        0.3462
  urdi            1  2.2528   2.2528        0.3462
  vektor uzunligi (L2): 1.000000

=== 4. sublinear_tf: takrorlangan so'z ta'siri ===
  sublinear_tf=False: 'gol' (2 marta) 0.6923, 'jamoa' (1 marta) 0.2839, nisbat 2.439
  sublinear_tf=True : 'gol' (2 marta) 0.6305, 'jamoa' (1 marta) 0.3054, nisbat 2.065
  tf: 2 -> 1 + ln 2 = 1.6931
  ⭐ TF-IDF: ko'p hujjatda uchraydigan so'z pasayadi, noyob so'z ko'tariladi

"""Kosinus o'xshashlik bilan eng yaqin hujjatlar: sanoq va TF-IDF."""

import re
import unicodedata

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

OTLAR = {
    "sport": ["jamoa", "o'yinchi", "murabbiy", "stadion", "o'yin", "g'alaba",
              "chempionat", "darvoza", "to'p", "musobaqa", "muxlis", "gol",
              "hakam", "sportchi", "mag'lubiyat", "klub"],
    "iqtisod": ["bank", "narx", "soliq", "kompaniya", "bozor", "kredit",
                "eksport", "sarmoya", "byudjet", "daromad", "savdo",
                "tadbirkor", "valyuta", "tarif", "sur'at", "da'vo"],
    "texnologiya": ["telefon", "dastur", "kompyuter", "internet", "ilova",
                    "ma'lumot", "tarmoq", "qurilma", "server", "dasturchi",
                    "robot", "platforma", "algoritm", "surat", "ta'rif",
                    "xavfsizlik"],
    "salomatlik": ["shifokor", "kasalxona", "dori", "bemor", "sog'liq",
                   "vaksina", "parhez", "uyqu", "yurak", "qon", "mashq",
                   "vitamin", "klinika", "davo", "tibbiyot", "og'riq"],
}
FELLAR = ["osh", "kamay", "o'zgar", "yaxshilan", "tayyorla", "ko'rsat",
          "boshla", "ishla", "yut", "qo'lla", "tekshir", "ta'minla",
          "o'rgan", "sot", "yoq"]
SIFATLAR = ["yangi", "katta", "muhim", "tez", "yaxshi", "kuchli", "zaif",
            "arzon", "qimmat", "sog'lom", "xavfli", "a'lo", "yuqori", "past",
            "mashhur", "qulay"]
UNLILAR = set("aeiou")


def ot_shakli(asos, rng):
    """Ot + ko'plik + egalik + kelishik (soddalashtirilgan qoidalar)."""
    s = asos
    if rng.random() < 0.3:
        s += "lar"
    if rng.random() < 0.3:
        unli = s[-1] in UNLILAR
        s += rng.choice(["si", "miz", "ngiz", "m"] if unli
                        else ["i", "imiz", "ingiz", "im"])
    if rng.random() < 0.6:
        k = str(rng.choice(["ning", "ni", "da", "dan", "ga", "dagi"]))
        if k == "ga" and s[-1] in "kq":
            k = "ka" if s[-1] == "k" else "qa"
        s += k
    return s


def fel_shakli(asos, rng):
    """Fe'l: o'tgan/hozirgi zamon, ba'zan -ma- inkori."""
    inkor = rng.random() < 0.25
    if rng.random() < 0.5:
        return asos + ("madi" if inkor else "di")
    if inkor:
        return asos + "maydi"
    return asos + ("ydi" if asos[-1] in UNLILAR else "adi")


APOSTROFLAR = ["'", "\u2018", "\u2019", "\u02bb", "\u02bc", "`"]
APOSTROF_P = [0.45, 0.12, 0.2, 0.13, 0.05, 0.05]
KIRILL = {"o": "\u043e", "a": "\u0430", "e": "\u0435", "c": "\u0441",
          "p": "\u0440", "x": "\u0445"}


def jumla(mavzu, rng):
    otlar = OTLAR[mavzu]
    sozlar = [ot_shakli(str(rng.choice(otlar)), rng)]
    if rng.random() < 0.5:
        sozlar.append(ot_shakli(str(rng.choice(otlar)), rng))
    tur = rng.random()
    if tur < 0.3:
        sozlar.append(str(rng.choice(SIFATLAR)))
        if rng.random() < 0.3:
            sozlar.append("emas")
        return " ".join(sozlar)
    if tur < 0.4:
        sozlar.append(str(rng.choice(["bor", "yo'q"])))
        return " ".join(sozlar)
    if rng.random() < 0.25:
        son = int(rng.integers(2, 100))
        sozlar.append(str(rng.choice([f"{son} foizga", f"{1990 + son % 35}-yilda",
                                      f"{son}.{son % 10} mln"])))
    sozlar.append(fel_shakli(str(rng.choice(FELLAR)), rng))
    return " ".join(sozlar)


def buz(soz, rng):
    """Imlo xatosi: harf tushib qolishi, takrorlanishi yoki o'rin almashishi."""
    if len(soz) < 4:
        return soz
    i = int(rng.integers(1, len(soz) - 1))
    tur = rng.integers(3)
    if tur == 0:
        return soz[:i] + soz[i + 1:]
    if tur == 1:
        return soz[:i] + soz[i] + soz[i:]
    return soz[:i - 1] + soz[i] + soz[i - 1] + soz[i + 1:]


def hujjat(rng, mavzu=None):
    if mavzu is None:
        mavzu = str(rng.choice(sorted(OTLAR)))
    jumlalar = [jumla(mavzu, rng) for _ in range(int(rng.integers(1, 5)))]
    matn = ""
    for j in jumlalar:
        sozlar = [buz(s, rng) if rng.random() < 0.03 else s for s in j.split()]
        j = " ".join(sozlar)
        matn += j[0].upper() + j[1:] + str(rng.choice([".", ".", ".", "!", "?", "..."]))
        matn += str(rng.choice([" ", " ", " ", " ", "  ", "\u00a0", "\t"]))
    matn = matn.strip()
    odat = int(rng.choice(len(APOSTROFLAR), p=APOSTROF_P))
    belgilar = []
    for ch in matn:
        if ch == "'":
            k = odat if rng.random() < 0.85 else int(rng.integers(len(APOSTROFLAR)))
            ch = APOSTROFLAR[k]
        belgilar.append(ch)
    matn = "".join(belgilar)
    if rng.random() < 0.06:
        matn = "".join(KIRILL[c] if c in KIRILL and rng.random() < 0.3 else c
                       for c in matn)
    if rng.random() < 0.04:
        matn = matn.upper()
    if rng.random() < 0.05:
        matn = matn.translate({ord(str(d)): 0xFF10 + d for d in range(10)})
    return mavzu, matn


APOSTROF_RE = re.compile("[\u2018\u2019\u02bb\u02bc`]")
LOTINGA = {v: k for k, v in KIRILL.items()}
LOTINGA.update({v.upper(): k.upper() for k, v in KIRILL.items()})


def kirill_tuzat(soz):
    yoz = {unicodedata.name(c).split()[0] for c in soz if c.isalpha()}
    if "LATIN" in yoz and "CYRILLIC" in yoz:
        return "".join(LOTINGA.get(c, c) for c in soz)
    return soz


def normalla(matn):
    """23.1-darsdagi quvur: bo'sh joy, NFKC, apostrof, kirill, kichik harf, raqam."""
    matn = re.sub(r"\s+", " ", matn).strip()
    matn = unicodedata.normalize("NFKC", matn)
    matn = APOSTROF_RE.sub("'", matn)
    matn = " ".join(kirill_tuzat(s) for s in matn.split(" "))
    return re.sub(r"[0-9]+", "0", matn.lower())


TOKEN_NAQSH = r"[a-z0-9]+(?:['-][a-z0-9]+)*"


def aniqlik_k(S, mavzu, k=5):
    """Har hujjat uchun eng yaqin k ta qo'shnining necha ulushi o'sha mavzudan."""
    S = S.copy()
    np.fill_diagonal(S, -np.inf)
    qoshni = np.argsort(-S, axis=1, kind="stable")[:, :k]
    return (mavzu[qoshni] == mavzu[:, None]).mean(axis=1)


def main() -> None:
    rng = np.random.default_rng(0)
    juftlar = [hujjat(rng) for _ in range(2000)]
    matnlar = [t for _, t in juftlar]
    mavzu = np.array([m for m, _ in juftlar])
    umumiy = dict(preprocessor=normalla, token_pattern=TOKEN_NAQSH)
    vektorlar = {
        "sanoq (Count)": CountVectorizer(**umumiy),
        "binar": CountVectorizer(binary=True, **umumiy),
        "TF-IDF": TfidfVectorizer(**umumiy),
        "TF-IDF sublinear": TfidfVectorizer(sublinear_tf=True, **umumiy),
        "standart TF-IDF": TfidfVectorizer(),
    }

    print("=== 1. Kosinus o'xshashlik: bitta so'rov ===")
    tv = TfidfVectorizer(**umumiy).fit(matnlar)
    X = tv.transform(matnlar)
    sorov = "Bankdagi KREDIT narxi o\u2018sdi"
    s = cosine_similarity(tv.transform([sorov]), X).ravel()
    print(f"  so'rov: {ascii(sorov)} -> {normalla(sorov)!r}")
    for i in np.argsort(-s, kind="stable")[:3]:
        print(f"  {s[i]:.3f} [{mavzu[i]}] {ascii(normalla(matnlar[i]))[:62]}")

    print("\n=== 2. Eng yaqin 5 qo'shni o'sha mavzudanmi? ===")
    natija = {}
    for nom, v in vektorlar.items():
        Xv = v.fit_transform(matnlar)
        natija[nom] = aniqlik_k(cosine_similarity(Xv), mavzu)
        print(f"  {nom:<18} precision@5 = {natija[nom].mean():.4f}")

    print("\n=== 3. Juftlashgan farq (2000 so'rov bo'yicha) ===")
    asos = natija["sanoq (Count)"]
    for nom in ["binar", "TF-IDF", "TF-IDF sublinear", "standart TF-IDF"]:
        farq = natija[nom] - asos
        se = farq.std(ddof=1) / np.sqrt(len(farq))
        if abs(farq.mean()) <= 2 * se:
            xulosa = "sezilarli farq yo'q"
        elif farq.mean() > 0:
            xulosa = "sanoqdan sezilarli yaxshi"
        else:
            xulosa = "sanoqdan sezilarli yomon"
        print(f"  {nom:<18} - sanoq: {farq.mean():+.4f} (SE {se:.4f}) {xulosa}")

    farq = natija["TF-IDF"] - natija["standart TF-IDF"]
    se = farq.std(ddof=1) / np.sqrt(len(farq))
    holat = "sezilarli" if abs(farq.mean()) > 2 * se else "sezilarli emas"
    print(f"  normalla + naqsh foydasi (TF-IDF - standart TF-IDF): "
          f"{farq.mean():+.4f} (SE {se:.4f}) {holat}")

    print("\n=== 4. Nega: eng ko'p hujjatdagi so'zlarning og'irligi ===")
    Xc = CountVectorizer(**umumiy).fit(matnlar)
    nomlar = tv.get_feature_names_out()
    df = np.asarray((X > 0).sum(axis=0)).ravel()
    for j in np.argsort(-df, kind="stable")[:5]:
        print(f"  {nomlar[j]:<8} {df[j] / len(matnlar):>6.1%} hujjatda, "
              f"idf {tv.idf_[j]:.3f} (maks {tv.idf_.max():.3f})")
    print(f"  Count va TF-IDF lug'ati bir xil ({len(Xc.vocabulary_)} so'z) - "
          f"farq faqat og'irlikda")
    print("  ⭐ Kosinus uzunlikka befarq; TF-IDF umumiy so'zlarni pasaytiradi")


if __name__ == "__main__":
    main()

=== 1. Kosinus o'xshashlik: bitta so'rov ===
  so'rov: 'Bankdagi KREDIT narxi o\u2018sdi' -> "bankdagi kredit narxi o'sdi"
  0.338 [iqtisod] 'kredit sotdi!'
  0.330 [iqtisod] "narxda tarifdagi 0-yilda o'rganadi. narxi eksportlar xavfli."
  0.309 [iqtisod] "da'voga ta'minladi! kreditga bankdagi o'zgardi."

=== 2. Eng yaqin 5 qo'shni o'sha mavzudanmi? ===
  sanoq (Count)      precision@5 = 0.3993
  binar              precision@5 = 0.4152
  TF-IDF             precision@5 = 0.8414
  TF-IDF sublinear   precision@5 = 0.8610
  standart TF-IDF    precision@5 = 0.7774

=== 3. Juftlashgan farq (2000 so'rov bo'yicha) ===
  binar              - sanoq: +0.0159 (SE 0.0036) sanoqdan sezilarli yaxshi
  TF-IDF             - sanoq: +0.4421 (SE 0.0059) sanoqdan sezilarli yaxshi
  TF-IDF sublinear   - sanoq: +0.4617 (SE 0.0059) sanoqdan sezilarli yaxshi
  standart TF-IDF    - sanoq: +0.3781 (SE 0.0064) sanoqdan sezilarli yaxshi
  normalla + naqsh foydasi (TF-IDF - standart TF-IDF): +0.0640 (SE 0.0049) sezilarli

=== 4. Nega: eng ko'p hujjatdagi so'zlarning og'irligi ===
  0         23.3% hujjatda, idf 2.455 (maks 7.908)
  emas      19.5% hujjatda, idf 2.633 (maks 7.908)
  mln       12.4% hujjatda, idf 3.084 (maks 7.908)
  0-yilda   12.2% hujjatda, idf 3.096 (maks 7.908)
  foizga    11.9% hujjatda, idf 3.121 (maks 7.908)
  Count va TF-IDF lug'ati bir xil (2753 so'z) - farq faqat og'irlikda
  ⭐ Kosinus uzunlikka befarq; TF-IDF umumiy so'zlarni pasaytiradi

"""N-gram inkorni qanday ushlaydi: "yaxshi" va "yaxshi emas"."""

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.model_selection import StratifiedKFold

TOKEN_NAQSH = r"[a-z0-9]+(?:['-][a-z0-9]+)*"
JIHATLAR = ["sifati", "narxi", "dizayni", "batareyasi", "ekrani", "xizmati",
            "yetkazish", "ovozi"]
IJOBIY = ["yaxshi", "a'lo", "zo'r", "qulay", "chiroyli", "tez"]
SALBIY = ["yomon", "sifatsiz", "noqulay", "xunuk", "sekin", "qimmat"]


def gap(qutb, rng):
    """qutb=+1 ijobiy, -1 salbiy gap. Inkor ma'noni teskari qiladi."""
    jihat = str(rng.choice(JIHATLAR))
    tur = rng.random()
    if tur < 0.55:
        inkor = rng.random() < 0.45
        sifat = IJOBIY if (qutb > 0) != inkor else SALBIY
        return f"{jihat} {rng.choice(sifat)}" + (" emas" if inkor else "")
    if tur < 0.8:
        return str(rng.choice(["menga yoqdi", "tavsiya qilaman", "yana olaman"]
                              if qutb > 0 else
                              ["menga yoqmadi", "tavsiya qilmayman", "yana olmayman"]))
    return f"{jihat}da muammo " + ("yo'q" if qutb > 0 else "ko'p")


def sharh(rng):
    yorliq = int(rng.random() < 0.5)
    qutb = 1 if yorliq else -1
    gaplar = [gap(qutb if rng.random() < 0.85 else -qutb, rng)
              for _ in range(int(rng.integers(1, 4)))]
    return ", ".join(gaplar), yorliq


def main() -> None:
    rng = np.random.default_rng(0)
    juftlar = [sharh(rng) for _ in range(3000)]
    matnlar = [m for m, _ in juftlar]
    y = np.array([t for _, t in juftlar])
    emasli = np.array([" emas" in m for m in matnlar])
    print("=== 1. Ma'lumot ===")
    for m, t in juftlar[:4]:
        print(f"  {t} | {m}")
    print(f"  sharhlar: {len(y)}, ijobiy ulushi {y.mean():.3f}, "
          f"'emas' bor: {emasli.mean():.1%}")

    print("\n=== 2. Unigramda 'yaxshi' va 'yaxshi emas' juda o'xshash ===")
    juft_gap = ["sifati yaxshi", "sifati yaxshi emas", "sifati yomon"]
    for ng in [(1, 1), (1, 2)]:
        cv = CountVectorizer(token_pattern=TOKEN_NAQSH, ngram_range=ng).fit(matnlar)
        S = cosine_similarity(cv.transform(juft_gap))
        print(f"  ngram={ng}: cos(yaxshi, yaxshi emas)={S[0, 1]:.3f}  "
              f"cos(yaxshi, yomon)={S[0, 2]:.3f}")

    print("\n=== 3. Logistik regressiya: unigram va uni+bigram (5 fold) ===")
    cv5 = StratifiedKFold(5, shuffle=True, random_state=0)
    natija = {(1, 1): [], (1, 2): []}
    emas_natija = {(1, 1): [], (1, 2): []}
    emassiz = {(1, 1): [], (1, 2): []}
    for oquv, test in cv5.split(matnlar, y):
        for ng in natija:
            v = CountVectorizer(token_pattern=TOKEN_NAQSH, ngram_range=ng)
            Xo = v.fit_transform([matnlar[i] for i in oquv])
            Xt = v.transform([matnlar[i] for i in test])
            m = LogisticRegression(C=10.0, max_iter=2000).fit(Xo, y[oquv])
            p = m.predict(Xt)
            natija[ng].append((p == y[test]).mean())
            e = emasli[test]
            emas_natija[ng].append((p[e] == y[test][e]).mean())
            emassiz[ng].append((p[~e] == y[test][~e]).mean())
    for ng in natija:
        print(f"  ngram={ng}: aniqlik {np.mean(natija[ng]):.4f}, "
              f"'emas'li {np.mean(emas_natija[ng]):.4f}, "
              f"'emas'siz {np.mean(emassiz[ng]):.4f}")
    for nom, d in [("hammasi", natija), ("'emas'li", emas_natija),
                   ("'emas'siz", emassiz)]:
        farq = np.array(d[(1, 2)]) - np.array(d[(1, 1)])
        se = farq.std(ddof=1) / np.sqrt(len(farq))
        holat = "sezilarli" if abs(farq.mean()) > 2 * se else "sezilarli emas"
        print(f"  bigram foydasi ({nom}): {farq.mean():+.4f} (SE {se:.4f}) {holat}")

    print("\n=== 4. Model nimani o'rgandi (uni+bigram, butun ma'lumot) ===")
    v = CountVectorizer(token_pattern=TOKEN_NAQSH, ngram_range=(1, 2))
    m = LogisticRegression(C=10.0, max_iter=2000).fit(v.fit_transform(matnlar), y)
    w = dict(zip(v.get_feature_names_out(), m.coef_[0]))
    for f in ["yaxshi", "yaxshi emas", "yomon", "yomon emas", "emas",
              "yoqdi", "yoqmadi", "muammo yo'q"]:
        print(f"  {f:<14} {w[f]:+.3f}")
    if w["yoqmadi"] < 0 < w["yoqdi"]:
        print("  -ma- inkori so'z ichida: 'yoqmadi' - alohida unigram, "
              "uning belgisi unigramning o'zida")
    print("  ⭐ 'emas' kabi alohida inkor so'zi uchun bigram kerak")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  0 | dizayni xunuk, tavsiya qilmayman
  0 | xizmatida muammo ko'p, yana olmayman
  0 | sifati a'lo emas
  1 | yana olaman
  sharhlar: 3000, ijobiy ulushi 0.502, 'emas' bor: 40.2%

=== 2. Unigramda 'yaxshi' va 'yaxshi emas' juda o'xshash ===
  ngram=(1, 1): cos(yaxshi, yaxshi emas)=0.816  cos(yaxshi, yomon)=0.500
  ngram=(1, 2): cos(yaxshi, yaxshi emas)=0.775  cos(yaxshi, yomon)=0.333

=== 3. Logistik regressiya: unigram va uni+bigram (5 fold) ===
  ngram=(1, 1): aniqlik 0.7510, 'emas'li 0.6162, 'emas'siz 0.8417
  ngram=(1, 2): aniqlik 0.8670, 'emas'li 0.8581, 'emas'siz 0.8727
  bigram foydasi (hammasi): +0.1160 (SE 0.0019) sezilarli
  bigram foydasi ('emas'li): +0.2419 (SE 0.0144) sezilarli
  bigram foydasi ('emas'siz): +0.0311 (SE 0.0095) sezilarli

=== 4. Model nimani o'rgandi (uni+bigram, butun ma'lumot) ===
  yaxshi         +2.060
  yaxshi emas    -4.258
  yomon          -2.120
  yomon emas     +4.575
  emas           +0.085
  yoqdi          +0.819
  yoqmadi        -0.935
  muammo yo'q    +1.098
  -ma- inkori so'z ichida: 'yoqmadi' - alohida unigram, uning belgisi unigramning o'zida
  ⭐ 'emas' kabi alohida inkor so'zi uchun bigram kerak

CountVectorizer().fit(matnlar)                       # "yinchi", "alaba"   # ⚠️
CountVectorizer(preprocessor=normalla, token_pattern=TOKEN_NAQSH)          # ✅

X = tv.fit_transform(barcha); Xo, Xt = X[oquv], X[test]                    # ⚠️
Xo = tv.fit_transform(oquv_matn); Xt = tv.transform(test_matn)             # ✅

TfidfVectorizer(stop_words=["emas", "yo'q", "bor"])  # inkor yo'qoladi     # ⚠️
TfidfVectorizer(ngram_range=(1, 2))                  # "yaxshi emas"       # ✅

X.toarray() @ X.toarray().T                          # yuzlab MB           # ⚠️
cosine_similarity(X)                                 # siyrak holda        # ✅

q = tv.fit_transform([sorov])                        # boshqa lug'at       # ⚠️
q = tv.transform([sorov])                                                  # ✅

np.linalg.norm(X_sanoq - q, axis=1).argmin()         # uzunlikka sezgir    # ⚠️
cosine_similarity(q, X_tfidf).argmax()                                     # ✅

CountVectorizer(preprocessor=lambda t: t.strip())    # lowercase ishlamaydi # ⚠️
CountVectorizer(preprocessor=lambda t: normalla(t))  # normalla lower qiladi # ✅

1.  # CountVectorizer() "o'yinchi" dan qanday token oladi?
2.  # "narx oshdi" va "oshdi narx" bag-of-words vektorlari?
3.  # ngram_range=(1, 2) da "yaxshi emas" dan nechta xususiyat?
4.  # min_df=2 ning ma'nosi?
5.  # max_df=0.05 nimani tashlaydi?
6.  # smooth_idf=True da idf formulasi?
7.  # n=6, df=1 bo'lsa idf (smooth)?
8.  # sublinear_tf=True da tf=2 qanday bo'ladi?
9.  # norm="l2" dan keyin hujjat vektori uzunligi?
10. # L2 normallangan vektorlar uchun kosinus?
11. # so'rov uchun fit_transform yoki transform?
12. # unigram modelda "yaxshi emas" va "yaxshi" farqi?

1.  cv = CountVectorizer()
    X = cv.fit_transform(ozbekcha_matnlar)

2.  X = TfidfVectorizer().fit_transform(barcha_matnlar)
    X_oquv, X_test = X[:800], X[800:]

3.  tv = TfidfVectorizer(stop_words=["emas", "yo'q"])   # sentiment uchun

4.  S = X.toarray() @ X.toarray().T                     # 50 000 hujjat

5.  q = tv.fit_transform([sorov])
    S = cosine_similarity(q, X)