17.6-dars: Matn belgilari

s = df["matn"].fillna("")
df["uzunlik"] = s.str.len()
df["sozlar"] = s.str.split().str.len()
df["ortacha_soz"] = df["uzunlik"] / df["sozlar"].clip(lower=1)
df["undov"] = s.str.count("!")
df["savol"] = s.str.count(r"\?")
df["katta_ulush"] = s.str.count(r"[A-Z]") / s.str.len().clip(lower=1)
df["raqam_ulush"] = s.str.count(r"\d") / s.str.len().clip(lower=1)
df["bosh_harf"] = s.str.count(r"\b[A-Z]")
df["url_bor"] = s.str.contains(r"http", case=False).astype(int)
df["bosh"] = (s.str.strip() == "").astype(int)

NEGA ARZON VA KUCHLI:
  + bir necha qator kod
  + tez hisoblanadi
  + til va lug'atga bog'liq emas
  + ko'pincha TF-IDF ning yarim foydasini beradi

QACHON ISHLAYDI:
  shoshilinchlik, spam, sifat, emotsional ohang

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

CountVectorizer(lowercase=True, min_df=5, max_df=0.9, max_features=20000,
                stop_words=None, token_pattern=r"(?u)\b\w\w+\b")

TfidfVectorizer(min_df=5, max_df=0.9, sublinear_tf=True, norm="l2")

COUNT: har hujjatda har so'z necha marta uchragan
TF-IDF: chastota * teskari hujjat chastotasi

  tf(t, d)  = so'zning hujjatdagi chastotasi
  idf(t)    = log(N / df(t)) + 1
  -> KAM hujjatda uchraydigan so'z KO'PROQ vazn oladi

sublinear_tf=True: tf o'rniga 1 + log(tf) - uzun hujjatlarni yumshatadi
norm="l2": har hujjat vektori birlik uzunlikka keltiriladi (masshtablash)

QAYSI MODELGA:
  chiziqli (LogReg, LinearSVC) + TF-IDF - klassik va kuchli
  Naive Bayes + Count - tez bazaviy
  daraxtlar + TF-IDF - yomon (siyrak, yuqori o'lcham)

ngram_range=(1, 1)  - faqat alohida so'zlar (unigram)
ngram_range=(1, 2)  - so'zlar + juftliklar (bigram)
ngram_range=(1, 3)  - + uchliklar

NEGA KERAK:
  "yaxshi emas" - unigram da "yaxshi" va "emas" alohida
                  bigram da "yaxshi emas" - inkor saqlanadi

NARXI: belgilar soni keskin oshadi
  unigram: 20 000
  + bigram: 200 000+
  -> min_df va max_features bilan cheklang

AMALIY: (1, 2) ko'p hollarda yetarli

TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), min_df=3)

So'z o'rniga HARFLAR ketma-ketligi:
  "kitob" -> "kit", "ito", "tob" (3-gram)

QACHON FOYDALI:
  + imlo xatolari ko'p ("kitob" / "kitab")
  + morfologik boy tillar (o'zbek, turk, rus) - qo'shimchalar
  + qisqa matnlar (nom, manzil, mahsulot nomi)
  + tokenizatsiya qiyin bo'lganda

char_wb - so'z chegaralarini hisobga oladi (afzalroq)

NARXI: belgilar soni ko'p, hisoblash sekinroq

min_df=5          - 5 tadan kam hujjatda uchragan so'zni tashlash
max_df=0.9        - hujjatlarning 90% idan ko'pida uchragan so'zni tashlash
max_features=20000 - eng ko'p uchraydigan N ta so'z

STOP-SO'ZLAR: "va", "bilan", "uchun" - odatda foydasiz
  sklearn da faqat ingliz tili uchun tayyor ro'yxat bor
  o'zbek tili uchun qo'lda ro'yxat yoki max_df bilan avtomatik

HashingVectorizer: lug'at saqlamaydi, xesh funksiya ishlatadi
  + xotira tejamkor, oqimda ishlaydi
  + fit kerak emas (stateless)
  - teskari aylantirib bo'lmaydi (qaysi so'z ekanini bilib bo'lmaydi)
  - to'qnashuvlar bo'lishi mumkin

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

tayyor = ColumnTransformer([
    ("matn", TfidfVectorizer(min_df=5, ngram_range=(1, 2)), "izoh"),
    ("kategoriya", OneHotEncoder(handle_unknown="ignore"), ["kanal"]),
    ("sonli", StandardScaler(), ["uzunlik", "undov"]),
])
Pipeline([("t", tayyor), ("m", LogisticRegression(max_iter=2000))])

MUHIM: ColumnTransformer da matn ustuni STRING sifatida beriladi
  ("matn", TfidfVectorizer(), "izoh")     - ustun NOMI (ro'yxat emas!)
  ["izoh"] bersangiz XATO bo'ladi

IKKI BOSQICHLI YONDASHUV (ko'pincha yaxshiroq):
  1. Matndan sodda statistika -> sonli belgilar
  2. TF-IDF dan alohida model -> uning bashoratini BELGI sifatida
  3. Ikkalasini gradient boosting ga berish

from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import (CountVectorizer, HashingVectorizer,
                                             TfidfVectorizer)
from sklearn.pipeline import Pipeline

s = df["izoh"].fillna("")
df["uzunlik"], df["sozlar"] = s.str.len(), s.str.split().str.len()
df["undov"] = s.str.count("!")

tfidf = TfidfVectorizer(min_df=5, max_df=0.9, ngram_range=(1, 2),
                        sublinear_tf=True, max_features=50000)
char = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), min_df=3)

tayyor = ColumnTransformer([
    ("matn", tfidf, "izoh"),              # ustun NOMI, ro'yxat emas
    ("sonli", "passthrough", ["uzunlik", "undov"])])
QOIDA: sodda statistikadan boshla · min_df qo'y · Pipeline ichida ·
       matn ustunini ro'yxatsiz ber

Sodda statistika: uzunlik, so'zlar, undov, katta harf - arzon va kuchli
TF-IDF + chiziqli model - klassik kombinatsiya
ngram_range=(1, 2) inkorni saqlaydi; char_wb morfologik tillarga
min_df / max_df / max_features bilan o'lchamni cheklang

"""Arzon belgilar, katta foyda (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score


def yarat(seed: int = 5, n: int = 4000) -> pd.DataFrame:
    """Qo'llab-quvvatlash murojaatlari: shoshilinchlik."""
    rng = np.random.default_rng(seed)
    shoshilinch = rng.random(n) < 0.25
    kanal = rng.choice(["ilova", "sayt", "email"], n, p=[0.5, 0.3, 0.2])
    matnlar = []
    for tez in shoshilinch:
        if tez:
            asos = rng.choice(["ISHLAMAYAPTI", "TEZDA yordam kerak",
                               "pul yechib olindi", "buyurtma kelmadi"])
            undov = "!" * int(rng.integers(1, 5))
            qoshimcha = rng.choice(["", " iltimos tezroq", " juda muhim"])
            matn = f"{asos}{undov}{qoshimcha}"
        else:
            asos = rng.choice(["savol bor edi", "qanday qilib almashtirsam",
                               "narxlar haqida ma'lumot",
                               "yetkazish muddati qancha"])
            matn = asos + rng.choice(["", " rahmat", " iltimos javob bering"])
        # uzunlikni tasodifiylashtirish
        if rng.random() < 0.4:
            matn += " " + " ".join(rng.choice(
                ["mahsulot", "buyurtma", "xizmat", "hisob", "karta"],
                int(rng.integers(3, 25))))
        matnlar.append(matn)
    return pd.DataFrame({"matn": matnlar, "kanal": kanal,
                         "shoshilinch": shoshilinch.astype(int)})


def matn_statistikasi(s: pd.Series) -> pd.DataFrame:
    s = s.fillna("")
    uzunlik = s.str.len().clip(lower=1)
    return pd.DataFrame({
        "uzunlik": s.str.len(),
        "sozlar": s.str.split().str.len(),
        "ortacha_soz": s.str.len() / s.str.split().str.len().clip(lower=1),
        "undov": s.str.count("!"),
        "savol": s.str.count(r"\?"),
        "katta_ulush": s.str.count(r"[A-Z]") / uzunlik,
        "raqam_ulush": s.str.count(r"\d") / uzunlik,
        "bosh_harfli_soz": s.str.count(r"\b[A-Z]"),
    })


def main() -> None:
    df = yarat()
    y = df["shoshilinch"].to_numpy()
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Ma'lumot ===")
    print(f"  {len(df)} murojaat, shoshilinch {y.mean():.1%}")
    for nechanchi in [0, 1]:
        namuna = df.loc[df["shoshilinch"] == nechanchi, "matn"].iloc[0]
        nom = "shoshilinch" if nechanchi else "oddiy"
        print(f"  {nom:<12}: {namuna[:60]}")

    print("\n=== 2. Statistik belgilar ===")
    stat = matn_statistikasi(df["matn"])
    print(f"  {'belgi':<18} {'oddiy':>10} {'shoshilinch':>13} "
          f"{'korrelyatsiya':>15}")
    for nom in stat.columns:
        a = stat.loc[y == 0, nom].mean()
        b = stat.loc[y == 1, nom].mean()
        # doimiy belgi uchun korrelyatsiya aniqlanmagan
        korr = (stat[nom].corr(pd.Series(y)) if stat[nom].std() > 0
                else float("nan"))
        print(f"  {nom:<18} {a:>10.3f} {b:>13.3f} {korr:>+15.4f}")

    print("\n=== 3. Belgilar guruhlari ===")
    def model():
        return HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                              random_state=0)
    kanal_kod = pd.get_dummies(df["kanal"], prefix="kanal").astype(int)
    guruhlar = {
        "faqat kanal": kanal_kod,
        "faqat uzunlik": stat[["uzunlik", "sozlar"]],
        "uzunlik + undov": stat[["uzunlik", "sozlar", "undov"]],
        "barcha statistika": stat,
        "statistika + kanal": pd.concat([stat, kanal_kod], axis=1),
    }
    print(f"  {'guruh':<22} {'belgilar':>9} {'CV ROC AUC':>12}")
    for nom, Xa in guruhlar.items():
        b = cross_val_score(model(), Xa, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {nom:<22} {Xa.shape[1]:>9} {b:>12.4f}")

    print("\n=== 4. Har belgining yakka kuchi ===")
    print(f"  {'belgi':<18} {'yakka CV AUC':>14}")
    for nom in stat.columns:
        b = cross_val_score(model(), stat[[nom]], y, cv=cv,
                            scoring="roc_auc").mean()
        print(f"  {nom:<18} {b:>14.4f}")
    print("  ⭐ Sodda statistika - arzon va ko'pincha yetarli")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  4000 murojaat, shoshilinch 26.2%
  oddiy       : yetkazish muddati qancha rahmat
  shoshilinch : TEZDA yordam kerak!!! iltimos tezroq

=== 2. Statistik belgilar ===
  belgi                   oddiy   shoshilinch   korrelyatsiya
  uzunlik                71.058        69.194         -0.0142
  sozlar                  9.797         9.332         -0.0262
  ortacha_soz             7.159         7.791         +0.2190
  undov                   0.000         2.434         +0.8831
  savol                   0.000         0.000            +nan
  katta_ulush             0.000         0.122         +0.4626
  raqam_ulush             0.000         0.000            +nan
  bosh_harfli_soz         0.000         0.492         +0.6459

=== 3. Belgilar guruhlari ===
  guruh                   belgilar   CV ROC AUC
  faqat kanal                    3       0.5041
  faqat uzunlik                  2       0.8821
  uzunlik + undov                3       1.0000
  barcha statistika              8       1.0000
  statistika + kanal            11       1.0000

=== 4. Har belgining yakka kuchi ===
  belgi                yakka CV AUC
  uzunlik                    0.7703
  sozlar                     0.7457
  ortacha_soz                0.8508
  undov                      1.0000
  savol                      0.5000
  katta_ulush                0.7462
  raqam_ulush                0.5000
  bosh_harfli_soz            0.7462
  ⭐ Sodda statistika - arzon va ko'pincha yetarli

"""Ikki vektorizator va parametrlar (real sklearn)."""

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline


def yarat(seed: int = 7, n: int = 2500):
    """Ikki sinf: ijobiy va salbiy izohlar."""
    rng = np.random.default_rng(seed)
    sinf = rng.integers(0, 2, n)
    ijobiy = ["zo'r", "ajoyib", "tez", "sifatli", "tavsiya", "mamnun",
              "arzon", "chiroyli"]
    salbiy = ["yomon", "sekin", "sifatsiz", "aldov", "qimmat", "buzuq",
              "kechikdi", "noto'g'ri"]
    umumiy = ["mahsulot", "buyurtma", "yetkazish", "narx", "xizmat", "sotuvchi",
              "do'kon", "karta"]
    matnlar = []
    for s in sinf:
        asosiy = ijobiy if s == 1 else salbiy
        boshqa = salbiy if s == 1 else ijobiy
        uzunlik = int(rng.integers(8, 40))
        sozlar = list(rng.choice(asosiy, max(int(uzunlik * 0.3), 1)))
        sozlar += list(rng.choice(umumiy, int(uzunlik * 0.5)))
        sozlar += list(rng.choice(boshqa, max(int(uzunlik * 0.2), 1)))
        # inkor: "emas" bilan ma'no teskari bo'ladi
        if rng.random() < 0.15:
            sozlar = [rng.choice(boshqa), "emas"] + sozlar
        rng.shuffle(sozlar)
        matnlar.append(" ".join(sozlar))
    return matnlar, sinf


def main() -> None:
    matnlar, y = yarat()
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Korpus ===")
    print(f"  {len(matnlar)} izoh, ijobiy {y.mean():.1%}")
    uzunliklar = [len(m.split()) for m in matnlar]
    print(f"  uzunlik: mediana {int(np.median(uzunliklar))}, "
          f"min {min(uzunliklar)}, max {max(uzunliklar)}")
    print(f"  namuna: {matnlar[0][:70]}")

    print("\n=== 2. Count va TF-IDF ===")
    variantlar = {
        "Count + LogReg": Pipeline([
            ("v", CountVectorizer(min_df=3)),
            ("m", LogisticRegression(max_iter=2000))]),
        "Count + NB": Pipeline([
            ("v", CountVectorizer(min_df=3)), ("m", MultinomialNB())]),
        "TF-IDF + LogReg": Pipeline([
            ("v", TfidfVectorizer(min_df=3)),
            ("m", LogisticRegression(max_iter=2000))]),
        "TF-IDF(sublinear) + LogReg": Pipeline([
            ("v", TfidfVectorizer(min_df=3, sublinear_tf=True)),
            ("m", LogisticRegression(max_iter=2000))]),
    }
    print(f"  {'variant':<28} {'belgilar':>9} {'CV ROC AUC':>12}")
    for nom, quvur in variantlar.items():
        nechta = quvur.named_steps["v"].fit(matnlar).transform(
            matnlar[:5]).shape[1]
        b = cross_val_score(quvur, matnlar, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {nom:<28} {nechta:>9} {b:>12.4f}")

    print("\n=== 3. min_df va max_df ===")
    print(f"  {'min_df':>7} {'max_df':>8} {'belgilar':>9} {'CV ROC AUC':>12}")
    for min_df in [1, 3, 10, 50]:
        for max_df in [1.0, 0.7]:
            v = TfidfVectorizer(min_df=min_df, max_df=max_df)
            nechta = v.fit(matnlar).transform(matnlar[:5]).shape[1]
            b = cross_val_score(Pipeline([("v", v),
                                          ("m", LogisticRegression(
                                              max_iter=2000))]),
                                matnlar, y, cv=cv, scoring="roc_auc").mean()
            print(f"  {min_df:>7} {max_df:>8} {nechta:>9} {b:>12.4f}")

    print("\n=== 4. Eng muhim so'zlar ===")
    v = TfidfVectorizer(min_df=3)
    X = v.fit_transform(matnlar)
    m = LogisticRegression(max_iter=2000).fit(X, y)
    nomlar = v.get_feature_names_out()
    tartib = np.argsort(m.coef_[0])
    print(f"  eng salbiy: "
          f"{[nomlar[i] for i in tartib[:6]]}")
    print(f"  eng ijobiy: "
          f"{[nomlar[i] for i in tartib[-6:]]}")
    print("  ⭐ TF-IDF + chiziqli model - klassik kombinatsiya")


if __name__ == "__main__":
    main()

=== 1. Korpus ===
  2500 izoh, ijobiy 50.9%
  uzunlik: mediana 22, min 7, max 39
  namuna: xizmat karta zo'r xizmat kechikdi karta arzon mahsulot chiroyli karta

=== 2. Count va TF-IDF ===
  variant                       belgilar   CV ROC AUC
  Count + LogReg                      27       0.9993
  Count + NB                          27       0.9917
  TF-IDF + LogReg                     27       0.9993
  TF-IDF(sublinear) + LogReg          27       0.9983

=== 3. min_df va max_df ===
   min_df   max_df  belgilar   CV ROC AUC
        1      1.0        27       0.9993
        1      0.7        18       0.9992
        3      1.0        27       0.9993
        3      0.7        18       0.9992
       10      1.0        27       0.9993
       10      0.7        18       0.9992
       50      1.0        27       0.9993
       50      0.7        18       0.9992

=== 4. Eng muhim so'zlar ===
  eng salbiy: ['yomon', 'sekin', 'qimmat', 'sifatsiz', 'kechikdi', 'aldov']
  eng ijobiy: ['chiroyli', 'mamnun', 'tez', 'sifatli', 'ajoyib', 'zo']
  ⭐ TF-IDF + chiziqli model - klassik kombinatsiya

"""Inkor va morfologiya muammolari (real sklearn)."""

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline


def inkorli_yarat(seed: int = 3, n: int = 2500):
    """Inkor ma'noni teskari qiladi: 'yaxshi emas' salbiy."""
    rng = np.random.default_rng(seed)
    matnlar, sinflar = [], []
    ijobiy = ["yaxshi", "tez", "sifatli", "arzon"]
    umumiy = ["mahsulot", "xizmat", "yetkazish", "narx", "do'kon"]
    for _ in range(n):
        soz = str(rng.choice(ijobiy))
        inkor = rng.random() < 0.5
        ibora = f"{soz} emas" if inkor else soz
        qolgan = " ".join(rng.choice(umumiy, int(rng.integers(4, 15))))
        matnlar.append(f"{qolgan} {ibora} {qolgan}")
        sinflar.append(0 if inkor else 1)
    return matnlar, np.array(sinflar)


def morfologik_yarat(seed: int = 4, n: int = 2500):
    """O'zbekcha qo'shimchalar: bir o'zak - ko'p shakl."""
    rng = np.random.default_rng(seed)
    ozaklar = {1: ["kitob", "daftar", "qalam"],
               0: ["telefon", "kompyuter", "quloqchin"]}
    qoshimchalar = ["", "ni", "ga", "da", "dan", "lar", "larni", "im", "ingiz"]
    umumiy = ["yangi", "eski", "katta", "kichik", "yaxshi"]
    matnlar, sinflar = [], []
    for _ in range(n):
        s = int(rng.integers(0, 2))
        sozlar = []
        for _ in range(int(rng.integers(3, 8))):
            ozak = str(rng.choice(ozaklar[s]))
            sozlar.append(ozak + str(rng.choice(qoshimchalar)))
        sozlar += list(rng.choice(umumiy, int(rng.integers(3, 10))))
        rng.shuffle(sozlar)
        matnlar.append(" ".join(sozlar))
        sinflar.append(s)
    return matnlar, np.array(sinflar)


def main() -> None:
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    def baho(matnlar, y, vektorizator):
        quvur = Pipeline([("v", vektorizator),
                          ("m", LogisticRegression(max_iter=2000))])
        nechta = vektorizator.fit(matnlar).transform(matnlar[:5]).shape[1]
        b = cross_val_score(quvur, matnlar, y, cv=cv, scoring="roc_auc").mean()
        return nechta, b

    print("=== 1. Inkor muammosi ===")
    matnlar, y = inkorli_yarat()
    print(f"  namuna (ijobiy): {matnlar[np.argmax(y)][:60]}")
    print(f"  namuna (salbiy): {matnlar[np.argmin(y)][:60]}")
    print(f"  {'ngram_range':>13} {'belgilar':>9} {'CV ROC AUC':>12}")
    for nr in [(1, 1), (1, 2), (1, 3), (2, 2)]:
        nechta, b = baho(matnlar, y, TfidfVectorizer(min_df=3, ngram_range=nr))
        print(f"  {str(nr):>13} {nechta:>9} {b:>12.4f}")
    print("  (unigram inkorni ko'rmaydi)")

    print("\n=== 2. Morfologiya muammosi ===")
    matnlar2, y2 = morfologik_yarat()
    print(f"  namuna: {matnlar2[0][:60]}")
    v = TfidfVectorizer(min_df=2)
    v.fit(matnlar2)
    nomlar = v.get_feature_names_out()
    kitob_shakllari = [s for s in nomlar if s.startswith("kitob")]
    print(f"  'kitob' o'zakli so'z shakllari: {len(kitob_shakllari)} ta")
    print(f"    {kitob_shakllari[:6]}")

    print("\n=== 3. So'z va belgi n-gramlari ===")
    variantlar = {
        "so'z (1,1)": TfidfVectorizer(min_df=2, ngram_range=(1, 1)),
        "so'z (1,2)": TfidfVectorizer(min_df=2, ngram_range=(1, 2)),
        "belgi (3,5)": TfidfVectorizer(analyzer="char_wb", min_df=3,
                                       ngram_range=(3, 5)),
        "belgi (2,4)": TfidfVectorizer(analyzer="char_wb", min_df=3,
                                       ngram_range=(2, 4)),
    }
    print(f"  {'vektorizator':<14} {'belgilar':>9} {'CV ROC AUC':>12}")
    for nom, vek in variantlar.items():
        nechta, b = baho(matnlar2, y2, vek)
        print(f"  {nom:<14} {nechta:>9} {b:>12.4f}")

    print("\n=== 4. Imlo xatolari bilan ===")
    rng = np.random.default_rng(0)
    buzuq = []
    for m in matnlar2:
        sozlar = m.split()
        for i in range(len(sozlar)):
            if rng.random() < 0.25 and len(sozlar[i]) > 3:
                j = int(rng.integers(1, len(sozlar[i]) - 1))
                sozlar[i] = sozlar[i][:j] + sozlar[i][j + 1:]
        buzuq.append(" ".join(sozlar))
    print(f"  asl:    {matnlar2[0][:55]}")
    print(f"  buzuq:  {buzuq[0][:55]}")
    print(f"  {'vektorizator':<14} {'toza CV':>9} {'buzuq CV':>10} "
          f"{'yo_qotish':>11}")
    for nom in ["so'z (1,1)", "belgi (3,5)"]:
        vek = variantlar[nom]
        _, toza = baho(matnlar2, y2, vek)
        _, b = baho(buzuq, y2, vek)
        print(f"  {nom:<14} {toza:>9.4f} {b:>10.4f} {toza - b:>11.4f}")
    print("  ⭐ Belgi n-gramlari imlo va morfologiyaga chidamli")


if __name__ == "__main__":
    main()

=== 1. Inkor muammosi ===
  namuna (ijobiy): xizmat xizmat narx narx narx do'kon xizmat do'kon mahsulot m
  namuna (salbiy): mahsulot do'kon do'kon yetkazish arzon emas mahsulot do'kon
    ngram_range  belgilar   CV ROC AUC
         (1, 1)        11       1.0000
         (1, 2)        86       1.0000
         (1, 3)       525       1.0000
         (2, 2)        75       1.0000
  (unigram inkorni ko'rmaydi)

=== 2. Morfologiya muammosi ===
  namuna: qalamingiz daftarlar qalam eski katta daftarim yaxshi eski q
  'kitob' o'zakli so'z shakllari: 9 ta
    ['kitob', 'kitobda', 'kitobdan', 'kitobga', 'kitobim', 'kitobingiz']

=== 3. So'z va belgi n-gramlari ===
  vektorizator    belgilar   CV ROC AUC
  so'z (1,1)            59       1.0000
  so'z (1,2)          1863       1.0000
  belgi (3,5)          455       1.0000
  belgi (2,4)          367       1.0000

=== 4. Imlo xatolari bilan ===
  asl:    qalamingiz daftarlar qalam eski katta daftarim yaxshi e
  buzuq:  qalamingiz daftarlar qlam eski katta daftarim yaxshi es
  vektorizator     toza CV   buzuq CV   yo_qotish
  so'z (1,1)        1.0000     1.0000      0.0000
  belgi (3,5)       1.0000     1.0000      0.0000
  ⭐ Belgi n-gramlari imlo va morfologiyaga chidamli

"""Matn va jadval belgilarini bitta Pipeline da (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.feature_extraction.text import HashingVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, cross_val_predict
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


class MatnStatistikasi(BaseEstimator, TransformerMixin):
    """Matn ustunidan sonli belgilar chiqaradi."""

    def __init__(self, ustun: str = "matn"):
        self.ustun = ustun

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        s = X[self.ustun].fillna("")
        uzunlik = s.str.len().clip(lower=1)
        return np.column_stack([
            s.str.len(),
            s.str.split().str.len(),
            s.str.count("!"),
            s.str.count(r"[A-Z]") / uzunlik,
            s.str.count(r"\d") / uzunlik,
        ])

    def get_feature_names_out(self, input_features=None):
        return np.array(["uzunlik", "sozlar", "undov", "katta", "raqam"])


def yarat(seed: int = 9, n: int = 4000) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    shoshilinch = rng.random(n) < 0.25
    kanal = rng.choice(["ilova", "sayt", "email"], n, p=[0.5, 0.3, 0.2])
    soat = rng.integers(0, 24, n)
    tez_sozlar = ["ISHLAMAYAPTI", "shoshilinch", "tezda", "muammo", "xato"]
    oddiy_sozlar = ["savol", "qiziqdim", "ma'lumot", "qanday", "rahmat"]
    umumiy = ["buyurtma", "mahsulot", "xizmat", "hisob", "karta", "narx"]
    matnlar = []
    for tez in shoshilinch:
        asosiy = tez_sozlar if tez else oddiy_sozlar
        sozlar = list(rng.choice(asosiy, int(rng.integers(1, 4))))
        sozlar += list(rng.choice(umumiy, int(rng.integers(3, 20))))
        rng.shuffle(sozlar)
        matn = " ".join(sozlar)
        if tez and rng.random() < 0.6:
            matn += "!" * int(rng.integers(1, 4))
        matnlar.append(matn)
    return pd.DataFrame({"matn": matnlar, "kanal": kanal, "soat": soat,
                         "shoshilinch": shoshilinch.astype(int)})


def main() -> None:
    df = yarat()
    y = df["shoshilinch"].to_numpy()
    X = df[["matn", "kanal", "soat"]]
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Ustunlar ===")
    print(f"  matn (string), kanal (kategoriya), soat (sonli)")
    print(f"  {len(df)} qator, shoshilinch {y.mean():.1%}")

    print("\n=== 2. ColumnTransformer da matn ustuni ===")
    try:
        ColumnTransformer([("t", TfidfVectorizer(), ["matn"])]).fit_transform(X)
        print("  ro'yxat bilan ishladi (kutilmagan)")
    except (ValueError, AttributeError) as xato:
        print(f"  ['matn'] (ro'yxat) -> {type(xato).__name__}: "
              f"{str(xato)[:55]}...")
    natija = ColumnTransformer([("t", TfidfVectorizer(), "matn")]).fit_transform(X)
    print(f"  'matn' (nom) -> ishladi, o'lcham {natija.shape}")

    print("\n=== 3. Belgilar to'plamlari ===")
    faqat_jadval = ColumnTransformer([
        ("k", OneHotEncoder(handle_unknown="ignore", sparse_output=False),
         ["kanal"]),
        ("s", StandardScaler(), ["soat"])])
    stat_bilan = ColumnTransformer([
        ("k", OneHotEncoder(handle_unknown="ignore", sparse_output=False),
         ["kanal"]),
        ("s", StandardScaler(), ["soat"]),
        ("st", MatnStatistikasi("matn"), ["matn"])])
    tfidf_bilan = ColumnTransformer([
        ("k", OneHotEncoder(handle_unknown="ignore"), ["kanal"]),
        ("s", StandardScaler(), ["soat"]),
        ("t", TfidfVectorizer(min_df=3, ngram_range=(1, 2)), "matn")])
    hammasi = ColumnTransformer([
        ("k", OneHotEncoder(handle_unknown="ignore"), ["kanal"]),
        ("s", StandardScaler(), ["soat"]),
        ("st", MatnStatistikasi("matn"), ["matn"]),
        ("t", TfidfVectorizer(min_df=3, ngram_range=(1, 2)), "matn")])
    variantlar = {
        "faqat jadval": faqat_jadval,
        "+ matn statistikasi": stat_bilan,
        "+ TF-IDF": tfidf_bilan,
        "hammasi": hammasi,
    }
    print(f"  {'to_plam':<22} {'belgilar':>9} {'CV ROC AUC':>12}")
    for nom, tayyor in variantlar.items():
        nechta = tayyor.fit(X, y).transform(X[:5]).shape[1]
        b = cross_val_score(Pipeline([("t", tayyor),
                                      ("m", LogisticRegression(max_iter=3000))]),
                            X, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {nom:<22} {nechta:>9} {b:>12.4f}")

    print("\n=== 4. Ikki bosqichli yondashuv ===")
    # 1-bosqich: matndan alohida model, uning OOF bashoratini belgi qilish
    matn_quvur = Pipeline([("t", TfidfVectorizer(min_df=3, ngram_range=(1, 2))),
                           ("m", LogisticRegression(max_iter=3000))])
    oof = cross_val_predict(matn_quvur, df["matn"], y, cv=cv,
                            method="predict_proba")[:, 1]
    stat = MatnStatistikasi("matn").transform(X)
    kanal_kod = pd.get_dummies(df["kanal"], prefix="k").astype(int).to_numpy()
    X2 = np.column_stack([kanal_kod, df["soat"].to_numpy(), stat, oof])
    b_gb = cross_val_score(HistGradientBoostingClassifier(learning_rate=0.1,
                                                          max_iter=200,
                                                          random_state=0),
                           X2, y, cv=cv, scoring="roc_auc").mean()
    b_lr = cross_val_score(Pipeline([("t", hammasi),
                                     ("m", LogisticRegression(max_iter=3000))]),
                           X, y, cv=cv, scoring="roc_auc").mean()
    print(f"  bitta Pipeline (LogReg + TF-IDF): {b_lr:.4f}")
    print(f"  ikki bosqichli (matn bali + GB): {b_gb:.4f}")
    print(f"  matn balining yakka kuchi: "
          f"{cross_val_score(HistGradientBoostingClassifier(max_iter=100, random_state=0), oof.reshape(-1, 1), y, cv=cv, scoring='roc_auc').mean():.4f}")
    print("  ⭐ Matn ustuni ColumnTransformer da ro'yxatsiz beriladi")


if __name__ == "__main__":
    main()

=== 1. Ustunlar ===
  matn (string), kanal (kategoriya), soat (sonli)
  4000 qator, shoshilinch 24.6%

=== 2. ColumnTransformer da matn ustuni ===
  ro'yxat bilan ishladi (kutilmagan)
  'matn' (nom) -> ishladi, o'lcham (4000, 17)

=== 3. Belgilar to'plamlari ===
  to_plam                 belgilar   CV ROC AUC
  faqat jadval                   4       0.5187
  + matn statistikasi            9       0.8629
  + TF-IDF                     227       1.0000
  hammasi                      232       1.0000

=== 4. Ikki bosqichli yondashuv ===
  bitta Pipeline (LogReg + TF-IDF): 1.0000
  ikki bosqichli (matn bali + GB): 1.0000
  matn balining yakka kuchi: 1.0000
  ⭐ Matn ustuni ColumnTransformer da ro'yxatsiz beriladi

X = TfidfVectorizer().fit_transform(matnlar); cross_val_score(m, X, y)  # ⚠️
cross_val_score(Pipeline([("v", TfidfVectorizer()), ("m", m)]), matnlar, y) # ✅

TfidfVectorizer(ngram_range=(1, 3))    # 500 000 belgi                 # ⚠️
TfidfVectorizer(ngram_range=(1, 2), min_df=5, max_features=50000)      # ✅

ColumnTransformer([("t", TfidfVectorizer(), ["matn"])])                # ⚠️
ColumnTransformer([("t", TfidfVectorizer(), "matn")])                  # ✅

TfidfVectorizer().fit(df["izoh"])      # NaN -> AttributeError         # ⚠️
TfidfVectorizer().fit(df["izoh"].fillna(""))                           # ✅

RandomForestClassifier().fit(tfidf_matritsa, y)                        # ⚠️
LogisticRegression()  yoki  TruncatedSVD dan keyin daraxt              # ✅

# darhol TF-IDF                                                        # ⚠️
# avval uzunlik, undov, katta harf - arzon va tushunarli               # ✅

TfidfVectorizer(analyzer="char", ngram_range=(2, 8))  # millionlab     # ⚠️
TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), min_df=3)      # ✅

1.  # matn belgilarining uch darajasi?
2.  # eng arzon daraja?
3.  # TF-IDF formulasi?
4.  # sublinear_tf nima qiladi?
5.  # TF-IDF qaysi modelga mos?
6.  # bigram nima uchun kerak?
7.  # char_wb qachon foydali?
8.  # min_df nima qiladi?
9.  # max_df nimani almashtiradi?
10. # HashingVectorizer kamchiligi?
11. # ColumnTransformer da matn ustuni?
12. # NaN bilan nima qilish kerak?

1.  X = TfidfVectorizer().fit_transform(matnlar); cross_val_score(m, X, y)

2.  TfidfVectorizer(ngram_range=(1, 3))

3.  ColumnTransformer([("t", TfidfVectorizer(), ["matn"])])

4.  TfidfVectorizer().fit(df["izoh"])   # NaN bor

5.  RandomForestClassifier().fit(tfidf_matritsa, y)