17.3-dars: Kategoriyali belgilarni kodlash

from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

OneHotEncoder(handle_unknown="ignore", sparse_output=False,
              min_frequency=20, max_categories=15, drop=None)

OrdinalEncoder(categories=[["past", "o_rta", "yuqori"]],
               handle_unknown="use_encoded_value", unknown_value=-1)

ONE-HOT: har daraja uchun alohida 0/1 ustun
  + tartib qo'shmaydi, har qanday model bilan ishlaydi
  - kardinallik bilan ustunlar soni oshadi
  - siyrak matritsa (chiziqli modellarga yaxshi, daraxtlarga yomon)
  drop="first" - chiziqli modellarda multikollinearlikni oldini oladi

ORDINAL: har darajaga butun son
  + bitta ustun
  - TARTIB qo'shadi -> faqat haqiqiy ordinal belgilarga
  - daraxtlar uchun nominal bo'lsa ham ba'zan ishlaydi (lekin optimal emas)

G'OYA: har darajani MAQSADNING shu darajadagi o'rtachasi bilan almashtirish

  hudud = "buxoro"  ->  buxorodagi o'rtacha defolt ulushi 0.23-bob

XAVF: qatorning O'ZI o'rtachaga hissa qo'shadi
  -> model o'z javobini ko'radi -> CV optimistik

IKKI HIMOYA:
1. OUT-OF-FOLD: har fold uchun o'rtacha QOLGAN foldlardan hisoblanadi
2. SMOOTHING (regulyarizatsiya): kam uchraydigan darajalar
   umumiy o'rtachaga tortiladi

   kodlangan = (n_k * o'rtacha_k + m * umumiy) / (n_k + m)
   m - smoothing kuchi

sklearn 1.3+: TargetEncoder(smooth="auto", cv=StratifiedKFold(...))
  - out-of-fold va smoothing ni O'ZI bajaradi
  - cv ga SEED li splitter bering, aks holda natija takrorlanmaydi

# har darajani uning uchrash soni/ulushi bilan almashtirish
sanoq = df["hudud"].value_counts()
df["hudud_soni"] = df["hudud"].map(sanoq)
df["hudud_ulushi"] = df["hudud"].map(sanoq / len(df))

+ maqsadni KO'RMAYDI -> leakage yo'q
+ bitta ustun, yuqori kardinallikda ham ishlaydi
+ ko'pincha kutilmaganda foydali ("mashhurlik" signali)
- turli darajalar bir xil songa ega bo'lishi mumkin (to'qnashuv)
- ma'no bilvosita

Ko'pincha target encoding BILAN BIRGA ishlatiladi

Kardinallik            Tavsiya
< 10                   OneHot
10 - 50                OneHot (min_frequency bilan) yoki Target
50 - 1000              Target encoding, Frequency, ichki (LightGBM/CatBoost)
> 1000                 Target + Frequency, embedding (NN), hashing

QO'SHIMCHA USULLAR:
  - GURUHLASH: kam uchraydiganlarni "boshqa" ga birlashtirish
  - IERARXIYA: pochta indeksi -> tuman -> viloyat
  - ICHKI mexanizm: LightGBM/XGBoost categorical 15.10-bob
  - HASHING: HashingVectorizer - o'lchamni qat'iy cheklaydi
  - EMBEDDING: neyron tarmoqlarda o'rganiladigan vektor

Ishlab chiqarishda o'quvda BO'LMAGAN daraja keladi

OneHotEncoder(handle_unknown="ignore")    -> barcha ustunlar 0
OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1)
TargetEncoder                              -> umumiy o'rtacha

TEKSHIRUV:
  - test/ishlab chiqarishda yangi darajalar ulushi qancha?
  - vaqt o'tishi bilan yangi darajalar paydo bo'ladimi?
  - "boshqa" guruhi oldindan yaratilganmi?

EHTIYOT: handle_unknown siz Pipeline ISHLAMAY QOLADI

                      Chiziqli model    Daraxtlar        Boosting (LGBM/XGB)
kam kardinallik       OneHot(drop=1st)  OneHot           ichki categorical
o'rta kardinallik     Target + OneHot   Target           ichki categorical
yuqori kardinallik    Target + Freq     Target + Freq    ichki categorical
ordinal belgi         Ordinal           Ordinal          Ordinal

ESLATMA: daraxtlar uchun OneHot yuqori kardinallikda ZARAR qiladi
  har ustun kam ma'lumot beradi -> daraxt sayozlashadi (15.10)

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, TargetEncoder

tayyor = ColumnTransformer([
    ("kam", OneHotEncoder(handle_unknown="ignore", min_frequency=20), kam_kard),
    ("ordinal", OrdinalEncoder(categories=[tartib],
                               handle_unknown="use_encoded_value",
                               unknown_value=-1), ordinal),
    ("yuqori", TargetEncoder(smooth="auto", cv=te_cv), yuqori_kard),
])
Pipeline([("t", tayyor), ("m", model)])

# frequency encoding
sanoq = df["ustun"].value_counts()
df["ustun_soni"] = df["ustun"].map(sanoq)
QOIDA: kardinallikni o'lcha · handle_unknown qo'y · TargetEncoder ni
       Pipeline ichida · daraxtga yuqori kardinallikda one-hot qilma

< 10 daraja: OneHot; ordinal belgi: OrdinalEncoder (tartib bilan)
50+ daraja: TargetEncoder (out-of-fold + smoothing) yoki Frequency
Yangi daraja: handle_unknown MAJBURIY
LightGBM/CatBoost: ichki categorical mexanizm eng yaxshi

"""Asosiy kodlash usullari (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler


def yarat(seed: int = 5, n: int = 6000) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    hudud = rng.choice(["toshkent", "samarqand", "buxoro", "fargona",
                        "namangan"], n, p=[0.35, 0.2, 0.15, 0.2, 0.1])
    daraja = rng.choice(["bronza", "kumush", "oltin", "platina"], n,
                        p=[0.5, 0.3, 0.15, 0.05])
    kanal = rng.choice(["ilova", "sayt", "qongiroq"], n, p=[0.5, 0.35, 0.15])
    summa = rng.lognormal(12.0, 0.6, n)
    hq = pd.Series(hudud).map({"toshkent": 0.0, "samarqand": 0.45,
                               "buxoro": 0.9, "fargona": 0.35,
                               "namangan": 0.6}).to_numpy()
    dq = pd.Series(daraja).map({"bronza": 0.0, "kumush": -0.4,
                                "oltin": -0.9, "platina": -1.5}).to_numpy()
    kuch = -1.6 + hq + dq + 0.35 * (kanal == "qongiroq") \
        + 0.4 * (np.log(summa) - 12.0)
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    return pd.DataFrame({"hudud": hudud, "daraja": daraja, "kanal": kanal,
                         "summa": summa, "kechikdi": y})


def main() -> None:
    df = yarat()
    y = df["kechikdi"].to_numpy()
    cv = StratifiedKFold(5, shuffle=True, random_state=0)
    kategoriyali = ["hudud", "daraja", "kanal"]

    print("=== 1. Kardinallik ===")
    print(f"  {'ustun':<10} {'darajalar':>10} {'eng kam uchraydigan':>21}")
    for ustun in kategoriyali:
        sanoq = df[ustun].value_counts()
        print(f"  {ustun:<10} {df[ustun].nunique():>10} "
              f"{sanoq.iloc[-1] / len(df):>20.2%}")

    print("\n=== 2. One-hot va ordinal (barcha ustunlarga) ===")
    tartib = ["bronza", "kumush", "oltin", "platina"]
    variantlar = {
        "OneHot (hammasi)": ColumnTransformer([
            ("k", OneHotEncoder(handle_unknown="ignore",
                                sparse_output=False), kategoriyali),
            ("s", StandardScaler(), ["summa"])]),
        "Ordinal (hammasi)": ColumnTransformer([
            ("k", OrdinalEncoder(handle_unknown="use_encoded_value",
                                 unknown_value=-1), kategoriyali),
            ("s", StandardScaler(), ["summa"])]),
        "Aralash (to'g'ri)": ColumnTransformer([
            ("nom", OneHotEncoder(handle_unknown="ignore",
                                  sparse_output=False),
             ["hudud", "kanal"]),
            ("ord", OrdinalEncoder(categories=[tartib],
                                   handle_unknown="use_encoded_value",
                                   unknown_value=-1), ["daraja"]),
            ("s", StandardScaler(), ["summa"])]),
    }
    print(f"  {'variant':<20} {'LogReg':>9} {'HistGB':>9}")
    for nom, tayyor in variantlar.items():
        lr = cross_val_score(Pipeline([("t", tayyor),
                                       ("m", LogisticRegression(max_iter=2000))]),
                             df, y, cv=cv, scoring="roc_auc").mean()
        gb = cross_val_score(Pipeline([("t", tayyor),
                                       ("m", HistGradientBoostingClassifier(
                                           learning_rate=0.1, max_iter=200,
                                           random_state=0))]),
                             df, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {nom:<20} {lr:>9.4f} {gb:>9.4f}")

    print("\n=== 3. drop='first' ning ta'siri ===")
    for drop in [None, "first"]:
        tayyor = ColumnTransformer([
            ("k", OneHotEncoder(handle_unknown="ignore", drop=drop,
                             sparse_output=False),
             kategoriyali),
            ("s", StandardScaler(), ["summa"])])
        ustunlar = tayyor.fit(df).transform(df[:5]).shape[1]
        lr = cross_val_score(Pipeline([("t", tayyor),
                                       ("m", LogisticRegression(max_iter=2000))]),
                             df, y, cv=cv, scoring="roc_auc").mean()
        nom = "None" if drop is None else drop
        print(f"  drop={nom:<6}: {ustunlar} ustun, LogReg AUC {lr:.4f}")

    print("\n=== 4. Ordinal kodlash nominal belgiga nima qiladi ===")
    kodlar = {h: i for i, h in enumerate(sorted(df["hudud"].unique()))}
    df["hudud_kod"] = df["hudud"].map(kodlar)
    haqiqiy = {"toshkent": 0.0, "samarqand": 0.45, "buxoro": 0.9,
               "fargona": 0.35, "namangan": 0.6}
    print(f"  {'hudud':<12} {'alifbo kodi':>12} {'haqiqiy risk':>14}")
    for h in sorted(df["hudud"].unique()):
        print(f"  {h:<12} {kodlar[h]:>12} {haqiqiy[h]:>14.2f}")
    korr = np.corrcoef(df["hudud_kod"],
                       df["hudud"].map(haqiqiy))[0, 1]
    print(f"  alifbo kodi va risk korrelyatsiyasi: {korr:+.4f}")
    print("  ⭐ Nominal belgiga tartib berish - yolg'on ma'lumot")


if __name__ == "__main__":
    main()

=== 1. Kardinallik ===
  ustun       darajalar   eng kam uchraydigan
  hudud               5               10.10%
  daraja              4                4.95%
  kanal               3               15.65%

=== 2. One-hot va ordinal (barcha ustunlarga) ===
  variant                 LogReg    HistGB
  OneHot (hammasi)        0.6260    0.5544
  Ordinal (hammasi)       0.6145    0.5604
  Aralash (to'g'ri)       0.6279    0.5538

=== 3. drop='first' ning ta'siri ===
  drop=None  : 13 ustun, LogReg AUC 0.6260
  drop=first : 10 ustun, LogReg AUC 0.6260

=== 4. Ordinal kodlash nominal belgiga nima qiladi ===
  hudud         alifbo kodi   haqiqiy risk
  buxoro                  0           0.90
  fargona                 1           0.35
  namangan                2           0.60
  samarqand               3           0.45
  toshkent                4           0.00
  alifbo kodi va risk korrelyatsiyasi: -0.8438
  ⭐ Nominal belgiga tartib berish - yolg'on ma'lumot

"""Kuchli usul va uning xavfi (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import TargetEncoder


def yarat(seed: int = 8, n: int = 8000, darajalar: int = 200) -> pd.DataFrame:
    """Yuqori kardinallikdagi belgi: har tumanning o'z riski."""
    rng = np.random.default_rng(seed)
    tuman = rng.integers(0, darajalar, n)
    tuman_riski = rng.normal(0, 1.0, darajalar)
    summa = rng.lognormal(12.0, 0.6, n)
    kuch = -1.8 + 1.2 * tuman_riski[tuman] + 0.4 * (np.log(summa) - 12.0)
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    return pd.DataFrame({"tuman": tuman.astype(str), "summa": summa,
                         "kechikdi": y})


# TargetEncoder ichki bo'linishi: takrorlanish uchun seed li splitter
te_cv = StratifiedKFold(5, shuffle=True, random_state=0)


def main() -> None:
    df = yarat()
    y = df["kechikdi"].to_numpy()
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    def model():
        return HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                              random_state=0)

    print("=== 1. Ma'lumot ===")
    print(f"  {len(df)} qator, tuman darajalari {df['tuman'].nunique()}")
    sanoq = df["tuman"].value_counts()
    print(f"  har tumanda: mediana {int(sanoq.median())}, "
          f"min {int(sanoq.min())}, max {int(sanoq.max())}")
    print(f"  kechikish ulushi {y.mean():.2%}")

    print("\n=== 2. NOTO'G'RI: qo'lda target encoding ===")
    ulush = df.groupby("tuman")["kechikdi"].mean()
    df["tuman_notogri"] = df["tuman"].map(ulush)
    b_notogri = cross_val_score(model(), df[["tuman_notogri", "summa"]], y,
                                cv=cv, scoring="roc_auc")
    print(f"  CV ROC AUC: {b_notogri.mean():.4f} (+-{b_notogri.std():.4f})")
    print("  SHUBHA: qatorning o'zi o'rtachaga hissa qo'shgan")

    print("\n=== 3. TO'G'RI: TargetEncoder (Pipeline ichida) ===")
    from sklearn.compose import ColumnTransformer
    tayyor = ColumnTransformer([
        ("te", TargetEncoder(smooth="auto", cv=te_cv), ["tuman"]),
        ("s", "passthrough", ["summa"])])
    b_togri = cross_val_score(Pipeline([("t", tayyor), ("m", model())]),
                              df[["tuman", "summa"]], y, cv=cv,
                              scoring="roc_auc")
    print(f"  CV ROC AUC: {b_togri.mean():.4f} (+-{b_togri.std():.4f})")

    print("\n=== 4. Halol test to'plamida tekshirish ===")
    Xtr, Xte, ytr, yte = train_test_split(df[["tuman", "summa"]], y,
                                          test_size=0.3, random_state=0,
                                          stratify=y)
    from sklearn.metrics import roc_auc_score
    # noto'g'ri usul: kodlash butun ma'lumotda
    ulush_hammasi = df.groupby("tuman")["kechikdi"].mean()
    umumiy = y.mean()
    Xtr2 = Xtr.assign(tuman=Xtr["tuman"].map(ulush_hammasi).fillna(umumiy))
    Xte2 = Xte.assign(tuman=Xte["tuman"].map(ulush_hammasi).fillna(umumiy))
    m1 = model().fit(Xtr2, ytr)
    a_notogri = roc_auc_score(yte, m1.predict_proba(Xte2)[:, 1])
    # to'g'ri usul: TargetEncoder faqat o'quvda o'qitiladi
    quvur = Pipeline([("t", tayyor), ("m", model())]).fit(Xtr, ytr)
    a_togri = roc_auc_score(yte, quvur.predict_proba(Xte)[:, 1])
    print(f"  {'usul':<34} {'CV AUC':>9} {'test AUC':>10}")
    print(f"  {'qo_lda (butun malumotda)':<34} {b_notogri.mean():>9.4f} "
          f"{a_notogri:>10.4f}")
    print(f"  {'TargetEncoder (Pipeline ichida)':<34} {b_togri.mean():>9.4f} "
          f"{a_togri:>10.4f}")
    print(f"  CV va test farqi: qo'lda {b_notogri.mean() - a_notogri:+.4f}, "
          f"TargetEncoder {b_togri.mean() - a_togri:+.4f}")
    print("  ⭐ TargetEncoder out-of-fold va smoothing ni o'zi bajaradi")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  8000 qator, tuman darajalari 200
  har tumanda: mediana 40, min 25, max 55
  kechikish ulushi 19.61%

=== 2. NOTO'G'RI: qo'lda target encoding ===
  CV ROC AUC: 0.7393 (+-0.0098)
  SHUBHA: qatorning o'zi o'rtachaga hissa qo'shgan

=== 3. TO'G'RI: TargetEncoder (Pipeline ichida) ===
  CV ROC AUC: 0.6885 (+-0.0119)

=== 4. Halol test to'plamida tekshirish ===
  usul                                  CV AUC   test AUC
  qo_lda (butun malumotda)              0.7393     0.7384
  TargetEncoder (Pipeline ichida)       0.6885     0.6794
  CV va test farqi: qo'lda +0.0009, TargetEncoder +0.0091
  ⭐ TargetEncoder out-of-fold va smoothing ni o'zi bajaradi

"""Besh usul, bitta vazifa (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, TargetEncoder


def yarat(seed: int = 12, n: int = 10000, darajalar: int = 800) -> pd.DataFrame:
    """Pochta indeksi: ierarxik tuzilma (viloyat -> tuman)."""
    rng = np.random.default_rng(seed)
    viloyat = rng.integers(0, 12, darajalar)          # har indeks bir viloyatda
    viloyat_riski = rng.normal(0, 0.8, 12)
    tuman_riski = viloyat_riski[viloyat] + rng.normal(0, 0.4, darajalar)
    indeks = rng.integers(0, darajalar, n)
    summa = rng.lognormal(12.0, 0.6, n)
    kuch = -1.7 + 1.2 * tuman_riski[indeks] + 0.35 * (np.log(summa) - 12.0)
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    return pd.DataFrame({"indeks": indeks.astype(str),
                         "viloyat": viloyat[indeks].astype(str),
                         "summa": summa, "kechikdi": y})


# TargetEncoder ichki bo'linishi: takrorlanish uchun seed li splitter
te_cv = StratifiedKFold(5, shuffle=True, random_state=0)


def main() -> None:
    df = yarat()
    y = df["kechikdi"].to_numpy()
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    def model():
        return HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                              random_state=0)

    print("=== 1. Kardinallik ===")
    print(f"  indeks: {df['indeks'].nunique()} daraja")
    print(f"  viloyat: {df['viloyat'].nunique()} daraja")
    sanoq = df["indeks"].value_counts()
    print(f"  indeks bo'yicha: mediana {int(sanoq.median())} qator, "
          f"min {int(sanoq.min())}")

    print("\n=== 2. Besh strategiya ===")
    df["indeks_soni"] = df["indeks"].map(df["indeks"].value_counts())
    strategiyalar = {
        "faqat summa": (["summa"], "passthrough"),
        "OneHot (indeks)": (["indeks", "summa"], ColumnTransformer([
            ("oh", OneHotEncoder(handle_unknown="ignore",
                                sparse_output=False), ["indeks"]),
            ("s", "passthrough", ["summa"])])),
        "OneHot (viloyat)": (["viloyat", "summa"], ColumnTransformer([
            ("oh", OneHotEncoder(handle_unknown="ignore",
                                sparse_output=False), ["viloyat"]),
            ("s", "passthrough", ["summa"])])),
        "Target (indeks)": (["indeks", "summa"], ColumnTransformer([
            ("te", TargetEncoder(smooth="auto", cv=te_cv),
             ["indeks"]),
            ("s", "passthrough", ["summa"])])),
        "Target + Frequency": (["indeks", "indeks_soni", "summa"],
                               ColumnTransformer([
                                   ("te", TargetEncoder(smooth="auto", cv=te_cv),
                                    ["indeks"]),
                                   ("s", "passthrough",
                                    ["indeks_soni", "summa"])])),
    }
    print(f"  {'strategiya':<22} {'belgilar':>10} {'CV ROC AUC':>12} "
          f"{'std':>8}")
    for nom, (ustunlar, tayyor) in strategiyalar.items():
        quvur = (model() if tayyor == "passthrough"
                 else Pipeline([("t", tayyor), ("m", model())]))
        Xa = df[ustunlar]
        nechta = (len(ustunlar) if tayyor == "passthrough"
                  else tayyor.fit(Xa, y).transform(Xa[:5]).shape[1])
        b = cross_val_score(quvur, Xa, y, cv=cv, scoring="roc_auc")
        print(f"  {nom:<22} {nechta:>10} {b.mean():>12.4f} {b.std():>8.4f}")

    print("\n=== 3. Ierarxiya: indeks va viloyat birga ===")
    tayyor = ColumnTransformer([
        ("te", TargetEncoder(smooth="auto", cv=te_cv),
         ["indeks", "viloyat"]),
        ("s", "passthrough", ["indeks_soni", "summa"])])
    b = cross_val_score(Pipeline([("t", tayyor), ("m", model())]),
                        df[["indeks", "viloyat", "indeks_soni", "summa"]], y,
                        cv=cv, scoring="roc_auc")
    print(f"  Target(indeks + viloyat) + Frequency: {b.mean():.4f} "
          f"(+-{b.std():.4f})")

    print("\n=== 4. smooth parametrining ta'siri ===")
    print(f"  {'smooth':>10} {'CV ROC AUC':>12}")
    for smooth in ["auto", 1.0, 10.0, 50.0, 200.0]:
        tayyor = ColumnTransformer([
            ("te", TargetEncoder(smooth=smooth, cv=te_cv),
             ["indeks"]),
            ("s", "passthrough", ["summa"])])
        b = cross_val_score(Pipeline([("t", tayyor), ("m", model())]),
                            df[["indeks", "summa"]], y, cv=cv,
                            scoring="roc_auc").mean()
        print(f"  {str(smooth):>10} {b:>12.4f}")
    print("  (katta smooth: kam uchraydigan darajalar umumiy o'rtachaga tortiladi)")
    print("  ⭐ Yuqori kardinallikda Target + Frequency + ierarxiya")


if __name__ == "__main__":
    main()

=== 1. Kardinallik ===
  indeks: 800 daraja
  viloyat: 12 daraja
  indeks bo'yicha: mediana 12 qator, min 4

=== 2. Besh strategiya ===
  strategiya               belgilar   CV ROC AUC      std
  faqat summa                     1       0.5309   0.0168
  OneHot (indeks)               801       0.5325   0.0167
  OneHot (viloyat)               13       0.6891   0.0115
  Target (indeks)                 2       0.6637   0.0085
  Target + Frequency              3       0.6627   0.0111

=== 3. Ierarxiya: indeks va viloyat birga ===
  Target(indeks + viloyat) + Frequency: 0.6990 (+-0.0086)

=== 4. smooth parametrining ta'siri ===
      smooth   CV ROC AUC
        auto       0.6637
         1.0       0.6625
        10.0       0.6621
        50.0       0.6611
       200.0       0.6643
  (katta smooth: kam uchraydigan darajalar umumiy o'rtachaga tortiladi)
  ⭐ Yuqori kardinallikda Target + Frequency + ierarxiya

"""handle_unknown va vaqt bo'yicha o'zgarish (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, TargetEncoder


def yarat(seed: int, n: int, darajalar: int, boshlanish: int = 0):
    """Vaqt o'tishi bilan yangi mahsulotlar paydo bo'ladi."""
    rng = np.random.default_rng(seed)
    mahsulot = rng.integers(boshlanish, boshlanish + darajalar, n)
    xavf = np.sin(mahsulot * 0.7) * 0.9
    summa = rng.lognormal(11.8, 0.6, n)
    kuch = -1.6 + 1.1 * xavf + 0.35 * (np.log(summa) - 11.8)
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    return pd.DataFrame({"mahsulot": mahsulot.astype(str), "summa": summa}), y


# TargetEncoder ichki bo'linishi: takrorlanish uchun seed li splitter
te_cv = StratifiedKFold(5, shuffle=True, random_state=0)


def main() -> None:
    Xtr, ytr = yarat(1, 6000, 60, 0)           # o'quv: 0-59 mahsulot
    Xte, yte = yarat(2, 3000, 80, 0)           # test: 0-79 (20 tasi yangi)

    print("=== 1. Yangi darajalar ===")
    oquv_darajalari = set(Xtr["mahsulot"])
    yangi = ~Xte["mahsulot"].isin(oquv_darajalari)
    print(f"  o'quvda {len(oquv_darajalari)} mahsulot")
    print(f"  testda {Xte['mahsulot'].nunique()} mahsulot")
    print(f"  yangi darajali qatorlar: {int(yangi.sum())} "
          f"({yangi.mean():.1%})")

    def model():
        return HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                              random_state=0)

    print("\n=== 2. handle_unknown siz nima bo'ladi ===")
    try:
        t = ColumnTransformer([("oh", OneHotEncoder(sparse_output=False),
                                ["mahsulot"]),
                               ("s", "passthrough", ["summa"])])
        Pipeline([("t", t), ("m", model())]).fit(Xtr, ytr).predict(Xte)
        print("  ishladi (kutilmagan)")
    except ValueError as xato:
        # xabardagi darajalar tartibi barqaror emas - boshini olamiz
        print(f"  ValueError: {str(xato).split('[')[0].strip()}")
        nomalum = sorted(set(Xte["mahsulot"]) - oquv_darajalari, key=int)
        print(f"  noma'lum darajalar: {len(nomalum)} ta, "
              f"birinchilari {nomalum[:4]}")

    print("\n=== 3. Uch strategiya bilan ===")
    variantlar = {
        "OneHot(ignore)": ColumnTransformer([
            ("oh", OneHotEncoder(handle_unknown="ignore",
                                sparse_output=False), ["mahsulot"]),
            ("s", "passthrough", ["summa"])]),
        "Ordinal(-1)": ColumnTransformer([
            ("od", OrdinalEncoder(handle_unknown="use_encoded_value",
                                  unknown_value=-1), ["mahsulot"]),
            ("s", "passthrough", ["summa"])]),
        "TargetEncoder": ColumnTransformer([
            ("te", TargetEncoder(smooth="auto", cv=te_cv),
             ["mahsulot"]),
            ("s", "passthrough", ["summa"])]),
    }
    print(f"  {'strategiya':<16} {'test AUC':>10} {'eski darajalar':>16} "
          f"{'yangi darajalar':>17}")
    for nom, tayyor in variantlar.items():
        quvur = Pipeline([("t", tayyor), ("m", model())]).fit(Xtr, ytr)
        p = quvur.predict_proba(Xte)[:, 1]
        eski_auc = roc_auc_score(yte[~yangi], p[~yangi])
        yangi_auc = (roc_auc_score(yte[yangi], p[yangi])
                     if len(set(yte[yangi])) > 1 else float("nan"))
        print(f"  {nom:<16} {roc_auc_score(yte, p):>10.4f} "
              f"{eski_auc:>16.4f} {yangi_auc:>17.4f}")

    print("\n=== 4. Kam uchraydigan darajalarni guruhlash ===")
    print(f"  {'min_frequency':>14} {'ustunlar':>10} {'test AUC':>10}")
    for mf in [None, 10, 50, 150]:
        tayyor = ColumnTransformer([
            ("oh", OneHotEncoder(handle_unknown="ignore", min_frequency=mf,
                           sparse_output=False),
             ["mahsulot"]),
            ("s", "passthrough", ["summa"])])
        quvur = Pipeline([("t", tayyor), ("m", model())]).fit(Xtr, ytr)
        ustunlar = tayyor.transform(Xtr[:5]).shape[1]
        a = roc_auc_score(yte, quvur.predict_proba(Xte)[:, 1])
        nom = "None" if mf is None else str(mf)
        print(f"  {nom:>14} {ustunlar:>10} {a:>10.4f}")
    print("  ⭐ handle_unknown siz Pipeline ishlab chiqarishda buziladi")


if __name__ == "__main__":
    main()

=== 1. Yangi darajalar ===
  o'quvda 60 mahsulot
  testda 80 mahsulot
  yangi darajali qatorlar: 773 (25.8%)

=== 2. handle_unknown siz nima bo'ladi ===
  ValueError: Found unknown categories
  noma'lum darajalar: 20 ta, birinchilari ['60', '61', '62', '63']

=== 3. Uch strategiya bilan ===
  strategiya         test AUC   eski darajalar   yangi darajalar
  OneHot(ignore)       0.6176           0.6469            0.5021
  Ordinal(-1)          0.6044           0.6401            0.4899
  TargetEncoder        0.6150           0.6447            0.4974

=== 4. Kam uchraydigan darajalarni guruhlash ===
   min_frequency   ustunlar   test AUC
            None         61     0.6176
              10         61     0.6176
              50         61     0.6176
             150          2     0.5332
  ⭐ handle_unknown siz Pipeline ishlab chiqarishda buziladi

OrdinalEncoder().fit_transform(df[["hudud"]])                     # ⚠️
OneHotEncoder(handle_unknown="ignore").fit_transform(df[["hudud"]]) # ✅

OneHotEncoder()                  # yangi daraja -> ValueError     # ⚠️
OneHotEncoder(handle_unknown="ignore")                            # ✅

df["kod"] = df.groupby("g")["y"].transform("mean")                # ⚠️
TargetEncoder(smooth="auto", cv=te_cv)   # Pipeline ichida            # ✅

OneHotEncoder().fit(df[["mahsulot_id"]])   # 12 000 ustun         # ⚠️
TargetEncoder() + frequency encoding                              # ✅

OneHotEncoder(drop="first")      # RandomForest uchun             # ⚠️
OneHotEncoder(handle_unknown="ignore")                            # ✅

OneHotEncoder()                  # 300 daraja, 200 tasi < 5 qator # ⚠️
OneHotEncoder(min_frequency=20, handle_unknown="ignore")          # ✅

# faqat pochta indeksi (8000 daraja)                              # ⚠️
# indeks + tuman + viloyat birga                                  # ✅

1.  # one-hot qachon mos?
2.  # ordinal qachon?
3.  # target encoding g'oyasi?
4.  # uning ikki xavfi qanday hal qilinadi?
5.  # sklearn dagi klass nomi?
6.  # frequency encoding afzalligi?
7.  # yuqori kardinallikda nima?
8.  # ierarxiya nima beradi?
9.  # yangi daraja uchun?
10. # min_frequency nima qiladi?
11. # drop='first' kimga?
12. # daraxtga yuqori kardinallikda one-hot?

1.  OrdinalEncoder().fit_transform(df[["hudud"]])

2.  OneHotEncoder()   # ishlab chiqarish uchun

3.  df["kod"] = df.groupby("g")["y"].transform("mean")

4.  OneHotEncoder().fit(df[["mahsulot_id"]])   # 12 000 daraja

5.  OneHotEncoder(drop="first")   # RandomForest