17.7-dars: Feature selection

FILTR (filter): modeldan MUSTAQIL statistik mezon
  + juda tez, katta ma'lumotda ishlaydi
  - belgilar O'ZARO TA'SIRINI ko'rmaydi
  misollar: dispersiya, korrelyatsiya, chi2, ANOVA, mutual information

O'RAM (wrapper): modelni QAYTA-QAYTA o'qitib, to'plamlarni sinash
  + o'zaro ta'sirni hisobga oladi, modelga moslashgan
  - juda qimmat (p ta yoki ko'proq o'qitish)
  misollar: RFE, ketma-ket oldinga/orqaga tanlash

ICHKI (embedded): model o'zi tanlaydi
  + arzon (bitta o'qitish), modelga moslashgan
  - modelga bog'liq
  misollar: Lasso (L1), daraxt muhimligi, SelectFromModel

from sklearn.feature_selection import (SelectKBest, VarianceThreshold, chi2,
                                       f_classif, f_regression,
                                       mutual_info_classif)

VarianceThreshold(threshold=0.0)      # konstantalarni olib tashlash
SelectKBest(f_classif, k=20)          # ANOVA F-testi (chiziqli bog'liqlik)
SelectKBest(mutual_info_classif, k=20)  # NOCHIZIQLI bog'liqlikni ham ko'radi
SelectKBest(chi2, k=20)               # faqat MANFIY BO'LMAGAN belgilar

f_classif / f_regression: CHIZIQLI bog'liqlikni o'lchaydi
  -> nochiziqli bog'liqlikni O'TKAZIB YUBORADI (x^2 kabi)

mutual_info: har qanday bog'liqlikni ko'radi
  + nochiziqlilikni topadi
  - sekinroq, baholash noaniqroq

CHEKLOV: har belgi ALOHIDA baholanadi
  -> yakka o'zi foydasiz, lekin birga kuchli belgilar yo'qoladi
     (klassik misol: XOR)

from sklearn.feature_selection import RFE, RFECV, SequentialFeatureSelector

RFE(estimator, n_features_to_select=20, step=0.1)
RFECV(estimator, min_features_to_select=5, cv=5, scoring="roc_auc")
SequentialFeatureSelector(estimator, n_features_to_select=20,
                          direction="forward", cv=3, n_jobs=-1)

RFE: barcha belgilar bilan o'qitib, eng KUCHSIZINI olib tashlash,
     va shu jarayonni takrorlash
  step=0.1 - har qadamda 10% ni tashlash (tezlashtirish)
  RFECV - optimal sonni CV bilan o'zi topadi

SEQUENTIAL:
  forward  - bo'sh to'plamdan boshlab, eng yaxshi belgini qo'shish
  backward - hammasidan boshlab, eng kamini tashlash

NARXI: RFE ~ p ta o'qitish, Sequential ~ p^2/2 ta o'qitish
  -> 100+ belgida amaliy emas (RFECV bilan step ishlating)

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LassoCV, LogisticRegression

SelectFromModel(LassoCV(cv=5), threshold="median")
SelectFromModel(LogisticRegression(l1_ratio=1.0, solver="liblinear", C=0.1))
SelectFromModel(RandomForestClassifier(300), threshold="1.25*mean")

LASSO (L1): koeffitsiyentlarni aniq NOLGA tushiradi 13.8-bob
  + tanlov avtomatik
  - korrelyatsiyali belgilardan BITTASINI tanlaydi (tasodifiy)
  - masshtablash MAJBURIY

DARAXT MUHIMLIGI: feature_importances_ bo'yicha
  - kardinallikka moyil 15.11-bob -> permutation afzalroq

threshold: "mean", "median", "1.25*mean" yoki aniq son
max_features: qat'iy chegara

from sklearn.inspection import permutation_importance

r = permutation_importance(model, Xval, yval, n_repeats=20,
                           scoring="roc_auc", random_state=0)
tanlangan = [i for i in range(X.shape[1])
             if r.importances_mean[i] > 0]

AFZALLIKLARI 15.11-bob:
  + modelga bog'liq, lekin uning ichki mexanizmidan mustaqil
  + VALIDATSIYA to'plamida hisoblanadi -> overfitting ni ko'radi
  + manfiy muhimlik = belgi ZARAR qiladi

TARTIB:
  1. To'liq model o'qitiladi
  2. Validatsiyada permutation importance
  3. Muhimligi <= 0 bo'lganlar tashlanadi
  4. Qolganlar bilan qayta o'qitiladi va CV da tekshiriladi
  5. Kerak bo'lsa takrorlanadi

EHTIYOT: korrelyatsiyali belgilar bir-birini "yashiradi" -> guruhlang

KORRELYATSIYA:
  0.95+ korrelyatsiyali belgilar bir-birini almashtiradi
  -> har birining muhimligi past ko'rinadi
  -> ikkalasi ham tashlanadi (XATO)

YECHIM 15.11-bob:
  1. Spearman korrelyatsiya matritsasi
  2. Ierarxik klasterlash
  3. Har guruhdan BITTA vakil qoldirish

LEAKAGE:
  NOTO'G'RI: butun ma'lumotda tanlash -> keyin CV
  TO'G'RI:   Pipeline([("tanlov", SelectKBest(...)), ("m", model)])

  Ta'sir: p >> n bo'lganda JUDA katta (0.5 -> 0.9 AUC)

from sklearn.feature_selection import (RFE, RFECV, SelectFromModel, SelectKBest,
                                       VarianceThreshold, f_classif,
                                       mutual_info_classif)
from sklearn.inspection import permutation_importance
from sklearn.pipeline import Pipeline

Pipeline([("v", VarianceThreshold()),
          ("s", SelectKBest(mutual_info_classif, k=30)),
          ("m", model)])                          # DOIM Pipeline ichida

SelectFromModel(LassoCV(cv=5), threshold="median")
RFECV(model, step=0.1, cv=5, scoring="roc_auc", min_features_to_select=5)

r = permutation_importance(model, Xval, yval, n_repeats=20, scoring="roc_auc")
tanlangan = np.where(r.importances_mean > 0)[0]
QOIDA: Pipeline ichida tanla · korrelyatsiyali guruhlarni birlashtir ·
       tanlovdan keyin CV ni qayta o'lcha

Filtr: tez, o'zaro ta'sirni ko'rmaydi (XOR muammosi)
O'ram (RFE/RFECV): aniq, lekin qimmat
Ichki (Lasso/SelectFromModel): arzon, modelga moslashgan
Permutation: validatsiyada, manfiy muhimlik = zararli belgi
Tanlov Pipeline ichida - aks holda leakage

"""Tez, lekin o'zaro ta'sirni ko'rmaydi (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.feature_selection import (SelectKBest, VarianceThreshold,
                                       f_classif, mutual_info_classif)
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline


def yarat(seed: int = 5, n: int = 3000, shovqin: int = 40):
    """Chiziqli, nochiziqli va o'zaro ta'sirli belgilar + shovqin."""
    rng = np.random.default_rng(seed)
    x_chiziqli = rng.normal(0, 1, n)
    x_kvadrat = rng.normal(0, 1, n)
    x_oz1 = rng.normal(0, 1, n)
    x_oz2 = rng.normal(0, 1, n)
    konstanta = np.ones(n)
    shovqin_belgilar = rng.normal(0, 1, (n, shovqin))
    X = np.column_stack([x_chiziqli, x_kvadrat, x_oz1, x_oz2, konstanta,
                         shovqin_belgilar])
    kuch = (1.5 * x_chiziqli + 1.8 * (x_kvadrat ** 2 - 1)
            + 2.2 * x_oz1 * x_oz2)
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    nomlar = ["chiziqli", "kvadrat", "ozaro1", "ozaro2", "konstanta"] \
        + [f"shovqin{i}" for i in range(shovqin)]
    return X, y, nomlar


def main() -> None:
    X, y, nomlar = yarat()
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Ma'lumot ===")
    print(f"  {X.shape[0]} qator, {X.shape[1]} belgi")
    print(f"  foydali: chiziqli, kvadrat (nochiziqli), "
          f"ozaro1*ozaro2 (o'zaro ta'sir)")
    print(f"  foydasiz: konstanta + 40 ta shovqin")

    print("\n=== 2. VarianceThreshold ===")
    vt = VarianceThreshold(threshold=0.0).fit(X)
    tashlangan = [nomlar[i] for i in range(len(nomlar)) if not vt.get_support()[i]]
    print(f"  tashlangan: {tashlangan}")
    print(f"  qolgan: {int(vt.get_support().sum())} belgi")

    print("\n=== 3. Filtr ballari ===")
    # konstanta ustuni f_classif da aniqlanmagan qiymat beradi -
    # shuning uchun ballar VarianceThreshold dan KEYIN hisoblanadi
    X_vt = vt.transform(X)
    f_ball = f_classif(X_vt, y)[0]
    mi_ball = mutual_info_classif(X_vt, y, random_state=0)
    print(f"  {'belgi':<12} {'F-test':>10} {'F o_rni':>9} "
          f"{'mutual info':>13} {'MI o_rni':>10}")
    f_tartib = np.argsort(-f_ball)
    mi_tartib = np.argsort(-mi_ball)
    for i in range(4):
        f_orin = int(np.where(f_tartib == i)[0][0]) + 1
        mi_orin = int(np.where(mi_tartib == i)[0][0]) + 1
        print(f"  {nomlar[i]:<12} {f_ball[i]:>10.2f} {f_orin:>9} "
              f"{mi_ball[i]:>13.4f} {mi_orin:>10}")
    print("  (F-test kvadrat va o'zaro ta'sirni ko'rmaydi)")

    print("\n=== 4. Tanlov natijasi ===")
    def model():
        return HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                              random_state=0)
    variantlar = {
        "barcha belgilar": None,
        "SelectKBest(f_classif, 4)": SelectKBest(f_classif, k=4),
        "SelectKBest(f_classif, 10)": SelectKBest(f_classif, k=10),
        "SelectKBest(mutual_info, 4)": SelectKBest(mutual_info_classif, k=4),
        "SelectKBest(mutual_info, 10)": SelectKBest(mutual_info_classif, k=10),
        "faqat 4 ta foydali": "qolda",
    }
    print(f"  {'variant':<30} {'belgilar':>9} {'CV ROC AUC':>12}")
    for nom, tanlov in variantlar.items():
        if tanlov is None:
            Xa, quvur, nechta = X, model(), X.shape[1]
        elif tanlov == "qolda":
            Xa, quvur, nechta = X[:, :4], model(), 4
        else:
            Xa = X
            quvur = Pipeline([("v", VarianceThreshold(threshold=0.0)),
                              ("s", tanlov), ("m", model())])
            nechta = tanlov.k
        b = cross_val_score(quvur, Xa, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {nom:<30} {nechta:>9} {b:>12.4f}")
    print("  ⭐ Filtr usullari o'zaro ta'sirni ko'rmaydi")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  3000 qator, 45 belgi
  foydali: chiziqli, kvadrat (nochiziqli), ozaro1*ozaro2 (o'zaro ta'sir)
  foydasiz: konstanta + 40 ta shovqin

=== 2. VarianceThreshold ===
  tashlangan: ['konstanta']
  qolgan: 44 belgi

=== 3. Filtr ballari ===
  belgi            F-test   F o_rni   mutual info   MI o_rni
  chiziqli         299.13         1        0.0497          2
  kvadrat            0.14        35        0.1059          1
  ozaro1             0.29        28        0.0000         29
  ozaro2             0.47        24        0.0084         10
  (F-test kvadrat va o'zaro ta'sirni ko'rmaydi)

=== 4. Tanlov natijasi ===
  variant                         belgilar   CV ROC AUC
  barcha belgilar                       45       0.8570
  SelectKBest(f_classif, 4)              4       0.6106
  SelectKBest(f_classif, 10)            10       0.6042
  SelectKBest(mutual_info, 4)            4       0.7661
  SelectKBest(mutual_info, 10)          10       0.7761
  faqat 4 ta foydali                     4       0.8848
  ⭐ Filtr usullari o'zaro ta'sirni ko'rmaydi

"""Uch yondashuvni taqqoslash (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.feature_selection import RFE, RFECV, SelectFromModel
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (StratifiedKFold, cross_val_score,
                                     train_test_split)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 9, n: int = 3000, foydali: int = 8, shovqin: int = 60):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, foydali + shovqin))
    ogirlik = rng.choice([-1.5, -1.0, 1.0, 1.5], foydali)
    kuch = X[:, :foydali] @ ogirlik
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    return X, y, foydali


def aniqlik_qamrov(tanlangan, foydali: int, jami: int):
    """Tanlangan to'plamda foydali belgilar qanchasi bor."""
    tanlangan = np.asarray(tanlangan)
    togri = int((tanlangan < foydali).sum())
    return togri / max(len(tanlangan), 1), togri / foydali


def main() -> None:
    X, y, foydali = yarat()
    cv = StratifiedKFold(5, shuffle=True, random_state=0)
    jami = X.shape[1]

    print("=== 1. Ma'lumot ===")
    print(f"  {X.shape[0]} qator, {jami} belgi "
          f"({foydali} foydali, {jami - foydali} shovqin)")
    asos = cross_val_score(
        HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                       random_state=0),
        X, y, cv=cv, scoring="roc_auc").mean()
    print(f"  barcha belgilar bilan CV ROC AUC: {asos:.4f}")

    print("\n=== 2. Lasso (L1) bilan tanlash ===")
    print(f"  {'C':>8} {'tanlangan':>11} {'aniqlik':>9} {'qamrov':>9} "
          f"{'CV AUC':>9}")
    for C in [1.0, 0.1, 0.03, 0.01]:
        sfm = SelectFromModel(LogisticRegression(l1_ratio=1.0,
                                                 solver="liblinear", C=C,
                                                 max_iter=2000))
        quvur = Pipeline([("sc", StandardScaler()), ("s", sfm),
                          ("m", HistGradientBoostingClassifier(
                              learning_rate=0.1, max_iter=200,
                              random_state=0))])
        sfm_fit = Pipeline([("sc", StandardScaler()),
                            ("s", sfm)]).fit(X, y)
        tanlangan = np.where(sfm_fit.named_steps["s"].get_support())[0]
        a, q = aniqlik_qamrov(tanlangan, foydali, jami)
        b = cross_val_score(quvur, X, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {C:>8.2f} {len(tanlangan):>11} {a:>9.2%} {q:>9.2%} "
              f"{b:>9.4f}")

    print("\n=== 3. RFE ===")
    print(f"  {'n_features':>11} {'aniqlik':>9} {'qamrov':>9} {'CV AUC':>9}")
    for k in [5, 8, 15, 30]:
        rfe = RFE(LogisticRegression(max_iter=2000), n_features_to_select=k,
                  step=0.2)
        quvur = Pipeline([("sc", StandardScaler()), ("s", rfe),
                          ("m", HistGradientBoostingClassifier(
                              learning_rate=0.1, max_iter=200,
                              random_state=0))])
        fit = Pipeline([("sc", StandardScaler()), ("s", rfe)]).fit(X, y)
        tanlangan = np.where(fit.named_steps["s"].get_support())[0]
        a, q = aniqlik_qamrov(tanlangan, foydali, jami)
        b = cross_val_score(quvur, X, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {k:>11} {a:>9.2%} {q:>9.2%} {b:>9.4f}")

    print("\n=== 4. Permutation importance bilan ===")
    Xtr, Xval, ytr, yval = train_test_split(X, y, test_size=0.3,
                                            random_state=0, stratify=y)
    m = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                       random_state=0).fit(Xtr, ytr)
    r = permutation_importance(m, Xval, yval, n_repeats=10, scoring="roc_auc",
                               random_state=0, n_jobs=1)
    print(f"  {'chegara':>12} {'tanlangan':>11} {'aniqlik':>9} "
          f"{'qamrov':>9} {'CV AUC':>9}")
    for chegara in [0.0, 0.001, 0.005]:
        tanlangan = np.where(r.importances_mean > chegara)[0]
        if len(tanlangan) < 2:
            continue
        a, q = aniqlik_qamrov(tanlangan, foydali, jami)
        b = cross_val_score(
            HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                           random_state=0),
            X[:, tanlangan], y, cv=cv, scoring="roc_auc").mean()
        print(f"  {chegara:>12.3f} {len(tanlangan):>11} {a:>9.2%} "
              f"{q:>9.2%} {b:>9.4f}")
    manfiy = int((r.importances_mean < 0).sum())
    print(f"  manfiy muhimlikdagi belgilar: {manfiy} "
          f"(ular ZARAR qiladi)")
    print("  ⭐ Permutation validatsiyada hisoblanadi")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  3000 qator, 68 belgi (8 foydali, 60 shovqin)
  barcha belgilar bilan CV ROC AUC: 0.9066

=== 2. Lasso (L1) bilan tanlash ===
         C   tanlangan   aniqlik    qamrov    CV AUC
      1.00          66    12.12%   100.00%    0.9067
      0.10          41    19.51%   100.00%    0.9097
      0.03          10    80.00%   100.00%    0.9095
      0.01           8   100.00%   100.00%    0.9089

=== 3. RFE ===
   n_features   aniqlik    qamrov    CV AUC
            5   100.00%    62.50%    0.8363
            8   100.00%   100.00%    0.9089
           15    53.33%   100.00%    0.9101
           30    26.67%   100.00%    0.9117

=== 4. Permutation importance bilan ===
       chegara   tanlangan   aniqlik    qamrov    CV AUC
         0.000          32    25.00%   100.00%    0.9109
         0.001           8   100.00%   100.00%    0.9089
         0.005           8   100.00%   100.00%    0.9089
  manfiy muhimlikdagi belgilar: 36 (ular ZARAR qiladi)
  ⭐ Permutation validatsiyada hisoblanadi

"""Bir-birini yashiradigan belgilar (real numpy/scipy/sklearn)."""

import numpy as np
from scipy.cluster.hierarchy import fcluster, linkage
from scipy.spatial.distance import squareform
from scipy.stats import spearmanr
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.inspection import permutation_importance
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split


def yarat(seed: int = 11, n: int = 3000):
    """Uchta korrelyatsiyali nusxa + mustaqil belgilar + shovqin."""
    rng = np.random.default_rng(seed)
    asos = rng.normal(0, 1, n)
    nusxalar = np.column_stack([asos + rng.normal(0, 0.15, n) for _ in range(3)])
    mustaqil = rng.normal(0, 1, (n, 2))
    shovqin = rng.normal(0, 1, (n, 25))
    X = np.column_stack([nusxalar, mustaqil, shovqin])
    kuch = 1.8 * asos + 1.2 * mustaqil[:, 0] - 1.0 * mustaqil[:, 1]
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    nomlar = ["nusxa0", "nusxa1", "nusxa2", "mustaqil0", "mustaqil1"] \
        + [f"shovqin{i}" for i in range(25)]
    return X, y, nomlar


def main() -> None:
    X, y, nomlar = yarat()
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Korrelyatsiya ===")
    print(f"  nusxa0 - nusxa1: {np.corrcoef(X[:, 0], X[:, 1])[0, 1]:.4f}")
    print(f"  nusxa0 - mustaqil0: {np.corrcoef(X[:, 0], X[:, 3])[0, 1]:.4f}")

    print("\n=== 2. Permutation muhimligi (guruhsiz) ===")
    Xtr, Xval, ytr, yval = train_test_split(X, y, test_size=0.3,
                                            random_state=0, stratify=y)
    m = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                       random_state=0).fit(Xtr, ytr)
    r = permutation_importance(m, Xval, yval, n_repeats=15, scoring="roc_auc",
                               random_state=0, n_jobs=1)
    print(f"  {'belgi':<12} {'muhimlik':>11}")
    for i in range(5):
        print(f"  {nomlar[i]:<12} {r.importances_mean[i]:>+11.4f}")
    print(f"  shovqin (max): {r.importances_mean[5:].max():+.4f}")
    print("  (uch nusxa bir-birini yashiradi - muhimligi past)")

    print("\n=== 3. Ierarxik klasterlash bilan guruhlash ===")
    korr = spearmanr(X).correlation
    masofa = 1 - np.abs(korr)
    np.fill_diagonal(masofa, 0.0)
    masofa = (masofa + masofa.T) / 2
    Z = linkage(squareform(masofa, checks=False), "average")
    guruhlar = fcluster(Z, 0.3, criterion="distance")
    katta_guruhlar = {}
    for g in np.unique(guruhlar):
        azolar = [i for i in range(len(nomlar)) if guruhlar[i] == g]
        if len(azolar) > 1:
            katta_guruhlar[g] = azolar
    print(f"  jami guruhlar: {len(np.unique(guruhlar))}")
    for g, azolar in katta_guruhlar.items():
        print(f"  guruh {g}: {[nomlar[i] for i in azolar]}")

    print("\n=== 4. Har guruhdan bitta vakil ===")
    vakillar = []
    for g in np.unique(guruhlar):
        azolar = [i for i in range(len(nomlar)) if guruhlar[i] == g]
        # guruhdagi eng yuqori muhimlikdagini olamiz
        vakillar.append(max(azolar, key=lambda i: r.importances_mean[i]))
    def model():
        return HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                              random_state=0)
    variantlar = {
        "barcha belgilar": list(range(len(nomlar))),
        "permutation > 0": list(np.where(r.importances_mean > 0)[0]),
        "guruh vakillari": vakillar,
        "vakillar + muhimlik": [i for i in vakillar
                                if r.importances_mean[i] > 0],
        "haqiqiy foydali": [0, 3, 4],
    }
    print(f"  {'variant':<22} {'belgilar':>9} {'CV ROC AUC':>12}")
    for nom, idx in variantlar.items():
        if len(idx) < 2:
            continue
        b = cross_val_score(model(), X[:, idx], y, cv=cv,
                            scoring="roc_auc").mean()
        print(f"  {nom:<22} {len(idx):>9} {b:>12.4f}")
    print("  ⭐ Korrelyatsiyali belgilarni guruhlab tanlang")


if __name__ == "__main__":
    main()

=== 1. Korrelyatsiya ===
  nusxa0 - nusxa1: 0.9780
  nusxa0 - mustaqil0: 0.0115

=== 2. Permutation muhimligi (guruhsiz) ===
  belgi           muhimlik
  nusxa0           +0.0350
  nusxa1           +0.0157
  nusxa2           +0.0319
  mustaqil0        +0.1004
  mustaqil1        +0.0662
  shovqin (max): +0.0027
  (uch nusxa bir-birini yashiradi - muhimligi past)

=== 3. Ierarxik klasterlash bilan guruhlash ===
  jami guruhlar: 28
  guruh 18: ['nusxa0', 'nusxa1', 'nusxa2']

=== 4. Har guruhdan bitta vakil ===
  variant                 belgilar   CV ROC AUC
  barcha belgilar               30       0.8594
  permutation > 0               19       0.8588
  guruh vakillari               28       0.8583
  vakillar + muhimlik           17       0.8583
  haqiqiy foydali                3       0.8541
  ⭐ Korrelyatsiyali belgilarni guruhlab tanlang

"""p >> n da halokatli xato (real numpy/sklearn)."""

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline


def main() -> None:
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. BUTUNLAY TASODIFIY ma'lumot ===")
    rng = np.random.default_rng(0)
    n, p = 120, 6000
    X = rng.normal(0, 1, (n, p))
    y = rng.integers(0, 2, n)               # maqsad X bilan bog'liq EMAS
    print(f"  {n} qator, {p} belgi, maqsad TASODIFIY")
    print(f"  haqiqiy AUC ~ 0.5 bo'lishi kerak")

    print("\n=== 2. NOTO'G'RI: butun ma'lumotda tanlash ===")
    print(f"  {'k':>6} {'CV ROC AUC':>12}")
    for k in [5, 20, 100]:
        Xs = SelectKBest(f_classif, k=k).fit_transform(X, y)
        b = cross_val_score(LogisticRegression(max_iter=2000), Xs, y, cv=cv,
                            scoring="roc_auc").mean()
        print(f"  {k:>6} {b:>12.4f}")
    print("  (tasodifiy ma'lumotda 0.9+ AUC - butunlay yolg'on)")

    print("\n=== 3. TO'G'RI: Pipeline ichida ===")
    print(f"  {'k':>6} {'CV ROC AUC':>12}")
    for k in [5, 20, 100]:
        quvur = Pipeline([("s", SelectKBest(f_classif, k=k)),
                          ("m", LogisticRegression(max_iter=2000))])
        b = cross_val_score(quvur, X, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {k:>6} {b:>12.4f}")
    print("  (0.5 atrofida - to'g'ri)")

    print("\n=== 4. Haqiqiy signal bo'lganda ===")
    rng2 = np.random.default_rng(1)
    n2, p2 = 400, 2000
    X2 = rng2.normal(0, 1, (n2, p2))
    kuch = X2[:, :5] @ np.array([1.5, -1.2, 1.0, -0.8, 1.1])
    y2 = (rng2.random(n2) < 1 / (1 + np.exp(-kuch))).astype(int)
    print(f"  {n2} qator, {p2} belgi, 5 tasi haqiqatan foydali")
    print(f"  {'k':>6} {'leakage bilan':>15} {'Pipeline bilan':>16} "
          f"{'farq':>9}")
    for k in [5, 20, 100]:
        Xs = SelectKBest(f_classif, k=k).fit_transform(X2, y2)
        leak = cross_val_score(LogisticRegression(max_iter=2000), Xs, y2,
                               cv=cv, scoring="roc_auc").mean()
        quvur = Pipeline([("s", SelectKBest(f_classif, k=k)),
                          ("m", LogisticRegression(max_iter=2000))])
        togri = cross_val_score(quvur, X2, y2, cv=cv, scoring="roc_auc").mean()
        print(f"  {k:>6} {leak:>15.4f} {togri:>16.4f} {leak - togri:>+9.4f}")
    print("  ⭐ Belgi tanlash Pipeline ichida bo'lishi SHART")


if __name__ == "__main__":
    main()

=== 1. BUTUNLAY TASODIFIY ma'lumot ===
  120 qator, 6000 belgi, maqsad TASODIFIY
  haqiqiy AUC ~ 0.5 bo'lishi kerak

=== 2. NOTO'G'RI: butun ma'lumotda tanlash ===
       k   CV ROC AUC
       5       0.8316
      20       0.9277
     100       0.9847
  (tasodifiy ma'lumotda 0.9+ AUC - butunlay yolg'on)

=== 3. TO'G'RI: Pipeline ichida ===
       k   CV ROC AUC
       5       0.4277
      20       0.5476
     100       0.5401
  (0.5 atrofida - to'g'ri)

=== 4. Haqiqiy signal bo'lganda ===
  400 qator, 2000 belgi, 5 tasi haqiqatan foydali
       k   leakage bilan   Pipeline bilan      farq
       5          0.9015           0.9015   +0.0000
      20          0.9187           0.8624   +0.0564
     100          0.9455           0.7297   +0.2158
  ⭐ Belgi tanlash Pipeline ichida bo'lishi SHART

Xs = SelectKBest(k=20).fit_transform(X, y); cross_val_score(m, Xs, y)  # ⚠️
cross_val_score(Pipeline([("s", SelectKBest(k=20)), ("m", m)]), X, y)  # ✅

tanlangan = importance > chegara   # uch nusxa ham tashlandi           # ⚠️
# spearmanr + klasterlash -> har guruhdan vakil                        # ✅

SelectFromModel(LassoCV()).fit(X, y)                                   # ⚠️
Pipeline([("sc", StandardScaler()), ("s", SelectFromModel(LassoCV()))]) # ✅

SelectFromModel(RandomForestClassifier(300))    # kardinallik moyilligi # ⚠️
# permutation_importance validatsiyada                                  # ✅

SelectKBest(f_classif, k=20)     # o'zaro ta'sir yo'qoladi             # ⚠️
SelectKBest(mutual_info_classif) yoki permutation                      # ✅

# "20 belgi tanladik, tayyor"                                          # ⚠️
# tanlangan to'plamda CV ni qayta o'lchang                             # ✅

SelectKBest(k=3)                 # AUC 0.84 -> 0.71                    # ⚠️
# bir necha k ni sinang, CV std bilan solishtiring                     # ✅

1.  # uch yondashuv?
2.  # filtr usullarining cheklovi?
3.  # klassik misol?
4.  # RFE qanday ishlaydi?
5.  # RFECV nima qiladi?
6.  # Lasso nimani beqaror qiladi?
7.  # permutation qayerda hisoblanadi?
8.  # manfiy muhimlik nimani anglatadi?
9.  # korrelyatsiyali guruhlar bilan?
10. # tanlov qayerda bo'lishi kerak?
11. # leakage qachon halokatli?
12. # tanlashning uch sababi?

1.  Xs = SelectKBest(k=20).fit_transform(X, y); cross_val_score(m, Xs, y)

2.  SelectFromModel(LassoCV()).fit(X, y)

3.  SelectFromModel(RandomForestClassifier(300))

4.  tanlangan = importance > chegara   # korrelyatsiyali nusxalar bor

5.  SelectKBest(k=3)   # AUC keskin tushdi