IlmHamroh
Data Science va sun'iy intellekt/Daraxtlar va ansambllar12/14-dars19 daqiqa
Mundarija (22)

15.12-dars: Voting va stacking

15-QISM — DARAXTLAR VA ANSAMBLLAR · 12-dars

1. Kirish va motivatsiya

Bagging va boosting bir xil turdagi modellarni birlashtiradi. Voting va stacking esa turli modellarni birlashtiradi: Random Forest, gradient boosting, logistik regressiya va KNN bir vazifada turli xatolar qiladi — ularni birlashtirsak, xatolar bir-birini qoplashi mumkin.

Bu — Kaggle musobaqalarining klassik "oxirgi 1%" usuli, lekin u ishlab chiqarishda ham foydali: turli modellarning xilma-xilligi ansamblning asosiy resursi.

Bu darsda: VotingClassifier (qattiq va yumshoq ovoz), og'irliklar, StackingClassifier, meta-model tanlash, cross_val_predict va leakage, model xilma-xilligini o'lchash hamda ansambl qachon foyda bermasligi.

Real vaziyat. Kredit skoringida LightGBM 0.812, Random Forest 0.796, logistik regressiya 0.781 AUC berdi. Ularning oddiy o'rtachasi 0.819, stacking esa 0.821 chiqdi. Lekin ishlab chiqarishga LightGBM yakka o'zi chiqdi — 0.007 AUC uchun uch modelni saqlash, kuzatish va tushuntirish narxi oqlanmadi.

Bu darsda voting va stacking ni o'rganamiz.

Bu darsda:

  • Voting
  • Stacking
  • Meta-model tanlash
  • Xilma-xillikni o'lchash
  • Leakage xavfi
  • Qachon foyda bermaydi
  • Tuzoqlar
  • Amaliy: to'liq taqqoslash

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Voting

text
VotingClassifier(estimators=[("rf", ...), ("gb", ...), ("lr", ...)],
                 voting="soft",            # yoki "hard"
                 weights=[2, 2, 1])

hard: ko'pchilik sinfi
soft: predict_proba larning (og'irlikli) o'rtachasi  <- odatda YAXSHIROQ

VotingRegressor: bashoratlarning og'irlikli o'rtachasi

Shart: modellar TAXMINAN teng kuchli bo'lsin
  juda kuchsiz model o'rtachani tortadi

Yumshoq ovoz uchun ehtimolliklar taqqoslanadigan bo'lishi kerak 14.10-bob: kalibrlanmagan model (masalan SVM yoki kuchli class_weight li RF) o'rtachani buzadi. Shubha bo'lsa, avval CalibratedClassifierCV bilan kalibrlang.

2.2. Stacking

text
StackingClassifier(estimators=[...], final_estimator=LogisticRegression(),
                   cv=5, stack_method="predict_proba", passthrough=False)

Mexanizm:
  1. Har bazaviy model CV bilan OUT-OF-FOLD bashoratlar beradi
  2. Shu bashoratlar meta-model uchun BELGI bo'ladi
  3. Meta-model ularni birlashtirishni o'rganadi
  4. Bazaviy modellar butun ma'lumotda qayta o'qitiladi

passthrough=True -> asl belgilar ham meta-modelga beriladi

Out-of-fold bashoratlar — stacking ning o'zagi: meta-model bazaviy modellarning ko'rmagan ma'lumotidagi bashoratlarini ko'radi. Aks holda u overfitting qilgan bashoratlarga o'rganadi va yangi ma'lumotda ishlamaydi.

2.3. Meta-model tanlash

text
Meta-model SODDA bo'lishi kerak:
  LogisticRegression (klassifikatsiya) - eng keng tarqalgan
  RidgeCV (regressiya)
  LogisticRegression(C kichik) - kuchliroq regulyarizatsiya

NEGA sodda:
  - kirish belgilari kam (bazaviy modellar soni)
  - ular yuqori korrelyatsiyali
  - murakkab meta-model tez overfitting qiladi

Murakkab meta-model (GB) - faqat ko'p bazaviy model va ko'p ma'lumotda

Meta-modelga LogisticRegression berish — deyarli har doim to'g'ri tanlov. Uning koeffitsiyentlari bonus sifatida qaysi bazaviy model qanchalik ishonchli ekanini ko'rsatadi.

2.4. Xilma-xillikni o'lchash

text
Ansambl foydasi modellar XILMA-XIL bo'lgandagina bo'ladi

O'lchash usullari:
  - bashoratlar korrelyatsiyasi (pastroq = yaxshiroq)
  - kelishmovchilik (disagreement): qancha namunada javob farq qiladi
  - Q-statistika, Kappa

Amaliy qoida:
  korrelyatsiya > 0.98 -> ansambl deyarli foyda bermaydi
  korrelyatsiya < 0.90 -> sezilarli foyda kutish mumkin

Xilma-xillikni oldindan tekshiring: agar uch modelning bashoratlari 0.99 korrelyatsiyaga ega bo'lsa, ansambl qurish vaqtni behuda sarflash bo'ladi.

2.5. Leakage xavfi

text
NOTO'G'RI stacking:
  1. Bazaviy modellarni butun o'quvda o'qitish
  2. Ularning O'QUVdagi bashoratlarini meta-modelga berish
  -> meta-model overfitting qilgan bashoratlarni ko'radi
  -> yangi ma'lumotda ishlamaydi

TO'G'RI:
  cross_val_predict yoki StackingClassifier(cv=5)

Qo'shimcha: tayyorlash (scaler, encoder) ham Pipeline ichida bo'lsin

Bu xatoni qo'lda stacking yozganda qilish oson. StackingClassifier uni o'zi to'g'ri bajaradi — shuning uchun qo'lda yozmang.

2.6. Qachon foyda bermaydi

text
Ansambl foyda BERMAYDI:
  - modellar juda o'xshash (korrelyatsiya > 0.98)
  - bitta model boshqalardan ancha kuchli
  - ma'lumot kam (meta-model overfitting qiladi)
  - xatolar bir xil namunalarda (umumiy sabab: shovqinli yorliq)

NARXI:
  - o'qitish vaqti (N barobar)
  - bashorat vaqti (kechikish)
  - saqlash va kuzatish murakkabligi
  - talqin qilish qiyinlashadi

Foyda odatda 0.3-1.5% - biznes uchun arziydimi?

Narx-foyda tahlili — ansambl bo'yicha qaror qabul qilishning asosiy mezoni. Kaggle da 0.5% yutuq g'alaba bo'lishi mumkin, ishlab chiqarishda esa u uch modelni kuzatish narxini oqlamaydi.

2.7. Tuzoqlar

Asosiy tuzoqlar: qo'lda stacking yozib, out-of-fold ni unutish; murakkab meta-model tanlash; kalibrlanmagan modellarni yumshoq ovozga qo'shish; juda o'xshash modellarni birlashtirish; kuchsiz modelni teng og'irlik bilan qo'shish; n_jobs ni unutish (juda sekin); ansambl foydasini test to'plamida tanlab, shu bilan baholash; ishlab chiqarish narxini hisobga olmaslik.

2.8. Xilma-xillik — asosiy resurs

Voting modellarning bashoratlarini o'rtachalashtiradi (yumshoq ovoz odatda yaxshiroq), stacking esa ularni birlashtirishni o'rganadi — out-of-fold bashoratlar ustida. Meta-model sodda bo'lishi kerak (LogisticRegression). Foyda faqat modellar xilma-xil bo'lganda bo'ladi: korrelyatsiya 0.98 dan yuqori bo'lsa, ansambl qurmang. Odatiy yutuq 0.3-1.5% — uni ishlab chiqarish narxi bilan solishtiring. Keyingi dars — ishlab chiqarishga chiqarish.

3. Tez ma'lumotnoma

python
from sklearn.ensemble import (StackingClassifier, VotingClassifier,
                              VotingRegressor)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict

v = VotingClassifier([("rf", rf), ("gb", gb), ("lr", lr)],
                     voting="soft", weights=[2, 2, 1], n_jobs=-1).fit(X, y)

s = StackingClassifier([("rf", rf), ("gb", gb), ("lr", lr)],
                       final_estimator=LogisticRegression(C=1.0),
                       cv=5, stack_method="predict_proba",
                       passthrough=False, n_jobs=-1).fit(X, y)
s.final_estimator_.coef_          # qaysi model ishonchli

# xilma-xillikni tekshirish
oof = {nom: cross_val_predict(m, X, y, cv=5, method="predict_proba")[:, 1]
       for nom, m in modellar.items()}
QOIDA: soft voting · sodda meta-model · out-of-fold ·
       korrelyatsiyani avval tekshir

Voting va stacking xulosasi

Voting: og'irlikli o'rtacha (soft > hard); modellar teng kuchli bo'lsin
Stacking: meta-model out-of-fold bashoratlardan o'rganadi
Meta-model sodda (LogisticRegression / RidgeCV)
Foyda faqat xilma-xillikdan; korr > 0.98 -> foyda yo'q

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14).

Misol 1 — Voting

python
"""Qattiq va yumshoq ovoz, og'irliklar (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import (HistGradientBoostingClassifier,
                              RandomForestClassifier, VotingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 5, n: int = 6000, p: int = 12, shovqin: float = 0.10):
    """Chiziqli va qoidaga asoslangan qismlar aralashmasi."""
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    chiziqli = 1.2 * X[:, 0] - 1.0 * X[:, 1] + 0.8 * X[:, 2]
    qoida = 2.0 * (((X[:, 3] > 0.4) & (X[:, 4] < 0.2))
                   | (X[:, 5] < -1.2)).astype(float)
    y = ((chiziqli + qoida) > 1.0).astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def modellar():
    return [
        ("rf", RandomForestClassifier(n_estimators=200, random_state=0,
                                      n_jobs=1)),
        ("gb", HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                              random_state=0)),
        ("lr", Pipeline([("sc", StandardScaler()),
                         ("m", LogisticRegression(max_iter=2000))])),
        ("knn", Pipeline([("sc", StandardScaler()),
                          ("m", KNeighborsClassifier(25))])),
    ]


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)

    print("=== 1. Alohida modellar ===")
    ballar, ehtimolliklar = {}, {}
    for nom, m in modellar():
        m.fit(Xtr, ytr)
        p = m.predict_proba(Xte)[:, 1]
        ehtimolliklar[nom] = p
        ballar[nom] = roc_auc_score(yte, p)
        print(f"  {nom:<5}: test AUC {ballar[nom]:.4f}")

    print("\n=== 2. Qattiq va yumshoq ovoz ===")
    for ovoz in ["hard", "soft"]:
        v = VotingClassifier(modellar(), voting=ovoz, n_jobs=1).fit(Xtr, ytr)
        if ovoz == "soft":
            a = roc_auc_score(yte, v.predict_proba(Xte)[:, 1])
            print(f"  {ovoz}: AUC {a:.4f}, aniqlik {v.score(Xte, yte):.4f}")
        else:
            print(f"  {ovoz}: aniqlik {v.score(Xte, yte):.4f}")

    print("\n=== 3. Bashoratlar korrelyatsiyasi ===")
    nomlar = list(ehtimolliklar)
    print(f"  {'':<6}" + "".join(f"{n:>8}" for n in nomlar))
    for a in nomlar:
        qator = "".join(
            f"{np.corrcoef(ehtimolliklar[a], ehtimolliklar[b])[0, 1]:>8.3f}"
            for b in nomlar)
        print(f"  {a:<6}" + qator)

    print("\n=== 4. Og'irliklar ===")
    variantlar = {
        "teng": [1, 1, 1, 1],
        "daraxtlarga ko'p": [3, 3, 1, 1],
        "faqat daraxtlar": [1, 1, 0, 0],
        "eng yaxshiga ko'p": [1, 5, 1, 1],
    }
    for nom, w in variantlar.items():
        v = VotingClassifier(modellar(), voting="soft", weights=w,
                             n_jobs=1).fit(Xtr, ytr)
        print(f"  {nom:<20} {w}: AUC "
              f"{roc_auc_score(yte, v.predict_proba(Xte)[:, 1]):.4f}")
    print(f"  eng yaxshi alohida model: "
          f"{max(ballar, key=ballar.get)} ({max(ballar.values()):.4f})")
    print("  ⭐ Yumshoq ovoz ehtimolliklarni hisobga oladi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Alohida modellar ===
  rf   : test AUC 0.8758
  gb   : test AUC 0.8682
  lr   : test AUC 0.8463
  knn  : test AUC 0.8416

=== 2. Qattiq va yumshoq ovoz ===
  hard: aniqlik 0.8294
  soft: AUC 0.8720, aniqlik 0.8411

=== 3. Bashoratlar korrelyatsiyasi ===
              rf      gb      lr     knn
  rf       1.000   0.941   0.884   0.871
  gb       0.941   1.000   0.863   0.841
  lr       0.884   0.863   1.000   0.901
  knn      0.871   0.841   0.901   1.000

=== 4. Og'irliklar ===
  teng                 [1, 1, 1, 1]: AUC 0.8720
  daraxtlarga ko'p     [3, 3, 1, 1]: AUC 0.8740
  faqat daraxtlar      [1, 1, 0, 0]: AUC 0.8742
  eng yaxshiga ko'p    [1, 5, 1, 1]: AUC 0.8711
  eng yaxshi alohida model: rf 0.8758-bob
  ⭐ Yumshoq ovoz ehtimolliklarni hisobga oladi

Nima ko'rsatdi: 2.1, 2.4-bo'limlar.

Misol 2 — Stacking

python
"""Meta-model birlashtirishni o'rganadi (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import (HistGradientBoostingClassifier,
                              RandomForestClassifier, StackingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold, train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 9, n: int = 6000, p: int = 12, shovqin: float = 0.10):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    chiziqli = 1.2 * X[:, 0] - 1.0 * X[:, 1] + 0.8 * X[:, 2]
    qoida = 2.0 * (((X[:, 3] > 0.4) & (X[:, 4] < 0.2))
                   | (X[:, 5] < -1.2)).astype(float)
    y = ((chiziqli + qoida) > 1.0).astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def modellar():
    return [
        ("rf", RandomForestClassifier(n_estimators=200, random_state=0,
                                      n_jobs=1)),
        ("gb", HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                              random_state=0)),
        ("lr", Pipeline([("sc", StandardScaler()),
                         ("m", LogisticRegression(max_iter=2000))])),
        ("knn", Pipeline([("sc", StandardScaler()),
                          ("m", KNeighborsClassifier(25))])),
    ]


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Bazaviy modellar ===")
    for nom, m in modellar():
        m.fit(Xtr, ytr)
        print(f"  {nom:<5}: {roc_auc_score(yte, m.predict_proba(Xte)[:, 1]):.4f}")

    print("\n=== 2. Stacking (LogisticRegression meta) ===")
    s = StackingClassifier(modellar(),
                           final_estimator=LogisticRegression(max_iter=2000),
                           cv=cv, stack_method="predict_proba",
                           n_jobs=1).fit(Xtr, ytr)
    print(f"  test AUC: "
          f"{roc_auc_score(yte, s.predict_proba(Xte)[:, 1]):.4f}")
    print(f"  meta-model koeffitsiyentlari:")
    for (nom, _), c in zip(modellar(), s.final_estimator_.coef_[0]):
        print(f"    {nom:<5}: {c:>+8.4f}")

    print("\n=== 3. Meta-model tanlash ===")
    metalar = {
        "LogReg(C=1)": LogisticRegression(max_iter=2000),
        "LogReg(C=0.1)": LogisticRegression(C=0.1, max_iter=2000),
        "RandomForest": RandomForestClassifier(n_estimators=100, max_depth=3,
                                               random_state=0, n_jobs=1),
    }
    for nom, meta in metalar.items():
        m = StackingClassifier(modellar(), final_estimator=meta, cv=cv,
                               stack_method="predict_proba",
                               n_jobs=1).fit(Xtr, ytr)
        print(f"  {nom:<16}: "
              f"{roc_auc_score(yte, m.predict_proba(Xte)[:, 1]):.4f}")

    print("\n=== 4. passthrough ===")
    for pt in [False, True]:
        m = StackingClassifier(modellar(),
                               final_estimator=LogisticRegression(max_iter=2000),
                               cv=cv, stack_method="predict_proba",
                               passthrough=pt, n_jobs=1).fit(Xtr, ytr)
        kirish = len(modellar()) + (X.shape[1] if pt else 0)
        print(f"  passthrough={str(pt):<5}: meta kirishlari {kirish}, AUC "
              f"{roc_auc_score(yte, m.predict_proba(Xte)[:, 1]):.4f}")
    print("  ⭐ Meta-model koeffitsiyentlari ishonch darajasini ko'rsatadi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Bazaviy modellar ===
  rf   : 0.8853
  gb   : 0.8825
  lr   : 0.8630
  knn  : 0.8401

=== 2. Stacking (LogisticRegression meta) ===
  test AUC: 0.8888
  meta-model koeffitsiyentlari:
    rf   :  +3.7268
    gb   :  +1.7846
    lr   :  +1.3189
    knn  :  -0.3670

=== 3. Meta-model tanlash ===
  LogReg(C=1)     : 0.8888
  LogReg(C=0.1)   : 0.8882
  RandomForest    : 0.8882

=== 4. passthrough ===
  passthrough=False: meta kirishlari 4, AUC 0.8888
  passthrough=True : meta kirishlari 16, AUC 0.8898
  ⭐ Meta-model koeffitsiyentlari ishonch darajasini ko'rsatadi

Nima ko'rsatdi: 2.2, 2.3-bo'limlar.

Misol 3 — Leakage: to'g'ri va noto'g'ri stacking

python
"""Out-of-fold nega majburiy (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_val_predict, train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 15, n: int = 5000, p: int = 10, shovqin: float = 0.12):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    chiziqli = 1.1 * X[:, 0] - 0.9 * X[:, 1]
    qoida = 1.8 * (((X[:, 2] > 0.4) & (X[:, 3] < 0.2))
                   | (X[:, 4] < -1.2)).astype(float)
    y = ((chiziqli + qoida) > 0.9).astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def modellar():
    return {
        "rf": RandomForestClassifier(n_estimators=200, random_state=0, n_jobs=1),
        "gb": HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                             random_state=0),
        "knn": Pipeline([("sc", StandardScaler()),
                         ("m", KNeighborsClassifier(15))]),
    }


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. NOTO'G'RI: o'quvdagi bashoratlar ===")
    ichki, tashqi = [], []
    for nom, m in modellar().items():
        m.fit(Xtr, ytr)
        ichki.append(m.predict_proba(Xtr)[:, 1])       # o'quvda - yodlangan
        tashqi.append(m.predict_proba(Xte)[:, 1])
    Mtr = np.column_stack(ichki)
    Mte = np.column_stack(tashqi)
    meta_notogri = LogisticRegression(max_iter=2000).fit(Mtr, ytr)
    print(f"  bazaviy modellarning o'quv AUC lari:")
    for (nom, _), s in zip(modellar().items(), ichki):
        print(f"    {nom:<5}: {roc_auc_score(ytr, s):.4f}")
    print(f"  meta koeffitsiyentlari: "
          f"{np.round(meta_notogri.coef_[0], 3).tolist()}")
    print(f"  test AUC: "
          f"{roc_auc_score(yte, meta_notogri.predict_proba(Mte)[:, 1]):.4f}")

    print("\n=== 2. TO'G'RI: out-of-fold bashoratlar ===")
    oof = []
    for nom, m in modellar().items():
        oof.append(cross_val_predict(m, Xtr, ytr, cv=cv,
                                     method="predict_proba", n_jobs=1)[:, 1])
    Otr = np.column_stack(oof)
    meta_togri = LogisticRegression(max_iter=2000).fit(Otr, ytr)
    print(f"  bazaviy modellarning OOF AUC lari:")
    for (nom, _), s in zip(modellar().items(), oof):
        print(f"    {nom:<5}: {roc_auc_score(ytr, s):.4f}")
    print(f"  meta koeffitsiyentlari: "
          f"{np.round(meta_togri.coef_[0], 3).tolist()}")
    print(f"  test AUC: "
          f"{roc_auc_score(yte, meta_togri.predict_proba(Mte)[:, 1]):.4f}")

    print("\n=== 3. Farq nimada ===")
    print("  Noto'g'ri variantda RandomForest o'quvda deyarli mukammal")
    print("  (AUC ~1.0) -> meta-model unga ortiqcha ishonadi")
    print("  OOF da esa uning haqiqiy kuchi ko'rinadi")
    print(f"  o'quv va OOF AUC farqi (rf): "
          f"{roc_auc_score(ytr, ichki[0]) - roc_auc_score(ytr, oof[0]):+.4f}")

    print("\n=== 4. Eng yaxshi alohida model bilan taqqoslash ===")
    eng = max((roc_auc_score(yte, p), nom)
              for nom, p in zip(modellar(), tashqi))
    print(f"  eng yaxshi alohida: {eng[1]} ({eng[0]:.4f})")
    print(f"  noto'g'ri stacking: "
          f"{roc_auc_score(yte, meta_notogri.predict_proba(Mte)[:, 1]):.4f}")
    print(f"  to'g'ri stacking:   "
          f"{roc_auc_score(yte, meta_togri.predict_proba(Mte)[:, 1]):.4f}")
    print("  ⭐ Out-of-fold - stacking ning majburiy sharti")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. NOTO'G'RI: o'quvdagi bashoratlar ===
  bazaviy modellarning o'quv AUC lari:
    rf   : 1.0000
    gb   : 1.0000
    knn  : 0.8705
  meta koeffitsiyentlari: [6.879, 7.257, -0.952]
  test AUC: 0.8585

=== 2. TO'G'RI: out-of-fold bashoratlar ===
  bazaviy modellarning OOF AUC lari:
    rf   : 0.8692
    gb   : 0.8662
    knn  : 0.8164
  meta koeffitsiyentlari: [3.444, 2.209, -0.097]
  test AUC: 0.8600

=== 3. Farq nimada ===
  Noto'g'ri variantda RandomForest o'quvda deyarli mukammal
  (AUC ~1.0) -> meta-model unga ortiqcha ishonadi
  OOF da esa uning haqiqiy kuchi ko'rinadi
  o'quv va OOF AUC farqi (rf): +0.1308

=== 4. Eng yaxshi alohida model bilan taqqoslash ===
  eng yaxshi alohida: rf 0.8611-bob
  noto'g'ri stacking: 0.8585
  to'g'ri stacking:   0.8600
  ⭐ Out-of-fold - stacking ning majburiy sharti

Nima ko'rsatdi: 2.5-bo'lim.

Misol 4 — Qachon foyda bermaydi

python
"""Xilma-xillik va narx-foyda (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import (HistGradientBoostingClassifier,
                              RandomForestClassifier, StackingClassifier,
                              VotingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def yarat(tur: str, seed: int = 4, n: int = 5000, p: int = 10,
          shovqin: float = 0.10):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    if tur == "chiziqli":
        kuch = 1.4 * X[:, 0] - 1.1 * X[:, 1] + 0.9 * X[:, 2]
    elif tur == "qoidaviy":
        kuch = 3.0 * (((X[:, 0] > 0.3) & (X[:, 1] < 0.2))
                      | (X[:, 2] < -1.1)).astype(float) - 1.4
    else:                                  # aralash
        kuch = (1.2 * X[:, 0] - 1.0 * X[:, 1]
                + 2.0 * (((X[:, 3] > 0.4) & (X[:, 4] < 0.2))
                         | (X[:, 5] < -1.2)).astype(float) - 1.0)
    y = (kuch > 0).astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def modellar():
    return [
        ("rf", RandomForestClassifier(n_estimators=200, random_state=0,
                                      n_jobs=1)),
        ("gb", HistGradientBoostingClassifier(learning_rate=0.1, max_iter=200,
                                              random_state=0)),
        ("lr", Pipeline([("sc", StandardScaler()),
                         ("m", LogisticRegression(max_iter=2000))])),
    ]


def main() -> None:
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Uch xil ma'lumotda ===")
    print(f"  {'ma_lumot':<12} {'eng yaxshi':>12} {'voting':>9} "
          f"{'stacking':>10} {'foyda':>9}")
    for tur in ["chiziqli", "qoidaviy", "aralash"]:
        X, y = yarat(tur)
        Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
                                              random_state=0, stratify=y)
        alohida = {}
        for nom, m in modellar():
            m.fit(Xtr, ytr)
            alohida[nom] = roc_auc_score(yte, m.predict_proba(Xte)[:, 1])
        v = VotingClassifier(modellar(), voting="soft",
                             n_jobs=1).fit(Xtr, ytr)
        av = roc_auc_score(yte, v.predict_proba(Xte)[:, 1])
        s = StackingClassifier(modellar(),
                               final_estimator=LogisticRegression(max_iter=2000),
                               cv=cv, n_jobs=1).fit(Xtr, ytr)
        asb = roc_auc_score(yte, s.predict_proba(Xte)[:, 1])
        eng = max(alohida.values())
        print(f"  {tur:<12} {eng:>12.4f} {av:>9.4f} {asb:>10.4f} "
              f"{max(av, asb) - eng:>+9.4f}")

    print("\n=== 2. Bir xil modellarni birlashtirish ===")
    X, y = yarat("aralash")
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    bir_xil = [(f"rf{i}", RandomForestClassifier(n_estimators=200,
                                                 random_state=i, n_jobs=1))
               for i in range(3)]
    P = []
    for nom, m in bir_xil:
        m.fit(Xtr, ytr)
        P.append(m.predict_proba(Xte)[:, 1])
    v = VotingClassifier(bir_xil, voting="soft", n_jobs=1).fit(Xtr, ytr)
    print(f"  uch RF korrelyatsiyasi: "
          f"{np.corrcoef(P[0], P[1])[0, 1]:.4f}")
    print(f"  alohida (o'rtacha): "
          f"{np.mean([roc_auc_score(yte, p) for p in P]):.4f}")
    print(f"  voting: {roc_auc_score(yte, v.predict_proba(Xte)[:, 1]):.4f}")

    print("\n=== 3. Kuchsiz modelni qo'shish ===")
    from sklearn.dummy import DummyClassifier
    from sklearn.tree import DecisionTreeClassifier
    asos = modellar()
    kuchsizlar = {
        "yo'q": asos,
        "stump": asos + [("stump", DecisionTreeClassifier(max_depth=1,
                                                          random_state=0))],
        "dummy": asos + [("dummy", DummyClassifier(strategy="prior"))],
    }
    for nom, ro_yxat in kuchsizlar.items():
        v = VotingClassifier(ro_yxat, voting="soft", n_jobs=1).fit(Xtr, ytr)
        print(f"  {nom:<8}: voting AUC "
              f"{roc_auc_score(yte, v.predict_proba(Xte)[:, 1]):.4f}")

    print("\n=== 4. Narx-foyda ===")
    alohida = {}
    for nom, m in modellar():
        m.fit(Xtr, ytr)
        alohida[nom] = roc_auc_score(yte, m.predict_proba(Xte)[:, 1])
    s = StackingClassifier(modellar(),
                           final_estimator=LogisticRegression(max_iter=2000),
                           cv=cv, n_jobs=1).fit(Xtr, ytr)
    asb = roc_auc_score(yte, s.predict_proba(Xte)[:, 1])
    eng_nom = max(alohida, key=alohida.get)
    print(f"  eng yaxshi alohida ({eng_nom}): {alohida[eng_nom]:.4f}")
    print(f"  stacking: {asb:.4f}, foyda {asb - alohida[eng_nom]:+.4f}")
    print(f"  narx: {len(modellar())} model o'qitish va saqlash,")
    print(f"        + meta-model, + {cv.get_n_splits()} karra CV")
    print("  ⭐ Foydani ishlab chiqarish narxi bilan solishtiring")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Uch xil ma'lumotda ===
  ma_lumot       eng yaxshi    voting   stacking     foyda
  chiziqli           0.9100    0.9033     0.9069   -0.0032
  qoidaviy           0.9017    0.9002     0.8987   -0.0016
  aralash            0.8957    0.9016     0.8970   +0.0059

=== 2. Bir xil modellarni birlashtirish ===
  uch RF korrelyatsiyasi: 0.9923
  alohida (o'rtacha): 0.8914
  voting: 0.8918

=== 3. Kuchsiz modelni qo'shish ===
  yo'q    : voting AUC 0.9016
  stump   : voting AUC 0.8994
  dummy   : voting AUC 0.9016

=== 4. Narx-foyda ===
  eng yaxshi alohida (gb): 0.8957
  stacking: 0.8970, foyda +0.0013
  narx: 3 model o'qitish va saqlash,
        + meta-model, + 5 karra CV
  ⭐ Foydani ishlab chiqarish narxi bilan solishtiring

Nima ko'rsatdi: 2.4, 2.6-bo'limlar.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Ansambl har doim yaxshilaydi" Faqat xilma-xillikda
"Ko'p model — yaxshiroq" Kuchsizlar zarar qiladi
"Qattiq ovoz yetarli" Yumshoq odatda yaxshiroq
"Meta-model murakkab bo'lsin" Sodda bo'lsin
"Qo'lda stacking oson" Out-of-fold ni unutish oson
"passthrough har doim yaxshi" Ko'pincha farqsiz
"Kalibrlash ahamiyatsiz" Yumshoq ovozda muhim
"0.5% — katta yutuq" Narxga qarang

6. Keng tarqalgan xatolar va yechimlari

1. Qo'lda stacking, out-of-fold siz

python
meta.fit(np.column_stack([m.predict_proba(Xtr)[:, 1] for m in modellar]), ytr) # ⚠️
StackingClassifier(estimators, final_estimator=LogisticRegression(), cv=5)     # ✅

2. Murakkab meta-model

python
StackingClassifier(..., final_estimator=RandomForestClassifier(500))  # ⚠️
StackingClassifier(..., final_estimator=LogisticRegression())         # ✅

3. Kalibrlanmagan model yumshoq ovozda

python
VotingClassifier([("svm", SVC(probability=True)), ...], voting="soft") # ⚠️
VotingClassifier([("svm", CalibratedClassifierCV(SVC())), ...])        # ✅

4. Juda o'xshash modellar

python
VotingClassifier([("rf1", rf), ("rf2", rf2), ("rf3", rf3)])           # ⚠️
VotingClassifier([("rf", rf), ("gb", gb), ("lr", lr)])                # ✅

5. Kuchsiz modelni teng og'irlik bilan

python
VotingClassifier([..., ("dummy", DummyClassifier())])                 # ⚠️
# avval har modelni alohida baholang                                  # ✅

6. n_jobs ni unutish

python
StackingClassifier(estimators, cv=5)                                  # ⚠️
StackingClassifier(estimators, cv=5, n_jobs=-1)                       # ✅

7. Testda tanlab, testda baholash

python
# test da voting va stacking dan yaxshisini tanlash                   # ⚠️
# CV da tanlang, testda faqat yakuniy baho                            # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 15.5-dars (o'tilgan): Random Forest
  • 15.10-dars (o'tilgan): XGBoost va LightGBM
  • 14.10-dars (o'tilgan): Kalibrlash
  • 12.9-dars (o'tilgan): Leakage
  • 15.13-dars: Ishlab chiqarish

8. Eng yaxshi amaliyotlar

  1. Xilma-xillikni avval tekshiring.

  2. Yumshoq ovoz ishlating.

  3. Meta-modelni sodda qiling.

  4. StackingClassifier dan foydalaning.

  5. Kalibrlangan modellarni qo'shing.

  6. Kuchsizlarni chiqarib tashlang.

  7. n_jobs qo'ying.

  8. Narx-foydani hisoblang.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # voting turlari?
2.  # qaysi biri yaxshiroq?
3.  # stacking mexanizmi?
4.  # out-of-fold nega kerak?
5.  # meta-model qanday bo'lsin?
6.  # passthrough nima?
7.  # xilma-xillikni qanday o'lchash?
8.  # korrelyatsiya chegarasi?
9.  # kuchsiz model nima qiladi?
10. # odatiy foyda?
11. # qo'lda stacking xavfi?
12. # yumshoq ovoz uchun shart?
Javoblar
  1. Qattiq va yumshoq
  2. Yumshoq
  3. Meta-model OOF bashoratlardan o'rganadi
  4. Overfitting qilgan bashoratlarni oldini olish
  5. Sodda (LogisticRegression)
  6. Asl belgilarni ham berish
  7. Korrelyatsiya, kelishmovchilik
  8. ~0.98 dan yuqori — foyda yo'q
  9. O'rtachani tortadi
  10. 0.3-1.5%
  11. Out-of-fold ni unutish
  12. Kalibrlangan ehtimolliklar

Vazifa 2: Xatolarni tuzating

python
1.  meta.fit(np.column_stack([m.predict_proba(Xtr)[:, 1] for m in ms]), ytr)

2.  StackingClassifier(..., final_estimator=RandomForestClassifier(500))

3.  VotingClassifier([("svm", SVC(probability=True))], voting="soft")

4.  VotingClassifier([("rf1", rf), ("rf2", rf2), ("rf3", rf3)])

5.  StackingClassifier(estimators, cv=5)   # juda sekin
Javoblar
python
1.  StackingClassifier(estimators, final_estimator=LogisticRegression(), cv=5)

2.  StackingClassifier(..., final_estimator=LogisticRegression())

3.  VotingClassifier([("svm", CalibratedClassifierCV(SVC()))], voting="soft")

4.  VotingClassifier([("rf", rf), ("gb", gb), ("lr", lr)])

5.  StackingClassifier(estimators, cv=5, n_jobs=-1)

Vazifa 3: Voting

Modellang:

  1. Alohida modellar
  2. Ikki ovoz
  3. Korrelyatsiya
  4. Og'irliklar

Vazifa 4: Stacking

Modellang:

  1. Bazaviy
  2. Stacking
  3. Meta-model
  4. passthrough

Vazifa 5: Leakage

Modellang:

  1. Noto'g'ri
  2. To'g'ri
  3. Farq
  4. Taqqoslash

Vazifa 6: Qachon foyda yo'q

Modellang:

  1. Uch ma'lumot
  2. Bir xil modellar
  3. Kuchsiz model
  4. Narx-foyda

Vazifa 7: O'ylash

Kaggle musobaqalarida g'olib yechimlar ko'pincha 50+ modelning murakkab stacking i bo'ladi. Nega ishlab chiqarishda bunday qilinmaydi?

Javob

Qisqa javob: musobaqada faqat bitta mezon bor — leaderboard balli, va uning narxi nolga teng. Ishlab chiqarishda esa aniqlik ko'p mezondan bittasi, qolganlari (kechikish, ishonchlilik, kuzatuv, tushuntirish, xarajat) murakkab ansamblga qarshi ishlaydi.

1. Mezonlar farqi

Mezon Musobaqa Ishlab chiqarish
Aniqlik Yagona Bittasi
Kechikish Ahamiyatsiz Ko'pincha kritik
Qayta o'qitish narxi Bir marta Har kuni/hafta
Tushuntirish Kerak emas Ko'pincha majburiy
Nosozlik nuqtalari Yo'q 50 model = 50 xavf
Kuzatuv Yo'q Har model uchun

2. Yashirin xarajatlar

  1. Drift: har model alohida eskiradi, ularni alohida kuzatish kerak
  2. Nosozlik: bitta model ishlamasa, butun ansambl to'xtaydi
  3. Versiyalash: 50 model + meta-model = murakkab bog'liqlik
  4. Xatolarni topish: qaysi model noto'g'ri bashorat berdi?

3. Musobaqa yutug'ining tabiati

  • Ko'pincha 0.001-0.005 metrika
  • Ko'p qismi test to'plamiga moslashish (leaderboard overfitting)
  • Yangi taqsimotda bu yutuq ko'pincha yo'qoladi

4. Ishlab chiqarishdagi oqilona chegara

  1. 1 ta asosiy model (odatda boosting)
  2. Kerak bo'lsa: 2-3 model yumshoq ovoz bilan
  3. Stacking — faqat foyda > 1% va narx oqlansa
  4. Har doim sodda muqobil bilan taqqoslang

5. Xulosa

  1. Musobaqada narx yo'q, ishlab chiqarishda bor
  2. Murakkablik operatsion xavf demak
  3. 0.5% uchun 50 model arzimaydi
  4. Sodda model ko'pincha to'g'ri tanlov

Nimani mustahkamlaydi: 2.6-bo'lim.

Xulosa

Bu darsda voting va stacking ni o'rgandik.

Eng muhim uch fikr:

  1. Voting — o'rtachalashtirish, stacking — o'rganish. VotingClassifier bashoratlarni (og'irlik bilan) o'rtachalashtiradi; yumshoq ovoz odatda yaxshiroq, lekin u kalibrlangan ehtimolliklarni talab qiladi. StackingClassifier esa meta-model orqali birlashtirishni o'rganadi va meta-model sodda bo'lishi kerak (LogisticRegression).

  2. Out-of-fold — stacking ning majburiy sharti. Meta-model bazaviy modellarning ko'rmagan ma'lumotidagi bashoratlarini ko'rishi kerak. O'quv bashoratlarini berish — to'g'ridan-to'g'ri leakage: meta-model overfitting qilgan modelga ortiqcha ishonadi. Qo'lda yozmang — StackingClassifier(cv=5) buni to'g'ri bajaradi.

  3. Foyda faqat xilma-xillikdan. Modellarning bashoratlari 0.98 dan yuqori korrelyatsiyaga ega bo'lsa, ansambl deyarli hech narsa bermaydi. Kuchsiz modelni teng og'irlik bilan qo'shish esa natijani yomonlashtiradi. Odatiy yutuq 0.3-1.5% — uni N ta modelni o'qitish, saqlash, kuzatish va tushuntirish narxi bilan solishtiring.

Keyingi darsda ansambllarni ishlab chiqarishga chiqarishni o'rganamiz: model hajmi, kechikish, drift va kuzatuv.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
15.12-dars: Voting va stacking — IlmHamroh