IlmHamroh
Data Science va sun'iy intellekt/Daraxtlar va ansambllar4/14-dars18 daqiqa
Mundarija (22)

15.4-dars: Bagging

15-QISM — DARAXTLAR VA ANSAMBLLAR · 4-dars

1. Kirish va motivatsiya

Bagging (Bootstrap AGGregatING) — Leo Breiman 1996 yilda taklif qilgan sodda g'oya: o'quv ma'lumotidan bootstrap namunalar olib, har birida model qurib, natijalarni o'rtachalashtirish. Bu — beqaror modellar 15.3-bob uchun to'g'ridan-to'g'ri yechim.

G'oyaning go'zalligi shundaki, u hech qanday qo'shimcha ma'lumot talab qilmaydi: bir xil ma'lumotdan tasodifiy qayta namuna olish orqali modellar bir-biridan farqlanadi, va bu farq o'rtachalashtirishda yo'qoladi. Bonus sifatida OOB (out-of-bag) baho paydo bo'ladi — CV siz, bepul validatsiya.

Bu darsda: bootstrap namunalash, agregatsiya (ovoz berish va o'rtacha), OOB baho, bagging nimani kamaytiradi (dispersiya, bias emas), BaggingClassifier/BaggingRegressor va n_estimators tanlash.

Real vaziyat. Ishlab chiqarishdagi nuqsonlarni bashorat qiluvchi model kuniga qayta o'qitiladi. Bitta daraxt bilan bashoratlar kundan kunga sakrar edi — ishlab chiqarish bo'limi modelga ishonmay qoldi. 200 daraxtli bagging joriy qilingach, kunlik bashoratlar barqarorlashdi va aniqlik 0.71 dan 0.79 ga ko'tarildi. Qaror qabul qilish uchun barqarorlik aniqlikdan kam muhim emas edi.

Bu darsda bagging ni o'rganamiz.

Bu darsda:

  • Bootstrap namunalash
  • Agregatsiya
  • OOB baho
  • Nimani kamaytiradi
  • BaggingClassifier
  • n_estimators
  • Tuzoqlar
  • Amaliy: qo'lda bagging

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Bootstrap namunalash

text
n ta namunadan n ta TANLASH, QAYTARISH BILAN

Natija: ba'zi namunalar bir necha marta, ba'zilari umuman tushmaydi

Bitta namunaning tushmaslik ehtimoli:
  (1 - 1/n)^n  ->  1/e  ~  0.368   (n katta bo'lganda)

Ya'ni har bootstrap namunada:
  ~63.2% noyob namuna ishlatiladi
  ~36.8% namuna CHETDA qoladi (out-of-bag, OOB)

36.8% chetda qolishi — bagging ning ikki ustuni: (1) modellar bir-biridan farq qiladi, (2) chetdagi namunalar bepul validatsiya to'plamini beradi. Bu son n ga deyarli bog'liq emas (n > 50 da 0.368 ga yaqin).

2.2. Agregatsiya

text
Regressiya:     bashorat = B ta modelning O'RTACHASI
Klassifikatsiya: ikki variant
  1. Qattiq ovoz (hard voting) — ko'pchilik sinfi
  2. Yumshoq ovoz (soft voting) — ehtimolliklar o'rtachasi   <- odatda YAXSHIROQ

sklearn: BaggingClassifier.predict_proba() — yumshoq ovoz

Yumshoq ovoz odatda yaxshiroq: u modellarning ishonch darajasini hisobga oladi. 0.51 va 0.99 ehtimolliklar qattiq ovozda teng, yumshoq ovozda esa ikkinchisi kuchliroq ta'sir qiladi. Kalibrlash 14.10-bob muhim bo'lganda ham yumshoq ovoz zarur.

2.3. OOB baho

text
Har namuna uchun: uni KO'RMAGAN daraxtlar bilan bashorat qilish

oob_score_ — shu bashoratlar bo'yicha aniqlik (yoki R^2)

Afzalligi: CV siz, qo'shimcha o'qitishsiz validatsiya
Kamchiligi: har namuna ~0.37*B daraxt bilan baholanadi
            B kichik bo'lsa (< 50) ishonchsiz
            sklearn da faqat n_estimators yetarli bo'lsa ishlaydi

BaggingClassifier(oob_score=True)  /  RandomForestClassifier(oob_score=True)

OOB baho CV ga yaqin natija beradi va bepul: ansambl qurish jarayonida o'z-o'zidan hosil bo'ladi. Katta ma'lumotlarda bu 5-karra CV ga nisbatan 5 barobar tejamkor. Lekin vaqt qatorlarida 12.3-bob OOB noto'g'ri — u tasodifiy bo'linishni nazarda tutadi.

2.4. Nimani kamaytiradi

text
E[o'rtacha] = E[bitta model]        -> BIAS o'zgarmaydi
var(o'rtacha) = rho*s^2 + (1-rho)*s^2/B  -> DISPERSIYA kamayadi

XULOSA: bagging faqat DISPERSIYANI kamaytiradi

Shuning uchun:
  chuqur daraxt (past bias, yuqori dispersiya) -> bagging KATTA foyda
  sayoz daraxt (yuqori bias) -> bagging kam foyda
  chiziqli model (past dispersiya) -> deyarli foyda yo'q

Bu qoida bagging ni qachon ishlatishni to'liq belgilaydi: bazaviy model beqaror bo'lishi kerak. Shuning uchun bagging da daraxtlar to'liq o'stiriladi (max_depth=None) — bu boshqa joyda xato bo'lgan narsa bu yerda to'g'ri.

2.5. BaggingClassifier

python
from sklearn.ensemble import BaggingClassifier, BaggingRegressor

b = BaggingClassifier(
    DecisionTreeClassifier(random_state=0),   # bazaviy model
    n_estimators=200,
    max_samples=1.0,        # har modelga namunalar ulushi
    max_features=1.0,       # har modelga belgilar ulushi
    bootstrap=True,         # qaytarish bilan (False -> pasting)
    bootstrap_features=False,
    oob_score=True,
    random_state=0, n_jobs=-1)

estimator sifatida istalgan model berish mumkin (KNN, SVM, LogReg), lekin foyda faqat beqaror modellarda seziladi. max_features < 1.0 bilan bagging Random Subspaces ga aylanadi — bu Random Forest ga yaqinlashish 15.5-bob.

2.6. n_estimators

text
Ko'proq daraxt = yomonroq EMAS (overfitting bermaydi)
  lekin foyda tez to'yinadi: 50-200 dan keyin sezilmaydi

Amaliy tanlov:
  boshlang'ich  100
  yakuniy model 300-500
  OOB egri chizig'ini chizib to'yinishni ko'ring

Narx: o'qitish va bashorat vaqti chiziqli oshadi

Bagging da n_estimators overfitting bermaydi — bu boosting dan 15.8-bob asosiy farq. Shuning uchun uni CV bilan sozlash shart emas: hisoblash imkoni qancha bo'lsa, shuncha qo'ying (to'yinishdan keyin foydasi yo'q).

2.7. Tuzoqlar

Asosiy tuzoqlar: bazaviy model sifatida sayoz daraxt berish (dispersiya yo'q — foyda yo'q); chiziqli modelni bagging qilish; n_estimators ni CV bilan sozlashga urinish; vaqt qatorlarida OOB ga ishonish; oob_score ni kichik n_estimators bilan o'qish; bagging ni bias muammosiga qarshi ishlatish; max_samples ni juda kichik qo'yish (har model kuchsizlanadi); n_jobs ni unutish (sekin).

2.8. Sodda va kuchli

Bagging — bootstrap namunalarda ko'p model qurib, natijalarni o'rtachalashtirish. Har namunada 63.2% noyob ma'lumot ishlatiladi, qolgan 36.8% esa OOB baho beradi — bepul validatsiya. Bagging faqat dispersiyani kamaytiradi, biasni emas; shuning uchun bazaviy model beqaror (chuqur daraxt) bo'lishi kerak. n_estimators ni oshirish hech qachon zarar qilmaydi, lekin foyda 100-300 dan keyin to'yinadi. Keyingi dars — Random Forest.

3. Tez ma'lumotnoma

python
from sklearn.ensemble import BaggingClassifier, BaggingRegressor
from sklearn.tree import DecisionTreeClassifier

b = BaggingClassifier(DecisionTreeClassifier(random_state=0),
                      n_estimators=300, bootstrap=True, oob_score=True,
                      random_state=0, n_jobs=-1).fit(X, y)
b.oob_score_                    # bepul validatsiya
b.estimators_                   # alohida daraxtlar
b.estimators_samples_           # har daraxtga tushgan indekslar
b.predict_proba(Xte)            # yumshoq ovoz

# qo'lda bagging
idx = rng.integers(0, n, n)     # bootstrap namuna
QOIDA: bazaviy model chuqur bo'lsin · n_estimators ni ko'p qo'y ·
       OOB dan foydalan · vaqt qatorida OOB ishlatma

Bagging xulosasi

Bootstrap: n dan n ta qaytarish bilan; 63.2% noyob, 36.8% OOB
Agregatsiya: o'rtacha (regressiya) yoki yumshoq ovoz (klassifikatsiya)
Faqat dispersiyani kamaytiradi -> bazaviy model beqaror bo'lsin
n_estimators overfitting bermaydi; 100-300 da to'yinadi

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14).

Misol 1 — Bootstrap va 63.2%

python
"""Bootstrap namunalashning asosiy xossalari (real numpy)."""

import numpy as np


def main() -> None:
    rng = np.random.default_rng(0)

    print("=== 1. Noyob namunalar ulushi ===")
    print(f"  {'n':>7} {'noyob %':>10} {'nazariy %':>11}")
    for n in [10, 50, 200, 1000, 10000]:
        ulushlar = []
        for _ in range(200):
            idx = rng.integers(0, n, n)
            ulushlar.append(len(np.unique(idx)) / n)
        print(f"  {n:>7} {np.mean(ulushlar):>9.1%} "
              f"{1 - (1 - 1 / n) ** n:>10.1%}")
    print(f"  chegara (n -> cheksiz): {1 - 1 / np.e:.4f}")

    print("\n=== 2. Namuna necha marta tushadi ===")
    n = 1000
    sanoq = np.zeros(n, dtype=int)
    for _ in range(1):
        idx = rng.integers(0, n, n)
        sanoq = np.bincount(idx, minlength=n)
    for marta in range(5):
        print(f"  {marta} marta: {(sanoq == marta).sum():>4} namuna "
              f"({(sanoq == marta).mean():>6.1%})")
    print(f"  5+ marta: {(sanoq >= 5).sum()} namuna")

    print("\n=== 3. OOB: har namuna nechta modelda chetda qoladi ===")
    B = 100
    oob = np.zeros(n, dtype=int)
    for _ in range(B):
        idx = rng.integers(0, n, n)
        ichida = np.zeros(n, dtype=bool)
        ichida[idx] = True
        oob += ~ichida
    print(f"  {B} model uchun o'rtacha OOB soni: {oob.mean():.1f}")
    print(f"  eng kam {oob.min()}, eng ko'p {oob.max()}")
    print(f"  hech qachon OOB bo'lmagan namunalar: {(oob == 0).sum()}")

    print("\n=== 4. Bootstrap o'rtachaning dispersiyasi ===")
    haqiqiy = rng.normal(5.0, 2.0, 400)
    ortachalar = [haqiqiy[rng.integers(0, 400, 400)].mean() for _ in range(2000)]
    print(f"  namuna o'rtachasi: {haqiqiy.mean():.4f}")
    print(f"  bootstrap o'rtachalari: {np.mean(ortachalar):.4f}")
    print(f"  bootstrap std: {np.std(ortachalar):.4f}")
    print(f"  nazariy SE (s/sqrt(n)): {haqiqiy.std(ddof=1) / np.sqrt(400):.4f}")
    print("  ⭐ Bootstrap taqsimotni qayta tiklaydi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Noyob namunalar ulushi ===
        n    noyob %   nazariy %
       10     65.7%      65.1%
       50     63.8%      63.6%
      200     63.5%      63.3%
     1000     63.3%      63.2%
    10000     63.2%      63.2%
  chegara (n -> cheksiz): 0.6321

=== 2. Namuna necha marta tushadi ===
  0 marta:  360 namuna ( 36.0%)
  1 marta:  379 namuna ( 37.9%)
  2 marta:  185 namuna ( 18.5%)
  3 marta:   56 namuna (  5.6%)
  4 marta:   17 namuna (  1.7%)
  5+ marta: 3 namuna

=== 3. OOB: har namuna nechta modelda chetda qoladi ===
  100 model uchun o'rtacha OOB soni: 36.8
  eng kam 22, eng ko'p 50
  hech qachon OOB bo'lmagan namunalar: 0

=== 4. Bootstrap o'rtachaning dispersiyasi ===
  namuna o'rtachasi: 5.0882
  bootstrap o'rtachalari: 5.0872
  bootstrap std: 0.0972
  nazariy SE (s/sqrt(n)): 0.0977
  ⭐ Bootstrap taqsimotni qayta tiklaydi

Nima ko'rsatdi: 2.1-bo'lim.

Misol 2 — Qo'lda bagging

python
"""Bagging ni noldan qurish va sklearn bilan solishtirish (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import BaggingClassifier
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier


def yarat(seed: int = 5, n: int = 2500, shovqin: float = 0.12):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 8))
    qoida = (((X[:, 0] > 0.4) & (X[:, 1] < 0.0))
             | ((X[:, 2] > 0.0) & (X[:, 3] > 0.5))
             | (X[:, 4] < -1.2))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    rng = np.random.default_rng(0)

    print("=== 1. Qo'lda bagging ===")
    B = 200
    P, ichida_hammasi = [], []
    for _ in range(B):
        idx = rng.integers(0, len(Xtr), len(Xtr))
        d = DecisionTreeClassifier(random_state=0).fit(Xtr[idx], ytr[idx])
        P.append(d.predict_proba(Xte)[:, 1])
        ichida = np.zeros(len(Xtr), dtype=bool)
        ichida[idx] = True
        ichida_hammasi.append(ichida)
    P = np.array(P)
    bitta = DecisionTreeClassifier(random_state=0).fit(Xtr, ytr)
    print(f"  bitta daraxt: {bitta.score(Xte, yte):.4f}")
    for b in [1, 5, 20, 50, 100, 200]:
        aniqlik = ((P[:b].mean(axis=0) > 0.5).astype(int) == yte).mean()
        print(f"  {b:>3} daraxt: {aniqlik:.4f}")

    print("\n=== 2. Qattiq va yumshoq ovoz ===")
    qattiq = ((P > 0.5).astype(int).mean(axis=0) > 0.5).astype(int)
    yumshoq = (P.mean(axis=0) > 0.5).astype(int)
    print(f"  qattiq ovoz:  {(qattiq == yte).mean():.4f}")
    print(f"  yumshoq ovoz: {(yumshoq == yte).mean():.4f}")
    print(f"  farq qilgan namunalar: {(qattiq != yumshoq).sum()}")

    print("\n=== 3. sklearn BaggingClassifier ===")
    b = BaggingClassifier(DecisionTreeClassifier(random_state=0),
                          n_estimators=200, oob_score=True,
                          random_state=0, n_jobs=1).fit(Xtr, ytr)
    print(f"  test aniqligi: {b.score(Xte, yte):.4f}")
    print(f"  OOB bahosi:    {b.oob_score_:.4f}")
    print(f"  daraxtlar soni: {len(b.estimators_)}")
    print(f"  o'rtacha barglar: "
          f"{np.mean([e.get_n_leaves() for e in b.estimators_]):.1f}")

    print("\n=== 4. OOB baho CV ga yaqinmi ===")
    from sklearn.model_selection import StratifiedKFold, cross_val_score
    cv = StratifiedKFold(5, shuffle=True, random_state=0)
    cvb = cross_val_score(BaggingClassifier(DecisionTreeClassifier(random_state=0),
                                            n_estimators=100, random_state=0),
                          Xtr, ytr, cv=cv).mean()
    b100 = BaggingClassifier(DecisionTreeClassifier(random_state=0),
                             n_estimators=100, oob_score=True,
                             random_state=0).fit(Xtr, ytr)
    print(f"  5-karra CV:    {cvb:.4f}")
    print(f"  OOB bahosi:    {b100.oob_score_:.4f}")
    print(f"  test aniqligi: {b100.score(Xte, yte):.4f}")
    print("  ⭐ OOB — bepul validatsiya")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Qo'lda bagging ===
  bitta daraxt: 0.7733
    1 daraxt: 0.7507
    5 daraxt: 0.8320
   20 daraxt: 0.8640
   50 daraxt: 0.8747
  100 daraxt: 0.8773
  200 daraxt: 0.8787

=== 2. Qattiq va yumshoq ovoz ===
  qattiq ovoz:  0.8787
  yumshoq ovoz: 0.8787
  farq qilgan namunalar: 0

=== 3. sklearn BaggingClassifier ===
  test aniqligi: 0.8813
  OOB bahosi:    0.8497
  daraxtlar soni: 200
  o'rtacha barglar: 173.7

=== 4. OOB baho CV ga yaqinmi ===
  5-karra CV:    0.8497
  OOB bahosi:    0.8451
  test aniqligi: 0.8787
  ⭐ OOB — bepul validatsiya

Nima ko'rsatdi: 2.2, 2.3-bo'limlar.

Misol 3 — Bagging nimani kamaytiradi

python
"""Bias va dispersiya alohida (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import BaggingRegressor
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsRegressor
from sklearn.tree import DecisionTreeRegressor


def yarat(seed: int, n: int = 500):
    rng = np.random.default_rng(seed)
    X = rng.uniform(-3, 3, (n, 3))
    f = np.sin(1.5 * X[:, 0]) + 0.5 * X[:, 1] - 0.3 * X[:, 2] ** 2
    return X, f + rng.normal(0, 0.8, n), f


def bias_dispersiya(model_yaratuvchi, Xte, fte, takror: int = 40):
    """Ko'p o'quv to'plamida bias^2 va dispersiyani ajratish."""
    P = []
    for k in range(takror):
        Xtr, ytr, _ = yarat(k + 1)
        P.append(model_yaratuvchi().fit(Xtr, ytr).predict(Xte))
    P = np.array(P)
    ortacha = P.mean(axis=0)
    bias2 = float(((ortacha - fte) ** 2).mean())
    dispersiya = float(P.var(axis=0).mean())
    return bias2, dispersiya, bias2 + dispersiya


def main() -> None:
    Xte, _, fte = yarat(999, 400)

    print("=== 1. Bazaviy modellar ===")
    modellar = {
        "daraxt(to'liq)": lambda: DecisionTreeRegressor(random_state=0),
        "daraxt(3)": lambda: DecisionTreeRegressor(max_depth=3, random_state=0),
        "chiziqli": lambda: LinearRegression(),
        "KNN(20)": lambda: KNeighborsRegressor(20),
    }
    print(f"  {'model':<15} {'bias^2':>9} {'dispersiya':>12} {'jami':>9}")
    asosiy = {}
    for nom, yaratuvchi in modellar.items():
        b2, d, j = bias_dispersiya(yaratuvchi, Xte, fte)
        asosiy[nom] = (b2, d, j)
        print(f"  {nom:<15} {b2:>9.4f} {d:>12.4f} {j:>9.4f}")

    print("\n=== 2. Bagging dan keyin ===")
    print(f"  {'model':<15} {'bias^2':>9} {'dispersiya':>12} {'jami':>9} "
          f"{'foyda %':>9}")
    for nom, yaratuvchi in modellar.items():
        def bagged(y=yaratuvchi):
            return BaggingRegressor(y(), n_estimators=50, random_state=0)
        b2, d, j = bias_dispersiya(bagged, Xte, fte, takror=20)
        foyda = 1 - j / asosiy[nom][2]
        print(f"  {nom:<15} {b2:>9.4f} {d:>12.4f} {j:>9.4f} {foyda:>8.1%}")

    print("\n=== 3. Bias o'zgarmaganini tekshirish ===")
    for nom in ["daraxt(to'liq)", "chiziqli"]:
        def bagged(y=modellar[nom]):
            return BaggingRegressor(y(), n_estimators=50, random_state=0)
        b2b, db, _ = bias_dispersiya(bagged, Xte, fte, takror=20)
        b2a, da, _ = asosiy[nom]
        print(f"  {nom:<15}: bias^2 {b2a:.4f} -> {b2b:.4f}, "
              f"dispersiya {da:.4f} -> {db:.4f}")

    print("\n=== 4. Daraxt chuqurligi va bagging foydasi ===")
    print(f"  {'max_depth':>10} {'bitta':>9} {'bagging':>9} {'foyda %':>9}")
    for chuqurlik in [2, 3, 5, 8, None]:
        def bitta(c=chuqurlik):
            return DecisionTreeRegressor(max_depth=c, random_state=0)

        def bagged(c=chuqurlik):
            return BaggingRegressor(DecisionTreeRegressor(max_depth=c,
                                                          random_state=0),
                                    n_estimators=50, random_state=0)
        _, _, j1 = bias_dispersiya(bitta, Xte, fte, takror=20)
        _, _, j2 = bias_dispersiya(bagged, Xte, fte, takror=20)
        nom = "None" if chuqurlik is None else str(chuqurlik)
        print(f"  {nom:>10} {j1:>9.4f} {j2:>9.4f} {1 - j2 / j1:>8.1%}")
    print("  ⭐ Chuqurroq daraxt -> bagging foydasi katta")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Bazaviy modellar ===
  model              bias^2   dispersiya      jami
  daraxt(to'liq)     0.0738       0.9051    0.9789
  daraxt(3)          0.6358       0.2710    0.9069
  chiziqli           1.0993       0.0172    1.1165
  KNN(20)            0.2681       0.0682    0.3363

=== 2. Bagging dan keyin ===
  model              bias^2   dispersiya      jami   foyda %
  daraxt(to'liq)     0.0795       0.1652    0.2447    75.0%
  daraxt(3)          0.6075       0.0763    0.6837    24.6%
  chiziqli           1.1011       0.0173    1.1185    -0.2%
  KNN(20)            0.2849       0.0590    0.3439    -2.3%

=== 3. Bias o'zgarmaganini tekshirish ===
  daraxt(to'liq) : bias^2 0.0738 -> 0.0795, dispersiya 0.9051 -> 0.1652
  chiziqli       : bias^2 1.0993 -> 1.1011, dispersiya 0.0172 -> 0.0173

=== 4. Daraxt chuqurligi va bagging foydasi ===
   max_depth     bitta   bagging   foyda %
           2    1.1725    0.9699    17.3%
           3    0.8986    0.6837    23.9%
           5    0.7414    0.4167    43.8%
           8    0.7191    0.2377    66.9%
        None    0.9909    0.2447    75.3%
  ⭐ Chuqurroq daraxt -> bagging foydasi katta

Nima ko'rsatdi: 2.4-bo'lim.

Misol 4 — n_estimators va parametrlar

python
"""To'yinish, max_samples va pasting (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import BaggingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier


def yarat(seed: int = 15, n: int = 3000, shovqin: float = 0.13):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 10))
    qoida = (((X[:, 0] > 0.3) & (X[:, 1] < 0.2))
             | ((X[:, 2] > 0.5) & (X[:, 3] > 0.0))
             | (X[:, 4] < -1.0))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)

    def baho(**p):
        b = BaggingClassifier(DecisionTreeClassifier(random_state=0),
                              random_state=0, n_jobs=1, **p).fit(Xtr, ytr)
        return b, roc_auc_score(yte, b.predict_proba(Xte)[:, 1])

    print("=== 1. n_estimators va to'yinish ===")
    print(f"  {'n_est':>7} {'test AUC':>10} {'OOB':>10}")
    for ne in [1, 5, 10, 25, 50, 100, 200, 400]:
        b, auc = baho(n_estimators=ne, oob_score=(ne >= 50))
        oob = f"{b.oob_score_:.4f}" if ne >= 50 else "-"
        print(f"  {ne:>7} {auc:>10.4f} {oob:>10}")

    print("\n=== 2. max_samples ===")
    print(f"  {'ulush':>7} {'test AUC':>10} {'o_rt barglar':>14}")
    for ms in [0.1, 0.3, 0.5, 0.8, 1.0]:
        b, auc = baho(n_estimators=100, max_samples=ms)
        barglar = np.mean([e.get_n_leaves() for e in b.estimators_])
        print(f"  {ms:>7.1f} {auc:>10.4f} {barglar:>14.1f}")

    print("\n=== 3. Bagging va pasting (bootstrap=False) ===")
    for nom, bs in [("bagging (True)", True), ("pasting (False)", False)]:
        b, auc = baho(n_estimators=100, bootstrap=bs, max_samples=0.63)
        print(f"  {nom:<18}: test AUC {auc:.4f}")

    print("\n=== 4. Bazaviy model chuqurligi ===")
    print(f"  {'max_depth':>10} {'bitta AUC':>11} {'bagging AUC':>13} "
          f"{'foyda':>8}")
    for chuqurlik in [1, 3, 5, 10, None]:
        d = DecisionTreeClassifier(max_depth=chuqurlik,
                                   random_state=0).fit(Xtr, ytr)
        a1 = roc_auc_score(yte, d.predict_proba(Xte)[:, 1])
        b = BaggingClassifier(DecisionTreeClassifier(max_depth=chuqurlik,
                                                     random_state=0),
                              n_estimators=150, random_state=0).fit(Xtr, ytr)
        a2 = roc_auc_score(yte, b.predict_proba(Xte)[:, 1])
        nom = "None" if chuqurlik is None else str(chuqurlik)
        print(f"  {nom:>10} {a1:>11.4f} {a2:>13.4f} {a2 - a1:>+8.4f}")
    print("  ⭐ Bagging uchun daraxtni CHEKLAMANG")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. n_estimators va to'yinish ===
    n_est   test AUC        OOB
        1     0.7486          -
        5     0.8582          -
       10     0.8664          -
       25     0.8637          -
       50     0.8632     0.8352
      100     0.8648     0.8438
      200     0.8647     0.8457
      400     0.8649     0.8467

=== 2. max_samples ===
    ulush   test AUC   o_rt barglar
      0.1     0.8679           33.6
      0.3     0.8636           85.3
      0.5     0.8640          126.7
      0.8     0.8667          177.8
      1.0     0.8648          204.1

=== 3. Bagging va pasting (bootstrap=False) ===
  bagging (True)    : test AUC 0.8626
  pasting (False)   : test AUC 0.8611

=== 4. Bazaviy model chuqurligi ===
   max_depth   bitta AUC   bagging AUC    foyda
           1      0.6052        0.7396  +0.1344
           3      0.8367        0.8543  +0.0176
           5      0.8477        0.8670  +0.0194
          10      0.7707        0.8658  +0.0952
        None      0.7271        0.8650  +0.1379
  ⭐ Bagging uchun daraxtni CHEKLAMANG

Nima ko'rsatdi: 2.6-bo'lim.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Bagging biasni ham kamaytiradi" Faqat dispersiyani
"Bazaviy daraxt sayoz bo'lsin" To'liq o'stiring
"n_estimators ni sozlash kerak" Overfitting bermaydi
"OOB — taxminiy, ishonchsiz" CV ga yaqin
"Chiziqli modelni bagging qilish foydali" Deyarli foyda yo'q
"Qattiq ovoz yetarli" Yumshoq odatda yaxshiroq
"Bootstrap ma'lumotni ko'paytiradi" Bir xil ma'lumot
"OOB har doim ishlatsa bo'ladi" Vaqt qatorida emas

6. Keng tarqalgan xatolar va yechimlari

1. Sayoz bazaviy daraxt

python
BaggingClassifier(DecisionTreeClassifier(max_depth=3))             # ⚠️
BaggingClassifier(DecisionTreeClassifier(random_state=0))          # ✅

2. n_estimators ni CV bilan sozlash

python
GridSearchCV(b, {"n_estimators": [50, 100, 200]})                  # ⚠️
BaggingClassifier(n_estimators=300)   # imkon qadar ko'p           # ✅

3. Chiziqli modelni bagging qilish

python
BaggingRegressor(LinearRegression(), n_estimators=100)             # ⚠️
BaggingRegressor(DecisionTreeRegressor(), n_estimators=100)        # ✅

4. Kichik n_estimators bilan OOB

python
BaggingClassifier(n_estimators=10, oob_score=True)                 # ⚠️
BaggingClassifier(n_estimators=200, oob_score=True)                # ✅

5. Vaqt qatorida OOB

python
b.oob_score_          # tasodifiy bo'linishni nazarda tutadi       # ⚠️
TimeSeriesSplit(5)    # vaqt bo'yicha                              # ✅

6. n_jobs ni unutish

python
BaggingClassifier(n_estimators=500)      # bitta yadro             # ⚠️
BaggingClassifier(n_estimators=500, n_jobs=-1)                     # ✅

7. max_samples ni juda kichik qo'yish

python
BaggingClassifier(max_samples=0.05)      # har model kuchsiz       # ⚠️
BaggingClassifier(max_samples=1.0)       # standart                # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 15.3-dars (o'tilgan): Beqarorlik
  • 15.5-dars: Random Forest (bagging + max_features)
  • 15.6-dars: OOB va Extra Trees
  • 15.7-dars: Boosting (boshqa yondashuv)
  • 15.11-dars: Permutation importance (OOB asosida)

8. Eng yaxshi amaliyotlar

  1. Bazaviy daraxtni cheklamang.

  2. n_estimators ni ko'p qo'ying.

  3. OOB dan foydalaning.

  4. Yumshoq ovoz ishlating.

  5. n_jobs=-1 qo'ying.

  6. Beqaror modelga qo'llang.

  7. Vaqt qatorida ehtiyot bo'ling.

  8. To'yinishni tekshiring.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # bagging to'liq nomi?
2.  # bootstrap nima?
3.  # noyob namunalar ulushi?
4.  # OOB ulushi?
5.  # agregatsiya usullari?
6.  # qaysi ovoz yaxshiroq?
7.  # bagging nimani kamaytiradi?
8.  # bias o'zgaradimi?
9.  # bazaviy model qanday bo'lsin?
10. # n_estimators overfitting beradimi?
11. # OOB qachon noto'g'ri?
12. # pasting nima?
Javoblar
  1. Bootstrap Aggregating
  2. Qaytarish bilan namunalash
  3. ~63.2%
  4. ~36.8%
  5. O'rtacha, qattiq/yumshoq ovoz
  6. Yumshoq
  7. Dispersiyani
  8. Yo'q
  9. Beqaror (chuqur daraxt)
  10. Yo'q
  11. Vaqt qatorida
  12. bootstrap=False

Vazifa 2: Xatolarni tuzating

python
1.  BaggingClassifier(DecisionTreeClassifier(max_depth=3))

2.  GridSearchCV(b, {"n_estimators": [50, 100, 200]})

3.  BaggingRegressor(LinearRegression(), n_estimators=200)

4.  BaggingClassifier(n_estimators=10, oob_score=True)

5.  BaggingClassifier(n_estimators=500)   # sekin
Javoblar
python
1.  BaggingClassifier(DecisionTreeClassifier(random_state=0))

2.  BaggingClassifier(n_estimators=300)

3.  BaggingRegressor(DecisionTreeRegressor(), n_estimators=200)

4.  BaggingClassifier(n_estimators=200, oob_score=True)

5.  BaggingClassifier(n_estimators=500, n_jobs=-1)

Vazifa 3: Bootstrap

Modellang:

  1. Noyob ulush
  2. Takrorlanish
  3. OOB soni
  4. Bootstrap taqsimoti

Vazifa 4: Qo'lda bagging

Modellang:

  1. B daraxt
  2. Ikki ovoz
  3. sklearn
  4. OOB va CV

Vazifa 5: Bias-dispersiya

Modellang:

  1. Bazaviy modellar
  2. Bagging dan keyin
  3. Bias tekshiruvi
  4. Chuqurlik

Vazifa 6: Parametrlar

Modellang:

  1. To'yinish
  2. max_samples
  3. Pasting
  4. Bazaviy chuqurlik

Vazifa 7: O'ylash

Bagging bir xil ma'lumotdan qayta namuna oladi — yangi hech narsa qo'shmaydi. Qanday qilib u aniqlikni oshiradi? Bu "bepul tushlik" emasmi?

Javob

Qisqa javob: bagging yangi ma'lumot qo'shmaydi, lekin u o'qitish algoritmining tasodifiyligini o'rtachalashtiradi. Bu bepul tushlik emas — u narx evaziga keladi: hisoblash vaqti va talqin qilinishning yo'qolishi. Va u faqat model beqaror bo'lgandagina ishlaydi.

1. Nima o'rtachalashtiriladi

Daraxt bashorati ikki manbadan kelib chiqadi:

Manba Bagging ta'siri
Haqiqiy signal O'zgarmaydi (hamma daraxtda bor)
O'quv to'plamining tasodifi O'rtachalashib yo'qoladi

Bitta daraxt shovqinga moslashadi; ikkinchi daraxt boshqa shovqinga moslashadi. O'rtachalashtirilganda tasodifiy komponentlar bir-birini bekor qiladi, signal esa qoladi.

2. Nega bootstrap kerak

  • Bir xil ma'lumotda bir xil algoritm bir xil daraxt beradi -> o'rtachalashtirish foydasiz
  • Bootstrap sun'iy xilma-xillik yaratadi
  • Lekin bu xilma-xillik cheklangan: daraxtlar hamon korrelyatsiyali (rho > 0)

3. Narxi

  1. Hisoblash: B barobar ko'p o'qitish va bashorat
  2. Talqin: 300 daraxtni ko'rib bo'lmaydi
  3. Xotira: modelning hajmi B barobar
  4. Foyda chegarasi: rho*sigma^2 hech qachon yo'qolmaydi

4. Qachon ishlamaydi

  • Bazaviy model barqaror (chiziqli, katta k li KNN)
  • Bazaviy model yuqori biasli (sayoz daraxt) — bias qoladi
  • Ma'lumot juda kichik (bootstrap namunalar deyarli bir xil)

5. Xulosa

  1. Bagging shovqinga moslashuvni o'rtachalashtiradi
  2. Signal qoladi, tasodif yo'qoladi
  3. Narxi — hisoblash va talqin
  4. Faqat beqaror modellarda ishlaydi

Nimani mustahkamlaydi: 2.1, 2.4-bo'limlar.

Xulosa

Bu darsda bagging ni o'rgandik.

Eng muhim uch fikr:

  1. Bootstrap 63.2% / 36.8%. Har bootstrap namunada noyob namunalarning 63.2% i ishlatiladi, 36.8% i esa chetda (out-of-bag) qoladi. Birinchi qism modellarni bir-biridan farqlantiradi, ikkinchisi esa bepul validatsiya (oob_score_) beradi — CV ga yaqin va qo'shimcha o'qitish talab qilmaydi.

  2. Bagging faqat dispersiyani kamaytiradi. E[o'rtacha] = E[bitta model] — bias o'zgarmaydi. Shuning uchun bazaviy model beqaror bo'lishi kerak: bagging da daraxtlar atayin to'liq o'stiriladi (max_depth=None). Chiziqli modelni yoki sayoz daraxtni bagging qilish deyarli foyda bermaydi.

  3. n_estimators overfitting bermaydi. Daraxtlar sonini oshirish natijani yomonlashtirmaydi — faqat hisoblash narxi oshadi va foyda 100-300 dan keyin to'yinadi. Bu boosting dan 15.8-bob asosiy farq: u yerda daraxtlar soni sozlanishi kerak bo'lgan giperparametr.

Keyingi darsda Random Forestni o'rganamiz: bagging ustiga tasodifiy belgi tanlash qo'shilganda nima o'zgaradi.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
15.4-dars: Bagging — IlmHamroh