IlmHamroh
Data Science va sun'iy intellekt/Daraxtlar va ansambllar7/14-dars19 daqiqa
Mundarija (22)

15.7-dars: Boosting g'oyasi va AdaBoost

15-QISM — DARAXTLAR VA ANSAMBLLAR · 7-dars

1. Kirish va motivatsiya

Bagging va Random Forest daraxtlarni parallel quradi: har biri mustaqil, keyin o'rtachalashtiriladi. Boosting boshqa yo'ldan boradi — modellarni ketma-ket quradi, har biri oldingilarning xatosiga e'tibor qaratadi.

Bu farq natijani tubdan o'zgartiradi: bagging dispersiyani kamaytiradi, boosting esa biasni. Shuning uchun boosting da bazaviy model atayin kuchsiz olinadi (1-6 darajali daraxt) va ansambl ularni bosqichma-bosqich kuchli modelga aylantiradi.

Bu darsda: kuchsiz o'rganuvchi tushunchasi, AdaBoost algoritmi (og'irliklarni yangilash), qo'shimcha modellashtirish (additive modeling), boosting va bagging farqi, learning_rate va overfitting xavfi.

Real vaziyat. Yuzni aniqlash bo'yicha 2001 yilgi Viola-Jones algoritmi — boosting ning eng mashhur qo'llanilishi: 180 000 ta juda sodda belgidan AdaBoost 6000 tasini tanlab, real vaqtda ishlaydigan detektor qurdi. Har belgi alohida deyarli foydasiz edi (aniqligi ~0.51), lekin ketma-ket qo'shilganda ular kuchli model berdi.

Bu darsda boosting g'oyasini o'rganamiz.

Bu darsda:

  • Kuchsiz o'rganuvchi
  • AdaBoost algoritmi
  • Qo'shimcha modellashtirish
  • Bagging bilan farq
  • learning_rate
  • Overfitting xavfi
  • Tuzoqlar
  • Amaliy: qo'lda AdaBoost

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Kuchsiz o'rganuvchi

text
KUCHSIZ o'rganuvchi (weak learner) — tasodifiydan BIROZ yaxshiroq model
  ikki sinfda: aniqlik > 0.5
  odatiy misol: "stump" — bitta bo'linishli daraxt (max_depth=1)

Boosting nazariyasi (Schapire, 1990):
  kuchsiz o'rganuvchilar KETMA-KET birlashtirilsa
  istalgan aniqlikdagi KUCHLI model olinadi

Shart: har qadamda oldingi xatoga e'tibor qaratish

Kuchsiz o'rganuvchi boosting uchun majburiy: agar bazaviy model allaqachon kuchli bo'lsa (to'liq daraxt), boosting uni yaxshilay olmaydi va tez overfitting qiladi. Bu bagging ga to'liq teskari talab.

2.2. AdaBoost algoritmi

text
1. Har namunaga teng og'irlik: w_i = 1/n
2. M marta takrorlash:
   a. Og'irliklar bilan kuchsiz model o'qitiladi
   b. Og'irlikli xato hisoblanadi:  err = sum(w_i * [xato]) / sum(w_i)
   c. Model og'irligi:              alpha = log((1 - err) / err)
   d. Namuna og'irliklari yangilanadi:
        xato bo'lganlar:  w_i *= exp(alpha)     -> OSHADI
        to'g'rilar:       o'zgarmaydi
   e. Og'irliklar normallashtiriladi
3. Yakuniy bashorat: sum(alpha_m * model_m(x)) ning belgisi

err > 0.5 bo'lsa -> alpha < 0 -> algoritm to'xtaydi
err -> 0 bo'lsa  -> alpha -> cheksiz (juda ishonchli model)

Mexanizmning o'zagi — og'irliklarni qayta taqsimlash: noto'g'ri tasniflangan namunalar og'irligi oshadi, shuning uchun keyingi model aynan ularni to'g'rilashga harakat qiladi. Model og'irligi alpha esa uning sifatiga qarab beriladi.

2.3. Qo'shimcha modellashtirish

text
Boosting — QO'SHIMCHA model (additive model):

  F_M(x) = sum_{m=1..M} alpha_m * h_m(x)

Har qadamda bitta had qo'shiladi va oldingilar O'ZGARMAYDI
  -> "oldinga bosqichma-bosqich" (forward stagewise) qurish

AdaBoost = eksponensial yo'qotish funksiyasi bilan qo'shimcha model
  L(y, F) = exp(-y * F(x)),  y in {-1, +1}

Bu kashfiyot (Friedman, 2000) gradient boosting ga yo'l ochdi (15.8)

AdaBoost ning eksponensial yo'qotish bilan bog'liqligi kashf etilgandan keyin boosting ni istalgan yo'qotish funksiyasiga umumlashtirish mumkin bo'ldi — bu gradient boosting ning tug'ilishi.

2.4. Bagging bilan farq

text
                     Bagging / RF          Boosting
qurish               parallel              ketma-ket
bazaviy model        kuchli (chuqur)       kuchsiz (sayoz)
nimani kamaytiradi   dispersiya            bias
n_estimators         ko'p bo'lsin          SOZLANADI
overfitting          deyarli yo'q          BOR
parallellashtirish   to'liq                cheklangan
shovqinga chidam     yuqori                past
sozlash              oson                  murakkab

Eng muhim amaliy farq — n_estimators boosting da overfitting beradi. Bagging da uni ko'p qo'yish bepul, boosting da esa u asosiy giperparametr (erta to'xtatish bilan tanlanadi — 15.9).

2.5. learning_rate

text
F_M(x) = sum(eta * alpha_m * h_m(x))     eta = learning_rate

eta kichik (0.01-0.1):
  + har qadam ehtiyotkor -> yaxshiroq umumlashtirish
  - ko'p model kerak (n_estimators katta)

eta katta (0.5-1.0):
  + tez o'qiydi
  - overfitting xavfi yuqori

QOIDA: eta va n_estimators BOG'LIQ
  eta ni 2 barobar kamaytirsangiz, n_estimators ni ~2 barobar oshiring

learning_rate va n_estimators — bog'langan juftlik. Amalda learning_rate ni kichik qilib (0.05-0.1), n_estimators ni erta to'xtatish bilan tanlash eng ishonchli usul.

2.6. Overfitting xavfi

text
AdaBoost shovqinga JUDA sezgir:
  noto'g'ri yorliqli namuna hech qachon to'g'ri tasniflanmaydi
  -> uning og'irligi eksponensial oshadi
  -> keyingi modellar faqat shu namunaga e'tibor qaratadi

Shuning uchun:
  toza ma'lumotda AdaBoost kuchli
  shovqinli ma'lumotda Random Forest xavfsizroq
  yoki gradient boosting (huber/log-loss bilan) - 15.8

Eksponensial yo'qotish xatoga juda katta jarima beradi, shuning uchun bitta noto'g'ri yorliq butun ansamblni buzishi mumkin. Bu AdaBoost ning eng jiddiy kamchiligi va gradient boosting ning paydo bo'lish sabablaridan biri.

2.7. Tuzoqlar

Asosiy tuzoqlar: bazaviy model sifatida chuqur daraxt berish; n_estimators ni sozlamasdan ko'p qo'yish; learning_rate ni e'tiborsiz qoldirish; shovqinli ma'lumotda AdaBoost ishlatish; yorliq xatolarini tekshirmaslik; boosting ni parallellashtirishga urinish; n_estimators va learning_rate ni alohida sozlash; AdaBoost ehtimolliklarini kalibrlangan deb hisoblash 14.10-bob.

2.8. Ketma-ket qurish

Boosting modellarni ketma-ket quradi: har biri oldingilarning xatosiga e'tibor qaratadi. Bazaviy model atayin kuchsiz (stump yoki sayoz daraxt) bo'lishi kerak. AdaBoost noto'g'ri tasniflangan namunalar og'irligini oshiradi va modellarni sifatiga qarab alpha bilan birlashtiradi. Boosting biasni kamaytiradi (bagging dispersiyani), lekin overfitting beradi va shovqinga sezgir. learning_rate va n_estimators — bog'langan juftlik. Keyingi dars — gradient boosting.

3. Tez ma'lumotnoma

python
from sklearn.ensemble import AdaBoostClassifier, AdaBoostRegressor
from sklearn.tree import DecisionTreeClassifier

a = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1),   # stump
                       n_estimators=300, learning_rate=0.1,
                       random_state=0).fit(X, y)
a.estimator_weights_             # alpha_m
a.estimator_errors_              # err_m
list(a.staged_score(Xte, yte))   # bosqichma-bosqich aniqlik
list(a.staged_predict_proba(Xte))

# qo'lda og'irlik yangilash
w[xato] *= np.exp(alpha)
QOIDA: bazaviy model kuchsiz · n_estimators ni sozla ·
       learning_rate bilan birga · shovqinga ehtiyot

Boosting xulosasi

Ketma-ket qurish; har model oldingi xatoga e'tibor qaratadi
Bazaviy model KUCHSIZ (stump / sayoz daraxt)
AdaBoost: w[xato] *= exp(alpha), alpha = log((1-err)/err)
Biasni kamaytiradi; overfitting bor; shovqinga sezgir

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14).

Misol 1 — Qo'lda AdaBoost

python
"""AdaBoost ni noldan qurish (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier


def yarat(seed: int = 4, n: int = 2000, shovqin: float = 0.05):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 6))
    qoida = (((X[:, 0] > 0.3) & (X[:, 1] < 0.2))
             | ((X[:, 2] > 0.5) & (X[:, 3] > 0.0)))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    ytr_pm = np.where(ytr == 1, 1, -1)          # {-1, +1} ko'rinish
    yte_pm = np.where(yte == 1, 1, -1)

    print("=== 1. Bitta stump ===")
    stump = DecisionTreeClassifier(max_depth=1, random_state=0).fit(Xtr, ytr)
    print(f"  o'quv aniqligi: {stump.score(Xtr, ytr):.4f}")
    print(f"  test aniqligi:  {stump.score(Xte, yte):.4f}")
    print("  (tasodifiydan biroz yaxshiroq - kuchsiz o'rganuvchi)")

    print("\n=== 2. Qo'lda AdaBoost (50 qadam) ===")
    n = len(Xtr)
    w = np.ones(n) / n
    alphalar, modellar = [], []
    F = np.zeros(len(Xte))
    print(f"  {'qadam':>6} {'err':>8} {'alpha':>8} {'test aniqlik':>14}")
    for m in range(50):
        h = DecisionTreeClassifier(max_depth=1,
                                   random_state=0).fit(Xtr, ytr, sample_weight=w)
        pred = np.where(h.predict(Xtr) == 1, 1, -1)
        xato = pred != ytr_pm
        err = float(np.clip(w[xato].sum() / w.sum(), 1e-10, 1 - 1e-10))
        alpha = float(np.log((1 - err) / err))
        w[xato] *= np.exp(alpha)
        w /= w.sum()
        alphalar.append(alpha)
        modellar.append(h)
        F += alpha * np.where(h.predict(Xte) == 1, 1, -1)
        if m in [0, 1, 4, 9, 24, 49]:
            aniqlik = (np.sign(F) == yte_pm).mean()
            print(f"  {m + 1:>6} {err:>8.4f} {alpha:>8.4f} {aniqlik:>14.4f}")

    print("\n=== 3. Og'irliklar qanday o'zgardi ===")
    print(f"  eng katta og'irlik: {w.max():.6f} (teng bo'lganda "
          f"{1 / n:.6f})")
    print(f"  eng kichik:         {w.min():.6f}")
    print(f"  og'irligi 5x dan katta namunalar: {(w > 5 / n).sum()}")
    print(f"  ularning {(w > 5 / n).sum()} tasidan nechtasi buzilgan yorliq: "
          f"kuzatib bo'lmaydi (yorliq yashirin)")

    print("\n=== 4. sklearn bilan solishtirish ===")
    a = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1, random_state=0),
                           n_estimators=50, learning_rate=1.0,
                           random_state=0).fit(Xtr, ytr)
    print(f"  qo'lda:   {(np.sign(F) == yte_pm).mean():.4f}")
    print(f"  sklearn:  {a.score(Xte, yte):.4f}")
    print(f"  alpha lar (qo'lda, birinchi 3): "
          f"{np.round(alphalar[:3], 4).tolist()}")
    print(f"  alpha lar (sklearn, birinchi 3): "
          f"{np.round(a.estimator_weights_[:3], 4).tolist()}")
    print("  ⭐ Og'irlik yangilash - boosting ning o'zagi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Bitta stump ===
  o'quv aniqligi: 0.7350
  test aniqligi:  0.7450
  (tasodifiydan biroz yaxshiroq - kuchsiz o'rganuvchi)

=== 2. Qo'lda AdaBoost (50 qadam) ===
   qadam      err    alpha   test aniqlik
       1   0.2650   1.0201         0.7450
       2   0.2477   1.1107         0.7283
       5   0.3818   0.4818         0.8433
      10   0.4496   0.2023         0.8483
      25   0.4882   0.0473         0.8567
      50   0.4917   0.0334         0.8483

=== 3. Og'irliklar qanday o'zgardi ===
  eng katta og'irlik: 0.012536 (teng bo'lganda 0.000714)
  eng kichik:         0.000071
  og'irligi 5x dan katta namunalar: 18
  ularning 18 tasidan nechtasi buzilgan yorliq: kuzatib bo'lmaydi (yorliq yashirin)

=== 4. sklearn bilan solishtirish ===
  qo'lda:   0.8483
  sklearn:  0.8483
  alpha lar (qo'lda, birinchi 3): [1.0201, 1.1107, 0.8379]
  alpha lar (sklearn, birinchi 3): [1.0201, 1.1107, 0.8379]
  ⭐ Og'irlik yangilash - boosting ning o'zagi

Nima ko'rsatdi: 2.1, 2.2-bo'limlar.

Misol 2 — Bazaviy model va learning_rate

python
"""Kuchsiz o'rganuvchi nega kerak (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier


def yarat(seed: int = 8, n: int = 3000, shovqin: float = 0.05):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 10))
    qoida = (((X[:, 0] > 0.2) & (X[:, 1] < 0.3))
             | ((X[:, 2] > 0.4) & (X[:, 3] > -0.1))
             | (X[:, 4] < -1.0))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)

    print("=== 1. Bazaviy model chuqurligi ===")
    print(f"  {'max_depth':>10} {'bitta AUC':>11} {'AdaBoost AUC':>14} "
          f"{'o_quv aniqlik':>15}")
    for chuqurlik in [1, 2, 3, 5, 10, None]:
        bitta = DecisionTreeClassifier(max_depth=chuqurlik,
                                       random_state=0).fit(Xtr, ytr)
        a1 = roc_auc_score(yte, bitta.predict_proba(Xte)[:, 1])
        ab = AdaBoostClassifier(DecisionTreeClassifier(max_depth=chuqurlik,
                                                       random_state=0),
                                n_estimators=100, learning_rate=0.5,
                                random_state=0).fit(Xtr, ytr)
        a2 = roc_auc_score(yte, ab.predict_proba(Xte)[:, 1])
        nom = "None" if chuqurlik is None else str(chuqurlik)
        print(f"  {nom:>10} {a1:>11.4f} {a2:>14.4f} "
              f"{ab.score(Xtr, ytr):>15.4f}")

    print("\n=== 2. learning_rate va n_estimators ===")
    print(f"  {'lr':>6} {'50':>9} {'100':>9} {'300':>9} {'600':>9}")
    for lr in [1.0, 0.5, 0.1, 0.05]:
        qator = []
        for ne in [50, 100, 300, 600]:
            ab = AdaBoostClassifier(DecisionTreeClassifier(max_depth=2,
                                                           random_state=0),
                                    n_estimators=ne, learning_rate=lr,
                                    random_state=0).fit(Xtr, ytr)
            qator.append(roc_auc_score(yte, ab.predict_proba(Xte)[:, 1]))
        print(f"  {lr:>6.2f} " + " ".join(f"{v:>9.4f}" for v in qator))

    print("\n=== 3. Bosqichma-bosqich o'rganish ===")
    ab = AdaBoostClassifier(DecisionTreeClassifier(max_depth=2, random_state=0),
                            n_estimators=300, learning_rate=0.5,
                            random_state=0).fit(Xtr, ytr)
    oquv = list(ab.staged_score(Xtr, ytr))
    test = list(ab.staged_score(Xte, yte))
    print(f"  {'qadam':>6} {'o_quv':>9} {'test':>9} {'farq':>9}")
    for i in [0, 9, 49, 99, 199, 299]:
        print(f"  {i + 1:>6} {oquv[i]:>9.4f} {test[i]:>9.4f} "
              f"{oquv[i] - test[i]:>9.4f}")
    eng = int(np.argmax(test))
    print(f"  eng yaxshi test: {eng + 1}-qadam ({test[eng]:.4f}), "
          f"oxirida {test[-1]:.4f}")

    print("\n=== 4. Model og'irliklari va xatolari ===")
    print(f"  {'qadam':>6} {'alpha':>9} {'err':>9}")
    for i in [0, 4, 49, 149, 299]:
        print(f"  {i + 1:>6} {ab.estimator_weights_[i]:>9.4f} "
              f"{ab.estimator_errors_[i]:>9.4f}")
    print(f"  o'rtacha err: {ab.estimator_errors_.mean():.4f}")
    print("  ⭐ Keyingi modellar qiyinroq vazifa oladi (err oshadi)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Bazaviy model chuqurligi ===
   max_depth   bitta AUC   AdaBoost AUC   o_quv aniqlik
           1      0.6549         0.8994          0.8629
           2      0.7893         0.9304          0.9514
           3      0.8962         0.9381          0.9552
           5      0.9316         0.9368          1.0000
          10      0.9200         0.9393          1.0000
        None      0.8978         0.8934          1.0000

=== 2. learning_rate va n_estimators ===
      lr        50       100       300       600
    1.00    0.9365    0.9360    0.9373    0.9350
    0.50    0.9331    0.9304    0.9299    0.9337
    0.10    0.9377    0.9317    0.9313    0.9301
    0.05    0.9024    0.9381    0.9352    0.9348

=== 3. Bosqichma-bosqich o'rganish ===
   qadam     o_quv      test      farq
       1    0.7786    0.7533    0.0252
      10    0.9524    0.9344    0.0179
      50    0.9533    0.9311    0.0222
     100    0.9514    0.9267    0.0248
     200    0.9533    0.9244    0.0289
     300    0.9538    0.9222    0.0316
  eng yaxshi test: 11-qadam 0.9367-bob, oxirida 0.9222

=== 4. Model og'irliklari va xatolari ===
   qadam     alpha       err
       1    0.6287    0.2214
       5    0.4793    0.2772
      50    0.0280    0.4860
     150    0.0090    0.4955
     300    0.0066    0.4967
  o'rtacha err: 0.4770
  ⭐ Keyingi modellar qiyinroq vazifa oladi (err oshadi)

Nima ko'rsatdi: 2.1, 2.5-bo'limlar.

Misol 3 — Shovqinga sezgirlik

python
"""AdaBoost ning asosiy kamchiligi (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import (AdaBoostClassifier, GradientBoostingClassifier,
                              RandomForestClassifier)
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier


def yarat(seed: int = 3, n: int = 3000, shovqin: float = 0.0):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 8))
    qoida = (((X[:, 0] > 0.2) & (X[:, 1] < 0.2))
             | ((X[:, 2] > 0.4) & (X[:, 3] > 0.0)))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y, alm


def main() -> None:
    print("=== 1. Shovqin darajasi bo'yicha uch model ===")
    print(f"  {'shovqin':>8} {'AdaBoost':>10} {'GradBoost':>11} {'RF':>9}")
    for sh in [0.0, 0.05, 0.15, 0.30]:
        X, y, _ = yarat(shovqin=sh)
        Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3,
                                              random_state=0, stratify=y)
        ab = AdaBoostClassifier(DecisionTreeClassifier(max_depth=2,
                                                       random_state=0),
                                n_estimators=200, learning_rate=0.5,
                                random_state=0).fit(Xtr, ytr)
        gb = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                        max_depth=3,
                                        random_state=0).fit(Xtr, ytr)
        rf = RandomForestClassifier(n_estimators=300, random_state=0,
                                    n_jobs=1).fit(Xtr, ytr)
        print(f"  {sh:>8.2f} "
              f"{roc_auc_score(yte, ab.predict_proba(Xte)[:, 1]):>10.4f} "
              f"{roc_auc_score(yte, gb.predict_proba(Xte)[:, 1]):>11.4f} "
              f"{roc_auc_score(yte, rf.predict_proba(Xte)[:, 1]):>9.4f}")

    print("\n=== 2. Buzilgan yorliqlar og'irligi ===")
    X, y, buzilgan = yarat(shovqin=0.10)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    # o'quv qismidagi buzilgan yorliqlarni topish uchun indekslarni saqlaymiz
    idx = np.arange(len(X))
    itr, _ = train_test_split(idx, test_size=0.3, random_state=0, stratify=y)
    buzilgan_tr = buzilgan[itr]
    ytr_pm = np.where(ytr == 1, 1, -1)

    n = len(Xtr)
    w = np.ones(n) / n
    for m in range(100):
        h = DecisionTreeClassifier(max_depth=2,
                                   random_state=0).fit(Xtr, ytr, sample_weight=w)
        pred = np.where(h.predict(Xtr) == 1, 1, -1)
        xato = pred != ytr_pm
        err = float(np.clip(w[xato].sum() / w.sum(), 1e-10, 1 - 1e-10))
        w[xato] *= np.exp(np.log((1 - err) / err))
        w /= w.sum()
        if m in [0, 9, 49, 99]:
            print(f"  {m + 1:>3} qadam: buzilganlarning og'irlik ulushi "
                  f"{w[buzilgan_tr].sum():.4f} "
                  f"(namunalar ulushi {buzilgan_tr.mean():.4f})")

    print("\n=== 3. Erta to'xtatish yordam beradimi ===")
    ab = AdaBoostClassifier(DecisionTreeClassifier(max_depth=2, random_state=0),
                            n_estimators=400, learning_rate=0.5,
                            random_state=0).fit(Xtr, ytr)
    test = list(ab.staged_score(Xte, yte))
    eng = int(np.argmax(test))
    print(f"  eng yaxshi qadam: {eng + 1} ({test[eng]:.4f})")
    print(f"  400-qadamda:      {test[-1]:.4f}")
    print(f"  yo'qotilgan aniqlik: {test[eng] - test[-1]:+.4f}")

    print("\n=== 4. Bazaviy model chuqurligi shovqinda ===")
    print(f"  {'max_depth':>10} {'test AUC':>10} {'o_quv aniqlik':>15}")
    for chuqurlik in [1, 2, 3, 6]:
        ab = AdaBoostClassifier(DecisionTreeClassifier(max_depth=chuqurlik,
                                                       random_state=0),
                                n_estimators=200, learning_rate=0.5,
                                random_state=0).fit(Xtr, ytr)
        print(f"  {chuqurlik:>10} "
              f"{roc_auc_score(yte, ab.predict_proba(Xte)[:, 1]):>10.4f} "
              f"{ab.score(Xtr, ytr):>15.4f}")
    print("  ⭐ Shovqinli ma'lumotda AdaBoost dan ehtiyot bo'ling")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Shovqin darajasi bo'yicha uch model ===
   shovqin   AdaBoost   GradBoost        RF
      0.00     1.0000      1.0000    1.0000
      0.05     0.9263      0.9298    0.9290
      0.15     0.8298      0.8324    0.8408
      0.30     0.7021      0.6955    0.7020

=== 2. Buzilgan yorliqlar og'irligi ===
    1 qadam: buzilganlarning og'irlik ulushi 0.2234 (namunalar ulushi 0.1005)
   10 qadam: buzilganlarning og'irlik ulushi 0.4212 (namunalar ulushi 0.1005)
   50 qadam: buzilganlarning og'irlik ulushi 0.4157 (namunalar ulushi 0.1005)
  100 qadam: buzilganlarning og'irlik ulushi 0.4037 (namunalar ulushi 0.1005)

=== 3. Erta to'xtatish yordam beradimi ===
  eng yaxshi qadam: 6 0.8978-bob
  400-qadamda:      0.8967
  yo'qotilgan aniqlik: +0.0011

=== 4. Bazaviy model chuqurligi shovqinda ===
   max_depth   test AUC   o_quv aniqlik
           1     0.8624          0.8343
           2     0.8881          0.8990
           3     0.8799          0.9052
           6     0.8817          1.0000
  ⭐ Shovqinli ma'lumotda AdaBoost dan ehtiyot bo'ling

Nima ko'rsatdi: 2.6-bo'lim.

Misol 4 — Bagging va boosting yonma-yon

python
"""Ikki ansambl falsafasi (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import (AdaBoostRegressor, BaggingRegressor,
                              RandomForestRegressor)
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor


def yarat(seed: int, n: int = 1500):
    rng = np.random.default_rng(seed)
    X = rng.uniform(-3, 3, (n, 5))
    f = (np.sin(1.5 * X[:, 0]) + 0.6 * X[:, 1] - 0.3 * X[:, 2] ** 2
         + 0.5 * X[:, 3] * (X[:, 4] > 0))
    return X, f + rng.normal(0, 0.7, n), f


def bias_dispersiya(yaratuvchi, Xte, fte, takror: int = 15):
    P = [yaratuvchi().fit(*yarat(k + 1)[:2]).predict(Xte)
         for k in range(takror)]
    P = np.array(P)
    bias2 = float(((P.mean(axis=0) - fte) ** 2).mean())
    dispersiya = float(P.var(axis=0).mean())
    return bias2, dispersiya


def main() -> None:
    Xte, yte, fte = yarat(999, 500)
    Xtr, ytr, _ = yarat(1)

    print("=== 1. Sayoz daraxt bazaviy model ===")
    variantlar = {
        "bitta daraxt(3)": lambda: DecisionTreeRegressor(max_depth=3,
                                                         random_state=0),
        "bagging(daraxt3)": lambda: BaggingRegressor(
            DecisionTreeRegressor(max_depth=3, random_state=0),
            n_estimators=100, random_state=0),
        "AdaBoost(daraxt3)": lambda: AdaBoostRegressor(
            DecisionTreeRegressor(max_depth=3, random_state=0),
            n_estimators=100, learning_rate=0.5, random_state=0),
    }
    print(f"  {'model':<20} {'bias^2':>9} {'dispersiya':>12} {'jami':>9}")
    for nom, y in variantlar.items():
        b2, d = bias_dispersiya(y, Xte, fte)
        print(f"  {nom:<20} {b2:>9.4f} {d:>12.4f} {b2 + d:>9.4f}")

    print("\n=== 2. Chuqur daraxt bazaviy model ===")
    variantlar2 = {
        "bitta daraxt(to'liq)": lambda: DecisionTreeRegressor(random_state=0),
        "bagging(to'liq)": lambda: BaggingRegressor(
            DecisionTreeRegressor(random_state=0), n_estimators=100,
            random_state=0),
        "AdaBoost(to'liq)": lambda: AdaBoostRegressor(
            DecisionTreeRegressor(random_state=0), n_estimators=100,
            learning_rate=0.5, random_state=0),
    }
    print(f"  {'model':<22} {'bias^2':>9} {'dispersiya':>12} {'jami':>9}")
    for nom, y in variantlar2.items():
        b2, d = bias_dispersiya(y, Xte, fte)
        print(f"  {nom:<22} {b2:>9.4f} {d:>12.4f} {b2 + d:>9.4f}")

    print("\n=== 3. Test R^2 ===")
    for nom, y in {**variantlar, **variantlar2}.items():
        m = y().fit(Xtr, ytr)
        print(f"  {nom:<22}: {r2_score(yte, m.predict(Xte)):>8.4f}")

    print("\n=== 4. n_estimators ta'siri ===")
    print(f"  {'n_est':>7} {'bagging R^2':>13} {'AdaBoost R^2':>14}")
    for ne in [10, 50, 100, 300, 600]:
        b = BaggingRegressor(DecisionTreeRegressor(max_depth=3, random_state=0),
                             n_estimators=ne, random_state=0).fit(Xtr, ytr)
        a = AdaBoostRegressor(DecisionTreeRegressor(max_depth=3,
                                                    random_state=0),
                              n_estimators=ne, learning_rate=0.5,
                              random_state=0).fit(Xtr, ytr)
        print(f"  {ne:>7} {r2_score(yte, b.predict(Xte)):>13.4f} "
              f"{r2_score(yte, a.predict(Xte)):>14.4f}")
    print("  ⭐ Bagging to'yinadi, boosting sozlanishi kerak")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Sayoz daraxt bazaviy model ===
  model                   bias^2   dispersiya      jami
  bitta daraxt(3)         1.0803       0.2876    1.3679
  bagging(daraxt3)        1.0325       0.0819    1.1144
  AdaBoost(daraxt3)       0.3700       0.1081    0.4781

=== 2. Chuqur daraxt bazaviy model ===
  model                     bias^2   dispersiya      jami
  bitta daraxt(to'liq)      0.2246       0.9720    1.1966
  bagging(to'liq)           0.2179       0.1140    0.3319
  AdaBoost(to'liq)          0.1662       0.1714    0.3376

=== 3. Test R^2 ===
  bitta daraxt(3)       :   0.3949
  bagging(daraxt3)      :   0.4646
  AdaBoost(daraxt3)     :   0.6946
  bitta daraxt(to'liq)  :   0.4909
  bagging(to'liq)       :   0.7221
  AdaBoost(to'liq)      :   0.7168

=== 4. n_estimators ta'siri ===
    n_est   bagging R^2   AdaBoost R^2
       10        0.4439         0.5199
       50        0.4533         0.6544
      100        0.4646         0.6946
      300        0.4653         0.7339
      600        0.4683         0.7405
  ⭐ Bagging to'yinadi, boosting sozlanishi kerak

Nima ko'rsatdi: 2.4-bo'lim.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Boosting ham dispersiyani kamaytiradi" Asosan biasni
"Bazaviy model kuchli bo'lsin" Kuchsiz bo'lsin
"n_estimators ko'p bo'lsa yaxshi" Overfitting beradi
"learning_rate mustaqil" n_estimators bilan bog'liq
"AdaBoost shovqinga chidamli" Juda sezgir
"Boosting parallellashadi" Ketma-ket
"AdaBoost eski, foydasiz" Toza ma'lumotda kuchli
"alpha modelning aniqligi" log((1-err)/err)

6. Keng tarqalgan xatolar va yechimlari

1. Chuqur bazaviy model

python
AdaBoostClassifier(DecisionTreeClassifier())                      # ⚠️
AdaBoostClassifier(DecisionTreeClassifier(max_depth=2))           # ✅

2. n_estimators ni sozlamaslik

python
AdaBoostClassifier(n_estimators=2000)                             # ⚠️
# staged_score bilan eng yaxshi qadamni toping                    # ✅

3. learning_rate ni e'tiborsiz qoldirish

python
AdaBoostClassifier(n_estimators=500)      # lr=1.0                # ⚠️
AdaBoostClassifier(n_estimators=500, learning_rate=0.1)           # ✅

4. Shovqinli ma'lumotda AdaBoost

python
AdaBoostClassifier(...)   # 20% yorliq xatosi                     # ⚠️
RandomForestClassifier(n_estimators=500)                          # ✅

5. lr va n_estimators ni alohida sozlash

python
GridSearchCV(a, {"learning_rate": [...]})  # n_est qotib qolgan   # ⚠️
GridSearchCV(a, {"learning_rate": [0.05, 0.1], "n_estimators": [200, 500]}) # ✅

6. Ehtimolliklarga ishonish

python
a.predict_proba(X)[:, 1]   # kalibrlanmagan                       # ⚠️
CalibratedClassifierCV(a, cv=5)                                   # ✅

7. Yorliq xatolarini tekshirmaslik

python
# to'g'ridan-to'g'ri boosting                                     # ⚠️
# avval shubhali yorliqlarni tekshiring 14.13-bob                   # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 15.4-dars (o'tilgan): Bagging
  • 15.8-dars: Gradient boosting
  • 15.9-dars: Sozlash va erta to'xtatish
  • 15.10-dars: XGBoost va LightGBM
  • 12.5-dars (o'tilgan): Bias-variance

8. Eng yaxshi amaliyotlar

  1. Bazaviy modelni kuchsiz qiling.

  2. n_estimators ni sozlang.

  3. learning_rate bilan birga tanlang.

  4. staged_score dan foydalaning.

  5. Shovqinni tekshiring.

  6. Kalibrlang.

  7. RF bilan taqqoslang.

  8. Erta to'xtatishni qo'llang.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # kuchsiz o'rganuvchi nima?
2.  # stump nima?
3.  # AdaBoost og'irlikni qanday yangilaydi?
4.  # alpha formulasi?
5.  # err > 0.5 bo'lsa?
6.  # boosting nimani kamaytiradi?
7.  # bagging nimani?
8.  # n_estimators overfitting beradimi?
9.  # learning_rate nima bilan bog'liq?
10. # AdaBoost qaysi yo'qotish funksiyasi?
11. # nega shovqinga sezgir?
12. # parallellashadimi?
Javoblar
  1. Tasodifiydan biroz yaxshiroq
  2. Bitta bo'linishli daraxt
  3. w[xato] *= exp(alpha)
  4. log((1-err)/err)
  5. Algoritm to'xtaydi
  6. Biasni
  7. Dispersiyani
  8. Ha
  9. n_estimators bilan
  10. Eksponensial
  11. exp jarima juda katta
  12. Yo'q

Vazifa 2: Xatolarni tuzating

python
1.  AdaBoostClassifier(DecisionTreeClassifier())

2.  AdaBoostClassifier(n_estimators=3000)

3.  AdaBoostClassifier(n_estimators=500)   # lr=1.0

4.  AdaBoostClassifier(...)   # 25% yorliq xatosi

5.  a.predict_proba(X)[:, 1]  # ehtimollik kerak
Javoblar
python
1.  AdaBoostClassifier(DecisionTreeClassifier(max_depth=2))

2.  # staged_score bilan eng yaxshi qadamni toping

3.  AdaBoostClassifier(n_estimators=500, learning_rate=0.1)

4.  RandomForestClassifier(n_estimators=500)

5.  CalibratedClassifierCV(a, cv=5)

Vazifa 3: Qo'lda AdaBoost

Modellang:

  1. Bitta stump
  2. 50 qadam
  3. Og'irliklar
  4. sklearn bilan

Vazifa 4: Parametrlar

Modellang:

  1. Bazaviy chuqurlik
  2. lr va n_estimators
  3. Bosqichma-bosqich
  4. alpha va err

Vazifa 5: Shovqin

Modellang:

  1. Uch model
  2. Buzilgan og'irliklar
  3. Erta to'xtatish
  4. Chuqurlik

Vazifa 6: Yonma-yon

Modellang:

  1. Sayoz bazaviy
  2. Chuqur bazaviy
  3. Test R^2
  4. n_estimators

Vazifa 7: O'ylash

AdaBoost 1995 yilda yaratilgan va uzoq vaqt "eng yaxshi tayyor algoritm" hisoblangan. Bugun uni deyarli hech kim ishlatmaydi. Nega — va nimani o'rganish kerak?

Javob

Qisqa javob: AdaBoost ning g'oyasi yashab qoldi, amalga oshirilishi esa gradient boosting bilan almashtirildi. Eksponensial yo'qotish funksiyasi shovqinga chidamsiz, va uni almashtirish imkoni yo'q edi — gradient boosting esa istalgan yo'qotish funksiyasi bilan ishlaydi.

1. Nima o'rnini egalladi

Jihat AdaBoost Gradient boosting
Yo'qotish funksiyasi Faqat eksponensial Istalgan
Shovqinga chidam Past O'rtacha (huber/log-loss)
Regressiya Cheklangan To'liq
Regulyarizatsiya Faqat lr lr, subsample, chuqurlik, L1/L2
Tezlik O'rtacha LightGBM/XGBoost bilan juda tez

2. Nima yashab qoldi

  1. Ketma-ket qurish g'oyasi — barcha boosting algoritmlarining asosi
  2. Kuchsiz o'rganuvchi tushunchasi
  3. Qo'shimcha model (additive) tuzilmasi
  4. learning_rate orqali qadamni kichraytirish

3. AdaBoost hali ham foydali joylar

  • Juda toza ma'lumot (sun'iy, o'lchangan)
  • Kichik ma'lumot va tez natija kerak
  • O'quv maqsadida — mexanizm ko'rinib turadi
  • Viola-Jones kabi maxsus qo'llanmalar

4. Nimani o'rganish kerak

  1. Og'irlik yangilash mexanizmi (gradient boosting da u "qoldiq" ga aylanadi)
  2. Kuchsiz/kuchli o'rganuvchi farqi
  3. n_estimators va learning_rate bog'liqligi
  4. Shovqinning boosting ga ta'siri

5. Xulosa

  1. G'oya qoldi, amalga oshirilish o'zgardi
  2. Eksponensial yo'qotish — asosiy cheklov
  3. Gradient boosting uni umumlashtirdi
  4. AdaBoost ni tushunish gradient boosting ni tushunishga yordam beradi

Nimani mustahkamlaydi: 2.3, 2.6-bo'limlar.

Xulosa

Bu darsda boosting g'oyasi va AdaBoost ni o'rgandik.

Eng muhim uch fikr:

  1. Boosting ketma-ket quradi. Bagging modellarni parallel qurib o'rtachalashtiradi, boosting esa ularni ketma-ket qo'shadi — har biri oldingilarning xatosiga e'tibor qaratadi. Shuning uchun bazaviy model atayin kuchsiz olinadi (stump yoki 2-3 darajali daraxt): chuqur daraxt bilan boosting ishlamaydi va tez overfitting qiladi.

  2. AdaBoost og'irliklarni qayta taqsimlaydi. Noto'g'ri tasniflangan namunalar og'irligi exp(alpha) ga ko'payadi, modellar esa sifatiga qarab alpha = log((1-err)/err) og'irligi bilan birlashtiriladi. Bu eksponensial yo'qotish bilan qo'shimcha modellashtirishga teng ekanligi keyinchalik gradient boosting ga yo'l ochdi.

  3. Boosting biasni kamaytiradi, lekin overfitting beradi. n_estimators bagging dagidek bepul emas — u sozlanishi kerak bo'lgan asosiy giperparametr va learning_rate bilan bog'langan. Eksponensial yo'qotish tufayli AdaBoost shovqinga juda sezgir: bitta buzilgan yorliq og'irligi eksponensial o'sib butun ansamblni buzishi mumkin.

Keyingi darsda gradient boostingni o'rganamiz: xatoga qarab og'irlik emas, qoldiqqa qarab yangi daraxt qurish.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
15.7-dars: Boosting g'oyasi va AdaBoost — IlmHamroh