IlmHamroh
Data Science va sun'iy intellekt/Daraxtlar va ansambllar8/14-dars18 daqiqa
Mundarija (22)

15.8-dars: Gradient boosting

15-QISM — DARAXTLAR VA ANSAMBLLAR · 8-dars

1. Kirish va motivatsiya

Gradient boosting — AdaBoost g'oyasining umumlashmasi: og'irliklarni yangilash o'rniga har qadamda qoldiqqa (aniqrog'i, yo'qotish funksiyasining manfiy gradientiga) daraxt moslanadi. Bu kichik o'zgarish algoritmni istalgan yo'qotish funksiyasi bilan ishlashga imkon beradi.

Natija — jadval ma'lumotlari uchun eng kuchli modellar oilasi. Kaggle musobaqalarining katta qismi gradient boosting (XGBoost, LightGBM, CatBoost) bilan yutilgan va sanoatdagi ko'p tavsiya, narx va risk modellari shu asosda qurilgan.

Bu darsda: gradient tushish va funksional fazo, qoldiq ga moslash, yo'qotish funksiyalari (squared_error, absolute_error, huber, log_loss), learning_rate, subsample, max_depth va HistGradientBoosting.

Real vaziyat. Sug'urta zararini bashorat qilishda squared_error bilan model katta da'volarga haddan ortiq e'tibor berardi — bir nechta 100 mln lik hodisa butun modelni tortib ketgan edi. loss="huber" ga o'tilgach, o'rtacha mutlaq xato 22% ga yaxshilandi. Yo'qotish funksiyasini almashtirish — gradient boosting ning asosiy afzalligi.

Bu darsda gradient boosting ni o'rganamiz.

Bu darsda:

  • Qoldiqqa moslash
  • Funksional gradient tushish
  • Yo'qotish funksiyalari
  • learning_rate va n_estimators
  • subsample
  • HistGradientBoosting
  • Tuzoqlar
  • Amaliy: qo'lda boosting

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Qoldiqqa moslash

text
Eng sodda ko'rinish (regressiya, squared_error):

1. F_0(x) = y ning o'rtachasi
2. m = 1..M uchun:
   a. qoldiq:  r_i = y_i - F_{m-1}(x_i)
   b. qoldiqqa daraxt moslanadi:  h_m ~ r
   c. F_m(x) = F_{m-1}(x) + eta * h_m(x)
3. Yakuniy: F_M(x)

Ya'ni har daraxt OLDINGI modelning XATOSINI bashorat qiladi

Bu — gradient boosting ning eng tushunarli ko'rinishi: har daraxt qoldiqni bashorat qiladi. squared_error da qoldiq aynan manfiy gradientga teng, shuning uchun bu sodda tasvir to'g'ri ishlaydi.

2.2. Funksional gradient tushish

text
Umumiy holat: qoldiq emas, MANFIY GRADIENT

  g_i = -dL(y_i, F(x_i)) / dF(x_i)

  squared_error:  g_i = y_i - F(x_i)          (qoldiq)
  absolute_error: g_i = sign(y_i - F(x_i))    (faqat belgi)
  log_loss:       g_i = y_i - p_i             (ehtimollik xatosi)

Daraxt g ga moslanadi -> bu funksiyalar fazosida gradient tushish 13.6-bob

Har barg uchun optimal qiymat alohida hisoblanadi (line search)

Funksional fazoda gradient tushish — Friedman ning asosiy g'oyasi (2001): parametrlar bo'yicha emas, funksiya bo'yicha optimallashtirish. Har qadam eta uzunlikdagi qadam tashlaydi, daraxt esa yo'nalishni beradi.

2.3. Yo'qotish funksiyalari

text
Regressiya:
  squared_error   — standart; chetlangan qiymatlarga sezgir
  absolute_error  — medianaga moslaydi; chidamli, sekinroq
  huber           — ikkalasining o'rtasi (alpha bilan)
  quantile        — kvantil regressiya (alpha bilan) - 13.11

Klassifikatsiya:
  log_loss        — standart (ehtimollik)
  exponential     — AdaBoost bilan bir xil

sklearn: GradientBoostingRegressor(loss="huber", alpha=0.9)

Yo'qotish funksiyasini almashtirish — gradient boosting ning AdaBoost dan asosiy afzalligi. Chetlangan qiymatlar bo'lgan ma'lumotda huber yoki absolute_error ko'pincha sezilarli yaxshilanish beradi.

2.4. learning_rate va n_estimators

text
F_M(x) = F_0 + eta * sum(h_m(x))

eta kichik  -> har qadam kichik -> ko'p daraxt kerak, yaxshi umumlashtirish
eta katta   -> tez o'qiydi      -> overfitting xavfi

AMALIY QOIDA:
  eta = 0.05..0.1 qo'ying
  n_estimators ni ERTA TO'XTATISH bilan toping 15.9-bob
  eta ni 2x kamaytirsangiz, n_estimators ni ~2x oshiring

Boosting da n_estimators overfitting beradi — bu bagging dan asosiy farq 15.4-bob. Uni CV yoki validatsiya to'plami bilan albatta tanlang.

2.5. subsample

text
subsample < 1.0 -> Stochastic Gradient Boosting (Friedman, 2002)
  har qadamda ma'lumotning tasodifiy qismi ishlatiladi

  + regulyarizatsiya (overfitting kamayadi)
  + tezlik
  + OOB baho imkoniyati (sklearn: oob_improvement_)

Odatiy qiymat: 0.5 - 0.8

max_features ham qo'llaniladi (RF dagi kabi)

subsample=0.8 — deyarli bepul yaxshilanish: u tezlikni ham oshiradi, overfitting ni ham kamaytiradi. Ko'p amaliyotchilar uni standart sifatida qo'yadi.

2.6. HistGradientBoosting

text
HistGradientBoostingClassifier / Regressor (sklearn 0.21+)

  belgilarni 255 ta "savat" (bin) ga bo'ladi -> bo'linish qidiruvi juda tez
  LightGBM g'oyasiga asoslangan
  10-100x TEZROQ (katta ma'lumotda)
  NaN ni o'zi boshqaradi
  categorical_features bilan kategoriyalarni to'g'ridan-to'g'ri qabul qiladi
  early_stopping="auto" (n > 10000 bo'lsa avtomatik)

Eski GradientBoosting* — kichik ma'lumot va staged_* uchun

HistGradientBoosting* — sklearn dagi zamonaviy tanlov: eski GradientBoosting* ni faqat kichik ma'lumotlarda yoki staged_predict kerak bo'lganda ishlating. Tezlik farqi 10 000 qatordan boshlab sezilarli.

2.7. Tuzoqlar

Asosiy tuzoqlar: n_estimators ni sozlamaslik; learning_rate=1.0 bilan ishlatish; chuqur daraxtlar (max_depth > 8) qo'yish; chetlangan qiymatlarda squared_error da qolish; katta ma'lumotda eski GradientBoosting* ni ishlatish; subsample ni unutish; erta to'xtatishni test to'plamida qilish (12.9 — leakage); boosting ni shovqinli ma'lumotda sozlamasdan qo'llash.

2.8. Qoldiqqa qarab o'sish

Gradient boosting har qadamda yo'qotish funksiyasining manfiy gradientiga (eng sodda holatda — qoldiqqa) daraxt moslaydi va uni eta bilan qo'shadi. Bu funksional fazoda gradient tushish va u istalgan yo'qotish funksiyasi bilan ishlaydi (huber, quantile, log_loss). learning_rate kichik, n_estimators erta to'xtatish bilan, max_depth 3-6, subsample 0.5-0.8 — odatiy retsept. Katta ma'lumotda HistGradientBoosting*. Keyingi dars — sozlash va erta to'xtatish.

3. Tez ma'lumotnoma

python
from sklearn.ensemble import (GradientBoostingClassifier, GradientBoostingRegressor,
                              HistGradientBoostingClassifier,
                              HistGradientBoostingRegressor)

g = GradientBoostingRegressor(loss="huber", alpha=0.9, learning_rate=0.05,
                              n_estimators=500, max_depth=3, subsample=0.8,
                              random_state=0).fit(X, y)
list(g.staged_predict(Xte))        # bosqichma-bosqich
g.train_score_, g.oob_improvement_  # subsample < 1 bo'lsa

h = HistGradientBoostingClassifier(learning_rate=0.05, max_iter=500,
                                   early_stopping=True, validation_fraction=0.1,
                                   random_state=0).fit(X, y)
h.n_iter_                          # erta to'xtagan qadam
QOIDA: eta kichik · n_estimators ni erta to'xtatish bilan ·
       max_depth 3-6 · subsample 0.8 · katta ma'lumotda Hist*

Gradient boosting xulosasi

Har daraxt manfiy gradientga (qoldiqqa) moslanadi
F_m = F_{m-1} + eta * h_m; funksional gradient tushish
Istalgan yo'qotish: squared/absolute/huber/quantile/log_loss
eta 0.05-0.1, max_depth 3-6, subsample 0.8, n_estimators - erta to'xtatish

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14).

Misol 1 — Qo'lda gradient boosting

python
"""Qoldiqqa moslashni noldan qurish (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor


def yarat(seed: int = 5, n: int = 1500):
    rng = np.random.default_rng(seed)
    X = rng.uniform(-3, 3, (n, 4))
    f = (np.sin(1.5 * X[:, 0]) + 0.6 * X[:, 1] - 0.3 * X[:, 2] ** 2
         + 0.5 * X[:, 3])
    return X, f + rng.normal(0, 0.5, n), f


def main() -> None:
    X, y, f = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

    print("=== 1. Boshlang'ich model ===")
    F_tr = np.full(len(ytr), ytr.mean())
    F_te = np.full(len(yte), ytr.mean())
    print(f"  F_0 = o'rtacha = {ytr.mean():.4f}")
    print(f"  test MSE: {mean_squared_error(yte, F_te):.4f}")

    print("\n=== 2. Qo'lda 100 qadam (eta = 0.1) ===")
    eta = 0.1
    print(f"  {'qadam':>6} {'qoldiq std':>12} {'o_quv MSE':>11} {'test MSE':>10}")
    for m in range(100):
        qoldiq = ytr - F_tr
        h = DecisionTreeRegressor(max_depth=3, random_state=0).fit(Xtr, qoldiq)
        F_tr = F_tr + eta * h.predict(Xtr)
        F_te = F_te + eta * h.predict(Xte)
        if m in [0, 4, 19, 49, 99]:
            print(f"  {m + 1:>6} {qoldiq.std():>12.4f} "
                  f"{mean_squared_error(ytr, F_tr):>11.4f} "
                  f"{mean_squared_error(yte, F_te):>10.4f}")

    print("\n=== 3. sklearn bilan solishtirish ===")
    g = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1,
                                  max_depth=3, random_state=0).fit(Xtr, ytr)
    print(f"  qo'lda:  test MSE {mean_squared_error(yte, F_te):.4f}")
    print(f"  sklearn: test MSE "
          f"{mean_squared_error(yte, g.predict(Xte)):.4f}")
    print(f"  (sklearn har bargda optimal qiymatni alohida hisoblaydi)")

    print("\n=== 4. eta ning ta'siri ===")
    print(f"  {'eta':>6} {'50 qadam':>10} {'100 qadam':>11} {'400 qadam':>11}")
    for e in [0.5, 0.2, 0.1, 0.03]:
        qator = []
        for ne in [50, 100, 400]:
            gm = GradientBoostingRegressor(n_estimators=ne, learning_rate=e,
                                           max_depth=3,
                                           random_state=0).fit(Xtr, ytr)
            qator.append(mean_squared_error(yte, gm.predict(Xte)))
        print(f"  {e:>6.2f} " + " ".join(f"{v:>10.4f}" for v in qator))
    print("  ⭐ Kichik eta ko'p qadam talab qiladi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Boshlang'ich model ===
  F_0 = o'rtacha = -0.9358
  test MSE: 2.9935

=== 2. Qo'lda 100 qadam (eta = 0.1) ===
   qadam   qoldiq std   o_quv MSE   test MSE
       1       1.7534      2.7566     2.6997
       5       1.4422      1.9184     1.8969
      20       0.9233      0.7947     0.9111
      50       0.5479      0.2954     0.4343
     100       0.4163      0.1724     0.3228

=== 3. sklearn bilan solishtirish ===
  qo'lda:  test MSE 0.3228
  sklearn: test MSE 0.3235
  (sklearn har bargda optimal qiymatni alohida hisoblaydi)

=== 4. eta ning ta'siri ===
     eta   50 qadam   100 qadam   400 qadam
    0.50     0.4573     0.4562     0.4939
    0.20     0.3420     0.3369     0.3683
    0.10     0.4348     0.3235     0.3311
    0.03     1.1816     0.6862     0.3290
  ⭐ Kichik eta ko'p qadam talab qiladi

Nima ko'rsatdi: 2.1, 2.4-bo'limlar.

Misol 2 — Yo'qotish funksiyalari

python
"""Chetlangan qiymatlarda qaysi yo'qotish yaxshiroq (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, mean_squared_error
from sklearn.model_selection import train_test_split


def yarat(seed: int = 7, n: int = 2000, chetlangan: float = 0.03):
    rng = np.random.default_rng(seed)
    X = rng.uniform(-3, 3, (n, 4))
    f = np.sin(1.5 * X[:, 0]) + 0.7 * X[:, 1] - 0.3 * X[:, 2] ** 2
    y = f + rng.normal(0, 0.5, n)
    buzuq = rng.random(n) < chetlangan
    y[buzuq] += rng.normal(0, 15, buzuq.sum())       # katta chetlanishlar
    return X, y, f, buzuq


def main() -> None:
    X, y, f, buzuq = yarat()
    itr, ite = train_test_split(np.arange(len(X)), test_size=0.3,
                                random_state=0)
    Xtr, Xte, ytr, yte = X[itr], X[ite], y[itr], y[ite]
    fte = f[ite]

    print("=== 1. Ma'lumot ===")
    print(f"  {len(X)} namuna, chetlangan {buzuq.sum()} ta "
          f"({buzuq.mean():.1%})")
    print(f"  y diapazoni: {y.min():.1f} .. {y.max():.1f}")

    print("\n=== 2. Yo'qotish funksiyalari (toza signalga nisbatan) ===")
    print(f"  {'loss':<18} {'MAE(f)':>9} {'RMSE(f)':>10} {'MAE(y)':>9}")
    for nom, parametrlar in [("squared_error", {"loss": "squared_error"}),
                             ("absolute_error", {"loss": "absolute_error"}),
                             ("huber 0.9-bob", {"loss": "huber", "alpha": 0.9}),
                             ("huber 0.7-bob", {"loss": "huber", "alpha": 0.7})]:
        g = GradientBoostingRegressor(n_estimators=300, learning_rate=0.05,
                                      max_depth=3, subsample=0.8,
                                      random_state=0, **parametrlar).fit(Xtr, ytr)
        pred = g.predict(Xte)
        print(f"  {nom:<18} {mean_absolute_error(fte, pred):>9.4f} "
              f"{np.sqrt(mean_squared_error(fte, pred)):>10.4f} "
              f"{mean_absolute_error(yte, pred):>9.4f}")

    print("\n=== 3. Chetlanish ulushi bo'yicha ===")
    print(f"  {'ulush':>7} {'squared':>10} {'huber':>10} {'absolute':>10}")
    for ch in [0.0, 0.01, 0.05, 0.10]:
        Xn, yn, fn, _ = yarat(chetlangan=ch)
        ia, ib = train_test_split(np.arange(len(Xn)), test_size=0.3,
                                  random_state=0)
        qator = []
        for parametrlar in [{"loss": "squared_error"},
                            {"loss": "huber", "alpha": 0.9},
                            {"loss": "absolute_error"}]:
            g = GradientBoostingRegressor(n_estimators=200, learning_rate=0.05,
                                          max_depth=3, subsample=0.8,
                                          random_state=0,
                                          **parametrlar).fit(Xn[ia], yn[ia])
            qator.append(mean_absolute_error(fn[ib], g.predict(Xn[ib])))
        print(f"  {ch:>7.2f} " + " ".join(f"{v:>10.4f}" for v in qator))

    print("\n=== 4. Kvantil regressiya ===")
    print(f"  {'kvantil':>8} {'qoplash %':>11} {'nazariy %':>11}")
    modellar = {}
    for q in [0.1, 0.5, 0.9]:
        g = GradientBoostingRegressor(loss="quantile", alpha=q,
                                      n_estimators=200, learning_rate=0.05,
                                      max_depth=3,
                                      random_state=0).fit(Xtr, ytr)
        modellar[q] = g.predict(Xte)
        qoplash = (yte <= modellar[q]).mean()
        print(f"  {q:>8.1f} {qoplash:>10.1%} {q:>10.1%}")
    kenglik = (modellar[0.9] - modellar[0.1])
    ichida = ((yte >= modellar[0.1]) & (yte <= modellar[0.9])).mean()
    print(f"  80% oraliq: qoplash {ichida:.1%}, "
          f"o'rtacha kenglik {kenglik.mean():.3f}")
    print("  ⭐ Yo'qotish funksiyasini almashtirish - asosiy afzallik")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ma'lumot ===
  2000 namuna, chetlangan 51 ta (2.5%)
  y diapazoni: -41.0 .. 27.5

=== 2. Yo'qotish funksiyalari (toza signalga nisbatan) ===
  loss                  MAE(f)    RMSE(f)    MAE(y)
  squared_error         0.5403     1.2319    1.0901
  absolute_error        0.1860     0.2375    0.7893
  huber 0.9-bob           0.1919     0.3302    0.8081
  huber 0.7-bob           0.1822     0.2466    0.7940

=== 3. Chetlanish ulushi bo'yicha ===
    ulush    squared      huber   absolute
     0.00     0.1700     0.1741     0.2022
     0.01     0.2767     0.1701     0.2010
     0.05     0.6518     0.2477     0.2172
     0.10     0.9967     0.2722     0.2107

=== 4. Kvantil regressiya ===
   kvantil   qoplash %   nazariy %
       0.1      13.0%      10.0%
       0.5      52.8%      50.0%
       0.9      87.8%      90.0%
  80% oraliq: qoplash 74.8%, o'rtacha kenglik 1.474
  ⭐ Yo'qotish funksiyasini almashtirish - asosiy afzallik

Nima ko'rsatdi: 2.3-bo'lim.

Misol 3 — Giperparametrlar

python
"""max_depth, subsample va n_estimators (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split


def yarat(seed: int = 11, n: int = 4000, p: int = 12, shovqin: float = 0.10):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    qoida = (((X[:, 0] > 0.2) & (X[:, 1] < 0.3))
             | ((X[:, 2] > 0.4) & (X[:, 3] > -0.1))
             | (X[:, 4] < -1.0))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)

    print("=== 1. max_depth ===")
    print(f"  {'max_depth':>10} {'o_quv AUC':>11} {'test AUC':>10} "
          f"{'farq':>8}")
    for chuqurlik in [1, 2, 3, 5, 8]:
        g = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                       max_depth=chuqurlik,
                                       random_state=0).fit(Xtr, ytr)
        a1 = roc_auc_score(ytr, g.predict_proba(Xtr)[:, 1])
        a2 = roc_auc_score(yte, g.predict_proba(Xte)[:, 1])
        print(f"  {chuqurlik:>10} {a1:>11.4f} {a2:>10.4f} {a1 - a2:>8.4f}")

    print("\n=== 2. subsample ===")
    print(f"  {'subsample':>10} {'test AUC':>10}")
    for ss in [1.0, 0.8, 0.5, 0.3]:
        g = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                       max_depth=3, subsample=ss,
                                       random_state=0).fit(Xtr, ytr)
        print(f"  {ss:>10.1f} "
              f"{roc_auc_score(yte, g.predict_proba(Xte)[:, 1]):>10.4f}")

    print("\n=== 3. n_estimators overfitting beradimi ===")
    g = GradientBoostingClassifier(n_estimators=600, learning_rate=0.2,
                                   max_depth=5, random_state=0).fit(Xtr, ytr)
    bosqichlar = list(g.staged_predict_proba(Xte))
    test_auc = [roc_auc_score(yte, p[:, 1]) for p in bosqichlar]
    oquv = list(g.staged_predict_proba(Xtr))
    oquv_auc = [roc_auc_score(ytr, p[:, 1]) for p in oquv]
    print(f"  {'qadam':>6} {'o_quv AUC':>11} {'test AUC':>10}")
    for i in [9, 49, 99, 299, 599]:
        print(f"  {i + 1:>6} {oquv_auc[i]:>11.4f} {test_auc[i]:>10.4f}")
    eng = int(np.argmax(test_auc))
    print(f"  eng yaxshi: {eng + 1}-qadam ({test_auc[eng]:.4f}), "
          f"oxirida {test_auc[-1]:.4f}")
    print("  (bagging dan farqli: ko'p daraxt ZARAR qilishi mumkin)")

    print("\n=== 4. learning_rate va n_estimators bog'liqligi ===")
    print(f"  {'eta':>6} {'eng yaxshi qadam':>18} {'test AUC':>10}")
    for e in [0.3, 0.1, 0.05, 0.02]:
        g = GradientBoostingClassifier(n_estimators=500, learning_rate=e,
                                       max_depth=3, subsample=0.8,
                                       random_state=0).fit(Xtr, ytr)
        auc = [roc_auc_score(yte, p[:, 1])
               for p in g.staged_predict_proba(Xte)]
        eng = int(np.argmax(auc))
        print(f"  {e:>6.2f} {eng + 1:>18} {auc[eng]:>10.4f}")
    print("  ⭐ eta kichrayganda kerakli qadamlar soni oshadi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. max_depth ===
   max_depth   o_quv AUC   test AUC     farq
           1      0.8872     0.8830   0.0042
           2      0.9422     0.9101   0.0322
           3      0.9777     0.9113   0.0664
           5      0.9996     0.9123   0.0874
           8      1.0000     0.9204   0.0796

=== 2. subsample ===
   subsample   test AUC
         1.0     0.9113
         0.8     0.9139
         0.5     0.9129
         0.3     0.9127

=== 3. n_estimators overfitting beradimi ===
   qadam   o_quv AUC   test AUC
      10      0.9520     0.9124
      50      0.9939     0.9126
     100      0.9997     0.9130
     300      1.0000     0.9143
     600      1.0000     0.9165
  eng yaxshi: 567-qadam 0.9165-bob, oxirida 0.9165
  (bagging dan farqli: ko'p daraxt ZARAR qilishi mumkin)

=== 4. learning_rate va n_estimators bog'liqligi ===
     eta   eng yaxshi qadam   test AUC
    0.30                  9     0.9187
    0.10                  9     0.9194
    0.05                 21     0.9193
    0.02                 54     0.9184
  ⭐ eta kichrayganda kerakli qadamlar soni oshadi

Nima ko'rsatdi: 2.4, 2.5-bo'limlar.

Misol 4 — HistGradientBoosting

python
"""Zamonaviy sklearn implementatsiyasi (real numpy/pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.ensemble import (GradientBoostingClassifier,
                              HistGradientBoostingClassifier,
                              RandomForestClassifier)
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split


def yarat(seed: int = 2, n: int = 20000) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    hudud = rng.choice(["toshkent", "samarqand", "buxoro", "fargona"], n,
                       p=[0.4, 0.25, 0.2, 0.15])
    tur = rng.choice(["oddiy", "tezkor", "yirik"], n, p=[0.6, 0.3, 0.1])
    masofa = rng.gamma(3, 60, n)
    ogirlik = rng.gamma(2, 4, n)
    soat = rng.integers(0, 24, n).astype(float)
    hq = pd.Series(hudud).map({"toshkent": 0.0, "samarqand": 0.4,
                               "buxoro": 0.8, "fargona": 0.5}).to_numpy()
    tt = pd.Series(tur).map({"oddiy": 0.0, "tezkor": -0.9,
                             "yirik": 1.0}).to_numpy()
    kuch = (-3.0 + 0.006 * masofa + 0.05 * ogirlik + hq + tt
            + 1.0 * ((soat >= 7) & (soat <= 10)))
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    df = pd.DataFrame({"hudud": pd.Categorical(hudud),
                       "tur": pd.Categorical(tur), "masofa": masofa,
                       "ogirlik": ogirlik, "soat": soat, "kechikdi": y})
    # 3% yo'qolgan qiymat
    yoq = rng.random(n) < 0.03
    df.loc[yoq, "ogirlik"] = np.nan
    return df


def main() -> None:
    df = yarat()
    X = df.drop(columns="kechikdi")
    y = df["kechikdi"]
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)

    print("=== 1. Ma'lumot ===")
    print(f"  {len(df)} qator, kechikish {y.mean():.2%}")
    print(f"  yo'qolgan qiymatlar: {int(df['ogirlik'].isna().sum())}")
    print(f"  kategoriyali: hudud, tur")

    print("\n=== 2. HistGradientBoosting (NaN va kategoriya avtomatik) ===")
    h = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=400,
                                       early_stopping=True,
                                       validation_fraction=0.15,
                                       categorical_features="from_dtype",
                                       random_state=0).fit(Xtr, ytr)
    ph = h.predict_proba(Xte)[:, 1]
    print(f"  erta to'xtagan qadam: {h.n_iter_} (max 400)")
    print(f"  test ROC AUC: {roc_auc_score(yte, ph):.4f}")

    print("\n=== 3. Boshqa modellar bilan solishtirish ===")
    # eski GradientBoosting NaN va kategoriyani qabul qilmaydi
    Xtr2 = pd.get_dummies(Xtr, columns=["hudud", "tur"]).fillna(-1)
    Xte2 = pd.get_dummies(Xte, columns=["hudud", "tur"]).fillna(-1)
    Xte2 = Xte2.reindex(columns=Xtr2.columns, fill_value=0)
    g = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                   max_depth=3, subsample=0.8,
                                   random_state=0).fit(Xtr2, ytr)
    r = RandomForestClassifier(n_estimators=300, random_state=0,
                               n_jobs=1).fit(Xtr2, ytr)
    print(f"  HistGradientBoosting: {roc_auc_score(yte, ph):.4f}")
    print(f"  GradientBoosting:     "
          f"{roc_auc_score(yte, g.predict_proba(Xte2)[:, 1]):.4f}")
    print(f"  RandomForest:         "
          f"{roc_auc_score(yte, r.predict_proba(Xte2)[:, 1]):.4f}")

    print("\n=== 4. Model hajmi va qadamlar ===")
    print(f"  Hist: {h.n_iter_} qadam, "
          f"max_leaf_nodes={h.max_leaf_nodes}")
    tugunlar = sum(e.tree_.node_count for e in r.estimators_)
    print(f"  RF:   {len(r.estimators_)} daraxt, jami {tugunlar:,} tugun")
    gb_tugun = sum(t[0].tree_.node_count for t in g.estimators_)
    print(f"  GB:   {len(g.estimators_)} daraxt, jami {gb_tugun:,} tugun")
    print("  ⭐ Hist* - katta ma'lumot uchun standart tanlov")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ma'lumot ===
  20000 qator, kechikish 26.85%
  yo'qolgan qiymatlar: 594
  kategoriyali: hudud, tur

=== 2. HistGradientBoosting (NaN va kategoriya avtomatik) ===
  erta to'xtagan qadam: 43 (max 400)
  test ROC AUC: 0.7388

=== 3. Boshqa modellar bilan solishtirish ===
  HistGradientBoosting: 0.7388
  GradientBoosting:     0.7393
  RandomForest:         0.6970

=== 4. Model hajmi va qadamlar ===
  Hist: 43 qadam, max_leaf_nodes=31
  RF:   300 daraxt, jami 1,778,356 tugun
  GB:   200 daraxt, jami 2,974 tugun
  ⭐ Hist* - katta ma'lumot uchun standart tanlov

Nima ko'rsatdi: 2.6-bo'lim.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Boosting har doim qoldiqqa moslanadi" Manfiy gradientga
"n_estimators ko'p bo'lsa yaxshi" Overfitting beradi
"squared_error har doim mos" Chetlanishda huber
"max_depth katta bo'lsin" 3-6
"subsample sifatni buzadi" Ko'pincha yaxshilaydi
"GradientBoosting va Hist* bir xil" Hist* 10-100x tez
"GB masshtablash talab qiladi" Yo'q
"GB ehtimolliklari kalibrlangan" Odatda yaxshi, lekin tekshiring

6. Keng tarqalgan xatolar va yechimlari

1. n_estimators ni sozlamaslik

python
GradientBoostingClassifier(n_estimators=2000)                     # ⚠️
# staged_* yoki early_stopping bilan toping 15.9-bob                # ✅

2. learning_rate=1.0

python
GradientBoostingRegressor(learning_rate=1.0)                      # ⚠️
GradientBoostingRegressor(learning_rate=0.05, n_estimators=500)   # ✅

3. Chuqur daraxtlar

python
GradientBoostingClassifier(max_depth=15)                          # ⚠️
GradientBoostingClassifier(max_depth=3)                           # ✅

4. Chetlanishda squared_error

python
GradientBoostingRegressor()          # chetlangan qiymatlar bor   # ⚠️
GradientBoostingRegressor(loss="huber", alpha=0.9)                # ✅

5. Katta ma'lumotda eski implementatsiya

python
GradientBoostingClassifier(n_estimators=500)   # 500k qator       # ⚠️
HistGradientBoostingClassifier(max_iter=500)                      # ✅

6. subsample ni unutish

python
GradientBoostingClassifier(n_estimators=500)                      # ⚠️
GradientBoostingClassifier(n_estimators=500, subsample=0.8)       # ✅

7. Testda erta to'xtatish

python
# staged_score(Xte, yte) bo'yicha qadam tanlash                   # ⚠️
# validation_fraction yoki alohida validatsiya to'plami           # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 15.7-dars (o'tilgan): Boosting g'oyasi
  • 13.6-dars (o'tilgan): Gradient tushish
  • 15.9-dars: Sozlash va erta to'xtatish
  • 15.10-dars: XGBoost va LightGBM
  • 13.11-dars (o'tilgan): Kvantil regressiya

8. Eng yaxshi amaliyotlar

  1. learning_rate ni kichik qo'ying.

  2. n_estimators ni erta to'xtatish bilan toping.

  3. max_depth 3-6.

  4. subsample 0.8 qo'ying.

  5. Yo'qotish funksiyasini vazifaga moslang.

  6. Katta ma'lumotda Hist ishlating.*

  7. Validatsiyani ajrating.

  8. RF bilan taqqoslang.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # gradient boosting nimaga moslanadi?
2.  # squared_error da gradient nima?
3.  # F_0 nima?
4.  # yangilash formulasi?
5.  # loss variantlari (regressiya)?
6.  # huber nima uchun?
7.  # quantile nima beradi?
8.  # max_depth odatiy?
9.  # subsample odatiy?
10. # n_estimators overfitting beradimi?
11. # eta va n_estimators bog'liqmi?
12. # Hist* qachon?
Javoblar
  1. Manfiy gradientga
  2. Qoldiq (y - F)
  3. O'rtacha (yoki log-odds)
  4. F_m = F_{m-1} + eta·h_m
  5. squared/absolute/huber/quantile
  6. Chetlangan qiymatlar
  7. Bashorat oralig'i
  8. 3-6
  9. 0.5-0.8
  10. Ha
  11. Ha, teskari
  12. Katta ma'lumotda

Vazifa 2: Xatolarni tuzating

python
1.  GradientBoostingClassifier(n_estimators=3000)

2.  GradientBoostingRegressor(learning_rate=1.0)

3.  GradientBoostingClassifier(max_depth=20)

4.  GradientBoostingRegressor()   # chetlangan qiymatlar bor

5.  GradientBoostingClassifier()  # 1 mln qator
Javoblar
python
1.  # early_stopping / staged_* bilan qadamni toping

2.  GradientBoostingRegressor(learning_rate=0.05, n_estimators=500)

3.  GradientBoostingClassifier(max_depth=3)

4.  GradientBoostingRegressor(loss="huber", alpha=0.9)

5.  HistGradientBoostingClassifier(max_iter=500)

Vazifa 3: Qo'lda boosting

Modellang:

  1. F_0
  2. 100 qadam
  3. sklearn
  4. eta

Vazifa 4: Yo'qotish

Modellang:

  1. Ma'lumot
  2. To'rt loss
  3. Chetlanish ulushi
  4. Kvantil

Vazifa 5: Giperparametrlar

Modellang:

  1. max_depth
  2. subsample
  3. n_estimators
  4. eta bog'liqligi

Vazifa 6: Hist

Modellang:

  1. Ma'lumot
  2. Hist
  3. Solishtirish
  4. Hajm

Vazifa 7: O'ylash

Gradient boosting da har daraxt oldingilarning xatosini tuzatadi. Nega u holda bitta katta, chuqur daraxt qurib qo'ya qolmaymiz — u ham barcha xatolarni tuzatadi-ku?

Javob

Qisqa javob: chuqur daraxt xatolarni yodlash orqali tuzatadi, boosting esa ularni bosqichma-bosqich va regulyarizatsiya bilan tuzatadi. Farq — qanday tuzatilishida, tuzatilish faktida emas.

1. Ikki yo'lning farqi

Jihat Bitta chuqur daraxt Boosting
Har bo'linish Butun ma'lumotni ko'radi Qoldiqqa qaraydi
Qadam kattaligi Cheklanmagan eta bilan kichraytirilgan
Regulyarizatsiya Faqat pruning eta, subsample, chuqurlik, n_est
Xatoni tuzatish Yodlash Umumiy naqshni topish

2. eta ning roli

Boosting har qadamda xatoning butunini emas, eta qismini tuzatadi. Bu:

  1. Bitta daraxtning tasodifiy qarorini yumshatadi
  2. Keyingi daraxtlarga tuzatish imkonini qoldiradi
  3. Ko'p daraxt bir naqshni "tasdiqlashi" kerak bo'ladi

Shovqin bunday tasdiqni ololmaydi — faqat haqiqiy naqsh oladi.

3. Qo'shimcha model ustunligi

  • Chuqur daraxt: bitta bo'lakli doimiy funksiya, qo'pol chegaralar
  • Boosting: yuzlab sayoz daraxt yig'indisi -> silliqroq funksiya
  • Natijada bir xil murakkablikda umumlashtirish yaxshiroq

4. Empirik tasdiq

Misol 3 da max_depth=8 bilan boosting max_depth=3 dan yomonroq natija berdi — chuqurroq daraxt bu yerda ham zarar keltiradi.

5. Xulosa

  1. Muhimi — xatoni tuzatish emas, uni QANDAY tuzatish
  2. eta kichik qadamlar orqali regulyarizatsiya qiladi
  3. Ko'p sayoz daraxt bitta chuqur daraxtdan silliqroq
  4. Shuning uchun boosting da max_depth kichik

Nimani mustahkamlaydi: 2.1, 2.4-bo'limlar.

Xulosa

Bu darsda gradient boosting ni o'rgandik.

Eng muhim uch fikr:

  1. Har daraxt qoldiqqa (gradientga) moslanadi. F_m = F_{m-1} + eta * h_m, bunda h_m yo'qotish funksiyasining manfiy gradientiga moslanadi. squared_error da bu aynan qoldiq, log_loss da — y - p. Bu funksional fazoda gradient tushish 13.6-bob va u AdaBoost ni istalgan yo'qotish funksiyasiga umumlashtiradi.

  2. Yo'qotish funksiyasini almashtirish mumkin. Chetlangan qiymatlarda huber yoki absolute_error, bashorat oralig'i kerak bo'lsa quantile 13.11-bob, klassifikatsiyada log_loss. Bu — gradient boosting ning AdaBoost dan asosiy amaliy afzalligi.

  3. Odatiy retsept: kichik qadam, sayoz daraxt. learning_rate 0.05-0.1, max_depth 3-6, subsample 0.5-0.8, n_estimators esa erta to'xtatish bilan tanlanadi — u bagging dagidek bepul emas va overfitting beradi. Katta ma'lumotda HistGradientBoosting* (10-100x tez, NaN va kategoriyalarni o'zi boshqaradi).

Keyingi darsda boosting ni sozlash va erta to'xtatishni o'rganamiz: qaysi parametrni qaysi tartibda va qanday tanlash kerak.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
15.8-dars: Gradient boosting — IlmHamroh