IlmHamroh
Data Science va sun'iy intellekt/Daraxtlar va ansambllar9/14-dars18 daqiqa
Mundarija (22)

15.9-dars: Boosting ni sozlash va erta to'xtatish

15-QISM — DARAXTLAR VA ANSAMBLLAR · 9-dars

1. Kirish va motivatsiya

Random Forest ni sozlash uchun bitta parametr (max_features) yetarli edi. Boosting da esa kamida beshtasi bir-biriga bog'liq: learning_rate, n_estimators, max_depth, subsample va regulyarizatsiya. To'liq grid qurish qimmat, tartibsiz qidiruv esa vaqtni behuda sarflaydi.

Yechim — tartib: qaysi parametrni qachon va nimaga qarab tanlashni bilish. Va eng muhimi — erta to'xtatish (early stopping): n_estimators ni grid ga qo'shmasdan, validatsiya to'plamida avtomatik topish.

Bu darsda: erta to'xtatish mexanizmi, validation_fraction va n_iter_no_change, sozlash tartibi, learning_rate va n_estimators almashinuvi, RandomizedSearchCV, o'rganish egri chiziqlari va leakage xavfi.

Real vaziyat. Jamoa gradient boosting ni sozlash uchun 5 parametrli to'liq grid qurdi — 1 440 konfiguratsiya, har biri 5-karra CV bilan: 7 200 model, ikki kun hisoblash. Erta to'xtatish + tasodifiy qidiruv (60 konfiguratsiya) bir soatda bir xil natijaga keldi.

Bu darsda boosting ni sozlashni o'rganamiz.

Bu darsda:

  • Erta to'xtatish
  • Sozlash tartibi
  • learning_rate almashinuvi
  • RandomizedSearchCV
  • O'rganish egri chiziqlari
  • Leakage xavfi
  • Tuzoqlar
  • Amaliy: to'liq oqim

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Erta to'xtatish

text
G'oya: validatsiya xatosi yaxshilanmay qolsa, qo'shishni to'xtatish

sklearn (Hist*):
  early_stopping=True
  validation_fraction=0.1        # o'quvdan ajratiladi
  n_iter_no_change=10            # sabr (patience)
  tol=1e-7
  -> h.n_iter_ tanlangan qadam

sklearn (eski GradientBoosting*):
  n_iter_no_change=10, validation_fraction=0.1

Qo'lda: staged_predict bilan eng yaxshi qadamni topish

Erta to'xtatish n_estimators ni gridga qo'shish zaruriyatini yo'q qiladi: n_estimators ni katta qo'yasiz (masalan 2000) va algoritm o'zi to'xtaydi. Bu qidiruvni 5-10 barobar arzonlashtiradi.

2.2. Sozlash tartibi

text
1. learning_rate = 0.1 (yoki 0.05) - QOTIB QOLADI
2. n_estimators = katta + erta to'xtatish
3. max_depth / max_leaf_nodes - eng ta'sirli tuzilma parametri
4. min_samples_leaf / min_child_weight - shovqinli ma'lumotda
5. subsample, colsample - regulyarizatsiya
6. L1/L2 (reg_alpha, reg_lambda) - oxirgi siqib olish
7. YAKUNIDA: learning_rate ni 2-4x kamaytirib, qayta o'qitish

Bu tartib 60-100 konfiguratsiya bilan deyarli optimal natija beradi

Oxirgi qadam muhim: eng yaxshi konfiguratsiya topilgach, learning_rate ni kamaytirib (masalan 0.1 dan 0.03 ga) qayta o'qitish odatda bepul 0.5-1% beradi — faqat ko'proq daraxt kerak bo'ladi.

2.3. learning_rate almashinuvi

text
eta * n_estimators ~ o'zgarmas (taxminan)

  eta = 0.3  -> ~100 daraxt
  eta = 0.1  -> ~300 daraxt
  eta = 0.03 -> ~1000 daraxt

Kichik eta:
  + yaxshiroq umumlashtirish (odatda)
  + optimum atrofida tekis -> qadam tanlash xatosi kam zarar
  - sekinroq

Amalda: sozlashda 0.1, yakuniy modelda 0.03-0.05

Kichik eta da test egri chizig'i optimum atrofida tekis bo'ladi — ya'ni qadamni biroz noto'g'ri tanlash deyarli zarar qilmaydi. Katta eta da esa egri chiziq o'tkir: bir nechta ortiqcha daraxt natijani sezilarli buzadi.

2.4. RandomizedSearchCV

python
from scipy.stats import loguniform, randint, uniform
from sklearn.model_selection import RandomizedSearchCV

taqsimot = {
    "max_depth": randint(2, 9),
    "min_samples_leaf": randint(1, 60),
    "subsample": uniform(0.5, 0.5),        # 0.5 .. 1.0
    "max_features": uniform(0.3, 0.7),
}
q = RandomizedSearchCV(model, taqsimot, n_iter=60, cv=5,
                       scoring="roc_auc", random_state=0)

Tasodifiy qidiruv gridd dan samaraliroq (Bergstra & Bengio, 2012): agar 5 parametrdan faqat 2 tasi muhim bo'lsa, grid muhim parametrlar uchun atigi bir nechta qiymatni sinaydi, tasodifiy qidiruv esa har birini 60 xil qiymatda ko'radi.

2.5. O'rganish egri chiziqlari

text
Ikki xil egri chiziq - ikki xil savol:

1. QADAMLAR bo'yicha (staged_*): n_estimators ni tanlash
   o'quv pasayadi, validatsiya pasayadi -> ko'tariladi

2. MA'LUMOT HAJMI bo'yicha (learning_curve): ko'proq ma'lumot foydalimi?
   ikkalasi yaqinlashsa -> ko'proq ma'lumot foyda bermaydi
   katta farq qolsa -> ma'lumot yig'ish foydali

Ikkinchisi loyiha rejalashtirishda muhimroq

learning_curve ma'lumot yig'ishga sarmoya kiritish kerakmi degan savolga javob beradi: egri chiziqlar yaqinlashgan bo'lsa, ko'proq ma'lumot foyda bermaydi — model biasga taqalgan.

2.6. Leakage xavfi

text
XAVFLI: erta to'xtatishni TEST to'plamida qilish
  -> test to'plami bilvosita o'qitishga aralashadi 12.9-bob

TO'G'RI:
  o'quv -> (ichki o'quv + validatsiya) + alohida test
  yoki: CV ichida erta to'xtatish (har foldda alohida validatsiya)

sklearn: validation_fraction o'QUV dan ajratadi - xavfsiz
XGBoost/LightGBM: eval_set ni O'ZINGIZ berasiz - ehtiyot bo'ling

eval_set ga test to'plamini berish — boosting bilan ishlashdagi eng keng tarqalgan leakage xatosi. Natijada test bahosi optimistik bo'ladi va ishlab chiqarishda model kutilganidan yomon ishlaydi.

2.7. Tuzoqlar

Asosiy tuzoqlar: n_estimators ni gridga qo'shish; test to'plamida erta to'xtatish; learning_rate ni oxirida kamaytirmaslik; to'liq grid qurish; n_iter_no_change ni juda kichik qo'yish (erta to'xtab qoladi); bitta bo'linishda sozlab, CV siz xulosa qilish; scoring ni vazifaga moslamaslik; sozlashdan keyin butun o'quv to'plamida qayta o'qitishni unutish.

2.8. Tartib bilan sozlash

Boosting ni sozlashda n_estimators ni gridga qo'shmang — uni erta to'xtatish bilan toping (validation_fraction, n_iter_no_change). learning_rate ni 0.1 da qotiring, so'ng tuzilma (max_depth), keyin regulyarizatsiya (min_samples_leaf, subsample) ni sozlang, yakunida learning_rate ni kamaytirib qayta o'qiting. Tasodifiy qidiruv to'liq griddan samaraliroq. Erta to'xtatishni hech qachon test to'plamida qilmang. Keyingi dars — XGBoost va LightGBM.

3. Tez ma'lumotnoma

python
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import RandomizedSearchCV, learning_curve
from scipy.stats import randint, uniform

h = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=2000,
                                   early_stopping=True, validation_fraction=0.15,
                                   n_iter_no_change=20, random_state=0).fit(X, y)
h.n_iter_, h.validation_score_

q = RandomizedSearchCV(h, {"max_depth": randint(2, 9),
                           "min_samples_leaf": randint(5, 80),
                           "l2_regularization": uniform(0, 5)},
                       n_iter=60, cv=5, scoring="roc_auc", random_state=0)
QOIDA: n_estimators ni gridga qo'shma · test da to'xtatma ·
       oxirida eta ni kamaytir

Sozlash xulosasi

1. eta = 0.1 qotiriladi
2. n_estimators - erta to'xtatish bilan
3. max_depth / max_leaf_nodes
4. min_samples_leaf, subsample
5. L2
6. eta ni 2-4x kamaytirib qayta o'qitish
Tasodifiy qidiruv > to'liq grid; test da to'xtatish = leakage

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14).

Misol 1 — Erta to'xtatish

python
"""Qadamni avtomatik tanlash (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split


def yarat(seed: int = 6, n: int = 8000, p: int = 14, shovqin: float = 0.10):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    qoida = (((X[:, 0] > 0.2) & (X[:, 1] < 0.3))
             | ((X[:, 2] > 0.4) & (X[:, 3] > -0.1))
             | (X[:, 4] < -1.0))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)

    print("=== 1. Erta to'xtatishsiz va bilan ===")
    h1 = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=1000,
                                        early_stopping=False,
                                        random_state=0).fit(Xtr, ytr)
    h2 = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=1000,
                                        early_stopping=True,
                                        validation_fraction=0.15,
                                        n_iter_no_change=20,
                                        random_state=0).fit(Xtr, ytr)
    print(f"  to'xtatishsiz: {h1.n_iter_:>4} qadam, test AUC "
          f"{roc_auc_score(yte, h1.predict_proba(Xte)[:, 1]):.4f}")
    print(f"  to'xtatish b.: {h2.n_iter_:>4} qadam, test AUC "
          f"{roc_auc_score(yte, h2.predict_proba(Xte)[:, 1]):.4f}")

    print("\n=== 2. n_iter_no_change (sabr) ===")
    print(f"  {'sabr':>6} {'qadam':>7} {'test AUC':>10}")
    for sabr in [3, 10, 20, 50]:
        h = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=1000,
                                           early_stopping=True,
                                           validation_fraction=0.15,
                                           n_iter_no_change=sabr,
                                           random_state=0).fit(Xtr, ytr)
        print(f"  {sabr:>6} {h.n_iter_:>7} "
              f"{roc_auc_score(yte, h.predict_proba(Xte)[:, 1]):>10.4f}")

    print("\n=== 3. validation_fraction ===")
    print(f"  {'ulush':>7} {'qadam':>7} {'test AUC':>10}")
    for vf in [0.05, 0.1, 0.2, 0.3]:
        h = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=1000,
                                           early_stopping=True,
                                           validation_fraction=vf,
                                           n_iter_no_change=20,
                                           random_state=0).fit(Xtr, ytr)
        print(f"  {vf:>7.2f} {h.n_iter_:>7} "
              f"{roc_auc_score(yte, h.predict_proba(Xte)[:, 1]):>10.4f}")

    print("\n=== 4. learning_rate va kerakli qadamlar ===")
    print(f"  {'eta':>6} {'qadam':>7} {'eta*qadam':>11} {'test AUC':>10}")
    for eta in [0.3, 0.1, 0.05, 0.02]:
        h = HistGradientBoostingClassifier(learning_rate=eta, max_iter=3000,
                                           early_stopping=True,
                                           validation_fraction=0.15,
                                           n_iter_no_change=25,
                                           random_state=0).fit(Xtr, ytr)
        print(f"  {eta:>6.2f} {h.n_iter_:>7} {eta * h.n_iter_:>11.1f} "
              f"{roc_auc_score(yte, h.predict_proba(Xte)[:, 1]):>10.4f}")
    print("  ⭐ eta kichrayganda qadamlar soni oshadi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Erta to'xtatishsiz va bilan ===
  to'xtatishsiz: 1000 qadam, test AUC 0.8840
  to'xtatish b.:   50 qadam, test AUC 0.8851

=== 2. n_iter_no_change (sabr) ===
    sabr   qadam   test AUC
       3      33     0.8880
      10      40     0.8877
      20      50     0.8851
      50      80     0.8833

=== 3. validation_fraction ===
    ulush   qadam   test AUC
     0.05      45     0.8861
     0.10      52     0.8837
     0.20      48     0.8872
     0.30      48     0.8839

=== 4. learning_rate va kerakli qadamlar ===
     eta   qadam   eta*qadam   test AUC
    0.30      34        10.2     0.8825
    0.10      55         5.5     0.8855
    0.05      86         4.3     0.8863
    0.02     175         3.5     0.8855
  ⭐ eta kichrayganda qadamlar soni oshadi

Nima ko'rsatdi: 2.1, 2.3-bo'limlar.

Misol 2 — Sozlash tartibi

python
"""Bosqichma-bosqich sozlash (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split


def yarat(seed: int = 12, n: int = 7000, p: int = 16, shovqin: float = 0.12):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    qoida = (((X[:, 0] > 0.3) & (X[:, 1] < 0.2))
             | ((X[:, 2] > 0.4) & (X[:, 3] > 0.0) & (X[:, 5] < 0.5))
             | (X[:, 4] < -1.1))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    cv = StratifiedKFold(4, shuffle=True, random_state=0)

    def baho(**p):
        m = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=1000,
                                           early_stopping=True,
                                           validation_fraction=0.15,
                                           n_iter_no_change=20,
                                           random_state=0, **p)
        return cross_val_score(m, Xtr, ytr, cv=cv, scoring="roc_auc").mean()

    print("=== 1. Bazaviy (barcha standart) ===")
    asos = baho()
    print(f"  CV ROC AUC: {asos:.4f}")

    print("\n=== 2. max_depth ===")
    eng_chuqurlik, eng_baho = None, asos
    for chuqurlik in [2, 3, 4, 6, None]:
        b = baho(max_depth=chuqurlik)
        belgi = ""
        if b > eng_baho:
            eng_baho, eng_chuqurlik, belgi = b, chuqurlik, "  <-"
        nom = "None" if chuqurlik is None else str(chuqurlik)
        print(f"  max_depth={nom:<5}: {b:.4f}{belgi}")

    print("\n=== 3. min_samples_leaf ===")
    eng_msl = 20
    for msl in [5, 20, 50, 120]:
        b = baho(max_depth=eng_chuqurlik, min_samples_leaf=msl)
        belgi = ""
        if b > eng_baho:
            eng_baho, eng_msl, belgi = b, msl, "  <-"
        print(f"  min_samples_leaf={msl:<4}: {b:.4f}{belgi}")

    print("\n=== 4. L2 va yakuniy eta ===")
    eng_l2 = 0.0
    for l2 in [0.0, 0.5, 2.0, 8.0]:
        b = baho(max_depth=eng_chuqurlik, min_samples_leaf=eng_msl,
                 l2_regularization=l2)
        belgi = ""
        if b > eng_baho:
            eng_baho, eng_l2, belgi = b, l2, "  <-"
        print(f"  l2={l2:<5.1f}: {b:.4f}{belgi}")
    print(f"  tanlangan: max_depth={eng_chuqurlik}, "
          f"min_samples_leaf={eng_msl}, l2={eng_l2}")

    print(f"  {'eta':>6} {'qadam':>7} {'test AUC':>10}")
    for eta in [0.1, 0.05, 0.03]:
        m = HistGradientBoostingClassifier(learning_rate=eta, max_iter=3000,
                                           early_stopping=True,
                                           validation_fraction=0.15,
                                           n_iter_no_change=25,
                                           max_depth=eng_chuqurlik,
                                           min_samples_leaf=eng_msl,
                                           l2_regularization=eng_l2,
                                           random_state=0).fit(Xtr, ytr)
        print(f"  {eta:>6.2f} {m.n_iter_:>7} "
              f"{roc_auc_score(yte, m.predict_proba(Xte)[:, 1]):>10.4f}")
    print("  ⭐ Oxirida eta ni kamaytirish deyarli bepul foyda")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Bazaviy (barcha standart) ===
  CV ROC AUC: 0.8711

=== 2. max_depth ===
  max_depth=2    : 0.8706
  max_depth=3    : 0.8734  <-
  max_depth=4    : 0.8731
  max_depth=6    : 0.8717
  max_depth=None : 0.8711

=== 3. min_samples_leaf ===
  min_samples_leaf=5   : 0.8726
  min_samples_leaf=20  : 0.8734
  min_samples_leaf=50  : 0.8741  <-
  min_samples_leaf=120 : 0.8740

=== 4. L2 va yakuniy eta ===
  l2=0.0  : 0.8741
  l2=0.5  : 0.8741  <-
  l2=2.0  : 0.8736
  l2=8.0  : 0.8740
  tanlangan: max_depth=3, min_samples_leaf=50, l2=0.5
     eta   qadam   test AUC
    0.10     115     0.8775
    0.05     179     0.8800
    0.03     281     0.8802
  ⭐ Oxirida eta ni kamaytirish deyarli bepul foyda

Nima ko'rsatdi: 2.2-bo'lim.

Misol 3 — Grid va tasodifiy qidiruv

python
"""Bir xil byudjetda qaysi biri yaxshiroq (real numpy/sklearn)."""

import numpy as np
from scipy.stats import randint, uniform
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import (GridSearchCV, RandomizedSearchCV,
                                     StratifiedKFold, train_test_split)


def yarat(seed: int = 21, n: int = 6000, p: int = 14, shovqin: float = 0.12):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    qoida = (((X[:, 0] > 0.25) & (X[:, 1] < 0.25))
             | ((X[:, 2] > 0.5) & (X[:, 3] > 0.0))
             | (X[:, 4] < -1.0))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    cv = StratifiedKFold(3, shuffle=True, random_state=0)
    asos = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=800,
                                          early_stopping=True,
                                          validation_fraction=0.15,
                                          n_iter_no_change=20, random_state=0)

    print("=== 1. To'liq grid (24 konfiguratsiya) ===")
    setka = {"max_depth": [3, 4, 6], "min_samples_leaf": [10, 40],
             "l2_regularization": [0.0, 1.0, 5.0], "max_bins": [255]}
    g = GridSearchCV(asos, setka, cv=cv, scoring="roc_auc",
                     n_jobs=1).fit(Xtr, ytr)
    print(f"  konfiguratsiyalar: {len(g.cv_results_['params'])}")
    print(f"  eng yaxshi CV: {g.best_score_:.4f}")
    print(f"  test AUC: "
          f"{roc_auc_score(yte, g.predict_proba(Xte)[:, 1]):.4f}")

    print("\n=== 2. Tasodifiy qidiruv (24 konfiguratsiya) ===")
    taqsimot = {"max_depth": randint(2, 9),
                "min_samples_leaf": randint(5, 100),
                "l2_regularization": uniform(0, 8),
                "max_features": uniform(0.4, 0.6)}
    r = RandomizedSearchCV(asos, taqsimot, n_iter=24, cv=cv, scoring="roc_auc",
                           random_state=0, n_jobs=1).fit(Xtr, ytr)
    print(f"  konfiguratsiyalar: {len(r.cv_results_['params'])}")
    print(f"  eng yaxshi CV: {r.best_score_:.4f}")
    print(f"  test AUC: "
          f"{roc_auc_score(yte, r.predict_proba(Xte)[:, 1]):.4f}")

    print("\n=== 3. Topilgan parametrlar ===")
    print(f"  grid:     {g.best_params_}")
    qisqa = {k: (round(v, 3) if isinstance(v, float) else v)
             for k, v in r.best_params_.items()}
    print(f"  tasodifiy: {qisqa}")

    print("\n=== 4. Har parametrning ta'siri (tasodifiy qidiruvdan) ===")
    natija = r.cv_results_["mean_test_score"]
    for nom in ["max_depth", "min_samples_leaf", "l2_regularization",
                "max_features"]:
        qiymatlar = np.array([p[nom] for p in r.cv_results_["params"]],
                             dtype=float)
        korr = float(np.corrcoef(qiymatlar, natija)[0, 1])
        print(f"  {nom:<20}: CV bilan korrelyatsiya {korr:+.3f}, "
              f"diapazon {qiymatlar.min():.2f}..{qiymatlar.max():.2f}")
    print("  ⭐ Tasodifiy qidiruv har parametrni ko'p qiymatda ko'radi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. To'liq grid (24 konfiguratsiya) ===
  konfiguratsiyalar: 18
  eng yaxshi CV: 0.8811
  test AUC: 0.8772

=== 2. Tasodifiy qidiruv (24 konfiguratsiya) ===
  konfiguratsiyalar: 24
  eng yaxshi CV: 0.8820
  test AUC: 0.8746

=== 3. Topilgan parametrlar ===
  grid:     {'l2_regularization': 5.0, 'max_bins': 255, 'max_depth': 6, 'min_samples_leaf': 40}
  tasodifiy: {'l2_regularization': np.float64(3.534), 'max_depth': 5, 'max_features': np.float64(0.472), 'min_samples_leaf': 28}

=== 4. Har parametrning ta'siri (tasodifiy qidiruvdan) ===
  max_depth           : CV bilan korrelyatsiya -0.058, diapazon 2.00..8.00
  min_samples_leaf    : CV bilan korrelyatsiya -0.089, diapazon 5.00..85.00
  l2_regularization   : CV bilan korrelyatsiya +0.128, diapazon 0.11..7.83
  max_features        : CV bilan korrelyatsiya -0.151, diapazon 0.45..0.98
  ⭐ Tasodifiy qidiruv har parametrni ko'p qiymatda ko'radi

Nima ko'rsatdi: 2.4-bo'lim.

Misol 4 — Leakage va o'rganish egri chizig'i

python
"""Testda to'xtatish nima qiladi va ma'lumot yig'ish foydalimi (real sklearn)."""

import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold, learning_curve, train_test_split


def yarat(seed: int = 31, n: int = 6000, p: int = 12, shovqin: float = 0.13):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    qoida = (((X[:, 0] > 0.2) & (X[:, 1] < 0.3))
             | ((X[:, 2] > 0.4) & (X[:, 3] > 0.0))
             | (X[:, 4] < -1.0))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xqol, ytr, yqol = train_test_split(X, y, test_size=0.4,
                                            random_state=0, stratify=y)
    Xval, Xte, yval, yte = train_test_split(Xqol, yqol, test_size=0.5,
                                            random_state=0, stratify=yqol)
    print("=== 1. Uch to'plam ===")
    print(f"  o'quv {len(Xtr)}, validatsiya {len(Xval)}, test {len(Xte)}")

    g = GradientBoostingClassifier(n_estimators=500, learning_rate=0.1,
                                   max_depth=3, subsample=0.8,
                                   random_state=0).fit(Xtr, ytr)
    val_auc = [roc_auc_score(yval, p[:, 1])
               for p in g.staged_predict_proba(Xval)]
    test_auc = [roc_auc_score(yte, p[:, 1])
                for p in g.staged_predict_proba(Xte)]

    print("\n=== 2. Validatsiya va test bo'yicha to'xtatish ===")
    v_eng = int(np.argmax(val_auc))
    t_eng = int(np.argmax(test_auc))
    print(f"  validatsiya bo'yicha eng yaxshi qadam: {v_eng + 1}")
    print(f"  shu qadamda test AUC: {test_auc[v_eng]:.4f}   <- HALOL baho")
    print(f"  test bo'yicha eng yaxshi qadam: {t_eng + 1}")
    print(f"  shu qadamda test AUC: {test_auc[t_eng]:.4f}   <- optimistik")
    print(f"  optimizm: {test_auc[t_eng] - test_auc[v_eng]:+.4f}")

    print("\n=== 3. Optimizm bir nechta ajratishda ===")
    farqlar = []
    for s in range(5):
        Xa, Xb, ya, yb = train_test_split(X, y, test_size=0.4, random_state=s,
                                          stratify=y)
        Xv, Xt, yv, yt = train_test_split(Xb, yb, test_size=0.5, random_state=s,
                                          stratify=yb)
        m = GradientBoostingClassifier(n_estimators=300, learning_rate=0.1,
                                       max_depth=3, subsample=0.8,
                                       random_state=0).fit(Xa, ya)
        va = [roc_auc_score(yv, p[:, 1]) for p in m.staged_predict_proba(Xv)]
        ta = [roc_auc_score(yt, p[:, 1]) for p in m.staged_predict_proba(Xt)]
        farqlar.append(max(ta) - ta[int(np.argmax(va))])
    print(f"  o'rtacha optimizm: {np.mean(farqlar):+.4f}")
    print(f"  eng katta: {max(farqlar):+.4f}")

    print("\n=== 4. Ma'lumot hajmi egri chizig'i ===")
    cv = StratifiedKFold(3, shuffle=True, random_state=0)
    olcham, oquv_b, cv_b = learning_curve(
        GradientBoostingClassifier(n_estimators=200, learning_rate=0.1,
                                   max_depth=3, subsample=0.8, random_state=0),
        Xtr, ytr, train_sizes=[0.1, 0.25, 0.5, 0.75, 1.0], cv=cv,
        scoring="roc_auc", n_jobs=1)
    print(f"  {'namuna':>8} {'o_quv AUC':>11} {'CV AUC':>9} {'farq':>8}")
    for i, o in enumerate(olcham):
        print(f"  {int(o):>8} {oquv_b[i].mean():>11.4f} {cv_b[i].mean():>9.4f} "
              f"{oquv_b[i].mean() - cv_b[i].mean():>8.4f}")
    print("  ⭐ Farq kichraysa - ko'proq ma'lumot foyda bermaydi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Uch to'plam ===
  o'quv 3600, validatsiya 1200, test 1200

=== 2. Validatsiya va test bo'yicha to'xtatish ===
  validatsiya bo'yicha eng yaxshi qadam: 79
  shu qadamda test AUC: 0.8703   <- HALOL baho
  test bo'yicha eng yaxshi qadam: 200
  shu qadamda test AUC: 0.8782   <- optimistik
  optimizm: +0.0079

=== 3. Optimizm bir nechta ajratishda ===
  o'rtacha optimizm: +0.0051
  eng katta: +0.0079

=== 4. Ma'lumot hajmi egri chizig'i ===
    namuna   o_quv AUC    CV AUC     farq
       240      1.0000    0.8576   0.1424
       600      1.0000    0.8788   0.1212
      1200      0.9974    0.8885   0.1089
      1800      0.9913    0.8884   0.1028
      2400      0.9831    0.8902   0.0929
  ⭐ Farq kichraysa - ko'proq ma'lumot foyda bermaydi

Nima ko'rsatdi: 2.5, 2.6-bo'limlar.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"n_estimators ni gridga qo'shish kerak" Erta to'xtatish bilan
"Test da to'xtatish zararsiz" Leakage
"Grid tasodifiydan yaxshiroq" Odatda aksincha
"eta ni oxirida o'zgartirmaslik kerak" Kamaytirish foydali
"Sabr kichik bo'lsin" Erta to'xtab qoladi
"Barcha parametr teng muhim" max_depth eng ta'sirli
"learning_curve keraksiz" Ma'lumot rejasini beradi
"CV siz bir bo'linish yetarli" Tasodifga bog'liq

6. Keng tarqalgan xatolar va yechimlari

1. n_estimators gridda

python
GridSearchCV(g, {"n_estimators": [100, 300, 1000], ...})          # ⚠️
HistGradientBoostingClassifier(max_iter=2000, early_stopping=True) # ✅

2. Testda erta to'xtatish

python
model.fit(Xtr, ytr, eval_set=[(Xte, yte)])                        # ⚠️
model.fit(Xtr, ytr, eval_set=[(Xval, yval)])                      # ✅

3. eta ni oxirida kamaytirmaslik

python
# grid natijasini shundayligicha qoldirish                        # ⚠️
# eng yaxshi parametrlar + eta/3 bilan qayta o'qitish             # ✅

4. To'liq grid

python
GridSearchCV(g, {5 parametr x 4 qiymat})   # 1024 konfiguratsiya  # ⚠️
RandomizedSearchCV(g, taqsimot, n_iter=60)                        # ✅

5. Kichik sabr

python
n_iter_no_change=2                                                # ⚠️
n_iter_no_change=20                                               # ✅

6. Noto'g'ri scoring

python
RandomizedSearchCV(..., scoring="accuracy")   # 2% musbat sinf    # ⚠️
RandomizedSearchCV(..., scoring="average_precision")              # ✅

7. Qayta o'qitishni unutish

python
# validation_fraction bilan o'qitilgan modelni shundayligicha ishlatish # ⚠️
# yakuniy modelni butun o'quv to'plamida, topilgan qadam bilan qayta o'qiting # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 15.8-dars (o'tilgan): Gradient boosting
  • 12.9-dars (o'tilgan): Leakage
  • 15.10-dars: XGBoost va LightGBM
  • 15.12-dars: Ansambllarni solishtirish
  • 15.14-dars: Amaliyot

8. Eng yaxshi amaliyotlar

  1. Erta to'xtatishdan foydalaning.

  2. Validatsiyani ajrating.

  3. Tartib bilan sozlang.

  4. Tasodifiy qidiruv ishlating.

  5. Oxirida eta ni kamaytiring.

  6. Sabrni yetarli qo'ying.

  7. learning_curve bilan rejalashtiring.

  8. Yakuniy modelni qayta o'qiting.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # erta to'xtatish nima qiladi?
2.  # validation_fraction qayerdan ajratadi?
3.  # n_iter_no_change nima?
4.  # sozlash tartibida birinchi nima?
5.  # eng ta'sirli tuzilma parametri?
6.  # eta*n_estimators?
7.  # nega tasodifiy qidiruv yaxshiroq?
8.  # testda to'xtatish nima beradi?
9.  # learning_curve qaysi savolga javob?
10. # oxirgi qadam nima?
11. # sabr kichik bo'lsa?
12. # yakuniy model qayerda o'qitiladi?
Javoblar
  1. Qadamni avtomatik tanlaydi
  2. O'quv to'plamidan
  3. Sabr (patience)
  4. learning_rate ni qotirish
  5. max_depth
  6. Taxminan o'zgarmas
  7. Muhim parametrni ko'p qiymatda ko'radi
  8. Optimistik baho (leakage)
  9. Ko'proq ma'lumot foydalimi
  10. eta ni kamaytirib qayta o'qitish
  11. Erta to'xtab qoladi
  12. Butun o'quv to'plamida

Vazifa 2: Xatolarni tuzating

python
1.  GridSearchCV(g, {"n_estimators": [100, 500, 1000]})

2.  model.fit(Xtr, ytr, eval_set=[(Xte, yte)])

3.  GridSearchCV(g, {5 parametr x 4 qiymat})

4.  n_iter_no_change=2

5.  RandomizedSearchCV(..., scoring="accuracy")   # 2% musbat
Javoblar
python
1.  HistGradientBoostingClassifier(max_iter=2000, early_stopping=True)

2.  model.fit(Xtr, ytr, eval_set=[(Xval, yval)])

3.  RandomizedSearchCV(g, taqsimot, n_iter=60)

4.  n_iter_no_change=20

5.  RandomizedSearchCV(..., scoring="average_precision")

Vazifa 3: Erta to'xtatish

Modellang:

  1. Bilan va bilansiz
  2. Sabr
  3. validation_fraction
  4. eta

Vazifa 4: Tartib

Modellang:

  1. Bazaviy
  2. max_depth
  3. min_samples_leaf
  4. L2 va eta

Vazifa 5: Qidiruv

Modellang:

  1. Grid
  2. Tasodifiy
  3. Parametrlar
  4. Ta'sir

Vazifa 6: Leakage

Modellang:

  1. Uch to'plam
  2. Ikki to'xtatish
  3. Optimizm
  4. learning_curve

Vazifa 7: O'ylash

Boosting ni sozlash uchun avtomatik vositalar bor (Optuna, Hyperopt, sklearn ning HalvingRandomSearchCV). Ular qo'lda sozlashni butunlay almashtira oladimi?

Javob

Qisqa javob: ular qidiruvni almashtiradi, qarorlarni emas. Avtomatik vosita qidiruv fazosini, baholash metrikasini, validatsiya sxemasini va to'xtash shartini siz belgilaganingizdan keyin ishlaydi — va aynan shular natijani belgilaydi.

1. Avtomatik vositalar nimani yaxshi bajaradi

Vazifa Avtomatik vosita
Ko'p o'lchovli qidiruv Qo'ldan ancha yaxshi
Istiqbolsiz konfiguratsiyani to'xtatish Halving / pruning
Byudjetni taqsimlash Samarali
Takrorlanuvchanlik Yaxshi (seed bilan)

2. Nima sizning zimmangizda qoladi

  1. Qidiruv fazosi: max_depth 2..9 mi, 2..30 mi? Noto'g'ri diapazon — noto'g'ri natija
  2. Metrika: roc_auc mi, average_precision mi, biznes metrikasi mi?
  3. Validatsiya: tasodifiy CV mi, vaqt bo'yicha mi, guruh bo'yicha mi 12.3-bob?
  4. Leakage: tayyorlash quvur ichidami 12.9-bob?
  5. To'xtash: qancha byudjet arziydi?

3. Yashirin xavf: overfitting to validation

  • 1000 konfiguratsiya sinalsa, eng yaxshisi tasodifan yaxshi bo'lishi mumkin
  • CV bahosi optimistik bo'ladi
  • Shuning uchun alohida test to'plami majburiy
  • Nested CV 12.3-bob aniqroq baho beradi

4. Amaliy yondashuv

  1. Qo'lda 1-2 soat: fazoni tushunish, tartibni qo'llash
  2. Avtomatik qidiruv: topilgan fazo ichida 100-300 sinov
  3. Alohida test to'plamida yakuniy baho
  4. Sodda modelni (RF) ham saqlang — farq kichik bo'lsa, uni tanlang

5. Xulosa

  1. Avtomatik vosita qidiruvni tezlashtiradi
  2. Qidiruv fazosi va metrikani siz belgilaysiz
  3. Validatsiya to'g'riligi — sizning javobgarligingiz
  4. Alohida test to'plamisiz natija ishonchsiz

Nimani mustahkamlaydi: 2.4, 2.6-bo'limlar.

Xulosa

Bu darsda boosting ni sozlashni o'rgandik.

Eng muhim uch fikr:

  1. n_estimators ni gridga qo'shmang. Erta to'xtatish (early_stopping=True, validation_fraction, n_iter_no_change) uni avtomatik topadi va qidiruvni 5-10 barobar arzonlashtiradi. max_iter ni katta qo'ying — algoritm o'zi to'xtaydi. Sabrni yetarli (15-25) qo'ying, aks holda model erta to'xtab qoladi.

  2. Tartib bilan sozlang. learning_rate=0.1 ni qotiring → max_depth (eng ta'sirli) → min_samples_leaf → subsample/max_features → L2 → yakunida learning_rate ni 2-4x kamaytirib qayta o'qiting (deyarli bepul 0.5-1%). Tasodifiy qidiruv to'liq griddan samaraliroq, chunki u har parametrni ko'p qiymatda ko'radi.

  3. Erta to'xtatishni test to'plamida qilmang. eval_set ga test to'plamini berish — boosting dagi eng keng tarqalgan leakage: test bahosi optimistik bo'ladi va ishlab chiqarishdagi natija kutilganidan past chiqadi. O'quvni ichki o'quv + validatsiyaga ajrating, testga hech qachon tegmang.

Keyingi darsda XGBoost va LightGBMni o'rganamiz: sanoat standarti bo'lgan ikki kutubxona va ularning sklearn dan farqlari.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
15.9-dars: Boosting ni sozlash va erta to'xtatish — IlmHamroh