IlmHamroh
Data Science va sun'iy intellekt/Daraxtlar va ansambllar2/14-dars19 daqiqa
Mundarija (22)

15.2-dars: O'sish va to'xtash shartlari

15-QISM — DARAXTLAR VA ANSAMBLLAR · 2-dars

1. Kirish va motivatsiya

Cheklovsiz daraxt har doim o'quv ma'lumotini 100% yodlaydi: u har bir namunani alohida bargga joylashtirguncha bo'linaveradi. Bu — overfitting ning eng toza ko'rinishi 12.4-bob. Shuning uchun daraxt bilan ishlashning asosiy qismi — uni qachon to'xtatishni hal qilish.

To'xtatishning ikki yo'li bor: oldindan (pre-pruning — max_depth, min_samples_leaf) va keyin (post-pruning — daraxtni to'liq o'stirib, keyin keraksiz shoxlarni kesish). Ikkinchisi nazariy jihatdan yaxshiroq, birinchisi amalda ko'proq ishlatiladi.

Bu darsda: max_depth, min_samples_split, min_samples_leaf, max_leaf_nodes, min_impurity_decrease, max_features, cost-complexity pruning (ccp_alpha), o'quv egri chizig'i va giperparametr tanlash.

Real vaziyat. Talabalarning o'qishni tashlash ehtimolini bashorat qiluvchi daraxt o'quvda 100% aniqlik berdi va universitet rahbariyati uni joriy qilishga qaror qildi. Test to'plamida esa aniqlik 0.61 chiqdi — tasodifiydan biroz yaxshiroq. max_depth=5 va min_samples_leaf=30 qo'yilgach, o'quv 0.79 ga tushdi, lekin test 0.77 ga ko'tarildi. O'quv aniqligining tushishi — yaxshilanish edi.

Bu darsda daraxt o'sishini nazorat qilishni o'rganamiz.

Bu darsda:

  • Cheklovsiz daraxt va overfitting
  • Pre-pruning giperparametrlari
  • Cost-complexity pruning
  • O'quv egri chizig'i
  • max_features
  • Giperparametr tanlash
  • Tuzoqlar
  • Amaliy: sozlash

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Cheklovsiz daraxt

text
DecisionTreeClassifier()  — standart: HECH QANDAY chekov yo'q
  max_depth=None, min_samples_split=2, min_samples_leaf=1

Natija: har barg TOZA bo'lguncha bo'linadi
  o'quv aniqligi = 1.0000 (takrorlanmaydigan namunalar bo'lsa)
  barglar soni ~ namunalar soni tartibida
  test aniqligi: shovqinli ma'lumotda keskin past

DecisionTreeClassifier() ni standart parametrlar bilan ishlatish — eng keng tarqalgan xato. O'quv aniqligi 1.0000 chiqishi modelning yaxshiligini emas, cheklov yo'qligini bildiradi.

2.2. Pre-pruning giperparametrlari

text
max_depth              — eng ko'p qo'llaniladigan; 3-10 odatiy
min_samples_split      — tugunni bo'lish uchun kerakli minimal namuna (>= 2)
min_samples_leaf       — bargdagi minimal namuna (eng ISHONCHLI chekov)
max_leaf_nodes         — barglarning umumiy soni (chuqurlikdan moslashuvchan)
min_impurity_decrease  — kamayish shu qiymatdan kichik bo'lsa bo'linmaydi
max_features           — har bo'linishda ko'riladigan belgilar soni
class_weight           — nomutanosib sinflar uchun 14.9-bob

min_samples_leaf: float bersangiz — ULUSH (0.01 = 1%)

min_samples_leaf — eng ishonchli chekov: u har bargda yetarlicha namuna bo'lishini kafolatlaydi, ya'ni barg bashorati statistik ma'noga ega bo'ladi. max_depth esa faqat yo'l uzunligini cheklaydi — chuqurlik kichik bo'lsa ham bitta bargda 2 ta namuna qolishi mumkin.

2.3. Cost-complexity pruning

text
Daraxtni TO'LIQ o'stirib, keyin kesish (post-pruning)

R_alpha(T) = R(T) + alpha * |barglar(T)|
  R(T)   — daraxt xatosi (nopoklik)
  alpha  — murakkablik jarimasi

alpha oshgan sari: kichikroq daraxt
alpha = 0      — to'liq daraxt
alpha -> katta — faqat ildiz

sklearn:
  yol = d.cost_complexity_pruning_path(X, y)   # ccp_alphas, impurities
  DecisionTreeClassifier(ccp_alpha=tanlangan)

cost_complexity_pruning_path mumkin bo'lgan barcha alpha larni beradi — ular orasidan CV bilan tanlanadi. Bu pre-pruning dan ustun tomoni: kesish ma'lumotga qarab amalga oshiriladi, oldindan taxmin qilingan chuqurlikka emas.

2.4. O'quv egri chizig'i

text
Chuqurlik oshgan sari:
  o'quv aniqligi   — monoton O'SADI (1.0 gacha)
  CV aniqligi      — oshadi, MAKSIMUMGA yetadi, keyin tushadi
  farq (gap)       — monoton oshadi

Eng yaxshi chuqurlik = CV maksimumi (yoki 1-SE qoidasi bo'yicha soddaroq)

Ko'p ma'lumot -> chuqurroq daraxt afzal
Ko'p shovqin  -> sayozroq daraxt afzal

O'quv aniqligiga qarab qaror qabul qilmang — u har doim chuqurlik bilan o'sadi va hech qachon overfitting ni ko'rsatmaydi. Faqat CV yoki alohida validatsiya to'plami 12.3-bob haqiqatni aytadi.

2.5. max_features

text
max_features — har bo'linishda TASODIFIY tanlangan belgilar soni
  None (standart) — barcha belgilar
  "sqrt"          — sqrt(p) ta
  "log2"          — log2(p) ta
  int / float     — aniq son / ulush

Bitta daraxtda: odatda None (kerak emas)
Random Forest da: MAJBURIY 15.5-bob — daraxtlarni bir-biridan farqlantiradi

max_features bitta daraxt uchun deyarli foydasiz, lekin ansambl uchun hal qiluvchi: aynan u daraxtlarni korrelyatsiyasiz qiladi va shu orqali ansambl dispersiyasini kamaytiradi 15.4-bob.

2.6. Giperparametr tanlash

python
setka = {"max_depth": [3, 5, 8, 12, None],
         "min_samples_leaf": [1, 5, 20, 50],
         "criterion": ["gini"]}          # criterion ni sozlamang
q = GridSearchCV(DecisionTreeClassifier(random_state=0), setka,
                 cv=StratifiedKFold(5, shuffle=True, random_state=0),
                 scoring="roc_auc").fit(X, y)

scoring ni vazifaga mos tanlang 12.7-bob: nomutanosib sinflarda accuracy emas, roc_auc yoki average_precision. Aks holda grid "hammani ko'pchilik sinfga tegishli" deb aytadigan daraxtni tanlashi mumkin.

2.7. Tuzoqlar

Asosiy tuzoqlar: standart parametrlar bilan qoldirish; o'quv aniqligiga qarab baholash; faqat max_depth ni sozlab, min_samples_leaf ni unutish; ccp_alpha ni CV siz tanlash; nomutanosib sinflarda accuracy bo'yicha sozlash; juda kichik min_samples_leaf (1-2) ni "aniqroq" deb hisoblash; sozlashni test to'plamida qilish (12.9 — leakage); bitta daraxtni uzoq sozlash (ansambl osonroq yaxshi natija beradi).

2.8. Qachon to'xtash — asosiy savol

Cheklovsiz daraxt o'quvni yodlaydi (aniqlik 1.0000) va testda yomon ishlaydi. Yechim — pre-pruning (max_depth, ayniqsa min_samples_leaf) yoki post-pruning (ccp_alpha, cost_complexity_pruning_path bilan). Tanlov CV orqali qilinadi, o'quv aniqligi bo'yicha emas. max_features bitta daraxtga kam foyda beradi, lekin ansambl uchun majburiy. Keyingi dars — daraxt beqarorligi.

3. Tez ma'lumotnoma

python
from sklearn.model_selection import GridSearchCV, StratifiedKFold, validation_curve
from sklearn.tree import DecisionTreeClassifier

DecisionTreeClassifier(max_depth=5, min_samples_leaf=20, max_leaf_nodes=None,
                       min_impurity_decrease=0.0, ccp_alpha=0.0,
                       class_weight=None, random_state=0)

# post-pruning
yol = DecisionTreeClassifier(random_state=0).cost_complexity_pruning_path(X, y)
alphalar = yol.ccp_alphas

# o'quv egri chizig'i
oquv, cvb = validation_curve(DecisionTreeClassifier(random_state=0), X, y,
                             param_name="max_depth",
                             param_range=[2, 4, 6, 8, 12], cv=5)
QOIDA: min_samples_leaf eng ishonchli · CV bo'yicha tanla ·
       o'quv aniqligiga ishonma

To'xtash xulosasi

Cheklovsiz daraxt o'quvni yodlaydi 1.0000-bob
Pre-pruning: max_depth, min_samples_leaf (eng ishonchli), max_leaf_nodes
Post-pruning: ccp_alpha (cost_complexity_pruning_path + CV)
Tanlov faqat CV bo'yicha; max_features ansambl uchun

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14).

Misol 1 — Cheklovsiz daraxt va chuqurlik

python
"""Overfitting ni ko'rish va chuqurlik bilan nazorat qilish (real numpy/sklearn)."""

import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier


def yarat(seed: int = 4, n: int = 2000, shovqin: float = 0.12):
    """Qoidaga asoslangan ma'lumot: daraxt uchun tabiiy vazifa."""
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 8))
    qoida = (((X[:, 0] > 0.4) & (X[:, 1] < 0.0))
             | ((X[:, 2] > 0.0) & (X[:, 3] > 0.5))
             | (X[:, 4] < -1.2))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin          # atayin buzilgan yorliqlar
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Cheklovsiz daraxt ===")
    d = DecisionTreeClassifier(random_state=0).fit(Xtr, ytr)
    print(f"  chuqurlik {d.get_depth()}, barglar {d.get_n_leaves()}")
    print(f"  o'quv aniqligi {d.score(Xtr, ytr):.4f}")
    print(f"  test aniqligi  {d.score(Xte, yte):.4f}")
    print(f"  farq {d.score(Xtr, ytr) - d.score(Xte, yte):.4f}")

    print("\n=== 2. Chuqurlik bo'yicha o'quv egri chizig'i ===")
    print(f"  {'depth':>6} {'barglar':>9} {'o_quv':>8} {'CV':>8} {'test':>8} "
          f"{'farq':>8}")
    eng_cv, eng_d = -1.0, None
    for chuqurlik in [2, 3, 4, 5, 6, 8, 12, 20, None]:
        m = DecisionTreeClassifier(max_depth=chuqurlik, random_state=0).fit(Xtr, ytr)
        oquv = m.score(Xtr, ytr)
        cvb = cross_val_score(DecisionTreeClassifier(max_depth=chuqurlik,
                                                     random_state=0),
                              Xtr, ytr, cv=cv).mean()
        test = m.score(Xte, yte)
        if cvb > eng_cv:
            eng_cv, eng_d = cvb, chuqurlik
        nom = "None" if chuqurlik is None else str(chuqurlik)
        print(f"  {nom:>6} {m.get_n_leaves():>9} {oquv:>8.4f} {cvb:>8.4f} "
              f"{test:>8.4f} {oquv - cvb:>8.4f}")
    print(f"  eng yaxshi CV: max_depth={eng_d} ({eng_cv:.4f})")

    print("\n=== 3. Bayes chegarasi ===")
    print(f"  yorliqlarning 12% i buzilgan -> nazariy maksimum 0.8800")
    print(f"  cheklovsiz daraxt: {d.score(Xte, yte):.4f}")
    m = DecisionTreeClassifier(max_depth=eng_d, random_state=0).fit(Xtr, ytr)
    print(f"  sozlangan daraxt:  {m.score(Xte, yte):.4f}")

    print("\n=== 4. Ma'lumot hajmi ta'siri ===")
    print(f"  {'n':>6} {'eng yaxshi depth':>18} {'CV':>8}")
    for n in [200, 500, 2000, 8000]:
        Xn, yn = yarat(n=n)
        eng, edepth = -1.0, None
        for chuqurlik in [2, 3, 4, 6, 8, 12]:
            b = cross_val_score(DecisionTreeClassifier(max_depth=chuqurlik,
                                                       random_state=0),
                                Xn, yn, cv=cv).mean()
            if b > eng:
                eng, edepth = b, chuqurlik
        print(f"  {n:>6} {edepth:>18} {eng:>8.4f}")
    print("  ⭐ Ko'proq ma'lumot -> chuqurroq daraxt")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Cheklovsiz daraxt ===
  chuqurlik 16, barglar 193
  o'quv aniqligi 1.0000
  test aniqligi  0.7500
  farq 0.2500

=== 2. Chuqurlik bo'yicha o'quv egri chizig'i ===
   depth   barglar    o_quv       CV     test     farq
       2         4   0.7264   0.7193   0.6900   0.0071
       3         7   0.7929   0.7786   0.7667   0.0143
       4        12   0.8657   0.8443   0.8333   0.0214
       5        22   0.8921   0.8450   0.8433   0.0471
       6        40   0.9086   0.8371   0.8367   0.0714
       8        94   0.9450   0.8100   0.8133   0.1350
      12       168   0.9843   0.7743   0.7767   0.2100
      20       193   1.0000   0.7650   0.7500   0.2350
    None       193   1.0000   0.7650   0.7500   0.2350
  eng yaxshi CV: max_depth=5 0.8450-bob

=== 3. Bayes chegarasi ===
  yorliqlarning 12% i buzilgan -> nazariy maksimum 0.8800
  cheklovsiz daraxt: 0.7500
  sozlangan daraxt:  0.8433

=== 4. Ma'lumot hajmi ta'siri ===
       n   eng yaxshi depth       CV
     200                  3   0.7500
     500                  4   0.8080
    2000                  6   0.8585
    8000                  6   0.8716
  ⭐ Ko'proq ma'lumot -> chuqurroq daraxt

Nima ko'rsatdi: 2.1, 2.4-bo'limlar.

Misol 2 — min_samples_leaf va boshqa cheklovlar

python
"""Qaysi chekov eng ishonchli (real numpy/sklearn)."""

import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier


def yarat(seed: int = 11, n: int = 2500, shovqin: float = 0.15):
    """Qoidaga asoslangan ma'lumot: daraxt uchun tabiiy vazifa."""
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 8))
    qoida = (((X[:, 0] > 0.4) & (X[:, 1] < 0.0))
             | ((X[:, 2] > 0.0) & (X[:, 3] > 0.5))
             | (X[:, 4] < -1.2))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin          # atayin buzilgan yorliqlar
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    def baho(**parametrlar):
        m = DecisionTreeClassifier(random_state=0, **parametrlar)
        cvb = cross_val_score(m, Xtr, ytr, cv=cv).mean()
        m.fit(Xtr, ytr)
        return m.get_n_leaves(), m.score(Xtr, ytr), cvb, m.score(Xte, yte)

    print("=== 1. min_samples_leaf ===")
    print(f"  {'qiymat':>8} {'barglar':>9} {'o_quv':>8} {'CV':>8} {'test':>8}")
    for v in [1, 5, 10, 25, 50, 100, 200]:
        barg, oquv, cvb, test = baho(min_samples_leaf=v)
        print(f"  {v:>8} {barg:>9} {oquv:>8.4f} {cvb:>8.4f} {test:>8.4f}")

    print("\n=== 2. max_leaf_nodes ===")
    print(f"  {'qiymat':>8} {'barglar':>9} {'o_quv':>8} {'CV':>8} {'test':>8}")
    for v in [4, 8, 16, 32, 64, 256]:
        barg, oquv, cvb, test = baho(max_leaf_nodes=v)
        print(f"  {v:>8} {barg:>9} {oquv:>8.4f} {cvb:>8.4f} {test:>8.4f}")

    print("\n=== 3. min_impurity_decrease ===")
    print(f"  {'qiymat':>8} {'barglar':>9} {'o_quv':>8} {'CV':>8} {'test':>8}")
    for v in [0.0, 0.0005, 0.001, 0.005, 0.01]:
        barg, oquv, cvb, test = baho(min_impurity_decrease=v)
        print(f"  {v:>8} {barg:>9} {oquv:>8.4f} {cvb:>8.4f} {test:>8.4f}")

    print("\n=== 4. max_depth yetarlimi ===")
    print("  (max_depth=6, lekin min_samples_leaf turlicha)")
    print(f"  {'msl':>8} {'barglar':>9} {'eng kichik barg':>17} {'CV':>8}")
    for v in [1, 5, 20, 50]:
        m = DecisionTreeClassifier(max_depth=6, min_samples_leaf=v,
                                   random_state=0).fit(Xtr, ytr)
        barglar = m.tree_.n_node_samples[m.tree_.feature < 0]
        cvb = cross_val_score(DecisionTreeClassifier(max_depth=6,
                                                     min_samples_leaf=v,
                                                     random_state=0),
                              Xtr, ytr, cv=cv).mean()
        print(f"  {v:>8} {m.get_n_leaves():>9} {int(barglar.min()):>17} "
              f"{cvb:>8.4f}")
    print("  ⭐ max_depth bargdagi namuna sonini KAFOLATLAMAYDI")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. min_samples_leaf ===
    qiymat   barglar    o_quv       CV     test
         1       281   1.0000   0.7240   0.7333
         5       160   0.8977   0.7211   0.7600
        10       107   0.8571   0.7686   0.8040
        25        50   0.8400   0.8274   0.8813
        50        27   0.8291   0.8257   0.8640
       100        14   0.8234   0.8206   0.8560
       200         7   0.7554   0.7160   0.7680

=== 2. max_leaf_nodes ===
    qiymat   barglar    o_quv       CV     test
         4         4   0.7166   0.7360   0.7013
         8         8   0.8400   0.8349   0.8813
        16        16   0.8514   0.8189   0.8600
        32        32   0.8680   0.8166   0.8587
        64        64   0.8926   0.7863   0.8387
       256       256   0.9926   0.7251   0.7480

=== 3. min_impurity_decrease ===
    qiymat   barglar    o_quv       CV     test
       0.0       281   1.0000   0.7240   0.7333
    0.0005       158   0.9514   0.7337   0.7920
     0.001        40   0.8749   0.7777   0.8560
     0.005         8   0.8400   0.8349   0.8813
      0.01         6   0.8217   0.8240   0.8573

=== 4. max_depth yetarlimi ===
  (max_depth=6, lekin min_samples_leaf turlicha)
       msl   barglar   eng kichik barg       CV
         1        41                 1   0.8120
         5        42                 5   0.7971
        20        33                20   0.8349
        50        21                50   0.8257
  ⭐ max_depth bargdagi namuna sonini KAFOLATLAMAYDI

Nima ko'rsatdi: 2.2-bo'lim.

Misol 3 — Cost-complexity pruning

python
"""To'liq o'stirib, keyin kesish (real numpy/sklearn)."""

import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier


def yarat(seed: int = 21, n: int = 2000, shovqin: float = 0.14):
    """Qoidaga asoslangan ma'lumot: daraxt uchun tabiiy vazifa."""
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 8))
    qoida = (((X[:, 0] > 0.4) & (X[:, 1] < 0.0))
             | ((X[:, 2] > 0.0) & (X[:, 3] > 0.5))
             | (X[:, 4] < -1.2))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin          # atayin buzilgan yorliqlar
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Pruning yo'li ===")
    yol = DecisionTreeClassifier(random_state=0).cost_complexity_pruning_path(Xtr, ytr)
    alphalar = yol.ccp_alphas[:-1]            # oxirgisi = faqat ildiz
    print(f"  mumkin bo'lgan alpha lar soni: {len(alphalar)}")
    print(f"  eng kichik {alphalar.min():.6f}, eng katta {alphalar.max():.6f}")

    print("\n=== 2. Tanlangan alpha lar bo'yicha ===")
    namunalar = np.unique(np.round(np.geomspace(max(alphalar.min(), 1e-5),
                                                alphalar.max(), 8), 6))
    print(f"  {'alpha':>10} {'barglar':>9} {'o_quv':>8} {'CV':>8} {'test':>8}")
    eng_cv, eng_a = -1.0, 0.0
    for a in namunalar:
        m = DecisionTreeClassifier(ccp_alpha=a, random_state=0).fit(Xtr, ytr)
        cvb = cross_val_score(DecisionTreeClassifier(ccp_alpha=a, random_state=0),
                              Xtr, ytr, cv=cv).mean()
        if cvb > eng_cv:
            eng_cv, eng_a = cvb, a
        print(f"  {a:>10.6f} {m.get_n_leaves():>9} {m.score(Xtr, ytr):>8.4f} "
              f"{cvb:>8.4f} {m.score(Xte, yte):>8.4f}")
    print(f"  eng yaxshi alpha: {eng_a:.6f} (CV {eng_cv:.4f})")

    print("\n=== 3. Post-pruning va pre-pruning solishtiruvi ===")
    variantlar = {
        "cheklovsiz": {},
        "max_depth=4": {"max_depth": 4},
        "min_samples_leaf=50": {"min_samples_leaf": 50},
        "ccp_alpha (tanlangan)": {"ccp_alpha": float(eng_a)},
    }
    print(f"  {'variant':<24} {'barglar':>9} {'CV':>8} {'test':>8}")
    for nom, p in variantlar.items():
        m = DecisionTreeClassifier(random_state=0, **p).fit(Xtr, ytr)
        cvb = cross_val_score(DecisionTreeClassifier(random_state=0, **p),
                              Xtr, ytr, cv=cv).mean()
        print(f"  {nom:<24} {m.get_n_leaves():>9} {cvb:>8.4f} "
              f"{m.score(Xte, yte):>8.4f}")

    print("\n=== 4. Alpha va daraxt hajmi bog'liqligi ===")
    for a in [0.0, 0.0002, 0.001, 0.005, 0.02, 0.1]:
        m = DecisionTreeClassifier(ccp_alpha=a, random_state=0).fit(Xtr, ytr)
        print(f"  alpha {a:>7.4f}: {m.get_n_leaves():>4} barg, "
              f"chuqurlik {m.get_depth():>2}")
    print("  ⭐ alpha oshgan sari daraxt kichrayadi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Pruning yo'li ===
  mumkin bo'lgan alpha lar soni: 79
  eng kichik 0.000000, eng katta 0.045333

=== 2. Tanlangan alpha lar bo'yicha ===
       alpha   barglar    o_quv       CV     test
    0.000010       211   1.0000   0.7514   0.7233
    0.000033       211   1.0000   0.7514   0.7233
    0.000111       211   1.0000   0.7514   0.7233
    0.000369       211   1.0000   0.7514   0.7233
    0.001229        85   0.9357   0.7729   0.7850
    0.004090         9   0.8729   0.8564   0.8550
    0.013617         6   0.8543   0.8479   0.8450
    0.045333         2   0.6750   0.7314   0.6883
  eng yaxshi alpha: 0.004090 (CV 0.8564)

=== 3. Post-pruning va pre-pruning solishtiruvi ===
  variant                    barglar       CV     test
  cheklovsiz                     211   0.7514   0.7233
  max_depth=4                     13   0.8414   0.8333
  min_samples_leaf=50             23   0.8479   0.8450
  ccp_alpha (tanlangan)            9   0.8564   0.8550

=== 4. Alpha va daraxt hajmi bog'liqligi ===
  alpha  0.0000:  211 barg, chuqurlik 18
  alpha  0.0002:  211 barg, chuqurlik 18
  alpha  0.0010:  131 barg, chuqurlik 16
  alpha  0.0050:    8 barg, chuqurlik  5
  alpha  0.0200:    6 barg, chuqurlik  4
  alpha  0.1000:    1 barg, chuqurlik  0
  ⭐ alpha oshgan sari daraxt kichrayadi

Nima ko'rsatdi: 2.3-bo'lim.

Misol 4 — To'liq sozlash va shovqin darajasi

python
"""GridSearchCV va shovqinning optimal murakkablikka ta'siri (real numpy/sklearn)."""

import numpy as np
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.tree import DecisionTreeClassifier


def yarat(seed: int, n: int, shovqin: float):
    """Qoidaga asoslangan ma'lumot: daraxt uchun tabiiy vazifa."""
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 8))
    qoida = (((X[:, 0] > 0.4) & (X[:, 1] < 0.0))
             | ((X[:, 2] > 0.0) & (X[:, 3] > 0.5))
             | (X[:, 4] < -1.2))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin          # atayin buzilgan yorliqlar
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    cv = StratifiedKFold(5, shuffle=True, random_state=0)
    X, y = yarat(6, 3000, 0.12)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)

    print("=== 1. GridSearchCV ===")
    setka = {"max_depth": [3, 4, 6, 8, 12, None],
             "min_samples_leaf": [1, 10, 30, 80]}
    q = GridSearchCV(DecisionTreeClassifier(random_state=0), setka, cv=cv,
                     scoring="roc_auc", n_jobs=1).fit(Xtr, ytr)
    print(f"  konfiguratsiyalar: {len(q.cv_results_['params'])}")
    print(f"  eng yaxshi: {q.best_params_}")
    print(f"  CV ROC AUC: {q.best_score_:.4f}")
    print(f"  test ROC AUC: "
          f"{roc_auc_score(yte, q.predict_proba(Xte)[:, 1]):.4f}")

    print("\n=== 2. Eng yaxshi 5 konfiguratsiya ===")
    tartib = np.argsort(-q.cv_results_["mean_test_score"])[:5]
    print(f"  {'max_depth':>10} {'msl':>6} {'CV AUC':>9} {'std':>8}")
    for i in tartib:
        p = q.cv_results_["params"][i]
        nom = "None" if p["max_depth"] is None else str(p["max_depth"])
        print(f"  {nom:>10} {p['min_samples_leaf']:>6} "
              f"{q.cv_results_['mean_test_score'][i]:>9.4f} "
              f"{q.cv_results_['std_test_score'][i]:>8.4f}")

    print("\n=== 3. Shovqin darajasi va optimal murakkablik ===")
    print(f"  {'shovqin':>8} {'eng yaxshi depth':>18} {'msl':>6} {'CV AUC':>9}")
    for shovqin in [0.0, 0.05, 0.15, 0.30]:
        Xn, yn = yarat(6, 3000, shovqin)
        qn = GridSearchCV(DecisionTreeClassifier(random_state=0), setka, cv=cv,
                          scoring="roc_auc").fit(Xn, yn)
        nom = ("None" if qn.best_params_["max_depth"] is None
               else str(qn.best_params_["max_depth"]))
        print(f"  {shovqin:>8.2f} {nom:>18} "
              f"{qn.best_params_['min_samples_leaf']:>6} {qn.best_score_:>9.4f}")

    print("\n=== 4. Sozlangan daraxt va ansambl ===")
    from sklearn.ensemble import RandomForestClassifier
    orm = RandomForestClassifier(n_estimators=300, random_state=0,
                                 n_jobs=1).fit(Xtr, ytr)
    print(f"  sozlangan daraxt: test AUC "
          f"{roc_auc_score(yte, q.predict_proba(Xte)[:, 1]):.4f}")
    print(f"  RandomForest (sozlanmagan): test AUC "
          f"{roc_auc_score(yte, orm.predict_proba(Xte)[:, 1]):.4f}")
    print("  ⭐ Sozlanmagan ansambl sozlangan daraxtdan kuchli 15.5-bob")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. GridSearchCV ===
  konfiguratsiyalar: 24
  eng yaxshi: {'max_depth': 8, 'min_samples_leaf': 80}
  CV ROC AUC: 0.8708
  test ROC AUC: 0.8844

=== 2. Eng yaxshi 5 konfiguratsiya ===
   max_depth    msl    CV AUC      std
           8     80    0.8708   0.0180
          12     80    0.8708   0.0180
        None     80    0.8708   0.0180
        None     10    0.8699   0.0144
          12     10    0.8698   0.0119

=== 3. Shovqin darajasi va optimal murakkablik ===
   shovqin   eng yaxshi depth    msl    CV AUC
      0.00                  6     10    0.9981
      0.05                  8     10    0.9458
      0.15                  8     30    0.8362
      0.30                  4     80    0.6979

=== 4. Sozlangan daraxt va ansambl ===
  sozlangan daraxt: test AUC 0.8844
  RandomForest (sozlanmagan): test AUC 0.8877
  ⭐ Sozlanmagan ansambl sozlangan daraxtdan kuchli (15.5)

Nima ko'rsatdi: 2.4, 2.6-bo'limlar.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"O'quv aniqligi 1.0 — yaxshi model" Cheklov yo'qligi belgisi
"max_depth yetarli" min_samples_leaf ishonchliroq
"min_samples_leaf=1 aniqroq" Yodlash
"ccp_alpha ni ko'zdan tanlash mumkin" CV kerak
"Chuqurroq daraxt har doim yomon" Ko'p ma'lumotda yaxshi
"Shovqin murakkablikka ta'sir qilmaydi" Shovqin -> sayozroq
"Pre va post-pruning bir xil" Har xil mexanizm
"Daraxtni uzoq sozlash kerak" Ansambl osonroq

6. Keng tarqalgan xatolar va yechimlari

1. Standart parametrlar

python
DecisionTreeClassifier().fit(X, y)                                # ⚠️
DecisionTreeClassifier(max_depth=6, min_samples_leaf=20, random_state=0) # ✅

2. O'quv aniqligi bo'yicha baholash

python
print(d.score(Xtr, ytr))                                          # ⚠️
print(cross_val_score(d, Xtr, ytr, cv=5).mean())                  # ✅

3. min_samples_leaf ni unutish

python
DecisionTreeClassifier(max_depth=10)     # bargda 1 namuna bo'lishi mumkin # ⚠️
DecisionTreeClassifier(max_depth=10, min_samples_leaf=20)         # ✅

4. ccp_alpha ni CV siz tanlash

python
DecisionTreeClassifier(ccp_alpha=0.01)   # qayerdan olindi?        # ⚠️
# cost_complexity_pruning_path + GridSearchCV                      # ✅

5. Testda sozlash

python
for d in [3, 5, 8]:  ...  m.score(Xte, yte)   # test bilan tanlash # ⚠️
GridSearchCV(..., cv=5).fit(Xtr, ytr)    # test faqat oxirida      # ✅

6. Nomutanosib sinflarda accuracy

python
GridSearchCV(..., scoring="accuracy")    # 95% sinf bo'lsa         # ⚠️
GridSearchCV(..., scoring="average_precision")                     # ✅

7. Bitta daraxtni haddan uzoq sozlash

python
# 500 konfiguratsiyali grid, bitta daraxt uchun                    # ⚠️
RandomForestClassifier(n_estimators=300)  # sozlanmasdan kuchliroq # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 12.4-dars (o'tilgan): Overfitting
  • 12.3-dars (o'tilgan): Validatsiya
  • 15.3-dars: Daraxt beqarorligi
  • 15.5-dars: Random Forest giperparametrlari
  • 15.9-dars: Boosting sozlash

8. Eng yaxshi amaliyotlar

  1. min_samples_leaf ni albatta qo'ying.

  2. max_depth bilan birga ishlating.

  3. CV bo'yicha tanlang.

  4. scoring ni vazifaga moslang.

  5. ccp_alpha ni yo'l orqali toping.

  6. Shovqinni hisobga oling.

  7. Testni oxirida ishlating.

  8. Ansamblga vaqtida o'ting.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # cheklovsiz daraxtning o'quv aniqligi?
2.  # eng ishonchli chekov?
3.  # max_depth nimani kafolatlamaydi?
4.  # min_samples_leaf float bersa?
5.  # ccp_alpha nima qiladi?
6.  # alpha oshsa daraxt qanday bo'ladi?
7.  # o'quv egri chizig'ida CV qanday harakat qiladi?
8.  # shovqin ko'p bo'lsa chuqurlik?
9.  # ko'p ma'lumot bo'lsa chuqurlik?
10. # max_features qayerda muhim?
11. # sozlash qaysi to'plamda?
12. # nomutanosib sinflarda scoring?
Javoblar
  1. 1.0000
  2. min_samples_leaf
  3. Bargdagi namunalar sonini
  4. Ulush sifatida
  5. Post-pruning jarimasi
  6. Kichrayadi
  7. Oshadi, maksimum, tushadi
  8. Sayozroq
  9. Chuqurroq
  10. Random Forest (ansambl)
  11. O'quv (CV bilan)
  12. roc_auc / average_precision

Vazifa 2: Xatolarni tuzating

python
1.  DecisionTreeClassifier().fit(X, y)

2.  print("aniqlik:", d.score(Xtr, ytr))

3.  DecisionTreeClassifier(max_depth=15)

4.  DecisionTreeClassifier(ccp_alpha=0.01)   # taxminan

5.  for d in [3, 5, 8]: m.fit(Xtr, ytr); print(m.score(Xte, yte))
Javoblar
python
1.  DecisionTreeClassifier(max_depth=6, min_samples_leaf=20, random_state=0)

2.  print(cross_val_score(d, Xtr, ytr, cv=5).mean())

3.  DecisionTreeClassifier(max_depth=15, min_samples_leaf=20)

4.  # cost_complexity_pruning_path + GridSearchCV

5.  GridSearchCV(d, {"max_depth": [3, 5, 8]}, cv=5).fit(Xtr, ytr)

Vazifa 3: Overfitting

Modellang:

  1. Cheklovsiz daraxt
  2. Chuqurlik egri chizig'i
  3. Bayes chegarasi
  4. Ma'lumot hajmi

Vazifa 4: Cheklovlar

Modellang:

  1. min_samples_leaf
  2. max_leaf_nodes
  3. min_impurity_decrease
  4. max_depth cheklovi

Vazifa 5: Pruning

Modellang:

  1. Pruning yo'li
  2. Alpha tanlash
  3. Pre va post
  4. Hajm bog'liqligi

Vazifa 6: Sozlash

Modellang:

  1. GridSearchCV
  2. Eng yaxshi konfiguratsiyalar
  3. Shovqin ta'siri
  4. Ansambl bilan solishtirish

Vazifa 7: O'ylash

Post-pruning (ccp_alpha) nazariy adabiyotda pre-pruning dan ustun deb ko'rsatiladi, lekin amalda deyarli hamma max_depth va min_samples_leaf ishlatadi. Nega?

Javob

Qisqa javob: post-pruning haqiqatan ham sifatliroq daraxt beradi, lekin uning afzalligi kichik (odatda 1-2%), narxi esa katta (to'liq daraxtni o'stirish + yo'lni hisoblash + CV) — va zamonaviy amaliyotda bitta daraxt o'rniga ansambl ishlatilgani uchun bu farq umuman ahamiyatsiz bo'lib qoladi.

1. Nazariy ustunlik nimada

Yondashuv Qaror asosi
Pre-pruning Oldindan belgilangan chegara
Post-pruning To'liq daraxtni ko'rib, xato/murakkablik muvozanati

Pre-pruning **"ufq muammosi"**ga uchraydi: hozirgi bo'linish foydasiz ko'rinishi mumkin, lekin undan keyingi bo'linish juda foydali bo'lishi mumkin (XOR ga o'xshash naqshlar). Ochko'z to'xtash bunday holatni boy beradi.

2. Amaliy narx

  • To'liq daraxtni o'stirish katta ma'lumotda qimmat
  • cost_complexity_pruning_path yuzlab alpha qaytaradi
  • Har alpha uchun CV kerak -> yuzlab model
  • max_depth bo'yicha grid esa 5-6 qiymat

3. Asosiy sabab: ansambllar

  1. Random Forest da daraxtlar atayin to'liq o'stiriladi (15.5)
  2. Boosting da daraxtlar atayin sayoz (15.8)
  3. Ikkala holatda ham pruning kerak emas
  4. Bitta daraxt esa faqat talqin uchun ishlatiladi — u yerda max_depth=3 shunchaki qulayroq

4. Qachon post-pruning arziydi

  • Bitta daraxt yakuniy model bo'lsa (talqin talabi qat'iy)
  • Ma'lumot kichik (o'stirish arzon)
  • Qoidalar ro'yxati chiqarish kerak

5. Xulosa

  1. Nazariy ustunlik bor, lekin kichik
  2. Hisoblash narxi yuqori
  3. Ansambllar masalani boshqa yo'l bilan hal qiladi
  4. Bitta daraxt kerak bo'lsa — post-pruning ni ko'rib chiqing

Nimani mustahkamlaydi: 2.3, 2.7-bo'limlar.

Xulosa

Bu darsda daraxt o'sishini nazorat qilishni o'rgandik.

Eng muhim uch fikr:

  1. Cheklovsiz daraxt yodlaydi. DecisionTreeClassifier() standart holatda hech qanday cheklovsiz o'sadi va o'quv aniqligi 1.0000 bo'ladi — bu modelning yaxshiligi emas, cheklov yo'qligining belgisi. Shovqinli ma'lumotda test aniqligi keskin pasayadi.

  2. min_samples_leaf — eng ishonchli chekov. U har bargda yetarlicha namuna bo'lishini kafolatlaydi, ya'ni barg bashorati statistik ma'noga ega bo'ladi. max_depth esa faqat yo'l uzunligini cheklaydi va bargdagi namunalar sonini kafolatlamaydi — shuning uchun ikkalasini birga ishlating.

  3. Tanlov faqat CV bo'yicha. O'quv aniqligi chuqurlik bilan monoton o'sadi va overfitting ni hech qachon ko'rsatmaydi. ccp_alpha (post-pruning) cost_complexity_pruning_path orqali topiladi va CV bilan tanlanadi. Shovqin ko'p bo'lsa sayozroq, ma'lumot ko'p bo'lsa chuqurroq daraxt afzal. Va eng muhimi: sozlanmagan ansambl ko'pincha uzoq sozlangan bitta daraxtdan kuchliroq.

Keyingi darsda daraxt beqarorligini o'rganamiz: nega kichik o'zgarish tuzilmani butunlay o'zgartiradi va bu ansambllarga qanday yo'l ochadi.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
15.2-dars: O'sish va to'xtash shartlari — IlmHamroh