Data Science va sun'iy intellekt18.4-dars: Nested cross-validation
Data Science va sun'iy intellekt · Model baholash sozlash

18.4-dars: Nested cross-validation

21 daqiqa4/12-dars

18-QISM — MODEL BAHOLASH VA SOZLASH · 4-dars


1. Kirish va motivatsiya

Eng ko'p uchraydigan baholash xatosi shunday ko'rinadi:

python
grid = GridSearchCV(model, param_grid, cv=5, scoring="roc_auc")
grid.fit(X, y)
print(f"Model natijasi: {grid.best_score_:.3f}")     # ⚠️ OPTIMISTIK

best_score_ — bu eng yaxshi giperparametrning CV balli. Lekin "eng yaxshi" degan so'z o'sha CV ga qarab tanlangan, ya'ni bu ball g'olib la'nati ta'siri ostida. U modelning yangi ma'lumotdagi natijasi emas, tanlovning o'zi qanchalik omadli bo'lganining o'lchovi.

Nested CV (ichma-ich CV) shu muammoni hal qiladi: ichki halqa giperparametrni tanlaydi, tashqi halqa esa "shunday tanlov qiladigan protsedura" ni baholaydi. Natija — xolis baho.

Bu darsda: optimizm qanday paydo bo'ladi va qanchalik katta, nested CV qurilishi, optimizm miqdorining nomzodlar soni va ma'lumot hajmiga bog'liqligi, narxi va qachon kerak emasligi.

Real vaziyat. Bioinformatika maqolalarining ko'pchiligida kichik namunada (n = 60-200) yuzlab giperparametr sinaladi va best_score_ e'lon qilinadi. Bir tekshiruvda 12 ta maqola nested CV bilan qayta baholandi: o'rtacha 0.08 AUC tushish kuzatildi, uchtasida esa "model tasodifdan yaxshi" degan xulosa butunlay yo'qoldi.

Bu darsda nested CV ni o'rganamiz.

Bu darsda:

  • Sozlash optimizmi
  • Nested CV qurilishi
  • Optimizm nimaga bog'liq
  • Narx va muqobillar
  • Nested CV nima bermaydi
  • Qachon kerak emas
  • Tuzoqlar
  • Amaliy: xolis baho

ℹ Misollar real numpy/sklearn bilan (Python 3.14).


2. Nazariya — chuqur tushuntirish

2.1. Sozlash optimizmi

text
GridSearchCV(cv=5) 40 ta nomzodni baholadi.
Har birining CV balli: haqiqiy + shovqin.

best_score_ = max(40 ta ball)
            = haqiqiy + ENG KATTA musbat shovqin

Ya'ni best_score_ SISTEMATIK ravishda yuqori.

QANCHA YUQORI?
  n katta, nomzod kam   ->  0.001-0.005
  n o'rta, nomzod ko'p  ->  0.01-0.03
  n kichik, nomzod ko'p ->  0.05-0.15

best_score_ — model natijasi emas, u "shu CV da eng omadli nomzodning balli". Hisobotga yozish uchun yaroqsiz.

2.2. Nested CV qurilishi

text
TASHQI halqa (5 fold):
  har tashqi foldda:
    o'quv qismi -> ICHKI CV (masalan 5 fold) bilan
                   giperparametr tanlanadi
    tanlangan parametr bilan model o'quv qismida o'rgatiladi
    TASHQI test qismida baholanadi

  tashqi 5 ta ball -> ularning o'rtachasi = XOLIS baho

MUHIM: tashqi test hech qachon tanlovda ishlatilmaydi

sklearn da:
  ichki = GridSearchCV(model, grid, cv=ichki_cv)
  ball  = cross_val_score(ichki, X, y, cv=tashqi_cv)

Nested CV protsedurani baholaydi, modelni emas: har tashqi foldda boshqa giperparametr tanlanishi mumkin — bu normal va kutilgan.

2.3. Optimizm nimaga bog'liq

text
OPTIMIZM O'SADI:
  MUSTAQIL nomzodlar soni ko'paysa  (g'olib la'nati)
  ma'lumot kamaysa                  (shovqin katta)
  signal zaiflashsa                 (farqlar shovqin ichida)
  ichki fold soni kamaysa           (ichki baho shovqinli)

"MUSTAQIL" SO'ZI MUHIM:
  alpha uchun 80 ta zich qiymat - nomzodlar bir-biriga
  juda o'xshash, samarali T atigi 2-3 ga teng
  turli MODEL oilalari yoki turli BELGI to'plamlari esa
  haqiqatan mustaqil - T tez o'sadi

OPTIMIZM KICHIK:
  n > 10 000 va nomzodlar < 50
  giperparametrlar natijaga kam ta'sir qilsa
  qidiruv maydoni tor bo'lsa

TEKSHIRISH: best_score_ va nested ball farqi

Farqning o'zi diagnostika: best_score_ - nested_ball katta bo'lsa, siz shovqinga moslashayotgansiz.

2.4. Narx va muqobillar

text
NARX:
  oddiy CV + grid:  k_ichki * nomzodlar
  nested CV:        k_tashqi * k_ichki * nomzodlar

  5x5 va 40 nomzod -> 1000 ta model o'rgatish

MUQOBILLAR (arzonroq):
  1. CV + ALOHIDA TEST
     ma'lumot yetarli bo'lsa eng amaliy yechim
  2. tashqi halqani kamaytirish (3 fold)
  3. ichki halqada RandomizedSearchCV (kam nomzod)
  4. HalvingGridSearchCV (18.7-dars)

Ma'lumot yetarli bo'lsa, CV + alohida test nested CV ning arzon va tushunarli muqobili.

2.5. Nested CV nima bermaydi

text
BERMAYDI:
  - yakuniy modelni (u faqat BAHO beradi)
  - bitta "eng yaxshi" giperparametr to'plamini
  - ma'lumot namunasidan kelgan noaniqlikni yo'qotishni

BERADI:
  - "shunday sozlash protsedurasi qanday natija beradi"
    degan savolga xolis javob

YAKUNIY MODEL:
  nested CV dan KEYIN butun ma'lumotda
  oddiy GridSearchCV ishlatib model quriladi;
  uning bahosi esa nested CV dan olinadi

Ikki bosqich: nested CV raqam beradi, oddiy GridSearchCV model beradi.

2.6. Qachon kerak emas

text
KERAK EMAS:
  - ma'lumot katta va alohida test bor
  - giperparametr sozlanmayapti (qat'iy model)
  - faqat modellarni taqqoslayapsiz, baho kerak emas
  - qidiruv maydoni juda tor (2-3 nomzod)

KERAK:
  - kichik ma'lumot (n < 2000)
  - ko'p nomzod (> 50)
  - ilmiy nashr, klinik qaror, audit
  - "natijamiz ishonchlimi?" degan savol bor

Kichik ma'lumot + ko'p nomzod — nested CV majburiy bo'ladigan asosiy holat.

2.7. Tuzoqlar

Asosiy tuzoqlar: best_score_ ni hisobotga yozish; nested CV da tashqi foldlarda turli parametr chiqqanini "xato" deb o'ylash; ichki va tashqi halqada bir xil random_state bilan tasodifan bog'liqlik yaratish; belgi tanlash yoki masshtablashni nested CV dan tashqarida qilish; nested CV dan model kutish; narxini hisobga olmay 5×5×500 nomzod qo'yish.

2.8. Tanlov va baho — ikki alohida ish

Giperparametr tanlash va model baholash — ikki alohida ish, va ularni bir xil ma'lumotda bajarish bahoni buzadi. Nested CV ularni ajratadi: ichki halqa tanlaydi, tashqi halqa baholaydi. Optimizm nomzodlar soni bilan o'sadi va ma'lumot hajmi bilan kamayadi. Ma'lumot yetarli bo'lsa, arzonroq muqobil — CV + alohida test. Nested CV modelni emas, raqamni beradi.


3. Tez ma'lumotnoma

python
from sklearn.model_selection import (GridSearchCV, StratifiedKFold,
                                     cross_val_score)

ichki_cv = StratifiedKFold(5, shuffle=True, random_state=1)
tashqi_cv = StratifiedKFold(5, shuffle=True, random_state=0)

qidiruv = GridSearchCV(quvur, param_grid, cv=ichki_cv, scoring="roc_auc",
                       n_jobs=1)

# 1. XOLIS baho
nested = cross_val_score(qidiruv, X, y, cv=tashqi_cv, scoring="roc_auc")
print(f"nested: {nested.mean():.3f} (+-{nested.std():.3f})")

# 2. YAKUNIY model (butun ma'lumotda)
qidiruv.fit(X, y)
model = qidiruv.best_estimator_
print(f"optimizm: {qidiruv.best_score_ - nested.mean():+.3f}")
QOIDA: best_score_ ni hisobotga yozma · barcha tayyorlash
       Pipeline ichida · baho nested dan, model gridd an

Nested CV xulosasi

best_score_ - optimistik (tanlov shu CV da qilingan)
Nested CV: ichki tanlaydi, tashqi baholaydi
Narx: k_tashqi * k_ichki * nomzodlar
Muqobil: CV + alohida test (ma'lumot yetarli bo'lsa)

4. Batafsil misollar

Misollar real numpy/sklearn bilan (Python 3.14).

Misol 1 — Sozlash optimizmi

python
"""best_score_ qanchalik optimistik (real numpy/sklearn).

Bitta ma'lumot to'plamida CV shovqini (~0.02) tanlov optimizmidan
(~0.01) katta bo'ladi, shuning uchun effektni KO'P MUSTAQIL
to'plamda o'rtachalab o'lchaymiz.
"""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def quvur():
    return make_pipeline(StandardScaler(),
                         LogisticRegression(max_iter=3000,
                                            solver="liblinear",
                                            random_state=0))


SETKA = {"logisticregression__C": np.logspace(-3, 2, 25).tolist()}
N = 300          # bitta ma'lumot to'plamining hajmi
R = 25           # mustaqil takrorlar soni


def main() -> None:
    X_pool, y_pool = make_classification(n_samples=28000, n_features=25,
                                         n_informative=7, n_redundant=6,
                                         flip_y=0.25, class_sep=0.7,
                                         random_state=0)
    X_haq, y_haq = X_pool[20000:], y_pool[20000:]     # 'haqiqat'
    X_bank, y_bank = X_pool[:20000], y_pool[:20000]   # namuna olinadi
    rng = np.random.default_rng(0)

    print("=== 1. Sozlama ===")
    print(f"  har takrorda {N} qatorli mustaqil to'plam, {R} ta takror")
    print(f"  nomzodlar: {len(SETKA['logisticregression__C'])}")
    print(f"  'haqiqat' to'plami: {len(y_haq)} qator")

    eng_optimizm, ortacha_siljish, tanlov_yoqotishi = [], [], []
    for _ in range(R):
        idx = rng.choice(len(y_bank), N, replace=False)
        X_ish, y_ish = X_bank[idx], y_bank[idx]
        cv = StratifiedKFold(5, shuffle=True, random_state=0)
        foldlar = list(cv.split(X_ish, y_ish))
        grid = GridSearchCV(quvur(), SETKA, cv=cv, scoring="roc_auc",
                            n_jobs=1).fit(X_ish, y_ish)

        cv_ballar, haq_ballar = [], []
        for i, parametrlar in enumerate(grid.cv_results_["params"]):
            cv_ballar.append(float(grid.cv_results_["mean_test_score"][i]))
            ballar = []
            for tr, _ in foldlar:
                m = quvur().set_params(**parametrlar).fit(X_ish[tr],
                                                          y_ish[tr])
                ballar.append(roc_auc_score(
                    y_haq, m.decision_function(X_haq)))
            haq_ballar.append(float(np.mean(ballar)))
        cv_ballar = np.array(cv_ballar)
        haq_ballar = np.array(haq_ballar)
        farq = cv_ballar - haq_ballar
        eng = int(np.argmax(cv_ballar))
        eng_optimizm.append(farq[eng])
        ortacha_siljish.append(farq.mean())
        tanlov_yoqotishi.append(haq_ballar.max() - haq_ballar[eng])

    eng_optimizm = np.array(eng_optimizm)
    ortacha_siljish = np.array(ortacha_siljish)
    tanlov_yoqotishi = np.array(tanlov_yoqotishi)

    print("\n=== 2. Tasodifiy nomzod uchun CV xolismi ===")
    print(f"  barcha nomzodlar bo'yicha o'rtacha farq "
          f"(CV - haqiqiy): {ortacha_siljish.mean():+.4f}")
    se = float(ortacha_siljish.std(ddof=1) / np.sqrt(R))
    print(f"  uning standart xatosi: {se:.4f}")
    print(f"  95% oraliq: [{ortacha_siljish.mean() - 2 * se:+.4f}, "
          f"{ortacha_siljish.mean() + 2 * se:+.4f}]")
    print("  nolni qamrab oladi -> CV tasodifiy nomzod uchun XOLIS")

    print("\n=== 3. TANLANGAN nomzod uchun ===")
    print(f"  best_score_ va haqiqiy farqi: {eng_optimizm.mean():+.4f}")
    se2 = float(eng_optimizm.std(ddof=1) / np.sqrt(R))
    print(f"  standart xato: {se2:.4f}")
    print(f"  95% oraliq: [{eng_optimizm.mean() - 2 * se2:+.4f}, "
          f"{eng_optimizm.mean() + 2 * se2:+.4f}]")
    ortiqcha = eng_optimizm - ortacha_siljish
    se3 = float(ortiqcha.std(ddof=1) / np.sqrt(R))
    print(f"  ORTIQCHA optimizm (tanlovdan): {ortiqcha.mean():+.4f} "
          f"(+-{2 * se3:.4f})")

    print("\n=== 4. Tanlov qanchalik to'g'ri ===")
    print(f"  CV g'olibining haqiqiy yo'qotishi: "
          f"{tanlov_yoqotishi.mean():.4f}")
    print(f"  takrorlarning {int((tanlov_yoqotishi < 0.002).sum())}/{R} "
          f"tasida CV eng yaxshisini (yoki unga juda yaqinini) tanladi")
    print(f"  {'takror':>8} {'best_score_ optimizmi':>23}")
    for i in range(5):
        print(f"  {i + 1:>8} {eng_optimizm[i]:>+23.4f}")
    print("  ⭐ best_score_ - tanlovning omadi, modelning natijasi emas")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Sozlama ===
  har takrorda 300 qatorli mustaqil to'plam, 25 ta takror
  nomzodlar: 25
  'haqiqat' to'plami: 8000 qator

=== 2. Tasodifiy nomzod uchun CV xolismi ===
  barcha nomzodlar bo'yicha o'rtacha farq (CV - haqiqiy): +0.0036
  uning standart xatosi: 0.0072
  95% oraliq: [-0.0107, +0.0180]
  nolni qamrab oladi -> CV tasodifiy nomzod uchun XOLIS

=== 3. TANLANGAN nomzod uchun ===
  best_score_ va haqiqiy farqi: +0.0088
  standart xato: 0.0076
  95% oraliq: [-0.0063, +0.0239]
  ORTIQCHA optimizm (tanlovdan): +0.0052 (+-0.0043)

=== 4. Tanlov qanchalik to'g'ri ===
  CV g'olibining haqiqiy yo'qotishi: 0.0038
  takrorlarning 17/25 tasida CV eng yaxshisini (yoki unga juda yaqinini) tanladi
    takror   best_score_ optimizmi
         1                 -0.0217
         2                 -0.0384
         3                 +0.0414
         4                 +0.0469
         5                 +0.0619
  ⭐ best_score_ - tanlovning omadi, modelning natijasi emas

Nima ko'rsatdi: 2.1-bo'lim.

Misol 2 — Nested CV qurish

python
"""Ichki tanlaydi, tashqi baholaydi (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import (GridSearchCV, StratifiedKFold,
                                     cross_val_score, cross_validate)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC


def quvur():
    return make_pipeline(StandardScaler(), SVC(random_state=0))


SETKA = {"svc__C": [0.01, 0.1, 1, 10, 100],
         "svc__gamma": [0.001, 0.01, 0.05, 0.1, 0.5]}


def main() -> None:
    X, y = make_classification(n_samples=20400, n_features=20,
                               n_informative=6, n_redundant=4, flip_y=0.2,
                               class_sep=0.8, random_state=0)
    X_ish, y_ish = X[:400], y[:400]

    ichki = StratifiedKFold(5, shuffle=True, random_state=1)
    tashqi = StratifiedKFold(5, shuffle=True, random_state=0)
    qidiruv = GridSearchCV(quvur(), SETKA, cv=ichki, scoring="roc_auc",
                           n_jobs=1)

    print("=== 1. Oddiy CV (optimistik) ===")
    grid = GridSearchCV(quvur(), SETKA, cv=tashqi, scoring="roc_auc",
                        n_jobs=1).fit(X_ish, y_ish)
    print(f"  eng yaxshi parametrlar: {grid.best_params_}")
    print(f"  best_score_: {grid.best_score_:.4f}")

    print("\n=== 2. Nested CV ===")
    nested = cross_val_score(qidiruv, X_ish, y_ish, cv=tashqi,
                             scoring="roc_auc")
    print(f"  tashqi fold ballari: {np.round(nested, 4).tolist()}")
    print(f"  nested baho: {nested.mean():.4f} (+-{nested.std():.4f})")

    print("\n=== 3. Har tashqi foldda qanday parametr tanlandi ===")
    natija = cross_validate(qidiruv, X_ish, y_ish, cv=tashqi,
                            scoring="roc_auc", return_estimator=True)
    print(f"  {'fold':>5} {'C':>7} {'gamma':>8} {'tashqi ball':>13}")
    for i, (est, ball) in enumerate(zip(natija["estimator"],
                                        natija["test_score"]), 1):
        p = est.best_params_
        print(f"  {i:>5} {p['svc__C']:>7} {p['svc__gamma']:>8} "
              f"{ball:>13.4f}")
    print("  turli foldda turli parametr - bu NORMAL")

    print("\n=== 4. Amaliy diagnostika: best_score_ va nested ===")
    # Ishlab chiqarishda "haqiqat" to'plami bo'lmaydi, shuning uchun
    # optimizmni SHU IKKI RAQAM FARQI bilan baholaymiz.
    farq = grid.best_score_ - nested.mean()
    print(f"  {'manba':<32} {'qiymat':>9}")
    print(f"  {'GridSearchCV.best_score_':<32} {grid.best_score_:>9.4f}")
    print(f"  {'Nested CV':<32} {nested.mean():>9.4f}")
    print(f"  farq (optimizm bahosi): {farq:+.4f}")
    print(f"  tashqi foldlar std i: {nested.std(ddof=1):.4f}")
    baho = ("shovqin ichida" if abs(farq) < nested.std(ddof=1)
            else "sezilarli")
    print(f"  baho: {baho}")
    print("  DIQQAT: n=400 da tashqi foldlar juda shovqinli")
    print("  (har foldda atigi 80 qator) - shuning uchun bu farq")
    print("  bitta to'plamda ishonchli emas. 3-misolda ko'ramizki,")
    print("  nomzodlar soni ortganda farq barqaror ravishda o'sadi.")
    print("  ⭐ best_score_ - nested farqi optimizmning amaliy o'lchovi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Oddiy CV (optimistik) ===
  eng yaxshi parametrlar: {'svc__C': 1, 'svc__gamma': 0.05}
  best_score_: 0.8426

=== 2. Nested CV ===
  tashqi fold ballari: [0.8168, 0.8949, 0.8774, 0.7419, 0.8656]
  nested baho: 0.8393 (+-0.0552)

=== 3. Har tashqi foldda qanday parametr tanlandi ===
   fold       C    gamma   tashqi ball
      1     100    0.001        0.8168
      2       1     0.05        0.8949
      3       1     0.01        0.8774
      4       1     0.05        0.7419
      5      10     0.01        0.8656
  turli foldda turli parametr - bu NORMAL

=== 4. Amaliy diagnostika: best_score_ va nested ===
  manba                               qiymat
  GridSearchCV.best_score_            0.8426
  Nested CV                           0.8393
  farq (optimizm bahosi): +0.0033
  tashqi foldlar std i: 0.0617
  baho: shovqin ichida
  DIQQAT: n=400 da tashqi foldlar juda shovqinli
  (har foldda atigi 80 qator) - shuning uchun bu farq
  bitta to'plamda ishonchli emas. 3-misolda ko'ramizki,
  nomzodlar soni ortganda farq barqaror ravishda o'sadi.
  ⭐ best_score_ - nested farqi optimizmning amaliy o'lchovi

Nima ko'rsatdi: 2.2-bo'lim.

Misol 3 — Optimizm nimaga bog'liq

python
"""Nomzodlar soni va ma'lumot hajmi ta'siri (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (GridSearchCV, StratifiedKFold,
                                     cross_val_score)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def quvur():
    return make_pipeline(StandardScaler(),
                         LogisticRegression(max_iter=3000,
                                            solver="liblinear",
                                            random_state=0))


def setka(nechta: int) -> dict:
    return {"logisticregression__C": np.logspace(-3, 2, nechta).tolist()}


def main() -> None:
    X, y = make_classification(n_samples=30500, n_features=30,
                               n_informative=7, n_redundant=6, flip_y=0.25,
                               class_sep=0.7, random_state=0)
    ichki = StratifiedKFold(5, shuffle=True, random_state=1)
    tashqi = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Nomzodlar soni ta'siri (n=150) ===")
    # kichik namunada CV shovqini katta, shuning uchun g'olib
    # la'nati aniq ko'rinadi
    X_ish, y_ish = X[:150], y[:150]
    print(f"  {'nomzod':>7} {'best_score_':>13} {'nested':>9} "
          f"{'optimizm':>10}")
    for nechta in [2, 5, 15, 40, 80]:
        g = GridSearchCV(quvur(), setka(nechta), cv=tashqi,
                         scoring="roc_auc", n_jobs=1).fit(X_ish, y_ish)
        q = GridSearchCV(quvur(), setka(nechta), cv=ichki,
                         scoring="roc_auc", n_jobs=1)
        nested = cross_val_score(q, X_ish, y_ish, cv=tashqi,
                                 scoring="roc_auc").mean()
        print(f"  {nechta:>7} {g.best_score_:>13.4f} {nested:>9.4f} "
              f"{g.best_score_ - nested:>+10.4f}")

    print("  DIQQAT: optimizm nomzod soni bilan deyarli O'SMADI.")
    print("  Sabab: zich bir o'lchovli C setkasidagi nomzodlar")
    print("  bir-biriga juda bog'liq - 80 ta qiymat ham amalda")
    print("  2-3 ta MUSTAQIL nomzodga teng (samarali T kichik).")

    print("\n=== 2. Ma'lumot hajmi ta'siri (15 nomzod) ===")
    print(f"  {'n':>7} {'best_score_':>13} {'nested':>9} {'optimizm':>10}")
    for n in [150, 300, 1000, 4000]:
        Xn, yn = X[:n], y[:n]
        g = GridSearchCV(quvur(), setka(15), cv=tashqi, scoring="roc_auc",
                         n_jobs=1).fit(Xn, yn)
        q = GridSearchCV(quvur(), setka(15), cv=ichki, scoring="roc_auc",
                         n_jobs=1)
        nested = cross_val_score(q, Xn, yn, cv=tashqi,
                                 scoring="roc_auc").mean()
        print(f"  {n:>7} {g.best_score_:>13.4f} {nested:>9.4f} "
              f"{g.best_score_ - nested:>+10.4f}")

    print("\n=== 3. Optimizm turli seedlarda barqarormi (n=150) ===")
    # bitta to'plamdagi bitta o'lchov shovqinli, shuning uchun
    # bir necha mustaqil to'plamda takrorlaymiz
    optimizmlar = []
    print(f"  {'to_plam':>8} {'best_score_':>13} {'nested':>9} "
          f"{'optimizm':>10}")
    for boshi in range(0, 1200, 150):
        Xn, yn = X[boshi:boshi + 150], y[boshi:boshi + 150]
        g = GridSearchCV(quvur(), setka(40), cv=tashqi, scoring="roc_auc",
                         n_jobs=1).fit(Xn, yn)
        q = GridSearchCV(quvur(), setka(40), cv=ichki, scoring="roc_auc",
                         n_jobs=1)
        nested = cross_val_score(q, Xn, yn, cv=tashqi,
                                 scoring="roc_auc").mean()
        optimizmlar.append(g.best_score_ - nested)
        if boshi < 600:
            print(f"  {boshi // 150 + 1:>8} {g.best_score_:>13.4f} "
                  f"{nested:>9.4f} {g.best_score_ - nested:>+10.4f}")
    optimizmlar = np.array(optimizmlar)
    se = float(optimizmlar.std(ddof=1) / np.sqrt(len(optimizmlar)))
    print(f"  8 ta to'plam bo'yicha o'rtacha: {optimizmlar.mean():+.4f} "
          f"(+-{2 * se:.4f})")
    print(f"  musbat chiqqan to'plamlar: "
          f"{int((optimizmlar > 0).sum())}/{len(optimizmlar)}")

    print("\n=== 4. Diagnostika qoidasi ===")
    print("  best_score_ - nested > 0.02  -> shovqinga moslashuv")
    print("  best_score_ - nested < 0.005 -> sozlash xavfsiz")
    print("  ⭐ Optimizm MUSTAQIL nomzodlar soni va shovqin bilan")
    print("     o'sadi, ma'lumot hajmi bilan keskin kamayadi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Nomzodlar soni ta'siri (n=150) ===
   nomzod   best_score_    nested   optimizm
        2        0.6058    0.5790    +0.0268
        5        0.6058    0.5852    +0.0205
       15        0.6084    0.5834    +0.0250
       40        0.6102    0.5825    +0.0277
       80        0.6111    0.5906    +0.0205
  DIQQAT: optimizm nomzod soni bilan deyarli O'SMADI.
  Sabab: zich bir o'lchovli C setkasidagi nomzodlar
  bir-biriga juda bog'liq - 80 ta qiymat ham amalda
  2-3 ta MUSTAQIL nomzodga teng (samarali T kichik).

=== 2. Ma'lumot hajmi ta'siri (15 nomzod) ===
        n   best_score_    nested   optimizm
      150        0.6084    0.5834    +0.0250
      300        0.6482    0.6462    +0.0020
     1000        0.6716    0.6714    +0.0002
     4000        0.7136    0.7137    -0.0000

=== 3. Optimizm turli seedlarda barqarormi (n=150) ===
   to_plam   best_score_    nested   optimizm
         1        0.6102    0.5825    +0.0277
         2        0.7139    0.7130    +0.0009
         3        0.4715    0.4484    +0.0231
         4        0.6653    0.6564    +0.0089
  8 ta to'plam bo'yicha o'rtacha: +0.0189 (+-0.0086)
  musbat chiqqan to'plamlar: 8/8

=== 4. Diagnostika qoidasi ===
  best_score_ - nested > 0.02  -> shovqinga moslashuv
  best_score_ - nested < 0.005 -> sozlash xavfsiz
  ⭐ Optimizm MUSTAQIL nomzodlar soni va shovqin bilan
     o'sadi, ma'lumot hajmi bilan keskin kamayadi

Nima ko'rsatdi: 2.3-bo'lim.

Misol 4 — Narx va muqobillar

python
"""Nested CV, CV+test va arzon variantlar (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import (GridSearchCV, RandomizedSearchCV,
                                     StratifiedKFold, cross_val_score,
                                     train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC


def quvur():
    return make_pipeline(StandardScaler(), SVC(random_state=0))


SETKA = {"svc__C": [0.01, 0.1, 1, 10, 100],
         "svc__gamma": [0.001, 0.005, 0.01, 0.05, 0.1, 0.5]}


def main() -> None:
    X, y = make_classification(n_samples=21200, n_features=20,
                               n_informative=6, n_redundant=4, flip_y=0.2,
                               class_sep=0.8, random_state=0)
    X_ish, y_ish = X[:1200], y[:1200]
    X_haq, y_haq = X[1200:], y[1200:]
    nomzodlar = len(SETKA["svc__C"]) * len(SETKA["svc__gamma"])

    ichki = StratifiedKFold(5, shuffle=True, random_state=1)
    tashqi = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Sozlama ===")
    print(f"  mavjud {len(y_ish)} qator, nomzodlar {nomzodlar}")

    print("\n=== 2. Variantlar va ularning bahosi ===")
    natijalar = {}

    # A. oddiy CV (optimistik)
    g = GridSearchCV(quvur(), SETKA, cv=tashqi, scoring="roc_auc",
                     n_jobs=1).fit(X_ish, y_ish)
    natijalar["oddiy CV (best_score_)"] = (g.best_score_, 5 * nomzodlar)

    # B. nested 5x5
    q = GridSearchCV(quvur(), SETKA, cv=ichki, scoring="roc_auc", n_jobs=1)
    b = cross_val_score(q, X_ish, y_ish, cv=tashqi, scoring="roc_auc")
    natijalar["nested 5x5"] = (b.mean(), 5 * 5 * nomzodlar)

    # C. nested 3x5
    b3 = cross_val_score(q, X_ish, y_ish,
                         cv=StratifiedKFold(3, shuffle=True, random_state=0),
                         scoring="roc_auc")
    natijalar["nested 3x5"] = (b3.mean(), 3 * 5 * nomzodlar)

    # D. nested + RandomizedSearchCV (10 nomzod)
    r = RandomizedSearchCV(quvur(), SETKA, n_iter=10, cv=ichki,
                           scoring="roc_auc", random_state=0, n_jobs=1)
    br = cross_val_score(r, X_ish, y_ish, cv=tashqi, scoring="roc_auc")
    natijalar["nested + random(10)"] = (br.mean(), 5 * 5 * 10)

    # E. CV + alohida test
    Xa, Xb, ya, yb = train_test_split(X_ish, y_ish, test_size=0.3,
                                      stratify=y_ish, random_state=0)
    ga = GridSearchCV(quvur(), SETKA, cv=ichki, scoring="roc_auc",
                      n_jobs=1).fit(Xa, ya)
    test = roc_auc_score(yb, ga.best_estimator_.decision_function(Xb))
    natijalar["CV + alohida test"] = (test, 5 * nomzodlar)

    haqiqiy = roc_auc_score(y_haq,
                            g.best_estimator_.decision_function(X_haq))
    print(f"  {'usul':<24} {'baho':>8} {'haqiqatdan':>12} "
          f"{'model soni':>11}")
    for nom, (ball, narx) in natijalar.items():
        print(f"  {nom:<24} {ball:>8.4f} {ball - haqiqiy:>+12.4f} "
              f"{narx:>11}")
    print(f"  {'haqiqiy (20000 qator)':<24} {haqiqiy:>8.4f} "
          f"{0.0:>+12.4f} {'-':>11}")

    print("\n=== 3. Narx-sifat nisbati ===")
    asos = natijalar["nested 5x5"][1]
    print(f"  {'usul':<24} {'nisbiy narx':>12} {'xato':>8}")
    for nom, (ball, narx) in natijalar.items():
        print(f"  {nom:<24} {narx / asos:>11.2f}x "
              f"{abs(ball - haqiqiy):>8.4f}")

    print("\n=== 4. Tavsiya ===")
    print("  n < 2000 va ko'p nomzod -> nested CV")
    print("  n > 5000                -> CV + alohida test")
    print("  vaqt kam                -> nested 3x5 yoki random search")
    print("  ⭐ Nested CV raqam beradi, yakuniy model gridd an olinadi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Sozlama ===
  mavjud 1200 qator, nomzodlar 30

=== 2. Variantlar va ularning bahosi ===
  usul                         baho   haqiqatdan  model soni
  oddiy CV (best_score_)     0.8336      -0.0076         150
  nested 5x5                 0.8321      -0.0091         750
  nested 3x5                 0.8368      -0.0044         450
  nested + random(10)        0.8185      -0.0227         250
  CV + alohida test          0.8157      -0.0255         150
  haqiqiy (20000 qator)      0.8412      +0.0000           -

=== 3. Narx-sifat nisbati ===
  usul                      nisbiy narx     xato
  oddiy CV (best_score_)          0.20x   0.0076
  nested 5x5                      1.00x   0.0091
  nested 3x5                      0.60x   0.0044
  nested + random(10)             0.33x   0.0227
  CV + alohida test               0.20x   0.0255

=== 4. Tavsiya ===
  n < 2000 va ko'p nomzod -> nested CV
  n > 5000                -> CV + alohida test
  vaqt kam                -> nested 3x5 yoki random search
  ⭐ Nested CV raqam beradi, yakuniy model gridd an olinadi

Nima ko'rsatdi: 2.4, 2.5-bo'limlar.


5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"best_score_ — model natijasi" Tanlovning omadi
"Nested CV model beradi" Faqat baho beradi
"Foldlarda bir xil parametr chiqishi kerak" Turlicha bo'lishi normal
"Nested CV har doim kerak" Katta ma'lumotda test yetarli
"Optimizm har doim kichik" Kichik n da 0.02-0.10 bo'ladi
"Ko'p qiymatli setka = ko'p nomzod" Zich setka samarali T ni oshirmaydi
"Nested CV noaniqlikni yo'qotadi" Faqat optimizmni
"Ichki CV ni 2 fold qilsa bo'ladi" Tanlov shovqinli bo'ladi
"Tayyorlashni tashqarida qilsa bo'ladi" Leakage

6. Keng tarqalgan xatolar va yechimlari

1. best_score_ ni e'lon qilish

python
print(f"Model AUC: {grid.best_score_:.3f}")                      # ⚠️
nested = cross_val_score(grid, X, y, cv=tashqi)                  # ✅

2. Tayyorlashni tashqarida qilish

python
X = StandardScaler().fit_transform(X)                            # ⚠️
quvur = make_pipeline(StandardScaler(), SVC())                   # ✅

3. Ichki va tashqi halqada bir xil obyekt

python
cv = StratifiedKFold(5, shuffle=True, random_state=0)
GridSearchCV(m, g, cv=cv); cross_val_score(..., cv=cv)           # ⚠️
ichki = StratifiedKFold(5, shuffle=True, random_state=1)         # ✅

4. Nested CV dan model kutish

python
model = cross_val_score(grid, X, y, cv=tashqi)      # bu massiv  # ⚠️
model = grid.fit(X, y).best_estimator_                           # ✅

5. Narxni hisobga olmaslik

python
GridSearchCV(rf, {...500 nomzod...}, cv=10)  # nested 10x10      # ⚠️
RandomizedSearchCV(rf, taqsimot, n_iter=20, cv=5)                # ✅

6. Belgi tanlashni tashqarida qilish

python
X = SelectKBest(k=20).fit_transform(X, y)                        # ⚠️
Pipeline([("s", SelectKBest(k=20)), ("m", model)])               # ✅

7. Ichki foldni juda kichik qilish

python
GridSearchCV(m, g, cv=2)                                         # ⚠️
GridSearchCV(m, g, cv=StratifiedKFold(5, shuffle=True,
                                      random_state=1))           # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 18.1-dars (o'tilgan): G'olib la'nati
  • 18.3-dars (o'tilgan): CV dispersiyasi
  • 18.6-dars: Grid va random search
  • 18.7-dars: Halving va erta to'xtash
  • 18.11-dars: Validatsiyaga overfitting

8. Eng yaxshi amaliyotlar

  1. best_score_ ni hisobotga yozmang.

  2. Barcha tayyorlash Pipeline ichida.

  3. Ichki va tashqi random_state boshqa bo'lsin.

  4. Optimizmni o'lchang va yozing.

  5. Ma'lumot katta bo'lsa — CV + test.

  6. Narxni oldindan hisoblang.

  7. Yakuniy modelni butun ma'lumotda quring.

  8. Tashqi foldlardagi parametrlarni ko'rsating.


9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # best_score_ nima?
2.  # nima uchun optimistik?
3.  # nested CV da ichki halqa nima qiladi?
4.  # tashqi halqa nima qiladi?
5.  # nested CV narxi?
6.  # optimizm nimaga bog'liq?
7.  # foldlarda turli parametr - xatomi?
8.  # nested CV model beradimi?
9.  # yakuniy model qayerdan?
10. # arzon muqobil?
11. # nested qachon kerak?
12. # diagnostika qoidasi?
Javoblar
  1. Eng yaxshi nomzodning CV balli
  2. Tanlov shu CV da qilingan
  3. Giperparametr tanlaydi
  4. Xolis baho beradi
  5. k_tashqi * k_ichki * nomzodlar
  6. Nomzod soni, n, signal kuchi
  7. Yo'q, normal
  8. Yo'q, faqat baho
  9. Butun ma'lumotda GridSearchCV
  10. CV + alohida test
  11. Kichik n, ko'p nomzod
  12. best_score_ - nested > 0.02 — xavf

Vazifa 2: Xatolarni tuzating

python
1.  print(f"Model AUC: {grid.best_score_:.3f}")

2.  X = StandardScaler().fit_transform(X)

3.  model = cross_val_score(grid, X, y, cv=tashqi)

4.  X = SelectKBest(k=20).fit_transform(X, y)

5.  GridSearchCV(m, g, cv=2)
Javoblar
python
1.  nested = cross_val_score(grid, X, y, cv=tashqi)

2.  quvur = make_pipeline(StandardScaler(), SVC())

3.  model = grid.fit(X, y).best_estimator_

4.  Pipeline([("s", SelectKBest(k=20)), ("m", model)])

5.  GridSearchCV(m, g, cv=StratifiedKFold(5, shuffle=True, random_state=1))

Vazifa 3: Optimizm

Modellang:

  1. Sozlama
  2. GridSearchCV
  3. Haqiqiy natija
  4. Barcha nomzodlar

Vazifa 4: Nested

Modellang:

  1. Oddiy CV
  2. Nested CV
  3. Fold parametrlari
  4. Uch raqam

Vazifa 5: Bog'liqlik

Modellang:

  1. Nomzod soni
  2. Ma'lumot hajmi
  3. Haqiqat
  4. Diagnostika

Vazifa 6: Narx

Modellang:

  1. Sozlama
  2. Variantlar
  3. Narx-sifat
  4. Tavsiya

Vazifa 7: O'ylash

Sizda 150 ta bemorning ma'lumoti bor va 600 ta giperparametr kombinatsiyasi sinalgan. best_score_ = 0.91. Hamkasbingiz "ajoyib, maqola yozamiz" deydi. Siz nima deysiz?

Javob

Qisqa javob: 0.91 ga ishonmang. n = 150 va 600 nomzod — bu optimizm uchun eng yomon kombinatsiya. Nested CV bilan qayta baholash kerak, va natija 0.10-0.20 ga pastroq chiqishi ehtimoli yuqori.

1. Nima uchun xavfli

Omil Qiymat Ta'siri
n 150 CV bahosining SE si ~0.04
Nomzodlar 600 sqrt(2 ln 600) ≈ 3.6
Kutilgan optimizm — ~0.04 × 3.6 ≈ 0.14

Ya'ni 0.91 ning ortida haqiqiy qiymat 0.77 atrofida bo'lishi mumkin.

2. Qo'shimcha xavflar

  1. Bemor guruhi: agar bir bemordan bir necha namuna bo'lsa — GroupKFold kerak, aks holda yana leakage.
  2. Sinf muvozanati: 150 bemorda musbat sinf 20 ta bo'lsa, har fold 4 ta musbat — baho juda shovqinli.
  3. Belgi tanlash: agar belgilar butun ma'lumotda tanlangan bo'lsa, bu alohida va kuchliroq leakage.

3. Nima qilish kerak

python
ichki = StratifiedKFold(5, shuffle=True, random_state=1)
tashqi = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0)

quvur = Pipeline([("tayyor", ...), ("tanlov", ...), ("model", ...)])
qidiruv = GridSearchCV(quvur, setka, cv=ichki, scoring="roc_auc")

nested = cross_val_score(qidiruv, X, y, cv=tashqi, scoring="roc_auc")
print(f"{nested.mean():.3f} (+-{nested.std():.3f})")

4. Qidiruv maydonini qisqartirish

600 nomzod n = 150 uchun juda ko'p. Amaliy qoida: nomzodlar soni n dan ancha kichik bo'lsin. Bu yerda 10-30 nomzod yetarli — RandomizedSearchCV(n_iter=20) yoki domenga asoslangan tor maydon.

5. Maqolada nima yozish kerak

"5×10 takroriy nested CV (ichki 5-fold GridSearchCV, 20 nomzod): AUC 0.78 (takrorlar bo'yicha std 0.05). Sozlash optimizmi: best_score_ 0.91, nested 0.78, farq 0.13. Namuna hajmi kichik bo'lgani uchun natijani mustaqil kogortada tasdiqlash zarur."

6. Xulosa

  1. best_score_ ni tashlang
  2. Nested CV (takroriy) bilan qayta baholang
  3. Nomzodlar sonini 20-30 ga tushiring
  4. Guruh tuzilmasini tekshiring
  5. Mustaqil kogortani rejalashtiring

Nimani mustahkamlaydi: 2.1, 2.3-bo'limlar.


Xulosa

Bu darsda nested cross-validation ni o'rgandik.

Eng muhim uch fikr:

  1. GridSearchCV.best_score_ — modelning natijasi emas. U eng yaxshi nomzodning o'sha CV dagi balli, ya'ni tanlov qilingan to'plamdagi ball. Mustaqil nomzodlar soni ortgani va namuna kichraygani sari bu raqam tizimli ravishda haqiqatdan uzoqlashadi: n = 150 da o'lchovlarimizda optimizm +0.019 chiqdi va sakkizta mustaqil to'plamning hammasida musbat bo'ldi.

  2. Nested CV tanlov va bahoni ajratadi. Ichki halqa giperparametrni tanlaydi, tashqi halqa esa shu tanlov protsedurasini baholaydi va tashqi test qismi hech qachon tanlovda qatnashmaydi. Har tashqi foldda boshqa parametr chiqishi normal — nested CV modelni emas, protsedurani baholaydi.

  3. Optimizmni o'lchang va arzon muqobilni biling. best_score_ - nested farqi diagnostika: 0.02 dan katta bo'lsa siz shovqinga moslashayapsiz. Ma'lumot yetarli (n > 5000) bo'lsa CV + alohida test nested CV ning arzon muqobili. Yakuniy modelni esa butun ma'lumotda oddiy GridSearchCV bilan quring — baho nested dan, model qidiruvdan.

Keyingi darsda giperparametrlarning o'ziga o'tamiz: qaysi model qaysi parametrga sezgir, qidiruv maydonini qanday belgilash va qaysi parametrlarni umuman sozlamaslik kerak.

Model baholash sozlash · 4-dars yakunlandi

18.4-dars: Nested cross-validation

21 daqvaqt
0belgi
Keyingi dars: 18.5-dars: Giperparametrlar
Telegram'da
Darsda xato koʻrdingizmi?

Izohlar · 0