18.4-dars: Nested cross-validation
18-QISM — MODEL BAHOLASH VA SOZLASH · 4-dars
1. Kirish va motivatsiya
Eng ko'p uchraydigan baholash xatosi shunday ko'rinadi:
grid = GridSearchCV(model, param_grid, cv=5, scoring="roc_auc")
grid.fit(X, y)
print(f"Model natijasi: {grid.best_score_:.3f}") # ⚠️ OPTIMISTIKbest_score_ — bu eng yaxshi giperparametrning CV balli. Lekin "eng yaxshi" degan so'z o'sha CV ga qarab tanlangan, ya'ni bu ball g'olib la'nati ta'siri ostida. U modelning yangi ma'lumotdagi natijasi emas, tanlovning o'zi qanchalik omadli bo'lganining o'lchovi.
Nested CV (ichma-ich CV) shu muammoni hal qiladi: ichki halqa giperparametrni tanlaydi, tashqi halqa esa "shunday tanlov qiladigan protsedura" ni baholaydi. Natija — xolis baho.
Bu darsda: optimizm qanday paydo bo'ladi va qanchalik katta, nested CV qurilishi, optimizm miqdorining nomzodlar soni va ma'lumot hajmiga bog'liqligi, narxi va qachon kerak emasligi.
Real vaziyat. Bioinformatika maqolalarining ko'pchiligida kichik namunada (n = 60-200) yuzlab giperparametr sinaladi va best_score_ e'lon qilinadi. Bir tekshiruvda 12 ta maqola nested CV bilan qayta baholandi: o'rtacha 0.08 AUC tushish kuzatildi, uchtasida esa "model tasodifdan yaxshi" degan xulosa butunlay yo'qoldi.
Bu darsda nested CV ni o'rganamiz.
Bu darsda:
- Sozlash optimizmi
- Nested CV qurilishi
- Optimizm nimaga bog'liq
- Narx va muqobillar
- Nested CV nima bermaydi
- Qachon kerak emas
- Tuzoqlar
- Amaliy: xolis baho
ℹ Misollar real numpy/sklearn bilan (Python 3.14).
2. Nazariya — chuqur tushuntirish
2.1. Sozlash optimizmi
GridSearchCV(cv=5) 40 ta nomzodni baholadi.
Har birining CV balli: haqiqiy + shovqin.
best_score_ = max(40 ta ball)
= haqiqiy + ENG KATTA musbat shovqin
Ya'ni best_score_ SISTEMATIK ravishda yuqori.
QANCHA YUQORI?
n katta, nomzod kam -> 0.001-0.005
n o'rta, nomzod ko'p -> 0.01-0.03
n kichik, nomzod ko'p -> 0.05-0.15 best_score_ — model natijasi emas, u "shu CV da eng omadli nomzodning balli". Hisobotga yozish uchun yaroqsiz.
2.2. Nested CV qurilishi
TASHQI halqa (5 fold):
har tashqi foldda:
o'quv qismi -> ICHKI CV (masalan 5 fold) bilan
giperparametr tanlanadi
tanlangan parametr bilan model o'quv qismida o'rgatiladi
TASHQI test qismida baholanadi
tashqi 5 ta ball -> ularning o'rtachasi = XOLIS baho
MUHIM: tashqi test hech qachon tanlovda ishlatilmaydi
sklearn da:
ichki = GridSearchCV(model, grid, cv=ichki_cv)
ball = cross_val_score(ichki, X, y, cv=tashqi_cv)Nested CV protsedurani baholaydi, modelni emas: har tashqi foldda boshqa giperparametr tanlanishi mumkin — bu normal va kutilgan.
2.3. Optimizm nimaga bog'liq
OPTIMIZM O'SADI:
MUSTAQIL nomzodlar soni ko'paysa (g'olib la'nati)
ma'lumot kamaysa (shovqin katta)
signal zaiflashsa (farqlar shovqin ichida)
ichki fold soni kamaysa (ichki baho shovqinli)
"MUSTAQIL" SO'ZI MUHIM:
alpha uchun 80 ta zich qiymat - nomzodlar bir-biriga
juda o'xshash, samarali T atigi 2-3 ga teng
turli MODEL oilalari yoki turli BELGI to'plamlari esa
haqiqatan mustaqil - T tez o'sadi
OPTIMIZM KICHIK:
n > 10 000 va nomzodlar < 50
giperparametrlar natijaga kam ta'sir qilsa
qidiruv maydoni tor bo'lsa
TEKSHIRISH: best_score_ va nested ball farqi Farqning o'zi diagnostika: best_score_ - nested_ball katta bo'lsa, siz shovqinga moslashayotgansiz.
2.4. Narx va muqobillar
NARX:
oddiy CV + grid: k_ichki * nomzodlar
nested CV: k_tashqi * k_ichki * nomzodlar
5x5 va 40 nomzod -> 1000 ta model o'rgatish
MUQOBILLAR (arzonroq):
1. CV + ALOHIDA TEST
ma'lumot yetarli bo'lsa eng amaliy yechim
2. tashqi halqani kamaytirish (3 fold)
3. ichki halqada RandomizedSearchCV (kam nomzod)
4. HalvingGridSearchCV (18.7-dars)Ma'lumot yetarli bo'lsa, CV + alohida test nested CV ning arzon va tushunarli muqobili.
2.5. Nested CV nima bermaydi
BERMAYDI:
- yakuniy modelni (u faqat BAHO beradi)
- bitta "eng yaxshi" giperparametr to'plamini
- ma'lumot namunasidan kelgan noaniqlikni yo'qotishni
BERADI:
- "shunday sozlash protsedurasi qanday natija beradi"
degan savolga xolis javob
YAKUNIY MODEL:
nested CV dan KEYIN butun ma'lumotda
oddiy GridSearchCV ishlatib model quriladi;
uning bahosi esa nested CV dan olinadi Ikki bosqich: nested CV raqam beradi, oddiy GridSearchCV model beradi.
2.6. Qachon kerak emas
KERAK EMAS:
- ma'lumot katta va alohida test bor
- giperparametr sozlanmayapti (qat'iy model)
- faqat modellarni taqqoslayapsiz, baho kerak emas
- qidiruv maydoni juda tor (2-3 nomzod)
KERAK:
- kichik ma'lumot (n < 2000)
- ko'p nomzod (> 50)
- ilmiy nashr, klinik qaror, audit
- "natijamiz ishonchlimi?" degan savol borKichik ma'lumot + ko'p nomzod — nested CV majburiy bo'ladigan asosiy holat.
2.7. Tuzoqlar
Asosiy tuzoqlar: best_score_ ni hisobotga yozish; nested CV da tashqi foldlarda turli parametr chiqqanini "xato" deb o'ylash; ichki va tashqi halqada bir xil random_state bilan tasodifan bog'liqlik yaratish; belgi tanlash yoki masshtablashni nested CV dan tashqarida qilish; nested CV dan model kutish; narxini hisobga olmay 5×5×500 nomzod qo'yish.
2.8. Tanlov va baho — ikki alohida ish
Giperparametr tanlash va model baholash — ikki alohida ish, va ularni bir xil ma'lumotda bajarish bahoni buzadi. Nested CV ularni ajratadi: ichki halqa tanlaydi, tashqi halqa baholaydi. Optimizm nomzodlar soni bilan o'sadi va ma'lumot hajmi bilan kamayadi. Ma'lumot yetarli bo'lsa, arzonroq muqobil — CV + alohida test. Nested CV modelni emas, raqamni beradi.
3. Tez ma'lumotnoma
from sklearn.model_selection import (GridSearchCV, StratifiedKFold,
cross_val_score)
ichki_cv = StratifiedKFold(5, shuffle=True, random_state=1)
tashqi_cv = StratifiedKFold(5, shuffle=True, random_state=0)
qidiruv = GridSearchCV(quvur, param_grid, cv=ichki_cv, scoring="roc_auc",
n_jobs=1)
# 1. XOLIS baho
nested = cross_val_score(qidiruv, X, y, cv=tashqi_cv, scoring="roc_auc")
print(f"nested: {nested.mean():.3f} (+-{nested.std():.3f})")
# 2. YAKUNIY model (butun ma'lumotda)
qidiruv.fit(X, y)
model = qidiruv.best_estimator_
print(f"optimizm: {qidiruv.best_score_ - nested.mean():+.3f}")
QOIDA: best_score_ ni hisobotga yozma · barcha tayyorlash
Pipeline ichida · baho nested dan, model gridd anNested CV xulosasi
best_score_ - optimistik (tanlov shu CV da qilingan)
Nested CV: ichki tanlaydi, tashqi baholaydi
Narx: k_tashqi * k_ichki * nomzodlar
Muqobil: CV + alohida test (ma'lumot yetarli bo'lsa)4. Batafsil misollar
Misollar real numpy/sklearn bilan (Python 3.14).
Misol 1 — Sozlash optimizmi
"""best_score_ qanchalik optimistik (real numpy/sklearn).
Bitta ma'lumot to'plamida CV shovqini (~0.02) tanlov optimizmidan
(~0.01) katta bo'ladi, shuning uchun effektni KO'P MUSTAQIL
to'plamda o'rtachalab o'lchaymiz.
"""
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
def quvur():
return make_pipeline(StandardScaler(),
LogisticRegression(max_iter=3000,
solver="liblinear",
random_state=0))
SETKA = {"logisticregression__C": np.logspace(-3, 2, 25).tolist()}
N = 300 # bitta ma'lumot to'plamining hajmi
R = 25 # mustaqil takrorlar soni
def main() -> None:
X_pool, y_pool = make_classification(n_samples=28000, n_features=25,
n_informative=7, n_redundant=6,
flip_y=0.25, class_sep=0.7,
random_state=0)
X_haq, y_haq = X_pool[20000:], y_pool[20000:] # 'haqiqat'
X_bank, y_bank = X_pool[:20000], y_pool[:20000] # namuna olinadi
rng = np.random.default_rng(0)
print("=== 1. Sozlama ===")
print(f" har takrorda {N} qatorli mustaqil to'plam, {R} ta takror")
print(f" nomzodlar: {len(SETKA['logisticregression__C'])}")
print(f" 'haqiqat' to'plami: {len(y_haq)} qator")
eng_optimizm, ortacha_siljish, tanlov_yoqotishi = [], [], []
for _ in range(R):
idx = rng.choice(len(y_bank), N, replace=False)
X_ish, y_ish = X_bank[idx], y_bank[idx]
cv = StratifiedKFold(5, shuffle=True, random_state=0)
foldlar = list(cv.split(X_ish, y_ish))
grid = GridSearchCV(quvur(), SETKA, cv=cv, scoring="roc_auc",
n_jobs=1).fit(X_ish, y_ish)
cv_ballar, haq_ballar = [], []
for i, parametrlar in enumerate(grid.cv_results_["params"]):
cv_ballar.append(float(grid.cv_results_["mean_test_score"][i]))
ballar = []
for tr, _ in foldlar:
m = quvur().set_params(**parametrlar).fit(X_ish[tr],
y_ish[tr])
ballar.append(roc_auc_score(
y_haq, m.decision_function(X_haq)))
haq_ballar.append(float(np.mean(ballar)))
cv_ballar = np.array(cv_ballar)
haq_ballar = np.array(haq_ballar)
farq = cv_ballar - haq_ballar
eng = int(np.argmax(cv_ballar))
eng_optimizm.append(farq[eng])
ortacha_siljish.append(farq.mean())
tanlov_yoqotishi.append(haq_ballar.max() - haq_ballar[eng])
eng_optimizm = np.array(eng_optimizm)
ortacha_siljish = np.array(ortacha_siljish)
tanlov_yoqotishi = np.array(tanlov_yoqotishi)
print("\n=== 2. Tasodifiy nomzod uchun CV xolismi ===")
print(f" barcha nomzodlar bo'yicha o'rtacha farq "
f"(CV - haqiqiy): {ortacha_siljish.mean():+.4f}")
se = float(ortacha_siljish.std(ddof=1) / np.sqrt(R))
print(f" uning standart xatosi: {se:.4f}")
print(f" 95% oraliq: [{ortacha_siljish.mean() - 2 * se:+.4f}, "
f"{ortacha_siljish.mean() + 2 * se:+.4f}]")
print(" nolni qamrab oladi -> CV tasodifiy nomzod uchun XOLIS")
print("\n=== 3. TANLANGAN nomzod uchun ===")
print(f" best_score_ va haqiqiy farqi: {eng_optimizm.mean():+.4f}")
se2 = float(eng_optimizm.std(ddof=1) / np.sqrt(R))
print(f" standart xato: {se2:.4f}")
print(f" 95% oraliq: [{eng_optimizm.mean() - 2 * se2:+.4f}, "
f"{eng_optimizm.mean() + 2 * se2:+.4f}]")
ortiqcha = eng_optimizm - ortacha_siljish
se3 = float(ortiqcha.std(ddof=1) / np.sqrt(R))
print(f" ORTIQCHA optimizm (tanlovdan): {ortiqcha.mean():+.4f} "
f"(+-{2 * se3:.4f})")
print("\n=== 4. Tanlov qanchalik to'g'ri ===")
print(f" CV g'olibining haqiqiy yo'qotishi: "
f"{tanlov_yoqotishi.mean():.4f}")
print(f" takrorlarning {int((tanlov_yoqotishi < 0.002).sum())}/{R} "
f"tasida CV eng yaxshisini (yoki unga juda yaqinini) tanladi")
print(f" {'takror':>8} {'best_score_ optimizmi':>23}")
for i in range(5):
print(f" {i + 1:>8} {eng_optimizm[i]:>+23.4f}")
print(" ⭐ best_score_ - tanlovning omadi, modelning natijasi emas")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Sozlama ===
har takrorda 300 qatorli mustaqil to'plam, 25 ta takror
nomzodlar: 25
'haqiqat' to'plami: 8000 qator
=== 2. Tasodifiy nomzod uchun CV xolismi ===
barcha nomzodlar bo'yicha o'rtacha farq (CV - haqiqiy): +0.0036
uning standart xatosi: 0.0072
95% oraliq: [-0.0107, +0.0180]
nolni qamrab oladi -> CV tasodifiy nomzod uchun XOLIS
=== 3. TANLANGAN nomzod uchun ===
best_score_ va haqiqiy farqi: +0.0088
standart xato: 0.0076
95% oraliq: [-0.0063, +0.0239]
ORTIQCHA optimizm (tanlovdan): +0.0052 (+-0.0043)
=== 4. Tanlov qanchalik to'g'ri ===
CV g'olibining haqiqiy yo'qotishi: 0.0038
takrorlarning 17/25 tasida CV eng yaxshisini (yoki unga juda yaqinini) tanladi
takror best_score_ optimizmi
1 -0.0217
2 -0.0384
3 +0.0414
4 +0.0469
5 +0.0619
⭐ best_score_ - tanlovning omadi, modelning natijasi emasNima ko'rsatdi: 2.1-bo'lim.
Misol 2 — Nested CV qurish
"""Ichki tanlaydi, tashqi baholaydi (real numpy/sklearn)."""
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import (GridSearchCV, StratifiedKFold,
cross_val_score, cross_validate)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
def quvur():
return make_pipeline(StandardScaler(), SVC(random_state=0))
SETKA = {"svc__C": [0.01, 0.1, 1, 10, 100],
"svc__gamma": [0.001, 0.01, 0.05, 0.1, 0.5]}
def main() -> None:
X, y = make_classification(n_samples=20400, n_features=20,
n_informative=6, n_redundant=4, flip_y=0.2,
class_sep=0.8, random_state=0)
X_ish, y_ish = X[:400], y[:400]
ichki = StratifiedKFold(5, shuffle=True, random_state=1)
tashqi = StratifiedKFold(5, shuffle=True, random_state=0)
qidiruv = GridSearchCV(quvur(), SETKA, cv=ichki, scoring="roc_auc",
n_jobs=1)
print("=== 1. Oddiy CV (optimistik) ===")
grid = GridSearchCV(quvur(), SETKA, cv=tashqi, scoring="roc_auc",
n_jobs=1).fit(X_ish, y_ish)
print(f" eng yaxshi parametrlar: {grid.best_params_}")
print(f" best_score_: {grid.best_score_:.4f}")
print("\n=== 2. Nested CV ===")
nested = cross_val_score(qidiruv, X_ish, y_ish, cv=tashqi,
scoring="roc_auc")
print(f" tashqi fold ballari: {np.round(nested, 4).tolist()}")
print(f" nested baho: {nested.mean():.4f} (+-{nested.std():.4f})")
print("\n=== 3. Har tashqi foldda qanday parametr tanlandi ===")
natija = cross_validate(qidiruv, X_ish, y_ish, cv=tashqi,
scoring="roc_auc", return_estimator=True)
print(f" {'fold':>5} {'C':>7} {'gamma':>8} {'tashqi ball':>13}")
for i, (est, ball) in enumerate(zip(natija["estimator"],
natija["test_score"]), 1):
p = est.best_params_
print(f" {i:>5} {p['svc__C']:>7} {p['svc__gamma']:>8} "
f"{ball:>13.4f}")
print(" turli foldda turli parametr - bu NORMAL")
print("\n=== 4. Amaliy diagnostika: best_score_ va nested ===")
# Ishlab chiqarishda "haqiqat" to'plami bo'lmaydi, shuning uchun
# optimizmni SHU IKKI RAQAM FARQI bilan baholaymiz.
farq = grid.best_score_ - nested.mean()
print(f" {'manba':<32} {'qiymat':>9}")
print(f" {'GridSearchCV.best_score_':<32} {grid.best_score_:>9.4f}")
print(f" {'Nested CV':<32} {nested.mean():>9.4f}")
print(f" farq (optimizm bahosi): {farq:+.4f}")
print(f" tashqi foldlar std i: {nested.std(ddof=1):.4f}")
baho = ("shovqin ichida" if abs(farq) < nested.std(ddof=1)
else "sezilarli")
print(f" baho: {baho}")
print(" DIQQAT: n=400 da tashqi foldlar juda shovqinli")
print(" (har foldda atigi 80 qator) - shuning uchun bu farq")
print(" bitta to'plamda ishonchli emas. 3-misolda ko'ramizki,")
print(" nomzodlar soni ortganda farq barqaror ravishda o'sadi.")
print(" ⭐ best_score_ - nested farqi optimizmning amaliy o'lchovi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Oddiy CV (optimistik) ===
eng yaxshi parametrlar: {'svc__C': 1, 'svc__gamma': 0.05}
best_score_: 0.8426
=== 2. Nested CV ===
tashqi fold ballari: [0.8168, 0.8949, 0.8774, 0.7419, 0.8656]
nested baho: 0.8393 (+-0.0552)
=== 3. Har tashqi foldda qanday parametr tanlandi ===
fold C gamma tashqi ball
1 100 0.001 0.8168
2 1 0.05 0.8949
3 1 0.01 0.8774
4 1 0.05 0.7419
5 10 0.01 0.8656
turli foldda turli parametr - bu NORMAL
=== 4. Amaliy diagnostika: best_score_ va nested ===
manba qiymat
GridSearchCV.best_score_ 0.8426
Nested CV 0.8393
farq (optimizm bahosi): +0.0033
tashqi foldlar std i: 0.0617
baho: shovqin ichida
DIQQAT: n=400 da tashqi foldlar juda shovqinli
(har foldda atigi 80 qator) - shuning uchun bu farq
bitta to'plamda ishonchli emas. 3-misolda ko'ramizki,
nomzodlar soni ortganda farq barqaror ravishda o'sadi.
⭐ best_score_ - nested farqi optimizmning amaliy o'lchoviNima ko'rsatdi: 2.2-bo'lim.
Misol 3 — Optimizm nimaga bog'liq
"""Nomzodlar soni va ma'lumot hajmi ta'siri (real numpy/sklearn)."""
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (GridSearchCV, StratifiedKFold,
cross_val_score)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
def quvur():
return make_pipeline(StandardScaler(),
LogisticRegression(max_iter=3000,
solver="liblinear",
random_state=0))
def setka(nechta: int) -> dict:
return {"logisticregression__C": np.logspace(-3, 2, nechta).tolist()}
def main() -> None:
X, y = make_classification(n_samples=30500, n_features=30,
n_informative=7, n_redundant=6, flip_y=0.25,
class_sep=0.7, random_state=0)
ichki = StratifiedKFold(5, shuffle=True, random_state=1)
tashqi = StratifiedKFold(5, shuffle=True, random_state=0)
print("=== 1. Nomzodlar soni ta'siri (n=150) ===")
# kichik namunada CV shovqini katta, shuning uchun g'olib
# la'nati aniq ko'rinadi
X_ish, y_ish = X[:150], y[:150]
print(f" {'nomzod':>7} {'best_score_':>13} {'nested':>9} "
f"{'optimizm':>10}")
for nechta in [2, 5, 15, 40, 80]:
g = GridSearchCV(quvur(), setka(nechta), cv=tashqi,
scoring="roc_auc", n_jobs=1).fit(X_ish, y_ish)
q = GridSearchCV(quvur(), setka(nechta), cv=ichki,
scoring="roc_auc", n_jobs=1)
nested = cross_val_score(q, X_ish, y_ish, cv=tashqi,
scoring="roc_auc").mean()
print(f" {nechta:>7} {g.best_score_:>13.4f} {nested:>9.4f} "
f"{g.best_score_ - nested:>+10.4f}")
print(" DIQQAT: optimizm nomzod soni bilan deyarli O'SMADI.")
print(" Sabab: zich bir o'lchovli C setkasidagi nomzodlar")
print(" bir-biriga juda bog'liq - 80 ta qiymat ham amalda")
print(" 2-3 ta MUSTAQIL nomzodga teng (samarali T kichik).")
print("\n=== 2. Ma'lumot hajmi ta'siri (15 nomzod) ===")
print(f" {'n':>7} {'best_score_':>13} {'nested':>9} {'optimizm':>10}")
for n in [150, 300, 1000, 4000]:
Xn, yn = X[:n], y[:n]
g = GridSearchCV(quvur(), setka(15), cv=tashqi, scoring="roc_auc",
n_jobs=1).fit(Xn, yn)
q = GridSearchCV(quvur(), setka(15), cv=ichki, scoring="roc_auc",
n_jobs=1)
nested = cross_val_score(q, Xn, yn, cv=tashqi,
scoring="roc_auc").mean()
print(f" {n:>7} {g.best_score_:>13.4f} {nested:>9.4f} "
f"{g.best_score_ - nested:>+10.4f}")
print("\n=== 3. Optimizm turli seedlarda barqarormi (n=150) ===")
# bitta to'plamdagi bitta o'lchov shovqinli, shuning uchun
# bir necha mustaqil to'plamda takrorlaymiz
optimizmlar = []
print(f" {'to_plam':>8} {'best_score_':>13} {'nested':>9} "
f"{'optimizm':>10}")
for boshi in range(0, 1200, 150):
Xn, yn = X[boshi:boshi + 150], y[boshi:boshi + 150]
g = GridSearchCV(quvur(), setka(40), cv=tashqi, scoring="roc_auc",
n_jobs=1).fit(Xn, yn)
q = GridSearchCV(quvur(), setka(40), cv=ichki, scoring="roc_auc",
n_jobs=1)
nested = cross_val_score(q, Xn, yn, cv=tashqi,
scoring="roc_auc").mean()
optimizmlar.append(g.best_score_ - nested)
if boshi < 600:
print(f" {boshi // 150 + 1:>8} {g.best_score_:>13.4f} "
f"{nested:>9.4f} {g.best_score_ - nested:>+10.4f}")
optimizmlar = np.array(optimizmlar)
se = float(optimizmlar.std(ddof=1) / np.sqrt(len(optimizmlar)))
print(f" 8 ta to'plam bo'yicha o'rtacha: {optimizmlar.mean():+.4f} "
f"(+-{2 * se:.4f})")
print(f" musbat chiqqan to'plamlar: "
f"{int((optimizmlar > 0).sum())}/{len(optimizmlar)}")
print("\n=== 4. Diagnostika qoidasi ===")
print(" best_score_ - nested > 0.02 -> shovqinga moslashuv")
print(" best_score_ - nested < 0.005 -> sozlash xavfsiz")
print(" ⭐ Optimizm MUSTAQIL nomzodlar soni va shovqin bilan")
print(" o'sadi, ma'lumot hajmi bilan keskin kamayadi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Nomzodlar soni ta'siri (n=150) ===
nomzod best_score_ nested optimizm
2 0.6058 0.5790 +0.0268
5 0.6058 0.5852 +0.0205
15 0.6084 0.5834 +0.0250
40 0.6102 0.5825 +0.0277
80 0.6111 0.5906 +0.0205
DIQQAT: optimizm nomzod soni bilan deyarli O'SMADI.
Sabab: zich bir o'lchovli C setkasidagi nomzodlar
bir-biriga juda bog'liq - 80 ta qiymat ham amalda
2-3 ta MUSTAQIL nomzodga teng (samarali T kichik).
=== 2. Ma'lumot hajmi ta'siri (15 nomzod) ===
n best_score_ nested optimizm
150 0.6084 0.5834 +0.0250
300 0.6482 0.6462 +0.0020
1000 0.6716 0.6714 +0.0002
4000 0.7136 0.7137 -0.0000
=== 3. Optimizm turli seedlarda barqarormi (n=150) ===
to_plam best_score_ nested optimizm
1 0.6102 0.5825 +0.0277
2 0.7139 0.7130 +0.0009
3 0.4715 0.4484 +0.0231
4 0.6653 0.6564 +0.0089
8 ta to'plam bo'yicha o'rtacha: +0.0189 (+-0.0086)
musbat chiqqan to'plamlar: 8/8
=== 4. Diagnostika qoidasi ===
best_score_ - nested > 0.02 -> shovqinga moslashuv
best_score_ - nested < 0.005 -> sozlash xavfsiz
⭐ Optimizm MUSTAQIL nomzodlar soni va shovqin bilan
o'sadi, ma'lumot hajmi bilan keskin kamayadiNima ko'rsatdi: 2.3-bo'lim.
Misol 4 — Narx va muqobillar
"""Nested CV, CV+test va arzon variantlar (real numpy/sklearn)."""
import numpy as np
from sklearn.datasets import make_classification
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import (GridSearchCV, RandomizedSearchCV,
StratifiedKFold, cross_val_score,
train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
def quvur():
return make_pipeline(StandardScaler(), SVC(random_state=0))
SETKA = {"svc__C": [0.01, 0.1, 1, 10, 100],
"svc__gamma": [0.001, 0.005, 0.01, 0.05, 0.1, 0.5]}
def main() -> None:
X, y = make_classification(n_samples=21200, n_features=20,
n_informative=6, n_redundant=4, flip_y=0.2,
class_sep=0.8, random_state=0)
X_ish, y_ish = X[:1200], y[:1200]
X_haq, y_haq = X[1200:], y[1200:]
nomzodlar = len(SETKA["svc__C"]) * len(SETKA["svc__gamma"])
ichki = StratifiedKFold(5, shuffle=True, random_state=1)
tashqi = StratifiedKFold(5, shuffle=True, random_state=0)
print("=== 1. Sozlama ===")
print(f" mavjud {len(y_ish)} qator, nomzodlar {nomzodlar}")
print("\n=== 2. Variantlar va ularning bahosi ===")
natijalar = {}
# A. oddiy CV (optimistik)
g = GridSearchCV(quvur(), SETKA, cv=tashqi, scoring="roc_auc",
n_jobs=1).fit(X_ish, y_ish)
natijalar["oddiy CV (best_score_)"] = (g.best_score_, 5 * nomzodlar)
# B. nested 5x5
q = GridSearchCV(quvur(), SETKA, cv=ichki, scoring="roc_auc", n_jobs=1)
b = cross_val_score(q, X_ish, y_ish, cv=tashqi, scoring="roc_auc")
natijalar["nested 5x5"] = (b.mean(), 5 * 5 * nomzodlar)
# C. nested 3x5
b3 = cross_val_score(q, X_ish, y_ish,
cv=StratifiedKFold(3, shuffle=True, random_state=0),
scoring="roc_auc")
natijalar["nested 3x5"] = (b3.mean(), 3 * 5 * nomzodlar)
# D. nested + RandomizedSearchCV (10 nomzod)
r = RandomizedSearchCV(quvur(), SETKA, n_iter=10, cv=ichki,
scoring="roc_auc", random_state=0, n_jobs=1)
br = cross_val_score(r, X_ish, y_ish, cv=tashqi, scoring="roc_auc")
natijalar["nested + random(10)"] = (br.mean(), 5 * 5 * 10)
# E. CV + alohida test
Xa, Xb, ya, yb = train_test_split(X_ish, y_ish, test_size=0.3,
stratify=y_ish, random_state=0)
ga = GridSearchCV(quvur(), SETKA, cv=ichki, scoring="roc_auc",
n_jobs=1).fit(Xa, ya)
test = roc_auc_score(yb, ga.best_estimator_.decision_function(Xb))
natijalar["CV + alohida test"] = (test, 5 * nomzodlar)
haqiqiy = roc_auc_score(y_haq,
g.best_estimator_.decision_function(X_haq))
print(f" {'usul':<24} {'baho':>8} {'haqiqatdan':>12} "
f"{'model soni':>11}")
for nom, (ball, narx) in natijalar.items():
print(f" {nom:<24} {ball:>8.4f} {ball - haqiqiy:>+12.4f} "
f"{narx:>11}")
print(f" {'haqiqiy (20000 qator)':<24} {haqiqiy:>8.4f} "
f"{0.0:>+12.4f} {'-':>11}")
print("\n=== 3. Narx-sifat nisbati ===")
asos = natijalar["nested 5x5"][1]
print(f" {'usul':<24} {'nisbiy narx':>12} {'xato':>8}")
for nom, (ball, narx) in natijalar.items():
print(f" {nom:<24} {narx / asos:>11.2f}x "
f"{abs(ball - haqiqiy):>8.4f}")
print("\n=== 4. Tavsiya ===")
print(" n < 2000 va ko'p nomzod -> nested CV")
print(" n > 5000 -> CV + alohida test")
print(" vaqt kam -> nested 3x5 yoki random search")
print(" ⭐ Nested CV raqam beradi, yakuniy model gridd an olinadi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Sozlama ===
mavjud 1200 qator, nomzodlar 30
=== 2. Variantlar va ularning bahosi ===
usul baho haqiqatdan model soni
oddiy CV (best_score_) 0.8336 -0.0076 150
nested 5x5 0.8321 -0.0091 750
nested 3x5 0.8368 -0.0044 450
nested + random(10) 0.8185 -0.0227 250
CV + alohida test 0.8157 -0.0255 150
haqiqiy (20000 qator) 0.8412 +0.0000 -
=== 3. Narx-sifat nisbati ===
usul nisbiy narx xato
oddiy CV (best_score_) 0.20x 0.0076
nested 5x5 1.00x 0.0091
nested 3x5 0.60x 0.0044
nested + random(10) 0.33x 0.0227
CV + alohida test 0.20x 0.0255
=== 4. Tavsiya ===
n < 2000 va ko'p nomzod -> nested CV
n > 5000 -> CV + alohida test
vaqt kam -> nested 3x5 yoki random search
⭐ Nested CV raqam beradi, yakuniy model gridd an olinadiNima ko'rsatdi: 2.4, 2.5-bo'limlar.
5. To'g'ri va noto'g'ri tushunishlar
| Noto'g'ri fikr | To'g'risi |
|---|---|
"best_score_ — model natijasi" |
Tanlovning omadi |
| "Nested CV model beradi" | Faqat baho beradi |
| "Foldlarda bir xil parametr chiqishi kerak" | Turlicha bo'lishi normal |
| "Nested CV har doim kerak" | Katta ma'lumotda test yetarli |
| "Optimizm har doim kichik" | Kichik n da 0.02-0.10 bo'ladi |
| "Ko'p qiymatli setka = ko'p nomzod" | Zich setka samarali T ni oshirmaydi |
| "Nested CV noaniqlikni yo'qotadi" | Faqat optimizmni |
| "Ichki CV ni 2 fold qilsa bo'ladi" | Tanlov shovqinli bo'ladi |
| "Tayyorlashni tashqarida qilsa bo'ladi" | Leakage |
6. Keng tarqalgan xatolar va yechimlari
1. best_score_ ni e'lon qilish
print(f"Model AUC: {grid.best_score_:.3f}") # ⚠️
nested = cross_val_score(grid, X, y, cv=tashqi) # ✅2. Tayyorlashni tashqarida qilish
X = StandardScaler().fit_transform(X) # ⚠️
quvur = make_pipeline(StandardScaler(), SVC()) # ✅3. Ichki va tashqi halqada bir xil obyekt
cv = StratifiedKFold(5, shuffle=True, random_state=0)
GridSearchCV(m, g, cv=cv); cross_val_score(..., cv=cv) # ⚠️
ichki = StratifiedKFold(5, shuffle=True, random_state=1) # ✅4. Nested CV dan model kutish
model = cross_val_score(grid, X, y, cv=tashqi) # bu massiv # ⚠️
model = grid.fit(X, y).best_estimator_ # ✅5. Narxni hisobga olmaslik
GridSearchCV(rf, {...500 nomzod...}, cv=10) # nested 10x10 # ⚠️
RandomizedSearchCV(rf, taqsimot, n_iter=20, cv=5) # ✅6. Belgi tanlashni tashqarida qilish
X = SelectKBest(k=20).fit_transform(X, y) # ⚠️
Pipeline([("s", SelectKBest(k=20)), ("m", model)]) # ✅7. Ichki foldni juda kichik qilish
GridSearchCV(m, g, cv=2) # ⚠️
GridSearchCV(m, g, cv=StratifiedKFold(5, shuffle=True,
random_state=1)) # ✅7. Integratsiya — bu bilim qayerda kerak bo'ladi
- 18.1-dars (o'tilgan): G'olib la'nati
- 18.3-dars (o'tilgan): CV dispersiyasi
- 18.6-dars: Grid va random search
- 18.7-dars: Halving va erta to'xtash
- 18.11-dars: Validatsiyaga overfitting
8. Eng yaxshi amaliyotlar
best_score_ni hisobotga yozmang.Barcha tayyorlash
Pipelineichida.Ichki va tashqi
random_stateboshqa bo'lsin.Optimizmni o'lchang va yozing.
Ma'lumot katta bo'lsa — CV + test.
Narxni oldindan hisoblang.
Yakuniy modelni butun ma'lumotda quring.
Tashqi foldlardagi parametrlarni ko'rsating.
9. Amaliy topshiriq
Vazifa 1: Bashorat qiling
1. # best_score_ nima?
2. # nima uchun optimistik?
3. # nested CV da ichki halqa nima qiladi?
4. # tashqi halqa nima qiladi?
5. # nested CV narxi?
6. # optimizm nimaga bog'liq?
7. # foldlarda turli parametr - xatomi?
8. # nested CV model beradimi?
9. # yakuniy model qayerdan?
10. # arzon muqobil?
11. # nested qachon kerak?
12. # diagnostika qoidasi?Javoblar
- Eng yaxshi nomzodning CV balli
- Tanlov shu CV da qilingan
- Giperparametr tanlaydi
- Xolis baho beradi
k_tashqi * k_ichki * nomzodlar- Nomzod soni,
n, signal kuchi - Yo'q, normal
- Yo'q, faqat baho
- Butun ma'lumotda
GridSearchCV - CV + alohida test
- Kichik
n, ko'p nomzod best_score_ - nested > 0.02— xavf
Vazifa 2: Xatolarni tuzating
1. print(f"Model AUC: {grid.best_score_:.3f}")
2. X = StandardScaler().fit_transform(X)
3. model = cross_val_score(grid, X, y, cv=tashqi)
4. X = SelectKBest(k=20).fit_transform(X, y)
5. GridSearchCV(m, g, cv=2)Javoblar
1. nested = cross_val_score(grid, X, y, cv=tashqi)
2. quvur = make_pipeline(StandardScaler(), SVC())
3. model = grid.fit(X, y).best_estimator_
4. Pipeline([("s", SelectKBest(k=20)), ("m", model)])
5. GridSearchCV(m, g, cv=StratifiedKFold(5, shuffle=True, random_state=1))Vazifa 3: Optimizm
Modellang:
- Sozlama
- GridSearchCV
- Haqiqiy natija
- Barcha nomzodlar
Vazifa 4: Nested
Modellang:
- Oddiy CV
- Nested CV
- Fold parametrlari
- Uch raqam
Vazifa 5: Bog'liqlik
Modellang:
- Nomzod soni
- Ma'lumot hajmi
- Haqiqat
- Diagnostika
Vazifa 6: Narx
Modellang:
- Sozlama
- Variantlar
- Narx-sifat
- Tavsiya
Vazifa 7: O'ylash
Sizda 150 ta bemorning ma'lumoti bor va 600 ta giperparametr kombinatsiyasi sinalgan. best_score_ = 0.91. Hamkasbingiz "ajoyib, maqola yozamiz" deydi. Siz nima deysiz?
Javob
Qisqa javob: 0.91 ga ishonmang. n = 150 va 600 nomzod — bu optimizm uchun eng yomon kombinatsiya. Nested CV bilan qayta baholash kerak, va natija 0.10-0.20 ga pastroq chiqishi ehtimoli yuqori.
1. Nima uchun xavfli
| Omil | Qiymat | Ta'siri |
|---|---|---|
n |
150 | CV bahosining SE si ~0.04 |
| Nomzodlar | 600 | sqrt(2 ln 600) ≈ 3.6 |
| Kutilgan optimizm | — | ~0.04 × 3.6 ≈ 0.14 |
Ya'ni 0.91 ning ortida haqiqiy qiymat 0.77 atrofida bo'lishi mumkin.
2. Qo'shimcha xavflar
- Bemor guruhi: agar bir bemordan bir necha namuna bo'lsa —
GroupKFoldkerak, aks holda yana leakage. - Sinf muvozanati: 150 bemorda musbat sinf 20 ta bo'lsa, har fold 4 ta musbat — baho juda shovqinli.
- Belgi tanlash: agar belgilar butun ma'lumotda tanlangan bo'lsa, bu alohida va kuchliroq leakage.
3. Nima qilish kerak
ichki = StratifiedKFold(5, shuffle=True, random_state=1)
tashqi = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0)
quvur = Pipeline([("tayyor", ...), ("tanlov", ...), ("model", ...)])
qidiruv = GridSearchCV(quvur, setka, cv=ichki, scoring="roc_auc")
nested = cross_val_score(qidiruv, X, y, cv=tashqi, scoring="roc_auc")
print(f"{nested.mean():.3f} (+-{nested.std():.3f})")4. Qidiruv maydonini qisqartirish
600 nomzod n = 150 uchun juda ko'p. Amaliy qoida: nomzodlar soni n dan ancha kichik bo'lsin. Bu yerda 10-30 nomzod yetarli — RandomizedSearchCV(n_iter=20) yoki domenga asoslangan tor maydon.
5. Maqolada nima yozish kerak
"5×10 takroriy nested CV (ichki 5-fold
GridSearchCV, 20 nomzod): AUC 0.78 (takrorlar bo'yicha std 0.05). Sozlash optimizmi:best_score_0.91, nested 0.78, farq 0.13. Namuna hajmi kichik bo'lgani uchun natijani mustaqil kogortada tasdiqlash zarur."
6. Xulosa
best_score_ni tashlang- Nested CV (takroriy) bilan qayta baholang
- Nomzodlar sonini 20-30 ga tushiring
- Guruh tuzilmasini tekshiring
- Mustaqil kogortani rejalashtiring
Nimani mustahkamlaydi: 2.1, 2.3-bo'limlar.
Xulosa
Bu darsda nested cross-validation ni o'rgandik.
Eng muhim uch fikr:
GridSearchCV.best_score_— modelning natijasi emas. U eng yaxshi nomzodning o'sha CV dagi balli, ya'ni tanlov qilingan to'plamdagi ball. Mustaqil nomzodlar soni ortgani va namuna kichraygani sari bu raqam tizimli ravishda haqiqatdan uzoqlashadi:n = 150da o'lchovlarimizda optimizm +0.019 chiqdi va sakkizta mustaqil to'plamning hammasida musbat bo'ldi.Nested CV tanlov va bahoni ajratadi. Ichki halqa giperparametrni tanlaydi, tashqi halqa esa shu tanlov protsedurasini baholaydi va tashqi test qismi hech qachon tanlovda qatnashmaydi. Har tashqi foldda boshqa parametr chiqishi normal — nested CV modelni emas, protsedurani baholaydi.
Optimizmni o'lchang va arzon muqobilni biling.
best_score_ - nestedfarqi diagnostika: 0.02 dan katta bo'lsa siz shovqinga moslashayapsiz. Ma'lumot yetarli (n > 5000) bo'lsa CV + alohida test nested CV ning arzon muqobili. Yakuniy modelni esa butun ma'lumotda oddiyGridSearchCVbilan quring — baho nested dan, model qidiruvdan.
Keyingi darsda giperparametrlarning o'ziga o'tamiz: qaysi model qaysi parametrga sezgir, qidiruv maydonini qanday belgilash va qaysi parametrlarni umuman sozlamaslik kerak.
Izohlar · 0
Izoh yozish uchun kiring.