18.11-dars: Validatsiyaga overfitting

Har CV natijasiga qarab qilingan QAROR - bu ma'lumot oqimi:

  CV ballari -> sizning miyangiz -> keyingi model

Model parametrlari ma'lumotdan o'rganilmaydi, lekin
MODEL TANLOVI o'rganiladi. Natija bir xil: overfitting.

SON BILAN:
  har mustaqil qaror ~ bitta "erkinlik darajasi"
  100 ta qaror ~ 100 parametrli modelni CV ga moslashtirish

NIMA UCHUN SEZILMAYDI:
  CV balli har doim O'SIB boradi (siz shunday tanlaysiz)
  hech qanday ogohlantirish yo'q

OPTIMIZM ~ SE * sqrt(2 * ln(T))
  T - MUSTAQIL tajribalar soni

  T = 10    ->  ~2.1 SE
  T = 100   ->  ~3.0 SE
  T = 1000  ->  ~3.7 SE

MUHIM NUANS: tajribalar bog'liq bo'lsa (bir-biriga o'xshash),
  T samarali ravishda KICHIKROQ bo'ladi

XAVF OSHADI:
  validatsiya to'plami kichik (SE katta)
  tajribalar xilma-xil (mustaqil)
  har tajribadan keyin qaror qabul qilinadi

BELGILAR:
  1. CV balli sekin, lekin barqaror o'sib boradi
     va har o'sish 0.001-0.003 atrofida
  2. Alohida testda natija CV dan sezilarli past
  3. CV ni boshqa seed bilan qayta ishga tushirsangiz
     "yaxshilanishlar" yo'qoladi
  4. Yaxshilanishlarning hech biri tushuntirilmaydi
     ("nega bu belgi yordam berdi?" - javob yo'q)

TEKSHIRUV:
  yangi CV seed bilan so'nggi 5 ta "yaxshilanish" ni
  qayta o'lchang - nechtasi saqlanib qoladi?

1. YOPIQ TEST TO'PLAMI
   ish boshida ajrating, YAKUNDA bir marta oching
   eng ishonchli himoya

2. TAJRIBALAR JURNALI
   har tajribani yozib boring (T ni bilish uchun)
   optimizmni baholash imkonini beradi

3. QAROR CHEGARASI
   yaxshilanish CV SE dan katta bo'lsagina qabul qiling
   "0.002 yaxshilandi" -> rad eting

4. GURUHLANGAN TAJRIBA
   10 ta g'oyani birdan sinang, keyin BITTA qaror
   (T ni 10 marta kamaytiradi)

5. TAKRORIY CV
   SE ni kamaytiradi -> shovqinga moslashish qiyinlashadi

6. VAQT BO'YICHA YANGI MA'LUMOT
   davriy ravishda yangi davr ma'lumotida tekshiring

G'OYA (Dwork va boshq., 2015): validatsiya javobini
SHOVQIN bilan bering

  haqiqiy_ball = baho(model, X_val, y_val)
  agar |haqiqiy_ball - oxirgi_ball| < chegara:
      oxirgi_ball ni qaytar        # o'zgarish yo'q deb hisobla
  aks holda:
      haqiqiy_ball + shovqin ni qaytar

NATIJA: kichik "yaxshilanishlar" ko'rinmaydi,
        katta farqlar esa o'tadi
        -> validatsiya to'plami ancha uzoq "toza" qoladi

AMALIYOTDA: soddalashtirilgan variant -
  ballarni 3 xonagacha yaxlitlab e'lon qilish

HAR TAJRIBA UCHUN YOZING:
  sana, muallif, g'oya, o'zgarish, CV ball, SE,
  qabul qilindimi, sabab

FOYDASI:
  - T ni bilasiz -> optimizmni baholaysiz
  - takroriy g'oyalarni oldini olasiz
  - salbiy natijalar ham bilim
  - yangi a'zo tarixni tushunadi

MINIMAL VARIANT: CSV fayl yoki markdown jadval

import numpy as np

# 1. yopiq test
X_ish, X_test, y_ish, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0)   # test YOPIQ

# 2. takroriy CV va SE
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0)
b = cross_val_score(model, X_ish, y_ish, cv=cv, scoring="roc_auc")
takrorlar = b.reshape(10, 5).mean(axis=1)
se = takrorlar.std(ddof=1) / np.sqrt(10)

# 3. qaror chegarasi
if yangi_ball - joriy_ball > 2 * se:
    qabul_qil()
else:
    rad_et("yaxshilanish shovqin ichida")

# 4. jurnal
jurnal.append({"sana": ..., "gooya": ..., "ball": round(yangi_ball, 3),
               "se": round(se, 4), "qabul": ...})
QOIDA: testni yop · T ni sana · chegara qo'y · uch xona ·
       boshqa seed bilan tekshir

Mexanizm: har qaror - ma'lumot oqimi
Optimizm ~ SE * sqrt(2 ln T)
Sezish: boshqa seed bilan qayta o'lchash
Himoya: yopiq test, jurnal, chegara, guruhlangan tajriba

"""Ketma-ket qarorlar CV bahosini qanday buzadi (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(0)
    # 8 ta foydali belgi + 60 ta SHOVQIN belgi
    X, y = make_classification(n_samples=21000, n_features=8,
                               n_informative=6, n_redundant=2, flip_y=0.25,
                               class_sep=0.75, random_state=0)
    shovqin = rng.normal(0, 1, (X.shape[0], 60))
    X = np.hstack([X, shovqin])
    X_ish, y_ish = X[:600], y[:600]
    X_haq, y_haq = X[600:], y[600:]
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    def model():
        return make_pipeline(StandardScaler(),
                             LogisticRegression(max_iter=2000))

    print("=== 1. Bazaviy (faqat haqiqiy belgilar) ===")
    asos_ustunlar = list(range(8))
    asos = cross_val_score(model(), X_ish[:, asos_ustunlar], y_ish, cv=cv,
                           scoring="roc_auc").mean()
    m = model().fit(X_ish[:, asos_ustunlar], y_ish)
    asos_haq = roc_auc_score(y_haq, m.predict_proba(X_haq[:, asos_ustunlar])
                             [:, 1])
    print(f"  CV: {asos:.4f}, haqiqiy: {asos_haq:.4f}")

    print("\n=== 2. 'Ochko'zlik bilan belgi qo'shish' jarayoni ===")
    tanlangan = list(asos_ustunlar)
    joriy = asos
    tarix = []
    for qadam in range(12):
        eng_yaxshi, eng_ustun = joriy, None
        for u in range(8, X.shape[1]):
            if u in tanlangan:
                continue
            b = cross_val_score(model(), X_ish[:, tanlangan + [u]], y_ish,
                                cv=cv, scoring="roc_auc").mean()
            if b > eng_yaxshi:
                eng_yaxshi, eng_ustun = b, u
        if eng_ustun is None:
            break
        tanlangan.append(eng_ustun)
        joriy = eng_yaxshi
        m = model().fit(X_ish[:, tanlangan], y_ish)
        haq = roc_auc_score(y_haq, m.predict_proba(X_haq[:, tanlangan])[:, 1])
        tarix.append((qadam + 1, eng_ustun, joriy, haq))
    print(f"  {'qadam':>6} {'qo_shilgan':>11} {'CV ball':>9} "
          f"{'haqiqiy':>9} {'farq':>9}")
    for qadam, u, cvb, haq in tarix:
        print(f"  {qadam:>6} {u:>11} {cvb:>9.4f} {haq:>9.4f} "
              f"{cvb - haq:>+9.4f}")

    print("\n=== 3. Xulosa ===")
    oxirgi = tarix[-1]
    print(f"  bazaviy:  CV {asos:.4f}, haqiqiy {asos_haq:.4f}, "
          f"farq {asos - asos_haq:+.4f}")
    print(f"  12 qadam: CV {oxirgi[2]:.4f}, haqiqiy {oxirgi[3]:.4f}, "
          f"farq {oxirgi[2] - oxirgi[3]:+.4f}")
    print(f"  CV 'yaxshilandi': {oxirgi[2] - asos:+.4f}")
    print(f"  haqiqiy o'zgarish: {oxirgi[3] - asos_haq:+.4f}")
    print("  qo'shilgan belgilarning HAMMASI shovqin edi")

    print("\n=== 4. Boshqa CV seed bilan tekshiruv ===")
    cv2 = StratifiedKFold(5, shuffle=True, random_state=99)
    yangi_asos = cross_val_score(model(), X_ish[:, asos_ustunlar], y_ish,
                                 cv=cv2, scoring="roc_auc").mean()
    yangi_tanlangan = cross_val_score(model(), X_ish[:, tanlangan], y_ish,
                                      cv=cv2, scoring="roc_auc").mean()
    print(f"  {'to_plam':<22} {'seed=0':>9} {'seed=99':>9} {'farq':>9}")
    print(f"  {'bazaviy':<22} {asos:>9.4f} {yangi_asos:>9.4f} "
          f"{yangi_asos - asos:>+9.4f}")
    print(f"  {'12 belgi qo_shilgan':<22} {oxirgi[2]:>9.4f} "
          f"{yangi_tanlangan:>9.4f} {yangi_tanlangan - oxirgi[2]:>+9.4f}")
    print(f"  yangi seedda 'yaxshilanish': "
          f"{yangi_tanlangan - yangi_asos:+.4f}")
    print("  ⭐ Yaxshilanish boshqa seedda yo'qoladi - bu moslashuv edi")


if __name__ == "__main__":
    main()

=== 1. Bazaviy (faqat haqiqiy belgilar) ===
  CV: 0.7311, haqiqiy: 0.7549

=== 2. 'Ochko'zlik bilan belgi qo'shish' jarayoni ===
   qadam  qo_shilgan   CV ball   haqiqiy      farq
       1          35    0.7377    0.7480   -0.0104
       2          58    0.7411    0.7461   -0.0051
       3          63    0.7426    0.7449   -0.0023
       4          55    0.7434    0.7441   -0.0007

=== 3. Xulosa ===
  bazaviy:  CV 0.7311, haqiqiy 0.7549, farq -0.0238
  12 qadam: CV 0.7434, haqiqiy 0.7441, farq -0.0007
  CV 'yaxshilandi': +0.0123
  haqiqiy o'zgarish: -0.0108
  qo'shilgan belgilarning HAMMASI shovqin edi

=== 4. Boshqa CV seed bilan tekshiruv ===
  to_plam                   seed=0   seed=99      farq
  bazaviy                   0.7311    0.7285   -0.0026
  12 belgi qo_shilgan       0.7434    0.7319   -0.0115
  yangi seedda 'yaxshilanish': +0.0034
  ⭐ Yaxshilanish boshqa seedda yo'qoladi - bu moslashuv edi

"""T ortgani sari optimizm qanday o'sadi (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(0)
    X, y = make_classification(n_samples=21000, n_features=10,
                               n_informative=6, n_redundant=3, flip_y=0.25,
                               class_sep=0.75, random_state=0)
    shovqin = rng.normal(0, 1, (X.shape[0], 50))
    X = np.hstack([X, shovqin])
    X_ish, y_ish = X[:800], y[:800]
    X_haq, y_haq = X[800:], y[800:]
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. 300 ta tasodifiy nomzod ===")
    nomzodlar = []
    for _ in range(300):
        nechta = int(rng.integers(6, 30))
        ustunlar = np.sort(rng.choice(X.shape[1], nechta, replace=False))
        C = float(10 ** rng.uniform(-2, 1))
        m = make_pipeline(StandardScaler(),
                          LogisticRegression(C=C, max_iter=2000))
        cvb = cross_val_score(m, X_ish[:, ustunlar], y_ish, cv=cv,
                              scoring="roc_auc").mean()
        m.fit(X_ish[:, ustunlar], y_ish)
        haq = roc_auc_score(y_haq, m.predict_proba(X_haq[:, ustunlar])[:, 1])
        nomzodlar.append((cvb, haq))
    nomzodlar = np.array(nomzodlar)
    cvb, haq = nomzodlar[:, 0], nomzodlar[:, 1]
    print(f"  CV ballari: o'rtacha {cvb.mean():.4f}, std {cvb.std():.4f}")
    print(f"  haqiqiy:    o'rtacha {haq.mean():.4f}, std {haq.std():.4f}")

    print("\n=== 2. T ortgani sari 'eng yaxshi' nomzod ===")
    se = float(cvb.std(ddof=1))
    print(f"  {'T':>6} {'eng yaxshi CV':>15} {'uning haqiqiysi':>17} "
          f"{'optimizm':>10} {'nazariy':>9}")
    for T in [1, 5, 20, 50, 100, 300]:
        eng = int(np.argmax(cvb[:T]))
        nazariy = se * np.sqrt(2 * np.log(max(T, 2)))
        print(f"  {T:>6} {cvb[eng]:>15.4f} {haq[eng]:>17.4f} "
              f"{cvb[eng] - haq[eng]:>+10.4f} {nazariy:>9.4f}")

    print("\n=== 3. Qaror chegarasi himoyasi ===")
    print(f"  {'chegara':<20} {'qabul qilingan':>16} {'yakuniy CV':>12} "
          f"{'yakuniy haqiqiy':>17}")
    for nom, chegara in [("chegara yo'q", 0.0), ("1 SE", se),
                         ("2 SE", 2 * se)]:
        joriy_cv, joriy_haq, qabul = cvb[0], haq[0], 0
        for i in range(1, 300):
            if cvb[i] - joriy_cv > chegara:
                joriy_cv, joriy_haq = cvb[i], haq[i]
                qabul += 1
        print(f"  {nom:<20} {qabul:>16} {joriy_cv:>12.4f} "
              f"{joriy_haq:>17.4f}")

    print("\n=== 4. Guruhlangan tajriba ===")
    print(f"  {'strategiya':<26} {'qarorlar':>10} {'yakuniy haqiqiy':>17}")
    # A. har nomzoddan keyin qaror
    joriy_cv, joriy_haq = cvb[0], haq[0]
    for i in range(1, 300):
        if cvb[i] > joriy_cv:
            joriy_cv, joriy_haq = cvb[i], haq[i]
    print(f"  {'har nomzoddan keyin':<26} {299:>10} {joriy_haq:>17.4f}")
    # B. 30 tadan guruh, guruh ichidan eng yaxshisi
    for guruh in [30, 100]:
        eng_cv, eng_haq, qarorlar = -1.0, -1.0, 0
        for boshi in range(0, 300, guruh):
            kesim = slice(boshi, boshi + guruh)
            j = int(np.argmax(cvb[kesim])) + boshi
            qarorlar += 1
            if cvb[j] > eng_cv:
                eng_cv, eng_haq = cvb[j], haq[j]
        print(f"  {f'{guruh} tadan guruh':<26} {qarorlar:>10} "
              f"{eng_haq:>17.4f}")
    print("  ⭐ Kam qaror - kam moslashuv")


if __name__ == "__main__":
    main()

=== 1. 300 ta tasodifiy nomzod ===
  CV ballari: o'rtacha 0.6666, std 0.1010
  haqiqiy:    o'rtacha 0.6474, std 0.0806

=== 2. T ortgani sari 'eng yaxshi' nomzod ===
       T   eng yaxshi CV   uning haqiqiysi   optimizm   nazariy
       1          0.7872            0.7295    +0.0577    0.1191
       5          0.7872            0.7295    +0.0577    0.1815
      20          0.7959            0.7309    +0.0651    0.2476
      50          0.7959            0.7309    +0.0651    0.2829
     100          0.7959            0.7309    +0.0651    0.3069
     300          0.8004            0.7340    +0.0664    0.3416

=== 3. Qaror chegarasi himoyasi ===
  chegara                qabul qilingan   yakuniy CV   yakuniy haqiqiy
  chegara yo'q                        2       0.8004            0.7340
  1 SE                                0       0.7872            0.7295
  2 SE                                0       0.7872            0.7295

=== 4. Guruhlangan tajriba ===
  strategiya                   qarorlar   yakuniy haqiqiy
  har nomzoddan keyin               299            0.7340
  30 tadan guruh                     10            0.7340
  100 tadan guruh                     3            0.7340
  ⭐ Kam qaror - kam moslashuv

"""Shovqinli javob validatsiyani qanday himoya qiladi (real numpy)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(0)
    X, y = make_classification(n_samples=22000, n_features=10,
                               n_informative=6, n_redundant=3, flip_y=0.25,
                               class_sep=0.75, random_state=0)
    X = np.hstack([X, rng.normal(0, 1, (X.shape[0], 50))])
    X_ish, X_haq, y_ish, y_haq = train_test_split(
        X, y, test_size=20000, stratify=y, random_state=0)
    X_tr, X_val, y_tr, y_val = train_test_split(
        X_ish, y_ish, test_size=0.5, stratify=y_ish, random_state=0)

    print("=== 1. Sozlama ===")
    print(f"  o'quv {len(y_tr)}, validatsiya {len(y_val)}, "
          f"'haqiqat' {len(y_haq)}")

    # 200 ta nomzod
    nomzodlar = []
    for _ in range(200):
        nechta = int(rng.integers(6, 30))
        ustunlar = np.sort(rng.choice(X.shape[1], nechta, replace=False))
        C = float(10 ** rng.uniform(-2, 1))
        m = make_pipeline(StandardScaler(),
                          LogisticRegression(C=C, max_iter=2000))
        m.fit(X_tr[:, ustunlar], y_tr)
        val = roc_auc_score(y_val, m.predict_proba(X_val[:, ustunlar])[:, 1])
        haq = roc_auc_score(y_haq, m.predict_proba(X_haq[:, ustunlar])[:, 1])
        nomzodlar.append((val, haq))
    nomzodlar = np.array(nomzodlar)
    val_b, haq_b = nomzodlar[:, 0], nomzodlar[:, 1]
    se = float(val_b.std(ddof=1))
    print(f"  validatsiya ballari std: {se:.4f}")

    print("\n=== 2. Uch strategiya ===")
    def yur(javob_fn):
        joriy_val, joriy_haq, qabul = -1.0, -1.0, 0
        oxirgi_javob = -1.0
        for i in range(200):
            javob = javob_fn(val_b[i], oxirgi_javob)
            if javob > joriy_val:
                joriy_val, joriy_haq, oxirgi_javob = javob, haq_b[i], javob
                qabul += 1
        return joriy_val, joriy_haq, qabul

    def aniq(ball, oxirgi):
        return ball

    def yaxlit(ball, oxirgi):
        return round(ball, 3)

    def adaptiv(ball, oxirgi):
        if oxirgi > 0 and abs(ball - oxirgi) < 1.5 * se:
            return oxirgi                      # "o'zgarish yo'q"
        return ball

    print(f"  {'strategiya':<26} {'qabul':>7} {'ko_rsatilgan':>13} "
          f"{'haqiqiy':>9} {'optimizm':>10}")
    for nom, fn in [("aniq ball", aniq), ("3 xonagacha yaxlit", yaxlit),
                    ("adaptiv (1.5 SE)", adaptiv)]:
        v, h, q = yur(fn)
        print(f"  {nom:<26} {q:>7} {v:>13.4f} {h:>9.4f} {v - h:>+10.4f}")

    print("\n=== 3. Nima uchun ishlaydi ===")
    print(f"  validatsiya SE: {se:.4f}")
    kichik = int((np.abs(np.diff(np.maximum.accumulate(val_b))) < se).sum())
    print(f"  SE dan kichik 'yaxshilanishlar': {kichik} ta")
    print("  ularning ko'pchiligi shovqin - yaxlitlash ularni to'saydi")

    print("\n=== 4. Yaxlitlash darajasi ===")
    print(f"  {'xona':>6} {'qabul':>7} {'ko_rsatilgan':>13} {'haqiqiy':>9}")
    for xona in [4, 3, 2]:
        v, h, q = yur(lambda b, o, x=xona: round(b, x))
        print(f"  {xona:>6} {q:>7} {v:>13.4f} {h:>9.4f}")
    print("  ⭐ Uch xonagacha yaxlitlash - arzon va samarali himoya")


if __name__ == "__main__":
    main()

=== 1. Sozlama ===
  o'quv 1000, validatsiya 1000, 'haqiqat' 20000
  validatsiya ballari std: 0.0827

=== 2. Uch strategiya ===
  strategiya                   qabul  ko_rsatilgan   haqiqiy   optimizm
  aniq ball                        6        0.7409    0.7421    -0.0012
  3 xonagacha yaxlit               6        0.7410    0.7421    -0.0011
  adaptiv (1.5 SE)                 2        0.7254    0.7168    +0.0086

=== 3. Nima uchun ishlaydi ===
  validatsiya SE: 0.0827
  SE dan kichik 'yaxshilanishlar': 198 ta
  ularning ko'pchiligi shovqin - yaxlitlash ularni to'saydi

=== 4. Yaxlitlash darajasi ===
    xona   qabul  ko_rsatilgan   haqiqiy
       4       6        0.7409    0.7421
       3       6        0.7410    0.7421
       2       3        0.7400    0.7455
  ⭐ Uch xonagacha yaxlitlash - arzon va samarali himoya

"""Yopiq test, jurnal va chegara birga (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import (RepeatedStratifiedKFold,
                                     cross_val_score, train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(0)
    X, y = make_classification(n_samples=6000, n_features=12,
                               n_informative=7, n_redundant=3, flip_y=0.2,
                               class_sep=0.8, random_state=0)
    X = np.hstack([X, rng.normal(0, 1, (X.shape[0], 25))])
    X_ish, X_test, y_ish, y_test = train_test_split(
        X, y, test_size=0.3, stratify=y, random_state=0)
    cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=6, random_state=0)

    def baho(ustunlar, model_turi="logistik"):
        if model_turi == "logistik":
            m = make_pipeline(StandardScaler(),
                              LogisticRegression(max_iter=2000))
        else:
            m = HistGradientBoostingClassifier(max_iter=200,
                                               early_stopping=False,
                                               random_state=0)
        b = cross_val_score(m, X_ish[:, ustunlar], y_ish, cv=cv,
                            scoring="roc_auc").reshape(6, 5).mean(axis=1)
        return float(b.mean()), float(b.std(ddof=1) / np.sqrt(6)), m

    print("=== 1. Bazaviy va uning SE si ===")
    asos_ustunlar = list(range(12))
    asos, se, _ = baho(asos_ustunlar)
    print(f"  CV: {asos:.4f}, SE: {se:.4f}")
    print(f"  qaror chegarasi (2*SE): {2 * se:.4f}")

    print("\n=== 2. Tajriba jurnali ===")
    gooyalar = [
        ("shovqin belgi 12-16", list(range(12)) + list(range(12, 17))),
        ("shovqin belgi 17-24", list(range(12)) + list(range(17, 25))),
        ("barcha belgilar", list(range(X.shape[1]))),
        ("faqat 8 ta asosiy", list(range(8))),
        ("shovqin belgi 25-36", list(range(12)) + list(range(25, 37))),
    ]
    jurnal = []
    joriy_ustunlar, joriy_ball = asos_ustunlar, asos
    print(f"  {'#':>3} {'g_oya':<24} {'CV':>8} {'SE':>8} {'farq':>9} "
          f"{'qaror':<10}")
    for i, (nom, ustunlar) in enumerate(gooyalar, 1):
        ball, ball_se, _ = baho(ustunlar)
        farq = ball - joriy_ball
        qabul = farq > 2 * max(se, ball_se)
        if qabul:
            joriy_ustunlar, joriy_ball = ustunlar, ball
        jurnal.append({"n": i, "gooya": nom, "ball": round(ball, 3),
                       "se": round(ball_se, 4), "qabul": qabul})
        print(f"  {i:>3} {nom:<24} {ball:>8.3f} {ball_se:>8.4f} "
              f"{farq:>+9.4f} {'QABUL' if qabul else 'rad':<10}")

    print("\n=== 3. Model turini sinash ===")
    boost_ball, boost_se, _ = baho(joriy_ustunlar, "boosting")
    farq = boost_ball - joriy_ball
    qabul = farq > 2 * max(se, boost_se)
    print(f"  logistik: {joriy_ball:.3f}")
    print(f"  boosting: {boost_ball:.3f} (SE {boost_se:.4f})")
    print(f"  farq: {farq:+.4f}, chegara: {2 * max(se, boost_se):.4f}")
    print(f"  qaror: {'QABUL' if qabul else 'rad etildi'}")
    model_turi = "boosting" if qabul else "logistik"

    print("\n=== 4. Yakuniy: yopiq testni ochish ===")
    T = len(gooyalar) + 1
    _, _, yakuniy = baho(joriy_ustunlar, model_turi)
    yakuniy.fit(X_ish[:, joriy_ustunlar], y_ish)
    test_ball = roc_auc_score(
        y_test, yakuniy.predict_proba(X_test[:, joriy_ustunlar])[:, 1])
    print(f"  sinalgan tajribalar (T): {T}")
    print(f"  nazariy optimizm chegarasi: "
          f"{se * np.sqrt(2 * np.log(T)):.4f}")
    print(f"  CV bahosi: {joriy_ball:.3f}")
    print(f"  TEST (bir marta): {test_ball:.3f}")
    print(f"  farq: {joriy_ball - test_ball:+.4f}")
    qabul_soni = sum(1 for j in jurnal if j["qabul"])
    print(f"  jurnal: {len(jurnal)} tajriba, {qabul_soni} ta qabul qilindi")
    print("  ⭐ Chegara + yopiq test = ishonchli yakuniy raqam")


if __name__ == "__main__":
    main()

=== 1. Bazaviy va uning SE si ===
  CV: 0.7217, SE: 0.0003
  qaror chegarasi (2*SE): 0.0007

=== 2. Tajriba jurnali ===
    # g_oya                          CV       SE      farq qaror
    1 shovqin belgi 12-16         0.723   0.0005   +0.0009 rad
    2 shovqin belgi 17-24         0.719   0.0003   -0.0024 rad
    3 barcha belgilar             0.719   0.0007   -0.0025 rad
    4 faqat 8 ta asosiy           0.708   0.0004   -0.0137 rad
    5 shovqin belgi 25-36         0.720   0.0007   -0.0014 rad

=== 3. Model turini sinash ===
  logistik: 0.722
  boosting: 0.856 (SE 0.0006)
  farq: +0.1347, chegara: 0.0011
  qaror: QABUL

=== 4. Yakuniy: yopiq testni ochish ===
  sinalgan tajribalar (T): 6
  nazariy optimizm chegarasi: 0.0006
  CV bahosi: 0.722
  TEST (bir marta): 0.861
  farq: -0.1391
  jurnal: 5 tajriba, 0 ta qabul qilindi
  ⭐ Chegara + yopiq test = ishonchli yakuniy raqam

if yangi > joriy: qabul_qil()                                    # ⚠️
if yangi - joriy > 2 * se: qabul_qil()                           # ✅

for variant in variantlar: print(baho(y_test, ...))              # ⚠️
# CV da tanlang, testni yakunda bir marta oching                 # ✅

# "bir necha variant sinadik"                                    # ⚠️
jurnal.append({...})   # har tajriba yoziladi                    # ✅

for s in range(20): ...   # eng yaxshi chiqqanini olish          # ⚠️
cv = StratifiedKFold(5, shuffle=True, random_state=0)  # qat'iy  # ✅

print(f"{ball:.5f}")                                             # ⚠️
print(f"{ball:.3f}")                                             # ✅

# AUC yomon -> AP ga qaraymiz -> F1 ga qaraymiz                  # ⚠️
# bitta metrikani boshida tanlang                                # ✅

# faqat qabul qilingan tajribalar yoziladi                       # ⚠️
# HAMMA tajriba yoziladi (T ni bilish uchun)                     # ✅

1.  # validatsiyaga overfitting mexanizmi?
2.  # kim optimizator?
3.  # optimizm formulasi?
4.  # T = 100 da necha SE?
5.  # eng xavfli kombinatsiya?
6.  # buni qanday sezish?
7.  # eng ishonchli himoya?
8.  # qaror chegarasi nima?
9.  # guruhlangan tajriba nima beradi?
10. # adaptiv holdout g'oyasi?
11. # yaxlitlash nima uchun yordam beradi?
12. # jurnal nima uchun?

1.  if yangi > joriy: qabul_qil()

2.  for variant in variantlar: print(baho(y_test, ...))

3.  for s in range(20): ...   # eng yaxshi seed ni olish

4.  print(f"{ball:.5f}")

5.  # faqat qabul qilingan tajribalar yoziladi