18.1-dars: Baholash dizayni

DISPERSIYA (variance) - baho tasodifiy tebranadi
  manba: qaysi qatorlar testga tushdi
  belgisi: seed ni o'zgartirsangiz natija o'zgaradi
  yechim: ko'proq test qatori, CV, takroriy CV

OPTIMIZM (bias) - baho tizimli ravishda YUQORI
  manba: baholash to'plamida tanlov/sozlash qilindi
  belgisi: CV yaxshi, yangi ma'lumot yomon
  yechim: tanlov uchun ALOHIDA to'plam, nested CV

MUHIM FARQ:
  dispersiyani ko'proq ma'lumot bilan kamaytirasiz
  optimizmni FAQAT dizayn bilan kamaytirasiz

m ta nomzodni bitta validatsiya to'plamida baholadingiz.
Har birining bahosi: haqiqiy qiymat + shovqin

ENG YUQORI ball olgan nomzod:
  - haqiqatan yaxshi bo'lishi mumkin
  - YOKI shunchaki shovqini ijobiy bo'lgan

Kutilgan optimizm ~ SE * sqrt(2 * ln(m))

  m = 10  nomzod  ->  ~2.1 * SE
  m = 100 nomzod  ->  ~3.0 * SE
  m = 1000 nomzod ->  ~3.7 * SE

XULOSA: nomzodlar soni ortsa, "eng yaxshi" ball
        haqiqatdan shuncha uzoqlashadi

Aniqlik uchun (binom):
  SE = sqrt(p * (1 - p) / n)

  n = 100   p = 0.80  ->  SE = 0.040   (+-0.08 oraliq!)
  n = 1000  p = 0.80  ->  SE = 0.013
  n = 10000 p = 0.80  ->  SE = 0.004

AUC uchun taxminan:
  SE ~ 0.5 / sqrt(min(n_musbat, n_manfiy))

QOIDA: 0.01 lik farqni ishonchli o'lchash uchun
       kamida bir necha ming test qatori kerak

1. HOLDOUT (train / test)
   + eng tez, eng sodda
   - yuqori dispersiya (kichik ma'lumotda)
   - sozlash uchun joy yo'q

2. CROSS-VALIDATION
   + har qator bir marta testda bo'ladi -> kam dispersiya
   + butun ma'lumot ishlatiladi
   - k barobar qimmat
   - sozlash qilinsa OPTIMISTIK

3. CV + ALOHIDA TEST
   + CV da sozlaysiz, testda BIR MARTA baholaysiz
   + optimizm yo'q
   - ma'lumotning bir qismi sozlashda ishlatilmaydi

MA'LUMOT HAJMI:
  < 1 000      takroriy CV (holdout juda beqaror)
  1 000-50 000 CV + alohida test
  > 50 000     holdout yetarli (train/val/test)

VAZIFA:
  bir nechta modelni taqqoslash  -> CV
  yakuniy raqam kerak            -> alohida test
  giperparametr sozlash          -> nested CV yoki CV + test
  ishlab chiqarishga chiqarish   -> vaqt bo'yicha test

MUHIM: test to'plamini BIR MARTA ishlating

BYUDJETNI OLDINDAN TAQSIMLANG:

  tadqiqot bosqichi:  CV, cheksiz tajriba
  tanlov bosqichi:    CV, nomzodlar ro'yxati qisqartiriladi
  tasdiqlash:         alohida test, 1 marta

YOZIB BORING:
  - nechta variant sinaldi (m)
  - qaysi to'plamda
  - yakuniy qaror qaysi raqamga asoslandi

m ni bilish optimizmni baholashga imkon beradi

import numpy as np
from sklearn.model_selection import (RepeatedStratifiedKFold, cross_val_score,
                                     train_test_split)

# 1. tadqiqot va tanlov uchun ajratish
X_ish, X_test, y_ish, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0)      # test YOPIQ

# 2. tanlov CV da
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
ball = cross_val_score(model, X_ish, y_ish, cv=cv, scoring="roc_auc")
print(ball.mean(), ball.std())

# 3. yakuniy baho - BIR MARTA
eng_yaxshi.fit(X_ish, y_ish)
yakuniy = roc_auc_score(y_test, eng_yaxshi.predict_proba(X_test)[:, 1])

# standart xato (aniqlik uchun)
se = np.sqrt(p * (1 - p) / len(y_test))
QOIDA: dizaynni oldindan yoz · nomzodlar sonini sana ·
       farqni SE bilan solishtir · testni bir marta och

Dispersiya -> ko'proq test qatori, CV, takroriy CV
Optimizm   -> dizayn (tanlov va baho ALOHIDA to'plamda)
G'olib la'nati ~ SE * sqrt(2 ln m)
Test to'plami: bir marta, yakunda

"""Holdout va CV dispersiyasini o'lchash (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import (StratifiedKFold, cross_val_score,
                                     train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def model():
    return make_pipeline(StandardScaler(),
                         LogisticRegression(max_iter=2000))


def main() -> None:
    X, y = make_classification(n_samples=600, n_features=20,
                               n_informative=6, n_redundant=4,
                               flip_y=0.15, class_sep=0.9, random_state=0)
    print("=== 1. Ma'lumot ===")
    print(f"  {X.shape[0]} qator, {X.shape[1]} belgi, "
          f"musbat ulushi {y.mean():.2%}")

    print("\n=== 2. 200 ta turli holdout bo'linishi ===")
    holdout = []
    for seed in range(200):
        Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25,
                                              stratify=y, random_state=seed)
        m = model().fit(Xtr, ytr)
        holdout.append(roc_auc_score(yte, m.predict_proba(Xte)[:, 1]))
    holdout = np.array(holdout)
    print(f"  o'rtacha: {holdout.mean():.4f}")
    print(f"  std:      {holdout.std():.4f}")
    print(f"  eng past: {holdout.min():.4f}   eng yuqori: {holdout.max():.4f}")
    print(f"  oraliq:   {holdout.max() - holdout.min():.4f}")
    p5, p95 = np.percentile(holdout, [5, 95])
    print(f"  90% oraliq: [{p5:.4f}, {p95:.4f}]")

    print("\n=== 3. 40 ta turli 5-fold CV ===")
    cv_ballar = []
    for seed in range(40):
        cv = StratifiedKFold(5, shuffle=True, random_state=seed)
        cv_ballar.append(cross_val_score(model(), X, y, cv=cv,
                                         scoring="roc_auc").mean())
    cv_ballar = np.array(cv_ballar)
    print(f"  o'rtacha: {cv_ballar.mean():.4f}")
    print(f"  std:      {cv_ballar.std():.4f}")
    print(f"  oraliq:   {cv_ballar.max() - cv_ballar.min():.4f}")

    print("\n=== 4. Taqqoslash ===")
    print(f"  {'dizayn':<22} {'std':>8} {'oraliq':>9} {'narx':>8}")
    print(f"  {'holdout (25%)':<22} {holdout.std():>8.4f} "
          f"{holdout.max() - holdout.min():>9.4f} {'1x':>8}")
    print(f"  {'5-fold CV':<22} {cv_ballar.std():>8.4f} "
          f"{cv_ballar.max() - cv_ballar.min():>9.4f} {'5x':>8}")
    print(f"  dispersiya kamayishi: "
          f"{holdout.std() / cv_ballar.std():.1f} barobar")
    print("  ⭐ Bitta holdout - bitta tasodifiy son")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  600 qator, 20 belgi, musbat ulushi 48.50%

=== 2. 200 ta turli holdout bo'linishi ===
  o'rtacha: 0.7821
  std:      0.0330
  eng past: 0.6992   eng yuqori: 0.8772
  oraliq:   0.1781
  90% oraliq: [0.7345, 0.8386]

=== 3. 40 ta turli 5-fold CV ===
  o'rtacha: 0.7758
  std:      0.0056
  oraliq:   0.0244

=== 4. Taqqoslash ===
  dizayn                      std    oraliq     narx
  holdout (25%)            0.0330    0.1781       1x
  5-fold CV                0.0056    0.0244       5x
  dispersiya kamayishi: 5.9 barobar
  ⭐ Bitta holdout - bitta tasodifiy son

"""Ko'p nomzoddan tanlash bahoni qanday suradi (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(0)
    X, y = make_classification(n_samples=6000, n_features=30,
                               n_informative=8, n_redundant=6,
                               flip_y=0.2, class_sep=0.8, random_state=0)
    # mavjud ma'lumot (2000) va "haqiqat" to'plami (4000)
    X_ish, X_haq, y_ish, y_haq = train_test_split(
        X, y, test_size=4000, stratify=y, random_state=0)
    X_tr, X_val, y_tr, y_val = train_test_split(
        X_ish, y_ish, test_size=0.5, stratify=y_ish, random_state=0)
    print("=== 1. Bo'linish ===")
    print(f"  o'quv {len(y_tr)}, validatsiya {len(y_val)}, "
          f"'haqiqat' {len(y_haq)}")

    print("\n=== 2. 200 ta nomzod (tasodifiy belgi to'plamlari) ===")
    nomzodlar = []
    for _ in range(200):
        nechta = int(rng.integers(6, 25))
        ustunlar = np.sort(rng.choice(X.shape[1], nechta, replace=False))
        C = float(10 ** rng.uniform(-2, 1))
        m = make_pipeline(StandardScaler(),
                          LogisticRegression(C=C, max_iter=2000))
        m.fit(X_tr[:, ustunlar], y_tr)
        val = roc_auc_score(y_val, m.predict_proba(X_val[:, ustunlar])[:, 1])
        haq = roc_auc_score(y_haq, m.predict_proba(X_haq[:, ustunlar])[:, 1])
        nomzodlar.append((val, haq))
    nomzodlar = np.array(nomzodlar)
    val_ballar, haq_ballar = nomzodlar[:, 0], nomzodlar[:, 1]
    xato = val_ballar - haq_ballar
    print(f"  validatsiya: o'rtacha {val_ballar.mean():.4f}, "
          f"std {val_ballar.std():.4f}")
    print(f"  'haqiqat':   o'rtacha {haq_ballar.mean():.4f}, "
          f"std {haq_ballar.std():.4f}")
    print(f"  UMUMIY SILJISH (bu bo'linishning xususiyati): "
          f"{xato.mean():+.4f}")
    se = float(xato.std(ddof=1))
    print(f"  bitta bahoning shovqini (SE): {se:.4f}")
    print("  tanlov optimizmi shu SILJISHDAN ORTIQCHA qismdir")

    print("\n=== 3. Nomzodlar soni va ortiqcha optimizm ===")
    print(f"  {'m':>6} {'eng yaxshi val':>16} {'uning haqiqiysi':>17} "
          f"{'xato':>9} {'ortiqcha':>10}")
    for m_soni in [1, 5, 20, 50, 100, 200]:
        eng = int(np.argmax(val_ballar[:m_soni]))
        ortiqcha = xato[eng] - xato.mean()
        print(f"  {m_soni:>6} {val_ballar[eng]:>16.4f} "
              f"{haq_ballar[eng]:>17.4f} {xato[eng]:>+9.4f} "
              f"{ortiqcha:>+10.4f}")

    print("\n=== 4. Nazariy chegara bilan solishtirish ===")
    print(f"  {'m':>6} {'ortiqcha optimizm':>19} {'SE*sqrt(2 ln m)':>17}")
    for m_soni in [5, 20, 50, 100, 200]:
        eng = int(np.argmax(val_ballar[:m_soni]))
        ortiqcha = xato[eng] - xato.mean()
        nazariy = se * np.sqrt(2 * np.log(m_soni))
        print(f"  {m_soni:>6} {ortiqcha:>+19.4f} {nazariy:>17.4f}")
    print("  nazariy qiymat - MUSTAQIL nomzodlar uchun YUQORI chegara;")
    print("  nomzodlar bir-biriga bog'liq bo'lgani uchun kuzatilgan")
    print("  optimizm undan pastroq chiqadi")
    print("  ⭐ Tanlovning o'zi optimizm yaratadi")


if __name__ == "__main__":
    main()

=== 1. Bo'linish ===
  o'quv 1000, validatsiya 1000, 'haqiqat' 4000

=== 2. 200 ta nomzod (tasodifiy belgi to'plamlari) ===
  validatsiya: o'rtacha 0.8146, std 0.0690
  'haqiqat':   o'rtacha 0.8170, std 0.0687
  UMUMIY SILJISH (bu bo'linishning xususiyati): -0.0025
  bitta bahoning shovqini (SE): 0.0075
  tanlov optimizmi shu SILJISHDAN ORTIQCHA qismdir

=== 3. Nomzodlar soni va ortiqcha optimizm ===
       m   eng yaxshi val   uning haqiqiysi      xato   ortiqcha
       1           0.8538            0.8578   -0.0040    -0.0015
       5           0.8591            0.8594   -0.0004    +0.0021
      20           0.8591            0.8594   -0.0004    +0.0021
      50           0.8611            0.8604   +0.0007    +0.0032
     100           0.8611            0.8604   +0.0007    +0.0032
     200           0.8611            0.8604   +0.0007    +0.0032

=== 4. Nazariy chegara bilan solishtirish ===
       m   ortiqcha optimizm   SE*sqrt(2 ln m)
       5             +0.0021            0.0134
      20             +0.0021            0.0183
      50             +0.0032            0.0210
     100             +0.0032            0.0227
     200             +0.0032            0.0244
  nazariy qiymat - MUSTAQIL nomzodlar uchun YUQORI chegara;
  nomzodlar bir-biriga bog'liq bo'lgani uchun kuzatilgan
  optimizm undan pastroq chiqadi
  ⭐ Tanlovning o'zi optimizm yaratadi

"""Necha qator test kerak (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(0)
    X, y = make_classification(n_samples=42000, n_features=20,
                               n_informative=6, n_redundant=4,
                               flip_y=0.15, class_sep=0.9, random_state=0)
    X_tr, X_pul, y_tr, y_pul = train_test_split(
        X, y, test_size=40000, stratify=y, random_state=0)
    model = make_pipeline(StandardScaler(),
                          LogisticRegression(max_iter=2000)).fit(X_tr, y_tr)
    p_pul = model.predict_proba(X_pul)[:, 1]
    haqiqiy_acc = accuracy_score(y_pul, (p_pul >= 0.5).astype(int))
    haqiqiy_auc = roc_auc_score(y_pul, p_pul)

    print("=== 1. 'Haqiqiy' qiymatlar (40000 qatorda) ===")
    print(f"  aniqlik: {haqiqiy_acc:.4f}")
    print(f"  ROC AUC: {haqiqiy_auc:.4f}")

    print("\n=== 2. Kichik test to'plamlari nima ko'rsatadi ===")
    print(f"  {'n_test':>8} {'acc o_rtacha':>13} {'acc std':>9} "
          f"{'nazariy SE':>11} {'auc std':>9}")
    for n_test in [50, 100, 500, 2000, 10000]:
        acc_lar, auc_lar = [], []
        for _ in range(300):
            idx = rng.choice(len(y_pul), n_test, replace=False)
            acc_lar.append(accuracy_score(y_pul[idx],
                                          (p_pul[idx] >= 0.5).astype(int)))
            auc_lar.append(roc_auc_score(y_pul[idx], p_pul[idx]))
        acc_lar = np.array(acc_lar)
        nazariy = np.sqrt(haqiqiy_acc * (1 - haqiqiy_acc) / n_test)
        print(f"  {n_test:>8} {acc_lar.mean():>13.4f} {acc_lar.std():>9.4f} "
              f"{nazariy:>11.4f} {np.std(auc_lar):>9.4f}")

    print("\n=== 3. 0.01 lik farqni ko'rish uchun necha qator kerak ===")
    print(f"  {'farq':>8} {'kerakli n (95% ishonch)':>26}")
    for farq in [0.05, 0.02, 0.01, 0.005]:
        # ikki mustaqil namuna uchun: farq > 1.96 * sqrt(2) * SE
        p = haqiqiy_acc
        n = 2 * (1.96 ** 2) * p * (1 - p) / (farq ** 2)
        print(f"  {farq:>8.3f} {int(np.ceil(n)):>26}")

    print("\n=== 4. Amaliy xulosa ===")
    for n_test in [100, 1000, 10000]:
        se = np.sqrt(haqiqiy_acc * (1 - haqiqiy_acc) / n_test)
        print(f"  n={n_test:>6}: aniqlik {haqiqiy_acc:.3f} "
              f"+- {1.96 * se:.3f} (95%)")
    print("  ⭐ Farqni SE bilan solishtirmagan taqqoslash - bekor")


if __name__ == "__main__":
    main()

=== 1. 'Haqiqiy' qiymatlar (40000 qatorda) ===
  aniqlik: 0.7563
  ROC AUC: 0.8239

=== 2. Kichik test to'plamlari nima ko'rsatadi ===
    n_test  acc o_rtacha   acc std  nazariy SE   auc std
        50        0.7501    0.0649      0.0607    0.0656
       100        0.7532    0.0427      0.0429    0.0433
       500        0.7563    0.0192      0.0192    0.0190
      2000        0.7562    0.0093      0.0096    0.0088
     10000        0.7560    0.0034      0.0043    0.0035

=== 3. 0.01 lik farqni ko'rish uchun necha qator kerak ===
      farq    kerakli n (95% ishonch)
     0.050                        567
     0.020                       3541
     0.010                      14161
     0.005                      56644

=== 4. Amaliy xulosa ===
  n=   100: aniqlik 0.756 +- 0.084 (95%)
  n=  1000: aniqlik 0.756 +- 0.027 (95%)
  n= 10000: aniqlik 0.756 +- 0.008 (95%)
  ⭐ Farqni SE bilan solishtirmagan taqqoslash - bekor

"""Holdout, CV va CV+test dizaynlari (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import (StratifiedKFold, cross_val_score,
                                     train_test_split)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def nomzodlar():
    return {
        "logistik C=0.03": make_pipeline(
            StandardScaler(), LogisticRegression(C=0.03, max_iter=2000)),
        "logistik C=1": make_pipeline(
            StandardScaler(), LogisticRegression(C=1.0, max_iter=2000)),
        "KNN k=15": make_pipeline(StandardScaler(),
                                  KNeighborsClassifier(n_neighbors=15)),
        "RF 120": RandomForestClassifier(n_estimators=120, min_samples_leaf=3,
                                         random_state=0, n_jobs=1),
    }


def main() -> None:
    X, y = make_classification(n_samples=21000, n_features=25,
                               n_informative=8, n_redundant=5,
                               flip_y=0.15, class_sep=0.85, random_state=0)
    X_ish, X_haq, y_ish, y_haq = train_test_split(
        X, y, test_size=20000, stratify=y, random_state=0)
    print("=== 1. Sozlama ===")
    print(f"  mavjud ma'lumot: {len(y_ish)} qator")
    print(f"  'haqiqat' to'plami: {len(y_haq)} qator (faqat tekshirish uchun)")

    haqiqiy = {}
    for nom, m in nomzodlar().items():
        m.fit(X_ish, y_ish)
        haqiqiy[nom] = roc_auc_score(y_haq, m.predict_proba(X_haq)[:, 1])

    print("\n=== 2. Haqiqiy natijalar (20000 qatorda) ===")
    print(f"  {'nomzod':<18} {'haqiqiy AUC':>12}")
    for nom, b in sorted(haqiqiy.items(), key=lambda kv: -kv[1]):
        print(f"  {nom:<18} {b:>12.4f}")
    eng_yaxshi_haq = max(haqiqiy, key=haqiqiy.get)

    print("\n=== 3. Uch dizayn, 12 ta takrorlash ===")
    natija = {"holdout": [], "CV": [], "CV+test": []}
    togri = {"holdout": 0, "CV": 0, "CV+test": 0}
    for seed in range(12):
        # A. holdout
        Xa, Xb, ya, yb = train_test_split(X_ish, y_ish, test_size=0.3,
                                          stratify=y_ish, random_state=seed)
        ballar = {}
        for nom, m in nomzodlar().items():
            m.fit(Xa, ya)
            ballar[nom] = roc_auc_score(yb, m.predict_proba(Xb)[:, 1])
        tanlov = max(ballar, key=ballar.get)
        natija["holdout"].append(ballar[tanlov] - haqiqiy[tanlov])
        togri["holdout"] += int(tanlov == eng_yaxshi_haq)

        # B. CV (o'sha CV bahosi yakuniy deb e'lon qilinadi)
        cv = StratifiedKFold(5, shuffle=True, random_state=seed)
        ballar = {nom: cross_val_score(m, X_ish, y_ish, cv=cv,
                                       scoring="roc_auc").mean()
                  for nom, m in nomzodlar().items()}
        tanlov = max(ballar, key=ballar.get)
        natija["CV"].append(ballar[tanlov] - haqiqiy[tanlov])
        togri["CV"] += int(tanlov == eng_yaxshi_haq)

        # C. CV + alohida test
        Xa, Xb, ya, yb = train_test_split(X_ish, y_ish, test_size=0.3,
                                          stratify=y_ish, random_state=seed)
        ballar = {nom: cross_val_score(m, Xa, ya, cv=cv,
                                       scoring="roc_auc").mean()
                  for nom, m in nomzodlar().items()}
        tanlov = max(ballar, key=ballar.get)
        m = nomzodlar()[tanlov].fit(Xa, ya)
        test = roc_auc_score(yb, m.predict_proba(Xb)[:, 1])
        natija["CV+test"].append(test - haqiqiy[tanlov])
        togri["CV+test"] += int(tanlov == eng_yaxshi_haq)

    print(f"  {'dizayn':<12} {'o_rtacha xato':>14} {'std':>8} "
          f"{'togri tanlov':>14}")
    for nom in ["holdout", "CV", "CV+test"]:
        a = np.array(natija[nom])
        print(f"  {nom:<12} {a.mean():>+14.4f} {a.std():>8.4f} "
              f"{togri[nom]:>10}/12")

    print("\n=== 4. Talqin ===")
    print(f"  eng yaxshi model (haqiqatda): {eng_yaxshi_haq}")
    print("  'o_rtacha xato' musbat bo'lsa - dizayn OPTIMISTIK")
    print("  CV eng yaxshi tanlovni qiladi, lekin o'z bahosi optimistik")
    print("  CV+test: tanlov ham yaxshi, baho ham xolis")
    print("  ⭐ Tanlov va yakuniy baho boshqa to'plamda bo'lsin")


if __name__ == "__main__":
    main()

=== 1. Sozlama ===
  mavjud ma'lumot: 1000 qator
  'haqiqat' to'plami: 20000 qator (faqat tekshirish uchun)

=== 2. Haqiqiy natijalar (20000 qatorda) ===
  nomzod              haqiqiy AUC
  RF 120                   0.9024
  KNN k=15                 0.8867
  logistik C=1             0.8588
  logistik C=0.03          0.8585

=== 3. Uch dizayn, 12 ta takrorlash ===
  dizayn        o_rtacha xato      std   togri tanlov
  holdout             +0.0093   0.0155          5/12
  CV                  +0.0008   0.0020         12/12
  CV+test             -0.0012   0.0159         10/12

=== 4. Talqin ===
  eng yaxshi model (haqiqatda): RF 120
  'o_rtacha xato' musbat bo'lsa - dizayn OPTIMISTIK
  CV eng yaxshi tanlovni qiladi, lekin o'z bahosi optimistik
  CV+test: tanlov ham yaxshi, baho ham xolis
  ⭐ Tanlov va yakuniy baho boshqa to'plamda bo'lsin

Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=42)     # ⚠️
cross_val_score(model, X, y, cv=RepeatedStratifiedKFold(...))    # ✅

for variant in variantlar: print(baho(yte, ...))                 # ⚠️
# CV da tanlang, testni yakunda bir marta oching                 # ✅

# 500 ta variantdan eng yaxshisi: 0.87 "aniq"                    # ⚠️
# m=500 -> optimizm ~ 3.5*SE; alohida testda tasdiqlang          # ✅

print("A 0.842, B 0.836 -> A yaxshi")                            # ⚠️
print(f"farq {a-b:.4f}, SE {se:.4f} -> {'muhim' if ... }")       # ✅

for s in range(50): ...  # eng yaxshi chiqqanini yozib qo'yish   # ⚠️
# seed ni oldindan belgilang, natijani o'rtachalang              # ✅

train_test_split(X, y, test_size=0.1)   # n=80 test qatori       # ⚠️
# kamida bir necha ming, yoki takroriy CV                        # ✅

print(f"Model AUC: {grid.best_score_}")                          # ⚠️
print(f"Model AUC: {roc_auc_score(yte, ...)}")   # alohida test  # ✅

1.  # ikki asosiy xavf?
2.  # dispersiya nima bilan kamayadi?
3.  # optimizm nima bilan kamayadi?
4.  # g'olib la'nati formulasi?
5.  # m=100 da optimizm necha SE?
6.  # aniqlik SE formulasi?
7.  # 0.01 farq uchun taxminan necha qator?
8.  # uchta dizayn?
9.  # qaysi dizayn sozlash bo'lsa kerak?
10. # test to'plami necha marta ochiladi?
11. # 500 qatorli ma'lumotda qaysi dizayn?
12. # hisobotda m nima uchun kerak?

1.  Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=42)

2.  for variant in variantlar: print(baho(yte, ...))

3.  # 500 variantdan eng yaxshisi: 0.87 "aniq"

4.  print("A 0.842, B 0.836 -> A yaxshi")

5.  print(f"Model AUC: {grid.best_score_}")