18.10-dars: Modellarni taqqoslash

NOTO'G'RI:
  a = cross_val_score(A, X, y, cv=KFold(5, shuffle=True))
  b = cross_val_score(B, X, y, cv=KFold(5, shuffle=True))
  # turli bo'linishlar -> farqda BO'LINISH shovqini bor

TO'G'RI:
  CV = StratifiedKFold(5, shuffle=True, random_state=0)
  a = cross_val_score(A, X, y, cv=CV)
  b = cross_val_score(B, X, y, cv=CV)
  farq = a - b          # HAR FOLDDA bir xil ma'lumot

NIMA UCHUN: bo'linish shovqini ikkalasiga BIR XIL ta'sir
  qiladi va ayirishda QISQARADI

RepeatedStratifiedKFold(n_splits=5, n_repeats=10)
  -> 50 ta o'lchov, takror bo'yicha 10 ta o'rtacha

  farq_takrorlari = A_takrorlari - B_takrorlari   (10 ta son)

BAHOLASH:
  o'rtacha farq:  farq.mean()
  noaniqlik:      farq.std(ddof=1) / sqrt(10)
  oraliq:         mean +- 2 * noaniqlik

QAROR:
  oraliq nolni O'Z ICHIGA OLSA -> farq isbotlanmagan
  oraliq to'liq musbat bo'lsa  -> A ishonchli yaxshiroq

DIQQAT: bu ham taxminiy - foldlar korrelyatsiyali (18.3)

Dietterich (1998) taklifi: 5 marta 2-fold CV

  har takrorda 2 fold -> 2 ta farq
  5 takror -> 10 ta farq

  t = farq_1_1 / sqrt(mean(s_i^2))
  s_i^2 - i-takrordagi ikki farqning dispersiyasi

NIMA UCHUN 2 FOLD:
  o'quv to'plamlari KESISHMAYDI -> korrelyatsiya kamroq
  -> test kamroq optimistik

AMALIYOTDA:
  5x2cv - konservativ va ishonchli
  lekin o'quv hajmi yarmiga tushadi (bias)

10 ta modelni juft-juft taqqoslash = 45 ta test
alpha = 0.05 bo'lsa, kutilgan NOTO'G'RI "muhim" soni ~2.25

TUZATISH:
  Bonferroni: alpha / m      (konservativ)
  Holm: ketma-ket, kuchliroq
  Yoki: BITTA bazaviy model bilan solishtiring (m = k-1)

AMALIY YO'L:
  1. CV da eng yaxshi 2-3 nomzodni tanlang
  2. FAQAT ularni jiddiy taqqoslang
  3. Yakuniy tanlovni alohida testda tasdiqlang

STATISTIK ahamiyat: farq shovqindan kattami?
AMALIY ahamiyat:    farq FOYDA keltiradimi?

MISOL:
  AUC +0.003, statistik muhim (n = 500 000)
  lekin biznes ta'siri: yiliga 1200 dollar
  yangi modelni qo'llab-quvvatlash: yiliga 40 000 dollar
  -> AMALIY ahamiyat YO'Q

METRIKANI PULGA AYLANTIRING:
  "AUC +0.01 -> top-1000 da +12 ta firibgarlik
   -> yiliga 180 000 dollar"

QO'SHIMCHA MEZONLAR:
  bashorat tezligi, xotira, talqin, qo'llab-quvvatlash,
  qayta o'rgatish narxi, drift barqarorligi

TENG NATIJADA tanlash mezonlari (tartib bo'yicha):
  1. soddalik va talqin
  2. bashorat tezligi
  3. o'rgatish va qayta o'rgatish narxi
  4. bog'liqliklar soni (kutubxonalar)
  5. jamoaning tajribasi
  6. drift ga barqarorlik

QAROR JADVALI:
  model | CV ball | noaniqlik | tezlik | talqin | narx

import numpy as np
from scipy import stats
from sklearn.model_selection import RepeatedStratifiedKFold, cross_val_score

CV = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=0)
a = cross_val_score(A, X, y, cv=CV, scoring="roc_auc").reshape(10, 5)
b = cross_val_score(B, X, y, cv=CV, scoring="roc_auc").reshape(10, 5)

farq = a.mean(axis=1) - b.mean(axis=1)        # 10 ta takror
se = farq.std(ddof=1) / np.sqrt(len(farq))
print(f"{farq.mean():+.4f} +- {2 * se:.4f}")
t, p = stats.ttest_rel(a.mean(axis=1), b.mean(axis=1))
QOIDA: bir xil CV · juftlashgan farq · oraliq nolni qamrasa
       farq isbotlanmagan · amaliy ahamiyatni ayting

Juftlashgan: bir xil bo'linish, farqni ayirish
Noaniqlik: takrorlar orasidagi std / sqrt(R)
5x2cv: konservativ, o'quv to'plamlari kesishmaydi
Ko'p taqqoslash: bazaviy bilan solishtiring
Amaliy ahamiyat: metrikani pulga aylantiring

"""Bir xil bo'linish shovqinni qanday kamaytiradi (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def modellar():
    return (make_pipeline(StandardScaler(),
                          LogisticRegression(C=1.0, max_iter=2000)),
            make_pipeline(StandardScaler(),
                          LogisticRegression(C=0.05, max_iter=2000)))


def main() -> None:
    X, y = make_classification(n_samples=1500, n_features=30,
                               n_informative=8, n_redundant=8, flip_y=0.2,
                               class_sep=0.8, random_state=0)

    print("=== 1. Juftlashmagan taqqoslash (turli seed) ===")
    farqlar = []
    for s in range(40):
        A, B = modellar()
        a = cross_val_score(A, X, y,
                            cv=StratifiedKFold(5, shuffle=True,
                                               random_state=2 * s),
                            scoring="roc_auc").mean()
        b = cross_val_score(B, X, y,
                            cv=StratifiedKFold(5, shuffle=True,
                                               random_state=2 * s + 1),
                            scoring="roc_auc").mean()
        farqlar.append(a - b)
    juftlashmagan = np.array(farqlar)
    print(f"  o'rtacha farq: {juftlashmagan.mean():+.4f}")
    print(f"  farqning std i: {juftlashmagan.std(ddof=1):.4f}")
    print(f"  A yutgan holatlar: "
          f"{int((juftlashmagan > 0).sum())}/{len(juftlashmagan)}")

    print("\n=== 2. Juftlashgan taqqoslash (bir xil seed) ===")
    farqlar = []
    for s in range(40):
        cv = StratifiedKFold(5, shuffle=True, random_state=s)
        A, B = modellar()
        a = cross_val_score(A, X, y, cv=cv, scoring="roc_auc").mean()
        b = cross_val_score(B, X, y, cv=cv, scoring="roc_auc").mean()
        farqlar.append(a - b)
    juftlashgan = np.array(farqlar)
    print(f"  o'rtacha farq: {juftlashgan.mean():+.4f}")
    print(f"  farqning std i: {juftlashgan.std(ddof=1):.4f}")
    print(f"  A yutgan holatlar: "
          f"{int((juftlashgan > 0).sum())}/{len(juftlashgan)}")

    print("\n=== 3. Taqqoslash ===")
    print(f"  {'usul':<22} {'o_rtacha':>10} {'std':>9} {'nisbat':>8}")
    print(f"  {'juftlashmagan':<22} {juftlashmagan.mean():>+10.4f} "
          f"{juftlashmagan.std(ddof=1):>9.4f} {1.0:>7.1f}x")
    print(f"  {'juftlashgan':<22} {juftlashgan.mean():>+10.4f} "
          f"{juftlashgan.std(ddof=1):>9.4f} "
          f"{juftlashmagan.std(ddof=1) / juftlashgan.std(ddof=1):>7.1f}x")
    print("  juftlashgan farqning shovqini ancha kam")

    print("\n=== 4. Fold darajasida juftlashish ===")
    cv = StratifiedKFold(5, shuffle=True, random_state=0)
    A, B = modellar()
    a = cross_val_score(A, X, y, cv=cv, scoring="roc_auc")
    b = cross_val_score(B, X, y, cv=cv, scoring="roc_auc")
    print(f"  {'fold':>5} {'A':>9} {'B':>9} {'farq':>9}")
    for i, (aa, bb) in enumerate(zip(a, b), 1):
        print(f"  {i:>5} {aa:>9.4f} {bb:>9.4f} {aa - bb:>+9.4f}")
    print(f"  A std: {a.std(ddof=1):.4f}, B std: {b.std(ddof=1):.4f}")
    print(f"  FARQ std: {(a - b).std(ddof=1):.4f}   <- ancha kichik")
    korr = float(np.corrcoef(a, b)[0, 1])
    print(f"  A va B ballari korrelyatsiyasi: {korr:.4f}")
    print("  ⭐ Bir xil bo'linish shovqinni qisqartiradi")


if __name__ == "__main__":
    main()

=== 1. Juftlashmagan taqqoslash (turli seed) ===
  o'rtacha farq: -0.0005
  farqning std i: 0.0027
  A yutgan holatlar: 21/40

=== 2. Juftlashgan taqqoslash (bir xil seed) ===
  o'rtacha farq: -0.0007
  farqning std i: 0.0004
  A yutgan holatlar: 2/40

=== 3. Taqqoslash ===
  usul                     o_rtacha       std   nisbat
  juftlashmagan             -0.0005    0.0027     1.0x
  juftlashgan               -0.0007    0.0004     6.4x
  juftlashgan farqning shovqini ancha kam

=== 4. Fold darajasida juftlashish ===
   fold         A         B      farq
      1    0.8351    0.8361   -0.0011
      2    0.8397    0.8409   -0.0012
      3    0.8577    0.8646   -0.0069
      4    0.8474    0.8484   -0.0010
      5    0.8225    0.8203   +0.0022
  A std: 0.0132, B std: 0.0163
  FARQ std: 0.0033   <- ancha kichik
  A va B ballari korrelyatsiyasi: 0.9965
  ⭐ Bir xil bo'linish shovqinni qisqartiradi

"""Farq ishonchlimi (real numpy/scipy/sklearn)."""

import numpy as np
from scipy import stats
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RepeatedStratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def taqqosla(A, B, X, y, R: int = 10) -> dict:
    cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=R, random_state=0)
    a = cross_val_score(A, X, y, cv=cv, scoring="roc_auc").reshape(R, 5)
    b = cross_val_score(B, X, y, cv=cv, scoring="roc_auc").reshape(R, 5)
    ta, tb = a.mean(axis=1), b.mean(axis=1)
    farq = ta - tb
    se = float(farq.std(ddof=1) / np.sqrt(R))
    t, p = stats.ttest_rel(ta, tb)
    return {"A": float(ta.mean()), "B": float(tb.mean()),
            "farq": float(farq.mean()), "se": se,
            "past": float(farq.mean() - 2 * se),
            "yuqori": float(farq.mean() + 2 * se),
            "p": float(p)}


def main() -> None:
    X, y = make_classification(n_samples=1500, n_features=25,
                               n_informative=8, n_redundant=6, flip_y=0.2,
                               class_sep=0.8, random_state=0)

    juftliklar = {
        "C=1 va C=0.05": (
            make_pipeline(StandardScaler(),
                          LogisticRegression(C=1.0, max_iter=2000)),
            make_pipeline(StandardScaler(),
                          LogisticRegression(C=0.05, max_iter=2000))),
        "C=1 va C=0.9": (
            make_pipeline(StandardScaler(),
                          LogisticRegression(C=1.0, max_iter=2000)),
            make_pipeline(StandardScaler(),
                          LogisticRegression(C=0.9, max_iter=2000))),
        "boosting va logistik": (
            HistGradientBoostingClassifier(max_iter=200,
                                           early_stopping=False,
                                           random_state=0),
            make_pipeline(StandardScaler(),
                          LogisticRegression(max_iter=2000))),
    }

    print("=== 1. Uch juftlik ===")
    print(f"  {'juftlik':<24} {'A':>8} {'B':>8} {'farq':>9} {'SE':>8}")
    natijalar = {}
    for nom, (A, B) in juftliklar.items():
        d = taqqosla(A, B, X, y)
        natijalar[nom] = d
        print(f"  {nom:<24} {d['A']:>8.4f} {d['B']:>8.4f} "
              f"{d['farq']:>+9.4f} {d['se']:>8.4f}")

    print("\n=== 2. Ishonch oraliqlari ===")
    print(f"  {'juftlik':<24} {'95% oraliq':>24} {'nolni qamraydimi':>18}")
    for nom, d in natijalar.items():
        oraliq = f"[{d['past']:+.4f}, {d['yuqori']:+.4f}]"
        qamraydi = d["past"] <= 0 <= d["yuqori"]
        print(f"  {nom:<24} {oraliq:>24} {str(qamraydi):>18}")

    print("\n=== 3. Juftlashgan t-test ===")
    print(f"  {'juftlik':<24} {'p-qiymat':>10} {'xulosa':<28}")
    for nom, d in natijalar.items():
        xulosa = ("farq ishonchli" if d["p"] < 0.05
                  else "farq isbotlanmagan")
        print(f"  {nom:<24} {d['p']:>10.4f} {xulosa:<28}")

    print("\n=== 4. Takrorlar soni noaniqlikka ta'siri ===")
    A, B = juftliklar["C=1 va C=0.05"]
    print(f"  {'R':>4} {'farq':>9} {'SE':>9} {'oraliq kengligi':>17}")
    for R in [2, 5, 10, 20]:
        d = taqqosla(A, B, X, y, R=R)
        print(f"  {R:>4} {d['farq']:>+9.4f} {d['se']:>9.4f} "
              f"{4 * d['se']:>17.4f}")
    print("  ⭐ Oraliq nolni qamrasa, farq isbotlanmagan")


if __name__ == "__main__":
    main()

=== 1. Uch juftlik ===
  juftlik                         A        B      farq       SE
  C=1 va C=0.05              0.8311   0.8314   -0.0003   0.0002
  C=1 va C=0.9               0.8311   0.8311   +0.0000   0.0000
  boosting va logistik       0.8440   0.8311   +0.0129   0.0011

=== 2. Ishonch oraliqlari ===
  juftlik                                95% oraliq   nolni qamraydimi
  C=1 va C=0.05                  [-0.0007, +0.0002]               True
  C=1 va C=0.9                   [-0.0000, +0.0000]               True
  boosting va logistik           [+0.0107, +0.0150]              False

=== 3. Juftlashgan t-test ===
  juftlik                    p-qiymat xulosa
  C=1 va C=0.05                0.3039 farq isbotlanmagan
  C=1 va C=0.9                 1.0000 farq isbotlanmagan
  boosting va logistik         0.0000 farq ishonchli

=== 4. Takrorlar soni noaniqlikka ta'siri ===
     R      farq        SE   oraliq kengligi
     2   -0.0008    0.0010            0.0039
     5   -0.0005    0.0004            0.0016
    10   -0.0003    0.0002            0.0009
    20   -0.0001    0.0001            0.0005
  ⭐ Oraliq nolni qamrasa, farq isbotlanmagan

"""Konservativ test va Bonferroni tuzatishi (real numpy/scipy/sklearn)."""

import itertools

import numpy as np
from scipy import stats
from sklearn.datasets import make_classification
from sklearn.ensemble import (HistGradientBoostingClassifier,
                              RandomForestClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def besh_ikki_cv(A, B, X, y):
    """Dietterich 5x2cv juftlashgan t-testi."""
    farqlar = []
    for takror in range(5):
        cv = StratifiedKFold(2, shuffle=True, random_state=takror)
        juft = []
        for tr, te in cv.split(X, y):
            a = roc_auc_score(y[te], A.fit(X[tr], y[tr])
                              .predict_proba(X[te])[:, 1])
            b = roc_auc_score(y[te], B.fit(X[tr], y[tr])
                              .predict_proba(X[te])[:, 1])
            juft.append(a - b)
        farqlar.append(juft)
    farqlar = np.array(farqlar)                      # (5, 2)
    s2 = farqlar.var(axis=1, ddof=1)                 # har takror dispersiyasi
    t = farqlar[0, 0] / np.sqrt(s2.mean())
    p = 2 * (1 - stats.t.cdf(abs(t), df=5))
    return float(farqlar.mean()), float(t), float(p)


def main() -> None:
    X, y = make_classification(n_samples=1500, n_features=25,
                               n_informative=8, n_redundant=6, flip_y=0.2,
                               class_sep=0.8, random_state=0)

    modellar = {
        "logistik": make_pipeline(StandardScaler(),
                                  LogisticRegression(max_iter=2000)),
        "KNN": make_pipeline(StandardScaler(),
                             KNeighborsClassifier(n_neighbors=25)),
        "RF": RandomForestClassifier(n_estimators=150, min_samples_leaf=5,
                                     random_state=0, n_jobs=1),
        "boosting": HistGradientBoostingClassifier(max_iter=200,
                                                   early_stopping=False,
                                                   random_state=0),
    }

    print("=== 1. 5x2cv: har juftlik ===")
    juftliklar = list(itertools.combinations(modellar, 2))
    print(f"  {'A':<10} {'B':<10} {'farq':>9} {'t':>8} {'p':>9}")
    natijalar = {}
    for na, nb in juftliklar:
        farq, t, p = besh_ikki_cv(modellar[na], modellar[nb], X, y)
        natijalar[(na, nb)] = (farq, t, p)
        print(f"  {na:<10} {nb:<10} {farq:>+9.4f} {t:>8.3f} {p:>9.4f}")

    print("\n=== 2. Ko'p taqqoslash tuzatishi ===")
    m = len(juftliklar)
    alpha = 0.05
    print(f"  taqqoslashlar soni: {m}")
    print(f"  tuzatilmagan alpha: {alpha}")
    print(f"  Bonferroni alpha: {alpha / m:.5f}")
    print(f"  {'juftlik':<22} {'p':>9} {'tuzatilmagan':>14} "
          f"{'Bonferroni':>12}")
    for (na, nb), (_, _, p) in natijalar.items():
        print(f"  {na + ' vs ' + nb:<22} {p:>9.4f} "
              f"{str(p < alpha):>14} {str(p < alpha / m):>12}")
    tuzatilmagan = sum(1 for v in natijalar.values() if v[2] < alpha)
    bonf = sum(1 for v in natijalar.values() if v[2] < alpha / m)
    print(f"  'muhim' deb topilgan: {tuzatilmagan} -> {bonf}")

    print("\n=== 3. Holm tuzatishi (kuchliroq) ===")
    tartib = sorted(natijalar.items(), key=lambda kv: kv[1][2])
    print(f"  {'o_rin':>6} {'juftlik':<22} {'p':>9} {'chegara':>10} "
          f"{'muhim':>7}")
    rad_etildi = True
    for i, ((na, nb), (_, _, p)) in enumerate(tartib):
        chegara = alpha / (m - i)
        muhim = rad_etildi and p < chegara
        if not muhim:
            rad_etildi = False
        print(f"  {i + 1:>6} {na + ' vs ' + nb:<22} {p:>9.4f} "
              f"{chegara:>10.5f} {str(muhim):>7}")

    print("\n=== 4. Bazaviy bilan solishtirish (m = k-1) ===")
    bazaviy = "logistik"
    boshqalar = [n for n in modellar if n != bazaviy]
    print(f"  bazaviy: {bazaviy}, taqqoslashlar: {len(boshqalar)}")
    print(f"  Bonferroni alpha: {alpha / len(boshqalar):.5f}")
    print(f"  {'model':<12} {'farq':>9} {'p':>9} {'muhim':>7}")
    for nom in boshqalar:
        farq, t, p = besh_ikki_cv(modellar[nom], modellar[bazaviy], X, y)
        print(f"  {nom:<12} {farq:>+9.4f} {p:>9.4f} "
              f"{str(p < alpha / len(boshqalar)):>7}")
    print("  ⭐ Bazaviy bilan solishtirish taqqoslashlar sonini kamaytiradi")


if __name__ == "__main__":
    main()

=== 1. 5x2cv: har juftlik ===
  A          B               farq        t         p
  logistik   KNN          -0.0119   -0.514    0.6293
  logistik   RF           -0.0215   -1.117    0.3148
  logistik   boosting     -0.0061    1.027    0.3516
  KNN        RF           -0.0096   -1.133    0.3087
  KNN        boosting     +0.0059    1.834    0.1261
  RF         boosting     +0.0155    2.210    0.0781

=== 2. Ko'p taqqoslash tuzatishi ===
  taqqoslashlar soni: 6
  tuzatilmagan alpha: 0.05
  Bonferroni alpha: 0.00833
  juftlik                        p   tuzatilmagan   Bonferroni
  logistik vs KNN           0.6293          False        False
  logistik vs RF            0.3148          False        False
  logistik vs boosting      0.3516          False        False
  KNN vs RF                 0.3087          False        False
  KNN vs boosting           0.1261          False        False
  RF vs boosting            0.0781          False        False
  'muhim' deb topilgan: 0 -> 0

=== 3. Holm tuzatishi (kuchliroq) ===
   o_rin juftlik                        p    chegara   muhim
       1 RF vs boosting            0.0781    0.00833   False
       2 KNN vs boosting           0.1261    0.01000   False
       3 KNN vs RF                 0.3087    0.01250   False
       4 logistik vs RF            0.3148    0.01667   False
       5 logistik vs boosting      0.3516    0.02500   False
       6 logistik vs KNN           0.6293    0.05000   False

=== 4. Bazaviy bilan solishtirish (m = k-1) ===
  bazaviy: logistik, taqqoslashlar: 3
  Bonferroni alpha: 0.01667
  model             farq         p   muhim
  KNN            +0.0119    0.6293   False
  RF             +0.0215    0.3148   False
  boosting       +0.0061    0.3516   False
  ⭐ Bazaviy bilan solishtirish taqqoslashlar sonini kamaytiradi

"""Metrikani foydaga aylantirish (real numpy/sklearn)."""

import io
import pickle

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import (HistGradientBoostingClassifier,
                              RandomForestClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RepeatedStratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    X, y = make_classification(n_samples=4000, n_features=25,
                               n_informative=8, n_redundant=6,
                               weights=[0.95, 0.05], flip_y=0.06,
                               class_sep=0.85, random_state=0)
    cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=4, random_state=0)

    modellar = {
        "logistik": make_pipeline(StandardScaler(),
                                  LogisticRegression(max_iter=2000)),
        "RF": RandomForestClassifier(n_estimators=150, min_samples_leaf=3,
                                     random_state=0, n_jobs=1),
        "boosting": HistGradientBoostingClassifier(max_iter=200,
                                                   early_stopping=False,
                                                   random_state=0),
    }

    print("=== 1. CV natijalari ===")
    ballar = {}
    print(f"  {'model':<12} {'AP':>9} {'takror std':>12} {'SE':>9}")
    for nom, m in modellar.items():
        b = cross_val_score(m, X, y, cv=cv,
                            scoring="average_precision").reshape(4, 5)
        t = b.mean(axis=1)
        ballar[nom] = t
        se = float(t.std(ddof=1) / np.sqrt(4))
        print(f"  {nom:<12} {t.mean():>9.4f} {t.std(ddof=1):>12.4f} "
              f"{se:>9.4f}")

    print("\n=== 2. Bazaviy (logistik) bilan juftlashgan farq ===")
    asos = ballar["logistik"]
    print(f"  {'model':<12} {'farq':>9} {'SE':>9} {'95% oraliq':>22}")
    for nom in ["RF", "boosting"]:
        farq = ballar[nom] - asos
        se = float(farq.std(ddof=1) / np.sqrt(len(farq)))
        oraliq = f"[{farq.mean() - 2 * se:+.4f}, {farq.mean() + 2 * se:+.4f}]"
        print(f"  {nom:<12} {farq.mean():>+9.4f} {se:>9.4f} {oraliq:>22}")

    print("\n=== 3. Model murakkabligi va hajmi ===")
    # VAQT o'lchovlari mashinaga bog'liq va takrorlanmaydi,
    # shuning uchun DETERMINISTIK ko'rsatkichlarni ishlatamiz
    print(f"  {'model':<12} {'parametr/tugun':>16} {'hajm (KB)':>11} "
          f"{'nisbiy hajm':>13}")
    murakkablik = {}
    for nom, m in modellar.items():
        m.fit(X, y)
        oxirgi = m[-1] if hasattr(m, "steps") else m
        if hasattr(oxirgi, "estimators_"):
            tugunlar = int(sum(t.tree_.node_count
                               for t in oxirgi.estimators_))
        elif hasattr(oxirgi, "coef_"):
            tugunlar = int(oxirgi.coef_.size + 1)
        else:
            tugunlar = int(sum(len(bosqich) for bosqich
                               in oxirgi._predictors))
        buf = io.BytesIO()
        pickle.dump(m, buf, protocol=pickle.HIGHEST_PROTOCOL)
        murakkablik[nom] = (tugunlar, buf.tell() / 1024)
    asos_kb = murakkablik["logistik"][1]
    for nom, (tugunlar, kb) in murakkablik.items():
        print(f"  {nom:<12} {tugunlar:>16} {kb:>11.1f} "
              f"{kb / asos_kb:>12.0f}x")
    print("  (bashorat narxi tugunlar soniga taxminan proporsional)")

    print("\n=== 4. Qaror jadvali ===")
    # AP +0.01 -> top-1000 da qo'shimcha topilgan musbatlar
    musbat_ulush = float(y.mean())
    print(f"  musbat ulushi: {musbat_ulush:.2%}")
    print(f"  {'model':<12} {'AP':>8} {'farq':>9} {'muhimmi':>9} "
          f"{'hajm x':>9} {'qaror':<22}")
    for nom in modellar:
        t = ballar[nom]
        if nom == "logistik":
            farq, muhim, qaror = 0.0, "-", "bazaviy"
        else:
            d = t - asos
            se = float(d.std(ddof=1) / np.sqrt(len(d)))
            farq = float(d.mean())
            muhim = str(abs(farq) > 2 * se)
            qaror = ("qabul qilish" if farq > 2 * se and farq > 0.01
                     else "amaliy ahamiyat yo'q")
        print(f"  {nom:<12} {t.mean():>8.4f} {farq:>+9.4f} {muhim:>9} "
              f"{murakkablik[nom][1] / asos_kb:>8.0f}x {qaror:<22}")
    print("  qoida: farq 2*SE dan katta VA amaliy chegaradan katta bo'lsin")
    print("  ⭐ Statistik ahamiyat - kirish sharti, yakuniy mezon emas")


if __name__ == "__main__":
    main()

=== 1. CV natijalari ===
  model               AP   takror std        SE
  logistik        0.4414       0.0018    0.0009
  RF              0.5584       0.0046    0.0023
  boosting        0.5479       0.0089    0.0044

=== 2. Bazaviy (logistik) bilan juftlashgan farq ===
  model             farq        SE             95% oraliq
  RF             +0.1171    0.0025     [+0.1120, +0.1222]
  boosting       +0.1065    0.0051     [+0.0964, +0.1167]

=== 3. Model murakkabligi va hajmi ===
  model          parametr/tugun   hajm (KB)   nisbiy hajm
  logistik                   26         1.8            1x
  RF                      44100      3488.2         1915x
  boosting                  200       742.4          408x
  (bashorat narxi tugunlar soniga taxminan proporsional)

=== 4. Qaror jadvali ===
  musbat ulushi: 7.72%
  model              AP      farq   muhimmi    hajm x qaror
  logistik       0.4414   +0.0000         -        1x bazaviy
  RF             0.5584   +0.1171      True     1915x qabul qilish
  boosting       0.5479   +0.1065      True      408x qabul qilish
  qoida: farq 2*SE dan katta VA amaliy chegaradan katta bo'lsin
  ⭐ Statistik ahamiyat - kirish sharti, yakuniy mezon emas

a = cross_val_score(A, X, y, cv=KFold(5, shuffle=True))
b = cross_val_score(B, X, y, cv=KFold(5, shuffle=True))          # ⚠️
CV = StratifiedKFold(5, shuffle=True, random_state=0)            # ✅

print(f"A {a.mean():.3f} > B {b.mean():.3f}")                    # ⚠️
print(f"farq {d.mean():+.4f} +- {2 * se:.4f}")                   # ✅

for a, b in itertools.combinations(modellar, 2): test(a, b)      # ⚠️
for m in modellar: test(m, bazaviy)      # + Bonferroni          # ✅

if p < 0.05: ishga_tushir(yangi_model)                           # ⚠️
if p < 0.05 and farq > amaliy_chegara: ishga_tushir(...)         # ✅

for m in modellar: print(roc_auc_score(yte, m.predict_proba(Xte)))  # ⚠️
# CV da taqqoslang, g'olibni testda BIR MARTA tasdiqlang           # ✅

farq = a.mean() - b.mean()    # noaniqlik yo'qoladi              # ⚠️
farq = (a - b)                # fold/takror darajasida           # ✅

# "boosting 0.003 yaxshi -> uni olamiz"                          # ⚠️
# 40x sekin va 0.003 yaxshi -> oqlanadimi?                       # ✅

1.  # juftlashgan taqqoslash nima?
2.  # nima uchun shovqin kamayadi?
3.  # farq SE si qanday hisoblanadi?
4.  # oraliq nolni qamrasa nima?
5.  # 5x2cv nima uchun konservativ?
6.  # 10 modelda necha juftlik?
7.  # Bonferroni nima qiladi?
8.  # Holm Bonferronidan nimasi bilan farq qiladi?
9.  # bazaviy bilan solishtirish foydasi?
10. # statistik va amaliy ahamiyat farqi?
11. # teng natijada nima tanlanadi?
12. # bir standart xato qoidasi?

1.  a = cross_val_score(A, X, y, cv=KFold(5, shuffle=True))
    b = cross_val_score(B, X, y, cv=KFold(5, shuffle=True))

2.  print(f"A {a.mean():.3f} > B {b.mean():.3f}")

3.  for a, b in itertools.combinations(modellar, 2): test(a, b)

4.  if p < 0.05: ishga_tushir(yangi_model)

5.  farq = a.mean() - b.mean()