18.12-dars: Amaliyot — to'liq baholash va sozlash

1. VALIDATSIYA DIZAYNI
   vaqt? guruh? stratifikatsiya? necha fold? takror?
   test to'plamini AJRATIB QULFLANG

2. METRIKA
   "chiqish bilan nima qilinadi?" -> sozlash metrikasi
   + kuzatiladigan metrikalar ro'yxati

3. BAZAVIY
   Dummy + eng sodda real model
   CV ball va SE ni o'lchang -> QAROR CHEGARASI

4. SOZLASH
   erta to'xtash -> tasodifiy qidiruv -> aniqlashtirish
   byudjetni oldindan belgilang

5. TAQQOSLASH
   juftlashgan, bir xil CV, ishonch oraliqlari
   bazaviy bilan solishtiring

6. YAKUNIY BAHO
   nested CV yoki yopiq testni BIR MARTA oching

7. HISOBOT
   dizayn, T, ballar, oraliqlar, cheklovlar

ODATIY LOYIHA (100 soat):

  ma'lumot va belgilar        50 soat   <- eng ko'p foyda
  validatsiya dizayni          5 soat   <- eng muhim
  bazaviy va diagnostika      10 soat
  sozlash                     15 soat
  taqqoslash va tasdiqlash    10 soat
  hisobot va hujjat           10 soat

XATO TAQSIMOT:
  sozlash                     60 soat   <- +0.01 beradi
  belgilar                    10 soat   <- +0.05 berardi

QOIDA: sozlashga umumiy vaqtning 20% idan ko'pini bermang

MINIMAL USTUNLAR:
  n, sana, g'oya, o'zgarish, CV ball (3 xona), SE,
  qaror (qabul/rad), sabab

QO'SHIMCHA:
  qaysi metrika, qancha vaqt, qaysi seed

FOYDASI:
  - T ni bilasiz -> optimizmni baholaysiz
  - takroriy g'oyalarni oldini olasiz
  - hisobotga tayyor material

SHAKL: CSV, markdown jadval yoki MLflow/W&B

STRUKTURA:

1. Vazifa va metrika
   nima bashorat qilinadi, chiqish qanday ishlatiladi

2. Validatsiya dizayni
   strategiya, fold soni, takrorlar, test to'plami

3. Natijalar
   bazaviy, yakuniy model, ishonch oralig'i
   alohida test natijasi

4. Jarayon
   sinalgan tajribalar soni (T), optimizm bahosi

5. Cheklovlar
   ma'lumot davri, qamrov, drift xavfi, kutilgan ishlash

6. Tavsiya
   qabul qilish/rad etish, keyingi qadamlar

[ ] Validatsiya strategiyasi vazifaga mos (vaqt/guruh)
[ ] Test to'plami ish boshida ajratilgan va yopiq
[ ] Metrika boshida tanlangan va o'zgarmagan
[ ] Barcha tayyorlash Pipeline ichida
[ ] Bazaviy model bor va SE o'lchangan
[ ] Qaror chegarasi belgilangan (2*SE)
[ ] Sozlash byudjeti oldindan belgilangan
[ ] best_score_ hisobotga YOZILMAGAN
[ ] Taqqoslash juftlashgan va oraliqlar berilgan
[ ] Tajribalar soni (T) yozilgan
[ ] Test bir marta ochilgan
[ ] Cheklovlar hujjatlashtirilgan

import numpy as np
from sklearn.model_selection import (GroupShuffleSplit,
                                     RepeatedStratifiedKFold,
                                     cross_val_score)

# 1. dizayn: test yopiladi
tr, te = next(GroupShuffleSplit(1, test_size=0.25, random_state=0)
              .split(X, y, groups=g))
CV = RepeatedStratifiedKFold(n_splits=5, n_repeats=6, random_state=0)

# 2. bazaviy va SE -> qaror chegarasi
b = cross_val_score(bazaviy, X[tr], y[tr], cv=CV,
                    scoring="average_precision").reshape(6, 5).mean(axis=1)
se = b.std(ddof=1) / np.sqrt(6)
chegara = 2 * se

# 3. har tajriba jurnalga
jurnal.append({"n": n, "gooya": ..., "ball": round(ball, 3),
               "se": round(se, 4), "qabul": ball - joriy > chegara})

# 4. yakunda test BIR MARTA
QOIDA: dizayn birinchi · metrikani o'zgartirma · chegara qo'y ·
       T ni sana · testni bir marta och

1 dizayn -> 2 metrika -> 3 bazaviy -> 4 sozlash
-> 5 taqqoslash -> 6 yakuniy baho -> 7 hisobot
Byudjet: belgilar 50%, sozlash < 20%
Topshirishda: jurnal + hisobot + T + cheklovlar

"""1-3 qadamlar (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (GroupKFold, GroupShuffleSplit,
                                     StratifiedKFold, cross_val_score)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 7, mijozlar: int = 1200) -> pd.DataFrame:
    """Obuna bekor qilish: har mijozning bir necha oylik yozuvi."""
    rng = np.random.default_rng(seed)
    qatorlar = []
    for m in range(mijozlar):
        imzo = rng.normal(0, 1)
        tarif = rng.choice(["asosiy", "kengaytirilgan", "korporativ"])
        tq = {"asosiy": 0.6, "kengaytirilgan": 0.0, "korporativ": -0.7}[tarif]
        oylar = int(rng.integers(4, 15))
        faollik = rng.gamma(3, 6)
        for oy in range(oylar):
            sessiyalar = max(0.0, faollik + rng.normal(0, 3) - 0.25 * oy)
            qollab = rng.poisson(0.5)
            tolov_kechikishi = rng.poisson(0.3)
            kuch = (-2.4 + tq - 0.06 * sessiyalar + 0.45 * qollab
                    + 0.7 * tolov_kechikishi + 0.8 * imzo + 0.05 * oy)
            bekor = int(rng.random() < 1 / (1 + np.exp(-kuch)))
            qatorlar.append([m, oy, tarif, sessiyalar, qollab,
                             tolov_kechikishi, bekor])
            if bekor:
                break
    return pd.DataFrame(qatorlar, columns=["mijoz", "oy", "tarif",
                                           "sessiyalar", "qollab",
                                           "kechikish", "bekor"])


def main() -> None:
    df = yarat()
    y = df["bekor"].to_numpy()
    guruh = df["mijoz"].to_numpy()
    sonli = ["oy", "sessiyalar", "qollab", "kechikish"]

    print("=== 1. Ma'lumot va tuzilma ===")
    print(f"  {len(df)} qator, {df['mijoz'].nunique()} mijoz")
    print(f"  mijozga o'rtacha {len(df) / df['mijoz'].nunique():.1f} qator")
    print(f"  bekor qilish ulushi: {y.mean():.2%}")
    print(f"  duplikatlar: {int(df.duplicated().sum())}")

    print("\n=== 2. Validatsiya strategiyasi tanlovi ===")
    model = make_pipeline(StandardScaler(),
                          LogisticRegression(max_iter=2000))
    oddiy = cross_val_score(model, df[sonli], y,
                            cv=StratifiedKFold(5, shuffle=True,
                                               random_state=0),
                            scoring="average_precision")
    guruhli = cross_val_score(model, df[sonli], y, cv=GroupKFold(5),
                              groups=guruh, scoring="average_precision")
    print(f"  {'strategiya':<20} {'AP':>8} {'std':>8}")
    print(f"  {'StratifiedKFold':<20} {oddiy.mean():>8.4f} "
          f"{oddiy.std():>8.4f}")
    print(f"  {'GroupKFold':<20} {guruhli.mean():>8.4f} "
          f"{guruhli.std():>8.4f}")
    print(f"  farq: {oddiy.mean() - guruhli.mean():+.4f}")
    print("  QAROR: GroupKFold (bir mijozning ko'p oyi bor)")

    print("\n=== 3. Test to'plamini ajratish va QULFLASH ===")
    gss = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=0)
    tr, te = next(gss.split(df, y, groups=guruh))
    print(f"  ish to'plami: {len(tr)} qator, "
          f"{len(np.unique(guruh[tr]))} mijoz")
    print(f"  TEST (yopiq): {len(te)} qator, "
          f"{len(np.unique(guruh[te]))} mijoz")
    print(f"  kesishish: {len(np.intersect1d(guruh[tr], guruh[te]))}")

    print("\n=== 4. Metrika tanlovi va bazaviy ===")
    print("  chiqish: har oy xavf ro'yxati -> top-N mijozga qo'ng'iroq")
    print("  => sozlash metrikasi: average_precision")
    print("  => kuzatiladi: roc_auc, neg_log_loss")
    dftr, ytr, gtr = df.iloc[tr], y[tr], guruh[tr]
    cv = GroupKFold(5)
    print(f"  {'model':<28} {'AP':>8} {'AUC':>8}")
    for nom, m in [("Dummy (prior)", DummyClassifier(strategy="prior")),
                   ("logistik", model),
                   ("boosting", HistGradientBoostingClassifier(
                       max_iter=200, early_stopping=False, random_state=0))]:
        ap = cross_val_score(m, dftr[sonli], ytr, cv=cv, groups=gtr,
                             scoring="average_precision").mean()
        auc = cross_val_score(m, dftr[sonli], ytr, cv=cv, groups=gtr,
                              scoring="roc_auc").mean()
        print(f"  {nom:<28} {ap:>8.4f} {auc:>8.4f}")
    print(f"  bazaviy daraja (musbat ulushi): {ytr.mean():.4f}")
    print("  ⭐ Dizayn va metrika - modeldan OLDIN")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot va tuzilma ===
  8088 qator, 1200 mijoz
  mijozga o'rtacha 6.7 qator
  bekor qilish ulushi: 7.07%
  duplikatlar: 0

=== 2. Validatsiya strategiyasi tanlovi ===
  strategiya                 AP      std
  StratifiedKFold        0.1449   0.0157
  GroupKFold             0.1459   0.0072
  farq: -0.0009
  QAROR: GroupKFold (bir mijozning ko'p oyi bor)

=== 3. Test to'plamini ajratish va QULFLASH ===
  ish to'plami: 6135 qator, 900 mijoz
  TEST (yopiq): 1953 qator, 300 mijoz
  kesishish: 0

=== 4. Metrika tanlovi va bazaviy ===
  chiqish: har oy xavf ro'yxati -> top-N mijozga qo'ng'iroq
  => sozlash metrikasi: average_precision
  => kuzatiladi: roc_auc, neg_log_loss
  model                              AP      AUC
  Dummy (prior)                  0.0703   0.5000
  logistik                       0.1468   0.6926
  boosting                       0.1112   0.6152
  bazaviy daraja (musbat ulushi): 0.0703
  ⭐ Dizayn va metrika - modeldan OLDIN

"""3-4 qadamlar: chegara va sozlash (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from scipy.stats import loguniform, randint
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import (GroupKFold, GroupShuffleSplit,
                                     RandomizedSearchCV, cross_val_score)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder


def yarat(seed: int = 7, mijozlar: int = 1200) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    qatorlar = []
    for m in range(mijozlar):
        imzo = rng.normal(0, 1)
        tarif = rng.choice(["asosiy", "kengaytirilgan", "korporativ"])
        tq = {"asosiy": 0.6, "kengaytirilgan": 0.0, "korporativ": -0.7}[tarif]
        oylar = int(rng.integers(4, 15))
        faollik = rng.gamma(3, 6)
        for oy in range(oylar):
            sessiyalar = max(0.0, faollik + rng.normal(0, 3) - 0.25 * oy)
            qollab = rng.poisson(0.5)
            kechikish = rng.poisson(0.3)
            kuch = (-2.4 + tq - 0.06 * sessiyalar + 0.45 * qollab
                    + 0.7 * kechikish + 0.8 * imzo + 0.05 * oy)
            bekor = int(rng.random() < 1 / (1 + np.exp(-kuch)))
            qatorlar.append([m, oy, tarif, sessiyalar, qollab, kechikish,
                             bekor])
            if bekor:
                break
    return pd.DataFrame(qatorlar, columns=["mijoz", "oy", "tarif",
                                           "sessiyalar", "qollab",
                                           "kechikish", "bekor"])


SONLI = ["oy", "sessiyalar", "qollab", "kechikish"]
KATEGORIYA = ["tarif"]


def quvur(**kw):
    tayyor = ColumnTransformer([
        ("s", "passthrough", SONLI),
        ("k", OneHotEncoder(handle_unknown="ignore", sparse_output=False),
         KATEGORIYA)])
    return Pipeline([("t", tayyor),
                     ("m", HistGradientBoostingClassifier(
                         max_iter=250, early_stopping=False,
                         random_state=0, **kw))])


def main() -> None:
    df = yarat()
    y = df["bekor"].to_numpy()
    guruh = df["mijoz"].to_numpy()
    tr, te = next(GroupShuffleSplit(1, test_size=0.25, random_state=0)
                  .split(df, y, groups=guruh))
    dftr, ytr, gtr = df.iloc[tr], y[tr], guruh[tr]
    cv = GroupKFold(5)

    def baho(model):
        b = cross_val_score(model, dftr, ytr, cv=cv, groups=gtr,
                            scoring="average_precision")
        return float(b.mean()), float(b.std(ddof=1) / np.sqrt(len(b)))

    print("=== 1. Bazaviy va qaror chegarasi ===")
    asos, se = baho(quvur())
    print(f"  bazaviy AP: {asos:.4f}")
    print(f"  foldlar bo'yicha SE: {se:.4f}")
    print(f"  QAROR CHEGARASI (2*SE): {2 * se:.4f}")
    print("  bundan kichik yaxshilanish RAD ETILADI")

    print("\n=== 2. Belgi g'oyalari (guruhlab sinaladi) ===")
    def belgilar_qosh(d: pd.DataFrame) -> pd.DataFrame:
        d = d.copy()
        g = d.groupby("mijoz")
        d["sessiya_oynasi"] = g["sessiyalar"].transform(
            lambda s: s.shift(1).rolling(3, min_periods=1).mean())
        # birinchi oyda tarix yo'q -> joriy qiymat bilan to'ldiramiz
        d["sessiya_oynasi"] = d["sessiya_oynasi"].fillna(d["sessiyalar"])
        # maxraj nolga tushishi mumkin -> cheklaymiz (aks holda inf)
        d["sessiya_nisbati"] = (d["sessiyalar"]
                                / d["sessiya_oynasi"].clip(lower=0.5))
        d["qollab_jami"] = g["qollab"].transform(
            lambda s: s.shift(1).cumsum()).fillna(0.0)
        return d

    df2 = belgilar_qosh(df)
    dftr2 = df2.iloc[tr]
    yangi_sonli = SONLI + ["sessiya_oynasi", "sessiya_nisbati",
                           "qollab_jami"]

    def quvur2(**kw):
        tayyor = ColumnTransformer([
            ("s", "passthrough", yangi_sonli),
            ("k", OneHotEncoder(handle_unknown="ignore",
                                sparse_output=False), KATEGORIYA)])
        return Pipeline([("t", tayyor),
                         ("m", HistGradientBoostingClassifier(
                             max_iter=250, early_stopping=False,
                             random_state=0, **kw))])

    b2 = cross_val_score(quvur2(), dftr2, ytr, cv=cv, groups=gtr,
                         scoring="average_precision")
    yangi_ball = float(b2.mean())
    print(f"  bazaviy:        {asos:.4f}")
    print(f"  belgilar bilan: {yangi_ball:.4f}")
    print(f"  farq: {yangi_ball - asos:+.4f}, chegara: {2 * se:.4f}")
    qabul = yangi_ball - asos > 2 * se
    print(f"  qaror: {'QABUL' if qabul else 'rad etildi'}")

    print("\n=== 3. Sozlash (byudjet: 20 nomzod) ===")
    taqsimot = {"m__learning_rate": loguniform(0.02, 0.4),
                "m__max_leaf_nodes": randint(4, 50),
                "m__min_samples_leaf": randint(5, 100)}
    asos_quvur = quvur2() if qabul else quvur()
    asos_df = dftr2 if qabul else dftr
    q = RandomizedSearchCV(asos_quvur, taqsimot, n_iter=20, cv=cv,
                           scoring="average_precision", random_state=0,
                           n_jobs=1).fit(asos_df, ytr, groups=gtr)
    joriy = yangi_ball if qabul else asos
    print(f"  sozlashdan oldin: {joriy:.4f}")
    print(f"  best_score_: {q.best_score_:.4f} "
          f"(hisobotga YOZILMAYDI)")
    print(f"  o'sish: {q.best_score_ - joriy:+.4f}, "
          f"chegara: {2 * se:.4f}")
    print(f"  qaror: "
          f"{'QABUL' if q.best_score_ - joriy > 2 * se else 'rad etildi'}")

    print("\n=== 4. Byudjet hisoboti ===")
    print(f"  {'bosqich':<24} {'nomzodlar':>11} {'o_sish':>9}")
    print(f"  {'bazaviy':<24} {1:>11} {'-':>9}")
    print(f"  {'belgilar guruhi':<24} {1:>11} "
          f"{yangi_ball - asos:>+9.4f}")
    print(f"  {'sozlash':<24} {20:>11} "
          f"{q.best_score_ - joriy:>+9.4f}")
    print(f"  jami tajriba (T): {22}")
    print("  ⭐ Chegara har bosqichda avtomatik qaror beradi")


if __name__ == "__main__":
    main()

=== 1. Bazaviy va qaror chegarasi ===
  bazaviy AP: 0.1153
  foldlar bo'yicha SE: 0.0073
  QAROR CHEGARASI (2*SE): 0.0146
  bundan kichik yaxshilanish RAD ETILADI

=== 2. Belgi g'oyalari (guruhlab sinaladi) ===
  bazaviy:        0.1153
  belgilar bilan: 0.1057
  farq: -0.0096, chegara: 0.0146
  qaror: rad etildi

=== 3. Sozlash (byudjet: 20 nomzod) ===
  sozlashdan oldin: 0.1153
  best_score_: 0.1600 (hisobotga YOZILMAYDI)
  o'sish: +0.0447, chegara: 0.0146
  qaror: QABUL

=== 4. Byudjet hisoboti ===
  bosqich                    nomzodlar    o_sish
  bazaviy                            1         -
  belgilar guruhi                    1   -0.0096
  sozlash                           20   +0.0447
  jami tajriba (T): 22
  ⭐ Chegara har bosqichda avtomatik qaror beradi

"""5-6 qadamlar: juftlashgan taqqoslash va yopiq test (real sklearn)."""

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import (HistGradientBoostingClassifier,
                              RandomForestClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.model_selection import (GroupKFold, GroupShuffleSplit,
                                     cross_val_score)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def yarat(seed: int = 7, mijozlar: int = 1200) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    qatorlar = []
    for m in range(mijozlar):
        imzo = rng.normal(0, 1)
        tarif = rng.choice(["asosiy", "kengaytirilgan", "korporativ"])
        tq = {"asosiy": 0.6, "kengaytirilgan": 0.0, "korporativ": -0.7}[tarif]
        oylar = int(rng.integers(4, 15))
        faollik = rng.gamma(3, 6)
        for oy in range(oylar):
            sessiyalar = max(0.0, faollik + rng.normal(0, 3) - 0.25 * oy)
            qollab = rng.poisson(0.5)
            kechikish = rng.poisson(0.3)
            kuch = (-2.4 + tq - 0.06 * sessiyalar + 0.45 * qollab
                    + 0.7 * kechikish + 0.8 * imzo + 0.05 * oy)
            bekor = int(rng.random() < 1 / (1 + np.exp(-kuch)))
            qatorlar.append([m, oy, tarif, sessiyalar, qollab, kechikish,
                             bekor])
            if bekor:
                break
    d = pd.DataFrame(qatorlar, columns=["mijoz", "oy", "tarif", "sessiyalar",
                                        "qollab", "kechikish", "bekor"])
    g = d.groupby("mijoz")
    d["sessiya_oynasi"] = g["sessiyalar"].transform(
        lambda s: s.shift(1).rolling(3, min_periods=1).mean())
    # birinchi oyda tarix yo'q -> joriy qiymat bilan to'ldiramiz
    d["sessiya_oynasi"] = d["sessiya_oynasi"].fillna(d["sessiyalar"])
    # maxraj nolga tushishi mumkin -> cheklaymiz (aks holda inf)
    d["sessiya_nisbati"] = (d["sessiyalar"]
                            / d["sessiya_oynasi"].clip(lower=0.5))
    d["qollab_jami"] = g["qollab"].transform(
        lambda s: s.shift(1).cumsum()).fillna(0.0)
    return d


SONLI = ["oy", "sessiyalar", "qollab", "kechikish", "sessiya_oynasi",
         "sessiya_nisbati", "qollab_jami"]
KATEGORIYA = ["tarif"]


def tayyorlagich(masshtab: bool):
    qadamlar = [("s", StandardScaler() if masshtab else "passthrough",
                 SONLI),
                ("k", OneHotEncoder(handle_unknown="ignore",
                                    sparse_output=False), KATEGORIYA)]
    return ColumnTransformer(qadamlar)


def main() -> None:
    df = yarat()
    y = df["bekor"].to_numpy()
    guruh = df["mijoz"].to_numpy()
    tr, te = next(GroupShuffleSplit(1, test_size=0.25, random_state=0)
                  .split(df, y, groups=guruh))
    dftr, ytr, gtr = df.iloc[tr], y[tr], guruh[tr]
    dfte, yte = df.iloc[te], y[te]
    cv = GroupKFold(5)

    nomzodlar = {
        "logistik": Pipeline([
            ("t", tayyorlagich(True)),
            ("m", LogisticRegression(max_iter=3000))]),
        "RF": Pipeline([
            ("t", tayyorlagich(False)),
            ("m", RandomForestClassifier(n_estimators=250,
                                         min_samples_leaf=5,
                                         random_state=0, n_jobs=1))]),
        "boosting": Pipeline([
            ("t", tayyorlagich(False)),
            ("m", HistGradientBoostingClassifier(
                learning_rate=0.08, max_leaf_nodes=16,
                min_samples_leaf=30, max_iter=250,
                early_stopping=False, random_state=0))]),
    }

    print("=== 1. Juftlashgan CV (bir xil GroupKFold) ===")
    fold_ballari = {}
    print(f"  {'model':<12} {'AP':>8} {'std':>8} {'SE':>8}")
    for nom, m in nomzodlar.items():
        b = cross_val_score(m, dftr, ytr, cv=cv, groups=gtr,
                            scoring="average_precision")
        fold_ballari[nom] = b
        print(f"  {nom:<12} {b.mean():>8.4f} {b.std(ddof=1):>8.4f} "
              f"{b.std(ddof=1) / np.sqrt(len(b)):>8.4f}")

    print("\n=== 2. Bazaviy (logistik) bilan juftlashgan farq ===")
    asos = fold_ballari["logistik"]
    print(f"  {'model':<12} {'farq':>9} {'SE':>8} {'95% oraliq':>22} "
          f"{'muhim':>7}")
    for nom in ["RF", "boosting"]:
        farq = fold_ballari[nom] - asos
        se = float(farq.std(ddof=1) / np.sqrt(len(farq)))
        past, yuqori = farq.mean() - 2 * se, farq.mean() + 2 * se
        print(f"  {nom:<12} {farq.mean():>+9.4f} {se:>8.4f} "
              f"{f'[{past:+.4f}, {yuqori:+.4f}]':>22} "
              f"{str(not (past <= 0 <= yuqori)):>7}")

    print("\n=== 3. G'olibni tanlash (bir standart xato qoidasi) ===")
    ortachalar = {n: float(b.mean()) for n, b in fold_ballari.items()}
    eng = max(ortachalar, key=ortachalar.get)
    eng_se = float(fold_ballari[eng].std(ddof=1)
                   / np.sqrt(len(fold_ballari[eng])))
    chegara = ortachalar[eng] - eng_se
    murakkablik = {"logistik": 1, "RF": 3, "boosting": 2}
    nomzod_royxati = [n for n, v in ortachalar.items() if v >= chegara]
    tanlov = min(nomzod_royxati, key=lambda n: murakkablik[n])
    print(f"  eng yuqori ball: {eng} ({ortachalar[eng]:.4f})")
    print(f"  1 SE chegarasi: {chegara:.4f}")
    print(f"  chegara ichidagilar: {sorted(nomzod_royxati)}")
    print(f"  TANLOV (eng sodda): {tanlov}")

    print("\n=== 4. Yopiq testni BIR MARTA ochish ===")
    yakuniy = nomzodlar[tanlov].fit(dftr, ytr)
    p = yakuniy.predict_proba(dfte)[:, 1]
    test_ap = average_precision_score(yte, p)
    test_auc = roc_auc_score(yte, p)
    print(f"  {'ko_rsatkich':<24} {'CV':>9} {'TEST':>9} {'farq':>9}")
    print(f"  {'average_precision':<24} {ortachalar[tanlov]:>9.4f} "
          f"{test_ap:>9.4f} {ortachalar[tanlov] - test_ap:>+9.4f}")
    auc_cv = cross_val_score(nomzodlar[tanlov], dftr, ytr, cv=cv,
                             groups=gtr, scoring="roc_auc").mean()
    print(f"  {'roc_auc':<24} {auc_cv:>9.4f} {test_auc:>9.4f} "
          f"{auc_cv - test_auc:>+9.4f}")
    print(f"  test bazaviy darajasi: {yte.mean():.4f}")
    print("  ⭐ Teng natijada soddaroq model tanlanadi")


if __name__ == "__main__":
    main()

=== 1. Juftlashgan CV (bir xil GroupKFold) ===
  model              AP      std       SE
  logistik       0.1714   0.0198   0.0088
  RF             0.1297   0.0081   0.0036
  boosting       0.1249   0.0161   0.0072

=== 2. Bazaviy (logistik) bilan juftlashgan farq ===
  model             farq       SE             95% oraliq   muhim
  RF             -0.0417   0.0055     [-0.0527, -0.0306]    True
  boosting       -0.0465   0.0051     [-0.0567, -0.0363]    True

=== 3. G'olibni tanlash (bir standart xato qoidasi) ===
  eng yuqori ball: logistik 0.1714-bob
  1 SE chegarasi: 0.1625
  chegara ichidagilar: ['logistik']
  TANLOV (eng sodda): logistik

=== 4. Yopiq testni BIR MARTA ochish ===
  ko_rsatkich                     CV      TEST      farq
  average_precision           0.1714    0.1606   +0.0108
  roc_auc                     0.7210    0.7008   +0.0202
  test bazaviy darajasi: 0.0722
  ⭐ Teng natijada soddaroq model tanlanadi

"""7-qadam: jurnal va topshirishga tayyor hisobot (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score
from sklearn.model_selection import (GroupKFold, GroupShuffleSplit,
                                     cross_val_score)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def yarat(seed: int = 7, mijozlar: int = 1200) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    qatorlar = []
    for m in range(mijozlar):
        imzo = rng.normal(0, 1)
        tarif = rng.choice(["asosiy", "kengaytirilgan", "korporativ"])
        tq = {"asosiy": 0.6, "kengaytirilgan": 0.0, "korporativ": -0.7}[tarif]
        oylar = int(rng.integers(4, 15))
        faollik = rng.gamma(3, 6)
        for oy in range(oylar):
            sessiyalar = max(0.0, faollik + rng.normal(0, 3) - 0.25 * oy)
            qollab = rng.poisson(0.5)
            kechikish = rng.poisson(0.3)
            kuch = (-2.4 + tq - 0.06 * sessiyalar + 0.45 * qollab
                    + 0.7 * kechikish + 0.8 * imzo + 0.05 * oy)
            bekor = int(rng.random() < 1 / (1 + np.exp(-kuch)))
            qatorlar.append([m, oy, tarif, sessiyalar, qollab, kechikish,
                             bekor])
            if bekor:
                break
    d = pd.DataFrame(qatorlar, columns=["mijoz", "oy", "tarif", "sessiyalar",
                                        "qollab", "kechikish", "bekor"])
    g = d.groupby("mijoz")
    d["sessiya_oynasi"] = g["sessiyalar"].transform(
        lambda s: s.shift(1).rolling(3, min_periods=1).mean())
    # birinchi oyda tarix yo'q -> joriy qiymat bilan to'ldiramiz
    d["sessiya_oynasi"] = d["sessiya_oynasi"].fillna(d["sessiyalar"])
    # maxraj nolga tushishi mumkin -> cheklaymiz (aks holda inf)
    d["sessiya_nisbati"] = (d["sessiyalar"]
                            / d["sessiya_oynasi"].clip(lower=0.5))
    d["qollab_jami"] = g["qollab"].transform(
        lambda s: s.shift(1).cumsum()).fillna(0.0)
    return d


BASE = ["oy", "sessiyalar", "qollab", "kechikish"]
YANGI = ["sessiya_oynasi", "sessiya_nisbati", "qollab_jami"]


def qur(sonli, model):
    return Pipeline([
        ("t", ColumnTransformer([
            ("s", StandardScaler() if isinstance(model, LogisticRegression)
             else "passthrough", list(sonli)),
            ("k", OneHotEncoder(handle_unknown="ignore",
                                sparse_output=False), ["tarif"])])),
        ("m", model)])


def main() -> None:
    df = yarat()
    y = df["bekor"].to_numpy()
    guruh = df["mijoz"].to_numpy()
    tr, te = next(GroupShuffleSplit(1, test_size=0.25, random_state=0)
                  .split(df, y, groups=guruh))
    dftr, ytr, gtr = df.iloc[tr], y[tr], guruh[tr]
    dfte, yte = df.iloc[te], y[te]
    cv = GroupKFold(5)

    def baho(model_quvur):
        b = cross_val_score(model_quvur, dftr, ytr, cv=cv, groups=gtr,
                            scoring="average_precision")
        return float(b.mean()), float(b.std(ddof=1) / np.sqrt(len(b)))

    print("=== 1. Tajriba jurnali ===")
    tajribalar = [
        ("bazaviy logistik", BASE, LogisticRegression(max_iter=3000)),
        ("logistik + yangi belgilar", BASE + YANGI,
         LogisticRegression(max_iter=3000)),
        ("boosting bazaviy", BASE, HistGradientBoostingClassifier(
            max_iter=250, early_stopping=False, random_state=0)),
        ("boosting + yangi belgilar", BASE + YANGI,
         HistGradientBoostingClassifier(max_iter=250, early_stopping=False,
                                        random_state=0)),
        ("boosting sozlangan", BASE + YANGI,
         HistGradientBoostingClassifier(learning_rate=0.08,
                                        max_leaf_nodes=16,
                                        min_samples_leaf=30, max_iter=250,
                                        early_stopping=False,
                                        random_state=0)),
    ]
    jurnal = []
    joriy, joriy_nom, joriy_quvur = -1.0, None, None
    asos_se = None
    print(f"  {'#':>3} {'tajriba':<28} {'AP':>7} {'SE':>7} {'farq':>9} "
          f"{'qaror':<8}")
    for i, (nom, sonli, model) in enumerate(tajribalar, 1):
        q = qur(sonli, model)
        ball, se = baho(q)
        if asos_se is None:
            asos_se = se
        chegara = 2 * asos_se
        farq = 0.0 if joriy < 0 else ball - joriy
        qabul = joriy < 0 or farq > chegara
        if qabul:
            joriy, joriy_nom, joriy_quvur = ball, nom, q
        jurnal.append({"n": i, "tajriba": nom, "ap": round(ball, 3),
                       "se": round(se, 4), "qabul": qabul})
        print(f"  {i:>3} {nom:<28} {ball:>7.3f} {se:>7.4f} "
              f"{farq:>+9.4f} {'QABUL' if qabul else 'rad':<8}")

    print(f"\n  qaror chegarasi (2*SE): {2 * asos_se:.4f}")
    print(f"  qabul qilingan: "
          f"{sum(1 for j in jurnal if j['qabul'])}/{len(jurnal)}")

    print("\n=== 2. Yakuniy model va yopiq test ===")
    joriy_quvur.fit(dftr, ytr)
    test_ap = average_precision_score(
        yte, joriy_quvur.predict_proba(dfte)[:, 1])
    T = len(tajribalar)
    optimizm_chegarasi = asos_se * np.sqrt(2 * np.log(T))
    print(f"  tanlangan: {joriy_nom}")
    print(f"  CV AP: {joriy:.3f}")
    print(f"  TEST AP (bir marta): {test_ap:.3f}")
    print(f"  farq: {joriy - test_ap:+.4f}")
    print(f"  T = {T}, nazariy optimizm chegarasi: "
          f"{optimizm_chegarasi:.4f}")

    print("\n=== 3. Jurnal jadvali ===")
    j = pd.DataFrame(jurnal)
    print(j.to_string(index=False))

    print("\n=== 4. Yakuniy hisobot ===")
    print("  VAZIFA: obuna bekor qilinishini oylik bashorat qilish")
    print("  CHIQISH: xavf reytingi -> top-N mijozga qo'ng'iroq")
    print("  METRIKA: average_precision (boshida tanlangan)")
    print("  DIZAYN: GroupKFold(5) mijoz bo'yicha; "
          "test GroupShuffleSplit 25%")
    print(f"  BAZAVIY: {jurnal[0]['ap']:.3f} (musbat ulushi "
          f"{ytr.mean():.3f})")
    print(f"  YAKUNIY: CV {joriy:.3f}, TEST {test_ap:.3f}")
    print(f"  TAJRIBALAR: T = {T}, qaror chegarasi 2*SE = "
          f"{2 * asos_se:.4f}")
    print("  CHEKLOVLAR: sun'iy ma'lumot; bir mijoz bekor qilgandan")
    print("    keyin kuzatilmaydi (o'ngdan kesilgan);")
    print("    vaqt bo'yicha drift tekshirilmagan")
    print("  TAVSIYA: modelni pilot rejimda 1 oy sinash, "
          "haqiqiy top-N")
    print("    ro'yxatida konversiyani o'lchash")
    print("  ⭐ Hisobotda T va cheklovlar bo'lishi shart")


if __name__ == "__main__":
    main()

=== 1. Tajriba jurnali ===
    # tajriba                           AP      SE      farq qaror
    1 bazaviy logistik               0.174  0.0090   +0.0000 QABUL
    2 logistik + yangi belgilar      0.171  0.0088   -0.0027 rad
    3 boosting bazaviy               0.115  0.0073   -0.0588 rad
    4 boosting + yangi belgilar      0.106  0.0060   -0.0683 rad
    5 boosting sozlangan             0.125  0.0072   -0.0492 rad

  qaror chegarasi (2*SE): 0.0180
  qabul qilingan: 1/5

=== 2. Yakuniy model va yopiq test ===
  tanlangan: bazaviy logistik
  CV AP: 0.174
  TEST AP (bir marta): 0.158
  farq: +0.0160
  T = 5, nazariy optimizm chegarasi: 0.0161

=== 3. Jurnal jadvali ===
 n                   tajriba    ap     se  qabul
 1          bazaviy logistik 0.174 0.0090   True
 2 logistik + yangi belgilar 0.171 0.0088  False
 3          boosting bazaviy 0.115 0.0073  False
 4 boosting + yangi belgilar 0.106 0.0060  False
 5        boosting sozlangan 0.125 0.0072  False

=== 4. Yakuniy hisobot ===
  VAZIFA: obuna bekor qilinishini oylik bashorat qilish
  CHIQISH: xavf reytingi -> top-N mijozga qo'ng'iroq
  METRIKA: average_precision (boshida tanlangan)
  DIZAYN: GroupKFold(5) mijoz bo'yicha; test GroupShuffleSplit 25%
  BAZAVIY: 0.174 (musbat ulushi 0.070)
  YAKUNIY: CV 0.174, TEST 0.158
  TAJRIBALAR: T = 5, qaror chegarasi 2*SE = 0.0180
  CHEKLOVLAR: sun'iy ma'lumot; bir mijoz bekor qilgandan
    keyin kuzatilmaydi (o'ngdan kesilgan);
    vaqt bo'yicha drift tekshirilmagan
  TAVSIYA: modelni pilot rejimda 1 oy sinash, haqiqiy top-N
    ro'yxatida konversiyani o'lchash
  ⭐ Hisobotda T va cheklovlar bo'lishi shart

# modellarni sinab bo'lgach GroupKFold kerakligini bilish       # ⚠️
# 1-qadamda strategiyani aniqlang                                # ✅

grid = GridSearchCV(boosting, katta_setka).fit(X, y)             # ⚠️
asos, se = baho(DummyClassifier()), baho(sodda_model)            # ✅

if yangi > joriy: qabul_qil()                                    # ⚠️
if yangi - joriy > 2 * se: qabul_qil()                           # ✅

print(f"Model AP: {q.best_score_:.3f}")                          # ⚠️
print(f"Model AP: {average_precision_score(yte, p):.3f}")        # ✅

# "bir necha variant sinadik"                                    # ⚠️
jurnal.append({"n": i, "tajriba": nom, "ap": ..., "qabul": ...}) # ✅

# "Model AP 0.42. Tayyor."                                       # ⚠️
# + ma'lumot davri, qamrov, drift xavfi, T                       # ✅

# 60 soat sozlash, 10 soat belgilar                              # ⚠️
# 15 soat sozlash, 50 soat belgilar                              # ✅

1.  # oqimning birinchi qadami?
2.  # ikkinchi qadam?
3.  # bazaviy nima uchun kerak?
4.  # qaror chegarasi qancha?
5.  # sozlashga qancha vaqt?
6.  # eng ko'p foyda qayerdan?
7.  # jurnalda nima bo'ladi?
8.  # T nima uchun kerak?
9.  # test necha marta ochiladi?
10. # hisobotda nima bo'lishi shart?
11. # teng natijada nima tanlanadi?
12. # cheklovlar nima uchun yoziladi?

1.  # modellarni sinab bo'lgach GroupKFold kerakligini bilish

2.  grid = GridSearchCV(boosting, katta_setka).fit(X, y)

3.  if yangi > joriy: qabul_qil()

4.  print(f"Model AP: {q.best_score_:.3f}")

5.  # "Model AP 0.42. Tayyor."