Data Science va sun'iy intellekt14.10-dars: Kalibrlash
Data Science va sun'iy intellekt · Klassifikatsiya

14.10-dars: Kalibrlash

20 daqiqa10/14-dars

14-QISM — KLASSIFIKATSIYA · 10-dars


1. Kirish va motivatsiya

Model "0.8 ehtimol bilan firibgarlik" desa, bu nimani anglatadi? Kalibrlangan modelda aniq narsani: shunday bashorat berilgan 100 ta holatdan taxminan 80 tasi haqiqatan firibgarlik bo'ladi. Kalibrlanmagan modelda esa u shunchaki "yuqori ball" — haqiqiy ulush 0.35 ham, 0.98 ham bo'lishi mumkin.

Farq amaliy: ehtimolni narxga aylantirsangiz 9.9-bob, chegarani narxdan hisoblasangiz 12.7-bob yoki bir nechta model ehtimolini birlashtirsangiz, kalibrlanmagan qiymat noto'g'ri qarorga olib keladi.

Bu darsda: kalibrlash ta'rifi va o'lchovlari (kalibrlash egri chizig'i, Brier, ECE), qaysi modellar kalibrlangan, Platt (sigmoid) va izotonik usullar, CalibratedClassifierCV, kalibrlash qachon kerak emas va tipik xatolar.

Real vaziyat. Sug'urta kompaniyasi da'vo ehtimolini narxga qo'shadi: narx = baza + p × kutilgan_zarar. Model — gradient boosting, AUC 0.83, hamma xursand. Audit ko'rsatdi: model p = 0.2 degan holatlarda haqiqiy ulush 0.31 edi — kompaniya har polisda o'rtacha 40 ming so'm yo'qotardi. Izotonik kalibrlash Brier ni 0.084 dan 0.071 ga tushirdi va narx formulasi to'g'ri ishlay boshladi.

Bu darsda kalibrlashni o'rganamiz.

Bu darsda:

  • Kalibrlash nima
  • O'lchovlar: egri chiziq, Brier, ECE
  • Qaysi modellar kalibrlangan
  • Platt va izotonik
  • CalibratedClassifierCV
  • Qachon kerak emas
  • Tuzoqlar
  • Amaliy: narx formulasi

ℹ Misollar real numpy/sklearn bilan (Python 3.14).


2. Nazariya — chuqur tushuntirish

2.1. Kalibrlash nima

text
Model kalibrlangan deyiladi, agar:
  P(y = 1 | bashorat = p) = p     har qanday p uchun

Ya'ni: "0.7" degan barcha holatlarni yig'sak, ularning 70% i musbat bo'lishi kerak

Diqqat: kalibrlash AJRATISH QOBILIYATI emas
  · AUC 0.95 bo'lgan model kalibrlanmagan bo'lishi mumkin
  · AUC 0.55 bo'lgan model mukammal kalibrlangan bo'lishi mumkin
  (masalan, har doim 0.5 chiqaradigan model 50/50 ma'lumotda kalibrlangan)

Kalibrlash va ajratish — ikki mustaqil sifat. Yaxshi model ikkalasiga ham ega bo'lishi kerak: reyting (AUC/PR AUC) qaysi namunalar xavfliroq ekanini, kalibrlash esa "qanchalik xavfli" ekanini aytadi. Ko'p qarorlar ikkinchisisiz to'g'ri qabul qilinmaydi.

2.2. O'lchovlar

python
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss, log_loss

haqiqiy, bashorat = calibration_curve(y, p, n_bins=10, strategy="quantile")
# ideal: haqiqiy == bashorat (diagonal chiziq)

brier_score_loss(y, p)      # mean((p - y)^2) — past yaxshi; ajratish + kalibrlash
log_loss(y, p)              # kalibrlashga juda sezgir (ishonchli xatoni jazolaydi)

ECE = sum_b (n_b / n) · |haqiqiy_b - bashorat_b|      # o'rtacha kalibrlash xatosi

Brier va log-loss — ikkalasi ham kalibrlash va ajratishni birga o'lchaydi (ular "to'g'ri" (proper) skorlar). Faqat kalibrlashni ko'rish uchun egri chiziq yoki ECE kerak. Amaliyotda: Brier — umumiy sifat, egri chiziq — diagnostika, ECE — bitta son bilan hisobot.

2.3. Qaysi modellar kalibrlangan

text
YAXSHI KALIBRLANGAN (odatda):
  · logistik regressiya (log-loss ni minimallashtiradi — 13.10)
  · to'g'ri sozlangan neyron tarmoq (lekin chuqur tarmoqlar ko'pincha haddan tashqari ishonchli)

YOMON KALIBRLANGAN:
  · Naive Bayes — 0/1 ga siljiydi 14.3-bob
  · SVM — decision_function umuman ehtimol emas 14.5-bob
  · Random Forest — chekkalarda ehtiyotkor (0 va 1 ga yetmaydi)
  · Boosting — ko'pincha haddan tashqari ishonchli
  · class_weight yoki resampling ishlatilgan har qanday model (14.9)

Qoida: log-loss ni minimallashtiradigan modellar tabiiy kalibrlangan bo'ladi (logistik regressiya). Boshqa yo'qotishlarni (hinge, Gini) optimallashtiruvchi modellar esa yo'q. Random Forest ning egri chizig'i S shaklida: u 0 va 1 ga yaqin bashorat bermaydi, chunki ovoz berish o'rtachasi chekkalarga yetmaydi.

2.4. Platt va izotonik

text
PLATT (sigmoid): kalibrlangan p = sigmoid(a · f(x) + b)
  2 ta parametr → kam ma'lumotda barqaror
  taxmin: xato sigmoid shaklida (SVM uchun tabiiy)

IZOTONIK: monoton o'suvchi bo'lakli doimiy funksiya
  parametrsiz → har qanday shaklni tuzatadi
  ko'p ma'lumot talab qiladi (< 1000 namunada overfitting)

Tanlov: n < 1000 → sigmoid;  n > 5000 → izotonik (yoki CV bilan solishtiring)

Ikkala usul ham monoton — ular reytingni (AUC ni) o'zgartirmaydi, faqat qiymatlarni qayta taqsimlaydi. Bu muhim xossa: kalibrlash ajratish qobiliyatini yo'qotmaydi (izotonikda juda kichik o'zgarish bo'lishi mumkin, chunki u teng qiymatlarni birlashtiradi).

2.5. CalibratedClassifierCV

python
from sklearn.calibration import CalibratedClassifierCV

# variant 1: CV ichida (tavsiya etiladi) — butun ma'lumotdan foydalanadi
kalib = CalibratedClassifierCV(asos_model, method="sigmoid", cv=5).fit(X_tr, y_tr)

# variant 2: alohida kalibrlash to'plami
asos = asos_model.fit(X_tr, y_tr)
kalib = CalibratedClassifierCV(asos, method="isotonic", cv="prefit").fit(X_kal, y_kal)

MUHIM: kalibrlash to'plami MODEL O'QITISHDA ISHLATILMAGAN bo'lishi kerak (12.9)

cv=5 varianti amaliy standart: u ichki bo'linishlar bilan kalibrlaydi va butun ma'lumotdan foydalanadi. cv="prefit" esa allaqachon o'qitilgan model uchun — lekin unda alohida kalibrlash to'plami kerak, aks holda kalibrlash optimistik bo'ladi (leakage — 12.9).

2.6. Qachon kerak emas

text
KERAK EMAS:
  · faqat YORLIQ kerak (chegara CV bilan tanlanadi)
  · faqat REYTING kerak (AUC, top-k tanlash — 12.7)
  · model allaqachon kalibrlangan (egri chiziq diagonalga yaqin)

KERAK:
  · ehtimol narx/zaxira formulasiga kiradi (9.9, 12.8)
  · chegara ehtimoldan hisoblanadi (t* = narx_FP / (narx_FP + narx_FN))
  · bir nechta model ehtimoli birlashtiriladi
  · ehtimol foydalanuvchiga ko'rsatiladi ("85% ehtimol bilan...")

Kalibrlash bepul emas: u qo'shimcha ma'lumot va murakkablik talab qiladi. Shuning uchun avval savol bering: ehtimol qiymati qarorga kiradimi? Agar yo'q bo'lsa — chegara tanlash yetarli 12.7-bob.

2.7. Tuzoqlar

Asosiy tuzoqlar: kalibrlashni o'qitish ma'lumotida baholash (leakage — 12.9); cv="prefit" ni o'quv to'plami bilan ishlatish; izotonikni kam ma'lumotda (overfitting); kalibrlashni AUC ni oshiradi deb kutish; nomutanosib sinfda n_bins ni katta qo'yish (bo'sh binlar); class_weight/resampling dan keyin kalibrlashni unutish 14.9-bob; kalibrlangan modelni boshqa davr ma'lumotida ishlatish (drift); ECE ni yagona mezon qilish.

2.8. Reyting va qiymat

Kalibrlash — bashorat qilingan ehtimolning haqiqiy chastotaga mos kelishi; u ajratish qobiliyatidan (AUC) mustaqil. O'lchovlar: kalibrlash egri chizig'i (diagnostika), Brier va log-loss (umumiy sifat), ECE (bitta son). Logistik regressiya tabiiy kalibrlangan, NB/SVM/RF/boosting esa yo'q. Tuzatish: Platt (sigmoid) — kam ma'lumotda, izotonik — ko'p ma'lumotda; ikkalasi ham monoton, shuning uchun AUC o'zgarmaydi. Kalibrlash alohida ma'lumotda bajarilishi shart. Keyingi dars — matn klassifikatsiyasi.


3. Tez ma'lumotnoma

python
import numpy as np
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss, log_loss

kalib = CalibratedClassifierCV(model, method="sigmoid", cv=5).fit(X_tr, y_tr)
p = kalib.predict_proba(X_te)[:, 1]

haqiqiy, bashorat = calibration_curve(y_te, p, n_bins=10, strategy="quantile")
brier_score_loss(y_te, p); log_loss(y_te, p)

def ece(y, p, n_bins=10):
    chegaralar = np.quantile(p, np.linspace(0, 1, n_bins + 1))
    xato = 0.0
    for i in range(n_bins):
        mos = (p >= chegaralar[i]) & (p <= chegaralar[i + 1])
        if mos.sum():
            xato += mos.mean() * abs(y[mos].mean() - p[mos].mean())
    return xato
QOIDA: egri chiziqni ko'r · alohida to'plamda kalibrla · n kichik bo'lsa sigmoid

Kalibrlash xulosasi

Kalibrlash: P(y=1 | bashorat=p) = p · Ajratishdan MUSTAQIL
Brier/log-loss — birga; egri chiziq/ECE — faqat kalibrlash
LogReg kalibrlangan · NB/SVM/RF/boosting — yo'q
Platt (kam ma'lumot) / izotonik (ko'p ma'lumot); ikkalasi monoton

4. Batafsil misollar

Misollar real numpy/sklearn bilan (Python 3.14).

Misol 1 — Kalibrlash va ajratish: mustaqil sifatlar

python
"""AUC yuqori, kalibrlash yomon bo'lishi mumkin (real numpy/sklearn)."""

import numpy as np
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score


def ece(y, p, n_bins: int = 10) -> float:
    chegaralar = np.quantile(p, np.linspace(0, 1, n_bins + 1))
    xato = 0.0
    for i in range(n_bins):
        mos = (p >= chegaralar[i]) & (p <= chegaralar[i + 1])
        if mos.sum():
            xato += mos.mean() * abs(y[mos].mean() - p[mos].mean())
    return float(xato)


def main() -> None:
    rng = np.random.default_rng(5)
    n = 20_000
    haqiqiy_p = rng.beta(2, 5, n)                  # haqiqiy ehtimollar
    y = (rng.random(n) < haqiqiy_p).astype(int)

    print("=== 1. To'rt 'model' ===")
    modellar = {
        "mukammal": haqiqiy_p,
        "ishonchli": np.clip((haqiqiy_p - 0.5) * 3 + 0.5, 0.001, 0.999),
        "ehtiyotkor": 0.5 + (haqiqiy_p - 0.5) * 0.35,
        "siljigan": np.clip(haqiqiy_p + 0.15, 0.001, 0.999),
    }
    print(f"  {'model':<12} {'AUC':>8} {'Brier':>9} {'log-loss':>10} {'ECE':>8}")
    for nom, p in modellar.items():
        print(f"  {nom:<12} {roc_auc_score(y, p):>8.4f} "
              f"{brier_score_loss(y, p):>9.5f} {log_loss(y, p):>10.5f} "
              f"{ece(y, p):>8.5f}")

    print("\n=== 2. AUC bir xil, kalibrlash har xil ===")
    print("  ishonchli/ehtiyotkor/siljigan — hammasi haqiqiy_p ning MONOTON")
    print("  funksiyasi, demak AUC bir xil; lekin Brier va ECE farq qiladi")

    print("\n=== 3. Kalibrlash egri chiziqlari ===")
    for nom in ["mukammal", "ishonchli", "ehtiyotkor"]:
        h, b = calibration_curve(y, modellar[nom], n_bins=5, strategy="quantile")
        juftlar = ", ".join(f"{bb:.2f}->{hh:.2f}" for bb, hh in zip(b, h))
        print(f"  {nom:<12}: {juftlar}")

    print("\n=== 4. Nima buzilgan ===")
    print("  ishonchli : chekkalarga cho'zilgan (0 va 1 ga yaqin — haddan tashqari)")
    print("  ehtiyotkor: 0.5 atrofida siqilgan (yetarli ishonch bermaydi)")
    print("  siljigan  : hamma joyda yuqori (sistematik xato)")
    print("  ⭐ AUC kalibrlash haqida hech narsa demaydi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. To'rt 'model' ===
  model             AUC     Brier   log-loss      ECE
  mukammal       0.7228   0.17698    0.52979  0.00581
  ishonchli      0.6758   0.21734    1.14231  0.89478
  ehtiyotkor     0.7228   0.20805    0.60766  0.15443
  siljigan       0.7228   0.20008    0.58835  0.15200

=== 2. AUC bir xil, kalibrlash har xil ===
  ishonchli/ehtiyotkor/siljigan — hammasi haqiqiy_p ning MONOTON
  funksiyasi, demak AUC bir xil; lekin Brier va ECE farq qiladi

=== 3. Kalibrlash egri chiziqlari ===
  mukammal    : 0.09->0.09, 0.18->0.18, 0.26->0.26, 0.36->0.36, 0.53->0.53
  ishonchli   : 0.00->0.18, 0.12->0.38, 0.58->0.53
  ehtiyotkor  : 0.36->0.09, 0.39->0.18, 0.42->0.26, 0.45->0.36, 0.51->0.53

=== 4. Nima buzilgan ===
  ishonchli : chekkalarga cho'zilgan (0 va 1 ga yaqin — haddan tashqari)
  ehtiyotkor: 0.5 atrofida siqilgan (yetarli ishonch bermaydi)
  siljigan  : hamma joyda yuqori (sistematik xato)
  ⭐ AUC kalibrlash haqida hech narsa demaydi

Nima ko'rsatdi: 2.1, 2.2-bo'limlar.

Misol 2 — Qaysi modellar kalibrlangan

python
"""Beshta algoritmning kalibrlash sifati (real numpy/sklearn)."""

import numpy as np
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC


def yarat(seed: int = 7, n: int = 12_000, p: int = 8):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    z = 0.9 * X[:, 0] - 0.8 * X[:, 1] + 0.6 * X[:, 2] - 0.5 * X[:, 3]
    pr = 1 / (1 + np.exp(-z))
    y = (rng.random(n) < pr).astype(int)
    return X, y


def ece(y, p, n_bins: int = 10) -> float:
    chegaralar = np.quantile(p, np.linspace(0, 1, n_bins + 1))
    xato = 0.0
    for i in range(n_bins):
        mos = (p >= chegaralar[i]) & (p <= chegaralar[i + 1])
        if mos.sum():
            xato += mos.mean() * abs(y[mos].mean() - p[mos].mean())
    return float(xato)


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    modellar = {
        "LogReg": Pipeline([("sc", StandardScaler()),
                            ("m", LogisticRegression(max_iter=3000))]),
        "GaussianNB": GaussianNB(),
        # SVM ehtimol bermaydi — CalibratedClassifierCV bilan beriladi (14.5)
        "SVM(+kalib)": Pipeline([("sc", StandardScaler()),
                                 ("m", CalibratedClassifierCV(
                                     SVC(random_state=0), ensemble=False, cv=5))]),
        "RandomForest": RandomForestClassifier(n_estimators=200,
                                               min_samples_leaf=5, random_state=0),
        "Boosting": GradientBoostingClassifier(random_state=0),
    }

    print("=== 1. Ajratish va kalibrlash ===")
    print(f"  {'model':<13} {'AUC':>8} {'Brier':>9} {'log-loss':>10} {'ECE':>8}")
    bashoratlar = {}
    for nom, m in modellar.items():
        m.fit(Xtr, ytr)
        p = m.predict_proba(Xte)[:, 1]
        bashoratlar[nom] = p
        print(f"  {nom:<13} {roc_auc_score(yte, p):>8.4f} "
              f"{brier_score_loss(yte, p):>9.5f} {log_loss(yte, p):>10.5f} "
              f"{ece(yte, p):>8.5f}")

    print("\n=== 2. Kalibrlash egri chiziqlari (5 bin) ===")
    for nom in ["LogReg", "GaussianNB", "RandomForest"]:
        h, b = calibration_curve(yte, bashoratlar[nom], n_bins=5,
                                 strategy="quantile")
        juftlar = ", ".join(f"{bb:.2f}->{hh:.2f}" for bb, hh in zip(b, h))
        print(f"  {nom:<13}: {juftlar}")

    print("\n=== 3. Ehtimollar diapazoni ===")
    for nom, p in bashoratlar.items():
        print(f"  {nom:<13}: [{p.min():.4f}, {p.max():.4f}], "
              f"chekka (<0.02 yoki >0.98) ulushi "
              f"{((p < 0.02) | (p > 0.98)).mean():.1%}")

    print("\n=== 4. Xulosa ===")
    print("  LogReg — tabiiy kalibrlangan (log-loss ni minimallashtiradi)")
    print("  NB — chekkalarga siljigan; RF — chekkalarga yetmaydi")
    print("  SVM xom holda ehtimol bermaydi — bu yerda u allaqachon kalibrlangan")
    print("  ⭐ Yuqori AUC kalibrlashni kafolatlamaydi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ajratish va kalibrlash ===
  model              AUC     Brier   log-loss      ECE
  LogReg          0.8214   0.17270    0.51920  0.01484
  GaussianNB      0.8213   0.17552    0.52757  0.04918
  SVM(+kalib)     0.8123   0.17659    0.53205  0.01811
  RandomForest    0.8125   0.17820    0.53411  0.03747
  Boosting        0.8151   0.17569    0.52731  0.02011

=== 2. Kalibrlash egri chiziqlari (5 bin) ===
  LogReg       : 0.13->0.12, 0.32->0.29, 0.51->0.51, 0.69->0.71, 0.87->0.88
  GaussianNB   : 0.18->0.12, 0.35->0.30, 0.51->0.50, 0.65->0.70, 0.82->0.89
  RandomForest : 0.17->0.13, 0.34->0.30, 0.50->0.49, 0.66->0.71, 0.83->0.87

=== 3. Ehtimollar diapazoni ===
  LogReg       : [0.0037, 0.9940], chekka (<0.02 yoki >0.98) ulushi 1.0%
  GaussianNB   : [0.0112, 0.9807], chekka (<0.02 yoki >0.98) ulushi 0.1%
  SVM(+kalib)  : [0.0616, 0.9280], chekka (<0.02 yoki >0.98) ulushi 0.0%
  RandomForest : [0.0393, 0.9752], chekka (<0.02 yoki >0.98) ulushi 0.0%
  Boosting     : [0.0273, 0.9738], chekka (<0.02 yoki >0.98) ulushi 0.0%

=== 4. Xulosa ===
  LogReg — tabiiy kalibrlangan (log-loss ni minimallashtiradi)
  NB — chekkalarga siljigan; RF — chekkalarga yetmaydi
  SVM xom holda ehtimol bermaydi — bu yerda u allaqachon kalibrlangan
  ⭐ Yuqori AUC kalibrlashni kafolatlamaydi

Nima ko'rsatdi: 2.3-bo'lim.

Misol 3 — Platt va izotonik

python
"""Ikki kalibrlash usuli va ma'lumot hajmi (real numpy/sklearn)."""

import numpy as np
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB


def yarat(n: int, seed: int = 11, p: int = 8):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    # belgilar o'zaro bog'liq — NB taxmini buziladi
    X[:, 4:8] = X[:, 0:4] + rng.normal(0, 0.15, (n, 4))
    z = 0.9 * X[:, 0] - 0.8 * X[:, 1] + 0.6 * X[:, 2] - 0.5 * X[:, 3]
    y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
    return X, y


def ece(y, p, n_bins: int = 10) -> float:
    chegaralar = np.quantile(p, np.linspace(0, 1, n_bins + 1))
    xato = 0.0
    for i in range(n_bins):
        mos = (p >= chegaralar[i]) & (p <= chegaralar[i + 1])
        if mos.sum():
            xato += mos.mean() * abs(y[mos].mean() - p[mos].mean())
    return float(xato)


def main() -> None:
    X, y = yarat(16_000)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)

    print("=== 1. Kalibrlashsiz Naive Bayes ===")
    xom = GaussianNB().fit(Xtr, ytr)
    p_xom = xom.predict_proba(Xte)[:, 1]
    print(f"  AUC {roc_auc_score(yte, p_xom):.4f}, "
          f"Brier {brier_score_loss(yte, p_xom):.5f}, "
          f"log-loss {log_loss(yte, p_xom):.5f}, ECE {ece(yte, p_xom):.5f}")

    print("\n=== 2. Ikki usul bilan kalibrlash ===")
    for usul in ["sigmoid", "isotonic"]:
        k = CalibratedClassifierCV(GaussianNB(), method=usul, cv=5).fit(Xtr, ytr)
        p = k.predict_proba(Xte)[:, 1]
        print(f"  {usul:<9}: AUC {roc_auc_score(yte, p):.4f}, "
              f"Brier {brier_score_loss(yte, p):.5f}, "
              f"log-loss {log_loss(yte, p):.5f}, ECE {ece(yte, p):.5f}")

    print("\n=== 3. Ma'lumot hajmiga bog'liqlik ===")
    print(f"  {'n':>7} {'xom ECE':>9} {'sigmoid':>9} {'isotonic':>9}")
    for n in [300, 1000, 4000, 16_000]:
        Xs, ys = yarat(n, seed=n)
        Xa, Xb, ya, yb = train_test_split(Xs, ys, test_size=0.4, random_state=0,
                                          stratify=ys)
        e_xom = ece(yb, GaussianNB().fit(Xa, ya).predict_proba(Xb)[:, 1], 5)
        qatorlar = [e_xom]
        for usul in ["sigmoid", "isotonic"]:
            k = CalibratedClassifierCV(GaussianNB(), method=usul, cv=3).fit(Xa, ya)
            qatorlar.append(ece(yb, k.predict_proba(Xb)[:, 1], 5))
        print(f"  {n:>7} " + "".join(f"{v:>9.5f}" for v in qatorlar))
    print("  (kam ma'lumotda izotonik beqarorroq)")

    print("\n=== 4. Kalibrlash AUC ni o'zgartirmaydi ===")
    k_sig = CalibratedClassifierCV(GaussianNB(), method="sigmoid", cv=5).fit(Xtr, ytr)
    k_izo = CalibratedClassifierCV(GaussianNB(), method="isotonic", cv=5).fit(Xtr, ytr)
    print(f"  xom AUC      : {roc_auc_score(yte, p_xom):.4f}")
    print(f"  sigmoid AUC  : "
          f"{roc_auc_score(yte, k_sig.predict_proba(Xte)[:, 1]):.4f}")
    print(f"  izotonik AUC : "
          f"{roc_auc_score(yte, k_izo.predict_proba(Xte)[:, 1]):.4f}")
    h, b = calibration_curve(yte, k_izo.predict_proba(Xte)[:, 1], n_bins=5,
                             strategy="quantile")
    print(f"  izotonik egri chizig'i: "
          f"{', '.join(f'{bb:.2f}->{hh:.2f}' for bb, hh in zip(b, h))}")
    print("  ⭐ Monoton o'zgartirish reytingni saqlaydi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Kalibrlashsiz Naive Bayes ===
  AUC 0.7987, Brier 0.18905, log-loss 0.57570, ECE 0.07564

=== 2. Ikki usul bilan kalibrlash ===
  sigmoid  : AUC 0.7987, Brier 0.18466, log-loss 0.54965, ECE 0.03651
  isotonic : AUC 0.7987, Brier 0.18342, log-loss 0.54643, ECE 0.02389

=== 3. Ma'lumot hajmiga bog'liqlik ===
        n   xom ECE   sigmoid  isotonic
      300   0.03075  0.11776  0.06136
     1000   0.08061  0.05425  0.04654
     4000   0.07376  0.02531  0.01753
    16000   0.05662  0.03356  0.01020
  (kam ma'lumotda izotonik beqarorroq)

=== 4. Kalibrlash AUC ni o'zgartirmaydi ===
  xom AUC      : 0.7987
  sigmoid AUC  : 0.7987
  izotonik AUC : 0.7987
  izotonik egri chizig'i: 0.12->0.15, 0.31->0.34, 0.50->0.46, 0.69->0.70, 0.87->0.86
  ⭐ Monoton o'zgartirish reytingni saqlaydi

Nima ko'rsatdi: 2.4, 2.5-bo'limlar.

Misol 4 — Narx formulasi: kalibrlash nega pul

python
"""Kalibrlanmagan ehtimol qanday zarar keltiradi (real numpy/sklearn)."""

import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import brier_score_loss, roc_auc_score
from sklearn.model_selection import train_test_split


def yarat(seed: int = 15, n: int = 20_000, p: int = 10):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    z = -1.6 + 1.0 * X[:, 0] - 0.9 * X[:, 1] + 0.7 * X[:, 2] * X[:, 3]
    pr = 1 / (1 + np.exp(-z))
    y = (rng.random(n) < pr).astype(int)
    return X, y, pr


def main() -> None:
    X, y, haqiqiy_p = yarat()
    idx = np.arange(len(X))
    Xtr, Xte, ytr, yte, itr, ite = train_test_split(X, y, idx, test_size=0.4,
                                                    random_state=0, stratify=y)
    p_haqiqiy = haqiqiy_p[ite]

    ZARAR = 900_000                     # da'vo bo'lsa o'rtacha to'lov
    BAZA_NARX = 120_000

    print("=== 1. Xom Random Forest ===")
    rf = RandomForestClassifier(n_estimators=300, min_samples_leaf=10,
                                random_state=0).fit(Xtr, ytr)
    p_xom = rf.predict_proba(Xte)[:, 1]
    print(f"  AUC {roc_auc_score(yte, p_xom):.4f}, "
          f"Brier {brier_score_loss(yte, p_xom):.5f}")

    print("\n=== 2. Kalibrlangan model ===")
    kalib = CalibratedClassifierCV(
        RandomForestClassifier(n_estimators=300, min_samples_leaf=10,
                               random_state=0),
        method="isotonic", cv=5).fit(Xtr, ytr)
    p_kal = kalib.predict_proba(Xte)[:, 1]
    print(f"  AUC {roc_auc_score(yte, p_kal):.4f}, "
          f"Brier {brier_score_loss(yte, p_kal):.5f}")

    print("\n=== 3. Narx formulasi: narx = baza + p × zarar ===")
    print(f"  {'model':<14} {'o_rtacha narx':>14} {'haqiqiy xarajat':>16} "
          f"{'farq':>12}")
    haqiqiy_xarajat = BAZA_NARX + p_haqiqiy * ZARAR
    for nom, p in [("xom RF", p_xom), ("kalibrlangan", p_kal),
                   ("haqiqiy p", p_haqiqiy)]:
        narx = BAZA_NARX + p * ZARAR
        print(f"  {nom:<14} {narx.mean():>14,.0f} {haqiqiy_xarajat.mean():>16,.0f} "
              f"{narx.mean() - haqiqiy_xarajat.mean():>12,.0f}")

    print("\n=== 4. Segmentlar bo'yicha xato ===")
    chegaralar = np.quantile(p_kal, [0, 0.25, 0.5, 0.75, 1.0])
    print(f"  {'segment':<12} {'xom RF':>10} {'kalibrl.':>10} {'haqiqiy':>10}")
    for i in range(4):
        mos = (p_kal >= chegaralar[i]) & (p_kal <= chegaralar[i + 1])
        print(f"  chorak {i + 1:<5} {p_xom[mos].mean():>10.4f} "
              f"{p_kal[mos].mean():>10.4f} {yte[mos].mean():>10.4f}")
    xato_xom = np.abs(p_xom - p_haqiqiy).mean() * ZARAR
    xato_kal = np.abs(p_kal - p_haqiqiy).mean() * ZARAR
    print(f"\n  polis boshiga o'rtacha narx xatosi:")
    print(f"    xom RF       {xato_xom:>12,.0f} so'm")
    print(f"    kalibrlangan {xato_kal:>12,.0f} so'm")
    print("  ⭐ Ehtimol qarorga kirsa — kalibrlash pulga aylanadi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Xom Random Forest ===
  AUC 0.7868, Brier 0.14720

=== 2. Kalibrlangan model ===
  AUC 0.7855, Brier 0.14710

=== 3. Narx formulasi: narx = baza + p × zarar ===
  model           o_rtacha narx  haqiqiy xarajat         farq
  xom RF                340,466          333,860        6,606
  kalibrlangan          337,842          333,860        3,982
  haqiqiy p             333,860          333,860            0

=== 4. Segmentlar bo'yicha xato ===
  segment          xom RF   kalibrl.    haqiqiy
  chorak 1         0.0586     0.0496     0.0559
  chorak 2         0.1488     0.1349     0.1339
  chorak 3         0.2695     0.2542     0.2615
  chorak 4         0.5032     0.5295     0.5300

  polis boshiga o'rtacha narx xatosi:
    xom RF             57,084 so'm
    kalibrlangan       53,415 so'm
  ⭐ Ehtimol qarorga kirsa — kalibrlash pulga aylanadi

Nima ko'rsatdi: 2.6-bo'lim.


5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"AUC yuqori — ehtimol to'g'ri" Mustaqil sifatlar
"Kalibrlash AUC ni oshiradi" O'zgartirmaydi (monoton)
"Har doim kalibrlash kerak" Faqat ehtimol qarorga kirsa
"Izotonik har doim yaxshiroq" Kam ma'lumotda overfitting
"predict_proba — ehtimol" Modelga bog'liq
"RF yaxshi kalibrlangan" Chekkalarga yetmaydi
"Kalibrlash o'quv ma'lumotida" Alohida to'plam kerak
"Brier faqat kalibrlashni o'lchaydi" Ajratishni ham

6. Keng tarqalgan xatolar va yechimlari

1. O'quv ma'lumotida kalibrlash

python
model.fit(X_tr, y_tr)
CalibratedClassifierCV(model, cv="prefit").fit(X_tr, y_tr)        # ⚠️
CalibratedClassifierCV(model, cv=5).fit(X_tr, y_tr)               # ✅

2. Kam ma'lumotda izotonik

python
CalibratedClassifierCV(m, method="isotonic", cv=5)   # n = 400     # ⚠️
CalibratedClassifierCV(m, method="sigmoid", cv=5)                 # ✅

3. Kalibrlashni AUC bilan baholash

python
print(roc_auc_score(y, p_kalibrlangan))   # o'zgarmaydi           # ⚠️
print(brier_score_loss(y, p), ece(y, p))                          # ✅

4. Egri chiziqni ko'rmaslik

python
print("Brier 0.09 — normal")                                      # ⚠️
calibration_curve(y, p, n_bins=10, strategy="quantile")           # ✅

5. Nomutanosiblikda ko'p bin

python
calibration_curve(y, p, n_bins=20)     # 1% musbat sinf           # ⚠️
calibration_curve(y, p, n_bins=5, strategy="quantile")            # ✅

6. Resampling dan keyin kalibrlashni unutish

python
foyda = over_model.predict_proba(X)[:, 1] * LTV                   # ⚠️
# asl taqsimotda kalibrlang 14.9-bob                                # ✅

7. Eski kalibrlashni ishlatish

python
# 6 oy oldin kalibrlangan model                                   # ⚠️
# muntazam qayta kalibrlash (drift)                               # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 9.9-dars (o'tilgan): Ehtimol va qaror
  • 12.7-dars (o'tilgan): Chegara va narx
  • 14.3, 14.5-darslar (o'tilgan): NB va SVM ehtimollari
  • 14.9-dars (o'tilgan): Resampling ta'siri
  • 14.14-dars: Amaliy loyiha

8. Eng yaxshi amaliyotlar

  1. Avval savol bering: ehtimol qarorga kiradimi?

  2. Egri chiziqni chizing.

  3. Brier va ECE ni birga bering.

  4. Alohida ma'lumotda kalibrlang.

  5. n kichik bo'lsa sigmoid.

  6. Resampling dan keyin qayta kalibrlang.

  7. Muntazam qayta kalibrlang.

  8. AUC bilan kalibrlashni chalkashtirmang.


9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # kalibrlash ta'rifi?
2.  # kalibrlash va AUC bog'liqmi?
3.  # Brier nima o'lchaydi?
4.  # ECE nima?
5.  # qaysi model tabiiy kalibrlangan?
6.  # RF egri chizig'i qanday?
7.  # Platt usuli nima?
8.  # izotonik nima?
9.  # qaysi biri kam ma'lumotda?
10. # kalibrlash AUC ni o'zgartiradimi?
11. # cv="prefit" qachon?
12. # kalibrlash qachon kerak emas?
Javoblar
  1. P(y=1 | bashorat=p) = p
  2. Mustaqil
  3. Ajratish va kalibrlashni birga
  4. O'rtacha kalibrlash xatosi
  5. Logistik regressiya
  6. S shaklida (chekkalarga yetmaydi)
  7. Sigmoid moslash (2 parametr)
  8. Monoton bo'lakli funksiya
  9. Sigmoid
  10. Yo'q (monoton)
  11. Alohida kalibrlash to'plami bo'lsa
  12. Faqat yorliq yoki reyting kerak bo'lsa

Vazifa 2: Xatolarni tuzating

python
1.  CalibratedClassifierCV(m, cv="prefit").fit(X_tr, y_tr)   # m X_tr da o'qitilgan

2.  CalibratedClassifierCV(m, method="isotonic", cv=5)   # n = 300

3.  print(roc_auc_score(y, p_kal))   # kalibrlash sifatini baholash

4.  calibration_curve(y, p, n_bins=20)   # 1% musbat sinf

5.  foyda = nb.predict_proba(X)[:, 1] * 500000
Javoblar
python
1.  CalibratedClassifierCV(m, cv=5).fit(X_tr, y_tr)

2.  CalibratedClassifierCV(m, method="sigmoid", cv=5)

3.  print(brier_score_loss(y, p_kal), ece(y, p_kal))

4.  calibration_curve(y, p, n_bins=5, strategy="quantile")

5.  CalibratedClassifierCV(nb, method="sigmoid", cv=5)

Vazifa 3: Sifatlar

Modellang:

  1. To'rt xil bashorat
  2. AUC va Brier
  3. Egri chiziqlar
  4. Xulosa

Vazifa 4: Modellar

Modellang:

  1. Beshta algoritm
  2. Kalibrlash o'lchovlari
  3. Ehtimollar diapazoni
  4. Taqqoslash

Vazifa 5: Usullar

Modellang:

  1. Sigmoid va izotonik
  2. Ma'lumot hajmi
  3. AUC saqlanishi
  4. Tavsiya

Vazifa 6: Narx

Modellang:

  1. Narx formulasi
  2. Xom va kalibrlangan
  3. Segmentlar
  4. Moliyaviy ta'sir

Vazifa 7: O'ylash

Chuqur neyron tarmoqlar ko'pincha haddan tashqari ishonchli bo'ladi: ular 0.99 ehtimol beradi, haqiqiy ulush esa 0.85. Bu nima uchun sodir bo'ladi va nega bu muammo katta tizimlarda xavfli?

Javob

Qisqa javob: zamonaviy tarmoqlar o'quv ma'lumotini deyarli mukammal o'rganadi va log-loss ni nolgacha siqishga harakat qiladi — natijada ular chiqishlarni chekkalarga itaradi. Xavf: model bilmaydigan narsani ham ishonch bilan aytadi.

1. Sabablari

  • Katta sig'im: model o'quv yorliqlarini yodlaydi (12.4)
  • Yo'qotish funksiyasi: log-loss ishonchni rag'batlantiradi
  • Batch normalization, o'lchamlar va o'qitish uzunligi ta'siri
  • Regularizatsiya kamayishi (zamonaviy tarmoqlarda kamroq ishlatiladi)

2. Nega xavfli

Soha Xavf
Tibbiyot "99% sog'lom" — tekshiruv o'tkazib yuboriladi
Avtonom transport Ishonchli, lekin noto'g'ri qaror
Moliya Narx va zaxira noto'g'ri hisoblanadi
Inson ko'rigiga yo'naltirish Ishonchsiz holatlar aniqlanmaydi

3. Yechimlar

  • Temperature scaling (bitta parametr — Platt ning soddalashgan varianti)
  • Label smoothing (o'qitish paytida)
  • Ansambl (bir nechta tarmoq o'rtachasi)
  • Bayescha yondashuvlar, MC dropout
  • Konformal bashorat (kafolatlangan qamrov)

4. Amaliy tartib

  1. Kalibrlash egri chizig'ini albatta chizing
  2. Validatsiyada temperature scaling qo'llang
  3. Ishonch chegarasini belgilab, past ishonchda insonga yo'naltiring
  4. Drift bo'lganda qayta kalibrlang

5. Xulosa

  1. Aniqlik va ishonch — boshqa narsalar
  2. Zamonaviy tarmoqlar tabiatan haddan tashqari ishonchli
  3. Kalibrlash arzon va samarali tuzatish
  4. Qarorga ta'sir qiladigan har qanday ehtimol kalibrlanishi kerak

Nimani mustahkamlaydi: 2.1, 2.6-bo'limlar.


Xulosa

Bu darsda kalibrlashni o'rgandik.

Eng muhim uch fikr:

  1. Kalibrlash — ajratishdan mustaqil. P(y=1 | bashorat=p) = p bo'lsa model kalibrlangan. AUC 0.95 bo'lgan model butunlay kalibrlanmagan bo'lishi mumkin va aksincha. Diagnostika: kalibrlash egri chizig'i va ECE; umumiy sifat: Brier va log-loss (ular ikkala xossani birga o'lchaydi).

  2. Ko'p modellar kalibrlanmagan. Logistik regressiya tabiiy kalibrlangan (u log-loss ni minimallashtiradi — 13.10). Naive Bayes ehtimollarni 0/1 ga siljitadi 14.3-bob, SVM umuman ehtimol bermaydi 14.5-bob, Random Forest chekkalarga yetmaydi, boosting ko'pincha haddan tashqari ishonchli, class_weight/resampling esa har qanday modelni siljitadi 14.9-bob.

  3. Tuzatish arzon, lekin alohida ma'lumot talab qiladi. Platt (sigmoid) — 2 parametr, kam ma'lumotda barqaror; izotonik — parametrsiz, ko'p ma'lumot kerak. Ikkalasi ham monoton, shuning uchun AUC o'zgarmaydi. CalibratedClassifierCV(..., cv=5) amaliy standart. Va asosiy savol: ehtimol qarorga kiradimi? — agar yo'q bo'lsa, chegara tanlash yetarli 12.7-bob.

Keyingi darsda matn klassifikatsiyasini o'rganamiz: TF-IDF, n-gramlar, matn tayyorlash va chiziqli modellarning matndagi kuchi.

Klassifikatsiya · 10-dars yakunlandi

14.10-dars: Kalibrlash

20 daqvaqt
0belgi
Keyingi dars: 14.11-dars: Matn klassifikatsiyasi
Telegram'da
Darsda xato koʻrdingizmi?

Izohlar · 0