14.10-dars: Kalibrlash
14-QISM — KLASSIFIKATSIYA · 10-dars
1. Kirish va motivatsiya
Model "0.8 ehtimol bilan firibgarlik" desa, bu nimani anglatadi? Kalibrlangan modelda aniq narsani: shunday bashorat berilgan 100 ta holatdan taxminan 80 tasi haqiqatan firibgarlik bo'ladi. Kalibrlanmagan modelda esa u shunchaki "yuqori ball" — haqiqiy ulush 0.35 ham, 0.98 ham bo'lishi mumkin.
Farq amaliy: ehtimolni narxga aylantirsangiz 9.9-bob, chegarani narxdan hisoblasangiz 12.7-bob yoki bir nechta model ehtimolini birlashtirsangiz, kalibrlanmagan qiymat noto'g'ri qarorga olib keladi.
Bu darsda: kalibrlash ta'rifi va o'lchovlari (kalibrlash egri chizig'i, Brier, ECE), qaysi modellar kalibrlangan, Platt (sigmoid) va izotonik usullar, CalibratedClassifierCV, kalibrlash qachon kerak emas va tipik xatolar.
Real vaziyat. Sug'urta kompaniyasi da'vo ehtimolini narxga qo'shadi: narx = baza + p × kutilgan_zarar. Model — gradient boosting, AUC 0.83, hamma xursand. Audit ko'rsatdi: model p = 0.2 degan holatlarda haqiqiy ulush 0.31 edi — kompaniya har polisda o'rtacha 40 ming so'm yo'qotardi. Izotonik kalibrlash Brier ni 0.084 dan 0.071 ga tushirdi va narx formulasi to'g'ri ishlay boshladi.
Bu darsda kalibrlashni o'rganamiz.
Bu darsda:
- Kalibrlash nima
- O'lchovlar: egri chiziq, Brier, ECE
- Qaysi modellar kalibrlangan
- Platt va izotonik
- CalibratedClassifierCV
- Qachon kerak emas
- Tuzoqlar
- Amaliy: narx formulasi
ℹ Misollar real numpy/sklearn bilan (Python 3.14).
2. Nazariya — chuqur tushuntirish
2.1. Kalibrlash nima
Model kalibrlangan deyiladi, agar:
P(y = 1 | bashorat = p) = p har qanday p uchun
Ya'ni: "0.7" degan barcha holatlarni yig'sak, ularning 70% i musbat bo'lishi kerak
Diqqat: kalibrlash AJRATISH QOBILIYATI emas
· AUC 0.95 bo'lgan model kalibrlanmagan bo'lishi mumkin
· AUC 0.55 bo'lgan model mukammal kalibrlangan bo'lishi mumkin
(masalan, har doim 0.5 chiqaradigan model 50/50 ma'lumotda kalibrlangan)Kalibrlash va ajratish — ikki mustaqil sifat. Yaxshi model ikkalasiga ham ega bo'lishi kerak: reyting (AUC/PR AUC) qaysi namunalar xavfliroq ekanini, kalibrlash esa "qanchalik xavfli" ekanini aytadi. Ko'p qarorlar ikkinchisisiz to'g'ri qabul qilinmaydi.
2.2. O'lchovlar
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss, log_loss
haqiqiy, bashorat = calibration_curve(y, p, n_bins=10, strategy="quantile")
# ideal: haqiqiy == bashorat (diagonal chiziq)
brier_score_loss(y, p) # mean((p - y)^2) — past yaxshi; ajratish + kalibrlash
log_loss(y, p) # kalibrlashga juda sezgir (ishonchli xatoni jazolaydi)
ECE = sum_b (n_b / n) · |haqiqiy_b - bashorat_b| # o'rtacha kalibrlash xatosiBrier va log-loss — ikkalasi ham kalibrlash va ajratishni birga o'lchaydi (ular "to'g'ri" (proper) skorlar). Faqat kalibrlashni ko'rish uchun egri chiziq yoki ECE kerak. Amaliyotda: Brier — umumiy sifat, egri chiziq — diagnostika, ECE — bitta son bilan hisobot.
2.3. Qaysi modellar kalibrlangan
YAXSHI KALIBRLANGAN (odatda):
· logistik regressiya (log-loss ni minimallashtiradi — 13.10)
· to'g'ri sozlangan neyron tarmoq (lekin chuqur tarmoqlar ko'pincha haddan tashqari ishonchli)
YOMON KALIBRLANGAN:
· Naive Bayes — 0/1 ga siljiydi 14.3-bob
· SVM — decision_function umuman ehtimol emas 14.5-bob
· Random Forest — chekkalarda ehtiyotkor (0 va 1 ga yetmaydi)
· Boosting — ko'pincha haddan tashqari ishonchli
· class_weight yoki resampling ishlatilgan har qanday model (14.9)Qoida: log-loss ni minimallashtiradigan modellar tabiiy kalibrlangan bo'ladi (logistik regressiya). Boshqa yo'qotishlarni (hinge, Gini) optimallashtiruvchi modellar esa yo'q. Random Forest ning egri chizig'i S shaklida: u 0 va 1 ga yaqin bashorat bermaydi, chunki ovoz berish o'rtachasi chekkalarga yetmaydi.
2.4. Platt va izotonik
PLATT (sigmoid): kalibrlangan p = sigmoid(a · f(x) + b)
2 ta parametr → kam ma'lumotda barqaror
taxmin: xato sigmoid shaklida (SVM uchun tabiiy)
IZOTONIK: monoton o'suvchi bo'lakli doimiy funksiya
parametrsiz → har qanday shaklni tuzatadi
ko'p ma'lumot talab qiladi (< 1000 namunada overfitting)
Tanlov: n < 1000 → sigmoid; n > 5000 → izotonik (yoki CV bilan solishtiring)Ikkala usul ham monoton — ular reytingni (AUC ni) o'zgartirmaydi, faqat qiymatlarni qayta taqsimlaydi. Bu muhim xossa: kalibrlash ajratish qobiliyatini yo'qotmaydi (izotonikda juda kichik o'zgarish bo'lishi mumkin, chunki u teng qiymatlarni birlashtiradi).
2.5. CalibratedClassifierCV
from sklearn.calibration import CalibratedClassifierCV
# variant 1: CV ichida (tavsiya etiladi) — butun ma'lumotdan foydalanadi
kalib = CalibratedClassifierCV(asos_model, method="sigmoid", cv=5).fit(X_tr, y_tr)
# variant 2: alohida kalibrlash to'plami
asos = asos_model.fit(X_tr, y_tr)
kalib = CalibratedClassifierCV(asos, method="isotonic", cv="prefit").fit(X_kal, y_kal)
MUHIM: kalibrlash to'plami MODEL O'QITISHDA ISHLATILMAGAN bo'lishi kerak (12.9) cv=5 varianti amaliy standart: u ichki bo'linishlar bilan kalibrlaydi va butun ma'lumotdan foydalanadi. cv="prefit" esa allaqachon o'qitilgan model uchun — lekin unda alohida kalibrlash to'plami kerak, aks holda kalibrlash optimistik bo'ladi (leakage — 12.9).
2.6. Qachon kerak emas
KERAK EMAS:
· faqat YORLIQ kerak (chegara CV bilan tanlanadi)
· faqat REYTING kerak (AUC, top-k tanlash — 12.7)
· model allaqachon kalibrlangan (egri chiziq diagonalga yaqin)
KERAK:
· ehtimol narx/zaxira formulasiga kiradi (9.9, 12.8)
· chegara ehtimoldan hisoblanadi (t* = narx_FP / (narx_FP + narx_FN))
· bir nechta model ehtimoli birlashtiriladi
· ehtimol foydalanuvchiga ko'rsatiladi ("85% ehtimol bilan...")Kalibrlash bepul emas: u qo'shimcha ma'lumot va murakkablik talab qiladi. Shuning uchun avval savol bering: ehtimol qiymati qarorga kiradimi? Agar yo'q bo'lsa — chegara tanlash yetarli 12.7-bob.
2.7. Tuzoqlar
Asosiy tuzoqlar: kalibrlashni o'qitish ma'lumotida baholash (leakage — 12.9); cv="prefit" ni o'quv to'plami bilan ishlatish; izotonikni kam ma'lumotda (overfitting); kalibrlashni AUC ni oshiradi deb kutish; nomutanosib sinfda n_bins ni katta qo'yish (bo'sh binlar); class_weight/resampling dan keyin kalibrlashni unutish 14.9-bob; kalibrlangan modelni boshqa davr ma'lumotida ishlatish (drift); ECE ni yagona mezon qilish.
2.8. Reyting va qiymat
Kalibrlash — bashorat qilingan ehtimolning haqiqiy chastotaga mos kelishi; u ajratish qobiliyatidan (AUC) mustaqil. O'lchovlar: kalibrlash egri chizig'i (diagnostika), Brier va log-loss (umumiy sifat), ECE (bitta son). Logistik regressiya tabiiy kalibrlangan, NB/SVM/RF/boosting esa yo'q. Tuzatish: Platt (sigmoid) — kam ma'lumotda, izotonik — ko'p ma'lumotda; ikkalasi ham monoton, shuning uchun AUC o'zgarmaydi. Kalibrlash alohida ma'lumotda bajarilishi shart. Keyingi dars — matn klassifikatsiyasi.
3. Tez ma'lumotnoma
import numpy as np
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss, log_loss
kalib = CalibratedClassifierCV(model, method="sigmoid", cv=5).fit(X_tr, y_tr)
p = kalib.predict_proba(X_te)[:, 1]
haqiqiy, bashorat = calibration_curve(y_te, p, n_bins=10, strategy="quantile")
brier_score_loss(y_te, p); log_loss(y_te, p)
def ece(y, p, n_bins=10):
chegaralar = np.quantile(p, np.linspace(0, 1, n_bins + 1))
xato = 0.0
for i in range(n_bins):
mos = (p >= chegaralar[i]) & (p <= chegaralar[i + 1])
if mos.sum():
xato += mos.mean() * abs(y[mos].mean() - p[mos].mean())
return xato
QOIDA: egri chiziqni ko'r · alohida to'plamda kalibrla · n kichik bo'lsa sigmoidKalibrlash xulosasi
Kalibrlash: P(y=1 | bashorat=p) = p · Ajratishdan MUSTAQIL
Brier/log-loss — birga; egri chiziq/ECE — faqat kalibrlash
LogReg kalibrlangan · NB/SVM/RF/boosting — yo'q
Platt (kam ma'lumot) / izotonik (ko'p ma'lumot); ikkalasi monoton4. Batafsil misollar
Misollar real numpy/sklearn bilan (Python 3.14).
Misol 1 — Kalibrlash va ajratish: mustaqil sifatlar
"""AUC yuqori, kalibrlash yomon bo'lishi mumkin (real numpy/sklearn)."""
import numpy as np
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score
def ece(y, p, n_bins: int = 10) -> float:
chegaralar = np.quantile(p, np.linspace(0, 1, n_bins + 1))
xato = 0.0
for i in range(n_bins):
mos = (p >= chegaralar[i]) & (p <= chegaralar[i + 1])
if mos.sum():
xato += mos.mean() * abs(y[mos].mean() - p[mos].mean())
return float(xato)
def main() -> None:
rng = np.random.default_rng(5)
n = 20_000
haqiqiy_p = rng.beta(2, 5, n) # haqiqiy ehtimollar
y = (rng.random(n) < haqiqiy_p).astype(int)
print("=== 1. To'rt 'model' ===")
modellar = {
"mukammal": haqiqiy_p,
"ishonchli": np.clip((haqiqiy_p - 0.5) * 3 + 0.5, 0.001, 0.999),
"ehtiyotkor": 0.5 + (haqiqiy_p - 0.5) * 0.35,
"siljigan": np.clip(haqiqiy_p + 0.15, 0.001, 0.999),
}
print(f" {'model':<12} {'AUC':>8} {'Brier':>9} {'log-loss':>10} {'ECE':>8}")
for nom, p in modellar.items():
print(f" {nom:<12} {roc_auc_score(y, p):>8.4f} "
f"{brier_score_loss(y, p):>9.5f} {log_loss(y, p):>10.5f} "
f"{ece(y, p):>8.5f}")
print("\n=== 2. AUC bir xil, kalibrlash har xil ===")
print(" ishonchli/ehtiyotkor/siljigan — hammasi haqiqiy_p ning MONOTON")
print(" funksiyasi, demak AUC bir xil; lekin Brier va ECE farq qiladi")
print("\n=== 3. Kalibrlash egri chiziqlari ===")
for nom in ["mukammal", "ishonchli", "ehtiyotkor"]:
h, b = calibration_curve(y, modellar[nom], n_bins=5, strategy="quantile")
juftlar = ", ".join(f"{bb:.2f}->{hh:.2f}" for bb, hh in zip(b, h))
print(f" {nom:<12}: {juftlar}")
print("\n=== 4. Nima buzilgan ===")
print(" ishonchli : chekkalarga cho'zilgan (0 va 1 ga yaqin — haddan tashqari)")
print(" ehtiyotkor: 0.5 atrofida siqilgan (yetarli ishonch bermaydi)")
print(" siljigan : hamma joyda yuqori (sistematik xato)")
print(" ⭐ AUC kalibrlash haqida hech narsa demaydi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. To'rt 'model' ===
model AUC Brier log-loss ECE
mukammal 0.7228 0.17698 0.52979 0.00581
ishonchli 0.6758 0.21734 1.14231 0.89478
ehtiyotkor 0.7228 0.20805 0.60766 0.15443
siljigan 0.7228 0.20008 0.58835 0.15200
=== 2. AUC bir xil, kalibrlash har xil ===
ishonchli/ehtiyotkor/siljigan — hammasi haqiqiy_p ning MONOTON
funksiyasi, demak AUC bir xil; lekin Brier va ECE farq qiladi
=== 3. Kalibrlash egri chiziqlari ===
mukammal : 0.09->0.09, 0.18->0.18, 0.26->0.26, 0.36->0.36, 0.53->0.53
ishonchli : 0.00->0.18, 0.12->0.38, 0.58->0.53
ehtiyotkor : 0.36->0.09, 0.39->0.18, 0.42->0.26, 0.45->0.36, 0.51->0.53
=== 4. Nima buzilgan ===
ishonchli : chekkalarga cho'zilgan (0 va 1 ga yaqin — haddan tashqari)
ehtiyotkor: 0.5 atrofida siqilgan (yetarli ishonch bermaydi)
siljigan : hamma joyda yuqori (sistematik xato)
⭐ AUC kalibrlash haqida hech narsa demaydiNima ko'rsatdi: 2.1, 2.2-bo'limlar.
Misol 2 — Qaysi modellar kalibrlangan
"""Beshta algoritmning kalibrlash sifati (real numpy/sklearn)."""
import numpy as np
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
def yarat(seed: int = 7, n: int = 12_000, p: int = 8):
rng = np.random.default_rng(seed)
X = rng.normal(0, 1, (n, p))
z = 0.9 * X[:, 0] - 0.8 * X[:, 1] + 0.6 * X[:, 2] - 0.5 * X[:, 3]
pr = 1 / (1 + np.exp(-z))
y = (rng.random(n) < pr).astype(int)
return X, y
def ece(y, p, n_bins: int = 10) -> float:
chegaralar = np.quantile(p, np.linspace(0, 1, n_bins + 1))
xato = 0.0
for i in range(n_bins):
mos = (p >= chegaralar[i]) & (p <= chegaralar[i + 1])
if mos.sum():
xato += mos.mean() * abs(y[mos].mean() - p[mos].mean())
return float(xato)
def main() -> None:
X, y = yarat()
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
stratify=y)
modellar = {
"LogReg": Pipeline([("sc", StandardScaler()),
("m", LogisticRegression(max_iter=3000))]),
"GaussianNB": GaussianNB(),
# SVM ehtimol bermaydi — CalibratedClassifierCV bilan beriladi (14.5)
"SVM(+kalib)": Pipeline([("sc", StandardScaler()),
("m", CalibratedClassifierCV(
SVC(random_state=0), ensemble=False, cv=5))]),
"RandomForest": RandomForestClassifier(n_estimators=200,
min_samples_leaf=5, random_state=0),
"Boosting": GradientBoostingClassifier(random_state=0),
}
print("=== 1. Ajratish va kalibrlash ===")
print(f" {'model':<13} {'AUC':>8} {'Brier':>9} {'log-loss':>10} {'ECE':>8}")
bashoratlar = {}
for nom, m in modellar.items():
m.fit(Xtr, ytr)
p = m.predict_proba(Xte)[:, 1]
bashoratlar[nom] = p
print(f" {nom:<13} {roc_auc_score(yte, p):>8.4f} "
f"{brier_score_loss(yte, p):>9.5f} {log_loss(yte, p):>10.5f} "
f"{ece(yte, p):>8.5f}")
print("\n=== 2. Kalibrlash egri chiziqlari (5 bin) ===")
for nom in ["LogReg", "GaussianNB", "RandomForest"]:
h, b = calibration_curve(yte, bashoratlar[nom], n_bins=5,
strategy="quantile")
juftlar = ", ".join(f"{bb:.2f}->{hh:.2f}" for bb, hh in zip(b, h))
print(f" {nom:<13}: {juftlar}")
print("\n=== 3. Ehtimollar diapazoni ===")
for nom, p in bashoratlar.items():
print(f" {nom:<13}: [{p.min():.4f}, {p.max():.4f}], "
f"chekka (<0.02 yoki >0.98) ulushi "
f"{((p < 0.02) | (p > 0.98)).mean():.1%}")
print("\n=== 4. Xulosa ===")
print(" LogReg — tabiiy kalibrlangan (log-loss ni minimallashtiradi)")
print(" NB — chekkalarga siljigan; RF — chekkalarga yetmaydi")
print(" SVM xom holda ehtimol bermaydi — bu yerda u allaqachon kalibrlangan")
print(" ⭐ Yuqori AUC kalibrlashni kafolatlamaydi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Ajratish va kalibrlash ===
model AUC Brier log-loss ECE
LogReg 0.8214 0.17270 0.51920 0.01484
GaussianNB 0.8213 0.17552 0.52757 0.04918
SVM(+kalib) 0.8123 0.17659 0.53205 0.01811
RandomForest 0.8125 0.17820 0.53411 0.03747
Boosting 0.8151 0.17569 0.52731 0.02011
=== 2. Kalibrlash egri chiziqlari (5 bin) ===
LogReg : 0.13->0.12, 0.32->0.29, 0.51->0.51, 0.69->0.71, 0.87->0.88
GaussianNB : 0.18->0.12, 0.35->0.30, 0.51->0.50, 0.65->0.70, 0.82->0.89
RandomForest : 0.17->0.13, 0.34->0.30, 0.50->0.49, 0.66->0.71, 0.83->0.87
=== 3. Ehtimollar diapazoni ===
LogReg : [0.0037, 0.9940], chekka (<0.02 yoki >0.98) ulushi 1.0%
GaussianNB : [0.0112, 0.9807], chekka (<0.02 yoki >0.98) ulushi 0.1%
SVM(+kalib) : [0.0616, 0.9280], chekka (<0.02 yoki >0.98) ulushi 0.0%
RandomForest : [0.0393, 0.9752], chekka (<0.02 yoki >0.98) ulushi 0.0%
Boosting : [0.0273, 0.9738], chekka (<0.02 yoki >0.98) ulushi 0.0%
=== 4. Xulosa ===
LogReg — tabiiy kalibrlangan (log-loss ni minimallashtiradi)
NB — chekkalarga siljigan; RF — chekkalarga yetmaydi
SVM xom holda ehtimol bermaydi — bu yerda u allaqachon kalibrlangan
⭐ Yuqori AUC kalibrlashni kafolatlamaydiNima ko'rsatdi: 2.3-bo'lim.
Misol 3 — Platt va izotonik
"""Ikki kalibrlash usuli va ma'lumot hajmi (real numpy/sklearn)."""
import numpy as np
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
def yarat(n: int, seed: int = 11, p: int = 8):
rng = np.random.default_rng(seed)
X = rng.normal(0, 1, (n, p))
# belgilar o'zaro bog'liq — NB taxmini buziladi
X[:, 4:8] = X[:, 0:4] + rng.normal(0, 0.15, (n, 4))
z = 0.9 * X[:, 0] - 0.8 * X[:, 1] + 0.6 * X[:, 2] - 0.5 * X[:, 3]
y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
return X, y
def ece(y, p, n_bins: int = 10) -> float:
chegaralar = np.quantile(p, np.linspace(0, 1, n_bins + 1))
xato = 0.0
for i in range(n_bins):
mos = (p >= chegaralar[i]) & (p <= chegaralar[i + 1])
if mos.sum():
xato += mos.mean() * abs(y[mos].mean() - p[mos].mean())
return float(xato)
def main() -> None:
X, y = yarat(16_000)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
stratify=y)
print("=== 1. Kalibrlashsiz Naive Bayes ===")
xom = GaussianNB().fit(Xtr, ytr)
p_xom = xom.predict_proba(Xte)[:, 1]
print(f" AUC {roc_auc_score(yte, p_xom):.4f}, "
f"Brier {brier_score_loss(yte, p_xom):.5f}, "
f"log-loss {log_loss(yte, p_xom):.5f}, ECE {ece(yte, p_xom):.5f}")
print("\n=== 2. Ikki usul bilan kalibrlash ===")
for usul in ["sigmoid", "isotonic"]:
k = CalibratedClassifierCV(GaussianNB(), method=usul, cv=5).fit(Xtr, ytr)
p = k.predict_proba(Xte)[:, 1]
print(f" {usul:<9}: AUC {roc_auc_score(yte, p):.4f}, "
f"Brier {brier_score_loss(yte, p):.5f}, "
f"log-loss {log_loss(yte, p):.5f}, ECE {ece(yte, p):.5f}")
print("\n=== 3. Ma'lumot hajmiga bog'liqlik ===")
print(f" {'n':>7} {'xom ECE':>9} {'sigmoid':>9} {'isotonic':>9}")
for n in [300, 1000, 4000, 16_000]:
Xs, ys = yarat(n, seed=n)
Xa, Xb, ya, yb = train_test_split(Xs, ys, test_size=0.4, random_state=0,
stratify=ys)
e_xom = ece(yb, GaussianNB().fit(Xa, ya).predict_proba(Xb)[:, 1], 5)
qatorlar = [e_xom]
for usul in ["sigmoid", "isotonic"]:
k = CalibratedClassifierCV(GaussianNB(), method=usul, cv=3).fit(Xa, ya)
qatorlar.append(ece(yb, k.predict_proba(Xb)[:, 1], 5))
print(f" {n:>7} " + "".join(f"{v:>9.5f}" for v in qatorlar))
print(" (kam ma'lumotda izotonik beqarorroq)")
print("\n=== 4. Kalibrlash AUC ni o'zgartirmaydi ===")
k_sig = CalibratedClassifierCV(GaussianNB(), method="sigmoid", cv=5).fit(Xtr, ytr)
k_izo = CalibratedClassifierCV(GaussianNB(), method="isotonic", cv=5).fit(Xtr, ytr)
print(f" xom AUC : {roc_auc_score(yte, p_xom):.4f}")
print(f" sigmoid AUC : "
f"{roc_auc_score(yte, k_sig.predict_proba(Xte)[:, 1]):.4f}")
print(f" izotonik AUC : "
f"{roc_auc_score(yte, k_izo.predict_proba(Xte)[:, 1]):.4f}")
h, b = calibration_curve(yte, k_izo.predict_proba(Xte)[:, 1], n_bins=5,
strategy="quantile")
print(f" izotonik egri chizig'i: "
f"{', '.join(f'{bb:.2f}->{hh:.2f}' for bb, hh in zip(b, h))}")
print(" ⭐ Monoton o'zgartirish reytingni saqlaydi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Kalibrlashsiz Naive Bayes ===
AUC 0.7987, Brier 0.18905, log-loss 0.57570, ECE 0.07564
=== 2. Ikki usul bilan kalibrlash ===
sigmoid : AUC 0.7987, Brier 0.18466, log-loss 0.54965, ECE 0.03651
isotonic : AUC 0.7987, Brier 0.18342, log-loss 0.54643, ECE 0.02389
=== 3. Ma'lumot hajmiga bog'liqlik ===
n xom ECE sigmoid isotonic
300 0.03075 0.11776 0.06136
1000 0.08061 0.05425 0.04654
4000 0.07376 0.02531 0.01753
16000 0.05662 0.03356 0.01020
(kam ma'lumotda izotonik beqarorroq)
=== 4. Kalibrlash AUC ni o'zgartirmaydi ===
xom AUC : 0.7987
sigmoid AUC : 0.7987
izotonik AUC : 0.7987
izotonik egri chizig'i: 0.12->0.15, 0.31->0.34, 0.50->0.46, 0.69->0.70, 0.87->0.86
⭐ Monoton o'zgartirish reytingni saqlaydiNima ko'rsatdi: 2.4, 2.5-bo'limlar.
Misol 4 — Narx formulasi: kalibrlash nega pul
"""Kalibrlanmagan ehtimol qanday zarar keltiradi (real numpy/sklearn)."""
import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import brier_score_loss, roc_auc_score
from sklearn.model_selection import train_test_split
def yarat(seed: int = 15, n: int = 20_000, p: int = 10):
rng = np.random.default_rng(seed)
X = rng.normal(0, 1, (n, p))
z = -1.6 + 1.0 * X[:, 0] - 0.9 * X[:, 1] + 0.7 * X[:, 2] * X[:, 3]
pr = 1 / (1 + np.exp(-z))
y = (rng.random(n) < pr).astype(int)
return X, y, pr
def main() -> None:
X, y, haqiqiy_p = yarat()
idx = np.arange(len(X))
Xtr, Xte, ytr, yte, itr, ite = train_test_split(X, y, idx, test_size=0.4,
random_state=0, stratify=y)
p_haqiqiy = haqiqiy_p[ite]
ZARAR = 900_000 # da'vo bo'lsa o'rtacha to'lov
BAZA_NARX = 120_000
print("=== 1. Xom Random Forest ===")
rf = RandomForestClassifier(n_estimators=300, min_samples_leaf=10,
random_state=0).fit(Xtr, ytr)
p_xom = rf.predict_proba(Xte)[:, 1]
print(f" AUC {roc_auc_score(yte, p_xom):.4f}, "
f"Brier {brier_score_loss(yte, p_xom):.5f}")
print("\n=== 2. Kalibrlangan model ===")
kalib = CalibratedClassifierCV(
RandomForestClassifier(n_estimators=300, min_samples_leaf=10,
random_state=0),
method="isotonic", cv=5).fit(Xtr, ytr)
p_kal = kalib.predict_proba(Xte)[:, 1]
print(f" AUC {roc_auc_score(yte, p_kal):.4f}, "
f"Brier {brier_score_loss(yte, p_kal):.5f}")
print("\n=== 3. Narx formulasi: narx = baza + p × zarar ===")
print(f" {'model':<14} {'o_rtacha narx':>14} {'haqiqiy xarajat':>16} "
f"{'farq':>12}")
haqiqiy_xarajat = BAZA_NARX + p_haqiqiy * ZARAR
for nom, p in [("xom RF", p_xom), ("kalibrlangan", p_kal),
("haqiqiy p", p_haqiqiy)]:
narx = BAZA_NARX + p * ZARAR
print(f" {nom:<14} {narx.mean():>14,.0f} {haqiqiy_xarajat.mean():>16,.0f} "
f"{narx.mean() - haqiqiy_xarajat.mean():>12,.0f}")
print("\n=== 4. Segmentlar bo'yicha xato ===")
chegaralar = np.quantile(p_kal, [0, 0.25, 0.5, 0.75, 1.0])
print(f" {'segment':<12} {'xom RF':>10} {'kalibrl.':>10} {'haqiqiy':>10}")
for i in range(4):
mos = (p_kal >= chegaralar[i]) & (p_kal <= chegaralar[i + 1])
print(f" chorak {i + 1:<5} {p_xom[mos].mean():>10.4f} "
f"{p_kal[mos].mean():>10.4f} {yte[mos].mean():>10.4f}")
xato_xom = np.abs(p_xom - p_haqiqiy).mean() * ZARAR
xato_kal = np.abs(p_kal - p_haqiqiy).mean() * ZARAR
print(f"\n polis boshiga o'rtacha narx xatosi:")
print(f" xom RF {xato_xom:>12,.0f} so'm")
print(f" kalibrlangan {xato_kal:>12,.0f} so'm")
print(" ⭐ Ehtimol qarorga kirsa — kalibrlash pulga aylanadi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Xom Random Forest ===
AUC 0.7868, Brier 0.14720
=== 2. Kalibrlangan model ===
AUC 0.7855, Brier 0.14710
=== 3. Narx formulasi: narx = baza + p × zarar ===
model o_rtacha narx haqiqiy xarajat farq
xom RF 340,466 333,860 6,606
kalibrlangan 337,842 333,860 3,982
haqiqiy p 333,860 333,860 0
=== 4. Segmentlar bo'yicha xato ===
segment xom RF kalibrl. haqiqiy
chorak 1 0.0586 0.0496 0.0559
chorak 2 0.1488 0.1349 0.1339
chorak 3 0.2695 0.2542 0.2615
chorak 4 0.5032 0.5295 0.5300
polis boshiga o'rtacha narx xatosi:
xom RF 57,084 so'm
kalibrlangan 53,415 so'm
⭐ Ehtimol qarorga kirsa — kalibrlash pulga aylanadiNima ko'rsatdi: 2.6-bo'lim.
5. To'g'ri va noto'g'ri tushunishlar
| Noto'g'ri fikr | To'g'risi |
|---|---|
| "AUC yuqori — ehtimol to'g'ri" | Mustaqil sifatlar |
| "Kalibrlash AUC ni oshiradi" | O'zgartirmaydi (monoton) |
| "Har doim kalibrlash kerak" | Faqat ehtimol qarorga kirsa |
| "Izotonik har doim yaxshiroq" | Kam ma'lumotda overfitting |
| "predict_proba — ehtimol" | Modelga bog'liq |
| "RF yaxshi kalibrlangan" | Chekkalarga yetmaydi |
| "Kalibrlash o'quv ma'lumotida" | Alohida to'plam kerak |
| "Brier faqat kalibrlashni o'lchaydi" | Ajratishni ham |
6. Keng tarqalgan xatolar va yechimlari
1. O'quv ma'lumotida kalibrlash
model.fit(X_tr, y_tr)
CalibratedClassifierCV(model, cv="prefit").fit(X_tr, y_tr) # ⚠️
CalibratedClassifierCV(model, cv=5).fit(X_tr, y_tr) # ✅2. Kam ma'lumotda izotonik
CalibratedClassifierCV(m, method="isotonic", cv=5) # n = 400 # ⚠️
CalibratedClassifierCV(m, method="sigmoid", cv=5) # ✅3. Kalibrlashni AUC bilan baholash
print(roc_auc_score(y, p_kalibrlangan)) # o'zgarmaydi # ⚠️
print(brier_score_loss(y, p), ece(y, p)) # ✅4. Egri chiziqni ko'rmaslik
print("Brier 0.09 — normal") # ⚠️
calibration_curve(y, p, n_bins=10, strategy="quantile") # ✅5. Nomutanosiblikda ko'p bin
calibration_curve(y, p, n_bins=20) # 1% musbat sinf # ⚠️
calibration_curve(y, p, n_bins=5, strategy="quantile") # ✅6. Resampling dan keyin kalibrlashni unutish
foyda = over_model.predict_proba(X)[:, 1] * LTV # ⚠️
# asl taqsimotda kalibrlang 14.9-bob # ✅7. Eski kalibrlashni ishlatish
# 6 oy oldin kalibrlangan model # ⚠️
# muntazam qayta kalibrlash (drift) # ✅7. Integratsiya — bu bilim qayerda kerak bo'ladi
- 9.9-dars (o'tilgan): Ehtimol va qaror
- 12.7-dars (o'tilgan): Chegara va narx
- 14.3, 14.5-darslar (o'tilgan): NB va SVM ehtimollari
- 14.9-dars (o'tilgan): Resampling ta'siri
- 14.14-dars: Amaliy loyiha
8. Eng yaxshi amaliyotlar
Avval savol bering: ehtimol qarorga kiradimi?
Egri chiziqni chizing.
Brier va ECE ni birga bering.
Alohida ma'lumotda kalibrlang.
n kichik bo'lsa sigmoid.
Resampling dan keyin qayta kalibrlang.
Muntazam qayta kalibrlang.
AUC bilan kalibrlashni chalkashtirmang.
9. Amaliy topshiriq
Vazifa 1: Bashorat qiling
1. # kalibrlash ta'rifi?
2. # kalibrlash va AUC bog'liqmi?
3. # Brier nima o'lchaydi?
4. # ECE nima?
5. # qaysi model tabiiy kalibrlangan?
6. # RF egri chizig'i qanday?
7. # Platt usuli nima?
8. # izotonik nima?
9. # qaysi biri kam ma'lumotda?
10. # kalibrlash AUC ni o'zgartiradimi?
11. # cv="prefit" qachon?
12. # kalibrlash qachon kerak emas?Javoblar
- P(y=1 | bashorat=p) = p
- Mustaqil
- Ajratish va kalibrlashni birga
- O'rtacha kalibrlash xatosi
- Logistik regressiya
- S shaklida (chekkalarga yetmaydi)
- Sigmoid moslash (2 parametr)
- Monoton bo'lakli funksiya
- Sigmoid
- Yo'q (monoton)
- Alohida kalibrlash to'plami bo'lsa
- Faqat yorliq yoki reyting kerak bo'lsa
Vazifa 2: Xatolarni tuzating
1. CalibratedClassifierCV(m, cv="prefit").fit(X_tr, y_tr) # m X_tr da o'qitilgan
2. CalibratedClassifierCV(m, method="isotonic", cv=5) # n = 300
3. print(roc_auc_score(y, p_kal)) # kalibrlash sifatini baholash
4. calibration_curve(y, p, n_bins=20) # 1% musbat sinf
5. foyda = nb.predict_proba(X)[:, 1] * 500000Javoblar
1. CalibratedClassifierCV(m, cv=5).fit(X_tr, y_tr)
2. CalibratedClassifierCV(m, method="sigmoid", cv=5)
3. print(brier_score_loss(y, p_kal), ece(y, p_kal))
4. calibration_curve(y, p, n_bins=5, strategy="quantile")
5. CalibratedClassifierCV(nb, method="sigmoid", cv=5)Vazifa 3: Sifatlar
Modellang:
- To'rt xil bashorat
- AUC va Brier
- Egri chiziqlar
- Xulosa
Vazifa 4: Modellar
Modellang:
- Beshta algoritm
- Kalibrlash o'lchovlari
- Ehtimollar diapazoni
- Taqqoslash
Vazifa 5: Usullar
Modellang:
- Sigmoid va izotonik
- Ma'lumot hajmi
- AUC saqlanishi
- Tavsiya
Vazifa 6: Narx
Modellang:
- Narx formulasi
- Xom va kalibrlangan
- Segmentlar
- Moliyaviy ta'sir
Vazifa 7: O'ylash
Chuqur neyron tarmoqlar ko'pincha haddan tashqari ishonchli bo'ladi: ular 0.99 ehtimol beradi, haqiqiy ulush esa 0.85. Bu nima uchun sodir bo'ladi va nega bu muammo katta tizimlarda xavfli?
Javob
Qisqa javob: zamonaviy tarmoqlar o'quv ma'lumotini deyarli mukammal o'rganadi va log-loss ni nolgacha siqishga harakat qiladi — natijada ular chiqishlarni chekkalarga itaradi. Xavf: model bilmaydigan narsani ham ishonch bilan aytadi.
1. Sabablari
- Katta sig'im: model o'quv yorliqlarini yodlaydi (12.4)
- Yo'qotish funksiyasi: log-loss ishonchni rag'batlantiradi
- Batch normalization, o'lchamlar va o'qitish uzunligi ta'siri
- Regularizatsiya kamayishi (zamonaviy tarmoqlarda kamroq ishlatiladi)
2. Nega xavfli
| Soha | Xavf |
|---|---|
| Tibbiyot | "99% sog'lom" — tekshiruv o'tkazib yuboriladi |
| Avtonom transport | Ishonchli, lekin noto'g'ri qaror |
| Moliya | Narx va zaxira noto'g'ri hisoblanadi |
| Inson ko'rigiga yo'naltirish | Ishonchsiz holatlar aniqlanmaydi |
3. Yechimlar
- Temperature scaling (bitta parametr — Platt ning soddalashgan varianti)
- Label smoothing (o'qitish paytida)
- Ansambl (bir nechta tarmoq o'rtachasi)
- Bayescha yondashuvlar, MC dropout
- Konformal bashorat (kafolatlangan qamrov)
4. Amaliy tartib
- Kalibrlash egri chizig'ini albatta chizing
- Validatsiyada temperature scaling qo'llang
- Ishonch chegarasini belgilab, past ishonchda insonga yo'naltiring
- Drift bo'lganda qayta kalibrlang
5. Xulosa
- Aniqlik va ishonch — boshqa narsalar
- Zamonaviy tarmoqlar tabiatan haddan tashqari ishonchli
- Kalibrlash arzon va samarali tuzatish
- Qarorga ta'sir qiladigan har qanday ehtimol kalibrlanishi kerak
Nimani mustahkamlaydi: 2.1, 2.6-bo'limlar.
Xulosa
Bu darsda kalibrlashni o'rgandik.
Eng muhim uch fikr:
Kalibrlash — ajratishdan mustaqil.
P(y=1 | bashorat=p) = pbo'lsa model kalibrlangan. AUC 0.95 bo'lgan model butunlay kalibrlanmagan bo'lishi mumkin va aksincha. Diagnostika: kalibrlash egri chizig'i va ECE; umumiy sifat: Brier va log-loss (ular ikkala xossani birga o'lchaydi).Ko'p modellar kalibrlanmagan. Logistik regressiya tabiiy kalibrlangan (u log-loss ni minimallashtiradi — 13.10). Naive Bayes ehtimollarni 0/1 ga siljitadi 14.3-bob, SVM umuman ehtimol bermaydi 14.5-bob, Random Forest chekkalarga yetmaydi, boosting ko'pincha haddan tashqari ishonchli,
class_weight/resampling esa har qanday modelni siljitadi 14.9-bob.Tuzatish arzon, lekin alohida ma'lumot talab qiladi. Platt (sigmoid) — 2 parametr, kam ma'lumotda barqaror; izotonik — parametrsiz, ko'p ma'lumot kerak. Ikkalasi ham monoton, shuning uchun AUC o'zgarmaydi.
CalibratedClassifierCV(..., cv=5)amaliy standart. Va asosiy savol: ehtimol qarorga kiradimi? — agar yo'q bo'lsa, chegara tanlash yetarli 12.7-bob.
Keyingi darsda matn klassifikatsiyasini o'rganamiz: TF-IDF, n-gramlar, matn tayyorlash va chiziqli modellarning matndagi kuchi.
Izohlar · 0
Izoh yozish uchun kiring.