Data Science va sun'iy intellekt/ML asoslari7/10-dars20 daqiqa

Mundarija (22)

12.7-dars: Klassifikatsiya metrikalari

12-QISM — MACHINE LEARNING ASOSLARI · 7-dars

1. Kirish va motivatsiya

"Modelimiz 98% aniqlikka erishdi" — bu gap ko'pincha hech narsa anglatmaydi. Agar firibgarlik 2% tranzaksiyada uchrasa, "hech kim firibgar emas" deydigan model ham 98% aniqlik beradi (12.1, Misol 4). Klassifikatsiyada metrika tanlash — texnik emas, biznes qarori: xato turlarining narxi qanday?

Bu darsda: chalkashlik matritsasi (confusion matrix) va undan kelib chiqadigan metrikalar (precision, recall, F1), ehtimol metrikalari (ROC AUC, PR AUC, log-loss, Brier — 9.9), chegara tanlash (narxlar bilan) va nomutanosib sinflar bilan ishlash. Har metrika boshqa savolga javob beradi — shuning uchun metrikani vazifadan oldin tanlash kerak 8.8-bob.

Real vaziyat. Bank firibgarlik modeli: aniqlik 99.2%, jamoa xursand. Risk bo'limi tekshiradi: model 1000 ta firibgarlikdan atigi 180 tasini ushlaydi (recall = 0.18), qolgan 820 tasi o'tib ketadi. Sabab: chegara 0.5 va sinflar nomutanosib. Chegara 0.08 ga tushirildi: recall 0.72 ga ko'tarildi, precision 0.31 bo'ldi (har 3 ogohlantirishdan 1 tasi haqiqiy) — bu bank uchun maqbul, chunki tekshiruv narxi past, o'tkazib yuborish narxi yuqori 9.9-bob.

Bu darsda klassifikatsiya metrikalarini o'rganamiz.

Bu darsda:

Chalkashlik matritsasi va asosiy metrikalar
Precision, recall, F1 — qachon qaysi biri
ROC AUC va PR AUC
Chegarani narxlar bilan tanlash
Nomutanosib sinflar
Ko'p sinfli metrikalar (macro/weighted)
Metrika tuzoqlari
Amaliy: firibgarlik modelini baholash

ℹ Misollar real numpy/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Chalkashlik matritsasi

text

                    BASHORAT: musbat    BASHORAT: manfiy
  HAQIQAT: musbat        TP                  FN  (o'tkazib yuborildi)
  HAQIQAT: manfiy        FP  (soxta signal)  TN

  Accuracy   = (TP + TN) / hammasi        — to'g'ri bashoratlar ulushi
  Precision  = TP / (TP + FP)             — "signal berdik — qanchasi haqiqiy?"
  Recall     = TP / (TP + FN)             — "haqiqiylarning qanchasini topdik?"
  Specificity= TN / (TN + FP)             — manfiylarni to'g'ri aniqlash
  F1         = 2 PR / (P + R)             — precision va recall garmonik o'rtachasi

Chalkashlik matritsasi — barcha metrikalarning manbai: to'rtta son (TP, FP, FN, TN) butun rasmni beradi. Precision — signallar sifati (FP narxi bilan bog'liq), recall — qamrov (FN narxi bilan). Ular orasida muvozanat bor: chegarani pasaytirsangiz recall o'sadi, precision tushadi. sklearn.metrics.confusion_matrix va classification_report — birinchi ko'riladigan narsa.

2.2. Qachon qaysi metrika

text

FN (o'tkazib yuborish) qimmat   → RECALL muhim
  kasallik skriningi, firibgarlik, xavfsizlik, nosozlik bashorati

FP (soxta signal) qimmat        → PRECISION muhim
  spam (muhim xat spamga tushsa), qimmat tekshiruv, mijozga bezovtalik

Ikkalasi ham muhim              → F1 yoki narxga asoslangan metrika
Reyting sifati (chegarasiz)     → ROC AUC / PR AUC
Ehtimol sifati                  → log-loss, Brier (9.9)

Metrika xato narxlaridan kelib chiqadi (9.6, 9.9). "F1 ni maksimallashtiramiz" — standart, lekin ko'pincha noto'g'ri tanlov: u precision va recall ni teng muhim deb oladi. Agar narxlar teng bo'lmasa, kutilgan narx metrikasi to'g'riroq: narx = FP × narx(FP) + FN × narx(FN). Biznes bilan birga metrikani oldindan kelishing 8.8-bob.

2.3. ROC AUC va PR AUC

text

ROC egri chizig'i: TPR (recall) va FPR (= FP / (FP + TN)) — barcha chegaralar uchun
  ROC AUC = "tasodifiy musbat namuna tasodifiy manfiydan yuqoriroq ball oladi" ehtimoli
  0.5 — tasodifiy; 1.0 — mukammal;  sinf ulushiga sezgir EMAS

PR egri chizig'i: precision va recall — barcha chegaralar uchun
  PR AUC (average_precision) — kam uchraydigan sinfda ANIQROQ ko'rsatkich
  baza darajasi = musbat sinf ulushi (ROC da esa har doim 0.5)

ROC AUC — modelning reyting sifati: chegaradan mustaqil. Kamchiligi: kuchli nomutanosiblikda u optimistik ko'rinadi (FPR maxrajida TN juda ko'p). PR AUC (average precision) — kam uchraydigan sinf uchun ma'lumotliroq: uning bazasi — musbat sinf ulushi (masalan, 0.02). Qoida: nomutanosib vazifalarda ikkalasini ham bering, lekin qaror uchun PR AUC va narx metrikasiga tayaning.

2.4. Chegarani tanlash

text

predict() — 0.5 chegarasi (standart), ko'pincha NOTO'G'RI

Chegarani tanlash usullari:
  1. Narxga asoslangan 9.9-bob: t* = narx(FP) / (narx(FP) + narx(FN))   [kalibrlangan p uchun]
  2. Maqsadli recall/precision: "recall >= 0.8 bo'lgan eng yuqori precision"
  3. Byudjet: "kuniga 200 ta tekshiruv" → top-200 ni tanlash
  4. F1 ni maksimallashtirish (narxlar teng bo'lsa)

Chegara VALIDATSIYA ma'lumotida tanlanadi (test emas!)

Chegara — modelning bir qismi emas, qaror qoidasi: uni alohida tanlash kerak. Ko'p loyihalarda model o'zi yaxshi, lekin standart 0.5 chegarasi biznes uchun mos emas (nomutanosib sinflarda deyarli har doim). Byudjet yondashuvi eng amaliy: "kuniga qancha tekshira olamiz?" → shuncha eng yuqori ballni tanlash (top-k).

2.5. Nomutanosib sinflar

text

Muammo: 2% musbat sinf → model "hammasi manfiy" deb o'rganishga moyil

Yechimlar:
  1. METRIKA: accuracy o'rniga PR AUC, recall@precision, narx metrikasi
  2. CHEGARA: 0.5 emas, past chegara
  3. class_weight="balanced" (modelga kam sinfni og'irroq qilish)
  4. Resampling: oversampling (SMOTE), undersampling — ehtiyotkorlik bilan
  5. Ko'proq ma'lumot yig'ish (kam sinf uchun)

Diqqat: resampling ehtimollarni buzadi 9.9-bob — kalibrlash kerak

Nomutanosiblik — metrika va chegara muammosi, ko'pincha algoritm muammosi emas. Birinchi qadam: to'g'ri metrika (PR AUC, recall) va to'g'ri chegara. class_weight="balanced" — oddiy va samarali; resampling (SMOTE) ko'pincha katta foyda bermaydi va ehtimollarni buzadi. Stratifikatsiya 12.3-bob — majburiy.

2.6. Ko'p sinfli metrikalar

text

macro    — har sinf uchun metrikani hisoblab, O'RTACHA (barcha sinflar teng muhim)
weighted — sinf hajmiga qarab vaznlangan o'rtacha
micro    — barcha TP/FP/FN ni yig'ib hisoblash (= accuracy, ko'p sinfda)

classification_report(y_true, y_pred)   — har sinf uchun precision/recall/F1
confusion_matrix(y_true, y_pred)        — qaysi sinf qaysi bilan chalkashadi

Ko'p sinfda macro va weighted farqi muhim: kam uchraydigan sinflar muhim bo'lsa — macro (ular teng vaznga ega), umumiy ishlash muhim bo'lsa — weighted. Chalkashlik matritsasi — eng foydali tashxis: qaysi sinflar bir-biri bilan aralashayotganini ko'rsatadi (masalan, "4" va "9" raqamlari).

2.7. Metrika tuzoqlari

Asosiy tuzoqlar: accuracy ni nomutanosib vazifada ishlatish; 0.5 chegarani standart deb olish; metrikani natijadan keyin tanlash 8.8-bob; ROC AUC ni kuchli nomutanosiblikda yakka ishlatish; chegarani test'da tanlash 12.3-bob; precision/recall ni bazasiz o'qish (PR AUC bazasi — sinf ulushi); kalibrlanmagan ehtimolga narx formulasini qo'llash 9.9-bob; resampling'dan keyin ehtimollarni to'g'ri deb bilish; ko'p sinfda average turini ko'rsatmaslik.

2.8. Metrikalar — qaror tili

Klassifikatsiya metrikalari chalkashlik matritsasidan kelib chiqadi: accuracy (nomutanosibda aldaydi), precision (signal sifati — FP narxi), recall (qamrov — FN narxi), F1 (ikkalasi teng bo'lsa). Chegarasiz baholash: ROC AUC (reyting sifati, sinf ulushiga sezgir emas) va PR AUC (kam sinfda ma'lumotliroq, bazasi — sinf ulushi). Chegara — alohida qaror: narxlar, maqsadli recall yoki byudjet bo'yicha, validatsiyada tanlanadi. Nomutanosiblik — avvalo metrika va chegara muammosi. Keyingi dars — regressiya metrikalari.

3. Tez ma'lumotnoma

python

import numpy as np
from sklearn.metrics import (average_precision_score, classification_report,
                             confusion_matrix, f1_score, precision_recall_curve,
                             precision_score, recall_score, roc_auc_score, roc_curve)

proba = model.predict_proba(X_te)[:, 1]
y_hat = (proba >= chegara).astype(int)

confusion_matrix(y_te, y_hat)                      # [[TN, FP], [FN, TP]]
print(classification_report(y_te, y_hat, digits=3))
roc_auc_score(y_te, proba)                         # reyting sifati
average_precision_score(y_te, proba)               # PR AUC (baza = sinf ulushi)

# chegara tanlash (validatsiyada)
p, r, t = precision_recall_curve(y_val, proba_val)
# recall >= 0.8 bo'lgan eng yuqori precision:
mos = r[:-1] >= 0.8
chegara = t[mos][np.argmax(p[:-1][mos])]

# nomutanosiblik
LogisticRegression(class_weight="balanced", max_iter=1000)
QOIDA: metrikani oldindan tanla · chegarani validatsiyada · PR AUC (kam sinf) · narxni hisobla

Metrikalar xulosasi

Chalkashlik matritsasi — TP, FP, FN, TN (hamma metrika manbai)
Precision — FP narxi; Recall — FN narxi; F1 — ikkalasi teng bo'lsa
ROC AUC — reyting; PR AUC — kam sinfda aniqroq (baza = sinf ulushi)
Chegara — narx/byudjet bo'yicha, validatsiyada
Nomutanosiblik: metrika + chegara + class_weight
Ko'p sinf: macro (teng) / weighted (hajmga qarab)

4. Batafsil misollar

Misollar real numpy/sklearn bilan (Python 3.14). Misollar bir xil yarat() generatoridan foydalanadi.

Misol 1 — Aniqlik aldaydi: chalkashlik matritsasi

python

"""Nomutanosib vazifada accuracy va boshqa metrikalar (real numpy/sklearn)."""

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (accuracy_score, classification_report, confusion_matrix,
                             precision_score, recall_score)
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 5, n: int = 20_000):
    """Firibgarlik: ~4% musbat sinf."""
    rng = np.random.default_rng(seed)
    summa = rng.lognormal(11.5, 1.1, n)
    tungi = (rng.random(n) < 0.18).astype(float)
    yangi_qurilma = (rng.random(n) < 0.12).astype(float)
    tezlik = rng.poisson(1.5, n).astype(float)
    ball = (-4.7 + 0.9 * (np.log(summa) - 11.5) + 0.9 * tungi
            + 1.3 * yangi_qurilma + 0.35 * tezlik)
    y = (rng.random(n) < 1 / (1 + np.exp(-ball))).astype(int)
    X = np.column_stack([np.log(summa), tungi, yangi_qurilma, tezlik])
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
    sc = StandardScaler().fit(Xtr)
    model = LogisticRegression(max_iter=1000).fit(sc.transform(Xtr), ytr)
    proba = model.predict_proba(sc.transform(Xte))[:, 1]

    print("=== 1. Sinflar ===")
    print(f"  musbat sinf ulushi: {yte.mean():.2%} ({yte.sum()} / {len(yte)})")

    print("\n=== 2. 'Hech kim firibgar emas' modeli ===")
    nol = np.zeros_like(yte)
    print(f"  accuracy = {accuracy_score(yte, nol):.4f}  ← yuqori, lekin foydasiz")
    print(f"  recall = {recall_score(yte, nol, zero_division=0):.3f}")

    print("\n=== 3. Model (chegara 0.5) ===")
    y_hat = (proba >= 0.5).astype(int)
    tn, fp, fn, tp = confusion_matrix(yte, y_hat).ravel()
    print(f"  TN={tn}, FP={fp}, FN={fn}, TP={tp}")
    print(f"  accuracy = {accuracy_score(yte, y_hat):.4f}, "
          f"precision = {precision_score(yte, y_hat, zero_division=0):.3f}, "
          f"recall = {recall_score(yte, y_hat):.3f}")

    print("\n=== 4. Chegara 0.08 ===")
    y_hat2 = (proba >= 0.08).astype(int)
    tn, fp, fn, tp = confusion_matrix(yte, y_hat2).ravel()
    print(f"  TN={tn}, FP={fp}, FN={fn}, TP={tp}")
    print(f"  accuracy = {accuracy_score(yte, y_hat2):.4f}, "
          f"precision = {precision_score(yte, y_hat2, zero_division=0):.3f}, "
          f"recall = {recall_score(yte, y_hat2):.3f}")
    print("  ⭐ Accuracy tushdi, lekin model foydaliroq bo'ldi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text

=== 1. Sinflar ===
  musbat sinf ulushi: 3.97% (238 / 6000)

=== 2. 'Hech kim firibgar emas' modeli ===
  accuracy = 0.9603  ← yuqori, lekin foydasiz
  recall = 0.000

=== 3. Model (chegara 0.5) ===
  TN=5757, FP=5, FN=232, TP=6
  accuracy = 0.9605, precision = 0.545, recall = 0.025

=== 4. Chegara 0.08 ===
  TN=5149, FP=613, FN=109, TP=129
  accuracy = 0.8797, precision = 0.174, recall = 0.542
  ⭐ Accuracy tushdi, lekin model foydaliroq bo'ldi

Nima ko'rsatdi: 2.1, 2.2, 2.5-bo'limlar.

Misol 2 — ROC AUC va PR AUC

python

"""Nomutanosib vazifada ROC va PR egri chiziqlari (real numpy/sklearn)."""

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (average_precision_score, precision_recall_curve,
                             roc_auc_score, roc_curve)
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 5, n: int = 20_000):
    rng = np.random.default_rng(seed)
    summa = rng.lognormal(11.5, 1.1, n)
    tungi = (rng.random(n) < 0.18).astype(float)
    yangi_qurilma = (rng.random(n) < 0.12).astype(float)
    tezlik = rng.poisson(1.5, n).astype(float)
    ball = (-4.7 + 0.9 * (np.log(summa) - 11.5) + 0.9 * tungi
            + 1.3 * yangi_qurilma + 0.35 * tezlik)
    y = (rng.random(n) < 1 / (1 + np.exp(-ball))).astype(int)
    X = np.column_stack([np.log(summa), tungi, yangi_qurilma, tezlik])
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
    sc = StandardScaler().fit(Xtr)
    proba = LogisticRegression(max_iter=1000).fit(sc.transform(Xtr), ytr) \
        .predict_proba(sc.transform(Xte))[:, 1]

    print("=== 1. Ikki AUC ===")
    print(f"  ROC AUC = {roc_auc_score(yte, proba):.3f}  (baza 0.5)")
    print(f"  PR AUC  = {average_precision_score(yte, proba):.3f}  "
          f"(baza = sinf ulushi {yte.mean():.3f})")

    print("\n=== 2. ROC egri chizig'idan bir necha nuqta ===")
    fpr, tpr, chegara = roc_curve(yte, proba)
    for maqsad in [0.5, 0.7, 0.9]:
        i = int(np.argmax(tpr >= maqsad))
        print(f"  recall {tpr[i]:.2f} uchun: FPR = {fpr[i]:.3f}, chegara = {chegara[i]:.4f}")

    print("\n=== 3. PR egri chizig'i ===")
    p, r, t = precision_recall_curve(yte, proba)
    for maqsad in [0.5, 0.7, 0.9]:
        mos = r[:-1] >= maqsad
        if mos.any():
            j = int(np.argmax(p[:-1][mos]))
            print(f"  recall >= {maqsad}: eng yaxshi precision = {p[:-1][mos][j]:.3f}, "
                  f"chegara = {t[mos][j]:.4f}")

    print("\n=== 4. Tasodifiy model bilan solishtirish ===")
    rng = np.random.default_rng(0)
    tasodif = rng.random(len(yte))
    print(f"  tasodifiy: ROC AUC = {roc_auc_score(yte, tasodif):.3f}, "
          f"PR AUC = {average_precision_score(yte, tasodif):.3f}")
    print("  ⭐ PR AUC bazasi — sinf ulushi, ROC AUC bazasi — 0.5")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text

=== 1. Ikki AUC ===
  ROC AUC = 0.814  (baza 0.5)
  PR AUC  = 0.242  (baza = sinf ulushi 0.040)

=== 2. ROC egri chizig'idan bir necha nuqta ===
  recall 0.50 uchun: FPR = 0.083, chegara = 0.0928
  recall 0.70 uchun: FPR = 0.224, chegara = 0.0456
  recall 0.90 uchun: FPR = 0.558, chegara = 0.0159

=== 3. PR egri chizig'i ===
  recall >= 0.5: eng yaxshi precision = 0.199, chegara = 0.0928
  recall >= 0.7: eng yaxshi precision = 0.115, chegara = 0.0448
  recall >= 0.9: eng yaxshi precision = 0.063, chegara = 0.0159

=== 4. Tasodifiy model bilan solishtirish ===
  tasodifiy: ROC AUC = 0.477, PR AUC = 0.039
  ⭐ PR AUC bazasi — sinf ulushi, ROC AUC bazasi — 0.5

Nima ko'rsatdi: 2.3-bo'lim.

Misol 3 — Chegarani narx bilan tanlash

python

"""Xato narxlari bo'yicha optimal chegara (real numpy/sklearn)."""

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 5, n: int = 20_000):
    rng = np.random.default_rng(seed)
    summa = rng.lognormal(11.5, 1.1, n)
    tungi = (rng.random(n) < 0.18).astype(float)
    yangi_qurilma = (rng.random(n) < 0.12).astype(float)
    tezlik = rng.poisson(1.5, n).astype(float)
    ball = (-4.7 + 0.9 * (np.log(summa) - 11.5) + 0.9 * tungi
            + 1.3 * yangi_qurilma + 0.35 * tezlik)
    y = (rng.random(n) < 1 / (1 + np.exp(-ball))).astype(int)
    X = np.column_stack([np.log(summa), tungi, yangi_qurilma, tezlik])
    return X, y


def main() -> None:
    X, y = yarat()
    X_qolgan, X_te, y_qolgan, y_te = train_test_split(X, y, test_size=0.3,
                                                      random_state=0, stratify=y)
    X_tr, X_val, y_tr, y_val = train_test_split(X_qolgan, y_qolgan, test_size=0.3,
                                                random_state=0, stratify=y_qolgan)
    sc = StandardScaler().fit(X_tr)
    model = LogisticRegression(max_iter=1000).fit(sc.transform(X_tr), y_tr)
    p_val = model.predict_proba(sc.transform(X_val))[:, 1]
    p_te = model.predict_proba(sc.transform(X_te))[:, 1]

    NARX_FP, NARX_FN = 20_000, 900_000        # tekshiruv va o'tkazib yuborish narxi

    print("=== 1. Chegara bo'yicha kutilgan narx (validatsiyada) ===")
    chegaralar = np.linspace(0.005, 0.5, 100)
    narxlar = []
    for t in chegaralar:
        tn, fp, fn, tp = confusion_matrix(y_val, (p_val >= t).astype(int)).ravel()
        narxlar.append(fp * NARX_FP + fn * NARX_FN)
    narxlar = np.array(narxlar)
    eng = int(np.argmin(narxlar))
    for t in [0.5, 0.1, 0.05, chegaralar[eng]]:
        tn, fp, fn, tp = confusion_matrix(y_val, (p_val >= t).astype(int)).ravel()
        print(f"  chegara {t:.3f}: FP={fp:>4}, FN={fn:>3}, "
              f"narx = {(fp * NARX_FP + fn * NARX_FN) / 1e6:6.1f} mln")

    print(f"\n=== 2. Optimal chegara (validatsiya) = {chegaralar[eng]:.3f} ===")
    print(f"  nazariy: narx(FP)/(narx(FP)+narx(FN)) = {NARX_FP / (NARX_FP + NARX_FN):.4f}")

    print("\n=== 3. Test to'plamida qo'llash ===")
    for nom, t in [("standart 0.5", 0.5), ("tanlangan", chegaralar[eng])]:
        tn, fp, fn, tp = confusion_matrix(y_te, (p_te >= t).astype(int)).ravel()
        print(f"  {nom:<13}: TP={tp:>3}, FP={fp:>4}, FN={fn:>3}, "
              f"narx = {(fp * NARX_FP + fn * NARX_FN) / 1e6:6.1f} mln")

    print("\n=== 4. Byudjet yondashuvi ===")
    byudjet = 300                                  # kuniga 300 tekshiruv
    tartib = np.argsort(p_te)[::-1][:byudjet]
    print(f"  eng yuqori {byudjet} ta ball ichida firibgarlar: {y_te[tartib].sum()} ta "
          f"({y_te[tartib].mean():.1%} precision)")
    print("  ⭐ Chegara — biznes qarori, 0.5 emas")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text

=== 1. Chegara bo'yicha kutilgan narx (validatsiyada) ===
  chegara 0.500: FP=   2, FN=164, narx =  147.6 mln
  chegara 0.100: FP= 317, FN= 93, narx =   90.0 mln
  chegara 0.050: FP= 832, FN= 52, narx =   63.4 mln
  chegara 0.020: FP=2003, FN= 18, narx =   56.3 mln

=== 2. Optimal chegara (validatsiya) = 0.020 ===
  nazariy: narx(FP)/(narx(FP)+narx(FN)) = 0.0217

=== 3. Test to'plamida qo'llash ===
  standart 0.5 : TP=  6, FP=   4, FN=232, narx =  208.9 mln
  tanlangan    : TP=205, FP=2765, FN= 33, narx =   85.0 mln

=== 4. Byudjet yondashuvi ===
  eng yuqori 300 ta ball ichida firibgarlar: 87 ta (29.0% precision)
  ⭐ Chegara — biznes qarori, 0.5 emas

Nima ko'rsatdi: 2.4-bo'lim.

Misol 4 — class_weight va ko'p sinfli metrikalar

python

"""class_weight ta'siri va ko'p sinfli hisobot (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (average_precision_score, classification_report,
                             confusion_matrix, f1_score, recall_score, roc_auc_score)
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 5, n: int = 20_000):
    rng = np.random.default_rng(seed)
    summa = rng.lognormal(11.5, 1.1, n)
    tungi = (rng.random(n) < 0.18).astype(float)
    yangi_qurilma = (rng.random(n) < 0.12).astype(float)
    tezlik = rng.poisson(1.5, n).astype(float)
    ball = (-4.7 + 0.9 * (np.log(summa) - 11.5) + 0.9 * tungi
            + 1.3 * yangi_qurilma + 0.35 * tezlik)
    y = (rng.random(n) < 1 / (1 + np.exp(-ball))).astype(int)
    X = np.column_stack([np.log(summa), tungi, yangi_qurilma, tezlik])
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
    sc = StandardScaler().fit(Xtr)

    print("=== 1. class_weight ta'siri (chegara 0.5) ===")
    for cw in [None, "balanced"]:
        m = LogisticRegression(max_iter=1000, class_weight=cw).fit(sc.transform(Xtr), ytr)
        p = m.predict_proba(sc.transform(Xte))[:, 1]
        y_hat = m.predict(sc.transform(Xte))
        print(f"  class_weight={str(cw):<9}: recall {recall_score(yte, y_hat):.3f}, "
              f"F1 {f1_score(yte, y_hat):.3f}, ROC AUC {roc_auc_score(yte, p):.3f}, "
              f"PR AUC {average_precision_score(yte, p):.3f}")
    print("  (AUC lar deyarli o'zgarmaydi — reyting bir xil, faqat chegara siljiydi)")

    print("\n=== 2. Ko'p sinfli vazifa (raqamlar) ===")
    Xd, yd = load_digits(return_X_y=True)
    Xdtr, Xdte, ydtr, ydte = train_test_split(Xd, yd, test_size=0.3, random_state=0,
                                              stratify=yd)
    md = LogisticRegression(max_iter=3000).fit(Xdtr, ydtr)
    pred = md.predict(Xdte)
    print(f"  accuracy = {(pred == ydte).mean():.3f}")
    print(f"  F1 macro = {f1_score(ydte, pred, average='macro'):.3f}, "
          f"weighted = {f1_score(ydte, pred, average='weighted'):.3f}")

    print("\n=== 3. Eng ko'p chalkashadigan sinflar ===")
    cm = confusion_matrix(ydte, pred)
    np.fill_diagonal(cm, 0)
    eng = np.unravel_index(np.argsort(cm, axis=None)[::-1][:3], cm.shape)
    for a, b in zip(*eng):
        print(f"  haqiqiy {a} → bashorat {b}: {cm[a, b]} marta")

    print("\n=== 4. Sinf bo'yicha hisobot (birinchi qatorlar) ===")
    hisobot = classification_report(ydte, pred, digits=3).splitlines()
    for qator in hisobot[:5]:
        print("  " + qator)
    print("  ⭐ Ko'p sinfda: macro/weighted va chalkashlik matritsasi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text

=== 1. class_weight ta'siri (chegara 0.5) ===
  class_weight=None     : recall 0.025, F1 0.048, ROC AUC 0.814, PR AUC 0.242
  class_weight=balanced : recall 0.748, F1 0.188, ROC AUC 0.814, PR AUC 0.242
  (AUC lar deyarli o'zgarmaydi — reyting bir xil, faqat chegara siljiydi)

=== 2. Ko'p sinfli vazifa (raqamlar) ===
  accuracy = 0.961
  F1 macro = 0.961, weighted = 0.961

=== 3. Eng ko'p chalkashadigan sinflar ===
  haqiqiy 8 → bashorat 1: 4 marta
  haqiqiy 7 → bashorat 9: 2 marta
  haqiqiy 9 → bashorat 5: 1 marta

=== 4. Sinf bo'yicha hisobot (birinchi qatorlar) ===
                precision    recall  f1-score   support

             0      1.000     0.981     0.991        54
             1      0.871     0.982     0.923        55
             2      1.000     0.981     0.990        53
  ⭐ Ko'p sinfda: macro/weighted va chalkashlik matritsasi

Nima ko'rsatdi: 2.5, 2.6-bo'limlar.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr	To'g'risi
"98% aniqlik — zo'r model"	Sinf ulushini ko'ring
"Chegara 0.5 — qoida"	Narx/byudjetdan tanlanadi
"F1 — universal metrika"	Narxlar teng bo'lsa
"ROC AUC hamma joyda yetarli"	Kam sinfda PR AUC
"class_weight AUC ni oshiradi"	Asosan chegarani siljitadi
"SMOTE — nomutanosiblik yechimi"	Avval metrika va chegara
"Chegarani testda tanlash mumkin"	Validatsiyada
"Ko'p sinfda accuracy yetarli"	macro/weighted va CM

6. Keng tarqalgan xatolar va yechimlari

1. Nomutanosibda accuracy

python

print("accuracy:", model.score(X_te, y_te))                       # ⚠️
print(classification_report(y_te, y_hat), average_precision_score(y_te, proba))  # ✅

2. Standart chegara

python

y_hat = model.predict(X_te)                                       # ⚠️
y_hat = (model.predict_proba(X_te)[:, 1] >= chegara).astype(int)  # ✅

3. Chegarani testda tanlash

python

# test bo'yicha F1 ni maksimallashtirish                          # ⚠️
# validatsiyada tanlab, testda faqat qo'llash                     # ✅

4. PR AUC ni bazasiz o'qish

python

print("PR AUC 0.35 — yomon")                                      # ⚠️
print(f"PR AUC {ap:.3f} (baza {y_te.mean():.3f})")                # ✅

5. Kalibrlanmagan ehtimolga narx formulasi

python

chegara = narx_fp / (narx_fp + narx_fn)   # kalibrlanmagan model   # ⚠️
# avval kalibrlash 9.9-bob yoki empirik qidiruv                      # ✅

6. zero_division ogohlantirishi

python

precision_score(y, y_hat)        # hech qanday musbat bashorat yo'q # ⚠️
precision_score(y, y_hat, zero_division=0)                        # ✅

7. Ko'p sinfda average ko'rsatmaslik

python

f1_score(y, pred)                # ko'p sinfda xato                # ⚠️
f1_score(y, pred, average="macro")                                # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

9.9-dars (o'tilgan): Ehtimol, kalibrlash, kutilgan narx
12.2-dars (o'tilgan): Vazifa va qaror bog'liqligi
12.3-dars (o'tilgan): Stratifikatsiya
12.8-dars: Regressiya metrikalari
Klassifikatsiya qismi: Modellar va kengaytirilgan metrikalar

8. Eng yaxshi amaliyotlar

Metrikani vazifadan oldin tanlang.
Chalkashlik matritsasini ko'ring.
Nomutanosibda PR AUC va recall.
Chegarani narx/byudjetdan tanlang.
Chegarani validatsiyada tanlang.
class_weight bilan sinab ko'ring.
Ko'p sinfda macro/weighted ni ko'rsating.
Natijani biznes birligida bering.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python

1.  # precision formulasi?
2.  # recall formulasi?
3.  # FN qimmat bo'lsa qaysi metrika?
4.  # FP qimmat bo'lsa?
5.  # F1 nima?
6.  # ROC AUC bazasi?
7.  # PR AUC bazasi?
8.  # chegara qayerda tanlanadi?
9.  # narxga asoslangan chegara formulasi?
10. # class_weight="balanced" nima qiladi?
11. # macro va weighted farqi?
12. # 2% sinfda accuracy 0.98 — ?

Javoblar

TP / (TP + FP)
TP / (TP + FN)
Recall
Precision
Garmonik o'rtacha
0.5
Musbat sinf ulushi
Validatsiyada
narx(FP) / (narx(FP) + narx(FN))
Kam sinfga ko'proq vazn
Teng / hajmga qarab
Baza bilan bir xil — foydasiz

Vazifa 2: Xatolarni tuzating

python

1.  print("aniqlik:", model.score(X_te, y_te))   # 1.5% sinf

2.  y_hat = model.predict(X_te)   # FN narxi FP dan 40 marta katta

3.  # test bo'yicha eng yaxshi F1 chegarasini tanlash

4.  print("PR AUC:", ap)   # bazasiz

5.  f1_score(y, pred)   # 10 sinf

Javoblar

python

1.  print(classification_report(y_te, y_hat), average_precision_score(y_te, proba))

2.  y_hat = (proba >= 1 / 41).astype(int)   # yoki empirik tanlangan chegara

3.  # validatsiyada tanlash, testda qo'llash

4.  print(f"PR AUC {ap:.3f} (baza {y_te.mean():.3f})")

5.  f1_score(y, pred, average="macro")

Vazifa 3: To'liq baholash

Modellang:

Nomutanosib vazifa
Chalkashlik matritsasi, hisobot
ROC va PR AUC
Chegara tanlash

Vazifa 4: Narx tahlili

Modellang:

Turli narx nisbatlari
Optimal chegara
Kutilgan narx grafigi
Tavsiya

Vazifa 5: Nomutanosiblik

Modellang:

1%, 5%, 20% musbat sinf
class_weight bilan/siz
PR AUC va recall
Xulosa

Vazifa 6: Integratsiya

Modellang:

Kalibrlash (9.9)
Chegara (9.9)
Stratifikatsiya (12.3)
Pipeline (12.6)

Vazifa 7: O'ylash

Ko'p tashkilotlarda ML modellari "aniqlik" bo'yicha hisobot qilinadi, chunki bu rahbariyatga tushunarli. Lekin bu ko'rsatkich ko'pincha qarorni noto'g'ri yo'naltiradi. Texnik metrikalarni biznes tiliga qanday tarjima qilish kerak va bu jarayonda nima yo'qoladi?

Javob

Qisqa javob: eng yaxshi tarjima — metrikani biznes birligida ifodalash: "kuniga 300 ta tekshiruv bilan firibgarlikning 72% ini ushlaymiz va oyiga ~180 mln so'm tejaymiz". Bu ham tushunarli, ham qarorga bog'langan. Sof "aniqlik" esa hech qanday qarorga ulanmaydi.

1. Tarjima jadvali

Texnik	Biznes tili
Recall 0.72	100 ta firibgarlikdan 72 tasini ushlaymiz
Precision 0.31	Har 3 ogohlantirishdan 1 tasi haqiqiy
Chegara 0.08	Kuniga ~300 ta tekshiruv
PR AUC	Reyting sifati (top-N da qancha topiladi)
Kutilgan narx	Oyiga tejalgan/yo'qotilgan pul

2. Nimani yo'qotmaslik kerak

Noaniqlik (CI — 11.8 bootstrap bilan)
Segmentlar bo'yicha farq (adolat, zarar)
Model eskirishi (monitoring)
Chegara o'zgarsa natija qanday o'zgaradi (sezgirlik)

3. Hisobot shabloni

Biznes savoli va qaror
Baza (hozir qanday ishlaydi)
Model natijasi biznes birligida
Xato turlari va ularning narxi
Cheklovlar va monitoring rejasi

4. Data Scientist qanday

Metrikani qaror bilan bog'laydi (12.2)
Bir nechta chegara ssenariysini ko'rsatadi
Natijani A/B test bilan tasdiqlaydi (11.10)

5. Xulosa

Aniqlik — kamdan-kam foydali metrika
Biznes birligi — eng yaxshi tarjima
Noaniqlik va segmentlar yo'qolmasligi kerak
Yakuniy tasdiq — real tajriba

Nimani mustahkamlaydi: 2.2, 2.4-bo'limlar.

Xulosa

Bu darsda klassifikatsiya metrikalarini o'rgandik.

Eng muhim uch fikr:

Chalkashlik matritsasi — manba. TP/FP/FN/TN dan barcha metrikalar kelib chiqadi: precision (signal sifati — FP narxi), recall (qamrov — FN narxi), F1 (ikkalasi teng muhim bo'lsa). Nomutanosib vazifada accuracy aldaydi: "hammasi manfiy" modeli ham 98% beradi.
Chegarasiz baholash. ROC AUC — reyting sifati (baza 0.5, sinf ulushiga sezgir emas); PR AUC (average precision) — kam uchraydigan sinf uchun ma'lumotliroq, bazasi — musbat sinf ulushi. Nomutanosib vazifalarda ikkalasini bering, qaror uchun PR AUC va narx metrikasiga tayaning.
Chegara — biznes qarori. 0.5 — standart, lekin kamdan-kam to'g'ri. Chegara narxlar (t* = narx(FP)/(narx(FP)+narx(FN)), kalibrlangan ehtimol uchun — 9.9), maqsadli recall yoki byudjet (top-k) bo'yicha, validatsiyada tanlanadi. Nomutanosiblik — avvalo metrika va chegara muammosi (class_weight — oddiy va samarali qo'shimcha).

Keyingi darsda regressiya metrikalarini o'rganamiz: MAE, RMSE, R², MAPE — qaysi biri qachon, outlierlarga sezgirlik va biznes talqini.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

Hozircha izoh yo'q. Birinchi bo'ling!