29.11-dars: Ma'lumot etikasi va adolatlilik

                     MA'LUMOT HAYOT SIKLI 29.1-bob
  yig'ish -> saqlash -> tahlil/model -> qaror -> monitoring -> o'chirish
     |          |            |             |           |
  MAXFIYLIK  XAVFSIZLIK  ADOLATLILIK   MAS'ULIYAT   SHAFFOFLIK
  rozilik,   kirish       guruh         inson        model kartasi,
  maqsad,    huquqi,      metrikalari,  nazorati,    datasheet,
  minimal    shifrlash,   proksi,       apellyatsiya, tushuntirish
  hajm       audit        yumshatish    javobgar shaxs

SAVOLLAR:
  kimning ma'lumoti?  ular rozimi?  kerakdan ortiq yig'yapmizmi?
  kim foyda ko'radi, kim zarar?  xato qilsak - kim tuzatadi va qanday?

BELGI TURLARI:
  TO'G'RIDAN IDENTIFIKATOR   ism, pasport raqami, telefon, email
                             -> o'chiriladi yoki psevdonim bilan almashtiriladi
  KVAZIIDENTIFIKATOR (QI)    yosh, jins, pochta indeksi, kasb, tug'ilgan sana
                             -> YOLG'IZ tanitmaydi, BIRGALIKDA - barmoq izi
  SEZGIR ATRIBUT             tashxis, daromad, qarz, sudlanganlik
                             -> himoya qilinadigan narsaning o'zi

PSEVDONIMLASHTIRISH != ANONIMLASHTIRISH
  psevdonim: ism -> "bemor_0042" (kalit kimdadir saqlanadi, qaytarsa bo'ladi)
  anonim: hech kim, hech qanday qo'shimcha ma'lumot bilan qaytara olmaydi
  -> ko'p qonunchilikda psevdonimlashgan ma'lumot HALI HAM shaxsiy ma'lumot

BOG'LASH HUJUMI (linkage):
  "anonim" baza:  [yosh, jins, indeks, TASHXIS]
  ochiq ro'yxat:  [ISM, yosh, jins, indeks]      (klub, saylov, ijtimoiy tarmoq)
  -> QI bo'yicha JOIN: noyob moslik = ism + tashxis

EKVIVALENTLIK SINFI: QI qiymatlari bir xil bo'lgan yozuvlar guruhi
k-ANONIMLIK: har sinfda kamida k ta yozuv
  -> hujumchi QI ni bilsa ham, odamni k tadan biriga toraytiradi xolos

UMUMLASHTIRISH IERARXIYALARI:
  yosh:    37  ->  35-39  ->  30-39  ->  20-39  ->  *
  indeks:  100417 -> 10041* -> 1004** -> 100*** -> *
  jins:    erkak  ->  *

O'CHIRISH (suppression): k dan kichik sinflardagi kam sonli yozuvni tashlash
  umumlashtirish bilan birga: "kichik darajada umumlashtir + ozgina o'chir"

FOYDALILIK NARXI (o'lchang!):
  yosh MAE (oraliq o'rtasi va haqiqiy yosh farqi), hududlar soni,
  tahliliy vazifa sifati (model AUC, kichik hudud statistikasi)

TANLOV QOIDASI: talab qilingan k ga yetadigan ENG KAM umumlashtirish

1. BIR XILLIK HUJUMI: sinfda 7 kishi, hammasi "diabet"
   -> k = 7, lekin tashxis baribir ochiq
   yechim g'oyasi: l-xilma-xillik (har sinfda kamida l xil sezgir qiymat)
2. FON BILIM: "qo'shnim 60 yoshda, gipertoniya kam uchraydigan sinfda emas"
   -> ehtimollar bo'yicha xulosa
   yechim g'oyasi: t-yaqinlik (sinfdagi taqsimot umumiy taqsimotga yaqin)
3. KO'P O'LCHAM: QI soni oshganda deyarli har yozuv noyob
   -> k ga yetish uchun shunchalik umumlashtirish kerakki, ma'lumot foydasiz
4. KO'P NASHR: bir bazaning ikki xil umumlashgan nusxasi birlashtirilsa
   -> k buziladi

TA'RIF (epsilon-DP): M mexanizm, D va D' - bitta odam bilan farq qiluvchi
bazalar. Har qanday natijalar to'plami S uchun:
    P[M(D) in S]  <=  e^eps * P[M(D') in S]
  eps kichik -> ikki olam deyarli ajralmaydi -> maxfiylik kuchli

SEZGIRLIK (sensitivity): bitta odam so'rov javobini ko'pi bilan qanchaga
o'zgartiradi
  sanoq ("nechta diabet?")                  Delta = 1
  chegaralangan o'rtacha, qiymat [a, b]      Delta = (b - a) / n
  yig'indi, qiymat [0, C]                    Delta = C

LAPLACE MEXANIZMI:
  javob = haqiqiy + Laplace(0, Delta / eps)
  o'rtacha absolyut xato = Delta / eps
  eps = 0.1 -> sanoqda ~10 xato;  eps = 1 -> ~1 xato

KOMPOZITSIYA: eps1 va eps2 li ikki so'rov birgalikda (eps1 + eps2)-DP
  -> bir so'rovni 100 marta so'rab o'rtachalash shovqinni yo'qotadi,
     lekin maxfiylik yo'qotilishi ham 100 barobar
  -> MAXFIYLIK BYUDJETI: jami eps oldindan belgilanadi va sarflanadi

MANBA                  MISOL                                  MODEL NIMA QILADI
TARIXIY                o'tmishda ayollarga kredit kam         tarixni "to'g'ri javob"
                       berilgan (inspektor tarafkashligi)     deb o'rganadi va takrorlaydi
YORLIQ / O'LCHASH      "yaxshi xodim" = rahbar bahosi;        o'lchov xatosini haqiqat
                       "jinoyat" = hibsga olish (nazorat      deb qabul qiladi
                       ko'p joyda ko'p hibs)
VAKILLIK (namuna)      qishloq mijozlari ma'lumotda kam       kam guruhda xato ko'p
AGREGATSIYA            bitta model hamma guruhga,             ko'pchilikka moslashadi
                       munosabatlar esa guruhda farqli
QAYTA ALOQA            rad etilgan mijozning natijasi         xato o'zini tasdiqlaydi
                       hech qachon kuzatilmaydi               (selective labels)
PROKSI                 soha, tuman, xarid turi - himoyalangan  olib tashlangan belgini
                       belgi bilan bog'liq                     "qayta tiklaydi"

DEMOGRAFIK PARITET (DP)          P(d=1 | g=A) = P(d=1 | g=B)
  "tasdiq ulushi teng"           farq yoki nisbat (B/A) bilan o'lchanadi
  y ga qaramaydi                 guruhlarda asosiy stavka farq qilsa - bahsli

TENG IMKONIYAT (equal opportunity)
                                 TPR_A = TPR_B,  TPR = P(d=1 | y=1, g)
  "qaytaradiganlar orasida       "munosib" odamga imkoniyat teng
   tasdiqlanish teng"

TENG KOEFFITSIYENTLAR (equalized odds)
                                 TPR_A = TPR_B  va  FPR_A = FPR_B
  ikkala xato turi teng          FPR = P(d=1 | y=0, g)

PREDIKTIV PARITET                PPV_A = PPV_B,  PPV = P(y=1 | d=1, g)
  "tasdiqlanganlar orasida
   qaytarish teng"

GURUH BO'YICHA KALIBRLASH        P(y=1 | ball=s, g) = s  har guruhda
  "0.7 ball ikkala guruhda ham   (14.10 dagi kalibrlash - endi guruh kesimida)
   70% ni bildiradi"

"Adolat ko'rlik orqali" (fairness through unawareness):
  modeldan g ni olib tashlaymiz -> model g ni "bilmaydi"

NEGA ISHLAMAYDI:
  1. YORLIQ tarafkash: y_tarix = f(qobiliyat) - jarima * g
     -> model qobiliyat bilan birga "jarima" ni ham o'rganishga intiladi
  2. PROKSI: soha, tuman, ism, xarid turi g bilan bog'liq
     -> model jarimani proksi orqali qayta tiklaydi
  3. KO'P ZAIF PROKSI birgalikda kuchli proksi bo'ladi

PROKSI TEKSHIRUVI:
  g ni qolgan belgilardan bashorat qiling -> AUC ~ 0.5 bo'lsa proksi kuchsiz,
  AUC 0.8+ bo'lsa - belgilarda g haqida ko'p ma'lumot bor

PARADOKS: adolatlilikni O'LCHASH uchun g KERAK
  -> g ni o'chirib tashlagan jamoa muammoni ko'rmaydi, lekin u yo'qolmaydi
  -> g ni alohida, cheklangan kirish bilan, faqat audit uchun saqlash

CHOULDECHOVA AYNIYATI (har guruh uchun, b - asosiy stavka P(y=1)):
    FPR = b / (1 - b) * (1 - PPV) / PPV * TPR

AGAR b_A != b_B bo'lsa va model mukammal bo'lmasa:
  PPV teng (prediktiv paritet)  va  TPR teng  ->  FPR TENG BO'LA OLMAYDI
  kalibrlangan ball + bitta chegara -> TPR va FPR odatda farq qiladi
  TPR ni guruhga xos chegara bilan tenglasak -> PPV va kalibrlash buziladi

XULOSA: "adolatli model" - bitta ta'rif emas; qaysi metrikani ustun
qo'yish - kontekst va qadriyat tanlovi, uni OSHKORA yozish kerak
  kredit: noto'g'ri RAD (FNR) kimga tushadi?  noto'g'ri TASDIQ narxi kimda?
  tibbiy skrining: kasalni o'tkazib yuborish (TPR) - ustuvor
  ishga olish: tasdiq ulushlari (DP) ko'pincha qonuniy tekshiruv mezoni

BOSQICH         USUL                        g QACHON KERAK     BAHSLI TOMONI
OLDIN (pre)     qayta og'irlash (reweighing) o'qitishda        yorliqqa "tuzatish" -
                w(g,y) = P(g) P(y) / P(g,y)                    tarixni qayta yozish
                qayta namuna, yorliqni tuzatish
ICHIDA (in)     adolatlilik cheklovli        o'qitishda        murakkab, izohlash
                optimallash, jarima hadi                       qiyin
KEYIN (post)    guruhga xos chegara          QAROR paytida     bir xil ball -> turli
                (DP yoki TPR ni tenglash)                      qaror: ko'p huquqiy
                                                               tizimlarda to'g'ridan
                                                               kamsitish deb
                                                               qaralishi mumkin
MA'LUMOT        proksini olib tashlash,      tahlilda           proksi ham qonuniy
                yangi, xolis yorliq yig'ish                     signal bo'lishi mumkin;
                                                                yangi yorliq - qimmat

MODEL KARTASI (26.9 2.8-bo'lim, 26.10, 29.3):
  maqsad va qo'llanish chegarasi | o'quv ma'lumoti va davri
  metrikalar - UMUMIY va GURUHLAR bo'yicha, SE bilan
  ma'lum cheklovlar va tuzoqlar  | qaysi qarorlarda ISHLATILMAYDI
  adolatlilik tahlili: qaysi metrika, nega, natija
  egasi, versiya, qayta ko'rib chiqish sanasi

DATASHEET (ma'lumot uchun):
  kim, nima uchun, qachon yig'di | rozilik va qonuniy asos
  kimlar vakili yetarli emas      | yorliq qanday olingan (kim baholagan?)
  ma'lum tarafkashliklar          | saqlash muddati, kirish huquqi

TUSHUNTIRISH (rad etilgan mijoz uchun):
  sabab kodlari: "qarz yuki yuqori", "kredit tarixi qisqa"
  (13.3 koeffitsiyent talqini, 15.11 belgi muhimligi)
  "nima o'zgarsa natija o'zgaradi" (kontrfaktual tushuntirish g'oyasi)

INSON NAZORATI (human-in-the-loop):
  yuqori ta'sirli qaror (kredit rad, tibbiy, ishdan bo'shatish) -> inson ko'radi
  TUZOQ: avtomatlashtirish tarafkashligi - inson modelga ko'r-ko'rona ergashadi
  -> inson qancha holatda modeldan farqli qaror qilganini o'lchang
APELLYATSIYA: rad etilgan odam qaror ustidan shikoyat qila oladi
  -> kanal, muddat, javobgar shaxs; apellyatsiya natijalari - yangi yorliq manbai
AUDIT JURNALI 25.13-bob: qaysi model versiyasi, qaysi kirish, qanday qaror
MONITORING (27.11, 27.12): metrikalar guruhlar bo'yicha, har oy
  -> guruh metrikasi chegaradan chiqsa - ogohlantirish, xuddi drift kabi
JAVOBGARLIK: har model uchun aniq egasi; "algoritm qaror qildi" - javob emas

- qonuniy asos (rozilik, shartnoma, qonuniy majburiyat va boshqalar)
- maqsad cheklovi, minimallashtirish, saqlash muddati
- sub'ekt huquqlari: o'z ma'lumotini ko'rish, tuzatish, o'chirishni so'rash
- faqat avtomatlashtirilgan, jiddiy oqibatli qarorlarga nisbatan himoya
  g'oyasi (inson ko'rib chiqishi, tushuntirish)
- yuqori xavfli qayta ishlashdan oldin ta'sirni baholash (DPIA g'oyasi)
- sizib chiqish holatida xabar berish majburiyati
- ma'lumotni chegaradan tashqariga uzatish cheklovlari
- kamsitishga qarshi qonunlar - himoyalangan belgilar va ta'sir tahlili

MAQSAD         [ ] qaror kimga ta'sir qiladi? xato narxi kimda?
               [ ] model umuman kerakmi (oddiy qoida yetarlimi)?
MA'LUMOT       [ ] qonuniy asos va rozilik bormi? maqsad mos keladimi?
               [ ] minimallashtirish: har belgi nima uchun kerak?
               [ ] kvaziidentifikatorlar: noyob yozuvlar ulushi, k
               [ ] yorliq qanday olingan? unda tarafkashlik bo'lishi mumkinmi?
               [ ] qaysi guruhlar vakili kam?
MODEL          [ ] himoyalangan belgi faqat audit uchun, cheklangan kirish bilan
               [ ] proksi tekshiruvi (g ni belgilardan bashorat qilish AUC)
               [ ] guruh metrikalari: DP, TPR, FPR, PPV, kalibrlash + SE
               [ ] qaysi metrika ustuvor va NEGA - yozma asoslash
               [ ] yumshatish variantlari: har birining narxi o'lchangan
QAROR          [ ] inson nazorati, apellyatsiya kanali, sabab kodlari
               [ ] model kartasi va datasheet
ISHLAB CHIQ.   [ ] guruh metrikalari monitoringi va chegaralari
               [ ] audit jurnali, egasi, qayta ko'rib chiqish sanasi
               [ ] saqlash muddati va o'chirish jarayoni

import numpy as np
import pandas as pd

# k-anonimlik: har yozuvning sinf hajmi
qi = ["yosh_g", "jins", "indeks_g"]
hajm = df.groupby(qi)["jins"].transform("size")
k = int(hajm.min())
noyob_ulush = float((hajm == 1).mean())
xavfli_ulush = float((hajm < 5).mean())          # k = 5 uchun o'chiriladigan


# umumlashtirish
def yosh_oraliq(yosh, w):
    q = (yosh // w) * w
    return q.astype(str) + "-" + (q + w - 1).astype(str)


indeks_g = df["indeks"].astype(str).str[:4] + "**"

# differensial maxfiylik: Laplace
rng = np.random.default_rng(0)
javob = haqiqiy_sanoq + rng.laplace(0.0, 1.0 / eps)          # sezgirlik 1
orta = x.clip(a, b).mean() + rng.laplace(0.0, (b - a) / len(x) / eps)


# guruh metrikalari
def guruh_metrikalari(d, y, g):
    r = {}
    for gg in np.unique(g):
        m = g == gg
        r[gg] = {"tasdiq": d[m].mean(),
                 "TPR": d[m & (y == 1)].mean(),
                 "FPR": d[m & (y == 0)].mean(),
                 "PPV": y[m & (d == 1)].mean()}
    return r


# qayta og'irlash (reweighing)
w = np.ones(len(y))
for gg in (0, 1):
    for yy in (0, 1):
        m = (g == gg) & (y == yy)
        w[m] = (g == gg).mean() * (y == yy).mean() / m.mean()
# model.fit(X, y, sample_weight=w)

# guruhga xos chegara: B guruhida tasdiq ulushini A ga tenglash
ulush_a = (p[g == 0] >= 0.5).mean()
t_b = np.quantile(p[g == 1], 1 - ulush_a)
d = np.where(g == 1, p >= t_b, p >= 0.5)

minimal yig' -> QI ni tekshir (noyob ulush, k) -> umumlashtir + o'chir -> foydalilikni o'lcha
e'lon: DP, eps oshkor, byudjet;  aniq so'rov interfeysi yo'q
adolat: g faqat audit uchun -> proksi AUC -> DP, TPR, FPR, PPV, kalibrlash + SE
belgini olib tashlash yetarli emas; metrikalar ziddiyatli - tanlovni yozing
yumshatish: narxini o'lchang, huquqiy tomonini ayting; inson nazorati va apellyatsiya

"""k-anonimlik noldan: kvaziidentifikatorlar, bog'lash hujumi, umumlashtirish."""

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

TASHXISLAR = ["sog'lom", "gipertoniya", "diabet", "astma", "depressiya"]
DARAJALAR = [  # (nom, yosh oralig'i, indeksdan yashiriladigan raqamlar)
    ("L0 xom", 1, 0),
    ("L1 yosh 5", 5, 0),
    ("L2 yosh 5, indeks 5 raqam", 5, 1),
    ("L3 yosh 10, indeks 4 raqam", 10, 2),
    ("L4 yosh 20, indeks 4 raqam", 20, 2),
    ("L5 yosh 20, indeks 3 raqam", 20, 3),
]


def yarat(seed=0, n=6000):
    """Tibbiy markaz bazasi: yosh, jins, pochta indeksi va tashxis."""
    rng = np.random.default_rng(seed)
    yosh = np.clip(rng.gamma(6.0, 7.0, n) + 12, 18, 90).astype(int)
    jins = rng.choice(["erkak", "ayol"], n)
    tuman = rng.choice(9, n, p=np.array([5, 4, 4, 3, 3, 2, 2, 1, 1]) / 25)
    uchastka = rng.integers(0, 30, n)
    indeks = 100000 + (tuman + 1) * 100 + uchastka
    risk = np.column_stack([
        np.full(n, 2.0),
        -2.5 + 0.045 * yosh,
        -4.0 + 0.05 * yosh,
        np.full(n, -1.2),
        np.full(n, -1.0),
    ])
    p = np.exp(risk) / np.exp(risk).sum(1, keepdims=True)
    tashxis = np.array([rng.choice(5, p=q) for q in p])
    return pd.DataFrame({"yosh": yosh, "jins": jins, "indeks": indeks,
                         "tashxis": np.array(TASHXISLAR)[tashxis]})


def umumlashtir(df, oraliq, yashir):
    """Yoshni oraliqqa, indeksni oxirgi raqamlarini yulduzcha bilan almashtiradi."""
    quyi = (df["yosh"] // oraliq) * oraliq
    yosh_g = (quyi.astype(str) + "-" + (quyi + oraliq - 1).astype(str)
              if oraliq > 1 else df["yosh"].astype(str))
    s = df["indeks"].astype(str)
    indeks_g = s.str[:6 - yashir] + "*" * yashir if yashir else s
    return pd.DataFrame({"yosh_g": yosh_g, "jins": df["jins"], "indeks_g": indeks_g,
                         "orta_yosh": quyi + (oraliq - 1) / 2})


def main() -> None:
    df = yarat()
    qi = ["yosh_g", "jins", "indeks_g"]
    print("=== 1. Baza: 6000 bemor, kvaziidentifikatorlar: yosh, jins, indeks ===")
    print(f"  noyob yosh {df.yosh.nunique()}, noyob indeks {df.indeks.nunique()}")
    print("  tashxislar:", ", ".join(f"{k} {v:.3f}" for k, v in
                                     df.tashxis.value_counts(normalize=True)
                                     .sort_index().items()))

    rng = np.random.default_rng(1)
    ochiq_i = np.sort(rng.choice(len(df), 1500, replace=False))
    ochiq = df.iloc[ochiq_i][["yosh", "jins", "indeks"]].copy()
    ochiq["ism"] = [f"shaxs_{i:04d}" for i in range(len(ochiq))]
    print("  ochiq ro'yxat (masalan, klub a'zolari): 1500 kishi - ism + yosh, "
          "jins, indeks")

    print("\n=== 2. Umumlashtirish darajalari: xavf va foydalilik ===")
    print(f"  {'daraja':<27} {'sinflar':>7} {'k':>3} {'noyob':>6} {'k<5':>6} "
          f"{'bog_landi':>9} {'bir_xil':>7} {'hudud':>5} {'yosh_MAE':>8} {'AUC':>6}")
    y_sur = df.tashxis.isin(["diabet", "gipertoniya"]).to_numpy().astype(int)
    natijalar = []
    for nom, oraliq, yashir in DARAJALAR:
        g = umumlashtir(df, oraliq, yashir)
        hajm = g.groupby(qi)["jins"].transform("size")
        sinf_soni = g.groupby(qi).ngroups
        noyob = float((hajm == 1).mean())
        kichik = float((hajm < 5).mean())
        # bog'lash hujumi: ochiq ro'yxatdagi odam umumlashgan bazada yolg'iz
        og = umumlashtir(ochiq, oraliq, yashir)
        kalit = g[qi].astype(str).agg("|".join, axis=1)
        okalit = og[qi].astype(str).agg("|".join, axis=1)
        sanoq = kalit.value_counts()
        bog = int((okalit.map(sanoq).fillna(0) == 1).sum())
        # bir xillik: k >= 5 bo'lsa ham sinfda bitta tashxis
        tur = g.assign(t=df.tashxis).groupby(qi)["t"].transform("nunique")
        bir_xil = float(((hajm >= 5) & (tur == 1)).mean())
        mae = float(np.abs(g["orta_yosh"] - df["yosh"]).mean())
        X = np.column_stack([g["orta_yosh"], g["orta_yosh"] ** 2 / 100,
                             (g["jins"] == "ayol").astype(float)])
        yarim = len(df) // 2
        m = LogisticRegression(max_iter=2000).fit(X[:yarim], y_sur[:yarim])
        auc = roc_auc_score(y_sur[yarim:], m.predict_proba(X[yarim:])[:, 1])
        k = int(hajm.min())
        natijalar.append((nom, k, kichik, auc))
        print(f"  {nom:<27} {sinf_soni:>7} {k:>3} {noyob:>6.3f} {kichik:>6.3f} "
              f"{bog:>9} {bir_xil:>7.3f} {g.indeks_g.nunique():>5} {mae:>8.2f} "
              f"{auc:>6.3f}")

    print("\n=== 3. k = 5 uchun: umumlashtirish + kichik sinflarni o'chirish ===")
    tanlov = None
    for nom, oraliq, yashir in DARAJALAR:
        g = umumlashtir(df, oraliq, yashir)
        hajm = g.groupby(qi)["jins"].transform("size")
        ochir = float((hajm < 5).mean())
        ok = ochir <= 0.02
        holat = "yetarli (<= 2%)" if ok else "juda ko'p"
        print(f"  {nom:<27} o'chiriladigan ulush {ochir:.3f}  {holat}")
        if ok and tanlov is None:
            tanlov = nom
    print(f"  eng kam umumlashtirish bilan k >= 5 (o'chirish <= 2%): {tanlov}")
    oraliq, yashir = [(o, y) for n, o, y in DARAJALAR if n == tanlov][0]
    g = umumlashtir(df, oraliq, yashir)
    hajm = g.groupby(qi)["jins"].transform("size")
    qoldi = g[hajm >= 5]
    print(f"  o'chirishdan keyin: {len(qoldi)} qator, k = "
          f"{qoldi.groupby(qi).size().min()}, har yozuv kamida 5 kishi bilan "
          f"bir xil ko'rinadi")
    auc0 = natijalar[0][3]
    auct = [a for n, _, _, a in natijalar if n == tanlov][0]
    print(f"  surunkali kasallik modeli AUC: xom {auc0:.3f} -> tanlangan {auct:.3f} "
          f"(farq {auct - auc0:+.3f})")
    print("  ⭐ Ismni o'chirish anonimlik emas: yosh + jins + indeks - barmoq izi")


if __name__ == "__main__":
    main()

=== 1. Baza: 6000 bemor, kvaziidentifikatorlar: yosh, jins, indeks ===
  noyob yosh 73, noyob indeks 270
  tashxislar: astma 0.033, depressiya 0.038, diabet 0.034, gipertoniya 0.120, sog'lom 0.775
  ochiq ro'yxat (masalan, klub a'zolari): 1500 kishi - ism + yosh, jins, indeks

=== 2. Umumlashtirish darajalari: xavf va foydalilik ===
  daraja                      sinflar   k  noyob    k<5 bog_landi bir_xil hudud yosh_MAE    AUC
  L0 xom                         5319   1  0.782  1.000      1190   0.000   270     0.00  0.715
  L1 yosh 5                      3681   1  0.370  0.944       557   0.014   270     1.22  0.715
  L2 yosh 5, indeks 5 raqam       761   1  0.013  0.120        19   0.075    27     1.22  0.715
  L3 yosh 10, indeks 4 raqam      150   1  0.001  0.005         2   0.007     9     2.57  0.713
  L4 yosh 20, indeks 4 raqam       78   1  0.001  0.001         2   0.003     9     4.85  0.707
  L5 yosh 20, indeks 3 raqam       10   2  0.000  0.001         0   0.000     1     4.85  0.707

=== 3. k = 5 uchun: umumlashtirish + kichik sinflarni o'chirish ===
  L0 xom                      o'chiriladigan ulush 1.000  juda ko'p
  L1 yosh 5                   o'chiriladigan ulush 0.944  juda ko'p
  L2 yosh 5, indeks 5 raqam   o'chiriladigan ulush 0.120  juda ko'p
  L3 yosh 10, indeks 4 raqam  o'chiriladigan ulush 0.005  yetarli (<= 2%)
  L4 yosh 20, indeks 4 raqam  o'chiriladigan ulush 0.001  yetarli (<= 2%)
  L5 yosh 20, indeks 3 raqam  o'chiriladigan ulush 0.001  yetarli (<= 2%)
  eng kam umumlashtirish bilan k >= 5 (o'chirish <= 2%): L3 yosh 10, indeks 4 raqam
  o'chirishdan keyin: 5967 qator, k = 5, har yozuv kamida 5 kishi bilan bir xil ko'rinadi
  surunkali kasallik modeli AUC: xom 0.715 -> tanlangan 0.713 (farq -0.002)
  ⭐ Ismni o'chirish anonimlik emas: yosh + jins + indeks - barmoq izi

"""Differensial maxfiylik noldan: Laplace mexanizmi, epsilon va aniqlik murosasi."""

import numpy as np

EPSLAR = [0.01, 0.1, 0.5, 1.0, 2.0, 5.0]


def laplace(rng, qiymat, sezgirlik, eps, size=None):
    """Laplace mexanizmi: qiymat + Laplace(0, sezgirlik / eps)."""
    return qiymat + rng.laplace(0.0, sezgirlik / eps, size)


def main() -> None:
    rng = np.random.default_rng(0)
    n = 6000
    yosh = np.clip(rng.gamma(6.0, 7.0, n) + 12, 18, 90)
    diabet = rng.random(n) < 1 / (1 + np.exp(-(-6.0 + 0.055 * yosh)))
    uchastka = rng.integers(0, 270, n)
    kichik = uchastka == 7

    print("=== 1. Farqlash hujumi: aniq javob beradigan so'rov tizimi ===")
    jami = int(diabet.sum())
    print(f"  so'rov A: 'bazadagi diabetlar soni' = {jami}")
    nishonlar = rng.choice(n, 1000, replace=False)
    togri = 0
    for i in nishonlar:
        b = jami - int(diabet[np.arange(n) != i].sum())  # 'i dan boshqa hammasi'
        togri += int((b == 1) == diabet[i])
    print(f"  so'rov B: 'X dan boshqa hamma' -> A - B X ning tashxisi")
    print(f"  1000 nishondan to'g'ri topilgani: {togri} ({togri / 10:.1f}%)")

    print("\n=== 2. Laplace mexanizmi: sanoq (sezgirlik 1), 2000 takror ===")
    print(f"  haqiqiy: jami diabet {jami}, kichik uchastkada {int(diabet[kichik].sum())}"
          f" ({int(kichik.sum())} kishi); o'rtacha yosh {yosh.mean():.2f}")
    print(f"  {'eps':>5} {'MAE':>8} {'1/eps':>8} {'jami nisbiy':>11} "
          f"{'uchastka nisbiy':>15} {'orta yosh MAE':>13}")
    sez_yosh = (90 - 18) / n  # chegaralangan o'rtacha: bitta odam ta'siri
    kich = int(diabet[kichik].sum())
    for eps in EPSLAR:
        s = laplace(rng, jami, 1.0, eps, 2000)
        mae = np.abs(s - jami).mean()
        u = laplace(rng, kich, 1.0, eps, 2000)
        y = laplace(rng, yosh.mean(), sez_yosh, eps, 2000)
        print(f"  {eps:>5.2f} {mae:>8.2f} {1 / eps:>8.2f} {mae / jami:>11.4f} "
              f"{np.abs(u - kich).mean() / max(kich, 1):>15.2f} "
              f"{np.abs(y - yosh.mean()).mean():>13.4f}")

    print("\n=== 3. Farqlash hujumi DP bilan (har so'rov eps, jami 2*eps) ===")
    diab_i = np.flatnonzero(diabet)
    sog_i = np.flatnonzero(~diabet)
    nish = np.concatenate([rng.choice(diab_i, 300, replace=False),
                           rng.choice(sog_i, 300, replace=False)])
    nish = np.repeat(nish, 10)  # har nishonga 10 ta mustaqil urinish
    print("  600 nishon (300 diabet, 300 sog'lom) x 10 urinish")
    print(f"  {'eps':>5} {'hujum aniqligi':>14} {'nazariy chegara':>15}")
    for eps in EPSLAR:
        a = laplace(rng, jami, 1.0, eps, len(nish))
        b = laplace(rng, jami - diabet[nish].astype(int), 1.0, eps, len(nish))
        aniq = float(np.mean(((a - b) > 0.5) == diabet[nish]))
        chegara = np.exp(2 * eps) / (1 + np.exp(2 * eps))
        print(f"  {eps:>5.2f} {aniq:>14.3f} {chegara:>15.3f}")

    print("\n=== 4. Kompozitsiya: bir so'rovni m marta so'rash (har biri eps=0.1) ===")
    print(f"  {'m':>4} {'jami eps':>8} {'o_rtachaning MAE':>16}")
    for m in [1, 10, 100]:
        s = laplace(rng, jami, 1.0, 0.1, (2000, m)).mean(1)
        print(f"  {m:>4} {0.1 * m:>8.1f} {np.abs(s - jami).mean():>16.2f}")
    print("  -> takroriy so'rovlar shovqinni o'rtachalab yo'qotadi: shuning uchun")
    print("     jami maxfiylik 'byudjeti' hisoblanadi va cheklanadi")
    print("  ⭐ Epsilon kichik - maxfiylik kuchli, aniqlik past; kichik guruhlar")
    print("     birinchi bo'lib 'yo'qoladi'")


if __name__ == "__main__":
    main()

=== 1. Farqlash hujumi: aniq javob beradigan so'rov tizimi ===
  so'rov A: 'bazadagi diabetlar soni' = 399
  so'rov B: 'X dan boshqa hamma' -> A - B X ning tashxisi
  1000 nishondan to'g'ri topilgani: 1000 (100.0%)

=== 2. Laplace mexanizmi: sanoq (sezgirlik 1), 2000 takror ===
  haqiqiy: jami diabet 399, kichik uchastkada 2 (25 kishi); o'rtacha yosh 53.90
    eps      MAE    1/eps jami nisbiy uchastka nisbiy orta yosh MAE
   0.01    95.60   100.00      0.2396           49.97        1.2096
   0.10    10.17    10.00      0.0255            5.00        0.1224
   0.50     2.02     2.00      0.0051            0.99        0.0236
   1.00     1.01     1.00      0.0025            0.51        0.0119
   2.00     0.52     0.50      0.0013            0.25        0.0060
   5.00     0.20     0.20      0.0005            0.10        0.0024

=== 3. Farqlash hujumi DP bilan (har so'rov eps, jami 2*eps) ===
  600 nishon (300 diabet, 300 sog'lom) x 10 urinish
    eps hujum aniqligi nazariy chegara
   0.01          0.505           0.505
   0.10          0.521           0.550
   0.50          0.553           0.731
   1.00          0.626           0.881
   2.00          0.734           0.982
   5.00          0.908           1.000

=== 4. Kompozitsiya: bir so'rovni m marta so'rash (har biri eps=0.1) ===
     m jami eps o_rtachaning MAE
     1      0.1             9.51
    10      1.0             3.45
   100     10.0             1.15
  -> takroriy so'rovlar shovqinni o'rtachalab yo'qotadi: shuning uchun
     jami maxfiylik 'byudjeti' hisoblanadi va cheklanadi
  ⭐ Epsilon kichik - maxfiylik kuchli, aniqlik past; kichik guruhlar
     birinchi bo'lib 'yo'qoladi'

"""Adolatlilik metrikalari noldan: tarixiy tarafkashlik va proksi belgi orqali sizish."""

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

SOHALAR = ["talim", "tibbiyot", "savdo", "IT", "qurilish", "transport"]
SONLI = ["log_daromad", "kredit_tarixi", "qarz_yuki", "staj"]


def yarat(seed, n=16000):
    """Haqiqiy to'lov qobiliyati jinsga bog'liq EMAS; tarixiy qarorlar esa bog'liq."""
    rng = np.random.default_rng(seed)
    ayol = rng.random(n) < 0.5
    p_soha = np.where(ayol[:, None], [0.4, 0.3, 0.15, 0.05, 0.05, 0.05],
                      [0.04, 0.04, 0.17, 0.2, 0.3, 0.25])
    soha = np.array([rng.choice(6, p=q) for q in p_soha])
    log_daromad = rng.normal(1.6, 0.45, n)            # mln so'm, log
    kredit_tarixi = rng.beta(4, 2, n)
    qarz_yuki = rng.beta(2, 5, n)
    staj = rng.gamma(2.0, 3.0, n)
    ball = (-2.65 + 1.6 * log_daromad + 2.2 * kredit_tarixi - 3.0 * qarz_yuki
            + 0.08 * staj)
    qaytaradi = rng.random(n) < 1 / (1 + np.exp(-ball))  # haqiqiy natija
    # tarixiy inspektor: haqiqiy ballni shovqin bilan ko'radi + ayollarga jarima
    tarixiy = (ball + rng.normal(0, 0.8, n) - 0.7 * ayol) > 0.3
    df = pd.DataFrame({"ayol": ayol.astype(int), "soha": soha,
                       "log_daromad": log_daromad, "kredit_tarixi": kredit_tarixi,
                       "qarz_yuki": qarz_yuki, "staj": staj,
                       "tarixiy": tarixiy.astype(int),
                       "qaytaradi": qaytaradi.astype(int)})
    return df


def belgilar(df, jins=False, soha=True):
    X = df[SONLI].copy()
    if soha:
        for j, s in enumerate(SOHALAR):
            X["soha_" + s] = (df["soha"] == j).astype(float)
    if jins:
        X["ayol"] = df["ayol"]
    return X


def metrikalar(d, y_true, g):
    """Guruh metrikalari: d - qaror (1 = tasdiq), y_true - haqiqiy qaytarish."""
    r = {}
    for nom, m in [("erkak", g == 0), ("ayol", g == 1)]:
        r[nom] = {
            "tasdiq": d[m].mean(),
            "TPR": d[m & (y_true == 1)].mean(),
            "FPR": d[m & (y_true == 0)].mean(),
            "PPV": y_true[m & (d == 1)].mean(),
        }
    return r


def main() -> None:
    df = yarat(0)
    oq, ts = df.iloc[:8000], df.iloc[8000:]
    g = ts["ayol"].to_numpy()
    y = ts["qaytaradi"].to_numpy()

    print("=== 1. Ma'lumot: haqiqat va tarix ===")
    for nom, m in [("erkak", df.ayol == 0), ("ayol", df.ayol == 1)]:
        print(f"  {nom:<6} haqiqiy qaytarish {df.qaytaradi[m].mean():.3f}, "
              f"tarixiy tasdiq {df.tarixiy[m].mean():.3f}")
    X_proksi = belgilar(df, jins=False, soha=True)
    X_toza = belgilar(df, jins=False, soha=False)
    for nom, X in [("soha bilan", X_proksi), ("sohasiz", X_toza)]:
        m = LogisticRegression(max_iter=2000).fit(X.iloc[:8000], oq["ayol"])
        auc = roc_auc_score(g, m.predict_proba(X.iloc[8000:])[:, 1])
        print(f"  jinsni boshqa belgilardan bashorat qilish ({nom}): AUC {auc:.3f}")

    print("\n=== 2. Tarixiy qarorlarni o'rgangan modellar (test, chegara 0.5) ===")
    variantlar = [("A: jins bilan", True, True), ("B: jinssiz, soha bor", False, True),
                  ("C: jinssiz, sohasiz", False, False)]
    print(f"  {'model':<22} {'tarixga mos':>11} {'haqiqatga mos':>13} "
          f"{'DP farq':>8} {'TPR farq':>8} {'FPR farq':>8} {'PPV farq':>8}")
    for nom, jins, soha in variantlar:
        X = belgilar(df, jins, soha)
        m = LogisticRegression(max_iter=2000).fit(X.iloc[:8000], oq["tarixiy"])
        d = (m.predict_proba(X.iloc[8000:])[:, 1] >= 0.5).astype(int)
        r = metrikalar(d, y, g)
        e, a = r["erkak"], r["ayol"]
        print(f"  {nom:<22} {np.mean(d == ts['tarixiy']):>11.3f} "
              f"{np.mean(d == y):>13.3f} {a['tasdiq'] - e['tasdiq']:>+8.3f} "
              f"{a['TPR'] - e['TPR']:>+8.3f} {a['FPR'] - e['FPR']:>+8.3f} "
              f"{a['PPV'] - e['PPV']:>+8.3f}")
    X = belgilar(df, False, True)
    m = LogisticRegression(max_iter=2000).fit(X.iloc[:8000], oq["tarixiy"])
    koef = dict(zip(X.columns, m.coef_[0]))
    print("  B modelida soha koeffitsientlari: "
          + ", ".join(f"{s} {koef['soha_' + s]:+.2f}" for s in SOHALAR))

    print("\n=== 3. Guruhlar bo'yicha kalibrlash (B modeli, haqiqiy qaytarish) ===")
    p = m.predict_proba(X.iloc[8000:])[:, 1]
    bins = np.array([0, 0.2, 0.4, 0.6, 0.8, 1.0])
    print(f"  {'ball oralig_i':<14} {'erkak: n, qaytardi':>20} {'ayol: n, qaytardi':>19}")
    for lo, hi in zip(bins[:-1], bins[1:]):
        qator = f"  [{lo:.1f}, {hi:.1f})    "
        for gg in [0, 1]:
            s = (p >= lo) & (p < hi) & (g == gg)
            qator += f" {int(s.sum()):>8} {y[s].mean() if s.sum() else float('nan'):>9.3f}"
        print(qator)

    print("\n=== 4. 10 urug': ayol - erkak farqlari (o'rtacha va SE) ===")
    natija = {v[0]: {"DP": [], "TPR": []} for v in variantlar}
    for seed in range(1, 11):
        d_ = yarat(seed)
        y_ = d_["qaytaradi"].to_numpy()[8000:]
        g_ = d_["ayol"].to_numpy()[8000:]
        for nom, jins, soha in variantlar:
            X = belgilar(d_, jins, soha)
            mm = LogisticRegression(max_iter=2000).fit(X.iloc[:8000],
                                                       d_["tarixiy"].iloc[:8000])
            dd = (mm.predict_proba(X.iloc[8000:])[:, 1] >= 0.5).astype(int)
            r = metrikalar(dd, y_, g_)
            natija[nom]["DP"].append(r["ayol"]["tasdiq"] - r["erkak"]["tasdiq"])
            natija[nom]["TPR"].append(r["ayol"]["TPR"] - r["erkak"]["TPR"])
    for nom in natija:
        dp, tpr = np.array(natija[nom]["DP"]), np.array(natija[nom]["TPR"])
        print(f"  {nom:<22} DP {dp.mean():+.3f} (SE {dp.std(ddof=1) / np.sqrt(10):.3f})"
              f"  TPR {tpr.mean():+.3f} (SE {tpr.std(ddof=1) / np.sqrt(10):.3f})")
    a_ = np.array(natija["A: jins bilan"]["DP"])
    b_ = np.array(natija["B: jinssiz, soha bor"]["DP"])
    c_ = np.array(natija["C: jinssiz, sohasiz"]["DP"])
    ulush = b_.mean() / a_.mean()
    print(f"  jinsni olib tashlash DP farqining {1 - ulush:.0%} ini yo'qotdi, "
          f"{ulush:.0%} i proksi orqali qoldi")
    f = c_ - b_
    se = f.std(ddof=1) / np.sqrt(len(f))
    print(f"  sohani ham olib tashlash: DP farqi o'zgarishi {f.mean():+.3f} (SE {se:.3f})"
          f", sezilarli: {abs(f.mean()) > 2 * se}")
    print("  ⭐ Belgini olib tashlash yetarli emas: tarafkashlik YORLIQDA va proksida")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot: haqiqat va tarix ===
  erkak  haqiqiy qaytarish 0.691, tarixiy tasdiq 0.712
  ayol   haqiqiy qaytarish 0.697, tarixiy tasdiq 0.505
  jinsni boshqa belgilardan bashorat qilish (soha bilan): AUC 0.854
  jinsni boshqa belgilardan bashorat qilish (sohasiz): AUC 0.496

=== 2. Tarixiy qarorlarni o'rgangan modellar (test, chegara 0.5) ===
  model                  tarixga mos haqiqatga mos  DP farq TPR farq FPR farq PPV farq
  A: jins bilan                0.807         0.692   -0.265   -0.254   -0.298   +0.068
  B: jinssiz, soha bor         0.794         0.700   -0.138   -0.137   -0.147   +0.033
  C: jinssiz, sohasiz          0.781         0.701   -0.013   -0.021   +0.000   +0.000
  B modelida soha koeffitsientlari: talim -0.66, tibbiyot -0.73, savdo -0.03, IT +0.41, qurilish +0.46, transport +0.53

=== 3. Guruhlar bo'yicha kalibrlash (B modeli, haqiqiy qaytarish) ===
  ball oralig_i    erkak: n, qaytardi   ayol: n, qaytardi
  [0.0, 0.2)          468     0.338      790     0.430
  [0.2, 0.4)          450     0.507      627     0.609
  [0.4, 0.6)          517     0.609      588     0.677
  [0.6, 0.8)          764     0.715      719     0.762
  [0.8, 1.0)         1835     0.851     1242     0.899

=== 4. 10 urug': ayol - erkak farqlari (o'rtacha va SE) ===
  A: jins bilan          DP -0.263 (SE 0.004)  TPR -0.245 (SE 0.003)
  B: jinssiz, soha bor   DP -0.122 (SE 0.004)  TPR -0.112 (SE 0.004)
  C: jinssiz, sohasiz    DP -0.006 (SE 0.003)  TPR -0.004 (SE 0.003)
  jinsni olib tashlash DP farqining 54% ini yo'qotdi, 46% i proksi orqali qoldi
  sohani ham olib tashlash: DP farqi o'zgarishi +0.116 (SE 0.003), sezilarli: True
  ⭐ Belgini olib tashlash yetarli emas: tarafkashlik YORLIQDA va proksida

"""Adolatlilik metrikalari ziddiyati va yumshatish: qayta og'irlash, guruhga xos chegara."""

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression

SOHALAR = ["talim", "tibbiyot", "savdo", "IT", "qurilish", "transport"]
SONLI = ["log_daromad", "kredit_tarixi", "qarz_yuki", "staj"]


def yarat(seed, n=16000):
    """3-misoldagi kredit ma'lumoti (haqiqiy qobiliyat jinsga bog'liq emas)."""
    rng = np.random.default_rng(seed)
    ayol = rng.random(n) < 0.5
    p_soha = np.where(ayol[:, None], [0.4, 0.3, 0.15, 0.05, 0.05, 0.05],
                      [0.04, 0.04, 0.17, 0.2, 0.3, 0.25])
    soha = np.array([rng.choice(6, p=q) for q in p_soha])
    log_daromad = rng.normal(1.6, 0.45, n)
    kredit_tarixi = rng.beta(4, 2, n)
    qarz_yuki = rng.beta(2, 5, n)
    staj = rng.gamma(2.0, 3.0, n)
    ball = (-2.65 + 1.6 * log_daromad + 2.2 * kredit_tarixi - 3.0 * qarz_yuki
            + 0.08 * staj)
    qaytaradi = rng.random(n) < 1 / (1 + np.exp(-ball))
    tarixiy = (ball + rng.normal(0, 0.8, n) - 0.7 * ayol) > 0.3
    df = pd.DataFrame({"ayol": ayol.astype(int), "log_daromad": log_daromad,
                       "kredit_tarixi": kredit_tarixi, "qarz_yuki": qarz_yuki,
                       "staj": staj, "tarixiy": tarixiy.astype(int),
                       "qaytaradi": qaytaradi.astype(int)})
    for j, s in enumerate(SOHALAR):
        df["soha_" + s] = (soha == j).astype(float)
    return df


def guruh(d, y, g):
    """Har guruh uchun tasdiq ulushi, TPR, FPR, PPV."""
    out = []
    for m in [g == 0, g == 1]:
        out.append((d[m].mean(), d[m & (y == 1)].mean(), d[m & (y == 0)].mean(),
                    y[m & (d == 1)].mean()))
    return out


def ziddiyat() -> None:
    print("=== 1. Ziddiyat: kalibrlangan ball, asosiy stavkalar farqli ===")
    rng = np.random.default_rng(0)
    n = 40000
    g = (rng.random(n) < 0.5).astype(int)
    p = np.where(g == 0, rng.beta(7, 3, n), rng.beta(5, 5, n))  # haqiqiy ehtimol
    y = (rng.random(n) < p).astype(int)
    print(f"  asosiy stavka: A {y[g == 0].mean():.3f}, B {y[g == 1].mean():.3f}")
    print("  kalibrlash (o'rtacha ball / haqiqiy ulush):")
    for lo, hi in [(0.3, 0.4), (0.5, 0.6), (0.7, 0.8)]:
        s = (p >= lo) & (p < hi)
        a, b = s & (g == 0), s & (g == 1)
        print(f"    ball [{lo:.1f}, {hi:.1f}): A {p[a].mean():.3f} / {y[a].mean():.3f}"
              f", B {p[b].mean():.3f} / {y[b].mean():.3f}")
    d = (p >= 0.6).astype(int)
    (ta, tpra, fpra, ppva), (tb, tprb, fprb, ppvb) = guruh(d, y, g)
    print(f"  bitta chegara 0.6: aniqlik {np.mean(d == y):.3f}, "
          f"tasdiqlangan qaytarmaydiganlar {int(((d == 1) & (y == 0)).sum())}")
    print(f"    {'':<4} {'tasdiq':>7} {'TPR':>6} {'FPR':>6} {'PPV':>6}")
    print(f"    {'A':<4} {ta:>7.3f} {tpra:>6.3f} {fpra:>6.3f} {ppva:>6.3f}")
    print(f"    {'B':<4} {tb:>7.3f} {tprb:>6.3f} {fprb:>6.3f} {ppvb:>6.3f}")
    for nom, m, tpr, ppv, fpr in [("A", g == 0, tpra, ppva, fpra),
                                  ("B", g == 1, tprb, ppvb, fprb)]:
        b = y[m].mean()
        print(f"    {nom}: FPR = b/(1-b) * (1-PPV)/PPV * TPR = "
              f"{b / (1 - b) * (1 - ppv) / ppv * tpr:.3f} (o'lchangan {fpr:.3f})")
    # B uchun TPR ni A ga tenglashtiradigan chegara
    t_b = float(np.quantile(p[(g == 1) & (y == 1)], 1 - tpra))
    d2 = np.where(g == 0, p >= 0.6, p >= t_b).astype(int)
    (ta, tpra, fpra, ppva), (tb, tprb, fprb, ppvb) = guruh(d2, y, g)
    print(f"  guruhga xos chegara: A 0.60, B {t_b:.2f} (TPR tenglashtirildi): "
          f"aniqlik {np.mean(d2 == y):.3f}, tasdiqlangan qaytarmaydiganlar "
          f"{int(((d2 == 1) & (y == 0)).sum())}")
    print(f"    {'A':<4} {ta:>7.3f} {tpra:>6.3f} {fpra:>6.3f} {ppva:>6.3f}")
    print(f"    {'B':<4} {tb:>7.3f} {tprb:>6.3f} {fprb:>6.3f} {ppvb:>6.3f}")
    print(f"    ball {t_b + 0.01:.2f} li ikki ariza: A - rad, B - tasdiq "
          f"(bir xil ehtimol, turli qaror)")


def yumshatish() -> None:
    print("\n=== 2. Yumshatish: kredit ma'lumoti, 10 urug' (test 8000) ===")
    usullar = ["asl (proksi bor)", "qayta og'irlash", "guruhga xos chegara",
               "proksisiz (soha yo'q)"]
    res = {u: {"tarix": [], "haqiqat": [], "DP": [], "TPR": []} for u in usullar}
    for seed in range(1, 11):
        df = yarat(seed)
        oq, va, ts = df.iloc[:6000], df.iloc[6000:8000], df.iloc[8000:]
        ust = SONLI + ["soha_" + s for s in SOHALAR]
        g_oq, y_oq = oq["ayol"].to_numpy(), oq["tarixiy"].to_numpy()
        g, y, yt = ts["ayol"].to_numpy(), ts["qaytaradi"].to_numpy(), ts["tarixiy"]
        asl = LogisticRegression(max_iter=2000).fit(oq[ust], y_oq)
        # qayta og'irlash: w(g, y) = P(g) P(y) / P(g, y)
        w = np.ones(len(oq))
        for gg in [0, 1]:
            for yy in [0, 1]:
                m = (g_oq == gg) & (y_oq == yy)
                w[m] = (g_oq == gg).mean() * (y_oq == yy).mean() / m.mean()
        rw = LogisticRegression(max_iter=2000).fit(oq[ust], y_oq, sample_weight=w)
        # guruhga xos chegara: validatsiyada ayollar tasdiq ulushi = erkaklarniki
        pv = asl.predict_proba(va[ust])[:, 1]
        gv = va["ayol"].to_numpy()
        erkak_ulush = (pv[gv == 0] >= 0.5).mean()
        t_ayol = float(np.quantile(pv[gv == 1], 1 - erkak_ulush))
        toza = [c for c in ust if not c.startswith("soha_")]
        pr = LogisticRegression(max_iter=2000).fit(oq[toza], y_oq)
        p_asl = asl.predict_proba(ts[ust])[:, 1]
        qarorlar = {
            usullar[0]: p_asl >= 0.5,
            usullar[1]: rw.predict_proba(ts[ust])[:, 1] >= 0.5,
            usullar[2]: np.where(g == 1, p_asl >= t_ayol, p_asl >= 0.5),
            usullar[3]: pr.predict_proba(ts[toza])[:, 1] >= 0.5,
        }
        for u, d in qarorlar.items():
            d = d.astype(int)
            (te, tpre, _, _), (ta, tpra, _, _) = guruh(d, y, g)
            res[u]["tarix"].append(np.mean(d == yt))
            res[u]["haqiqat"].append(np.mean(d == y))
            res[u]["DP"].append(ta - te)
            res[u]["TPR"].append(tpra - tpre)
    print(f"  {'usul':<22} {'tarixga mos':>11} {'haqiqatga mos':>13} "
          f"{'DP farq':>8} {'TPR farq':>8}")
    for u in usullar:
        r = {k: np.mean(v) for k, v in res[u].items()}
        print(f"  {u:<22} {r['tarix']:>11.3f} {r['haqiqat']:>13.3f} "
              f"{r['DP']:>+8.3f} {r['TPR']:>+8.3f}")
    print("  juftlashgan farq (usul - asl), 10 urug':")
    for u in usullar[1:]:
        qator = f"    {u:<22}"
        for k in ["tarix", "haqiqat"]:
            f = np.array(res[u][k]) - np.array(res[usullar[0]][k])
            se = f.std(ddof=1) / np.sqrt(len(f))
            belgi = "sezilarli" if abs(f.mean()) > 2 * se else "sezilarsiz"
            qator += f" {k} {f.mean():+.4f} (SE {se:.4f}, {belgi})"
        print(qator)
    print("  ⭐ 'Aniqlik' qaysi yorliqqa nisbatan o'lchanganini doim yozing")


def main() -> None:
    ziddiyat()
    yumshatish()


if __name__ == "__main__":
    main()

=== 1. Ziddiyat: kalibrlangan ball, asosiy stavkalar farqli ===
  asosiy stavka: A 0.696, B 0.499
  kalibrlash (o'rtacha ball / haqiqiy ulush):
    ball [0.3, 0.4): A 0.361 / 0.347, B 0.353 / 0.349
    ball [0.5, 0.6): A 0.555 / 0.551, B 0.549 / 0.551
    ball [0.7, 0.8): A 0.750 / 0.745, B 0.741 / 0.736
  bitta chegara 0.6: aniqlik 0.646, tasdiqlangan qaytarmaydiganlar 5526
          tasdiq    TPR    FPR    PPV
    A      0.771  0.834  0.625  0.753
    B      0.268  0.366  0.171  0.680
    A: FPR = b/(1-b) * (1-PPV)/PPV * TPR = 0.625 (o'lchangan 0.625)
    B: FPR = b/(1-b) * (1-PPV)/PPV * TPR = 0.171 (o'lchangan 0.171)
  guruhga xos chegara: A 0.60, B 0.40 (TPR tenglashtirildi): aniqlik 0.647, tasdiqlangan qaytarmaydiganlar 10174
    A      0.771  0.834  0.625  0.753
    B      0.735  0.834  0.637  0.566
    ball 0.41 li ikki ariza: A - rad, B - tasdiq (bir xil ehtimol, turli qaror)

=== 2. Yumshatish: kredit ma'lumoti, 10 urug' (test 8000) ===
  usul                   tarixga mos haqiqatga mos  DP farq TPR farq
  asl (proksi bor)             0.791         0.696   -0.123   -0.113
  qayta og'irlash              0.787         0.700   -0.054   -0.048
  guruhga xos chegara          0.771         0.715   +0.001   +0.003
  proksisiz (soha yo'q)        0.781         0.703   -0.006   -0.004
  juftlashgan farq (usul - asl), 10 urug':
    qayta og'irlash        tarix -0.0036 (SE 0.0006, sezilarli) haqiqat +0.0042 (SE 0.0007, sezilarli)
    guruhga xos chegara    tarix -0.0196 (SE 0.0017, sezilarli) haqiqat +0.0188 (SE 0.0007, sezilarli)
    proksisiz (soha yo'q)  tarix -0.0102 (SE 0.0008, sezilarli) haqiqat +0.0062 (SE 0.0009, sezilarli)
  ⭐ 'Aniqlik' qaysi yorliqqa nisbatan o'lchanganini doim yozing

df = df.drop(columns=["ism", "telefon"])                        # ⚠️ QI qoldi
hajm = df.groupby(["yosh", "jins", "indeks"])["jins"].transform("size")
print((hajm == 1).mean())   # noyob ulush -> umumlashtir, o'chir, qayta tekshir  # ✅

def sanoq(filtr): return int(df[filtr].diabet.sum())            # ⚠️ farqlash hujumi
def sanoq(filtr, eps): return df[filtr].diabet.sum() + rng.laplace(0, 1 / eps)  # ✅ + byudjet

X = df.drop(columns=["jins"]); model.fit(X, y_tarix)            # ⚠️ "hal qilindi"
auc = roc_auc_score(jins_test, LogisticRegression().fit(X_oq, jins_oq)
                    .predict_proba(X_test)[:, 1])              # ✅ proksi tekshiruvi
# + guruh metrikalari: DP, TPR, FPR, PPV, kalibrlash, SE bilan

print(d[g == 1].mean() - d[g == 0].mean())                     # ⚠️ faqat DP
r = guruh_metrikalari(d, y, g)   # tasdiq, TPR, FPR, PPV + kalibrlash jadvali  # ✅

if tpr_farq < -0.05: print("adolatsiz")                         # ⚠️ 40 kishilik guruhda
# bootstrap yoki bir necha urug' bilan SE; |farq| > 2*SE bo'lsa sezilarli     # ✅

acc = (d == y_tarix).mean()   # "adolat aniqlikni pasaytirdi"   # ⚠️
# yorliq qanday olingan? xolis yorliqli namuna bormi? ikkalasini ham yozing   # ✅

d = np.where(g == 1, p >= t_b, p >= 0.5)   # jim-jit deployga             # ⚠️
# model kartasida: usul, sabab, narx (o'lchangan), huquqiy maslahat natijasi  # ✅

1.  # yosh, jins, 6 xonali indeks - 6000 kishida noyob ulush katta yoki kichik?
2.  # k = 7 bo'lgan sinfda hammasi bir xil tashxisli - maxfiylik bormi?
3.  # "hamma" va "X dan boshqa hamma" agregatlari aniq - nima bo'ladi?
4.  # eps = 0.1, sanoq so'rovi - o'rtacha absolyut xato qancha?
5.  # 25 kishilik guruhdagi 2 ta holat sanog'i DP bilan - nima bo'ladi?
6.  # eps = 0.1 li so'rovni 100 marta so'rash - jami eps?
7.  # jins modelda yo'q, soha bor, tarixiy yorliq tarafkash - DP farqi?
8.  # tasdiqlangan ayollar ko'proq qaytaradi (PPV yuqori) - bu nimaning belgisi?
9.  # asosiy stavkalar farqli, ball mukammal kalibrlangan, bitta chegara - TPR teng?
10. # TPR ni guruhga xos chegara bilan tenglasak - PPV?
11. # qayta og'irlash DP farqini to'liq yo'qotadimi?
12. # tarafkash yorliqli olamda adolatli model tarixga mosligi va haqiqatga mosligi?

1.  df = df.drop(columns=["ism", "pasport"])
    df.to_csv("ochiq_baza.csv")          # yosh, jins, indeks, tashxis qoldi

2.  @app.get("/sanoq")
    def sanoq(tuman: str, yosh_min: int, yosh_max: int):
        return int(df.query("...").diabet.sum())

3.  X = df.drop(columns=["jins", "yosh"])
    model.fit(X, df.tarixiy_qaror)       # "endi model adolatli"

4.  print("TPR farqi:", tpr_ayol - tpr_erkak)   # 35 ta ayol arizasi
    print("model adolatsiz")

5.  d = np.where(ayol == 1, p >= 0.35, p >= 0.5)   # deploy, izohsiz