28.6-dars: Ilg'or anomaliya aniqlash

16.10 DA O'RGANILGAN (qayta tushuntirilmaydi):
  IsolationForest - tasodifiy bo'linishlarda tez izolyatsiya = anomaliya
  LOF - mahalliy zichlik; One-Class SVM; PCA qayta tiklash xatosi
  contamination - faqat chegara; baholash - PR-AUC, precision@k
  kontekstual anomaliya - belgi muhandisligi bilan nuqtaviyga aylanadi

BU DARSDA YANGI:
  vaqt konteksti (soat, hafta kuni, bayram) va KETMA-KETLIK
  sababiylik: detektor faqat O'TMISHNI ko'ra oladimi?
  hodisa darajasida baholash va aniqlash kechikishi
  juda nomutanosib yorliqli ma'lumot: nazoratli usullar, sig'im, narx
  yorliqlar kechikib keladi

NUQTALI (point):   bitta soatda keskin sakrash
   ---/\---        global darajada ham g'ayrioddiy bo'lishi MUMKIN,
                   lekin kun ichidagi tebranish katta bo'lsa - yo'q

KONTEKSTLI (contextual):  qiymat o'zi normal, VAQTIGA nisbatan g'ayrioddiy
   juma 20:00 da 39 MW - kunduzgi daraja; shu soatda odatda 57 MW
   global statistika uni KO'RMAYDI (39 - umumiy o'rtachadan past ham emas)
   kontekst = soat, hafta kuni, bayram, harorat

KOLLEKTIV (collective):   har nuqta normal, KETMA-KETLIK g'ayrioddiy
   sensor qotib qoldi: 10 soat bir xil qiymat
   ---_____________---   darajasi normal, lekin kunlik profil yo'qoldi
   aniqlash uchun bir necha nuqta kerak -> KECHIKISH muqarrar

ANOMALIYA EMAS (lekin detektorni aldaydi):
   bayram kunlari, dam olish kunlari, mavsum - KUTILGAN o'zgarish
   ular modelda kontekst sifatida bo'lishi kerak, aks holda yolg'on signal

1. GLOBAL z:   |x - mean| / std  (tarixdan)
   kontekstsiz; kun ichidagi katta tebranish std ni shishiradi

2. ROLLING z:  oxirgi w soatning o'rtachasi va std si bilan  (sababiy)
   o'tgan 24 soat - kun profilining HAMMASI -> "kechki cho'qqi" normal
   anomaliyaning o'zi oynaga kirib, std ni shishiradi (niqoblanish)

3. ROBUST: median va MAD (median absolute deviation)
   z_robust = |x - median| / (1.4826 * MAD)     1.4826 - normal uchun std ga moslash
   bitta sakrash median va MAD ni deyarli o'zgartirmaydi

4. STL QOLDIG'I:  x = trend + mavsum + qoldiq   (statsmodels STL)
   qoldiqda robust z; mavsumiy kontekstni o'zi hisobga oladi
   ⚠️ STL BUTUN qatorni ko'radi (oldingi va KEYINGI qiymatlar) -
      oflayn tahlil uchun zo'r, real vaqtda bu shaklda ishlamaydi

5. BASHORAT QOLDIG'I:  e_t = x_t - x_hat_t,  x_hat - faqat o'tmishdan
   eng sodda: o'tgan 4 haftaning SHU SOATI mediana (mavsumiy naive, 28.1-28.2)
   yaxshiroq: 28.2-28.3 dagi modellar (bayram belgisi bilan)
   qoldiqda robust z; sababiy -> real vaqtda ishlaydi

NUQTA darajasi emas, HODISA darajasi:
   hodisa topildi = hodisa oynasida (+2 soat) kamida bitta signal
   recall = topilgan hodisalar / hamma hodisalar  (tur bo'yicha alohida)
   yolg'on signal / hafta = hech bir hodisaga tegishli bo'lmagan signallar
   KECHIKISH = birinchi signal soati - hodisa boshlanishi

CHEGARA - oldindan belgilangan byudjet bilan (27.11 alerting):
   "haftasiga ~1 yolg'on signal" -> toza KALIBRLASH davrida
   ball taqsimotining 1 - 1/168 kvantili
   har detektorga O'Z chegarasi, lekin BIR XIL yolg'on signal byudjeti
   -> detektorlarni teng sharoitda solishtirish mumkin

KALIBRLASH davri toza bo'lishi kerak; bo'lmasa (real hayotda) -
   robust kvantil yoki qo'lda tozalangan tarix

ulush 0.3%:  60 000 tranzaksiyada ~200 firib
ROC-AUC:     "tasodifiy firib tasodifiy normaldan yuqorimi" - normal
             ko'p bo'lgani uchun FPR = 1% ham 600 yolg'on signal!
PR-AUC:      signallar ichida firib ulushi (precision) - recall bo'yicha
             tasodifiy model: PR-AUC = ulush 0.003-bob, ROC-AUC = 0.5

AMALIY METRIKALAR:
   kunlik sig'im n: tekshiruvchilar kuniga n ta ko'radi
     -> har kuni eng shubhali n ta: kunlik precision, topilgan firib ulushi
   xarajat: FN narxi (yo'qolgan summa) va FP narxi (tekshirish, mijoz
     noqulayligi) - chegara shu yig'indini minimallashtiradi

XOM BELGILAR:   summa, soat, kategoriya
   500 ming so'm - bir mijoz uchun oddiy, boshqasi uchun g'ayrioddiy

SHAXSIY BAZAVIY (har mijoz O'Z O'TMISHI bilan):
   summa_z   = (log summa - o'tmish o'rtachasi) / o'tmish std
   soat_farq = |soat - o'tmishdagi o'rtacha soat|
   yangi_shahar, yangi_qurilma - o'tmishda ko'rilmaganmi
   soni_24s  = oxirgi 24 soatdagi tranzaksiyalar soni
   oraliq    = oldingi tranzaksiyadan beri o'tgan vaqt
   tarix_soni - o'tmish qancha uzun (bazaviy qanchalik ishonchli)

   ⚠️ FAQAT O'TMISH: cumsum - joriy qiymat; groupby().shift()
      16.10-4-misoldagi groupby().transform("median") butun davrni
      (KELAJAKNI ham) ishlatadi - tahlil uchun mayli, ishlab chiqarishda
      bunday belgi yo'q (17.8 leakage)

NAZORATSIZ (IsolationForest, LOF):   yorliq kerak emas
   "g'ayrioddiy" != "firib": sayohatchi, katta xarid ham g'ayrioddiy
   yangi turdagi firibni ham ushlashi mumkin

NAZORATLI (HistGB, logistik regressiya):  tarixiy yorliqlar bilan
   "firib qanday ko'rinadi" ni o'rganadi - odatda ANCHA aniqroq
   nomutanosiblik: class_weight="balanced" (sinf og'irligi)
   faqat MA'LUM turdagi firiblarni yaxshi taniydi

YARIM NAZORATLI:
   a) faqat tasdiqlangan NORMAL misollarda o'qitilgan nazoratsiz model
      (novelty detection, 16.10 dagi novelty=True g'oyasi)
   b) nazoratsiz ball -> nazoratli modelga belgi sifatida
   c) kam yorliq + ko'p yorliqsiz ma'lumot (self-training)

AMALDA: ikkalasi birga - nazoratli model asosiy ball, nazoratsiz model
        "yangi narsa" uchun qo'shimcha signal

signal (tekshirish yoki bloklash):  narx FP (tekshirish, mijoz noqulayligi)
o'tkazib yuborilgan firib:          narx = firib summasi

jami(c) = sum_{firib, ball < c} summa  +  (ball >= c lar soni) * narx_FP

c ni VALIDATSIYA davrida tanlaymiz (test - bir marta)
0.5 - ehtimol uchun "tabiiy" chegara, lekin narxlar teng bo'lmasa
      optimal chegara undan ancha farq qilishi mumkin
narxlar biznes bilan kelishiladi; bu darsdagi raqamlar FARAZIY

FIRIB YORLIG'I qachon ma'lum bo'ladi?
   mijoz shikoyati / chargeback - kunlar yoki HAFTALAR keyin
   o'qitish kuni T da oxirgi haftalardagi firiblarning ko'pi hali "normal"

SODDA YO'L (xato):   noma'lum = 0 -> firiblar "normal" deb o'qitiladi
YETILGAN YO'L:       faqat T - L gacha bo'lgan davr (L - odatiy kechikish)
                     yorliqlar ishonchli, lekin ma'lumot eskiroq va kamroq
BAHOLASH:            test davri yorliqlari ham yetilishi kerak -
                     aks holda precision past ko'rinadi

BO'LISH: har doim vaqt bo'yicha (train - o'tmish, test - kelajak);
         tasodifiy bo'lish bitta epizodning firiblarini ikkala tomonga
         taqsimlaydi va natijani oshirib ko'rsatadi

26.2 dagi avtoenkoder:  x -> enkoder -> z (kichik) -> dekoder -> x_hat
   faqat (asosan) normal ma'lumotda o'qitiladi
   anomaliya bali = |x - x_hat|^2  (qayta tiklash xatosi)
   PCA qayta tiklash xatosining NOCHIZIQLI umumlashmasi

QACHON FOYDALI:
   ustunlar orasida murakkab (nochiziqli) bog'liqlik bor
   anomaliya har ustunda alohida normal, lekin BIRGALIKDA mos emas
   yuqori o'lchamli ma'lumot (rasm, spektr, ko'p sensor)

QACHON ORTIQCHA:
   kichik jadval, anomaliya bitta-ikkita ustunda chetga chiqishda
   -> IsolationForest oddiyroq, tezroq, masshtablash kerak emas
   AE: masshtablash, arxitektura, epoxalar - sozlanadigan narsa ko'p

import numpy as np
import pandas as pd

# vaqt qatori: rolling va robust (faqat o'tmish)
oldin = x.shift(1)
z_roll = (x - oldin.rolling(24).mean()) / oldin.rolling(24).std()
med = oldin.rolling(24).median()
mad = (oldin - med).abs().rolling(24).median()
z_rob = (x - med) / (1.4826 * mad)

# mavsumiy naive bashorat qoldig'i: o'tgan 4 haftaning shu soati
bashorat = np.median([x.shift(168 * k) for k in range(1, 5)], axis=0)
qoldiq = x - bashorat

# chegara: toza kalibrlash davrida haftasiga ~1 yolg'on signal
chegara = np.nanquantile(ball[kalibrlash], 1 - 1 / 168)

# firibgarlik: shaxsiy bazaviy (o'tmish o'rtachasi, joriy qator kirmaydi)
n_old = df.groupby("mijoz").cumcount()
yig = df["ls"].groupby(df["mijoz"]).cumsum() - df["ls"]
df["summa_z"] = (df["ls"] - yig / n_old) / std_otmish

# nazoratli, nomutanosib
m = HistGradientBoostingClassifier(class_weight="balanced", early_stopping=False)
pr_auc = average_precision_score(y_te, m.fit(X_tr, y_tr).predict_proba(X_te)[:, 1])

# kunlik sig'im: har kuni eng shubhali n ta
top = te.assign(b=ball).groupby("kun", group_keys=False).apply(
    lambda g: g.nlargest(20, "b"))

# xarajatli chegara (validatsiyada)
jami = summa[(y == 1) & (ball < c)].sum() + (ball >= c).sum() * narx_fp

kontekst: soat/kun/bayram - "normal" ning o'zi kontekstga bog'liq
sababiylik: real vaqt detektori faqat o'tmishni ko'radi
baholash: hodisa recall, yolg'on signal/hafta, kechikish - bir xil byudjetda
firib: shaxsiy bazaviy, PR-AUC, top-k sig'im, narx bo'yicha chegara
yorliqlar kechikadi: yetilmagan davrni ehtiyot bilan
AE - murakkab bog'liqlikda; oddiy jadvalda IF ko'pincha yetarli

"""Vaqt qatorida anomaliyalar: nuqtali, kontekstli, kollektiv - besh detektor."""

import warnings

import numpy as np
import pandas as pd
from statsmodels.tsa.seasonal import STL

HAFTA = 168


def qator(seed):
    """Tuman elektr iste'moli (MW), soatlik, 20 hafta; 9-20-haftalarga anomaliyalar."""
    rng = np.random.default_rng(seed)
    n = 20 * HAFTA
    t = np.arange(n)
    soat, kun = t % 24, (t // 24) % 7
    profil = (40 + 12 * np.exp(-(soat - 9) ** 2 / 8) + 20 * np.exp(-(soat - 20) ** 2 / 6)
              - 8 * (soat < 6))
    x = profil * np.where(kun >= 5, 0.85, 1.0) + 0.02 * t / 24 \
        + 3 * np.sin(2 * np.pi * t / (24 * 30))
    shovqin = np.zeros(n)
    e = rng.normal(0, 1.5, n)
    for i in range(1, n):
        shovqin[i] = 0.6 * shovqin[i - 1] + e[i]
    x = x + shovqin
    bayram = np.zeros(n, dtype=bool)                   # sintetik taqvimda bayram kunlari
    b0 = 12 * HAFTA + 24                               # 13-hafta seshanba-chorshanba
    bayram[b0:b0 + 48] = True
    x[bayram] *= 0.8
    hodisalar, band = [], np.zeros(n, dtype=bool)

    def joy(uzunlik, shart):
        while True:
            s = int(rng.integers(8 * HAFTA + 24, n - uzunlik - 24))
            if shart(s) and not band[s - 24:s + uzunlik + 24].any() and not bayram[s - 24:s + uzunlik + 24].any():
                band[s:s + uzunlik] = True
                return s
    for _ in range(8):                                  # nuqtali: bitta soatda sakrash
        s = joy(1, lambda s: True)
        x[s] += rng.choice([-1, 1]) * rng.uniform(12, 20)
        hodisalar.append(("nuqtali", s, s + 1))
    for _ in range(6):                                  # kontekstli: kechki cho'qqida kunduzgi daraja
        s = joy(3, lambda s: s % 24 == 19 and (s // 24) % 7 < 5)
        x[s:s + 3] = x[s - 5:s - 2].mean() - 3          # 14-16 soatdagi daraja - umumiy normal
        hodisalar.append(("kontekstli", s, s + 3))
    for _ in range(4):                                  # kollektiv: sensor qotib qoldi
        s = joy(10, lambda s: True)
        x[s:s + 10] = x[s]
        hodisalar.append(("kollektiv", s, s + 10))
    return pd.Series(x, index=pd.date_range("2025-01-06", periods=n, freq="h")), hodisalar, bayram


def ballar(x):
    """Har detektor: har soat uchun 'g'ayrioddiylik' bali (katta = shubhali)."""
    v = x.to_numpy()
    tarix = v[:8 * HAFTA]
    b = {"global z": np.abs(v - tarix.mean()) / tarix.std()}
    oldin = x.shift(1)
    o, s = oldin.rolling(24).mean(), oldin.rolling(24).std()
    b["rolling z (24s)"] = (np.abs(x - o) / s).to_numpy()
    med = oldin.rolling(24).median()
    mad = (oldin - med).abs().rolling(24).median()
    b["robust (median/MAD)"] = (np.abs(x - med) / (1.4826 * mad)).to_numpy()
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        q = STL(x, period=24, robust=True).fit().resid.to_numpy()
    b["STL qoldig'i"] = np.abs(q - np.median(q[:8 * HAFTA])) / (
        1.4826 * np.median(np.abs(q[:8 * HAFTA] - np.median(q[:8 * HAFTA]))))
    haftalar = np.stack([x.shift(HAFTA * k).to_numpy() for k in range(1, 5)])
    bashorat = np.median(haftalar, axis=0)             # o'tgan 4 haftaning shu soati
    qold = pd.Series(v - bashorat, index=x.index)
    qmad = (qold - qold.shift(1).rolling(HAFTA).median()).abs().shift(1).rolling(HAFTA).median()
    b["bashorat qoldig'i"] = (np.abs(qold - qold.shift(1).rolling(HAFTA).median())
                              / (1.4826 * qmad)).to_numpy()
    return b


def baholash(ball, hodisalar, bayram, n):
    """Chegara: toza kalibrlash haftalarida (5-8) haftasiga ~1 yolg'on signal."""
    kal = ball[4 * HAFTA:8 * HAFTA]
    chegara = np.nanquantile(kal, 1 - 1 / HAFTA)
    signal = np.nan_to_num(ball[8 * HAFTA:], nan=0.0) > chegara
    ind = np.flatnonzero(signal) + 8 * HAFTA
    oyna = np.zeros(n, dtype=bool)
    topildi, kechikish = {}, []
    for tur, s, e in hodisalar:
        oyna[s:e + 3] = True
        hit = ind[(ind >= s) & (ind < e + 3)]
        topildi.setdefault(tur, []).append(len(hit) > 0)
        if tur == "kollektiv" and len(hit):
            kechikish.append(hit[0] - s)
    yolgon = [i for i in ind if not oyna[i] and not bayram[i]]
    return ({k: np.mean(v) for k, v in topildi.items()},
            np.mean([t for v in topildi.values() for t in v]),
            len(yolgon) / 12, int(bayram[ind].sum()), kechikish)


def main() -> None:
    x, hodisalar, bayram = qator(0)
    print("=== 1. Qator va anomaliyalar (urug' 0) ===")
    print(f"  {len(x)} soat (20 hafta): 1-4 tarix, 5-8 kalibrlash (toza), 9-20 baholash")
    print(f"  o'rtacha {x.mean():.1f} MW, std {x.std():.1f}; kun ichida 3-soat "
          f"{x[x.index.hour == 3].mean():.1f}, 20-soat {x[x.index.hour == 20].mean():.1f}")
    for tur in ["nuqtali", "kontekstli", "kollektiv"]:
        s = [h for h in hodisalar if h[0] == tur]
        print(f"  {tur:<11} {len(s)} ta hodisa, masalan soat {s[0][1]}: "
              f"qiymat {x.iloc[s[0][1]]:.1f} (shu soatda odatda "
              f"{np.median([x.iloc[s[0][1] - HAFTA * k] for k in (1, 2, 3)]):.1f})")
    print(f"  bayram kunlari (anomaliya EMAS, 48 soat): o'rtacha "
          f"{x[bayram].mean():.1f}, oldingi haftaning shu soatlari "
          f"{x.shift(HAFTA)[bayram].mean():.1f}")

    print("\n=== 2. Besh detektor, 6 ta qator (urug'lar) - o'rtacha ===")
    jadval = {}
    for seed in range(6):
        x, hodisalar, bayram = qator(seed)
        for nom, b in ballar(x).items():
            jadval.setdefault(nom, []).append(baholash(b, hodisalar, bayram, len(x)))
    print(f"  {'detektor':<20} {'nuqtali':>8} {'kontekst':>9} {'kollektiv':>10} "
          f"{'hammasi':>8} {'yolg_on/hafta':>14} {'bayramda':>9}")
    for nom, r in jadval.items():
        tur = {k: np.mean([q[0][k] for q in r]) for k in ["nuqtali", "kontekstli", "kollektiv"]}
        print(f"  {nom:<20} {tur['nuqtali']:>8.2f} {tur['kontekstli']:>9.2f} "
              f"{tur['kollektiv']:>10.2f} {np.mean([q[1] for q in r]):>8.2f} "
              f"{np.mean([q[2] for q in r]):>14.2f} {np.mean([q[3] for q in r]):>9.1f}")

    print("\n=== 3. Kollektiv anomaliya: aniqlash kechikishi (soat) ===")
    for nom, r in jadval.items():
        k = [d for q in r for d in q[4]]
        print(f"  {nom:<20} topildi {len(k):>2}/24, kechikish mediana "
              f"{np.median(k) if k else float('nan'):>4.1f}")

    print("\n=== 4. Qaror: faqat REAL VAQTDA ishlay oladigan detektorlar ichida ===")
    rec = {nom: np.array([q[1] for q in r]) for nom, r in jadval.items()}
    tartib = ["global z", "rolling z (24s)", "robust (median/MAD)",
              "bashorat qoldig'i"]                      # soddadan murakkabga, sababiy
    eng = max(tartib, key=lambda n: rec[n].mean())
    tanlov = None
    for nom in tartib:
        d = rec[nom] - rec[eng]
        se = d.std(ddof=1) / np.sqrt(len(d))
        yomon = d.mean() < -2 * se
        if nom != eng:
            print(f"  {nom:<20} - {eng}: {d.mean():+.3f}, SE {se:.3f}, "
                  f"sezilarli yomon: {yomon}")
        if tanlov is None and not yomon:
            tanlov = nom
    print(f"  qaror (eng sodda munosib): {tanlov}")
    d = rec["STL qoldig'i"] - rec[eng]
    print(f"  STL (butun qatorni, ya'ni KELAJAKNI ham ko'radi) - {eng}: {d.mean():+.3f}, "
          f"SE {d.std(ddof=1) / np.sqrt(len(d)):.3f} - faqat oflayn tahlil uchun")
    print("  ⭐ Kontekst (soat, hafta kuni) bilan solishtirmaguncha kontekstli anomaliya ko'rinmaydi")


if __name__ == "__main__":
    main()

=== 1. Qator va anomaliyalar (urug' 0) ===
  3360 soat (20 hafta): 1-4 tarix, 5-8 kalibrlash (toza), 9-20 baholash
  o'rtacha 43.4 MW, std 9.1; kun ichida 3-soat 32.5, 20-soat 57.9
  nuqtali     8 ta hodisa, masalan soat 1485: qiymat 34.5 (shu soatda odatda 48.6)
  kontekstli  6 ta hodisa, masalan soat 2803: qiymat 39.1 (shu soatda odatda 57.2)
  kollektiv   4 ta hodisa, masalan soat 2471: qiymat 47.4 (shu soatda odatda 45.1)
  bayram kunlari (anomaliya EMAS, 48 soat): o'rtacha 34.9, oldingi haftaning shu soatlari 42.3

=== 2. Besh detektor, 6 ta qator (urug'lar) - o'rtacha ===
  detektor              nuqtali  kontekst  kollektiv  hammasi  yolg_on/hafta  bayramda
  global z                 0.33      0.00       0.08     0.17           2.06       0.5
  rolling z (24s)          0.33      0.06       0.04     0.18           2.36       0.2
  robust (median/MAD)      0.27      0.08       0.29     0.21           2.33       0.5
  STL qoldig'i             0.98      1.00       0.96     0.98           1.28       1.5
  bashorat qoldig'i        0.88      0.94       0.92     0.91           1.35       8.5

=== 3. Kollektiv anomaliya: aniqlash kechikishi (soat) ===
  global z             topildi  2/24, kechikish mediana  5.0
  rolling z (24s)      topildi  1/24, kechikish mediana 10.0
  robust (median/MAD)  topildi  7/24, kechikish mediana 10.0
  STL qoldig'i         topildi 23/24, kechikish mediana  4.0
  bashorat qoldig'i    topildi 22/24, kechikish mediana  4.0

=== 4. Qaror: faqat REAL VAQTDA ishlay oladigan detektorlar ichida ===
  global z             - bashorat qoldig'i: -0.741, SE 0.062, sezilarli yomon: True
  rolling z (24s)      - bashorat qoldig'i: -0.731, SE 0.056, sezilarli yomon: True
  robust (median/MAD)  - bashorat qoldig'i: -0.694, SE 0.059, sezilarli yomon: True
  qaror (eng sodda munosib): bashorat qoldig'i
  STL (butun qatorni, ya'ni KELAJAKNI ham ko'radi) - bashorat qoldig'i: +0.074, SE 0.031 - faqat oflayn tahlil uchun
  ⭐ Kontekst (soat, hafta kuni) bilan solishtirmaguncha kontekstli anomaliya ko'rinmaydi

"""Firibgarlik: shaxsiy bazaviy belgilar, vaqt bo'yicha bo'lish, nazoratsiz vs nazoratli."""

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier, IsolationForest
from sklearn.metrics import average_precision_score, roc_auc_score, roc_curve

KATEG = ["oziq", "transport", "restoran", "kiyim", "elektronika", "kommunal",
         "otkazma", "onlayn"]


def tranzaksiyalar(seed=0, n_mijoz=1200, kunlar=180):
    """Bank kartasi tranzaksiyalari; firibgarlik - akkaunt egallash epizodlari (~0.3%)."""
    rng = np.random.default_rng(seed)
    tezlik = rng.lognormal(np.log(0.45), 0.5, n_mijoz)          # kuniga tranzaksiya
    odat_summa = rng.lognormal(11.3, 0.6, n_mijoz)               # ~80 ming so'm
    odat_soat = rng.uniform(9, 20, n_mijoz)
    uy = rng.integers(0, 6, n_mijoz)
    kat_pref = rng.dirichlet(np.full(len(KATEG), 0.7), n_mijoz)
    kat_pref[:, 4] *= 0.3                                        # elektronika kam
    kat_pref /= kat_pref.sum(1, keepdims=True)
    soni = rng.poisson(tezlik * kunlar)
    m = np.repeat(np.arange(n_mijoz), soni)
    n = len(m)
    vaqt = rng.uniform(0, kunlar, n)
    kat = (rng.random(n)[:, None] > np.cumsum(kat_pref[m], 1)).sum(1)
    kat_koef = np.array([0.6, 0.3, 1.0, 2.0, 4.0, 1.2, 2.5, 1.5])
    summa = odat_summa[m] * kat_koef[kat] * rng.lognormal(0, 0.6, n)
    soat = np.mod(odat_soat[m] + rng.normal(0, 3, n), 24)
    safar = rng.random(n) < 0.03
    shahar = np.where(safar, (uy[m] + rng.integers(1, 6, n)) % 6, uy[m])
    qurilma = np.where(rng.random(n) < 0.015, rng.integers(1, 1000, n), 0)
    firib = np.zeros(n, dtype=bool)
    # firibgarlik epizodlari: bir necha soat ichida 1-5 ta tranzaksiya
    ep = int(n * 0.003 / 2.6)
    em = rng.integers(0, n_mijoz, ep)
    et = rng.uniform(0, kunlar - 1, ep)
    ek = rng.integers(1, 6, ep)
    fm = np.repeat(em, ek)
    fv = np.repeat(et, ek) + rng.uniform(0, 0.25, ek.sum())
    nf = len(fm)
    fk = rng.choice(len(KATEG), nf, p=[0.05, 0.02, 0.05, 0.1, 0.35, 0.03, 0.25, 0.15])
    fs = odat_summa[fm] * kat_koef[fk] * rng.lognormal(np.log(1.8), 0.8, nf)
    fsoat = np.where(rng.random(nf) < 0.45, rng.uniform(0, 6, nf),
                     np.mod(odat_soat[fm] + rng.normal(0, 5, nf), 24))
    fsh = np.where(rng.random(nf) < 0.5, (uy[fm] + rng.integers(1, 6, nf)) % 6, uy[fm])
    fq = np.where(rng.random(nf) < 0.7, rng.integers(1000, 2000, nf), 0)
    df = pd.DataFrame({
        "mijoz": np.r_[m, fm], "vaqt": np.r_[vaqt, fv], "summa": np.r_[summa, fs],
        "soat": np.r_[soat, fsoat], "kategoriya": np.r_[kat, fk],
        "shahar": np.r_[shahar, fsh], "qurilma": np.r_[qurilma, fq],
        "firib": np.r_[firib, np.ones(nf, dtype=bool)].astype(int)})
    kechikish = 3 + rng.exponential(20, len(df))                # chargeback kelishi (kun)
    df["belgi_kuni"] = np.where(df["firib"] == 1, df["vaqt"] + kechikish, df["vaqt"])
    return df.sort_values(["mijoz", "vaqt"]).reset_index(drop=True)


def belgilar(df, sizish=False):
    """Shaxsiy bazaviy: har tranzaksiya mijozning FAQAT O'TMISHI bilan solishtiriladi."""
    df = df.copy()
    g = df.groupby("mijoz")
    ls = np.log(df["summa"])
    df["log_summa"] = ls
    if sizish:                                  # XATO: butun davr (kelajak ham) ishlatiladi
        o = ls.groupby(df["mijoz"]).transform("mean")
        s = ls.groupby(df["mijoz"]).transform("std")
        os_ = g["soat"].transform("mean")
    else:
        n_old = g.cumcount()
        yig = ls.groupby(df["mijoz"]).cumsum() - ls
        yig2 = (ls ** 2).groupby(df["mijoz"]).cumsum() - ls ** 2
        o = yig / n_old.replace(0, np.nan)
        s = np.sqrt((yig2 / n_old.replace(0, np.nan) - o ** 2).clip(lower=0.01))
        os_ = (df["soat"].groupby(df["mijoz"]).cumsum() - df["soat"]) / n_old.replace(0, np.nan)
    df["summa_z"] = ((ls - o) / s).fillna(0)
    df["soat_farq"] = (df["soat"] - os_).abs().fillna(0)
    df["yangi_shahar"] = (df.groupby(["mijoz", "shahar"]).cumcount() == 0).astype(int)
    df["yangi_qurilma"] = (df.groupby(["mijoz", "qurilma"]).cumcount() == 0).astype(int)
    birinchi = g.cumcount() == 0
    df.loc[birinchi, ["yangi_shahar", "yangi_qurilma"]] = 0
    kalit = df["mijoz"].to_numpy() * 1000.0 + df["vaqt"].to_numpy()
    df["soni_24s"] = np.arange(len(df)) - np.searchsorted(kalit, kalit - 1.0)
    df["oraliq"] = np.log1p(g["vaqt"].diff().fillna(30).to_numpy() * 24)
    df["tarix_soni"] = g.cumcount()
    return df


XOM = ["log_summa", "soat", "kategoriya"]
SHAXSIY = ["log_summa", "soat", "kategoriya", "summa_z", "soat_farq", "yangi_shahar",
           "yangi_qurilma", "soni_24s", "oraliq", "tarix_soni"]


def ap_vazn(y, w, tartib):
    """Og'irlikli average precision: bootstrap da kun necha marta tanlangan bo'lsa."""
    y, w = y[tartib], w[tartib].astype(float)
    tp = np.cumsum(w * y)
    precision = tp / np.maximum(np.cumsum(w), 1e-12)
    return float(np.sum(w * y * precision) / max(tp[-1], 1e-12))


def main() -> None:
    xom = tranzaksiyalar()
    df = belgilar(xom)
    print("=== 1. Ma'lumot ===")
    print(f"  {len(df)} tranzaksiya, {df['mijoz'].nunique()} mijoz, 180 kun; "
          f"firibgarlik {int(df['firib'].sum())} ta ({df['firib'].mean():.2%})")
    f, n = df[df["firib"] == 1], df[df["firib"] == 0]
    print(f"  summa mediana: normal {n['summa'].median():,.0f}, firib {f['summa'].median():,.0f} so'm")
    print(f"  firib ichida: tungi (0-6) {np.mean(f['soat'] < 6):.0%}, yangi qurilma "
          f"{f['yangi_qurilma'].mean():.0%}, yangi shahar {f['yangi_shahar'].mean():.0%}")
    print(f"  normal ichida: tungi {np.mean(n['soat'] < 6):.0%}, yangi qurilma "
          f"{n['yangi_qurilma'].mean():.1%}, yangi shahar {n['yangi_shahar'].mean():.1%}")

    tr, te = df[df["vaqt"] < 120], df[df["vaqt"] >= 120]
    y_tr, y_te = tr["firib"].to_numpy(), te["firib"].to_numpy()
    print(f"  vaqt bo'yicha: train 0-119 kun ({len(tr)}, firib {y_tr.sum()}), "
          f"test 120-179 ({len(te)}, firib {y_te.sum()})")

    print("\n=== 2. Usullar (test, 60 kun) ===")
    ballar = {}
    for nom, cols in [("IF xom belgilar", XOM), ("IF shaxsiy belgilar", SHAXSIY)]:
        m = IsolationForest(n_estimators=100, random_state=0, n_jobs=1).fit(tr[cols])
        ballar[nom] = -m.score_samples(te[cols])
    m = IsolationForest(n_estimators=100, random_state=0, n_jobs=1).fit(
        tr.loc[tr["firib"] == 0, SHAXSIY])
    ballar["IF faqat normal (yarim)"] = -m.score_samples(te[SHAXSIY])
    for nom, cols, vazn in [("HistGB xom, balanced", XOM, "balanced"),
                            ("HistGB", SHAXSIY, None),
                            ("HistGB balanced", SHAXSIY, "balanced")]:
        m = HistGradientBoostingClassifier(max_iter=80, learning_rate=0.1, max_leaf_nodes=15,
                                           early_stopping=False, class_weight=vazn,
                                           random_state=0)
        ballar[nom] = m.fit(tr[cols], y_tr).predict_proba(te[cols])[:, 1]
    print(f"  {'usul':<24} {'ROC-AUC':>8} {'PR-AUC':>7} {'P@100':>6}")
    print(f"  {'tasodifiy (nazariy)':<24} {0.5:>8.4f} {y_te.mean():>7.4f} {y_te.mean():>6.3f}")
    for nom, b in ballar.items():
        top = np.argsort(-b, kind="stable")[:100]
        print(f"  {nom:<24} {roc_auc_score(y_te, b):>8.4f} "
              f"{average_precision_score(y_te, b):>7.4f} {y_te[top].mean():>6.3f}")
    print("\n=== 3. ROC-AUC nega aldaydi (IF shaxsiy belgilar) ===")
    b = ballar["IF shaxsiy belgilar"]
    fpr, tpr, chegara = roc_curve(y_te, b)
    for maqsad in [0.01, 0.05]:
        i = np.searchsorted(fpr, maqsad)
        tp, fp = tpr[i] * y_te.sum(), fpr[i] * (len(y_te) - y_te.sum())
        print(f"  FPR {maqsad:.0%}: firiblarning {tpr[i]:.0%} i topildi ({tp:.0f} ta), "
              f"yolg'on signal {fp:.0f} ta -> precision {tp / (tp + fp):.1%}")
    print(f"  ROC-AUC {roc_auc_score(y_te, b):.3f} 'yaxshi' ko'rinadi, lekin signallarning "
          f"aksariyati yolg'on - PR-AUC {average_precision_score(y_te, b):.3f}")

    print("\n=== 4. Juftlashgan bootstrap (kunlar bo'yicha, 200 marta), PR-AUC ===")
    kun = te["vaqt"].astype(int).to_numpy() - 120
    rng = np.random.default_rng(0)
    juftlar = [("HistGB balanced", "HistGB"), ("HistGB balanced", "IF shaxsiy belgilar"),
               ("IF shaxsiy belgilar", "IF faqat normal (yarim)")]
    tartib = {nom: np.argsort(-b, kind="stable") for nom, b in ballar.items()}
    for a, c in juftlar:
        d = []
        for _ in range(200):
            w = np.bincount(rng.integers(0, 60, 60), minlength=60)[kun]   # kun vazni
            d.append(ap_vazn(y_te, w, tartib[a]) - ap_vazn(y_te, w, tartib[c]))
        nuqta = (average_precision_score(y_te, ballar[a])
                 - average_precision_score(y_te, ballar[c]))
        se = np.std(d, ddof=1)
        print(f"  {a} - {c}: {nuqta:+.4f}, SE {se:.4f}, sezilarli: {abs(nuqta) > 2 * se}")
    print("  ⭐ Yorliq bo'lsa - nazoratli model; baholash - PR-AUC va top-k")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  107685 tranzaksiya, 1200 mijoz, 180 kun; firibgarlik 388 ta (0.36%)
  summa mediana: normal 93,548, firib 335,491 so'm
  firib ichida: tungi (0-6) 48%, yangi qurilma 68%, yangi shahar 36%
  normal ichida: tungi 3%, yangi qurilma 1.5%, yangi shahar 2.2%
  vaqt bo'yicha: train 0-119 kun (71885, firib 244), test 120-179 (35800, firib 144)

=== 2. Usullar (test, 60 kun) ===
  usul                      ROC-AUC  PR-AUC  P@100
  tasodifiy (nazariy)        0.5000  0.0040  0.004
  IF xom belgilar            0.7779  0.0707  0.150
  IF shaxsiy belgilar        0.9670  0.4908  0.620
  IF faqat normal (yarim)    0.9661  0.4877  0.590
  HistGB xom, balanced       0.8503  0.1761  0.300
  HistGB                     0.8597  0.2162  0.310
  HistGB balanced            0.9847  0.7152  0.830

=== 3. ROC-AUC nega aldaydi (IF shaxsiy belgilar) ===
  FPR 1%: firiblarning 68% i topildi (98 ta), yolg'on signal 367 ta -> precision 21.1%
  FPR 5%: firiblarning 85% i topildi (123 ta), yolg'on signal 1787 ta -> precision 6.4%
  ROC-AUC 0.967 'yaxshi' ko'rinadi, lekin signallarning aksariyati yolg'on - PR-AUC 0.491

=== 4. Juftlashgan bootstrap (kunlar bo'yicha, 200 marta), PR-AUC ===
  HistGB balanced - HistGB: +0.4990, SE 0.0375, sezilarli: True
  HistGB balanced - IF shaxsiy belgilar: +0.2245, SE 0.0347, sezilarli: True
  IF shaxsiy belgilar - IF faqat normal (yarim): +0.0031, SE 0.0138, sezilarli: False
  ⭐ Yorliq bo'lsa - nazoratli model; baholash - PR-AUC va top-k

"""Firibgarlik amaliyotda: belgi kechikishi, kunlik sig'im (top-k) va xarajatli chegara."""

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier, IsolationForest
from sklearn.metrics import average_precision_score

KATEG = ["oziq", "transport", "restoran", "kiyim", "elektronika", "kommunal",
         "otkazma", "onlayn"]


def tranzaksiyalar(seed=0, n_mijoz=1200, kunlar=180):
    """Bank kartasi tranzaksiyalari; firibgarlik - akkaunt egallash epizodlari (~0.3%)."""
    rng = np.random.default_rng(seed)
    tezlik = rng.lognormal(np.log(0.45), 0.5, n_mijoz)          # kuniga tranzaksiya
    odat_summa = rng.lognormal(11.3, 0.6, n_mijoz)               # ~80 ming so'm
    odat_soat = rng.uniform(9, 20, n_mijoz)
    uy = rng.integers(0, 6, n_mijoz)
    kat_pref = rng.dirichlet(np.full(len(KATEG), 0.7), n_mijoz)
    kat_pref[:, 4] *= 0.3                                        # elektronika kam
    kat_pref /= kat_pref.sum(1, keepdims=True)
    soni = rng.poisson(tezlik * kunlar)
    m = np.repeat(np.arange(n_mijoz), soni)
    n = len(m)
    vaqt = rng.uniform(0, kunlar, n)
    kat = (rng.random(n)[:, None] > np.cumsum(kat_pref[m], 1)).sum(1)
    kat_koef = np.array([0.6, 0.3, 1.0, 2.0, 4.0, 1.2, 2.5, 1.5])
    summa = odat_summa[m] * kat_koef[kat] * rng.lognormal(0, 0.6, n)
    soat = np.mod(odat_soat[m] + rng.normal(0, 3, n), 24)
    safar = rng.random(n) < 0.03
    shahar = np.where(safar, (uy[m] + rng.integers(1, 6, n)) % 6, uy[m])
    qurilma = np.where(rng.random(n) < 0.015, rng.integers(1, 1000, n), 0)
    firib = np.zeros(n, dtype=bool)
    # firibgarlik epizodlari: bir necha soat ichida 1-5 ta tranzaksiya
    ep = int(n * 0.003 / 2.6)
    em = rng.integers(0, n_mijoz, ep)
    et = rng.uniform(0, kunlar - 1, ep)
    ek = rng.integers(1, 6, ep)
    fm = np.repeat(em, ek)
    fv = np.repeat(et, ek) + rng.uniform(0, 0.25, ek.sum())
    nf = len(fm)
    fk = rng.choice(len(KATEG), nf, p=[0.05, 0.02, 0.05, 0.1, 0.35, 0.03, 0.25, 0.15])
    fs = odat_summa[fm] * kat_koef[fk] * rng.lognormal(np.log(1.8), 0.8, nf)
    fsoat = np.where(rng.random(nf) < 0.45, rng.uniform(0, 6, nf),
                     np.mod(odat_soat[fm] + rng.normal(0, 5, nf), 24))
    fsh = np.where(rng.random(nf) < 0.5, (uy[fm] + rng.integers(1, 6, nf)) % 6, uy[fm])
    fq = np.where(rng.random(nf) < 0.7, rng.integers(1000, 2000, nf), 0)
    df = pd.DataFrame({
        "mijoz": np.r_[m, fm], "vaqt": np.r_[vaqt, fv], "summa": np.r_[summa, fs],
        "soat": np.r_[soat, fsoat], "kategoriya": np.r_[kat, fk],
        "shahar": np.r_[shahar, fsh], "qurilma": np.r_[qurilma, fq],
        "firib": np.r_[firib, np.ones(nf, dtype=bool)].astype(int)})
    kechikish = 3 + rng.exponential(20, len(df))                # chargeback kelishi (kun)
    df["belgi_kuni"] = np.where(df["firib"] == 1, df["vaqt"] + kechikish, df["vaqt"])
    return df.sort_values(["mijoz", "vaqt"]).reset_index(drop=True)


def belgilar(df, sizish=False):
    """Shaxsiy bazaviy: har tranzaksiya mijozning FAQAT O'TMISHI bilan solishtiriladi."""
    df = df.copy()
    g = df.groupby("mijoz")
    ls = np.log(df["summa"])
    df["log_summa"] = ls
    if sizish:                                  # XATO: butun davr (kelajak ham) ishlatiladi
        o = ls.groupby(df["mijoz"]).transform("mean")
        s = ls.groupby(df["mijoz"]).transform("std")
        os_ = g["soat"].transform("mean")
    else:
        n_old = g.cumcount()
        yig = ls.groupby(df["mijoz"]).cumsum() - ls
        yig2 = (ls ** 2).groupby(df["mijoz"]).cumsum() - ls ** 2
        o = yig / n_old.replace(0, np.nan)
        s = np.sqrt((yig2 / n_old.replace(0, np.nan) - o ** 2).clip(lower=0.01))
        os_ = (df["soat"].groupby(df["mijoz"]).cumsum() - df["soat"]) / n_old.replace(0, np.nan)
    df["summa_z"] = ((ls - o) / s).fillna(0)
    df["soat_farq"] = (df["soat"] - os_).abs().fillna(0)
    df["yangi_shahar"] = (df.groupby(["mijoz", "shahar"]).cumcount() == 0).astype(int)
    df["yangi_qurilma"] = (df.groupby(["mijoz", "qurilma"]).cumcount() == 0).astype(int)
    birinchi = g.cumcount() == 0
    df.loc[birinchi, ["yangi_shahar", "yangi_qurilma"]] = 0
    kalit = df["mijoz"].to_numpy() * 1000.0 + df["vaqt"].to_numpy()
    df["soni_24s"] = np.arange(len(df)) - np.searchsorted(kalit, kalit - 1.0)
    df["oraliq"] = np.log1p(g["vaqt"].diff().fillna(30).to_numpy() * 24)
    df["tarix_soni"] = g.cumcount()
    return df


XOM = ["log_summa", "soat", "kategoriya"]
SHAXSIY = ["log_summa", "soat", "kategoriya", "summa_z", "soat_farq", "yangi_shahar",
           "yangi_qurilma", "soni_24s", "oraliq", "tarix_soni"]


def hgb(X, y):
    return HistGradientBoostingClassifier(max_iter=80, learning_rate=0.1, max_leaf_nodes=15,
                                          early_stopping=False, class_weight="balanced",
                                          random_state=0).fit(X, y)


def ap_vazn(y, w, tartib):
    """Og'irlikli average precision (kunlar bo'yicha bootstrap uchun)."""
    y, w = y[tartib], w[tartib].astype(float)
    tp = np.cumsum(w * y)
    return float(np.sum(w * y * tp / np.maximum(np.cumsum(w), 1e-12)) / max(tp[-1], 1e-12))


def kunlik_top(ball, kun, y, n):
    """Har kuni eng shubhali n ta tekshiriladi: kunlik precision va topilganlar."""
    prec, top_soni = [], []
    for d in np.unique(kun):
        i = np.flatnonzero(kun == d)
        tanlandi = i[np.argsort(-ball[i], kind="stable")[:n]]
        prec.append(y[tanlandi].mean())
        top_soni.append(y[tanlandi].sum())
    return np.array(prec), np.array(top_soni)


def xarajat(ball, chegara, y, summa, fp_narx=20_000):
    """FN - firib summasi yo'qotiladi; har signal (TP yoki FP) - tekshirish narxi."""
    signal = ball >= chegara
    return float(summa[(y == 1) & ~signal].sum() + signal.sum() * fp_narx)


def main() -> None:
    df = belgilar(tranzaksiyalar())
    T = 120                                          # o'qitish kuni
    tr, te = df[df["vaqt"] < T], df[df["vaqt"] >= T]
    y_te = te["firib"].to_numpy()
    kun = te["vaqt"].astype(int).to_numpy()

    print("=== 1. Belgi kechikishi: 120-kunda qaysi firiblar MA'LUM? ===")
    malum = tr["belgi_kuni"] < T
    f = tr["firib"] == 1
    print(f"  train dagi firib: {int(f.sum())}, 120-kungacha tasdiqlangani: "
          f"{int((f & malum).sum())}")
    for a, b in [(0, 60), (60, 90), (90, 105), (105, 120)]:
        m = f & (tr["vaqt"] >= a) & (tr["vaqt"] < b)
        print(f"   {a:>3}-{b:<3} kun: firib {int(m.sum()):>3}, ma'lum {(m & malum).sum() / m.sum():.0%}")
    strategiyalar = {
        "ideal (hamma yorliq)": (tr, tr["firib"]),
        "sodda (noma'lum = 0)": (tr, (tr["firib"] == 1) & malum),
        "faqat yetilgan (<75)": (tr[tr["vaqt"] < 75], tr.loc[tr["vaqt"] < 75, "firib"]),
    }
    ballar = {}
    for nom, (qism, y) in strategiyalar.items():
        ballar[nom] = hgb(qism[SHAXSIY], y.astype(int)).predict_proba(te[SHAXSIY])[:, 1]
        print(f"  {nom:<22} o'qitish firiblari {int(y.sum()):>3}, test PR-AUC "
              f"{average_precision_score(y_te, ballar[nom]):.4f}")
    rng = np.random.default_rng(0)
    tartib = {n: np.argsort(-b, kind="stable") for n, b in ballar.items()}
    nomlar = list(ballar)
    for a, c in [(nomlar[1], nomlar[0]), (nomlar[2], nomlar[1])]:
        d = []
        for _ in range(200):
            w = np.bincount(rng.integers(0, 60, 60), minlength=60)[kun - T]
            d.append(ap_vazn(y_te, w, tartib[a]) - ap_vazn(y_te, w, tartib[c]))
        nuqta = average_precision_score(y_te, ballar[a]) - average_precision_score(y_te, ballar[c])
        print(f"  {a} - {c}: {nuqta:+.4f}, SE {np.std(d, ddof=1):.4f}")

    print("\n=== 2. Kunlik sig'im: tekshiruvchilar kuniga n ta ko'radi ===")
    ifm = IsolationForest(n_estimators=100, random_state=0, n_jobs=1).fit(tr[SHAXSIY])
    usul = {"qoida (summa_z>2 va yangi qurilma)":
            (te["summa_z"] > 2).to_numpy() * 2.0 + te["yangi_qurilma"].to_numpy()
            + 0.01 * te["summa_z"].to_numpy(),
            "IF shaxsiy": -ifm.score_samples(te[SHAXSIY]),
            "HistGB (yetilgan)": ballar["faqat yetilgan (<75)"]}
    print(f"  test: 60 kun, kuniga o'rtacha {len(te) / 60:.0f} tranzaksiya, "
          f"{y_te.sum() / 60:.1f} firib")
    print(f"  {'usul':<36}" + "".join(f"{f'n={n}':>14}" for n in [5, 10, 20]))
    kunlik = {}
    for nom, b in usul.items():
        qator = []
        for n in [5, 10, 20]:
            p, s = kunlik_top(b, kun, y_te, n)
            kunlik[(nom, n)] = s
            qator.append(f"{p.mean():.2f}/{s.sum() / y_te.sum():.0%}")
        print(f"  {nom:<36}" + "".join(f"{q:>14}" for q in qator))
    print("  (kunlik precision / topilgan firiblar ulushi)")
    for a in ["qoida (summa_z>2 va yangi qurilma)", "IF shaxsiy"]:
        d = kunlik[("HistGB (yetilgan)", 10)] - kunlik[(a, 10)]
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  n=10: HistGB - {a}: kuniga {d.mean():+.2f} firib, SE {se:.2f}, "
              f"sezilarli: {abs(d.mean()) > 2 * se}")

    print("\n=== 3. Xarajatga asoslangan chegara (faraziy narxlar) ===")
    print("  FN: firib summasi yo'qoladi; har signal: 20 000 so'm (tekshirish, mijoz noqulayligi)")
    trv, va = df[df["vaqt"] < 90], df[(df["vaqt"] >= 90) & (df["vaqt"] < T)]
    b_va = hgb(trv[SHAXSIY], trv["firib"]).predict_proba(va[SHAXSIY])[:, 1]
    y_va, s_va = va["firib"].to_numpy(), va["summa"].to_numpy()
    nomzod = np.quantile(b_va, [0.9, 0.97, 0.99, 0.995, 0.998, 0.999])
    xv = [xarajat(b_va, c, y_va, s_va) for c in nomzod]
    eng = nomzod[int(np.argmin(xv))]
    print("  validatsiya (90-119 kun): " + ", ".join(
        f"{c:.3f}: {x / 1e6:.1f}" for c, x in zip(nomzod, xv)) + " (mln so'm)")
    print(f"  tanlangan chegara: {eng:.3f}")
    b = ballar["faqat yetilgan (<75)"]
    s_te = te["summa"].to_numpy()
    variantlar = {"hech narsa tekshirmaslik": np.inf, "chegara 0.5": 0.5,
                  "xarajat bo'yicha (val)": eng}
    for nom, c in variantlar.items():
        signal = b >= c
        print(f"  TEST {nom:<24} signal {int(signal.sum()):>5}, topildi "
              f"{int(y_te[signal].sum()):>3}/{int(y_te.sum())}, xarajat "
              f"{xarajat(b, c, y_te, s_te) / 1e6:>6.1f} mln")
    print("  ⭐ Chegara - biznes qarori: narxlar va sig'im bilan, validatsiyada")


if __name__ == "__main__":
    main()

=== 1. Belgi kechikishi: 120-kunda qaysi firiblar MA'LUM? ===
  train dagi firib: 244, 120-kungacha tasdiqlangani: 178
     0-60  kun: firib 116, ma'lum 99%
    60-90  kun: firib  62, ma'lum 79%
    90-105 kun: firib  13, ma'lum 46%
   105-120 kun: firib  53, ma'lum 15%
  ideal (hamma yorliq)   o'qitish firiblari 244, test PR-AUC 0.7152
  sodda (noma'lum = 0)   o'qitish firiblari 178, test PR-AUC 0.7002
  faqat yetilgan (<75)   o'qitish firiblari 148, test PR-AUC 0.6931
  sodda (noma'lum = 0) - ideal (hamma yorliq): -0.0150, SE 0.0108
  faqat yetilgan (<75) - sodda (noma'lum = 0): -0.0071, SE 0.0156

=== 2. Kunlik sig'im: tekshiruvchilar kuniga n ta ko'radi ===
  test: 60 kun, kuniga o'rtacha 597 tranzaksiya, 2.4 firib
  usul                                           n=5          n=10          n=20
  qoida (summa_z>2 va yangi qurilma)        0.13/27%      0.08/34%      0.07/62%
  IF shaxsiy                                0.25/52%      0.17/69%      0.09/78%
  HistGB (yetilgan)                         0.30/62%      0.20/83%      0.10/87%
  (kunlik precision / topilgan firiblar ulushi)
  n=10: HistGB - qoida (summa_z>2 va yangi qurilma): kuniga +1.18 firib, SE 0.21, sezilarli: True
  n=10: HistGB - IF shaxsiy: kuniga +0.33 firib, SE 0.10, sezilarli: True

=== 3. Xarajatga asoslangan chegara (faraziy narxlar) ===
  FN: firib summasi yo'qoladi; har signal: 20 000 so'm (tekshirish, mijoz noqulayligi)
  validatsiya (90-119 kun): 0.011: 38.0, 0.081: 15.8, 0.334: 9.6, 0.697: 8.7, 0.983: 15.6, 0.994: 24.1 (mln so'm)
  tanlangan chegara: 0.697
  TEST hech narsa tekshirmaslik signal     0, topildi   0/144, xarajat   81.2 mln
  TEST chegara 0.5              signal   206, topildi 103/144, xarajat   25.9 mln
  TEST xarajat bo'yicha (val)   signal   169, topildi 100/144, xarajat   26.6 mln
  ⭐ Chegara - biznes qarori: narxlar va sig'im bilan, validatsiyada

"""Avtoenkoder bilan anomaliya: IsolationForest va PCA bilan halol taqqoslash."""

import numpy as np
import pandas as pd
import torch
from sklearn.decomposition import PCA
from sklearn.ensemble import IsolationForest
from sklearn.metrics import average_precision_score
from sklearn.preprocessing import StandardScaler

KATEG = ["oziq", "transport", "restoran", "kiyim", "elektronika", "kommunal",
         "otkazma", "onlayn"]


def tranzaksiyalar(seed=0, n_mijoz=1200, kunlar=180):
    """Bank kartasi tranzaksiyalari; firibgarlik - akkaunt egallash epizodlari (~0.3%)."""
    rng = np.random.default_rng(seed)
    tezlik = rng.lognormal(np.log(0.45), 0.5, n_mijoz)          # kuniga tranzaksiya
    odat_summa = rng.lognormal(11.3, 0.6, n_mijoz)               # ~80 ming so'm
    odat_soat = rng.uniform(9, 20, n_mijoz)
    uy = rng.integers(0, 6, n_mijoz)
    kat_pref = rng.dirichlet(np.full(len(KATEG), 0.7), n_mijoz)
    kat_pref[:, 4] *= 0.3                                        # elektronika kam
    kat_pref /= kat_pref.sum(1, keepdims=True)
    soni = rng.poisson(tezlik * kunlar)
    m = np.repeat(np.arange(n_mijoz), soni)
    n = len(m)
    vaqt = rng.uniform(0, kunlar, n)
    kat = (rng.random(n)[:, None] > np.cumsum(kat_pref[m], 1)).sum(1)
    kat_koef = np.array([0.6, 0.3, 1.0, 2.0, 4.0, 1.2, 2.5, 1.5])
    summa = odat_summa[m] * kat_koef[kat] * rng.lognormal(0, 0.6, n)
    soat = np.mod(odat_soat[m] + rng.normal(0, 3, n), 24)
    safar = rng.random(n) < 0.03
    shahar = np.where(safar, (uy[m] + rng.integers(1, 6, n)) % 6, uy[m])
    qurilma = np.where(rng.random(n) < 0.015, rng.integers(1, 1000, n), 0)
    firib = np.zeros(n, dtype=bool)
    # firibgarlik epizodlari: bir necha soat ichida 1-5 ta tranzaksiya
    ep = int(n * 0.003 / 2.6)
    em = rng.integers(0, n_mijoz, ep)
    et = rng.uniform(0, kunlar - 1, ep)
    ek = rng.integers(1, 6, ep)
    fm = np.repeat(em, ek)
    fv = np.repeat(et, ek) + rng.uniform(0, 0.25, ek.sum())
    nf = len(fm)
    fk = rng.choice(len(KATEG), nf, p=[0.05, 0.02, 0.05, 0.1, 0.35, 0.03, 0.25, 0.15])
    fs = odat_summa[fm] * kat_koef[fk] * rng.lognormal(np.log(1.8), 0.8, nf)
    fsoat = np.where(rng.random(nf) < 0.45, rng.uniform(0, 6, nf),
                     np.mod(odat_soat[fm] + rng.normal(0, 5, nf), 24))
    fsh = np.where(rng.random(nf) < 0.5, (uy[fm] + rng.integers(1, 6, nf)) % 6, uy[fm])
    fq = np.where(rng.random(nf) < 0.7, rng.integers(1000, 2000, nf), 0)
    df = pd.DataFrame({
        "mijoz": np.r_[m, fm], "vaqt": np.r_[vaqt, fv], "summa": np.r_[summa, fs],
        "soat": np.r_[soat, fsoat], "kategoriya": np.r_[kat, fk],
        "shahar": np.r_[shahar, fsh], "qurilma": np.r_[qurilma, fq],
        "firib": np.r_[firib, np.ones(nf, dtype=bool)].astype(int)})
    kechikish = 3 + rng.exponential(20, len(df))                # chargeback kelishi (kun)
    df["belgi_kuni"] = np.where(df["firib"] == 1, df["vaqt"] + kechikish, df["vaqt"])
    return df.sort_values(["mijoz", "vaqt"]).reset_index(drop=True)


def belgilar(df, sizish=False):
    """Shaxsiy bazaviy: har tranzaksiya mijozning FAQAT O'TMISHI bilan solishtiriladi."""
    df = df.copy()
    g = df.groupby("mijoz")
    ls = np.log(df["summa"])
    df["log_summa"] = ls
    if sizish:                                  # XATO: butun davr (kelajak ham) ishlatiladi
        o = ls.groupby(df["mijoz"]).transform("mean")
        s = ls.groupby(df["mijoz"]).transform("std")
        os_ = g["soat"].transform("mean")
    else:
        n_old = g.cumcount()
        yig = ls.groupby(df["mijoz"]).cumsum() - ls
        yig2 = (ls ** 2).groupby(df["mijoz"]).cumsum() - ls ** 2
        o = yig / n_old.replace(0, np.nan)
        s = np.sqrt((yig2 / n_old.replace(0, np.nan) - o ** 2).clip(lower=0.01))
        os_ = (df["soat"].groupby(df["mijoz"]).cumsum() - df["soat"]) / n_old.replace(0, np.nan)
    df["summa_z"] = ((ls - o) / s).fillna(0)
    df["soat_farq"] = (df["soat"] - os_).abs().fillna(0)
    df["yangi_shahar"] = (df.groupby(["mijoz", "shahar"]).cumcount() == 0).astype(int)
    df["yangi_qurilma"] = (df.groupby(["mijoz", "qurilma"]).cumcount() == 0).astype(int)
    birinchi = g.cumcount() == 0
    df.loc[birinchi, ["yangi_shahar", "yangi_qurilma"]] = 0
    kalit = df["mijoz"].to_numpy() * 1000.0 + df["vaqt"].to_numpy()
    df["soni_24s"] = np.arange(len(df)) - np.searchsorted(kalit, kalit - 1.0)
    df["oraliq"] = np.log1p(g["vaqt"].diff().fillna(30).to_numpy() * 24)
    df["tarix_soni"] = g.cumcount()
    return df


XOM = ["log_summa", "soat", "kategoriya"]
SHAXSIY = ["log_summa", "soat", "kategoriya", "summa_z", "soat_farq", "yangi_shahar",
           "yangi_qurilma", "soni_24s", "oraliq", "tarix_soni"]


def sensor(seed, n=5000, ulush=0.02):
    """Uskuna sensorlari: 8 ustun 2 ta yashirin holatdan (burchak t, yuklama s) NOCHIZIQLI."""
    rng = np.random.default_rng(seed)
    t, s = rng.uniform(0, 2 * np.pi, n), rng.normal(0, 1, n)
    X = np.column_stack([np.cos(t), np.sin(t), np.cos(2 * t), np.sin(t) * s,
                         s, np.tanh(2 * s), 0.5 * np.cos(t) + 0.5 * s, np.exp(0.5 * s)])
    X += rng.normal(0, 0.05, X.shape)
    y = np.zeros(n, dtype=int)
    a = rng.choice(n, int(n * ulush), replace=False)
    for i in a:                                  # 2 ta ustun boshqa qatordan: har biri
        j = rng.choice(8, 2, replace=False)      # alohida normal, birga - g'ayrioddiy
        X[i, j] = X[rng.integers(0, n), j]
    y[a] = 1
    return X, y


def avtoenkoder(Xtr, Xte, seed, yashirin=3, qadam=600):
    torch.manual_seed(seed)
    torch.set_flush_denormal(True)
    d = Xtr.shape[1]
    model = torch.nn.Sequential(torch.nn.Linear(d, 16), torch.nn.Tanh(),
                                torch.nn.Linear(16, yashirin), torch.nn.Tanh(),
                                torch.nn.Linear(yashirin, 16), torch.nn.Tanh(),
                                torch.nn.Linear(16, d))
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    A = torch.tensor(Xtr, dtype=torch.float32)
    for _ in range(qadam):                       # to'liq partiya, MSE
        loss = ((model(A) - A) ** 2).mean()
        opt.zero_grad()
        loss.backward()
        opt.step()
    with torch.no_grad():
        B = torch.tensor(Xte, dtype=torch.float32)
        return ((model(B) - B) ** 2).mean(1).numpy(), float(loss)


def uch_usul(Xtr, Xte, seed):
    sk = StandardScaler().fit(Xtr)
    A, B = sk.transform(Xtr), sk.transform(Xte)
    b = {"IsolationForest": -IsolationForest(n_estimators=100, random_state=seed, n_jobs=1)
         .fit(Xtr).score_samples(Xte)}
    p = PCA(3, random_state=seed).fit(A)
    b["PCA (3) qayta tiklash"] = ((B - p.inverse_transform(p.transform(B))) ** 2).mean(1)
    b["avtoenkoder (3)"], _ = avtoenkoder(A, B, seed)
    return b


def main() -> None:
    print("=== 1. Avtoenkoder ichkarisi: qayta tiklash xatosi (sensor, urug' 0) ===")
    X, y = sensor(0)
    Xtr, Xte, yte = X[:3000], X[3000:], y[3000:]
    sk = StandardScaler().fit(Xtr)
    xato, loss = avtoenkoder(sk.transform(Xtr), sk.transform(Xte), 0)
    print(f"  8 ustun -> 3 yashirin -> 8 ustun; oxirgi train MSE {loss:.4f}")
    print(f"  test qayta tiklash xatosi: normal mediana {np.median(xato[yte == 0]):.4f}, "
          f"anomaliya mediana {np.median(xato[yte == 1]):.4f}")
    z = np.abs(StandardScaler().fit(Xtr).transform(Xte))
    print(f"  alohida ustunlarda |z| > 3 bo'lgan anomaliyalar: "
          f"{np.mean(z[yte == 1].max(1) > 3):.0%} (normal: {np.mean(z[yte == 0].max(1) > 3):.0%})")

    print("\n=== 2. Ikki ma'lumot, 4 urug', PR-AUC ===")
    natija = {}
    for seed in range(4):
        X, y = sensor(seed)
        natija.setdefault("sensor (nochiziqli bog'liqlik)", []).append(
            (uch_usul(X[:3000], X[3000:], seed), y[3000:]))
        df = belgilar(tranzaksiyalar(seed=seed, n_mijoz=400))
        tr, te = df[df["vaqt"] < 120], df[df["vaqt"] >= 120]
        tr = tr.sample(4000, random_state=seed)          # kichik jadval: 4000 qator
        natija.setdefault("tranzaksiyalar (shaxsiy belgilar)", []).append(
            (uch_usul(tr[SHAXSIY].to_numpy(float), te[SHAXSIY].to_numpy(float), seed),
             te["firib"].to_numpy()))
    tartib = ["IsolationForest", "PCA (3) qayta tiklash", "avtoenkoder (3)"]
    for nom, qatorlar in natija.items():
        pr = {u: np.array([average_precision_score(yy, b[u]) for b, yy in qatorlar])
              for u in tartib}
        print(f"  {nom}: anomaliya ulushi {np.mean([yy.mean() for _, yy in qatorlar]):.2%}")
        for u in tartib:
            print(f"    {u:<24} PR-AUC {pr[u].mean():.4f}  (urug'lar: "
                  f"{', '.join(f'{v:.3f}' for v in pr[u])})")
        eng = max(tartib, key=lambda u: pr[u].mean())
        tanlov = None
        for u in tartib:
            d = pr[u] - pr[eng]
            se = d.std(ddof=1) / np.sqrt(len(d))
            if u != eng:
                print(f"    {u} - {eng}: {d.mean():+.4f}, SE {se:.4f}")
            if tanlov is None and d.mean() >= -2 * se:
                tanlov = u
        print(f"    qaror (eng sodda munosib): {tanlov}")
    print("  ⭐ Avtoenkoder - murakkab bog'liqlik bo'lsa; oddiy jadvalda IF ko'pincha yetarli")


if __name__ == "__main__":
    main()

=== 1. Avtoenkoder ichkarisi: qayta tiklash xatosi (sensor, urug' 0) ===
  8 ustun -> 3 yashirin -> 8 ustun; oxirgi train MSE 0.0226
  test qayta tiklash xatosi: normal mediana 0.0123, anomaliya mediana 0.2606
  alohida ustunlarda |z| > 3 bo'lgan anomaliyalar: 3% (normal: 2%)

=== 2. Ikki ma'lumot, 4 urug', PR-AUC ===
  sensor (nochiziqli bog'liqlik): anomaliya ulushi 1.98%
    IsolationForest          PR-AUC 0.0365  (urug'lar: 0.031, 0.034, 0.047, 0.035)
    PCA (3) qayta tiklash    PR-AUC 0.0487  (urug'lar: 0.028, 0.047, 0.083, 0.035)
    avtoenkoder (3)          PR-AUC 0.6414  (urug'lar: 0.647, 0.699, 0.488, 0.732)
    IsolationForest - avtoenkoder (3): -0.6049, SE 0.0570
    PCA (3) qayta tiklash - avtoenkoder (3): -0.5928, SE 0.0647
    qaror (eng sodda munosib): avtoenkoder (3)
  tranzaksiyalar (shaxsiy belgilar): anomaliya ulushi 0.34%
    IsolationForest          PR-AUC 0.5288  (urug'lar: 0.620, 0.641, 0.355, 0.499)
    PCA (3) qayta tiklash    PR-AUC 0.1339  (urug'lar: 0.040, 0.060, 0.237, 0.199)
    avtoenkoder (3)          PR-AUC 0.1514  (urug'lar: 0.175, 0.021, 0.094, 0.315)
    PCA (3) qayta tiklash - IsolationForest: -0.3949, SE 0.1135
    avtoenkoder (3) - IsolationForest: -0.3774, SE 0.0978
    qaror (eng sodda munosib): IsolationForest
  ⭐ Avtoenkoder - murakkab bog'liqlik bo'lsa; oddiy jadvalda IF ko'pincha yetarli

z = (x - x.mean()) / x.std()                                          # ⚠️
qoldiq = x - np.median([x.shift(168 * k) for k in range(1, 5)], axis=0)   # ✅ shu soat

q = STL(x, period=24).fit().resid           # ⚠️ real vaqt detektori sifatida
# real vaqtda: faqat o'tmishdan bashorat qoldig'i                   # ✅

signal = z > 3                                                        # ⚠️
chegara = np.nanquantile(ball[kalibrlash], 1 - 1 / 168)              # ✅ byudjet

df["nisbat"] = df["summa"] / df.groupby("mijoz")["summa"].transform("median")  # ⚠️
yig = ls.groupby(df["mijoz"]).cumsum() - ls                          # ✅ faqat o'tmish

eng = max(modellar, key=lambda m: roc_auc_score(y, m))                # ⚠️
eng = max(modellar, key=lambda m: average_precision_score(y, m))      # ✅ + top-k

y = df["firib_malum"].fillna(0)                                       # ⚠️ firib = 0
tr = df[df["vaqt"] < T - L]                                           # ✅ yetilgan davr

HistGradientBoostingClassifier()                                      # ⚠️
HistGradientBoostingClassifier(class_weight="balanced")               # ✅ va PR-AUC da tekshiring

1.  # 20:00 da 39 MW (odatda 57), umumiy o'rtacha 43 - global z qanday?
2.  # rolling 24 soatlik oyna kontekstli anomaliyani nega ko'rmaydi?
3.  # MAD ni std ga aylantirish koeffitsienti?
4.  # STL qoldig'i real vaqt detektori bo'la oladimi?
5.  # sensor qotib qolsa, detektor nega kechikadi?
6.  # firib ulushi 0.4% - tasodifiy modelning PR-AUC i?
7.  # 36000 normal, FPR 1% - nechta yolg'on signal?
8.  # 16.10 dagi groupby().transform("median") nima uchun xavfli?
9.  # nomutanosib boosting ga nima qo'shish kerak?
10. # o'qitish kunida oxirgi 2 haftadagi firiblar qanday belgilangan bo'ladi?
11. # FN narxi katta, FP narxi kichik - chegara qaysi tomonga siljiydi?
12. # anomaliya bitta ustunda katta chetlanish - IF yoki avtoenkoder?

1.  signal = np.abs(x - x.mean()) / x.std() > 3            # soatlik elektr

2.  df["nisbat"] = df["summa"] / df.groupby("mijoz")["summa"].transform("mean")

3.  eng = max(modellar, key=lambda m: roc_auc_score(y_te, ball[m]))

4.  y_train = df_train["firib_tasdiqlangan"].fillna(0)      # oxirgi haftalar ham

5.  chegara = 0.5                                          # FN = summa, FP = 20 000