28.12-dars: Amaliyot — do'konlar tarmog'i uchun talab bashorati va anomaliya monitoringi

MANBA: kassa tizimlaridan kunlik savdo (do'kon x guruh x kun) ---- (1-misol)
  |      + aksiya taqvimi (oldindan rejalashtiriladi)
  |      + bayram taqvimi, hujjatlashtirilgan yopilishlar (ta'mir)
  v
SIFAT: takror -> olib tashlash; manfiy -> NaN; to'liq panjara (yo'q kunlar)
  anomaliya: robust qoldiq (o'tgan 4 haftaning shu kuni medianasi)
  kutilgan hodisa (bayram, arafa, aksiya, yopilish) -> anomaliya EMAS
  anomaliya va yo'q kun -> to'ldirish (hafta kuni medianasi) + belgi
  |
  v
BACKTEST: 4 ta 28 kunlik oyna, oxirgi 28 kun - TEST (tegilmaydi) -- (2-misol)
  seasonal naive  <  ETS (har qator)  <  global HistGB   (soddalik tartibi)
  metrika: WAPE (do'kon x oyna birligida), juftlashgan farq + SE
  QOIDA: eng yaxshisidan sezilarli yomon bo'lmagan eng sodda
  |
  v
NOANIQLIK: konformal (backtest qoldiqlari) va kvantil HistGB -------- (3-misol)
  qamrov: nominal 80% - haqiqatda nechchi?
  ZAXIRA: kritik nisbat = c_yetishmaslik / (c_yetishmaslik + c_ortiqcha)
  |
  v
YAKUN: test bir marta -> hisobot                                   (4-misol)
  aksiya ta'siri: naiv / regressiya / kontrfaktual model / moslashtirish
  paket (joblib): model + belgilar + metadata + nazorat namunalari
  monitoring rejasi: ma'lumot, sifat, qamrov, trigger (27.11-27.13)

VALIDATSIYA 27.3-bob:
  kalit (do'kon, guruh, sana) noyob     -> takrorlar olib tashlanadi
  savdo >= 0                           -> manfiy (qaytarish yozuvi) -> NaN
  to'liq panjara: har kun uchun qator   -> yo'q qator = NaN (0 EMAS!)

ANOMALIYA (28.6 g'oyasi - kutilgan qiymat + robust og'ish):
  kutilgan b_t = median( y_{t-7}, y_{t-14}, y_{t-21}, y_{t-28} )  (log1p da)
  qoldiq  r_t = log1p(y_t) - b_t
  robust z = (r - median(r)) / (1.4826 * MAD(r))     (qator bo'yicha)
  |z| > chegara -> shubhali
  KUTILGAN HODISALAR (bayram, arafa, aksiya, hujjatlashtirilgan yopilish):
     anomaliya ro'yxatidan chiqariladi - ular modelga BELGI sifatida beriladi

TO'LDIRISH SIYOSATI:
  yo'q kun, kassa noli, birlik xatosi  -> NaN -> hafta kuni medianasi (+ belgi)
  hujjatlashtirilgan yopilish          -> 0 qoladi, lekin o'quvdan chiqariladi
  hamma tuzatish jurnalga yoziladi (nima, qayerda, qanday)

UFQ: h = 1..28 kun (ombor 4 haftalik reja bo'yicha buyurtma beradi)
ORIGIN: har 28 kunda, bir xil hafta kuni (masalan yakshanba kechqurun)
OYNALAR: oxirgi 28 kun - TEST; undan oldingi 4 ta 28 kunlik oyna - backtest
  oynalar qoplanmaydi -> xatolar mustaqilroq, SE halolroq 28.1-bob

NOMZODLAR (soddalik tartibida):
  1. seasonal naive:  f(o+h) = y(o + h - 7*ceil(h/7))   "o'tgan haftaning shu kuni"
  2. ETS (A,N,A), log1p shkalada, har qator alohida, oxirgi 182 kun   28.2-bob
  3. global HistGB: BITTA model hamma 24 qator uchun                  28.3-bob

METRIKA: WAPE = sum|y - f| / sum y   (nollarga chidamli, birliksiz)
BIRLIK: do'kon x oyna (8 x 4 = 32) - har do'konning 3 guruhi x 28 kuni
TAQQOSLASH: d = WAPE_model - WAPE_eng_yaxshi, SE = std(d)/sqrt(32)
QOIDA: |d| <= 2*SE bo'lgan (sezilarli yomon bo'lmagan) eng sodda model

ORIGIN DA MA'LUM (o gacha):
  daraja28 = log1p(o'rtacha y[o-27 .. o])             - qator darajasi
  daraja7 - daraja28                                  - oxirgi haftaning siljishi
  sn = log1p(y[o + h - 7*ceil(h/7)]) - daraja28       - seasonal naive nisbiy
  otgan_yil = log1p(y[t-367 .. t-361]) - log1p(y[o-391 .. o-364])
                                                      - o'tgan yilning shakli
OLDINDAN MA'LUM (kelajak kuni t = o + h uchun):
  h, hafta kuni, yil kuni (sin, cos), bayram, arafa, AKSIYA (reja!)
IDENTIFIKATOR: do'kon, guruh (kategoriya)
MAQSAD: log1p(y_t) - daraja28  (nisbiy - do'konlar hajmi turlicha)

SIZISH TUZOQLARI 17.8-bob:
  y[o+1 ..] dan har qanday statistika (sirpanuvchi o'rtacha "markazlashgan")
  o'quv misollarining maqsad kuni t > backtest origin O  ->  o'quv faqat
    o <= O - 28 originlardan (maqsad kunlari O dan oshmaydi)
  aksiya belgisi - faqat REJADA bo'lsa (kechikib kiritiladigan aksiya - sizish)

KONFORMAL (split, qoldiq kvantillari):
  kalibrovka oynalarida qoldiq r = log1p(y) - log1p(f)  (ufq haftasi bo'yicha)
  interval: expm1( log1p(f) + [q_0.1(r), q_0.9(r)] )    -> nominal 80%
  kafolat: yangi ma'lumot kalibrovkaga "o'xshash" bo'lsa (almashinuvchanlik)
  bayram/aksiya kunlari kam -> ular uchun qamrov alohida tekshiriladi

KVANTIL HistGB: loss="quantile", quantile=0.1 va 0.9 - alohida modellar
  kafolat yo'q, lekin sharoitga (belgilarga) moslashadi

ZAXIRA (newsvendor, 27-qism oldidagi biznes misollar):
  c_u - 1 dona yetishmaslik narxi (yo'qotilgan marja, xaridor noroziligi)
  c_o - 1 dona ortiqcha narxi (buzilish, chegirma bilan sotish)
  optimal buyurtma = talabning  c_u / (c_u + c_o)  kvantili
  c_u = 3, c_o = 1  ->  0.75 kvantil;   c_u = c_o  ->  mediana
  NUQTALI bashorat (o'rtacha/mediana) buyurtma sifatida - odatda NOTO'G'RI kvantil

NAIV:  mean(y | aksiya) / mean(y | aksiyasiz) - 1
       aksiya kunlari o'zi sust -> ta'sir KAM ko'rinadi (hatto manfiy)

TUZATILGAN 28.11-bob:
  regressiya: log1p(y) ~ aksiya + qator + hafta_kuni x guruh + oy x guruh
              + bayram + arafa        (SE - qator bo'yicha klaster)
  kontrfaktual model (g-hisoblash): aksiyasiz kunlarda o'qitilgan model
              aksiya kunlari uchun "aksiya bo'lmaganda" savdoni bashorat qiladi
              ta'sir = mean( log1p(y) - log1p(f_aksiyasiz) )  -> ATT
  moslashtirish: aksiya kuni <-> shu qator, shu hafta kuni, +-1-2 hafta,
              aksiyasiz va bayramsiz kunlar
FARAZ: aksiya tanlovi taqvim va mavsum orqali tushuntiriladi
       (menejer "bugun sust bo'ladi" degan boshqa ma'lumotdan foydalansa -
        o'lchanmagan chalkashtiruvchi, 28.11 3-misol)

PAKET 27.6-bob:
  model, belgilar ro'yxati va tartibi, kategoriya ustunlari
  metadata: ma'lumot davri, oynalar, backtest WAPE, tanlov qoidasi natijasi,
            interval kalibrovkasi (q_0.1, q_0.9 ufq haftasi bo'yicha), versiya
  nazorat namunalari: 50 ta kirish + kutilgan chiqish -> yuklashda tekshiriladi

MONITORING (27.11, 27.12):
  ma'lumot:  yo'q qatorlar, anomaliya ulushi, kassa nollari (har kuni)
  sifat:     haftalik do'kon WAPE > chegara (backtest dan) 2 hafta ketma-ket
  intervallar: 4 haftalik 80% interval qamrovi < 70%
  drift:     aksiya ulushi, yangi do'kon, assortiment o'zgarishi
  TRIGGER -> qayta o'qitish 27.13-bob: jadval (har 4 hafta) + sifat triggeri

# 1) sifat
xom = xom.drop_duplicates(["qator", "kun"])
W = xom.pivot(index="qator", columns="kun", values="savdo").reindex(
    index=range(S), columns=range(T)).to_numpy()          # yo'q kun -> NaN
b = np.nanmedian([siljit(np.log1p(W), 7 * k) for k in range(1, 5)], axis=0)
r = np.log1p(W) - b
z = (r - med) / (1.4826 * mad)
anomaliya = (np.abs(z) > 4) & ~kutilgan                   # bayram/arafa/aksiya/yopilish

# 2) backtest
sn = Y[s, O + h - 7 * np.ceil(h / 7).astype(int)]         # seasonal naive
ets = ExponentialSmoothing(qator, trend=None, seasonal="add",
                           seasonal_periods=7).fit().forecast(28)
model = HistGradientBoostingRegressor(categorical_features=[0, 1]).fit(X, y)
wape = np.abs(y - f).sum() / y.sum()
d = wape_a - wape_b; se = d.std(ddof=1) / np.sqrt(len(d))  # do'kon x oyna

# 3) noaniqlik va zaxira
q_lo, q_hi = np.quantile(qoldiq_kal, [0.1, 0.9])
past, yuqori = np.expm1(np.log1p(f) + q_lo), np.expm1(np.log1p(f) + q_hi)
kvantil = c_u / (c_u + c_o)                                 # 3 / (3 + 1) = 0.75
buyurtma = np.expm1(np.log1p(f) + np.quantile(qoldiq_kal, kvantil))

# 4) aksiya ta'siri va paket
tasir = np.mean(np.log1p(y_aksiya) - np.log1p(f_aksiyasiz))  # kontrfaktual
joblib.dump({"model": model, "belgilar": BELGILAR, "metadata": meta,
             "nazorat": {"X": X[:50], "bashorat": model.predict(X[:50])}}, yol)

talab_bashorati/
  konfig/qoidalar.yaml    ufq, oynalar, tanlov qoidasi, c_u/c_o, monitoring chegaralari
  src/talab/
    manba.py        kassa, aksiya va bayram taqvimlari
    sifat.py        validatsiya, anomaliya, to'ldirish, jurnal
    belgilar.py     origin ga asoslangan belgilar (sizish testlari bilan)
    modellar.py     seasonal naive, ETS, global HistGB
    backtest.py     oynalar, metrikalar, juftlashgan taqqoslash
    noaniqlik.py    konformal kalibrovka, kvantillar, buyurtma
    sababiy.py      aksiya ta'siri
    paket.py        yasash, yuklash, nazorat
    monitoring.py   kunlik va haftalik tekshiruvlar
  tests/            har modul uchun; belgilar uchun "kelajakdan sizish" testi

xom -> takror/manfiy/yo'q -> anomaliya (kutilgan hodisalar - anomaliya emas) -> to'ldirish + belgi
backtest: 4 oyna, SN < ETS < HistGB, WAPE do'kon x oyna, juftlashgan qaror
interval: konformal (qoldiq kvantillari) / kvantil model -> qamrovni tekshir
buyurtma = c_u/(c_u+c_o) kvantili; aksiya ta'siri - tuzatilgan
test bir marta; paket + monitoring rejasi

"""1-qadam: xom savdo ma'lumoti - validatsiya, anomaliyalar, to'ldirish siyosati."""

import warnings

import numpy as np
import pandas as pd

DOKONLAR = ["Toshkent-Chilonzor", "Toshkent-Yunusobod", "Toshkent-Sergeli",
            "Samarqand-Markaz", "Samarqand-Siyob", "Buxoro-Markaz",
            "Namangan-Markaz", "Andijon-Markaz"]
GURUHLAR = ["non", "sut", "meva"]
# sintetik taqvim: "bayram kunlari" (sanalar o'quv maqsadida to'qilgan)
BAYRAMLAR = ["2023-01-01", "2023-03-21", "2023-04-21", "2023-06-28",
             "2023-09-01", "2024-01-01", "2024-03-21", "2024-04-10",
             "2024-06-16", "2024-09-01", "2025-01-01", "2025-03-21",
             "2025-03-30", "2025-06-06", "2025-09-01"]
HAFTALIK = {"non": [1.0, 0.95, 0.95, 1.0, 1.05, 1.2, 1.1],
            "sut": [1.0, 1.0, 0.95, 1.0, 1.05, 1.15, 1.1],
            "meva": [0.9, 0.9, 0.9, 0.95, 1.05, 1.25, 1.3]}
YILLIK = {"non": (0.05, 15), "sut": (0.08, 30), "meva": (0.35, 200)}
ASOS = {"non": 120, "sut": 80, "meva": 60}


def malumot(seed=0):
    """Toza kunlik savdo: Y (24 qator x kun), aksiya A, taqvim, kutilgan mu."""
    rng = np.random.default_rng(seed)
    sana = pd.date_range("2023-01-01", "2025-12-31", freq="D")
    t = np.arange(len(sana))
    dow, doy = sana.dayofweek.to_numpy(), sana.dayofyear.to_numpy()
    bayram = sana.isin(pd.to_datetime(BAYRAMLAR))
    arafa = (np.roll(bayram, -1) | np.roll(bayram, -2)) & ~bayram
    Y, A, MU = [], [], []
    for _ in DOKONLAR:
        olcham, osish = rng.uniform(0.6, 1.6), rng.uniform(0.03, 0.15)
        for g in GURUHLAR:
            amp, choqqi = YILLIK[g]
            mu = (ASOS[g] * olcham * (1 + osish) ** (t / 365.25)
                  * np.array(HAFTALIK[g])[dow]
                  * (1 + amp * np.cos(2 * np.pi * (doy - choqqi) / 365.25))
                  * np.where(bayram, 0.4, 1.0) * np.where(arafa, 1.8, 1.0))
            nisbiy = mu / pd.Series(mu).rolling(29, center=True,
                                                min_periods=1).mean().to_numpy()
            p = np.clip(0.05 * nisbiy ** -6.0, 0, 0.35) * ~bayram * ~arafa
            aksiya = rng.random(len(t)) < p               # sust kunlarda ko'proq
            Y.append(rng.poisson(mu * np.where(aksiya, 1.25, 1.0)
                                 * rng.gamma(20, 1 / 20, len(t))))
            A.append(aksiya)
            MU.append(mu)
    kal = {"sana": sana, "dow": dow, "doy": doy, "bayram": bayram,
           "arafa": arafa}
    return np.array(Y, dtype=float), np.array(A), kal, np.array(MU)


def xom_jadval(Y, kal, rng):
    """Toza ma'lumotdan kassa tizimi 'xom' eksporti: xatolar bilan."""
    S, T = Y.shape
    xom = pd.DataFrame({"qator": np.repeat(np.arange(S), T),
                        "kun": np.tile(np.arange(T), S),
                        "savdo": Y.ravel(), "tur": "toza"})
    t0 = kal["sana"].get_loc(pd.Timestamp("2024-08-05"))
    yop = (xom.qator // 3 == 5) & xom.kun.between(t0, t0 + 6)   # ta'mir
    xom.loc[yop, "savdo"] = 0.0
    xom.loc[yop, "tur"] = "yopilish"
    toza = np.flatnonzero((xom.tur == "toza").to_numpy() & (xom.kun >= 60).to_numpy())
    pos_kun = rng.choice(np.arange(60, T), 12, replace=False)
    pos_dokon = rng.integers(0, len(DOKONLAR), 12)
    for d, k in zip(pos_dokon, pos_kun):                        # kassa noli
        m = (xom.qator // 3 == d) & (xom.kun == k) & (xom.tur == "toza")
        xom.loc[m, "savdo"] = 0.0
        xom.loc[m, "tur"] = "kassa_nol"
    toza = np.flatnonzero((xom.tur == "toza").to_numpy() & (xom.kun >= 60).to_numpy())
    tanlov = rng.choice(toza, 50, replace=False)
    xom.loc[tanlov[:30], "savdo"] *= 10                         # birlik xatosi
    xom.loc[tanlov[:30], "tur"] = "birlik"
    xom.loc[tanlov[30:], "savdo"] *= -1                         # qaytarish yozuvi
    xom.loc[tanlov[30:], "tur"] = "manfiy"
    toza = np.flatnonzero((xom.tur == "toza").to_numpy())
    yoqolgan = rng.choice(toza, int(0.015 * len(xom)), replace=False)
    xom = xom.drop(index=yoqolgan)
    takror = xom.sample(40, random_state=1)
    return pd.concat([xom, takror], ignore_index=True), len(yoqolgan)


def siljit(M, k):
    out = np.full(M.shape, np.nan)
    out[:, k:] = M[:, :-k]
    return out


def main() -> None:
    Y, A, kal, _ = malumot(0)
    S, T = Y.shape
    xom, n_yoq = xom_jadval(Y, kal, np.random.default_rng(1))

    print("=== 1. Xom ma'lumot ===")
    print(f"  qatorlar: {len(DOKONLAR)} do'kon x {len(GURUHLAR)} guruh = {S}, "
          f"kunlar {T} ({kal['sana'][0].date()} - {kal['sana'][-1].date()})")
    print(f"  kutilgan yozuvlar {S * T:,}, xom eksportda {len(xom):,}")
    print(f"  aksiya kunlari ulushi {A.mean():.3f}, bayram kunlari "
          f"{int(kal['bayram'].sum())}, arafa kunlari {int(kal['arafa'].sum())}")

    print("\n=== 2. Validatsiya: takror, manfiy, yo'q kunlar ===")
    takror = int(xom.duplicated(["qator", "kun"]).sum())
    xom = xom.drop_duplicates(["qator", "kun"])
    manfiy = int((xom.savdo < 0).sum())
    xom.loc[xom.savdo < 0, "savdo"] = np.nan
    W = (xom.pivot(index="qator", columns="kun", values="savdo")
         .reindex(index=range(S), columns=range(T)).to_numpy())
    tur = (xom.pivot(index="qator", columns="kun", values="tur")
           .reindex(index=range(S), columns=range(T)).to_numpy())
    yoq = np.isnan(W) & pd.isna(tur)
    print(f"  takror yozuvlar: {takror} -> olib tashlandi")
    print(f"  manfiy savdo: {manfiy} -> NaN")
    print(f"  yo'q kunlar (panjarada NaN): {int(yoq.sum())} "
          f"(yashirilgan: {n_yoq}), ulushi {yoq.mean():.4f}")
    eng = np.argsort(-yoq.sum(1), kind="stable")[:3]
    print(f"  eng ko'p yo'q: " + ", ".join(
        f"{DOKONLAR[s // 3]}/{GURUHLAR[s % 3]} {int(yoq[s].sum())}" for s in eng))

    print("\n=== 3. Anomaliyalar: robust qoldiq (o'tgan 4 haftaning shu kuni) ===")
    L = np.log1p(W)
    with warnings.catch_warnings():          # birinchi 4 hafta: hamma qiymat NaN
        warnings.simplefilter("ignore", RuntimeWarning)
        baza = np.nanmedian(np.stack([siljit(L, 7 * k) for k in range(1, 5)]),
                            axis=0)
    r = L - baza
    med = np.nanmedian(r, axis=1, keepdims=True)
    mad = np.nanmedian(np.abs(r - med), axis=1, keepdims=True)
    z = (r - med) / (1.4826 * mad)
    haqiqiy = np.isin(tur, ["birlik", "kassa_nol"])
    kutilgan = (kal["bayram"] | kal["arafa"])[None, :] | A | (tur == "yopilish")
    baholanadi = ~np.isnan(z)
    print(f"  haqiqiy anomaliyalar: {int(haqiqiy.sum())} "
          f"(birlik {int((tur == 'birlik').sum())}, "
          f"kassa noli {int((tur == 'kassa_nol').sum())})")
    print(f"  {'chegara':>7} {'rejim':<10} {'belgilandi':>10} {'aniqlik':>8} "
          f"{'topildi':>8} {'kutilgan hodisa':>16}")
    for chegara in [3.0, 4.0, 5.0]:
        for rejim in ["taqvimsiz", "taqvimli"]:
            b = baholanadi & (np.abs(np.nan_to_num(z)) > chegara)
            if rejim == "taqvimli":
                b &= ~kutilgan
            tp = int((b & haqiqiy).sum())
            print(f"  {chegara:>7.1f} {rejim:<10} {int(b.sum()):>10} "
                  f"{tp / max(int(b.sum()), 1):>8.3f} "
                  f"{tp / int(haqiqiy.sum()):>8.3f} {int((b & kutilgan).sum()):>16}")
    bayroq = baholanadi & (np.abs(np.nan_to_num(z)) > 4.0) & ~kutilgan
    for nom in ["birlik", "kassa_nol"]:
        m = tur == nom
        print(f"  chegara 4, taqvimli: {nom} topildi {int((bayroq & m).sum())}"
              f"/{int(m.sum())}")
    otdi = haqiqiy & ~bayroq
    print(f"  o'tkazib yuborilgan {int(otdi.sum())} tadan kutilgan hodisa kunida: "
          f"{int((otdi & kutilgan).sum())}, ularning |z| minimumi "
          f"{np.abs(z[otdi & kutilgan]).min():.1f}")

    print("\n=== 4. To'ldirish siyosati va uning xatosi ===")
    tuzat = yoq | bayroq | np.isnan(W) & ~yoq
    medianasi = np.expm1(baza)
    ok = tuzat & ~np.isnan(medianasi)
    for nom, qiymat in [("0 bilan", np.zeros_like(W)),
                        ("hafta kuni medianasi", medianasi)]:
        wape = np.abs(qiymat[ok] - Y[ok]).sum() / Y[ok].sum()
        print(f"  {nom:<22} to'ldirilgan {int(ok.sum())} katak, WAPE "
              f"(haqiqiy qiymatga nisbatan) {wape:.3f}")
    toza = np.where(ok, medianasi, W)
    yopiq = tur == "yopilish"
    print(f"  yopilish kunlari: {int(yopiq.sum())} katak -> 0 qoladi, "
          f"o'quvdan chiqariladi")
    qoldi = int(np.isnan(toza).sum())
    print(f"  tozalangan panjarada qolgan NaN: {qoldi} (o'tgan 4 haftada "
          f"qiymat yo'q - baza hisoblanmadi)")
    xato_ulush = float(np.abs(np.nan_to_num(toza) - Y)[~yopiq].sum()
                       / Y[~yopiq].sum())
    print(f"  tozalangan va haqiqiy ma'lumot farqi (WAPE): {xato_ulush:.4f}")
    print("  ⭐ Kutilgan hodisa - anomaliya emas; to'ldirilgan qiymat - belgilanadi")


if __name__ == "__main__":
    main()

=== 1. Xom ma'lumot ===
  qatorlar: 8 do'kon x 3 guruh = 24, kunlar 1096 (2023-01-01 - 2025-12-31)
  kutilgan yozuvlar 26,304, xom eksportda 25,950
  aksiya kunlari ulushi 0.061, bayram kunlari 15, arafa kunlari 30

=== 2. Validatsiya: takror, manfiy, yo'q kunlar ===
  takror yozuvlar: 40 -> olib tashlandi
  manfiy savdo: 20 -> NaN
  yo'q kunlar (panjarada NaN): 394 (yashirilgan: 394), ulushi 0.0150
  eng ko'p yo'q: Toshkent-Yunusobod/non 25, Namangan-Markaz/non 22, Toshkent-Chilonzor/non 20

=== 3. Anomaliyalar: robust qoldiq (o'tgan 4 haftaning shu kuni) ===
  haqiqiy anomaliyalar: 66 (birlik 30, kassa noli 36)
  chegara rejim      belgilandi  aniqlik  topildi  kutilgan hodisa
      3.0 taqvimsiz         371    0.178    1.000              255
      3.0 taqvimli          116    0.491    0.864                0
      4.0 taqvimsiz         148    0.439    0.985               88
      4.0 taqvimli           60    0.933    0.848                0
      5.0 taqvimsiz          96    0.667    0.970               41
      5.0 taqvimli           55    1.000    0.833                0
  chegara 4, taqvimli: birlik topildi 23/30
  chegara 4, taqvimli: kassa_nol topildi 33/36
  o'tkazib yuborilgan 10 tadan kutilgan hodisa kunida: 9, ularning |z| minimumi 5.4

=== 4. To'ldirish siyosati va uning xatosi ===
  0 bilan                to'ldirilgan 473 katak, WAPE (haqiqiy qiymatga nisbatan) 1.000
  hafta kuni medianasi   to'ldirilgan 473 katak, WAPE (haqiqiy qiymatga nisbatan) 0.260
  yopilish kunlari: 21 katak -> 0 qoladi, o'quvdan chiqariladi
  tozalangan panjarada qolgan NaN: 1 (o'tgan 4 haftada qiymat yo'q - baza hisoblanmadi)
  tozalangan va haqiqiy ma'lumot farqi (WAPE): 0.0070
  ⭐ Kutilgan hodisa - anomaliya emas; to'ldirilgan qiymat - belgilanadi

"""2-qadam: rolling-origin backtest - seasonal naive, ETS va global HistGB."""

import warnings

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from statsmodels.tsa.holtwinters import ExponentialSmoothing

DOKONLAR = ["Toshkent-Chilonzor", "Toshkent-Yunusobod", "Toshkent-Sergeli",
            "Samarqand-Markaz", "Samarqand-Siyob", "Buxoro-Markaz",
            "Namangan-Markaz", "Andijon-Markaz"]
GURUHLAR = ["non", "sut", "meva"]
# sintetik taqvim: "bayram kunlari" (sanalar o'quv maqsadida to'qilgan)
BAYRAMLAR = ["2023-01-01", "2023-03-21", "2023-04-21", "2023-06-28",
             "2023-09-01", "2024-01-01", "2024-03-21", "2024-04-10",
             "2024-06-16", "2024-09-01", "2025-01-01", "2025-03-21",
             "2025-03-30", "2025-06-06", "2025-09-01"]
HAFTALIK = {"non": [1.0, 0.95, 0.95, 1.0, 1.05, 1.2, 1.1],
            "sut": [1.0, 1.0, 0.95, 1.0, 1.05, 1.15, 1.1],
            "meva": [0.9, 0.9, 0.9, 0.95, 1.05, 1.25, 1.3]}
YILLIK = {"non": (0.05, 15), "sut": (0.08, 30), "meva": (0.35, 200)}
ASOS = {"non": 120, "sut": 80, "meva": 60}
H = 28


def malumot(seed=0):
    """Toza kunlik savdo: Y (24 qator x kun), aksiya A, taqvim, kutilgan mu."""
    rng = np.random.default_rng(seed)
    sana = pd.date_range("2023-01-01", "2025-12-31", freq="D")
    t = np.arange(len(sana))
    dow, doy = sana.dayofweek.to_numpy(), sana.dayofyear.to_numpy()
    bayram = sana.isin(pd.to_datetime(BAYRAMLAR))
    arafa = (np.roll(bayram, -1) | np.roll(bayram, -2)) & ~bayram
    Y, A, MU = [], [], []
    for _ in DOKONLAR:
        olcham, osish = rng.uniform(0.6, 1.6), rng.uniform(0.03, 0.15)
        for g in GURUHLAR:
            amp, choqqi = YILLIK[g]
            mu = (ASOS[g] * olcham * (1 + osish) ** (t / 365.25)
                  * np.array(HAFTALIK[g])[dow]
                  * (1 + amp * np.cos(2 * np.pi * (doy - choqqi) / 365.25))
                  * np.where(bayram, 0.4, 1.0) * np.where(arafa, 1.8, 1.0))
            nisbiy = mu / pd.Series(mu).rolling(29, center=True,
                                                min_periods=1).mean().to_numpy()
            p = np.clip(0.05 * nisbiy ** -6.0, 0, 0.35) * ~bayram * ~arafa
            aksiya = rng.random(len(t)) < p               # sust kunlarda ko'proq
            Y.append(rng.poisson(mu * np.where(aksiya, 1.25, 1.0)
                                 * rng.gamma(20, 1 / 20, len(t))))
            A.append(aksiya)
            MU.append(mu)
    kal = {"sana": sana, "dow": dow, "doy": doy, "bayram": bayram,
           "arafa": arafa}
    return np.array(Y, dtype=float), np.array(A), kal, np.array(MU)


def jadval(Y, A, kal, originlar):
    """Direct strategiya belgilari: faqat origin gacha ma'lum yoki rejadagi."""
    S = Y.shape[0]
    s, o, h = (a.ravel() for a in np.meshgrid(np.arange(S), originlar,
                                              np.arange(1, H + 1), indexing="ij"))
    t = o + h
    cs = np.concatenate([np.zeros((S, 1)), np.cumsum(Y, axis=1)], axis=1)

    def orta(a, b):                                   # Y[s, a..b] o'rtachasi
        return (cs[s, b + 1] - cs[s, a]) / (b - a + 1)

    d28 = np.log1p(orta(o - 27, o))
    X = np.column_stack([
        s // 3, s % 3, h, kal["dow"][t],
        np.sin(2 * np.pi * kal["doy"][t] / 365.25),
        np.cos(2 * np.pi * kal["doy"][t] / 365.25),
        kal["bayram"][t], kal["arafa"][t], A[s, t],
        np.log1p(orta(o - 6, o)) - d28,
        np.log1p(Y[s, t - 7 * np.ceil(h / 7).astype(int)]) - d28,
        np.log1p(orta(t - 367, t - 361)) - np.log1p(orta(o - 391, o - 364)),
    ])
    return X, s, t, d28


def hgb(Y, A, kal, O, **kw):
    """O gacha bo'lgan ma'lumotda o'qitish: maqsad kunlari O dan oshmaydi."""
    originlar = np.arange(O - H, 399, -14)[::-1]
    X, s, t, d28 = jadval(Y, A, kal, originlar)
    param = dict(max_iter=120, learning_rate=0.1, max_leaf_nodes=15,
                 min_samples_leaf=40, early_stopping=False,
                 categorical_features=[0, 1], random_state=0)
    param.update(kw)
    return HistGradientBoostingRegressor(**param).fit(
        X, np.log1p(Y[s, t]) - d28), len(X)


def ets(Y, kal, O):
    f = np.empty((Y.shape[0], H))
    indeks = pd.date_range(kal["sana"][O - 181], periods=182, freq="D")
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        for s in range(Y.shape[0]):
            qator = pd.Series(np.log1p(Y[s, O - 181:O + 1]), index=indeks)
            fit = ExponentialSmoothing(qator, trend=None, seasonal="add",
                                       seasonal_periods=7).fit(use_brute=False)
            f[s] = np.expm1(fit.forecast(H).to_numpy())
    return f.clip(0)


def main() -> None:
    Y, A, kal, _ = malumot(0)
    S, T = Y.shape
    O_test = T - 1 - H
    oynalar = [O_test - H * k for k in range(4, 0, -1)]
    modellar = ["seasonal naive", "ETS", "HistGB"]

    print("=== 1. Backtest dizayni ===")
    print(f"  {S} qator, {T} kun; test (tegilmaydi): "
          f"{kal['sana'][O_test + 1].date()} - {kal['sana'][-1].date()}")
    print(f"  4 oyna originlari: {kal['sana'][oynalar[0]].date()} - "
          f"{kal['sana'][oynalar[-1]].date()} (har 28 kun, hafta kuni "
          f"{int(kal['dow'][oynalar[0]])}: 0 - dushanba)")

    yozuv = []                   # (model, oyna, qator, t, y, f)
    for w, O in enumerate(oynalar):
        m, n_orq = hgb(Y, A, kal, O)
        X, s, t, d28 = jadval(Y, A, kal, np.array([O]))
        f_h = np.expm1(m.predict(X) + d28).clip(0)
        h = t - O
        f_sn = Y[s, t - 7 * np.ceil(h / 7).astype(int)]
        f_e = ets(Y, kal, O)[s, h - 1]
        for nom, f in zip(modellar, [f_sn, f_e, f_h]):
            yozuv.append(pd.DataFrame({"model": nom, "oyna": w, "qator": s,
                                       "t": t, "y": Y[s, t], "f": f}))
        if w == 0:
            print(f"  HistGB o'quv misollari (1-oyna): {n_orq:,}")
    df = pd.concat(yozuv, ignore_index=True)
    df["dokon"] = df.qator // 3
    df["xato"] = (df.y - df.f).abs()
    df["hafta"] = (df.t - df.oyna.map(dict(enumerate(oynalar))) - 1) // 7 + 1
    df["kun_turi"] = np.select(
        [kal["bayram"][df.t] | kal["arafa"][df.t], A[df.qator, df.t]],
        ["bayram/arafa", "aksiya"], "oddiy")

    birlik = (df.groupby(["model", "dokon", "oyna"])[["xato", "y"]].sum()
              .assign(wape=lambda d: d.xato / d.y)["wape"].unstack("model"))
    print(f"\n=== 2. WAPE: {len(birlik)} ta do'kon x oyna birligi ===")
    print(f"  {'model':<15} {'o_rtacha':>9} {'std':>7}   " + "  ".join(
        f"{k:>12}" for k in ["oddiy", "bayram/arafa", "aksiya"]))
    for nom in modellar:
        q = df[df.model == nom]
        turlar = [q[q.kun_turi == k] for k in ["oddiy", "bayram/arafa",
                                              "aksiya"]]
        print(f"  {nom:<15} {birlik[nom].mean():>9.4f} {birlik[nom].std():>7.4f}"
              f"   " + "  ".join(f"{x.xato.sum() / x.y.sum():>12.4f}"
                                 for x in turlar))

    print("\n=== 3. Juftlashgan taqqoslash va qaror ===")
    eng = birlik.mean().idxmin()
    print(f"  eng yaxshi (o'rtacha WAPE): {eng}")
    tanlov = None
    for nom in modellar:                                    # soddalik tartibi
        if nom == eng:
            yomon = False
            print(f"  {nom:<15} - eng yaxshi")
        else:
            d = birlik[nom] - birlik[eng]
            se = d.std(ddof=1) / np.sqrt(len(d))
            yomon = d.mean() > 2 * se
            print(f"  {nom:<15} - {eng}: {d.mean():+.4f} (SE {se:.4f}), "
                  f"sezilarli yomon: {yomon}")
        if tanlov is None and not yomon:
            tanlov = nom
    print(f"  QOIDA -> tanlov: {tanlov}")
    yutgan = (birlik["HistGB"] < birlik["seasonal naive"]).mean()
    print(f"  HistGB seasonal naive dan yaxshi bo'lgan birliklar: {yutgan:.2f}")

    print("\n=== 4. Ufq bo'yicha WAPE (hafta) ===")
    print(f"  {'model':<15} " + " ".join(f"{f'{k}-hafta':>9}" for k in range(1, 5)))
    for nom in modellar:
        q = df[df.model == nom].groupby("hafta")[["xato", "y"]].sum()
        print(f"  {nom:<15} " + " ".join(f"{v:>9.4f}"
                                         for v in (q.xato / q.y).to_numpy()))
    print("  ⭐ Qaror - juftlashgan farq va oldindan yozilgan qoida bilan")


if __name__ == "__main__":
    main()

=== 1. Backtest dizayni ===
  24 qator, 1096 kun; test (tegilmaydi): 2025-12-04 - 2025-12-31
  4 oyna originlari: 2025-08-13 - 2025-11-05 (har 28 kun, hafta kuni 2: 0 - dushanba)
  HistGB o'quv misollari (1-oyna): 25,536

=== 2. WAPE: 32 ta do'kon x oyna birligi ===
  model            o_rtacha     std          oddiy  bayram/arafa        aksiya
  seasonal naive     0.2905  0.0373         0.2804        0.5391        0.2934
  ETS                0.2272  0.0288         0.2074        0.5409        0.2645
  HistGB             0.1994  0.0216         0.1973        0.1819        0.1822

=== 3. Juftlashgan taqqoslash va qaror ===
  eng yaxshi (o'rtacha WAPE): HistGB
  seasonal naive  - HistGB: +0.0911 (SE 0.0071), sezilarli yomon: True
  ETS             - HistGB: +0.0278 (SE 0.0046), sezilarli yomon: True
  HistGB          - eng yaxshi
  QOIDA -> tanlov: HistGB
  HistGB seasonal naive dan yaxshi bo'lgan birliklar: 1.00

=== 4. Ufq bo'yicha WAPE (hafta) ===
  model             1-hafta   2-hafta   3-hafta   4-hafta
  seasonal naive     0.2796    0.2780    0.3086    0.2977
  ETS                0.2104    0.2096    0.2615    0.2131
  HistGB             0.1990    0.1943    0.1947    0.1945
  ⭐ Qaror - juftlashgan farq va oldindan yozilgan qoida bilan

"""3-qadam: konformal va kvantil intervallar, qamrov, buyurtma kvantilini tanlash."""

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor

DOKONLAR = ["Toshkent-Chilonzor", "Toshkent-Yunusobod", "Toshkent-Sergeli",
            "Samarqand-Markaz", "Samarqand-Siyob", "Buxoro-Markaz",
            "Namangan-Markaz", "Andijon-Markaz"]
GURUHLAR = ["non", "sut", "meva"]
# sintetik taqvim: "bayram kunlari" (sanalar o'quv maqsadida to'qilgan)
BAYRAMLAR = ["2023-01-01", "2023-03-21", "2023-04-21", "2023-06-28",
             "2023-09-01", "2024-01-01", "2024-03-21", "2024-04-10",
             "2024-06-16", "2024-09-01", "2025-01-01", "2025-03-21",
             "2025-03-30", "2025-06-06", "2025-09-01"]
HAFTALIK = {"non": [1.0, 0.95, 0.95, 1.0, 1.05, 1.2, 1.1],
            "sut": [1.0, 1.0, 0.95, 1.0, 1.05, 1.15, 1.1],
            "meva": [0.9, 0.9, 0.9, 0.95, 1.05, 1.25, 1.3]}
YILLIK = {"non": (0.05, 15), "sut": (0.08, 30), "meva": (0.35, 200)}
ASOS = {"non": 120, "sut": 80, "meva": 60}
H = 28
C_U, C_O = 3.0, 1.0            # faraziy: yetishmaslik va ortiqcha narxi (1 dona)


def malumot(seed=0):
    """Toza kunlik savdo: Y (24 qator x kun), aksiya A, taqvim, kutilgan mu."""
    rng = np.random.default_rng(seed)
    sana = pd.date_range("2023-01-01", "2025-12-31", freq="D")
    t = np.arange(len(sana))
    dow, doy = sana.dayofweek.to_numpy(), sana.dayofyear.to_numpy()
    bayram = sana.isin(pd.to_datetime(BAYRAMLAR))
    arafa = (np.roll(bayram, -1) | np.roll(bayram, -2)) & ~bayram
    Y, A, MU = [], [], []
    for _ in DOKONLAR:
        olcham, osish = rng.uniform(0.6, 1.6), rng.uniform(0.03, 0.15)
        for g in GURUHLAR:
            amp, choqqi = YILLIK[g]
            mu = (ASOS[g] * olcham * (1 + osish) ** (t / 365.25)
                  * np.array(HAFTALIK[g])[dow]
                  * (1 + amp * np.cos(2 * np.pi * (doy - choqqi) / 365.25))
                  * np.where(bayram, 0.4, 1.0) * np.where(arafa, 1.8, 1.0))
            nisbiy = mu / pd.Series(mu).rolling(29, center=True,
                                                min_periods=1).mean().to_numpy()
            p = np.clip(0.05 * nisbiy ** -6.0, 0, 0.35) * ~bayram * ~arafa
            aksiya = rng.random(len(t)) < p               # sust kunlarda ko'proq
            Y.append(rng.poisson(mu * np.where(aksiya, 1.25, 1.0)
                                 * rng.gamma(20, 1 / 20, len(t))))
            A.append(aksiya)
            MU.append(mu)
    kal = {"sana": sana, "dow": dow, "doy": doy, "bayram": bayram,
           "arafa": arafa}
    return np.array(Y, dtype=float), np.array(A), kal, np.array(MU)


def jadval(Y, A, kal, originlar):
    """Direct strategiya belgilari: faqat origin gacha ma'lum yoki rejadagi."""
    S = Y.shape[0]
    s, o, h = (a.ravel() for a in np.meshgrid(np.arange(S), originlar,
                                              np.arange(1, H + 1), indexing="ij"))
    t = o + h
    cs = np.concatenate([np.zeros((S, 1)), np.cumsum(Y, axis=1)], axis=1)

    def orta(a, b):                                   # Y[s, a..b] o'rtachasi
        return (cs[s, b + 1] - cs[s, a]) / (b - a + 1)

    d28 = np.log1p(orta(o - 27, o))
    X = np.column_stack([
        s // 3, s % 3, h, kal["dow"][t],
        np.sin(2 * np.pi * kal["doy"][t] / 365.25),
        np.cos(2 * np.pi * kal["doy"][t] / 365.25),
        kal["bayram"][t], kal["arafa"][t], A[s, t],
        np.log1p(orta(o - 6, o)) - d28,
        np.log1p(Y[s, t - 7 * np.ceil(h / 7).astype(int)]) - d28,
        np.log1p(orta(t - 367, t - 361)) - np.log1p(orta(o - 391, o - 364)),
    ])
    return X, s, t, d28


def hgb(Y, A, kal, O, **kw):
    """O gacha bo'lgan ma'lumotda o'qitish: maqsad kunlari O dan oshmaydi."""
    originlar = np.arange(O - H, 399, -14)[::-1]
    X, s, t, d28 = jadval(Y, A, kal, originlar)
    param = dict(max_iter=120, learning_rate=0.1, max_leaf_nodes=15,
                 min_samples_leaf=40, early_stopping=False,
                 categorical_features=[0, 1], random_state=0)
    param.update(kw)
    return HistGradientBoostingRegressor(**param).fit(
        X, np.log1p(Y[s, t]) - d28)


def narx(y, q):
    return C_U * np.maximum(y - q, 0) + C_O * np.maximum(q - y, 0)


def main() -> None:
    Y, A, kal, _ = malumot(0)
    S, T = Y.shape
    O_test = T - 1 - H
    oynalar = [O_test - H * k for k in range(4, 0, -1)]
    kalibr, baho = oynalar[:2], oynalar[2:]

    qoldiq = {k: [] for k in range(1, 5)}
    qatorlar = []
    n_bayram_kal = 0
    for O in oynalar:
        X, s, t, d28 = jadval(Y, A, kal, np.array([O]))
        f = np.expm1(hgb(Y, A, kal, O).predict(X) + d28).clip(0)
        hafta = (t - O - 1) // 7 + 1
        r = np.log1p(Y[s, t]) - np.log1p(f)
        if O in kalibr:
            for k in range(1, 5):
                qoldiq[k].extend(r[hafta == k].tolist())
            n_bayram_kal = n_bayram_kal + int(
                (kal["bayram"][t] | kal["arafa"][t]).sum())
            continue
        q = {a: np.expm1(hgb(Y, A, kal, O, loss="quantile", quantile=a)
                         .predict(X) + d28).clip(0) for a in (0.1, 0.9)}
        qatorlar.append(pd.DataFrame({
            "O": O, "dokon": s // 3, "t": t, "hafta": hafta, "y": Y[s, t],
            "f": f, "kq10": q[0.1], "kq90": q[0.9],
            "tur": np.select([kal["bayram"][t] | kal["arafa"][t], A[s, t]],
                             ["bayram/arafa", "aksiya"], "oddiy")}))
    df = pd.concat(qatorlar, ignore_index=True)
    kq = {k: np.quantile(qoldiq[k], [0.1, 0.9]) for k in qoldiq}
    lf = np.log1p(df.f)
    df["cq10"] = np.expm1(lf + df.hafta.map(lambda k: kq[k][0]))
    df["cq90"] = np.expm1(lf + df.hafta.map(lambda k: kq[k][1]))

    print("=== 1. Konformal kalibrovka (2 oyna qoldiqlari, log shkala) ===")
    for k in range(1, 5):
        print(f"  {k}-hafta: n={len(qoldiq[k])}, q0.1 {kq[k][0]:+.3f}, "
              f"q0.9 {kq[k][1]:+.3f}")

    print("\n=== 2. Nominal 80% interval: haqiqiy qamrov (2 baholash oynasi) ===")
    print(f"  {'usul':<10} {'umumiy':>7} {'oddiy':>7} {'aksiya':>7} "
          f"{'nisbiy kenglik':>15}")
    for nom, lo, hi in [("konformal", "cq10", "cq90"),
                        ("kvantil", "kq10", "kq90")]:
        ichida = (df.y >= df[lo]) & (df.y <= df[hi])
        kenglik = ((df[hi] - df[lo]) / df.f.clip(1)).mean()
        turlar = [ichida[df.tur == k].mean() for k in ["oddiy", "aksiya"]]
        print(f"  {nom:<10} {ichida.mean():>7.3f} {turlar[0]:>7.3f} "
              f"{turlar[1]:>7.3f} {kenglik:>15.3f}")
    n_tur = df.tur.value_counts()
    print(f"  baholash kunlari: oddiy {n_tur.get('oddiy', 0)}, aksiya "
          f"{n_tur.get('aksiya', 0)}, bayram/arafa {n_tur.get('bayram/arafa', 0)}"
          f"; kalibrovkada bayram/arafa qoldiqlari {n_bayram_kal}")
    for nom, lo, hi in [("konformal", "cq10", "cq90"), ("kvantil", "kq10", "kq90")]:
        haf = [((df.y >= df[lo]) & (df.y <= df[hi]))[df.hafta == k].mean()
               for k in range(1, 5)]
        print(f"  {nom:<10} hafta bo'yicha qamrov: "
              + " ".join(f"{v:.3f}" for v in haf))

    print(f"\n=== 3. Zaxira: c_u={C_U:.0f}, c_o={C_O:.0f} -> kritik nisbat "
          f"{C_U / (C_U + C_O):.2f} ===")
    print(f"  {'buyurtma kvantili':<18} {'o_rtacha narx':>13} {'y <= buyurtma':>14}")
    natija = {}
    for a in [0.5, 0.6, 0.75, 0.9]:
        qa = {k: np.quantile(qoldiq[k], a) for k in qoldiq}
        buyurtma = np.expm1(lf + df.hafta.map(qa))
        natija[a] = narx(df.y, buyurtma)
        print(f"  {a:<18} {natija[a].mean():>13.3f} "
              f"{(df.y <= buyurtma).mean():>14.3f}")
    natija["nuqta"] = narx(df.y, df.f)
    print(f"  {'nuqtali bashorat':<18} {natija['nuqta'].mean():>13.3f} "
          f"{(df.y <= df.f).mean():>14.3f}")
    eng = min((natija[a].mean(), a) for a in [0.5, 0.6, 0.75, 0.9])[1]
    print(f"  eng arzon kvantil: {eng}")

    print("\n=== 4. Juftlashgan: 0.75 kvantil - nuqtali bashorat (do'kon x oyna) ===")
    df["n75"], df["nq"] = natija[0.75], natija["nuqta"]
    birlik = df.groupby(["dokon", "O"])[["n75", "nq"]].sum()
    d = (birlik.n75 - birlik.nq) / birlik.nq
    se = d.std(ddof=1) / np.sqrt(len(d))
    print(f"  {len(d)} birlik: nisbiy narx o'zgarishi {d.mean():+.3f} "
          f"(SE {se:.3f}), sezilarli: {abs(d.mean()) > 2 * se}")
    print(f"  0.75 kvantil arzonroq bo'lgan birliklar: {(d < 0).mean():.2f}")
    print("  ⭐ Buyurtma - nuqtali bashorat emas, narxlar nisbatidagi kvantil")


if __name__ == "__main__":
    main()

=== 1. Konformal kalibrovka (2 oyna qoldiqlari, log shkala) ===
  1-hafta: n=336, q0.1 -0.323, q0.9 +0.324
  2-hafta: n=336, q0.1 -0.313, q0.9 +0.298
  3-hafta: n=336, q0.1 -0.353, q0.9 +0.316
  4-hafta: n=336, q0.1 -0.314, q0.9 +0.264

=== 2. Nominal 80% interval: haqiqiy qamrov (2 baholash oynasi) ===
  usul        umumiy   oddiy  aksiya  nisbiy kenglik
  konformal    0.749   0.749   0.747           0.635
  kvantil      0.745   0.746   0.723           0.650
  baholash kunlari: oddiy 1261, aksiya 83, bayram/arafa 0; kalibrovkada bayram/arafa qoldiqlari 72
  konformal  hafta bo'yicha qamrov: 0.762 0.738 0.801 0.693
  kvantil    hafta bo'yicha qamrov: 0.741 0.744 0.774 0.720

=== 3. Zaxira: c_u=3, c_o=1 -> kritik nisbat 0.75 ===
  buyurtma kvantili  o_rtacha narx  y <= buyurtma
  0.5                       58.522          0.493
  0.6                       50.661          0.620
  0.75                      47.707          0.751
  0.9                       54.706          0.876
  nuqtali bashorat          59.625          0.486
  eng arzon kvantil: 0.75

=== 4. Juftlashgan: 0.75 kvantil - nuqtali bashorat (do'kon x oyna) ===
  16 birlik: nisbiy narx o'zgarishi -0.200 (SE 0.022), sezilarli: True
  0.75 kvantil arzonroq bo'lgan birliklar: 0.94
  ⭐ Buyurtma - nuqtali bashorat emas, narxlar nisbatidagi kvantil

"""4-qadam: test bir marta, aksiya ta'siri (sababiy), paket va monitoring rejasi."""

import tempfile
from pathlib import Path

import joblib
import numpy as np
import pandas as pd
import statsmodels.api as sm
from sklearn.ensemble import HistGradientBoostingRegressor

DOKONLAR = ["Toshkent-Chilonzor", "Toshkent-Yunusobod", "Toshkent-Sergeli",
            "Samarqand-Markaz", "Samarqand-Siyob", "Buxoro-Markaz",
            "Namangan-Markaz", "Andijon-Markaz"]
GURUHLAR = ["non", "sut", "meva"]
# sintetik taqvim: "bayram kunlari" (sanalar o'quv maqsadida to'qilgan)
BAYRAMLAR = ["2023-01-01", "2023-03-21", "2023-04-21", "2023-06-28",
             "2023-09-01", "2024-01-01", "2024-03-21", "2024-04-10",
             "2024-06-16", "2024-09-01", "2025-01-01", "2025-03-21",
             "2025-03-30", "2025-06-06", "2025-09-01"]
HAFTALIK = {"non": [1.0, 0.95, 0.95, 1.0, 1.05, 1.2, 1.1],
            "sut": [1.0, 1.0, 0.95, 1.0, 1.05, 1.15, 1.1],
            "meva": [0.9, 0.9, 0.9, 0.95, 1.05, 1.25, 1.3]}
YILLIK = {"non": (0.05, 15), "sut": (0.08, 30), "meva": (0.35, 200)}
ASOS = {"non": 120, "sut": 80, "meva": 60}
H = 28
BELGILAR = ["dokon", "guruh", "h", "hafta_kuni", "yil_sin", "yil_cos",
            "bayram", "arafa", "aksiya", "daraja7_28", "sn_nisbiy", "otgan_yil"]


def malumot(seed=0):
    """Toza kunlik savdo: Y (24 qator x kun), aksiya A, taqvim, kutilgan mu."""
    rng = np.random.default_rng(seed)
    sana = pd.date_range("2023-01-01", "2025-12-31", freq="D")
    t = np.arange(len(sana))
    dow, doy = sana.dayofweek.to_numpy(), sana.dayofyear.to_numpy()
    bayram = sana.isin(pd.to_datetime(BAYRAMLAR))
    arafa = (np.roll(bayram, -1) | np.roll(bayram, -2)) & ~bayram
    Y, A, MU = [], [], []
    for _ in DOKONLAR:
        olcham, osish = rng.uniform(0.6, 1.6), rng.uniform(0.03, 0.15)
        for g in GURUHLAR:
            amp, choqqi = YILLIK[g]
            mu = (ASOS[g] * olcham * (1 + osish) ** (t / 365.25)
                  * np.array(HAFTALIK[g])[dow]
                  * (1 + amp * np.cos(2 * np.pi * (doy - choqqi) / 365.25))
                  * np.where(bayram, 0.4, 1.0) * np.where(arafa, 1.8, 1.0))
            nisbiy = mu / pd.Series(mu).rolling(29, center=True,
                                                min_periods=1).mean().to_numpy()
            p = np.clip(0.05 * nisbiy ** -6.0, 0, 0.35) * ~bayram * ~arafa
            aksiya = rng.random(len(t)) < p               # sust kunlarda ko'proq
            Y.append(rng.poisson(mu * np.where(aksiya, 1.25, 1.0)
                                 * rng.gamma(20, 1 / 20, len(t))))
            A.append(aksiya)
            MU.append(mu)
    kal = {"sana": sana, "dow": dow, "doy": doy, "bayram": bayram,
           "arafa": arafa}
    return np.array(Y, dtype=float), np.array(A), kal, np.array(MU)


def jadval(Y, A, kal, originlar):
    """Direct strategiya belgilari: faqat origin gacha ma'lum yoki rejadagi."""
    S = Y.shape[0]
    s, o, h = (a.ravel() for a in np.meshgrid(np.arange(S), originlar,
                                              np.arange(1, H + 1), indexing="ij"))
    t = o + h
    cs = np.concatenate([np.zeros((S, 1)), np.cumsum(Y, axis=1)], axis=1)

    def orta(a, b):                                   # Y[s, a..b] o'rtachasi
        return (cs[s, b + 1] - cs[s, a]) / (b - a + 1)

    d28 = np.log1p(orta(o - 27, o))
    X = np.column_stack([
        s // 3, s % 3, h, kal["dow"][t],
        np.sin(2 * np.pi * kal["doy"][t] / 365.25),
        np.cos(2 * np.pi * kal["doy"][t] / 365.25),
        kal["bayram"][t], kal["arafa"][t], A[s, t],
        np.log1p(orta(o - 6, o)) - d28,
        np.log1p(Y[s, t - 7 * np.ceil(h / 7).astype(int)]) - d28,
        np.log1p(orta(t - 367, t - 361)) - np.log1p(orta(o - 391, o - 364)),
    ])
    return X, s, t, d28


def hgb(Y, A, kal, O, **kw):
    """O gacha bo'lgan ma'lumotda o'qitish: maqsad kunlari O dan oshmaydi."""
    originlar = np.arange(O - H, 399, -14)[::-1]
    X, s, t, d28 = jadval(Y, A, kal, originlar)
    param = dict(max_iter=120, learning_rate=0.1, max_leaf_nodes=15,
                 min_samples_leaf=40, early_stopping=False,
                 categorical_features=[0, 1], random_state=0)
    param.update(kw)
    return HistGradientBoostingRegressor(**param).fit(
        X, np.log1p(Y[s, t]) - d28)


def prognoz(m, Y, A, kal, O):
    X, s, t, d28 = jadval(Y, A, kal, np.array([O]))
    return np.expm1(m.predict(X) + d28).clip(0), s, t, X


def main() -> None:
    Y, A, kal, MU = malumot(0)
    S, T = Y.shape
    O_test = T - 1 - H

    print("=== 1. Yakuniy test: oxirgi 28 kun, BIR MARTA ===")
    model = hgb(Y, A, kal, O_test)
    f, s, t, X = prognoz(model, Y, A, kal, O_test)
    h = t - O_test
    sn = Y[s, t - 7 * np.ceil(h / 7).astype(int)]
    y = Y[s, t]
    dk = s // 3
    w_m = np.array([np.abs(y - f)[dk == d].sum() / y[dk == d].sum() for d in range(8)])
    w_s = np.array([np.abs(y - sn)[dk == d].sum() / y[dk == d].sum() for d in range(8)])
    d = w_m - w_s
    print(f"  WAPE: HistGB {np.abs(y - f).sum() / y.sum():.4f}, seasonal naive "
          f"{np.abs(y - sn).sum() / y.sum():.4f}")
    print(f"  do'konlar bo'yicha farq {d.mean():+.4f} (SE "
          f"{d.std(ddof=1) / np.sqrt(8):.4f}); HistGB yaxshi do'konlar "
          f"{int((d < 0).sum())}/8")
    print(f"  eng yomon do'kon: {DOKONLAR[int(np.argmax(w_m))]} "
          f"(WAPE {w_m.max():.4f})")

    print("\n=== 2. Aksiya ta'siri (o'quv davri; haqiqiy +25.0%) ===")
    davr = np.arange(400, O_test + 1)
    sq, tq = (a.ravel() for a in np.meshgrid(np.arange(S), davr, indexing="ij"))
    ak, yq = A[sq, tq], Y[sq, tq]
    bayramsiz = ~(kal["bayram"][tq] | kal["arafa"][tq])
    dow = kal["dow"][tq]
    print(f"  aksiya ulushi: dush-chor {ak[dow <= 2].mean():.3f}, "
          f"shan-yak {ak[dow >= 5].mean():.3f}")
    natija = {}
    naiv = [yq[(sq == i) & ak & bayramsiz].mean()
            / yq[(sq == i) & ~ak & bayramsiz].mean() - 1 for i in range(S)]
    natija["naiv farq"] = (np.mean(naiv), np.std(naiv, ddof=1) / np.sqrt(S))
    oy = kal["sana"][tq].month.to_numpy()
    g = sq % 3
    nazorat = ([(dow == k) & (g == j) for k in range(1, 7) for j in range(3)]
               + [(oy == k) & (g == j) for k in range(2, 13) for j in range(3)]
               + [sq == i for i in range(1, S)]
               + [tq / 365.25 * (sq // 3 == k) for k in range(8)])  # o'sish
    Xr = sm.add_constant(np.column_stack([ak, kal["bayram"][tq],
                                          kal["arafa"][tq]] + nazorat)
                         .astype(float))
    r = sm.OLS(np.log1p(yq), Xr).fit(cov_type="cluster",
                                     cov_kwds={"groups": sq})
    natija["regressiya"] = (np.expm1(r.params[1]), r.bse[1])
    Xb = np.column_stack([sq // 3, sq % 3, dow,
                          np.sin(2 * np.pi * kal["doy"][tq] / 365.25),
                          np.cos(2 * np.pi * kal["doy"][tq] / 365.25),
                          kal["bayram"][tq], kal["arafa"][tq], tq])
    baza = HistGradientBoostingRegressor(
        max_iter=200, learning_rate=0.1, max_leaf_nodes=31, min_samples_leaf=20,
        early_stopping=False, categorical_features=[0, 1], random_state=0,
    ).fit(Xb[~ak], np.log1p(yq[~ak]))
    kontr = np.log1p(yq[ak]) - baza.predict(Xb[ak])
    per = [kontr[sq[ak] == i].mean() for i in range(S)]
    natija["kontrfaktual model"] = (np.expm1(np.mean(per)),
                                    np.std(per, ddof=1) / np.sqrt(S))
    per = []
    for i in range(S):
        farqlar = []
        for tt in davr[A[i, davr] & ~(kal["bayram"] | kal["arafa"])[davr]]:
            nomzod = [tt + k for k in (-14, -7, 7, 14)
                      if 400 <= tt + k <= O_test and not A[i, tt + k]
                      and not (kal["bayram"] | kal["arafa"])[tt + k]]
            if nomzod:
                farqlar.append(np.log1p(Y[i, tt]) - np.log1p(Y[i, nomzod]).mean())
        per.append(np.mean(farqlar))
    natija["moslashtirish"] = (np.expm1(np.mean(per)),
                               np.std(per, ddof=1) / np.sqrt(S))
    for nom, (v, se) in natija.items():
        print(f"  {nom:<19} {v:+.3f} (SE {se:.3f})  2*SE ichida: "
              f"{abs(v - 0.25) <= 2 * se}")

    print("\n=== 3. Model paketi (joblib + metadata + nazorat namunalari) ===")
    paket = {
        "format": 1, "model": model, "belgilar": BELGILAR,
        "metadata": {"model": "HistGB global, direct, h=1..28",
                     "oqitish_davri": f"{kal['sana'][400].date()} - "
                                      f"{kal['sana'][O_test].date()}",
                     "qatorlar": S, "maqsad": "log1p(y) - daraja28",
                     "tanlov": "backtest, juftlashgan qoida (2-qadam)",
                     "zaxira_kvantili": 0.75},
        "nazorat": {"X": X[:50], "bashorat": model.predict(X[:50])},
    }
    with tempfile.TemporaryDirectory() as papka:
        yol = Path(papka) / "talab_modeli.joblib"
        joblib.dump(paket, yol)
        p2 = joblib.load(yol)
    mos = np.allclose(p2["model"].predict(p2["nazorat"]["X"]),
                      p2["nazorat"]["bashorat"], rtol=0, atol=1e-12)
    print(f"  kalitlar: {sorted(p2)}")
    print(f"  belgilar ({len(p2['belgilar'])}): {p2['belgilar'][:4]} (+8 ta)")
    for k in sorted(p2["metadata"]):
        print(f"  {k}: {p2['metadata'][k]}")
    print(f"  nazorat namunalari (50) yuklangach mos: {mos}")

    print("\n=== 4. Monitoring rejasi: chegaralar backtestdan ===")
    haftalik = []
    for k in (3, 2, 1):
        O = O_test - H * k
        fb, sb, tb, _ = prognoz(hgb(Y, A, kal, O), Y, A, kal, O)
        yb = Y[sb, tb]
        for dd in range(8):
            for w in range(4):
                m = (sb // 3 == dd) & ((tb - O - 1) // 7 == w)
                haftalik.append(np.abs(yb - fb)[m].sum() / yb[m].sum())
    haftalik = np.array(haftalik)
    chegara = float(np.quantile(haftalik, 0.95))
    print(f"  backtest: do'kon-hafta WAPE mediana {np.median(haftalik):.3f}, "
          f"95-persentil {chegara:.3f} ({len(haftalik)} ta)")
    test_h = np.array([[np.abs(y - f)[(dk == dd) & ((t - O_test - 1) // 7 == w)].sum()
                        / y[(dk == dd) & ((t - O_test - 1) // 7 == w)].sum()
                        for w in range(4)] for dd in range(8)])
    oshgan = test_h > chegara
    ketma = int(np.sum(oshgan[:, 1:] & oshgan[:, :-1]))
    print(f"  test davri: chegaradan oshgan do'kon-haftalar {int(oshgan.sum())}/32,"
          f" 2 hafta ketma-ket: {ketma}")
    reja = [
        ("har kuni", "yo'q qatorlar > 1% yoki kassa noli", "insident, to'ldirish"),
        ("har kuni", "anomaliya ulushi > 0.5%", "ma'lumot egasiga xabar"),
        ("har hafta", f"do'kon WAPE > {chegara:.3f}, 2 hafta ketma-ket",
         "tekshiruv, trigger"),
        ("har 4 hafta", "80% interval qamrovi < 0.70", "qayta kalibrlash"),
        ("har 4 hafta", "jadval bo'yicha qayta o'qitish", "backtest darvozasi"),
    ]
    for davriy, shart, harakat in reja:
        print(f"  {davriy:<12} {shart:<40} -> {harakat}")
    print("  ⭐ Test - hisobot uchun; aksiya ta'siri - tuzatilgan; paket o'zini tekshiradi")


if __name__ == "__main__":
    main()

=== 1. Yakuniy test: oxirgi 28 kun, BIR MARTA ===
  WAPE: HistGB 0.2092, seasonal naive 0.3245
  do'konlar bo'yicha farq -0.1170 (SE 0.0123); HistGB yaxshi do'konlar 8/8
  eng yomon do'kon: Andijon-Markaz (WAPE 0.2297)

=== 2. Aksiya ta'siri (o'quv davri; haqiqiy +25.0%) ===
  aksiya ulushi: dush-chor 0.084, shan-yak 0.023
  naiv farq           +0.191 (SE 0.015)  2*SE ichida: False
  regressiya          +0.259 (SE 0.010)  2*SE ichida: True
  kontrfaktual model  +0.257 (SE 0.010)  2*SE ichida: True
  moslashtirish       +0.254 (SE 0.011)  2*SE ichida: True

=== 3. Model paketi (joblib + metadata + nazorat namunalari) ===
  kalitlar: ['belgilar', 'format', 'metadata', 'model', 'nazorat']
  belgilar (12): ['dokon', 'guruh', 'h', 'hafta_kuni'] (+8 ta)
  maqsad: log1p(y) - daraja28
  model: HistGB global, direct, h=1..28
  oqitish_davri: 2024-02-05 - 2025-12-03
  qatorlar: 24
  tanlov: backtest, juftlashgan qoida (2-qadam)
  zaxira_kvantili: 0.75
  nazorat namunalari (50) yuklangach mos: True

=== 4. Monitoring rejasi: chegaralar backtestdan ===
  backtest: do'kon-hafta WAPE mediana 0.203, 95-persentil 0.254 (96 ta)
  test davri: chegaradan oshgan do'kon-haftalar 3/32, 2 hafta ketma-ket: 0
  har kuni     yo'q qatorlar > 1% yoki kassa noli       -> insident, to'ldirish
  har kuni     anomaliya ulushi > 0.5%                  -> ma'lumot egasiga xabar
  har hafta    do'kon WAPE > 0.254, 2 hafta ketma-ket   -> tekshiruv, trigger
  har 4 hafta  80% interval qamrovi < 0.70              -> qayta kalibrlash
  har 4 hafta  jadval bo'yicha qayta o'qitish           -> backtest darvozasi
  ⭐ Test - hisobot uchun; aksiya ta'siri - tuzatilgan; paket o'zini tekshiradi

W = df.pivot(...).fillna(0)                                     # ⚠️
W = df.pivot(...).reindex(columns=range(T))                     # ✅ NaN, keyin siyosat bilan

df.loc[np.abs(z) > 3, "savdo"] = np.nan                         # ⚠️ bayramlar ham ketadi
anomaliya = (np.abs(z) > 4) & ~kutilgan                         # ✅ bayram/aksiya - belgi

df["orta7"] = df.savdo.rolling(7, center=True).mean()           # ⚠️ kelajak qiymatlari
d7 = orta(o - 6, o)                                             # ✅ faqat origin gacha

train_test_split(X, y, test_size=0.2)                           # ⚠️
oynalar = [O_test - 28 * k for k in range(4, 0, -1)]            # ✅ rolling-origin

buyurtma = f                                                    # ⚠️ mediana atrofida
buyurtma = np.expm1(np.log1p(f) + np.quantile(qoldiq, C_U / (C_U + C_O)))  # ✅

tasir = y[aksiya].mean() / y[~aksiya].mean() - 1                # ⚠️ sust kunlar
tasir = np.expm1(np.mean(np.log1p(y[aksiya]) - f_aksiyasiz))    # ✅ kontrfaktual

if wape_hafta > 0.3: ogohlantir()                               # ⚠️ qayerdan 0.3?
chegara = np.quantile(backtest_haftalik_wape, 0.95)             # ✅ + 2 hafta ketma-ket

1.  # yo'q kunni 0 bilan to'ldirsak, keyingi haftalar bashorati?
2.  # taqvimsiz anomaliya detektori bayram kunlarini nima qiladi?
3.  # kutilgan kunlarni butunlay chiqarib tashlashning narxi?
4.  # bayram kunlarida seasonal naive xatosi qanday?
5.  # nega o'quv originlari O - 28 dan oshmaydi?
6.  # HistGB xatosi ufq bo'yicha qanday o'zgaradi?
7.  # nominal 80% konformal interval qamrovi nimaga bog'liq?
8.  # c_u = 3, c_o = 1 - buyurtma kvantili?
9.  # nuqtali bashoratni buyurtma qilsak, talab qoplanadigan kunlar ulushi?
10. # aksiyalar sust kunlarda - naiv baho qaysi tomonga siljiydi?
11. # test davri yomon chiqsa, model almashtiramizmi?
12. # monitoring chegarasi qayerdan olinadi?

1.  W = xom.pivot(index="qator", columns="kun", values="savdo").fillna(0)

2.  df["orta7"] = df.groupby("qator").savdo.transform(lambda s: s.rolling(7, center=True).mean())

3.  X_tr, X_te = train_test_split(X, test_size=0.2, random_state=0)

4.  buyurtma = model.predict(X_kelajak)

5.  tasir = df[df.aksiya].savdo.mean() / df[~df.aksiya].savdo.mean() - 1