27.13-dars: Qayta o'qitish va A/B test

1. JADVAL BO'YICHA (har hafta / har oy)
   + oddiy, bashorat qilinadigan, resurs rejalashtiriladi
   - drift bo'lmasa - bekor resurs; to'satdan driftda - kechikish
   - jadval "keyingi o'qitish" gacha zararni to'xtatmaydi

2. TRIGGER BO'YICHA
   trigger: sifat tushishi (belgilar kelgach) > chegara
            muhim ustun / bashorat drifti > chegara 27.12-bob
            yetarli yangi belgi to'plandi (masalan, 10 000 ta)
   + resurs faqat kerak bo'lganda
   - trigger chegaralarini sozlash kerak; belgilar kechikadi

3. UZLUKSIZ (onlayn o'rganish, har hodisadan keyin)
   + eng tez moslashadi
   - barqarorlik xavfi, zaharlanish (poisoning), sinash qiyin
   - faqat ba'zi vazifalar uchun (reklama, tavsiya)

AMALDA: jadval + trigger aralashmasi
  "har oy, yoki sifat 0.03 dan ko'p tushsa - darhol"
  har holda: qayta o'qitilgan model DARVOZA dan o'tadi, avtomatik emas

BUTUN TARIX:   [=========================]  hamma ma'lumot, teng vazn
OXIRGI N OY:            [======]            faqat yangi ma'lumot
VAZNLI:        [.:::::|||||||||||]          vazn = 0.5^(yosh / yarim_umr)

                barqaror   to'satdan   asta-sekin   qaytuvchi
butun tarix       yaxshi      sekin       sekin       o'rtacha
oxirgi N oy      biroz shov.  tez         tez         har almashishda
                                                      kechikadi
vaznli           yaxshi       o'rtacha    o'rtacha    o'rtacha
statik (hech)    yaxshi      yomon       yomon       ?

VAQT BO'YICHA BACKTEST:
  for t in baholash_oylari:
      o'quv = strategiya(0..t-1 oylar)     # kelajakka qarash YO'Q
      model = orgat(o'quv)
      ball[t] = metrika(model, t-oy)       # keyingi oy
  taqqoslash: bir xil oylar va urug'lar bo'yicha JUFTLASHGAN farq

                  so'rov
                    |
          +---------+----------+
          v                    v
     CHAMPION (v1)        CHALLENGER (v2)       <- shadow
          |                    |
     javob -> foydalanuvchi   javob -> faqat LOG
                               (xato bo'lsa ham foydalanuvchi sezmaydi)

  belgilar kelgach: BIR XIL so'rovlarda juftlashgan taqqoslash

DARVOZA (oldindan yoziladi, kodda, versiyalanadi):
  1. asosiy metrika: challenger - champion  sezilarli yaxshi
     (juftlashgan farq, |d| > 2*SE)
  2. hech bir muhim metrika yomonlashmadi (AUC, kalibrovka)
  3. SEGMENTLAR: har segmentda yomonlashuv <= chegara
     (mintaqa, qurilma, yangi/eski mijoz)
  4. operatsion: shadow xatolari <= 0.1%, kechikish p99 <= SLA
  5. hujjat: model kartasi, ma'lumot versiyasi, konfig

QOIDA: natijani ko'rgandan keyin qoidani o'zgartirish - p-hacking
       qoida yomon bo'lsa - KEYINGI tajriba uchun o'zgartiriladi

CANARY:  1% -> 10% -> 50% -> 100% trafik, har bosqichda guardrail
  guardrail (belgisiz, TEZ):
    xizmat xatolari, kechikish
    bashorat taqsimoti - O'Z shadow bashoratiga nisbatan (PSI)
    NAZORAT NAMUNALARI: registrda saqlangan kirish -> kutilgan chiqish
      (paketlash darsidagi g'oya, 27.6) - maks farq ~ 0 bo'lishi shart

AVTOMATIK ROLLBACK:
  guardrail buzildi -> registrda production = oldingi versiya
  oldingi versiya ARTEFAKTI saqlangan bo'lishi shart 27.5-bob
  rollback - bitta buyruq, sinab ko'rilgan, qo'lda emas

1. GIPOTEZA va METRIKA (oldindan): asosiy - konversiya; guardrail -
   qaytarish, shikoyat, kechikish; MDE - eng kichik muhim farq
2. NAMUNA HAJMI (quvvat tahlili, 11.2):
   n = (z_{1-a/2} * sqrt(2 p(1-p)) + z_{1-b} * sqrt(p1 q1 + p2 q2))^2 / d^2
   MDE 2 marta kichik -> n ~4 marta katta
3. TAQSIMLASH: foydalanuvchi ID si XESHI bilan
   guruh = hash(tajriba_nomi + ":" + user_id) -> [0, 1) -> < ulush ? B : A
   BARQAROR: bir foydalanuvchi har doim bir guruhda
   MUSTAQIL: har tajribaga o'z "tuzi" (salt)
   Python ning hash() - YO'Q: har jarayonda tasodifiy (PYTHONHASHSEED)
4. TAHLIL (belgilangan kunda, BIR MARTA):
   farq = p_B - p_A,  z-test,  95% CI = farq +- 1.96 * SE
5. QAROR: asosiy sezilarli yaxshi VA guardraillar chegaradan yomon emas

PEEKING: har kuni p-qiymatga qarab, p < 0.05 bo'lsa to'xtatish
  har qarash - yangi imkoniyat tasodifan 0.05 dan pastga tushish uchun
  30 kun har kuni qarash -> yolg'on musbat ~5% emas, ancha yuqori

YECHIMLAR:
  a) hajm va sanani oldindan belgilash, faqat oxirida qarash
  b) ketma-ket testlar: alfa ni qarashlarga taqsimlash
     (O'Brien-Fleming, Pocock), mSPRT / "har doim yaroqli" p-qiymatlar
  c) oddiy, lekin qo'pol: har qarashda alfa / qarashlar_soni (Bonferroni)
     -> yolg'on musbat nazoratda, lekin quvvat keskin tushadi

GUARDRAIL: asosiy metrika yaxshilandi, lekin nimaning hisobiga?
  konversiya +1.1 p.p., qaytarish ham +0.55 p.p. -> sof foyda?
  qoida: guardrail CI ning yomon cheti oldindan kelishilgan chegaradan oshmasin

REGRET = sum_t (eng yaxshi variant konversiyasi - tanlangan variant konversiyasi)
        "yomon tanlov tufayli yo'qotilgan konversiyalar"

EPSILON-GREEDY:
  ehtimol eps bilan - tasodifiy variant (o'rganish)
  aks holda - hozirgacha o'rtachasi eng yuqori variant (foydalanish)
  oddiy; lekin eps doimiy -> yomon variantlarga abadiy eps/K trafik

THOMPSON SAMPLING (Beta-Bernoulli):
  har variant uchun posterior: Beta(1 + muvaffaqiyat, 1 + muvaffaqiyatsizlik)
  har qadamda har posteriordan bitta namuna -> eng kattasini tanlash
  noaniq variant ba'zan katta namuna beradi -> o'zi sinaladi
  ishonch oshgan sari o'rganish tabiiy kamayadi

CHEKLOVLAR:
  kuchsiz variantlar haqida ma'lumot kam -> ishonch intervali keng
  kechikkan mukofot (konversiya 7 kundan keyin) murakkablashtiradi
  NOSTATSIONAR dunyo: eski dalillar "unutilmaydi" -> sekin moslashadi
    yechim: diskont (n, s ni har qadamda gamma ga ko'paytirish) yoki oyna

     trigger (jadval / drift / sifat / yangi belgilar)
        |
        v
  [ma'lumot] -> validatsiya 27.3-bob -> versiya/xesh
        |
        v
  [o'qitish] -> eksperiment jurnali 27.4-bob -> kandidat artefakt
        |
        v
  [oflayn baholash] -> champion bilan juftlashgan, segmentlar
        |
        v
  [darvoza 1: oflayn] --RAD--> hisobot, champion qoladi
        | OK
        v
  [registr: Staging] 27.5-bob -> [shadow N kun]
        |
        v
  [darvoza 2: shadow] --RAD--> hisobot
        | OK
        v
  [canary 1% -> 10% -> 100%] --guardrail buzildi--> AVTOMATIK ROLLBACK
        | OK
        v
  [registr: Production], eski versiya Archived (o'chirilmaydi!)
        |
        v
  monitoring (27.11, 27.12) --> yangi trigger ...

VAQT CHIZIG'I (mijoz ketishi, belgi 1 oydan keyin):
  oy t:    bashoratlar beriladi, kirishlar loglanadi
  oy t+1:  t-oy belgilari keladi -> t-oy sifati ma'lum
  oy t+1:  qayta o'qitish (t-oygacha belgili ma'lumot bilan)
  oy t+2:  shadow (t+2 so'rovlarida), belgilar t+3 da
  => drift boshlanganidan yangi model production ga chiqquncha
     kamida 2-3 oy

TEZLASHTIRISH YO'LLARI:
  erta signal: kirish/bashorat drifti 27.12-bob -> tayyorgarlik
  proksi belgi: tezroq keladigan o'rinbosar (masalan, 7 kunlik
                faollik -> 30 kunlik ketish uchun)
  shadow ni tarixiy so'rovlarda ham o'tkazish (replay)

TRIGGER QOIDASI (misol, oldindan yoziladi):
  sifat: AUC(oxirgi belgili oy) < AUC(reference) - 0.03
  YOKI  muhim ustunda PSI > 0.25 ikki oy ketma-ket
  VA    yangi belgili qatorlar >= 5000 (o'qitishga yetarli)

import hashlib

import numpy as np
from scipy import stats


def guruh(user_id, tajriba, ulush_b=0.5):
    h = hashlib.sha256(f"{tajriba}:{user_id}".encode()).hexdigest()
    return "B" if int(h[:8], 16) / 16**8 < ulush_b else "A"


def namuna_hajmi(p1, p2, alfa=0.05, quvvat=0.8):
    za, zb = stats.norm.ppf(1 - alfa / 2), stats.norm.ppf(quvvat)
    pb = (p1 + p2) / 2
    return int(np.ceil((za * np.sqrt(2 * pb * (1 - pb))
                        + zb * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
                       / (p2 - p1) ** 2))


def z_test(x_a, n_a, x_b, n_b):
    pa, pb = x_a / n_a, x_b / n_b
    p = (x_a + x_b) / (n_a + n_b)
    z = (pb - pa) / np.sqrt(p * (1 - p) * (1 / n_a + 1 / n_b))
    se = np.sqrt(pa * (1 - pa) / n_a + pb * (1 - pb) / n_b)
    return pb - pa, 2 * stats.norm.sf(abs(z)), (pb - pa - 1.96 * se, pb - pa + 1.96 * se)


# vaznli o'quv oynasi: yosh - necha oy oldin
vazn = 0.5 ** (yosh / yarim_umr)
model.fit(X, y, sample_weight=vazn)

# juftlashgan shadow taqqoslash (bir xil so'rovlar)
d = log_loss_har_biri(y, p_challenger) - log_loss_har_biri(y, p_champion)
yaxshi = d.mean() < -2 * d.std(ddof=1) / np.sqrt(len(d))

# Thompson sampling (bitta qadam)
theta = rng.beta(1 + muvaffaqiyat, 1 + urinish - muvaffaqiyat)
tanlov = int(np.argmax(theta))

qayta o'qitish != ishga tushirish; darvoza majburiy
oyna - backtest bilan, drift turiga qarab
shadow: bir xil so'rovlar, juftlashgan; segmentlar
canary: PSI + nazorat namunalari; rollback avtomatik
A/B: xesh, quvvat, bir marta qarash, guardrail
bandit: regret kam, bilim kam; nostatsionarda diskont

"""O'quv oynasi: statik, butun tarix, oxirgi 3 oy, vaznli - drift turiga qarab."""

import warnings

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

OYLAR, OY_N, BOSH = 20, 800, 6          # 20 oy, har oyda 800 mijoz
W0 = np.array([1.2, -1.0, 0.6, 0.0])
W1 = np.array([0.2, 0.8, 0.6, 1.0])       # yangi rejim


def koef(ssenariy, t):
    if ssenariy == "barqaror":
        return W0
    if ssenariy == "to'satdan":
        return W0 if t < 10 else W1
    if ssenariy == "asta-sekin":
        a = min(max((t - 4) / 14, 0.0), 1.0)
        return (1 - a) * W0 + a * W1
    if ssenariy == "qaytuvchi":            # 4 oy eski, 4 oy yangi, ...
        return W0 if (t // 4) % 2 == 0 else W1
    raise ValueError(ssenariy)


def oy_partiyasi(rng, ssenariy, t):
    X = rng.normal(0, 1, (OY_N, 4))
    f = -0.8 + X @ koef(ssenariy, t)
    y = (rng.random(OY_N) < 1 / (1 + np.exp(-f))).astype(int)
    return X, y


def orgat(X, y, w=None):
    return LogisticRegression(max_iter=500).fit(X, y, sample_weight=w)


def yurish(ssenariy, seed):
    rng = np.random.default_rng(seed)
    data = [oy_partiyasi(rng, ssenariy, t) for t in range(OYLAR)]
    statik = orgat(np.vstack([d[0] for d in data[:BOSH]]),
                   np.concatenate([d[1] for d in data[:BOSH]]))
    natija = {k: [] for k in ["statik", "butun tarix", "oxirgi 3 oy",
                              "vaznli"]}
    for t in range(BOSH, OYLAR):             # t-oy uchun bashorat
        X_t, y_t = data[t]
        Xs = np.vstack([d[0] for d in data[:t]])
        ys = np.concatenate([d[1] for d in data[:t]])
        yosh = np.repeat(np.arange(t - 1, -1, -1), OY_N)   # necha oy oldin
        modellar = {
            "statik": statik,
            "butun tarix": orgat(Xs, ys),
            "oxirgi 3 oy": orgat(Xs[yosh < 3], ys[yosh < 3]),
            "vaznli": orgat(Xs, ys, w=0.5 ** (yosh / 3)),   # yarim umr 3 oy
        }
        for nom, m in modellar.items():
            natija[nom].append(log_loss(y_t, m.predict_proba(X_t)[:, 1]))
    return {k: np.array(v) for k, v in natija.items()}


def main() -> None:
    warnings.simplefilter("ignore")
    seedlar = [0, 1, 2]
    tartib = ["statik", "butun tarix", "oxirgi 3 oy", "vaznli"]  # soddadan
    print("=== 1. Sozlama ===")
    print(f"  {OYLAR} oy x {OY_N} mijoz; {BOSH}-oydan boshlab har oy qayta "
          f"o'qitiladi va KEYINGI oy baholanadi")
    print(f"  metrika: log loss (kichik - yaxshi); {len(seedlar)} urug' x "
          f"{OYLAR - BOSH} oy = {len(seedlar) * (OYLAR - BOSH)} juft")

    print("\n=== 2. O'rtacha log loss ===")
    print(f"  {'ssenariy':<11}" + "".join(f"{s:>13}" for s in tartib))
    hammasi = {}
    for ss in ["barqaror", "to'satdan", "asta-sekin", "qaytuvchi"]:
        yig = {k: [] for k in tartib}
        for s in seedlar:
            r = yurish(ss, s)
            for k in tartib:
                yig[k].append(r[k])
        hammasi[ss] = {k: np.concatenate(v) for k, v in yig.items()}
        print(f"  {ss:<11}" + "".join(
            f"{hammasi[ss][k].mean():>13.4f}" for k in tartib))

    print("\n=== 3. Juftlashgan farq eng yaxshisidan va qaror ===")
    for ss, r in hammasi.items():
        eng = min(tartib, key=lambda k: r[k].mean())
        tanlov = None
        qatorlar = []
        for k in tartib:
            d = r[k] - r[eng]
            se = d.std(ddof=1) / np.sqrt(len(d)) if k != eng else 0.0
            yomon = d.mean() > 2 * se and k != eng
            qatorlar.append(f"{k} {d.mean():+.4f}"
                            + (f" (SE {se:.4f})" if k != eng else ""))
            if tanlov is None and not yomon:
                tanlov = k
        print(f"  {ss}: eng yaxshi {eng}")
        for q in qatorlar:
            print(f"    {q}")
        print(f"    TANLOV: {tanlov}")
    print("  ⭐ Oyna drift turiga mos tanlanadi; barqarorda statik ham yetadi")


if __name__ == "__main__":
    main()

=== 1. Sozlama ===
  20 oy x 800 mijoz; 6-oydan boshlab har oy qayta o'qitiladi va KEYINGI oy baholanadi
  metrika: log loss (kichik - yaxshi); 3 urug' x 14 oy = 42 juft

=== 2. O'rtacha log loss ===
  ssenariy          statik  butun tarix  oxirgi 3 oy       vaznli
  barqaror          0.4751       0.4750       0.4759       0.4752
  to'satdan         0.8261       0.6506       0.5445       0.5716
  asta-sekin        0.7349       0.6112       0.5386       0.5586
  qaytuvchi         0.5863       0.5918       0.6289       0.5948

=== 3. Juftlashgan farq eng yaxshisidan va qaror ===
  barqaror: eng yaxshi butun tarix
    statik +0.0000 (SE 0.0001)
    butun tarix +0.0000
    oxirgi 3 oy +0.0008 (SE 0.0002)
    vaznli +0.0002 (SE 0.0001)
    TANLOV: statik
  to'satdan: eng yaxshi oxirgi 3 oy
    statik +0.2816 (SE 0.0339)
    butun tarix +0.1061 (SE 0.0138)
    oxirgi 3 oy +0.0000
    vaznli +0.0271 (SE 0.0052)
    TANLOV: oxirgi 3 oy
  asta-sekin: eng yaxshi oxirgi 3 oy
    statik +0.1963 (SE 0.0248)
    butun tarix +0.0726 (SE 0.0081)
    oxirgi 3 oy +0.0000
    vaznli +0.0200 (SE 0.0019)
    TANLOV: oxirgi 3 oy
  qaytuvchi: eng yaxshi statik
    statik +0.0000
    butun tarix +0.0055 (SE 0.0057)
    oxirgi 3 oy +0.0426 (SE 0.0290)
    vaznli +0.0084 (SE 0.0132)
    TANLOV: statik
  ⭐ Oyna drift turiga mos tanlanadi; barqarorda statik ham yetadi

"""Champion/challenger: shadow rejim, oldindan yozilgan darvoza, canary va rollback."""

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score

USTUNLAR = ["x1", "x2", "x3", "x4", "hudud"]
DARVOZA = {                                   # natijani ko'rishdan OLDIN
    "log loss sezilarli yaxshi": "mean(d) < -2*SE",
    "AUC pasaymadi": "AUC_ch >= AUC_champ - 0.005",
    "har hududda yomonlashuv <= 0.02": "mean(d_hudud) <= 0.02",
    "shadow xatolari <= 0.1%": "xato / jami <= 0.001",
}


def partiya(rng, n, rejim):
    X = rng.normal(0, 1, (n, 5))
    X[:, 4] = rng.integers(0, 3, n)                      # hudud 0/1/2
    b1 = 1.2 if rejim == "eski" else 0.2                 # drift: x1 ta'siri
    b3 = np.where(X[:, 4] == 2, -0.9, 0.6)               # 2-hududda teskari
    f = -0.8 + b1 * X[:, 0] - 0.9 * X[:, 1] + b3 * X[:, 2] + 0.7 * X[:, 3]
    y = (rng.random(n) < 1 / (1 + np.exp(-f))).astype(int)
    return X, y


def orgat(X, y):
    return HistGradientBoostingClassifier(
        max_iter=120, learning_rate=0.08, categorical_features=[4],
        random_state=0).fit(X, y)


class Model:
    """Model + kirish tayyorlash (versiyaga xos xatti-harakat bilan)."""

    def __init__(self, nom, clf, qatiy=False, tartib_xato=False):
        self.nom, self.clf = nom, clf
        self.qatiy, self.tartib_xato = qatiy, tartib_xato

    def bashorat(self, X):
        X = X.copy()
        if np.isnan(X).any():
            if self.qatiy:
                raise ValueError("bo'sh qiymat")
            X = np.nan_to_num(X, nan=0.0)                 # o'quvdagi median ~ 0
        if self.tartib_xato:
            X[:, [0, 1]] = X[:, [1, 0]]                   # ustun tartibi xatosi
        return self.clf.predict_proba(X)[:, 1]


def shadow(champion, challenger, X):
    """Foydalanuvchiga champion javobi; challenger faqat logga."""
    javob = champion.bashorat(X)
    log, xato = np.full(len(X), np.nan), 0
    for i in range(0, len(X), 100):                      # 100 talik partiya
        try:
            log[i:i + 100] = challenger.bashorat(X[i:i + 100])
        except ValueError:                               # partiya yiqildi ->
            for j in range(i, min(i + 100, len(X))):     # bittalab qayta
                try:
                    log[j] = challenger.bashorat(X[j:j + 1])[0]
                except ValueError:
                    xato += 1
    return javob, log, xato / len(X)


def ll(y, p):
    p = np.clip(p, 1e-6, 1 - 1e-6)
    return -(y * np.log(p) + (1 - y) * np.log(1 - p))


def psi(ref, cur, bins=10, eps=1e-4):
    ichki = np.unique(np.quantile(ref, np.linspace(0, 1, bins + 1)[1:-1]))
    k = len(ichki) + 1
    p = np.bincount(np.searchsorted(ichki, ref, side="right"),
                    minlength=k) / len(ref)
    q = np.bincount(np.searchsorted(ichki, cur, side="right"),
                    minlength=k) / len(cur)
    p, q = np.clip(p, eps, None), np.clip(q, eps, None)
    return float(np.sum((q - p) * np.log(q / p)))


def main() -> None:
    rng = np.random.default_rng(5)
    X_eski, y_eski = partiya(rng, 6000, "eski")
    X_yangi, y_yangi = partiya(rng, 4000, "yangi")        # drift dan keyin
    champion = Model("v1", orgat(X_eski, y_eski))
    ch_a = Model("A", orgat(X_yangi, y_yangi))
    bor = X_yangi[:, 4] != 2                             # quvur 2-hududni tashladi
    ch_b = Model("B", orgat(X_yangi[bor], y_yangi[bor]), qatiy=True)

    print("=== 1. Shadow rejim: 3000 so'rov (keyingi oy) ===")
    X_s, y_s = partiya(rng, 3000, "yangi")
    X_s[rng.choice(3000, 6, replace=False), 3] = np.nan  # 6 ta buzuq so'rov
    loglar = {}
    for ch in [ch_a, ch_b]:
        javob, log, xato = shadow(champion, ch, X_s)
        loglar[ch.nom] = (log, xato)
        bir_xil = np.allclose(javob, champion.bashorat(X_s))
        print(f"  {ch.nom}: foydalanuvchi javobi champion bilan bir xil: "
              f"{bir_xil}; challenger xato so'rovlari {xato:.2%}")

    print("\n=== 2. Belgilar keldi: juftlashgan taqqoslash ===")
    p_ch = champion.bashorat(X_s)
    print(f"  champion {champion.nom}: AUC {roc_auc_score(y_s, p_ch):.4f}")
    qarorlar = {}
    for nom, (log, xato) in loglar.items():
        ok = ~np.isnan(log)
        d = ll(y_s[ok], log[ok]) - ll(y_s[ok], p_ch[ok])
        se = d.std(ddof=1) / np.sqrt(len(d))
        auc_c = roc_auc_score(y_s[ok], log[ok])
        auc_0 = roc_auc_score(y_s[ok], p_ch[ok])
        hudud_d = [float(d[X_s[ok, 4] == h].mean()) for h in range(3)]
        print(f"  {nom}: d = {d.mean():+.4f} (SE {se:.4f}), AUC {auc_c:.4f}, "
              f"hududlar {[round(v, 4) for v in hudud_d]}")
        qarorlar[nom] = {
            "log loss sezilarli yaxshi": d.mean() < -2 * se,
            "AUC pasaymadi": auc_c >= auc_0 - 0.005,
            "har hududda yomonlashuv <= 0.02": max(hudud_d) <= 0.02,
            "shadow xatolari <= 0.1%": xato <= 0.001,
        }

    print("\n=== 3. Darvoza (qoidalar oldindan yozilgan) ===")
    for qoida in DARVOZA:
        print(f"  {qoida:<34} " + "  ".join(
            f"{n}: {'OK' if q[qoida] else 'RAD'}" for n, q in qarorlar.items()))
    otdi = [n for n, q in qarorlar.items() if all(q.values())]
    print(f"  darvozadan o'tdi: {otdi}")

    print("\n=== 4. Registr, canary (10% trafik) va avtomatik rollback ===")
    modellar = {"v1": champion, "v2": Model("v2", ch_a.clf),
                "v3": Model("v3", ch_a.clf, tartib_xato=True)}  # tezlik patchi
    shadow_ref = {"v2": loglar["A"][0], "v3": loglar["A"][0]}   # o'sha model
    registr = {"production": "v1", "tarix": ["v1"], "rad": []}
    for yangi in ["v2", "v3"]:
        eski = registr["production"]
        X_c, _ = partiya(rng, 1000, "yangi")                    # canary kuni
        p_c = modellar[yangi].bashorat(X_c)
        ref = shadow_ref[yangi][~np.isnan(shadow_ref[yangi])]
        k = psi(ref, p_c)
        nazorat = ~np.isnan(shadow_ref[yangi])
        nazorat[200:] = False                                   # 200 ta namuna
        farq = np.max(np.abs(modellar[yangi].bashorat(X_s[nazorat])
                             - shadow_ref[yangi][nazorat]))
        buzildi = k > 0.1 or farq > 1e-9
        print(f"  {eski} -> {yangi}: bashorat PSI (o'z shadow iga nisbatan) "
              f"{k:.3f}; nazorat namunalari maks farq {farq:.3g}")
        if buzildi:
            registr["rad"].append(yangi)
            sabab = [n for n, b in [("PSI", k > 0.1),
                                    ("nazorat namunalari", farq > 1e-9)] if b]
            print(f"    guardrail BUZILDI ({', '.join(sabab)}) -> avtomatik "
                  f"ROLLBACK, production = {eski}")
        else:
            registr["production"] = yangi
            registr["tarix"].append(yangi)
            print(f"    guardrail OK -> 100% trafik, production = {yangi}")
    print(f"  yakuniy: production {registr['production']}, "
          f"tarix {registr['tarix']}, rad {registr['rad']}")
    print("  ⭐ Qaror - oldindan yozilgan qoidalar bilan, rollback - avtomatik")


if __name__ == "__main__":
    main()

=== 1. Shadow rejim: 3000 so'rov (keyingi oy) ===
  A: foydalanuvchi javobi champion bilan bir xil: True; challenger xato so'rovlari 0.00%
  B: foydalanuvchi javobi champion bilan bir xil: True; challenger xato so'rovlari 0.20%

=== 2. Belgilar keldi: juftlashgan taqqoslash ===
  champion v1: AUC 0.7292
  A: d = -0.0711 (SE 0.0113), AUC 0.7593, hududlar [-0.0637, -0.0719, -0.0776]
  B: d = +0.0289 (SE 0.0147), AUC 0.7021, hududlar [-0.0333, -0.0417, 0.162]

=== 3. Darvoza (qoidalar oldindan yozilgan) ===
  log loss sezilarli yaxshi          A: OK  B: RAD
  AUC pasaymadi                      A: OK  B: RAD
  har hududda yomonlashuv <= 0.02    A: OK  B: RAD
  shadow xatolari <= 0.1%            A: OK  B: RAD
  darvozadan o'tdi: ['A']

=== 4. Registr, canary (10% trafik) va avtomatik rollback ===
  v1 -> v2: bashorat PSI (o'z shadow iga nisbatan) 0.009; nazorat namunalari maks farq 0
    guardrail OK -> 100% trafik, production = v2
  v2 -> v3: bashorat PSI (o'z shadow iga nisbatan) 0.016; nazorat namunalari maks farq 0.872
    guardrail BUZILDI (nazorat namunalari) -> avtomatik ROLLBACK, production = v2
  yakuniy: production v2, tarix ['v1', 'v2'], rad ['v3']
  ⭐ Qaror - oldindan yozilgan qoidalar bilan, rollback - avtomatik

"""Onlayn A/B test: xesh bilan taqsimlash, quvvat, test, peeking va guardrail."""

import hashlib

import numpy as np
from scipy import stats


def guruh(user_id, tajriba, ulush_b=0.5):
    """Barqaror taqsimlash: bir foydalanuvchi har doim bir guruhda."""
    h = hashlib.sha256(f"{tajriba}:{user_id}".encode()).hexdigest()
    return "B" if int(h[:8], 16) / 16**8 < ulush_b else "A"


def namuna_hajmi(p1, p2, alfa=0.05, quvvat=0.8):
    """Ikki proporsiya, ikki tomonlama test - har guruhga 11.2-bob."""
    za, zb = stats.norm.ppf(1 - alfa / 2), stats.norm.ppf(quvvat)
    p_bar = (p1 + p2) / 2
    n = (za * np.sqrt(2 * p_bar * (1 - p_bar))
         + zb * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p2 - p1) ** 2
    return int(np.ceil(n))


def z_test(x_a, n_a, x_b, n_b):
    """Proporsiyalar farqi: p-qiymat va 95% ishonch intervali 11.4-bob."""
    pa, pb = x_a / n_a, x_b / n_b
    p = (x_a + x_b) / (n_a + n_b)
    z = (pb - pa) / np.sqrt(p * (1 - p) * (1 / n_a + 1 / n_b))
    se = np.sqrt(pa * (1 - pa) / n_a + pb * (1 - pb) / n_b)
    return pb - pa, 2 * stats.norm.sf(abs(z)), (pb - pa - 1.96 * se,
                                                pb - pa + 1.96 * se)


def main() -> None:
    rng = np.random.default_rng(11)

    print("=== 1. Xesh bilan taqsimlash ===")
    idlar = [f"u{i}" for i in range(20000)]
    g1 = [guruh(u, "tajriba_tugma") for u in idlar]
    g2 = [guruh(u, "tajriba_narx") for u in idlar]
    print(f"  B ulushi: {np.mean([g == 'B' for g in g1]):.4f}")
    print(f"  barqarorlik (qayta hisoblash bir xil): "
          f"{all(guruh(u, 'tajriba_tugma') == g for u, g in zip(idlar[:500], g1))}")
    for a in "AB":
        for b in "AB":
            ulush = np.mean([(x == a) and (y == b) for x, y in zip(g1, g2)])
            print(f"  tugma={a}, narx={b}: {ulush:.4f}")
    print("  ikki tajriba mustaqil (har katak ~0.25) - boshqa tuz (salt)")

    print("\n=== 2. Quvvat tahlili: konversiya 10% -> 11% ===")
    n = namuna_hajmi(0.10, 0.11)
    print(f"  har guruhga n = {n} (alfa 0.05, quvvat 0.8)")
    print(f"  kuniga 2000 foydalanuvchi -> {int(np.ceil(2 * n / 2000))} kun")
    for mde in [0.005, 0.02]:
        print(f"  MDE {mde:.3f} bo'lsa: n = {namuna_hajmi(0.10, 0.10 + mde)}")

    print("\n=== 3. Bitta tajriba (haqiqiy farq +1 p.p.) ===")
    x_a, x_b = rng.binomial(n, 0.10), rng.binomial(n, 0.11)
    farq, p, (q, y) = z_test(x_a, n, x_b, n)
    print(f"  A {x_a / n:.4f}, B {x_b / n:.4f}; farq {farq:+.4f}, "
          f"95% CI [{q:+.4f}, {y:+.4f}], p = {p:.4f}")
    print(f"  qaror: {'B ni chiqarish' if p < 0.05 and farq > 0 else 'farq isbotlanmadi'}")
    T = 4000
    xa, xb = rng.binomial(n, 0.10, T), rng.binomial(n, 0.11, T)
    xa0 = rng.binomial(n, 0.10, T)
    kuch = np.mean([z_test(a, n, b, n)[1] < 0.05 for a, b in zip(xa, xb)])
    alfa = np.mean([z_test(a, n, b, n)[1] < 0.05 for a, b in zip(xa, xa0)])
    print(f"  {T} simulyatsiya: quvvat {kuch:.3f}, A/A da yolg'on musbat "
          f"{alfa:.3f}")

    print("\n=== 4. Peeking: har kuni qarab p<0.05 da to'xtatish ===")
    kun, kunlik, T = 30, 500, 4000                       # har guruhga kuniga
    nn = kunlik * np.arange(1, kun + 1)

    def kunlik_p(p_b):
        a = rng.binomial(kunlik, 0.10, (T, kun)).cumsum(1)
        b = rng.binomial(kunlik, p_b, (T, kun)).cumsum(1)
        pool = (a + b) / (2 * nn)
        z = (b - a) / nn / np.sqrt(pool * (1 - pool) * 2 / nn)
        return 2 * stats.norm.sf(np.abs(z))

    p_aa, p_ab = kunlik_p(0.10), kunlik_p(0.11)          # A/A va haqiqiy +1pp
    usullar = [("faqat 30-kuni (oldindan belgilangan)",
                lambda p: p[:, -1] < 0.05)]
    for d in [7, 14, 30]:
        usullar.append((f"dastlabki {d} kun har kuni qarash",
                        lambda p, d=d: (p[:, :d] < 0.05).any(1)))
    usullar.append(("har kuni, lekin alfa/30 (Bonferroni)",
                    lambda p: (p < 0.05 / 30).any(1)))
    print(f"  {'usul':<37} {'yolg_on musbat':>14} {'quvvat':>7}")
    for nom, f in usullar:
        print(f"  {nom:<37} {np.mean(f(p_aa)):>14.3f} "
              f"{np.mean(f(p_ab)):>7.3f}")

    print("\n=== 5. Guardrail: asosiy metrika yaxshi - qaytarishlar-chi? ===")
    n5 = 30000
    kon_a, kon_b = rng.binomial(n5, 0.100), rng.binomial(n5, 0.108)
    qay_a, qay_b = rng.binomial(n5, 0.020), rng.binomial(n5, 0.026)
    f1, p1, ci1 = z_test(kon_a, n5, kon_b, n5)
    f2, p2, ci2 = z_test(qay_a, n5, qay_b, n5)
    print(f"  konversiya (asosiy):  {f1:+.4f}, p={p1:.4f}, "
          f"CI [{ci1[0]:+.4f}, {ci1[1]:+.4f}]")
    print(f"  qaytarish (guardrail): {f2:+.4f}, p={p2:.4f}, "
          f"CI [{ci2[0]:+.4f}, {ci2[1]:+.4f}]")
    chegara = 0.002                                     # oldindan kelishilgan
    asosiy_ok = p1 < 0.05 and f1 > 0
    guard_ok = ci2[1] <= chegara                        # yomonlashuv <= 0.2 p.p.
    print(f"  asosiy OK: {asosiy_ok}; guardrail (CI yuqori <= +{chegara}) OK: "
          f"{guard_ok}")
    print(f"  QAROR: {'chiqarish' if asosiy_ok and guard_ok else 'CHIQARMASLIK'}")
    print("  ⭐ Hajm oldindan, qarash bir marta, guardrail - majburiy")


if __name__ == "__main__":
    main()

=== 1. Xesh bilan taqsimlash ===
  B ulushi: 0.4931
  barqarorlik (qayta hisoblash bir xil): True
  tugma=A, narx=A: 0.2510
  tugma=A, narx=B: 0.2558
  tugma=B, narx=A: 0.2477
  tugma=B, narx=B: 0.2455
  ikki tajriba mustaqil (har katak ~0.25) - boshqa tuz (salt)

=== 2. Quvvat tahlili: konversiya 10% -> 11% ===
  har guruhga n = 14751 (alfa 0.05, quvvat 0.8)
  kuniga 2000 foydalanuvchi -> 15 kun
  MDE 0.005 bo'lsa: n = 57763
  MDE 0.020 bo'lsa: n = 3841

=== 3. Bitta tajriba (haqiqiy farq +1 p.p.) ===
  A 0.0984, B 0.1142; farq +0.0159, 95% CI [+0.0088, +0.0229], p = 0.0000
  qaror: B ni chiqarish
  4000 simulyatsiya: quvvat 0.809, A/A da yolg'on musbat 0.054

=== 4. Peeking: har kuni qarab p<0.05 da to'xtatish ===
  usul                                  yolg_on musbat  quvvat
  faqat 30-kuni (oldindan belgilangan)           0.057   0.809
  dastlabki 7 kun har kuni qarash                0.172   0.404
  dastlabki 14 kun har kuni qarash               0.231   0.640
  dastlabki 30 kun har kuni qarash               0.293   0.892
  har kuni, lekin alfa/30 (Bonferroni)           0.017   0.481

=== 5. Guardrail: asosiy metrika yaxshi - qaytarishlar-chi? ===
  konversiya (asosiy):  +0.0110, p=0.0000, CI [+0.0061, +0.0158]
  qaytarish (guardrail): +0.0055, p=0.0000, CI [+0.0031, +0.0079]
  asosiy OK: True; guardrail (CI yuqori <= +0.002) OK: False
  QAROR: CHIQARMASLIK
  ⭐ Hajm oldindan, qarash bir marta, guardrail - majburiy

"""Ko'p qo'lli bandit: teng taqsimot (A/B/n), epsilon-greedy, Thompson sampling."""

import numpy as np

HAQIQIY = np.array([0.050, 0.055, 0.070])    # 3 variant konversiyasi (noma'lum)
QADAM, SEEDLAR = 20000, 200
STRATEGIYALAR = ["teng (A/B/n)", "epsilon-greedy", "Thompson",
                 "Thompson unutuvchi"]


def simulyatsiya(strategiya, seed, eps=0.1, keyin=None):
    """Hamma urug'lar parallel: har qadamda har urug' bitta variant tanlaydi."""
    rng = np.random.default_rng(seed)
    K, S = len(HAQIQIY), SEEDLAR
    n = np.zeros((S, K))
    s = np.zeros((S, K))
    regret = np.zeros(S)
    tanlov_soni = np.zeros((S, K))
    satr = np.arange(S)
    haqiqiy = HAQIQIY
    for t in range(QADAM):
        if keyin is not None and t == QADAM // 2:
            haqiqiy = keyin                                     # dunyo o'zgardi
        if strategiya == "teng (A/B/n)":
            a = np.full(S, t % K)
        elif strategiya == "epsilon-greedy":
            ortacha = np.where(n > 0, s / np.maximum(n, 1), 1.0)  # avval sinash
            a = np.argmax(ortacha + 1e-9 * rng.random((S, K)), axis=1)
            tasodif = rng.random(S) < eps
            a[tasodif] = rng.integers(0, K, tasodif.sum())
        else:                                                   # Thompson
            theta = rng.beta(1 + s, 1 + n - s)
            a = np.argmax(theta, axis=1)
        if strategiya == "Thompson unutuvchi":
            n *= 0.9998                                         # eski dalil
            s *= 0.9998                                         # so'nadi
        mukofot = rng.random(S) < haqiqiy[a]
        n[satr, a] += 1
        s[satr, a] += mukofot
        tanlov_soni[satr, a] += 1
        regret += haqiqiy.max() - haqiqiy[a]
    topdi = np.argmax(s / np.maximum(n, 1), axis=1) == np.argmax(haqiqiy)
    return regret, tanlov_soni / QADAM, topdi


def main() -> None:
    print("=== 1. Sozlama ===")
    print(f"  haqiqiy konversiyalar: {HAQIQIY.tolist()} (algoritmga noma'lum)")
    print(f"  {QADAM} tashrif, {SEEDLAR} takror; regret = yo'qotilgan "
          f"konversiyalar (eng yaxshisiga nisbatan)")

    print("\n=== 2. Natijalar ===")
    natija = {}
    print(f"  {'strategiya':<18} {'regret':>6} {'eng yaxshiga trafik':>20} "
          f"{'eng yaxshini topdi':>19}")
    for nom in STRATEGIYALAR:
        r, ulush, topdi = simulyatsiya(nom, seed=1)
        natija[nom] = r
        print(f"  {nom:<18} {r.mean():>6.1f} {ulush[:, -1].mean():>20.3f} "
              f"{topdi.mean():>19.3f}")

    print("\n=== 3. Juftlashgan farq (bir xil urug' indekslari) ===")
    for a, b in [("epsilon-greedy", "teng (A/B/n)"),
                 ("Thompson", "teng (A/B/n)"),
                 ("Thompson", "epsilon-greedy"),
                 ("Thompson unutuvchi", "Thompson")]:
        d = natija[a] - natija[b]
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  {a} - {b}: {d.mean():+.1f} (SE {se:.1f}), "
              f"sezilarli: {abs(d.mean()) > 2 * se}")
    eng = min(natija, key=lambda k: natija[k].mean())
    print(f"  eng kam regret: {eng}")

    print("\n=== 4. Nostatsionar dunyo: 10000-qadamda 1-variant 0.080 bo'ldi ===")
    keyin = np.array([0.080, 0.055, 0.070])
    print(f"  {'strategiya':<18} {'regret':>6} {'yangi eng yaxshini topdi':>25}")
    ns = {}
    for nom in STRATEGIYALAR:
        r, _, topdi = simulyatsiya(nom, seed=2, keyin=keyin)
        ns[nom] = r
        print(f"  {nom:<18} {r.mean():>6.1f} {topdi.mean():>25.3f}")
    d = ns["Thompson unutuvchi"] - ns["Thompson"]
    se = d.std(ddof=1) / np.sqrt(len(d))
    print(f"  unutuvchi - Thompson: {d.mean():+.1f} (SE {se:.1f}), "
          f"sezilarli: {abs(d.mean()) > 2 * se}")
    print("  oddiy bandit eski dalilni unutmaydi -> drift da sekin moslashadi")

    print("\n=== 5. Qachon qaysi biri ===")
    tejam = natija["teng (A/B/n)"].mean() - natija[eng].mean()
    print(f"  {eng} teng taqsimotga nisbatan ~{tejam:.0f} konversiya tejadi")
    print("  lekin: bandit kuchsiz variantlarga kam trafik beradi ->")
    print("  ular haqida ishonch intervali keng, 'nega yutdi' tahlili qiyin")
    print("  ⭐ Bilim kerak - A/B test; daromad kerak - bandit")


if __name__ == "__main__":
    main()

=== 1. Sozlama ===
  haqiqiy konversiyalar: [0.05, 0.055, 0.07] (algoritmga noma'lum)
  20000 tashrif, 200 takror; regret = yo'qotilgan konversiyalar (eng yaxshisiga nisbatan)

=== 2. Natijalar ===
  strategiya         regret  eng yaxshiga trafik  eng yaxshini topdi
  teng (A/B/n)        233.3                0.333               1.000
  epsilon-greedy       62.5                0.815               0.965
  Thompson             39.6                0.882               0.995
  Thompson unutuvchi   74.9                0.780               0.985

=== 3. Juftlashgan farq (bir xil urug' indekslari) ===
  epsilon-greedy - teng (A/B/n): -170.9 (SE 4.9), sezilarli: True
  Thompson - teng (A/B/n): -193.7 (SE 1.9), sezilarli: True
  Thompson - epsilon-greedy: -22.9 (SE 5.4), sezilarli: True
  Thompson unutuvchi - Thompson: +35.3 (SE 2.3), sezilarli: True
  eng kam regret: Thompson

=== 4. Nostatsionar dunyo: 10000-qadamda 1-variant 0.080 bo'ldi ===
  strategiya         regret  yangi eng yaxshini topdi
  teng (A/B/n)        233.4                     0.130
  epsilon-greedy      143.2                     0.145
  Thompson            120.5                     0.325
  Thompson unutuvchi  100.4                     0.950
  unutuvchi - Thompson: -20.1 (SE 3.0), sezilarli: True
  oddiy bandit eski dalilni unutmaydi -> drift da sekin moslashadi

=== 5. Qachon qaysi biri ===
  Thompson teng taqsimotga nisbatan ~194 konversiya tejadi
  lekin: bandit kuchsiz variantlarga kam trafik beradi ->
  ular haqida ishonch intervali keng, 'nega yutdi' tahlili qiyin
  ⭐ Bilim kerak - A/B test; daromad kerak - bandit

model = orgat(yangi_malumot); registr.production = model            # ⚠️
kandidat = orgat(yangi_malumot)
if darvoza(kandidat, champion, shadow_log): registr.promote(kandidat)  # ✅

model = orgat(hamma_oylar); baho = metrika(model, oy[t])            # ⚠️
model = orgat(oylar[:t]);   baho = metrika(model, oy[t])            # ✅

ok = d.mean() < -2 * se                                             # ⚠️
ok = d.mean() < -2 * se and max(d[seg == s].mean() for s in segmentlar) <= 0.02  # ✅

guruh = "B" if random.random() < 0.5 else "A"          # ⚠️ har so'rovda boshqa
guruh = "B" if hash(user_id) % 2 else "A"              # ⚠️ jarayonga bog'liq
guruh = xesh_guruh(user_id, "tajriba_nomi")            # ✅ sha256 + tuz

if p_qiymat(bugun) < 0.05: toxtat_va_chiqar()          # ⚠️ har kuni
if kun == REJA_KUNI: qaror(p_qiymat(kun))              # ✅ bir marta

if p_konversiya < 0.05: chiqar()                       # ⚠️
if p_konversiya < 0.05 and qaytarish_ci[1] <= CHEGARA: chiqar()   # ✅

os.remove("modellar/v1.joblib")                        # ⚠️ rollback imkonsiz
registr.arxivla("v1")                                  # ✅ saqlanadi

1.  # barqaror dunyoda statik va qayta o'qitilgan model?
2.  # to'satdan driftda qaysi oyna yaxshi?
3.  # qaytuvchi rejimda qisqa oyna?
4.  # shadow rejimda foydalanuvchi qaysi javobni oladi?
5.  # darvoza qoidalari qachon yoziladi?
6.  # ustunlar almashsa bashorat PSI?
7.  # A/B uchun Python hash() yaraydimi?
8.  # MDE ikki marta kichraysa n?
9.  # 30 kun har kuni qarash - yolg'on musbat?
10. # asosiy metrika sezilarli, guardrail yomon - qaror?
11. # Thompson vs teng taqsimot - regret?
12. # nostatsionar dunyoda oddiy bandit?

1.  model = orgat(yangi); registr.production = model

2.  guruh = "B" if hash(user_id) % 2 else "A"

3.  if p_qiymat(bugun) < 0.05: toxtat()

4.  ok = d.mean() < -2 * se          # faqat umumiy

5.  os.remove("modellar/v1.joblib")