IlmHamroh
Data Science va sun'iy intellekt/MLOps va deploy14/14-dars50 daqiqa
Mundarija (22)

27.14-dars: Amaliyot — mijoz ketishini bashorat qilish xizmati

27-QISM — MLOPS VA DEPLOY · 14-dars


1. Kirish va motivatsiya

27-qismda modelni noutbukdan ishlab chiqarishga olib chiqishning barcha bo'laklarini ko'rdik: MLOps nima va nega kerak, reproduksiya va muhit, ma'lumot quvuri, eksperiment kuzatuvi, model versiyalash va registr, paketlash, FastAPI bilan API, uni mustahkamlash va testlash, Docker, bulut va deploy strategiyalari, monitoring, drift aniqlash, qayta o'qitish va A/B test. Endi ularni bitta tizimga yig'amiz va eng muhim savolga javob beramiz: bu bo'laklar birgalikda ishlaganda, dunyo o'zgarganini tizim o'zi sezadimi va o'zini xavfsiz yangilay oladimi?

Vazifa — telekom kompaniyasi uchun mijoz ketishini bashorat qilish xizmati. Har oy yangi mijozlar kesimi keladi: oylik to'lov, xizmat muddati, qo'ng'iroqlar, shikoyatlar, tarif va hudud. Mijoz ketdimi — bu belgi bir oy kechikib keladi. Xizmat har oy barcha mijozlar uchun ketish ehtimolini beradi, saqlash bo'limi eng xavfli mijozlarga taklif yuboradi.

Loyiha to'rt qadamdan iborat, har biri alohida misol:

  1. Ma'lumot manbai va quvur — oylik partiyalar, validatsiya, karantin, idempotent va atomik yuklash sqlite ga.
  2. O'qitish va registr — vaqt bo'yicha bo'lish, ikki nomzod, eksperiment jurnali, juftlashgan taqqoslash, paket, darvoza va Production.
  3. Xizmat — FastAPI: /predict, /health, /version, validatsiya, API kaliti muhit o'zgaruvchisidan, so'rov loglari.
  4. Hayot sikli — keyingi 8 oy: drift, kechikkan belgilar, sifat tushishi, trigger, challenger shadow rejimda, darvoza, registrda versiya almashuvi, xizmat yangi versiyani yuklaydi, rollback imkoniyati.

Ma'lumotda 9-oydan boshlab dunyo o'zgaradi: raqobatchi arzon tarif chiqaradi (endi A tarifi mijozlari ketadi, avval C tarifi ketardi), narxlar oshadi, shikoyatlar ko'payadi va ularning ta'siri kuchayadi. Bu — kovariat, konsept va belgilar driftining aralashmasi, xuddi haqiqiy hayotdagidek. Tizim buni oldindan bilmaydi — barcha qaror qoidalari 4-qadam boshida, natijalarni ko'rishdan oldin yoziladi.

Real vaziyat. Telekom kompaniyasi ketish modelini bir marta o'qitib, ishga tushirdi va unutdi. Model kvartal hisobotlarida "AUC 0.73" deb yozilaverdi — bu raqam o'qitish paytidagi validatsiyadan olingan edi. Yarim yildan keyin saqlash bo'limi taklif yuborilgan mijozlarning deyarli hech biri ketmoqchi emasligini, ketganlar esa taklif olmaganini payqadi. Model haqiqiy AUC ni hech kim o'lchamagan: belgilar boshqa tizimda edi, bashoratlar loglanmagan edi, qayta o'qitish uchun quvur yo'q edi. Bu darsdagi tizimda xuddi shu drift 11-oyda trigger bilan ushlanadi va 12-oyda yangi model ishga tushadi.

Bu darsda to'liq ML xizmatini quramiz: xom partiyadan tortib o'zini kuzatadigan va xavfsiz yangilanadigan xizmatgacha.

Bu darsda:

  • Loyiha xaritasi va qaror qoidalari (oldindan)
  • Oylik partiyalar, validatsiya, karantin, idempotent va atomik yuklash
  • Vaqt bo'yicha bo'lish, eksperiment jurnali, juftlashgan taqqoslash
  • Paket: Pipeline + sxema + metadata + reference profil + nazorat namunalari
  • Registr: bosqichlar, ruxsat etilgan o'tishlar, audit
  • FastAPI xizmati: validatsiya, API kaliti, so'rov loglari, /version
  • Monitoring: PSI, kechikkan belgilar, sifat, trigger
  • Shadow, darvoza, versiya almashuvi va rollback
  • Dockerfile va CI/CT konfiguratsiyasi
  • Tuzoqlar

ℹ Misollar real numpy/pandas/sklearn/FastAPI bilan (Python 3.14). Hammasi bitta vaqtinchalik papkada va bitta jarayon ichida: server ishga tushirilmaydi, xizmat TestClient orqali sinaladi.


2. Nazariya — chuqur tushuntirish

2.1. Loyiha xaritasi

text
MANBA: oylik partiyalar (CSV) ----------------------------- (1-misol)
  |
  v
QUVUR: validatsiya (sxema, diapazon, toifalar, takror, hajm)
  |-- xato -> KARANTIN (fayl saqlanadi, sabab yoziladi)
  v
OMBOR (sqlite): idempotent (xesh bo'yicha), atomik (tranzaksiya)
  |
  v
O'QITISH: vaqt bo'yicha bo'lish (1-5 oy o'quv, 6-oy validatsiya)  (2-misol)
  nomzodlar: LogReg < HistGB (soddalik tartibida)
  EKSPERIMENT JURNALI (sqlite): yurish_id = xesh(model, param, ma'lumot, kod)
  juftlashgan bootstrap -> eng sodda munosib model
  |
  v
PAKET (joblib): Pipeline + sxema + metadata + profil + nazorat namunalari
  |
  v
REGISTR: None -> Staging -> [darvoza] -> Production; audit
  |
  v
XIZMAT (FastAPI): /predict /health /version /admin/yangila     (3-misol)
  kalit env dan, Pydantic validatsiya, har so'rov LOG ga (versiya bilan)
  |
  v
MONITORING (har oy):                                             (4-misol)
  kirish PSI + bashorat PSI  -> erta ogohlantirish
  (oy-1) belgilari keldi -> haqiqiy AUC -> TRIGGER qoidasi
  challenger (oxirgi 2 oy) -> Staging -> SHADOW -> darvoza
  -> Production, eski -> Archived -> xizmat yangilanadi -> /version
  rollback: Archived -> Production (bitta amal, auditda)

Har bosqich — alohida, sinaladigan komponent; tizim ular orasidagi shartnomalar (sxema, paket formati, registr bosqichlari) bilan birlashadi.

2.2. Ma'lumot quvuri

text
VALIDATSIYA (partiya darajasida - hammasi yoki hech narsa):
  sxema:     ustunlar ro'yxati va tartibi aynan mos
  hajm:      >= 500 qator
  bo'shlar:  har ustunda <= 1%
  takror:    mijoz_id noyob
  diapazon:  oylik_tolov 1..2000 (ming so'm), muddat_oy >= 0
  toifalar:  tarif in {A,B,C}, hudud in {4 ta}
  belgi:     ketdi in {0,1}

YUKLASH:
  xesh = sha256(kanonik CSV: saralangan, qat'iy format)
  xesh o'zgarmagan -> o'tkazib yuboriladi      (IDEMPOTENT)
  xesh yangi -> BEGIN; DELETE oy; INSERT; jurnal; COMMIT   (ATOMIK)
  uzilish -> ROLLBACK: eski ma'lumot butun qoladi

Validatsiyadan o'tmagan partiya o'chirilmaydi — u karantin papkasiga ko'chiriladi, sababi yoziladi. Manba tuzatilib qayta yetkazilsa, quvur uni oddiy partiya kabi yuklaydi. Birlik xatosi (so'm o'rniga ming so'm) — eng xavfli xato turlaridan biri: sxema to'g'ri, turlar to'g'ri, faqat qiymatlar 1000 marta katta. Uni faqat diapazon tekshiruvi ushlaydi. Model esa bunday partiyada jim ishlayveradi — va hamma mijozni "juda qimmat to'laydi, ketadi" deb baholaydi.

2.3. O'qitish, eksperiment jurnali va paket

text
BO'LISH - VAQT BO'YICHA (tasodifiy emas!):
  o'quv 1-5 oy, validatsiya 6-oy   ("kelajakni" bashorat qilish sinovi)
  yakuniy model 1-6 oyda qayta o'qitiladi (xuddi shu konfig)

EKSPERIMENT JURNALI (MLflow g'oyasi, sqlite bilan):
  yurish_id = sha256(model, parametrlar, ma'lumot xeshi, kod versiyasi)[:10]
  bir xil yurish qayta yozilsa - TAKRORLANMAYDI (INSERT OR REPLACE)

TAQQOSLASH: 6-oyda juftlashgan bootstrap (300 marta)
  farq = AUC_HistGB - AUC_LogReg,  SE = bootstrap std
  QOIDA: eng yaxshisidan sezilarli yomon bo'lmagan ENG SODDA

PAKET (joblib):
  format      - versiya, yuklashda tekshiriladi
  pipeline    - tayyorlash + model BITTA obyekt (train/serve farqi yo'q)
  sxema       - ustunlar va ruxsat etilgan toifalar
  metadata    - oylar, ma'lumot xeshi, kod versiyasi, val_auc
  profil      - reference taqsimotlar (monitoring uchun, 4-misol)
  nazorat     - 200 ta kirish + kutilgan chiqish (yuklashda tekshiriladi)

Haqiqiy loyihada jurnal va registr uchun MLflow, ma'lumot versiyasi uchun DVC ishlatiladi — bu darsda xuddi shu g'oyalar bir necha o'nlab qator sqlite kodi bilan ko'rsatiladi, shunda ichida nima bo'layotgani ko'rinadi.

2.4. Xizmat

text
ENDPOINTLAR:
  GET  /health         -> {"holat": "ok", "model_yuklangan": true}
  GET  /version        -> {"nom", "versiya", ...}   (qaysi model javob beryapti)
  POST /predict        -> X-API-Key sarlavhasi + {"mijozlar": [...]}
  POST /admin/yangila  -> registrdan Production ni qayta o'qish (kalit bilan)

HIMOYA:
  kalit: os.environ["CHURN_API_KEY"] - kodda, image da, gitda YO'Q
         o'rnatilmagan bo'lsa - xizmat ISHGA TUSHMAYDI
         solishtirish: hmac.compare_digest (vaqt bo'yicha sizib chiqmaydi)
         loglarda: faqat "********"
  kirish: Pydantic, extra="forbid", diapazonlar, Literal toifalar -> 422
  yuklash: paket nazorat namunalari mos kelmasa - yuklanmaydi

SO'ROV LOGI (monitoringning xom ashyosi):
  oy, mijoz_id, versiya, p, shadow_versiya, shadow_p

Eslatma: bu darsda kalit tekshiruvi endpoint ichida yozilgan — qisqa bo'lishi uchun. 27.8-darsdagidek uni Depends bilan umumiy bog'liqlikka chiqarish tavsiya etiladi.

2.5. Monitoring va qaror qoidalari

Qoidalar 4-misolning boshida, natijalardan oldin konstantalar sifatida yozilgan:

text
ERTA OGOHLANTIRISH (belgisiz, har oy):
  PSI(oylik_tolov), PSI(tarif), PSI(bashorat) > 0.10 -> "drift ogohl."
  reference = JORIY Production modelning o'quv taqsimoti (paketdagi profil)
  -> model almashsa, reference ham almashadi

SIFAT (belgilar 1 oy kechikadi):
  oy boshida (oy-1) belgilari keladi -> so'rov logi bilan birlashtiriladi
  AUC(oy-1) - xizmat HAQIQATAN bergan bashoratlar bo'yicha

TRIGGER:
  AUC < reference_AUC - 0.03  IKKI oy ketma-ket  (bitta oy - shovqin bo'lishi mumkin)
  -> challenger: oxirgi 2 belgili oyda (27.13: to'satdan drift -> qisqa oyna)
  -> registr: Staging -> xizmat uni SHADOW rejimda ishlatadi

DARVOZA (shadow oyi belgilari kelgach, BIR XIL so'rovlarda):
  AUC farqi > 2*SE (bootstrap)
  log loss juftlashgan farqi < -2*SE
  har tarifda log loss yomonlashuvi <= 0.02
  shadow xatolari = 0
  -> Production; eski -> Archived; xizmat /admin/yangila

ROLLBACK: Archived -> Production ruxsat etilgan o'tish (auditda)

Kechikish zanjiri halol hisoblanadi: drift 9-oyda boshlanadi, 9-oy belgilari 10-oy boshida keladi, qoida ikki oy talab qiladi — shuning uchun trigger eng erta 11-oyda, yangi model esa shadow oyidan keyin, eng erta 12-oyda ishga tushishi mumkin. Tizimni "tezroq" qilishning halol yo'llari 27.13 da ko'rilgan: proksi belgilar, erta ogohlantirishga tayyorgarlik, tarixiy so'rovlarda replay.

2.6. Dockerfile va CI/CT

Misollarda Docker va CI ishga tushirilmaydi, lekin loyiha ularsiz to'liq emas. 27.9-darsdagi qoidalar bo'yicha:

dockerfile
# Dockerfile - mijoz ketishi xizmati
FROM python:3.12.7-slim AS quruvchi
COPY requirements.lock .
RUN pip install --no-cache-dir --prefix=/install -r requirements.lock

FROM python:3.12.7-slim
RUN useradd --create-home --uid 10001 xizmat
COPY --from=quruvchi /install /usr/local
WORKDIR /app
COPY src/ /app/src/
# model artefakti image ga KIRMAYDI: registrdan ishga tushishda yuklanadi
# CHURN_API_KEY - faqat ishga tushirishda (secret), image da emas
ENV PYTHONUNBUFFERED=1
USER xizmat
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=3s CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health')"
CMD ["uvicorn", "churn.xizmat:app", "--host", "0.0.0.0", "--port", "8000"]
yaml
# .github/workflows/ci.yml - har push va PR da
name: ci
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.lock
      - run: python -m pytest -q tests/          # quvur, paket, xizmat testlari
      - run: python -m churn.paket --tekshir     # nazorat namunalari
  image:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t churn-xizmat:${{ github.sha }} .
yaml
# .github/workflows/ct.yml - uzluksiz o'qitish (CT): jadval + qo'lda
name: ct
on:
  schedule:
    - cron: "0 3 2 * *"          # har oyning 2-kuni (belgilar kelgandan keyin)
  workflow_dispatch: {}
jobs:
  monitoring-va-trigger:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.lock
      - run: python -m churn.monitoring --oy oxirgi   # AUC, PSI, trigger
      - run: python -m churn.orgatish --agar-trigger  # challenger -> Staging

Muhim nuqtalar: image teg sifatida commit xeshi oladi (:latest emas); model artefakti image ichida emas — xizmat uni registrdan oladi, shuning uchun model almashuvi yangi image talab qilmaydi (4-misoldagi /admin/yangila shu g'oya); kalit image ga yozilmaydi.

2.7. Hisobot va model kartasi

text
MODEL KARTASI (registr yozuviga biriktiriladi, 26.10 va 27.5):
  vazifa, foydalanuvchi, cheklovlar
  ma'lumot: oylar, xesh, ketish ulushi
  metrika: val AUC (vaqt bo'yicha), kalibrovka
  qaror: nomzodlar, juftlashgan farq, qoida
  monitoring: reference profil, trigger va darvoza qoidalari
  tarix: versiyalar, sabablar (audit)

2.8. Tuzoqlar

Asosiy tuzoqlar: validatsiyasiz yuklash (birlik xatosi modelga jim kiradi); buzuq partiyani o'chirib yuborish (karantin o'rniga); qayta yuklashda takrorlanadigan qatorlar; tranzaksiyasiz yuklash; tasodifiy bo'lish bilan vaqtli ma'lumotda baholash; qarorni bitta AUC bilan qabul qilish; tayyorlash va modelni alohida saqlash (train/serve farqi); kalitni kodda yoki image da saqlash; bashoratlarni loglamaslik (keyin sifatni o'lchab bo'lmaydi); monitoring reference ni model almashganda yangilamaslik; bitta past oyda qayta o'qitish (shovqin); shadow siz almashtirish; eski artefaktni o'chirish.


3. Tez ma'lumotnoma

python
# 1) quvur
xatolar = validatsiya(df)                       # [] bo'lsa - toza
holat, xesh = yukla(db, oy, df)                 # idempotent + atomik

# 2) o'qitish va registr
pipe = logreg().fit(tr, tr.ketdi)
yid = yurish_yoz(db, "LogReg", param, metrika, malumot_xeshi(tr))
joblib.dump(paket_yasa(pipe, df.iloc[:200], metadata), yol)
bosqich_ozgartir(db, 1, "Staging", "yangi nomzod")
bosqich_ozgartir(db, 1, "Production", "darvoza OK")

# 3) xizmat
os.environ["CHURN_API_KEY"]                      # sir faqat muhitdan
mijoz = TestClient(ilova_yarat(Xizmat(papka)))
mijoz.post("/predict", json={"mijozlar": [...]}, headers={"X-API-Key": kalit})

# 4) monitoring va hayot sikli
psi(paket["profil"]["oylik_tolov"], df.oylik_tolov.to_numpy())
auc = roc_auc_score(j.ketdi, j.p)                 # j = so'rov logi + belgilar
if past_ketma_ket >= 2:                           # trigger -> challenger shadow
    registr.otkaz(oy, 2, "Staging", "trigger: AUC 2 oy past")
if all(ok for ok, _ in darvoza_tekshir(j).values()):
    registr.otkaz(oy, 2, "Production", "darvoza OK")
    mijoz.post("/admin/yangila", headers=H)        # /version -> 2

Loyiha tuzilishi (haqiqiy repoda)

text
mijoz_ketishi/
  requirements.lock          pinlangan versiyalar 27.2-bob
  Dockerfile  .dockerignore  27.9-bob
  konfig/qoidalar.yaml       trigger, darvoza, PSI chegaralari (versiyalanadi!)
  src/churn/
    manba.py      partiyalarni olish
    quvur.py      validatsiya, karantin, yuklash
    orgatish.py   nomzodlar, bo'lish, taqqoslash
    kuzatuv.py    eksperiment jurnali
    registr.py    bosqichlar, audit
    paket.py      yasash, yuklash, nazorat
    xizmat.py     FastAPI ilova
    monitoring.py PSI, sifat, trigger, darvoza
  tests/          har modul uchun 27.8-bob
  .github/workflows/ci.yml, ct.yml

Amaliyot xulosasi

partiya -> validatsiya (karantin) -> ombor (idempotent, atomik)
vaqt bo'yicha bo'lish -> jurnal -> juftlashgan qaror -> paket (nazorat)
registr: Staging -> darvoza -> Production (audit)
xizmat: kalit env dan, validatsiya, log (versiya bilan)
monitoring: PSI (erta) + kechikkan belgilar (AUC) -> trigger
challenger -> shadow -> darvoza -> Production -> /version; rollback

4. Batafsil misollar

Misollar real numpy/pandas/sklearn/FastAPI bilan (Python 3.14). Har misol mustaqil ishlaydi, shuning uchun umumiy kod (ma'lumot manbai, model, paket, registr) har birida takrorlanadi. Haqiqiy loyihada bu kod 3-bo'limdagi modullarda bir marta yoziladi.

Misol 1 — Ma'lumot manbai, validatsiya va idempotent yuklash

python
"""1-qadam: ma'lumot manbai (oylik partiyalar), validatsiya va idempotent yuklash."""

import hashlib
import io
import sqlite3
import tempfile
from pathlib import Path

import numpy as np
import pandas as pd

SON = ["oylik_tolov", "muddat_oy", "qongiroqlar", "shikoyatlar"]
KAT = ["tarif", "hudud"]
USTUNLAR = ["mijoz_id", "oy"] + SON + KAT + ["ketdi"]
TARIFLAR = ["A", "B", "C"]
HUDUDLAR = ["Toshkent", "Samarqand", "Buxoro", "Namangan"]
DRIFT_OYI = 9                      # shu oydan dunyo o'zgaradi (27.12)


def oy_partiyasi(oy, n=1500):
    """Har oy - mijozlar kesimi; ketdi = keyingi oyda ketdimi (kechikib keladi)."""
    rng = np.random.default_rng(1000 + oy)
    d = oy >= DRIFT_OYI                  # raqobatchi arzon tarif chiqardi
    tarif = rng.choice(TARIFLAR, n, p=[0.35, 0.30, 0.35] if d else [0.5, 0.3, 0.2])
    hudud = rng.choice(HUDUDLAR, n, p=[0.4, 0.25, 0.2, 0.15])
    tolov = rng.lognormal(np.log(120), 0.35, n) * (1.15 if d else 1.0)
    muddat = rng.gamma(2.0, 12.0, n)
    qongiroq = rng.poisson(25, n)
    shikoyat = rng.poisson(0.8 if d else 0.5, n)
    z = (np.log(tolov) - np.log(120)) / 0.35
    f = (-1.8 + 1.0 * (muddat < 4) - 0.015 * muddat
         + 0.3 * (hudud == "Namangan")
         + (1.0 * (tarif == "A") - 0.3 * (tarif == "C") if d
            else 1.0 * (tarif == "C"))
         + (0.9 if d else 0.5) * shikoyat + (0.3 if d else 0.7) * z)
    ketdi = (rng.random(n) < 1 / (1 + np.exp(-f))).astype(int)
    return pd.DataFrame({
        "mijoz_id": [f"M{oy:02d}{i:05d}" for i in range(n)], "oy": oy,
        "oylik_tolov": tolov.round(2), "muddat_oy": muddat.round(1),
        "qongiroqlar": qongiroq, "shikoyatlar": shikoyat,
        "tarif": tarif, "hudud": hudud, "ketdi": ketdi})


# ---------------- quvur: validatsiya ----------------
def validatsiya(df):
    xatolar = []
    if list(df.columns) != USTUNLAR:
        return [f"sxema: kutilgan {len(USTUNLAR)} ustun, "
                f"kelgan {sorted(set(df.columns) ^ set(USTUNLAR))} farq"]
    if len(df) < 500:
        xatolar.append(f"hajm: {len(df)} < 500")
    if df.isna().mean().max() > 0.01:
        xatolar.append("bo'sh qiymatlar > 1%")
    if df["mijoz_id"].duplicated().any():
        xatolar.append(f"takror id: {int(df['mijoz_id'].duplicated().sum())}")
    if not df["oylik_tolov"].between(1, 2000).all():
        xatolar.append(f"oylik_tolov diapazondan tashqari "
                       f"(maks {df['oylik_tolov'].max():.0f})")
    if (df["muddat_oy"] < 0).any():
        xatolar.append("muddat_oy < 0")
    for c, ruxsat in [("tarif", TARIFLAR), ("hudud", HUDUDLAR)]:
        yangi = sorted(set(df[c]) - set(ruxsat))
        if yangi:
            xatolar.append(f"{c}: noma'lum qiymatlar {yangi}")
    if not df["ketdi"].isin([0, 1]).all():
        xatolar.append("ketdi faqat 0/1")
    return xatolar


def kanonik_xesh(df):
    """Tartib va format bir xil bo'lsa - xesh bir xil 27.5-bob."""
    buf = io.StringIO()
    df.sort_values("mijoz_id").to_csv(buf, index=False, float_format="%.4f",
                                      lineterminator="\n")
    return hashlib.sha256(buf.getvalue().encode()).hexdigest()[:12]


# ---------------- quvur: idempotent yuklash ----------------
def db_och(yol):
    db = sqlite3.connect(yol)
    db.executescript(
        "CREATE TABLE IF NOT EXISTS mijozlar (mijoz_id TEXT, oy INT,"
        " oylik_tolov REAL, muddat_oy REAL, qongiroqlar INT, shikoyatlar INT,"
        " tarif TEXT, hudud TEXT, ketdi INT, PRIMARY KEY (oy, mijoz_id));"
        "CREATE TABLE IF NOT EXISTS yuklashlar (oy INT PRIMARY KEY,"
        " xesh TEXT, qatorlar INT, holat TEXT);")
    return db


def yukla(db, oy, df, buzilish=False):
    xesh = kanonik_xesh(df)
    eski = db.execute("SELECT xesh FROM yuklashlar WHERE oy = ?",
                      (oy,)).fetchone()
    if eski and eski[0] == xesh:
        return "o'zgarmagan - o'tkazildi", xesh
    with db:                                         # tranzaksiya
        db.execute("DELETE FROM mijozlar WHERE oy = ?", (oy,))
        if buzilish:
            raise RuntimeError("yuklash o'rtasida uzilish")
        db.executemany("INSERT INTO mijozlar VALUES (?,?,?,?,?,?,?,?,?)",
                       df[USTUNLAR].itertuples(index=False, name=None))
        db.execute("INSERT OR REPLACE INTO yuklashlar VALUES (?,?,?,?)",
                   (oy, xesh, len(df), "yuklandi"))
    return ("almashtirildi" if eski else "yuklandi"), xesh


def quvur(db, manba, karantin):
    natija = []
    for fayl in sorted(manba.glob("oy_*.csv")):
        oy = int(fayl.stem.split("_")[1])
        df = pd.read_csv(fayl)
        xatolar = validatsiya(df)
        if xatolar:
            (karantin / fayl.name).write_bytes(fayl.read_bytes())
            natija.append((fayl.name, "KARANTIN", "; ".join(xatolar)))
            continue
        holat, xesh = yukla(db, oy, df)
        natija.append((fayl.name, holat, xesh))
    return natija


def main() -> None:
    with tempfile.TemporaryDirectory() as tmp:
        manba, karantin = Path(tmp, "manba"), Path(tmp, "karantin")
        manba.mkdir()
        karantin.mkdir()
        db = db_och(Path(tmp, "ombor.sqlite"))

        print("=== 1. Manba: 1-6 oy partiyalari (ikkitasi buzuq) ===")
        for oy in range(1, 7):
            df = oy_partiyasi(oy)
            if oy == 4:
                df["oylik_tolov"] *= 1000            # so'mda keldi (birlik xatosi)
            if oy == 5:
                df = df.drop(columns=["hudud"])      # ustun tushib qoldi
            df.to_csv(manba / f"oy_{oy:02d}.csv", index=False)
        print(f"  fayllar: {sorted(p.name for p in manba.iterdir())}")

        print("\n=== 2. Quvur: 1-yurish ===")
        for nom, holat, izoh in quvur(db, manba, karantin):
            print(f"  {nom}: {holat:<24} {izoh}")

        print("\n=== 3. Quvur: 2-yurish (xuddi shu fayllar) - idempotentlik ===")
        oldin = db.execute("SELECT COUNT(*) FROM mijozlar").fetchone()[0]
        for nom, holat, izoh in quvur(db, manba, karantin):
            if holat != "KARANTIN":
                print(f"  {nom}: {holat}")
        keyin = db.execute("SELECT COUNT(*) FROM mijozlar").fetchone()[0]
        print(f"  qatorlar: {oldin} -> {keyin} (takrorlanmadi: {oldin == keyin})")

        print("\n=== 4. Manba tuzatildi: 4 va 5-oy qayta yetkazildi ===")
        for oy in [4, 5]:
            oy_partiyasi(oy).to_csv(manba / f"oy_{oy:02d}.csv", index=False)
            (karantin / f"oy_{oy:02d}.csv").unlink()
        for nom, holat, izoh in quvur(db, manba, karantin):
            if holat != "o'zgarmagan - o'tkazildi":
                print(f"  {nom}: {holat:<24} {izoh}")

        print("\n=== 5. Atomiklik: yuklash o'rtasida uzilish ===")
        df2 = oy_partiyasi(2)
        df2.loc[0, "shikoyatlar"] = 3                 # tuzatilgan versiya
        oldin = db.execute("SELECT COUNT(*) FROM mijozlar WHERE oy = 2"
                           ).fetchone()[0]
        try:
            yukla(db, 2, df2, buzilish=True)
        except RuntimeError as e:
            print(f"  xato: {e}")
        keyin = db.execute("SELECT COUNT(*) FROM mijozlar WHERE oy = 2"
                           ).fetchone()[0]
        print(f"  2-oy qatorlari: {oldin} -> {keyin} (tranzaksiya qaytarildi)")
        holat, xesh = yukla(db, 2, df2)
        print(f"  qayta urinish: {holat}, xesh {xesh}")

        print("\n=== 6. Ombor holati ===")
        jad = pd.read_sql("SELECT oy, COUNT(*) AS qator, AVG(ketdi) AS ketish,"
                          " AVG(oylik_tolov) AS tolov FROM mijozlar"
                          " GROUP BY oy ORDER BY oy", db)
        print(jad.round(3).to_string(index=False))
        print(f"  yuklashlar jurnali: "
              f"{db.execute('SELECT COUNT(*) FROM yuklashlar').fetchone()[0]}"
              f" oy, karantinda: {len(list(karantin.iterdir()))} fayl")
        db.close()
    print("  ⭐ Validatsiya - darvoza, yuklash - idempotent va atomik")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Manba: 1-6 oy partiyalari (ikkitasi buzuq) ===
  fayllar: ['oy_01.csv', 'oy_02.csv', 'oy_03.csv', 'oy_04.csv', 'oy_05.csv', 'oy_06.csv']

=== 2. Quvur: 1-yurish ===
  oy_01.csv: yuklandi                 610779f5e002
  oy_02.csv: yuklandi                 d8a70865178a
  oy_03.csv: yuklandi                 f1bff71f671e
  oy_04.csv: KARANTIN                 oylik_tolov diapazondan tashqari (maks 316510)
  oy_05.csv: KARANTIN                 sxema: kutilgan 9 ustun, kelgan ['hudud'] farq
  oy_06.csv: yuklandi                 1e64911df0ac

=== 3. Quvur: 2-yurish (xuddi shu fayllar) - idempotentlik ===
  oy_01.csv: o'zgarmagan - o'tkazildi
  oy_02.csv: o'zgarmagan - o'tkazildi
  oy_03.csv: o'zgarmagan - o'tkazildi
  oy_06.csv: o'zgarmagan - o'tkazildi
  qatorlar: 6000 -> 6000 (takrorlanmadi: True)

=== 4. Manba tuzatildi: 4 va 5-oy qayta yetkazildi ===
  oy_04.csv: yuklandi                 baf34efc64c9
  oy_05.csv: yuklandi                 78fcee6b69db

=== 5. Atomiklik: yuklash o'rtasida uzilish ===
  xato: yuklash o'rtasida uzilish
  2-oy qatorlari: 1500 -> 1500 (tranzaksiya qaytarildi)
  qayta urinish: almashtirildi, xesh 098c77c05202

=== 6. Ombor holati ===
 oy  qator  ketish   tolov
  1   1500   0.198 129.019
  2   1500   0.214 126.245
  3   1500   0.185 126.401
  4   1500   0.193 126.462
  5   1500   0.192 128.142
  6   1500   0.213 128.860
  yuklashlar jurnali: 6 oy, karantinda: 0 fayl
  ⭐ Validatsiya - darvoza, yuklash - idempotent va atomik

Natija tahlili.

1-2-bo'limlar — manbadan olti oylik partiya keldi, ikkitasi buzuq. 4-oy partiyasida to'lov so'mda kelgan (birlik xatosi) — sxema va turlar to'g'ri, lekin diapazon tekshiruvi maksimum 316510 ni ushladi. 5-oyda hudud ustuni tushib qolgan — sxema tekshiruvi. Ikkala fayl karantinga ko'chirildi, qolgan to'rttasi yuklandi va har biri uchun kanonik xesh yozildi.

3-bo'lim — xuddi shu fayllar bilan quvur qayta ishga tushirildi (masalan, rejalashtiruvchi ikki marta ishga tushirib yubordi). Har partiya "o'zgarmagan — o'tkazildi", qatorlar soni 6000 -> 6000. Bu — idempotentlik: quvurni necha marta ishga tushirsangiz ham natija bir xil.

4-bo'lim — manba xatolarni tuzatib, 4 va 5-oyni qayta yetkazdi. Karantindagi nusxalar olib tashlandi, yangi fayllar validatsiyadan o'tib yuklandi.

5-bo'lim — atomiklik. 2-oyning tuzatilgan versiyasini yuklash DELETE dan keyin, INSERT dan oldin uzildi. Tranzaksiya qaytarildi — 2-oyda baribir 1500 qator, ya'ni eski ma'lumot butun. Qayta urinishda partiya "almashtirildi" — xesh o'zgargani uchun (bitta qatorda shikoyatlar tuzatilgan).

6-bo'lim — omborda olti oy, har birida 1500 qator; ketish ulushi 0.18-0.21 atrofida barqaror. Karantin bo'sh — hamma muammo hal qilingan.

Misol 2 — O'qitish, eksperiment jurnali, registr va paket

python
"""2-qadam: o'qitish, eksperiment kuzatuvi (sqlite), registr va paket (joblib)."""

import hashlib
import json
import sqlite3
import tempfile
import warnings
from pathlib import Path

import joblib
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss, roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

SON = ["oylik_tolov", "muddat_oy", "qongiroqlar", "shikoyatlar"]
KAT = ["tarif", "hudud"]
USTUNLAR = ["mijoz_id", "oy"] + SON + KAT + ["ketdi"]
TARIFLAR = ["A", "B", "C"]
HUDUDLAR = ["Toshkent", "Samarqand", "Buxoro", "Namangan"]
DRIFT_OYI = 9
KOD_VERSIYASI = "churn-1.0.0"
NOM = "mijoz_ketishi"


def oy_partiyasi(oy, n=1500):
    """Har oy - mijozlar kesimi; ketdi = keyingi oyda ketdimi (kechikib keladi)."""
    rng = np.random.default_rng(1000 + oy)
    d = oy >= DRIFT_OYI                  # raqobatchi arzon tarif chiqardi
    tarif = rng.choice(TARIFLAR, n, p=[0.35, 0.30, 0.35] if d else [0.5, 0.3, 0.2])
    hudud = rng.choice(HUDUDLAR, n, p=[0.4, 0.25, 0.2, 0.15])
    tolov = rng.lognormal(np.log(120), 0.35, n) * (1.15 if d else 1.0)
    muddat = rng.gamma(2.0, 12.0, n)
    qongiroq = rng.poisson(25, n)
    shikoyat = rng.poisson(0.8 if d else 0.5, n)
    z = (np.log(tolov) - np.log(120)) / 0.35
    f = (-1.8 + 1.0 * (muddat < 4) - 0.015 * muddat
         + 0.3 * (hudud == "Namangan")
         + (1.0 * (tarif == "A") - 0.3 * (tarif == "C") if d
            else 1.0 * (tarif == "C"))
         + (0.9 if d else 0.5) * shikoyat + (0.3 if d else 0.7) * z)
    ketdi = (rng.random(n) < 1 / (1 + np.exp(-f))).astype(int)
    return pd.DataFrame({
        "mijoz_id": [f"M{oy:02d}{i:05d}" for i in range(n)], "oy": oy,
        "oylik_tolov": tolov.round(2), "muddat_oy": muddat.round(1),
        "qongiroqlar": qongiroq, "shikoyatlar": shikoyat,
        "tarif": tarif, "hudud": hudud, "ketdi": ketdi})


def ombor_qur(yol, oylar):
    """1-qadam natijasi: validatsiyadan o'tgan partiyalar sqlite da."""
    db = sqlite3.connect(yol)
    pd.concat([oy_partiyasi(o) for o in oylar]).to_sql("mijozlar", db,
                                                        index=False)
    return db


def malumot_xeshi(df):
    return hashlib.sha256(pd.util.hash_pandas_object(
        df[USTUNLAR], index=False).values.tobytes()).hexdigest()[:12]


# ---------------- nomzodlar ----------------
def logreg():
    return Pipeline([
        ("tayyor", ColumnTransformer([
            ("son", StandardScaler(), SON),
            ("kat", OneHotEncoder(handle_unknown="ignore"), KAT)])),
        ("model", LogisticRegression(max_iter=2000))])


def histgb():
    return Pipeline([
        ("tayyor", ColumnTransformer([
            ("son", "passthrough", SON),
            ("kat", OrdinalEncoder(handle_unknown="use_encoded_value",
                                   unknown_value=-1), KAT)])),
        ("model", HistGradientBoostingClassifier(
            max_iter=150, learning_rate=0.05, categorical_features=[4, 5],
            random_state=0))])


NOMZODLAR = {"LogReg": (logreg, {"C": 1.0}),
             "HistGB": (histgb, {"max_iter": 150, "learning_rate": 0.05})}
SODDALIK = ["LogReg", "HistGB"]


# ---------------- eksperiment kuzatuvi (MLflow g'oyasi) ----------------
def kuzatuv_och(db):
    db.execute("CREATE TABLE IF NOT EXISTS yurishlar (yurish_id TEXT PRIMARY"
               " KEY, model TEXT, parametrlar TEXT, metrikalar TEXT,"
               " malumot_xeshi TEXT, kod TEXT)")


def yurish_yoz(db, model, param, metrika, mx):
    yid = hashlib.sha256(json.dumps([model, param, mx, KOD_VERSIYASI],
                                    sort_keys=True).encode()).hexdigest()[:10]
    with db:
        db.execute("INSERT OR REPLACE INTO yurishlar VALUES (?,?,?,?,?,?)",
                   (yid, model, json.dumps(param, sort_keys=True),
                    json.dumps(metrika, sort_keys=True), mx, KOD_VERSIYASI))
    return yid


# ---------------- registr 27.5-bob ----------------
RUXSAT = {("None", "Staging"), ("Staging", "Production"),
          ("Production", "Archived"), ("Staging", "Archived")}


def registr_och(db):
    db.executescript(
        "CREATE TABLE IF NOT EXISTS versiyalar (nom TEXT, versiya INT,"
        " bosqich TEXT, yurish_id TEXT, artefakt TEXT, artefakt_xeshi TEXT,"
        " metrikalar TEXT, PRIMARY KEY (nom, versiya));"
        "CREATE TABLE IF NOT EXISTS audit (id INTEGER PRIMARY KEY"
        " AUTOINCREMENT, nom TEXT, versiya INT, eski TEXT, yangi TEXT,"
        " sabab TEXT);")


def bosqich_ozgartir(db, versiya, yangi, sabab):
    eski = db.execute("SELECT bosqich FROM versiyalar WHERE nom=? AND"
                      " versiya=?", (NOM, versiya)).fetchone()[0]
    if (eski, yangi) not in RUXSAT:
        raise ValueError(f"ruxsat yo'q: {eski} -> {yangi}")
    with db:
        db.execute("UPDATE versiyalar SET bosqich=? WHERE nom=? AND"
                   " versiya=?", (yangi, NOM, versiya))
        db.execute("INSERT INTO audit (nom, versiya, eski, yangi, sabab)"
                   " VALUES (?,?,?,?,?)", (NOM, versiya, eski, yangi, sabab))


def paket_yasa(pipe, df_nazorat, metadata):
    return {"format": 1, "pipeline": pipe,
            "sxema": {"son": SON, "kat": KAT,
                      "kategoriyalar": {"tarif": TARIFLAR, "hudud": HUDUDLAR}},
            "metadata": metadata,
            "nazorat": {"X": df_nazorat[SON + KAT].to_dict("list"),
                        "p": pipe.predict_proba(df_nazorat)[:, 1].tolist()}}


def paket_yukla(yol):
    p = joblib.load(yol)
    if p["format"] != 1:
        raise ValueError("format mos emas")
    X = pd.DataFrame(p["nazorat"]["X"])
    farq = np.max(np.abs(p["pipeline"].predict_proba(X)[:, 1]
                         - np.array(p["nazorat"]["p"])))
    return p, float(farq)


def main() -> None:
    warnings.simplefilter("ignore")
    with tempfile.TemporaryDirectory() as tmp:
        ombor = ombor_qur(Path(tmp, "ombor.sqlite"), range(1, 7))
        df = pd.read_sql("SELECT * FROM mijozlar ORDER BY oy, mijoz_id", ombor)
        ombor.close()
        tr, va = df[df.oy <= 5], df[df.oy == 6]
        mx = malumot_xeshi(df)

        print("=== 1. Dizayn: vaqt bo'yicha bo'lish ===")
        print(f"  o'quv: 1-5 oy ({len(tr)} qator), validatsiya: 6-oy "
              f"({len(va)}), ketish ulushi {df.ketdi.mean():.3f}")
        print(f"  ma'lumot xeshi (1-6 oy): {mx}")

        mlops = sqlite3.connect(Path(tmp, "mlops.sqlite"))
        kuzatuv_och(mlops)
        print("\n=== 2. Nomzodlar va eksperiment jurnali ===")
        p_val, metrikalar = {}, {}
        for nom, (yasa, param) in NOMZODLAR.items():
            pipe = yasa().fit(tr, tr.ketdi)
            p = pipe.predict_proba(va)[:, 1]
            p_val[nom] = p
            m = {"val_auc": round(float(roc_auc_score(va.ketdi, p)), 4),
                 "val_logloss": round(float(log_loss(va.ketdi, p)), 4),
                 "kalibrovka": round(float(abs(p.mean() - va.ketdi.mean())), 4)}
            metrikalar[nom] = m
            yid = yurish_yoz(mlops, nom, param, m, malumot_xeshi(tr))
            print(f"  {nom:<7} yurish {yid}: {m}")
        yurish_yoz(mlops, "LogReg", NOMZODLAR["LogReg"][1],     # xuddi shu
                   metrikalar["LogReg"], malumot_xeshi(tr))     # yurish yana
        n = mlops.execute("SELECT COUNT(*) FROM yurishlar").fetchone()[0]
        print(f"  jurnalda {n} yurish (qayta yozilgan yurish takrorlanmadi - "
              f"id = xesh)")

        print("\n=== 3. Juftlashgan bootstrap: HistGB - LogReg (6-oy) ===")
        rng = np.random.default_rng(0)
        y = va.ketdi.to_numpy()
        farqlar = []
        for _ in range(300):
            i = rng.integers(0, len(y), len(y))
            farqlar.append(roc_auc_score(y[i], p_val["HistGB"][i])
                           - roc_auc_score(y[i], p_val["LogReg"][i]))
        farqlar = np.array(farqlar)
        d = roc_auc_score(y, p_val["HistGB"]) - roc_auc_score(y, p_val["LogReg"])
        se = farqlar.std(ddof=1)
        print(f"  AUC farqi {d:+.4f}, bootstrap SE {se:.4f}, "
              f"sezilarli: {abs(d) > 2 * se}")
        eng = "HistGB" if d > 0 else "LogReg"
        tanlov = "LogReg" if (eng == "LogReg" or d <= 2 * se) else "HistGB"
        print(f"  qoida: eng yaxshisidan sezilarli yomon bo'lmagan ENG SODDA "
              f"-> {tanlov}")

        print("\n=== 4. Yakuniy model (1-6 oy), paket va nazorat ===")
        yakuniy = NOMZODLAR[tanlov][0]().fit(df, df.ketdi)
        artefaktlar = Path(tmp, "artefaktlar")
        artefaktlar.mkdir()
        metadata = {"nom": NOM, "model": tanlov, "oylar": [1, 6],
                    "malumot_xeshi": mx, "kod": KOD_VERSIYASI,
                    "val_auc": round(float(roc_auc_score(y, p_val[tanlov])), 4),
                    "reference_ketish": round(float(df.ketdi.mean()), 4)}
        paket = paket_yasa(yakuniy, df.iloc[:200], metadata)
        yol = artefaktlar / f"{NOM}-v1.joblib"
        joblib.dump(paket, yol)
        artefakt_xeshi = hashlib.sha256(yol.read_bytes()).hexdigest()[:12]
        _, farq = paket_yukla(yol)
        print(f"  paket kalitlari: {sorted(paket)}")
        print(f"  metadata: {metadata}")
        print(f"  qayta yuklangach nazorat namunalari maks farq: {farq:.2e}")

        print("\n=== 5. Registr va darvoza ===")
        registr_och(mlops)
        yid = mlops.execute("SELECT yurish_id FROM yurishlar WHERE model=?",
                            (tanlov,)).fetchone()[0]
        with mlops:
            mlops.execute("INSERT INTO versiyalar VALUES (?,?,?,?,?,?,?)",
                          (NOM, 1, "None", yid, yol.name, artefakt_xeshi,
                           json.dumps({"val_auc": metadata["val_auc"]})))
        bosqich_ozgartir(mlops, 1, "Staging", "yangi nomzod")
        darvoza = {"val_auc >= 0.70": metadata["val_auc"] >= 0.70,
                   "nazorat farqi <= 1e-9": farq <= 1e-9,
                   "sxema to'liq": set(paket["sxema"]["son"]) == set(SON)}
        for q, ok in darvoza.items():
            print(f"  {q:<24} {'OK' if ok else 'RAD'}")
        if all(darvoza.values()):
            bosqich_ozgartir(mlops, 1, "Production", "darvoza OK")
        try:
            bosqich_ozgartir(mlops, 1, "Staging", "qo'lda")
        except ValueError as e:
            print(f"  noto'g'ri o'tish rad etildi: {e}")
        print(pd.read_sql("SELECT nom, versiya, bosqich, yurish_id, artefakt"
                          " FROM versiyalar", mlops).to_string(index=False))
        print(pd.read_sql("SELECT versiya, eski, yangi, sabab FROM audit",
                          mlops).to_string(index=False))
        mlops.close()
    print("  ⭐ Qaror jurnalda, model registrda, paket o'zini tekshiradi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Dizayn: vaqt bo'yicha bo'lish ===
  o'quv: 1-5 oy (7500 qator), validatsiya: 6-oy (1500), ketish ulushi 0.199
  ma'lumot xeshi (1-6 oy): 53c21818a727

=== 2. Nomzodlar va eksperiment jurnali ===
  LogReg  yurish d9576f065a: {'val_auc': 0.7263, 'val_logloss': 0.4615, 'kalibrovka': 0.0156}
  HistGB  yurish 90978cf8c6: {'val_auc': 0.7198, 'val_logloss': 0.467, 'kalibrovka': 0.017}
  jurnalda 2 yurish (qayta yozilgan yurish takrorlanmadi - id = xesh)

=== 3. Juftlashgan bootstrap: HistGB - LogReg (6-oy) ===
  AUC farqi -0.0066, bootstrap SE 0.0090, sezilarli: False
  qoida: eng yaxshisidan sezilarli yomon bo'lmagan ENG SODDA -> LogReg

=== 4. Yakuniy model (1-6 oy), paket va nazorat ===
  paket kalitlari: ['format', 'metadata', 'nazorat', 'pipeline', 'sxema']
  metadata: {'nom': 'mijoz_ketishi', 'model': 'LogReg', 'oylar': [1, 6], 'malumot_xeshi': '53c21818a727', 'kod': 'churn-1.0.0', 'val_auc': 0.7263, 'reference_ketish': 0.1992}
  qayta yuklangach nazorat namunalari maks farq: 0.00e+00

=== 5. Registr va darvoza ===
  val_auc >= 0.70          OK
  nazorat farqi <= 1e-9    OK
  sxema to'liq             OK
  noto'g'ri o'tish rad etildi: ruxsat yo'q: Production -> Staging
          nom  versiya    bosqich  yurish_id                artefakt
mijoz_ketishi        1 Production d9576f065a mijoz_ketishi-v1.joblib
 versiya    eski      yangi        sabab
       1    None    Staging yangi nomzod
       1 Staging Production   darvoza OK
  ⭐ Qaror jurnalda, model registrda, paket o'zini tekshiradi

Natija tahlili.

1-bo'lim — bo'lish vaqt bo'yicha: 1-5 oy o'quv, 6-oy validatsiya. Bu "kelajak oyni bashorat qilish" sinovi — ishlab chiqarishdagi vaziyatning aynan o'zi. Ma'lumot xeshi keyingi hamma yozuvlarga (yurishlar, paket metadata, registr) bog'lanadi.

2-bo'lim — ikki nomzod: LogReg AUC 0.7263, HistGB 0.7198. Ikkala yurish jurnalga yozildi; yurish identifikatori — model, parametrlar, ma'lumot va kod versiyasining xeshi, shuning uchun xuddi shu yurishni qayta yozish jurnalda takror hosil qilmadi (2 yurish).

3-bo'lim — 6-oyda juftlashgan bootstrap: HistGB - LogReg = -0.0066, SE 0.0090 — sezilarli emas. Qoida bo'yicha eng sodda — LogReg. Bu ma'lumotdagi bog'liqliklar asosan chiziqli (logit bo'yicha), shuning uchun boosting ustunlik bermadi — bu halol natija, va u xizmatni soddalashtiradi (tezroq, tushuntirish oson).

4-bo'lim — yakuniy model 1-6 oyda qayta o'qitildi va paketlandi: Pipeline (tayyorlash + model bitta obyekt), sxema, metadata (oylar, ma'lumot xeshi, kod versiyasi, val AUC, reference ketish ulushi) va 200 ta nazorat namunasi. Qayta yuklangach nazorat namunalari aynan mos (0.00e+00).

5-bo'lim — registr. Versiya None -> Staging ga o'tdi, uchta darvoza qoidasi tekshirildi (val_auc >= 0.70, nazorat, sxema) va Staging -> Production. Production -> Staging kabi ruxsat etilmagan o'tish rad etildi. Audit jadvalida har o'tish sababi bilan yozilgan.

Misol 3 — FastAPI xizmati: validatsiya, API kaliti va so'rov loglari

python
"""3-qadam: FastAPI xizmati - /predict, /health, /version, validatsiya, API kaliti, loglar."""

import hmac
import json
import logging
import os
import secrets
import sqlite3
import sys
import tempfile
import warnings
from pathlib import Path
from typing import Literal

import joblib
import numpy as np
import pandas as pd
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, ConfigDict, Field
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

with warnings.catch_warnings():             # TestClient importi ogohlantirish
    warnings.simplefilter("ignore")         # chiqaradi (stderr ni toza tutamiz)
    from fastapi.testclient import TestClient

SON = ["oylik_tolov", "muddat_oy", "qongiroqlar", "shikoyatlar"]
KAT = ["tarif", "hudud"]
USTUNLAR = ["mijoz_id", "oy"] + SON + KAT + ["ketdi"]
TARIFLAR = ["A", "B", "C"]
HUDUDLAR = ["Toshkent", "Samarqand", "Buxoro", "Namangan"]
DRIFT_OYI = 9
NOM = "mijoz_ketishi"

logging.basicConfig(stream=sys.stdout, level=logging.INFO,
                    format="  LOG %(levelname)s %(message)s")
log = logging.getLogger("xizmat")
logging.getLogger("httpx").setLevel(logging.WARNING)     # TestClient shovqini


def oy_partiyasi(oy, n=1500):
    """Har oy - mijozlar kesimi; ketdi = keyingi oyda ketdimi (kechikib keladi)."""
    rng = np.random.default_rng(1000 + oy)
    d = oy >= DRIFT_OYI                  # raqobatchi arzon tarif chiqardi
    tarif = rng.choice(TARIFLAR, n, p=[0.35, 0.30, 0.35] if d else [0.5, 0.3, 0.2])
    hudud = rng.choice(HUDUDLAR, n, p=[0.4, 0.25, 0.2, 0.15])
    tolov = rng.lognormal(np.log(120), 0.35, n) * (1.15 if d else 1.0)
    muddat = rng.gamma(2.0, 12.0, n)
    qongiroq = rng.poisson(25, n)
    shikoyat = rng.poisson(0.8 if d else 0.5, n)
    z = (np.log(tolov) - np.log(120)) / 0.35
    f = (-1.8 + 1.0 * (muddat < 4) - 0.015 * muddat
         + 0.3 * (hudud == "Namangan")
         + (1.0 * (tarif == "A") - 0.3 * (tarif == "C") if d
            else 1.0 * (tarif == "C"))
         + (0.9 if d else 0.5) * shikoyat + (0.3 if d else 0.7) * z)
    ketdi = (rng.random(n) < 1 / (1 + np.exp(-f))).astype(int)
    return pd.DataFrame({
        "mijoz_id": [f"M{oy:02d}{i:05d}" for i in range(n)], "oy": oy,
        "oylik_tolov": tolov.round(2), "muddat_oy": muddat.round(1),
        "qongiroqlar": qongiroq, "shikoyatlar": shikoyat,
        "tarif": tarif, "hudud": hudud, "ketdi": ketdi})


# ---------------- 2-qadam natijasi: paket va registr ----------------
def logreg():
    return Pipeline([
        ("tayyor", ColumnTransformer([
            ("son", StandardScaler(), SON),
            ("kat", OneHotEncoder(handle_unknown="ignore"), KAT)])),
        ("model", LogisticRegression(max_iter=2000))])


def tayyorla(papka):
    df = pd.concat([oy_partiyasi(o) for o in range(1, 7)])
    pipe = logreg().fit(df, df.ketdi)
    nazorat = df.iloc[:200]
    paket = {"format": 1, "pipeline": pipe,
             "sxema": {"son": SON, "kat": KAT},
             "metadata": {"nom": NOM, "model": "LogReg", "oylar": [1, 6]},
             "nazorat": {"X": nazorat[SON + KAT].to_dict("list"),
                         "p": pipe.predict_proba(nazorat)[:, 1].tolist()}}
    joblib.dump(paket, papka / f"{NOM}-v1.joblib")
    db = sqlite3.connect(papka / "mlops.sqlite")
    db.execute("CREATE TABLE versiyalar (nom TEXT, versiya INT, bosqich TEXT,"
               " artefakt TEXT, PRIMARY KEY (nom, versiya))")
    with db:
        db.execute("INSERT INTO versiyalar VALUES (?,?,?,?)",
                   (NOM, 1, "Production", f"{NOM}-v1.joblib"))
    db.close()


# ---------------- xizmat ----------------
class Mijoz(BaseModel):
    model_config = ConfigDict(extra="forbid")
    mijoz_id: str = Field(min_length=1, max_length=32)
    oylik_tolov: float = Field(gt=0, lt=2000)
    muddat_oy: float = Field(ge=0, le=600)
    qongiroqlar: int = Field(ge=0, le=10_000)
    shikoyatlar: int = Field(ge=0, le=100)
    tarif: Literal["A", "B", "C"]
    hudud: Literal["Toshkent", "Samarqand", "Buxoro", "Namangan"]


class Sorov(BaseModel):
    mijozlar: list[Mijoz] = Field(min_length=1, max_length=5000)


class Xizmat:
    """Registrdagi Production versiyani yuklaydi; so'rovlarni loglaydi."""

    def __init__(self, papka):
        self.papka = papka
        self.soat = 0                                   # virtual soat
        self.log_db = sqlite3.connect(papka / "sorovlar.sqlite",
                                      check_same_thread=False)
        self.log_db.execute("CREATE TABLE IF NOT EXISTS sorovlar (id INTEGER"
                            " PRIMARY KEY AUTOINCREMENT, soat INT, versiya INT,"
                            " mijoz_id TEXT, kirish TEXT, p REAL)")
        self.yangila()

    def yangila(self):
        db = sqlite3.connect(self.papka / "mlops.sqlite")
        versiya, artefakt = db.execute(
            "SELECT versiya, artefakt FROM versiyalar WHERE nom=? AND"
            " bosqich='Production'", (NOM,)).fetchone()
        db.close()
        paket = joblib.load(self.papka / artefakt)
        X = pd.DataFrame(paket["nazorat"]["X"])
        farq = np.max(np.abs(paket["pipeline"].predict_proba(X)[:, 1]
                             - np.array(paket["nazorat"]["p"])))
        if farq > 1e-9:
            raise RuntimeError("nazorat namunalari mos emas")
        self.versiya, self.paket = versiya, paket
        log.info("yuklandi: %s v%d (nazorat OK)", NOM, versiya)

    def bashorat(self, mijozlar):
        df = pd.DataFrame([m.model_dump() for m in mijozlar])
        p = self.paket["pipeline"].predict_proba(df)[:, 1]
        self.soat += 1
        with self.log_db:
            self.log_db.executemany(
                "INSERT INTO sorovlar (soat, versiya, mijoz_id, kirish, p)"
                " VALUES (?,?,?,?,?)",
                [(self.soat, self.versiya, m.mijoz_id,
                  json.dumps(m.model_dump(), sort_keys=True), float(q))
                 for m, q in zip(mijozlar, p)])
        return p


def niqobla(kalit):
    return "*" * 8 if kalit else "(yo'q)"


def ilova_yarat(xizmat):
    kalit = os.environ.get("CHURN_API_KEY")
    if not kalit:                                      # 12-factor: sir env dan
        raise RuntimeError("CHURN_API_KEY o'rnatilmagan")
    log.info("API kaliti: %s (uzunligi %d)", niqobla(kalit), len(kalit))
    app = FastAPI(title="Mijoz ketishi xizmati")

    @app.get("/health")
    def health():
        return {"holat": "ok", "model_yuklangan": xizmat.paket is not None}

    @app.get("/version")
    def version():
        m = xizmat.paket["metadata"]
        return {"nom": NOM, "versiya": xizmat.versiya,
                "algoritm": m["model"], "oylar": m["oylar"]}

    @app.post("/predict")
    def predict(sorov: Sorov, x_api_key: str | None = Header(default=None)):
        if not hmac.compare_digest((x_api_key or "").encode(), kalit.encode()):
            log.warning("rad etildi: kalit %s", niqobla(x_api_key))
            raise HTTPException(status_code=401, detail="kalit noto'g'ri")
        p = xizmat.bashorat(sorov.mijozlar)
        return {"versiya": xizmat.versiya,
                "bashoratlar": [{"mijoz_id": m.mijoz_id,
                                 "ketish_ehtimoli": round(float(q), 4)}
                                for m, q in zip(sorov.mijozlar, p)]}

    return app


def main() -> None:
    warnings.simplefilter("ignore")
    with tempfile.TemporaryDirectory() as tmp:
        papka = Path(tmp)
        tayyorla(papka)

        print("=== 1. Kalitsiz ishga tushirish ===")
        os.environ.pop("CHURN_API_KEY", None)
        xizmat = Xizmat(papka)
        try:
            ilova_yarat(xizmat)
        except RuntimeError as e:
            print(f"  rad etildi: {e}")

        print("\n=== 2. Kalit muhit o'zgaruvchisidan ===")
        os.environ["CHURN_API_KEY"] = secrets.token_hex(16)  # sinov uchun
        kalit = os.environ["CHURN_API_KEY"]
        mijoz = TestClient(ilova_yarat(xizmat))
        print(f"  /health  -> {mijoz.get('/health').json()}")
        print(f"  /version -> {mijoz.get('/version').json()}")

        print("\n=== 3. /predict ===")
        df = oy_partiyasi(7).iloc[:3]
        yuk = {"mijozlar": df[["mijoz_id"] + SON + KAT].to_dict("records")}
        r = mijoz.post("/predict", json=yuk)
        print(f"  kalitsiz: {r.status_code} {r.json()}")
        r = mijoz.post("/predict", json=yuk, headers={"X-API-Key": "xato"})
        print(f"  noto'g'ri kalit: {r.status_code}")
        r = mijoz.post("/predict", json=yuk, headers={"X-API-Key": kalit})
        print(f"  to'g'ri kalit: {r.status_code}")
        for b in r.json()["bashoratlar"]:
            print(f"    {b}")

        print("\n=== 4. Validatsiya xatolari (422) ===")
        buzuq = [("manfiy to'lov", {"oylik_tolov": -5.0}),
                 ("noma'lum tarif", {"tarif": "Z"}),
                 ("ortiqcha maydon", {"yosh": 30}),
                 ("matn o'rniga son", {"shikoyatlar": "ko'p"})]
        for nom, ozgarish in buzuq:
            m = dict(yuk["mijozlar"][0], **ozgarish)
            r = mijoz.post("/predict", json={"mijozlar": [m]},
                           headers={"X-API-Key": kalit})
            x = r.json()["detail"][0]
            print(f"  {nom:<17} {r.status_code}  {x['loc'][-1]}: {x['type']}")
        r = mijoz.post("/predict", json={"mijozlar": []},
                       headers={"X-API-Key": kalit})
        print(f"  {'bo_sh ro_yxat':<17} {r.status_code}")

        print("\n=== 5. Partiya va so'rov loglari ===")
        katta = oy_partiyasi(7).iloc[:500]
        r = mijoz.post("/predict", json={"mijozlar": katta[
            ["mijoz_id"] + SON + KAT].to_dict("records")},
            headers={"X-API-Key": kalit})
        p = np.array([b["ketish_ehtimoli"] for b in r.json()["bashoratlar"]])
        print(f"  500 mijoz: o'rtacha ehtimol {p.mean():.4f}, "
              f"> 0.5: {int((p > 0.5).sum())}")
        jad = pd.read_sql("SELECT soat, versiya, COUNT(*) AS n, AVG(p) AS p"
                          " FROM sorovlar GROUP BY soat", xizmat.log_db)
        print(jad.round(4).to_string(index=False))
        xizmat.log_db.close()
        del os.environ["CHURN_API_KEY"]
    print("  ⭐ Kalit env dan, loglarda niqoblangan; har so'rov versiya bilan")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Kalitsiz ishga tushirish ===
  LOG INFO yuklandi: mijoz_ketishi v1 (nazorat OK)
  rad etildi: CHURN_API_KEY o'rnatilmagan

=== 2. Kalit muhit o'zgaruvchisidan ===
  LOG INFO API kaliti: ******** (uzunligi 32)
  /health  -> {'holat': 'ok', 'model_yuklangan': True}
  /version -> {'nom': 'mijoz_ketishi', 'versiya': 1, 'algoritm': 'LogReg', 'oylar': [1, 6]}

=== 3. /predict ===
  LOG WARNING rad etildi: kalit (yo'q)
  kalitsiz: 401 {'detail': "kalit noto'g'ri"}
  LOG WARNING rad etildi: kalit ********
  noto'g'ri kalit: 401
  to'g'ri kalit: 200
    {'mijoz_id': 'M0700000', 'ketish_ehtimoli': 0.0856}
    {'mijoz_id': 'M0700001', 'ketish_ehtimoli': 0.2437}
    {'mijoz_id': 'M0700002', 'ketish_ehtimoli': 0.0565}

=== 4. Validatsiya xatolari (422) ===
  manfiy to'lov     422  oylik_tolov: greater_than
  noma'lum tarif    422  tarif: literal_error
  ortiqcha maydon   422  yosh: extra_forbidden
  matn o'rniga son  422  shikoyatlar: int_parsing
  bo_sh ro_yxat     422

=== 5. Partiya va so'rov loglari ===
  500 mijoz: o'rtacha ehtimol 0.2023, > 0.5: 29
 soat  versiya   n      p
    1        1   3 0.1286
    2        1 500 0.2023
  ⭐ Kalit env dan, loglarda niqoblangan; har so'rov versiya bilan

Natija tahlili.

fastapi.testclient ni import qilish Starlette ning eskirganlik haqidagi ogohlantirishini stderr ga chiqaradi — shuning uchun import warnings.catch_warnings() ichida. httpx ning har so'rov haqidagi INFO loglari ham WARNING darajasiga tushirilgan, aks holda natija shovqinga to'lib ketadi.

1-bo'lim — CHURN_API_KEY o'rnatilmagan: model yuklandi (nazorat namunalari OK), lekin ilova yaratishdan bosh tortdi. Kalitsiz xizmatni ishga tushirishdan ko'ra, umuman ishga tushirmaslik xavfsizroq.

2-bo'lim — kalit muhit o'zgaruvchisidan olindi (sinovda secrets.token_hex(16) bilan yaratilgan, hech qayerga chop etilmagan). Logda faqat ******** va uzunligi. /version qaysi model javob berayotganini aytadi — monitoring va hodisalarni tekshirishda birinchi savol shu.

3-bo'lim — kalitsiz va noto'g'ri kalit bilan 401, to'g'ri kalit bilan 200 va har mijozga ehtimol.

4-bo'lim — Pydantic sxemasi to'rt xil buzuq kirishni 422 bilan rad etdi va har biri uchun qaysi maydon va qanday xato ekanini qaytardi: manfiy to'lov (greater_than), noma'lum tarif (literal_error), ortiqcha maydon (extra_forbidden — extra="forbid" tufayli), noto'g'ri tur (int_parsing). Bo'sh ro'yxat ham 422. Bu tekshiruvlar model ichiga "chiqindi" kirishining oldini oladi.

5-bo'lim — 500 mijozlik partiya. So'rov logida ikki yozuv guruhi: 3 ta va 500 ta mijoz, har biri versiya bilan. Aynan shu log 4-misolda belgilar bilan birlashtirilib, haqiqiy sifatni o'lchashga xizmat qiladi.

Misol 4 — Hayot sikli: drift, trigger, shadow, darvoza va versiya almashuvi

python
"""4-qadam: monitoring, drift, kechikkan belgilar, trigger, shadow, darvoza, versiya almashuvi."""

import json
import logging
import os
import secrets
import sqlite3
import sys
import tempfile
import warnings
from pathlib import Path
from typing import Literal

import joblib
import numpy as np
import pandas as pd
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, ConfigDict, Field
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    from fastapi.testclient import TestClient

SON = ["oylik_tolov", "muddat_oy", "qongiroqlar", "shikoyatlar"]
KAT = ["tarif", "hudud"]
USTUNLAR = ["mijoz_id", "oy"] + SON + KAT + ["ketdi"]
TARIFLAR = ["A", "B", "C"]
HUDUDLAR = ["Toshkent", "Samarqand", "Buxoro", "Namangan"]
DRIFT_OYI = 9
NOM = "mijoz_ketishi"

logging.basicConfig(stream=sys.stdout, level=logging.WARNING,
                    format="  LOG %(levelname)s %(message)s")

# ---------------- QAROR QOIDALARI (oldindan, natijani ko'rishdan oldin) ----------------
TRIGGER_AUC_TUSHISHI = 0.03       # AUC(oy) < reference - 0.03, 2 oy ketma-ket
PSI_OGOHLANTIRISH = 0.10          # erta ogohlantirish (qaror emas)
OYNA = 2                          # challenger: oxirgi 2 belgili oy (27.13)
DARVOZA_SEGMENT = 0.02            # har tarifda log loss yomonlashuvi <= 0.02


def oy_partiyasi(oy, n=1500):
    """Har oy - mijozlar kesimi; ketdi = keyingi oyda ketdimi (kechikib keladi)."""
    rng = np.random.default_rng(1000 + oy)
    d = oy >= DRIFT_OYI                  # raqobatchi arzon tarif chiqardi
    tarif = rng.choice(TARIFLAR, n, p=[0.35, 0.30, 0.35] if d else [0.5, 0.3, 0.2])
    hudud = rng.choice(HUDUDLAR, n, p=[0.4, 0.25, 0.2, 0.15])
    tolov = rng.lognormal(np.log(120), 0.35, n) * (1.15 if d else 1.0)
    muddat = rng.gamma(2.0, 12.0, n)
    qongiroq = rng.poisson(25, n)
    shikoyat = rng.poisson(0.8 if d else 0.5, n)
    z = (np.log(tolov) - np.log(120)) / 0.35
    f = (-1.8 + 1.0 * (muddat < 4) - 0.015 * muddat
         + 0.3 * (hudud == "Namangan")
         + (1.0 * (tarif == "A") - 0.3 * (tarif == "C") if d
            else 1.0 * (tarif == "C"))
         + (0.9 if d else 0.5) * shikoyat + (0.3 if d else 0.7) * z)
    ketdi = (rng.random(n) < 1 / (1 + np.exp(-f))).astype(int)
    return pd.DataFrame({
        "mijoz_id": [f"M{oy:02d}{i:05d}" for i in range(n)], "oy": oy,
        "oylik_tolov": tolov.round(2), "muddat_oy": muddat.round(1),
        "qongiroqlar": qongiroq, "shikoyatlar": shikoyat,
        "tarif": tarif, "hudud": hudud, "ketdi": ketdi})


# ---------------- model, profil, paket ----------------
def logreg():
    return Pipeline([
        ("tayyor", ColumnTransformer([
            ("son", StandardScaler(), SON),
            ("kat", OneHotEncoder(handle_unknown="ignore"), KAT)])),
        ("model", LogisticRegression(max_iter=2000))])


def profil(x):
    ichki = np.unique(np.quantile(x, np.linspace(0, 1, 11)[1:-1]))
    ulush = np.bincount(np.searchsorted(ichki, x, side="right"),
                        minlength=len(ichki) + 1) / len(x)
    return {"ichki": ichki.tolist(), "ulush": ulush.tolist()}


def psi(prof, x, eps=1e-4):
    ichki = np.array(prof["ichki"])
    q = np.bincount(np.searchsorted(ichki, x, side="right"),
                    minlength=len(ichki) + 1) / len(x)
    p = np.clip(np.array(prof["ulush"]), eps, None)
    q = np.clip(q, eps, None)
    return float(np.sum((q - p) * np.log(q / p)))


def psi_kat(ref, cur, eps=1e-4):
    p = np.clip([ref.get(t, 0) for t in TARIFLAR], eps, None)
    q = np.clip([np.mean(cur == t) for t in TARIFLAR], eps, None)
    return float(np.sum((q - p) * np.log(q / p)))


def paket_saqla(papka, versiya, df, oylar, ref_auc):
    pipe = logreg().fit(df, df.ketdi)
    p = pipe.predict_proba(df)[:, 1]
    nazorat = df.iloc[:200]
    paket = {"format": 1, "pipeline": pipe, "sxema": {"son": SON, "kat": KAT},
             "metadata": {"nom": NOM, "versiya": versiya, "oylar": oylar,
                          "ref_auc": ref_auc},
             "profil": {"oylik_tolov": profil(df.oylik_tolov.to_numpy()),
                        "shikoyatlar": profil(df.shikoyatlar.to_numpy()),
                        "tarif": {t: float(np.mean(df.tarif == t))
                                  for t in TARIFLAR},
                        "p": profil(p)},
             "nazorat": {"X": nazorat[SON + KAT].to_dict("list"),
                         "p": pipe.predict_proba(nazorat)[:, 1].tolist()}}
    artefakt = f"{NOM}-v{versiya}.joblib"
    joblib.dump(paket, papka / artefakt)
    return artefakt


# ---------------- registr ----------------
RUXSAT = {("None", "Staging"), ("Staging", "Production"),
          ("Production", "Archived"), ("Staging", "Archived"),
          ("Archived", "Production")}                      # rollback


class Registr:
    def __init__(self, yol):
        self.db = sqlite3.connect(yol, check_same_thread=False)
        self.db.executescript(
            "CREATE TABLE versiyalar (nom TEXT, versiya INT, bosqich TEXT,"
            " artefakt TEXT, PRIMARY KEY (nom, versiya));"
            "CREATE TABLE audit (id INTEGER PRIMARY KEY AUTOINCREMENT,"
            " oy INT, versiya INT, eski TEXT, yangi TEXT, sabab TEXT);")

    def qosh(self, versiya, artefakt):
        with self.db:
            self.db.execute("INSERT INTO versiyalar VALUES (?,?,?,?)",
                            (NOM, versiya, "None", artefakt))

    def otkaz(self, oy, versiya, yangi, sabab):
        eski = self.db.execute("SELECT bosqich FROM versiyalar WHERE"
                               " versiya=?", (versiya,)).fetchone()[0]
        if (eski, yangi) not in RUXSAT:
            raise ValueError(f"{eski} -> {yangi}")
        with self.db:
            self.db.execute("UPDATE versiyalar SET bosqich=? WHERE versiya=?",
                            (yangi, versiya))
            self.db.execute("INSERT INTO audit (oy, versiya, eski, yangi,"
                            " sabab) VALUES (?,?,?,?,?)",
                            (oy, versiya, eski, yangi, sabab))

    def bosqichda(self, bosqich):
        r = self.db.execute("SELECT versiya, artefakt FROM versiyalar WHERE"
                            " bosqich=?", (bosqich,)).fetchone()
        return r


# ---------------- xizmat (3-qadam + shadow) ----------------
class Mijoz(BaseModel):
    model_config = ConfigDict(extra="forbid")
    mijoz_id: str = Field(min_length=1, max_length=32)
    oylik_tolov: float = Field(gt=0, lt=2000)
    muddat_oy: float = Field(ge=0, le=600)
    qongiroqlar: int = Field(ge=0, le=10_000)
    shikoyatlar: int = Field(ge=0, le=100)
    tarif: Literal["A", "B", "C"]
    hudud: Literal["Toshkent", "Samarqand", "Buxoro", "Namangan"]


class Sorov(BaseModel):
    oy: int = Field(ge=1, le=99)
    mijozlar: list[Mijoz] = Field(min_length=1, max_length=5000)


def paket_yukla(papka, artefakt):
    paket = joblib.load(papka / artefakt)
    X = pd.DataFrame(paket["nazorat"]["X"])
    if np.max(np.abs(paket["pipeline"].predict_proba(X)[:, 1]
                     - np.array(paket["nazorat"]["p"]))) > 1e-9:
        raise RuntimeError("nazorat namunalari mos emas")
    return paket


class Xizmat:
    def __init__(self, papka, registr):
        self.papka, self.registr = papka, registr
        self.db = sqlite3.connect(papka / "sorovlar.sqlite",
                                  check_same_thread=False)
        self.db.execute("CREATE TABLE sorovlar (oy INT, mijoz_id TEXT,"
                        " versiya INT, p REAL, shadow_versiya INT,"
                        " shadow_p REAL)")
        self.yangila()

    def yangila(self):
        v, a = self.registr.bosqichda("Production")
        self.versiya, self.paket = v, paket_yukla(self.papka, a)
        s = self.registr.bosqichda("Staging")
        self.shadow = (s[0], paket_yukla(self.papka, s[1])) if s else None

    def bashorat(self, oy, mijozlar):
        df = pd.DataFrame([m.model_dump() for m in mijozlar])
        p = self.paket["pipeline"].predict_proba(df)[:, 1]
        sv, sp = None, [None] * len(df)
        if self.shadow:                                  # javob foydalanuvchiga
            try:                                         # bormaydi, faqat log
                sv = self.shadow[0]
                sp = self.shadow[1]["pipeline"].predict_proba(df)[:, 1].tolist()
            except Exception:                            # shadow xatosi xizmatni
                sv, sp = None, [None] * len(df)          # to'xtatmaydi
        with self.db:
            self.db.executemany("INSERT INTO sorovlar VALUES (?,?,?,?,?,?)",
                                zip([oy] * len(df), df.mijoz_id,
                                    [self.versiya] * len(df), p.tolist(),
                                    [sv] * len(df), sp))
        return p


def ilova_yarat(xizmat):
    kalit = os.environ["CHURN_API_KEY"]
    app = FastAPI()

    def tekshir(k):
        if not secrets.compare_digest((k or "").encode(), kalit.encode()):
            raise HTTPException(status_code=401)

    @app.get("/version")
    def version():
        return {"nom": NOM, "versiya": xizmat.versiya,
                "shadow": xizmat.shadow[0] if xizmat.shadow else None}

    @app.post("/predict")
    def predict(sorov: Sorov, x_api_key: str | None = Header(default=None)):
        tekshir(x_api_key)
        p = xizmat.bashorat(sorov.oy, sorov.mijozlar)
        return {"versiya": xizmat.versiya, "p": [round(float(q), 4) for q in p]}

    @app.post("/admin/yangila")
    def yangila(x_api_key: str | None = Header(default=None)):
        tekshir(x_api_key)
        xizmat.yangila()
        return {"versiya": xizmat.versiya}

    return app


def ll(y, p):
    p = np.clip(p, 1e-6, 1 - 1e-6)
    return -(y * np.log(p) + (1 - y) * np.log(1 - p))


def darvoza_tekshir(j):
    """Shadow oyi: bir xil so'rovlarda champion (p) va challenger (shadow_p)."""
    y = j.ketdi.to_numpy()
    a, b = j.p.to_numpy(), j.shadow_p.to_numpy(dtype=float)
    rng = np.random.default_rng(0)
    farq = roc_auc_score(y, b) - roc_auc_score(y, a)
    bs = []
    for _ in range(300):
        i = rng.integers(0, len(y), len(y))
        bs.append(roc_auc_score(y[i], b[i]) - roc_auc_score(y[i], a[i]))
    se = float(np.std(bs, ddof=1))
    d = ll(y, b) - ll(y, a)
    se_d = d.std(ddof=1) / np.sqrt(len(d))
    seg = {t: float(d[j.tarif.to_numpy() == t].mean()) for t in TARIFLAR}
    return {
        "AUC sezilarli yaxshi (> 2*SE)": (farq > 2 * se,
                                          f"{farq:+.4f}, SE {se:.4f}"),
        "log loss sezilarli yaxshi": (d.mean() < -2 * se_d,
                                      f"{d.mean():+.4f}, SE {se_d:.4f}"),
        "har tarifda yomonlashuv <= 0.02": (
            max(seg.values()) <= DARVOZA_SEGMENT,
            str({k: round(v, 4) for k, v in seg.items()})),
        "shadow xatolari = 0": (bool(np.isfinite(b).all()),
                                f"{int((~np.isfinite(b)).sum())} ta"),
    }


def main() -> None:
    warnings.simplefilter("ignore")
    os.environ["CHURN_API_KEY"] = secrets.token_hex(16)     # sinov kaliti
    H = {"X-API-Key": os.environ["CHURN_API_KEY"]}
    with tempfile.TemporaryDirectory() as tmp:
        papka = Path(tmp)
        oylar = {o: oy_partiyasi(o) for o in range(1, 15)}   # manba (1-qadam)
        registr = Registr(papka / "mlops.sqlite")
        tr = pd.concat([oylar[o] for o in range(1, 6)])
        ref_auc = roc_auc_score(oylar[6].ketdi, logreg().fit(
            tr, tr.ketdi).predict_proba(oylar[6])[:, 1])
        v1 = pd.concat([oylar[o] for o in range(1, 7)])       # 2-qadam
        registr.qosh(1, paket_saqla(papka, 1, v1, [1, 6], ref_auc))
        registr.otkaz(6, 1, "Staging", "yangi nomzod")
        registr.otkaz(6, 1, "Production", "darvoza OK (2-qadam)")
        xizmat = Xizmat(papka, registr)
        mijoz = TestClient(ilova_yarat(xizmat))              # 3-qadam

        print("=== 1. Boshlang'ich holat ===")
        print(f"  /version -> {mijoz.get('/version').json()}")
        print(f"  reference AUC (6-oy): {ref_auc:.4f}; trigger: AUC < "
              f"{ref_auc - TRIGGER_AUC_TUSHISHI:.4f} 2 oy ketma-ket")

        print("\n=== 2. Oylar: oy boshida (oy-1) belgilari, keyin so'rovlar ===")
        print(f"  {'oy':>3} {'belgi':>5} {'AUC':>6} {'xizmat':>6} "
              f"{'PSI tolov':>9} {'PSI tarif':>9} {'PSI p':>6}  harakat")
        past_ketma_ket, oy_auc, darvoza = 0, {}, None
        for oy in range(7, 16):
            harakat, auc_txt = [], "-"
            # A) (oy-1) belgilari keldi -> xizmat bergan bashoratlar bilan
            log = pd.read_sql("SELECT mijoz_id, p, shadow_p FROM sorovlar"
                              " WHERE oy = ?", xizmat.db, params=(oy - 1,))
            if len(log):
                j = log.merge(oylar[oy - 1][["mijoz_id", "ketdi", "tarif"]],
                              on="mijoz_id")
                oy_auc[oy - 1] = roc_auc_score(j.ketdi, j.p)
                auc_txt = f"{oy_auc[oy - 1]:.4f}"
                past = oy_auc[oy - 1] < ref_auc - TRIGGER_AUC_TUSHISHI
                past_ketma_ket = past_ketma_ket + 1 if past else 0
                if j.shadow_p.notna().all() and xizmat.shadow:
                    darvoza = darvoza_tekshir(j)
                    ok = all(v for v, _ in darvoza.values())
                    harakat.append(f"darvoza {'OK' if ok else 'RAD'}")
                    if ok:
                        registr.otkaz(oy, 1, "Archived", "v2 ga almashdi")
                        registr.otkaz(oy, 2, "Production", "darvoza OK")
                        mijoz.post("/admin/yangila", headers=H)
                        harakat.append("v2 -> Production")
                elif past_ketma_ket >= 2 and xizmat.shadow is None \
                        and xizmat.versiya == 1:
                    oyna = list(range(oy - OYNA, oy))
                    yangi = pd.concat([oylar[o] for o in oyna])
                    registr.qosh(2, paket_saqla(papka, 2, yangi,
                                                [oyna[0], oyna[-1]],
                                                oy_auc[oy - 1]))
                    registr.otkaz(oy, 2, "Staging", "trigger: AUC 2 oy past")
                    mijoz.post("/admin/yangila", headers=H)
                    harakat.append(f"TRIGGER: v2 ({oyna[0]}-{oyna[-1]} oy)"
                                   f" -> shadow")
            # B) shu oy so'rovlari (15-oy so'rovlari bu simulyatsiyada yo'q)
            if oy <= 14:
                prof = xizmat.paket["profil"]       # reference = JORIY model
                df = oylar[oy]
                r = mijoz.post("/predict", headers=H, json={
                    "oy": oy,
                    "mijozlar": df[["mijoz_id"] + SON + KAT].to_dict("records")})
                p = np.array(r.json()["p"])
                psi_t = psi(prof["oylik_tolov"], df.oylik_tolov.to_numpy())
                psi_k = psi_kat(prof["tarif"], df.tarif.to_numpy())
                psi_p = psi(prof["p"], p)
                if max(psi_t, psi_k, psi_p) > PSI_OGOHLANTIRISH:
                    harakat.insert(0, "drift ogohl.")
                ustunlar = (f"{'v' + str(r.json()['versiya']):>6} {psi_t:>9.3f}"
                            f" {psi_k:>9.3f} {psi_p:>6.3f}")
            else:
                ustunlar = f"{'-':>6} {'-':>9} {'-':>9} {'-':>6}"
            print(f"  {oy:>3} {oy - 1:>5} {auc_txt:>6} {ustunlar}  "
                  f"{'; '.join(harakat)}")

        print("\n=== 3. Darvoza tafsiloti (shadow oyi, bir xil so'rovlar) ===")
        for q, (ok, izoh) in darvoza.items():
            print(f"  {q:<32} {'OK' if ok else 'RAD'}  {izoh}")

        print("\n=== 4. Almashtirish foydasi: v2 xizmat qilgan oylar ===")
        v1_pipe = paket_yukla(papka, f"{NOM}-v1.joblib")["pipeline"]
        for oy in [12, 13, 14]:
            v1_auc = roc_auc_score(oylar[oy].ketdi,
                                   v1_pipe.predict_proba(oylar[oy])[:, 1])
            print(f"  {oy}-oy AUC: xizmat (v2) {oy_auc[oy]:.4f}, "
                  f"agar v1 qolganida {v1_auc:.4f}")

        print("\n=== 5. Rollback imkoniyati ===")
        registr.otkaz(15, 2, "Archived", "rollback sinovi")
        registr.otkaz(15, 1, "Production", "rollback sinovi")
        mijoz.post("/admin/yangila", headers=H)
        print(f"  rollback  -> /version {mijoz.get('/version').json()}")
        registr.otkaz(15, 1, "Archived", "sinov tugadi")
        registr.otkaz(15, 2, "Production", "sinov tugadi")
        mijoz.post("/admin/yangila", headers=H)
        print(f"  qaytarildi -> /version {mijoz.get('/version').json()}")
        print(pd.read_sql("SELECT oy, versiya, eski, yangi, sabab FROM audit",
                          registr.db).to_string(index=False))
        xizmat.db.close()
        registr.db.close()
    del os.environ["CHURN_API_KEY"]
    print("  ⭐ Qarorlar oldindan yozilgan qoidalar bilan, har qadam auditda")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Boshlang'ich holat ===
  /version -> {'nom': 'mijoz_ketishi', 'versiya': 1, 'shadow': None}
  reference AUC (6-oy): 0.7263; trigger: AUC < 0.6963 2 oy ketma-ket

=== 2. Oylar: oy boshida (oy-1) belgilari, keyin so'rovlar ===
   oy belgi    AUC xizmat PSI tolov PSI tarif  PSI p  harakat
    7     6      -     v1     0.005     0.002  0.004
    8     7 0.7230     v1     0.004     0.002  0.005
    9     8 0.7300     v1     0.145     0.131  0.270  drift ogohl.
   10     9 0.6285     v1     0.171     0.173  0.363  drift ogohl.
   11    10 0.6212     v1     0.178     0.142  0.331  drift ogohl.; TRIGGER: v2 (9-10 oy) -> shadow
   12    11 0.6291     v2     0.013     0.001  0.012  darvoza OK; v2 -> Production
   13    12 0.7779     v2     0.014     0.002  0.011
   14    13 0.7791     v2     0.008     0.006  0.011
   15    14 0.7436      -         -         -      -

=== 3. Darvoza tafsiloti (shadow oyi, bir xil so'rovlar) ===
  AUC sezilarli yaxshi (> 2*SE)    OK  +0.1395, SE 0.0164
  log loss sezilarli yaxshi        OK  -0.1097, SE 0.0125
  har tarifda yomonlashuv <= 0.02  OK  {'A': -0.1483, 'B': -0.0408, 'C': -0.1336}
  shadow xatolari = 0              OK  0 ta

=== 4. Almashtirish foydasi: v2 xizmat qilgan oylar ===
  12-oy AUC: xizmat (v2) 0.7779, agar v1 qolganida 0.6262
  13-oy AUC: xizmat (v2) 0.7791, agar v1 qolganida 0.6223
  14-oy AUC: xizmat (v2) 0.7436, agar v1 qolganida 0.6332

=== 5. Rollback imkoniyati ===
  rollback  -> /version {'nom': 'mijoz_ketishi', 'versiya': 1, 'shadow': None}
  qaytarildi -> /version {'nom': 'mijoz_ketishi', 'versiya': 2, 'shadow': None}
 oy  versiya       eski      yangi                  sabab
  6        1       None    Staging           yangi nomzod
  6        1    Staging Production   darvoza OK (2-qadam)
 11        2       None    Staging trigger: AUC 2 oy past
 12        1 Production   Archived         v2 ga almashdi
 12        2    Staging Production             darvoza OK
 15        2 Production   Archived        rollback sinovi
 15        1   Archived Production        rollback sinovi
 15        1 Production   Archived           sinov tugadi
 15        2   Archived Production           sinov tugadi
  ⭐ Qarorlar oldindan yozilgan qoidalar bilan, har qadam auditda

Natija tahlili.

Qaror qoidalari fayl boshida, natijadan oldin yozilgan: PSI ogohlantirishi 0.10, trigger — AUC reference dan 0.03 pastda ikki oy ketma-ket, challenger oynasi — oxirgi 2 belgili oy, segment chegarasi 0.02.

2-bo'lim — oyma-oy jadval. Har oy boshida oldingi oy belgilari keladi va xizmat haqiqatan bergan bashoratlar bilan birlashtiriladi, keyin shu oy so'rovlari keladi.

  • 7-8 oy — hammasi tinch: PSI 0.002-0.005, AUC 0.7230 va 0.7300 (reference 0.7263).
  • 9-oy — raqobatchi tarifi chiqdi. Belgilar hali yo'q, lekin erta ogohlantirish darhol yondi: to'lov PSI 0.145, tarif 0.131, bashorat PSI 0.270. Bu ogohlantirish, qaror emas — 27.12 dagi kabi kirish drifti har doim ham sifatni tushirmaydi.
  • 10-oy — 9-oy belgilari keldi: AUC 0.6285 — chegaradan (0.6963) past. Birinchi past oy.
  • 11-oy — AUC 0.6212, ikkinchi past oy → trigger. Challenger (v2) 9-10 oylarda o'qitildi, Staging ga o'tdi, xizmat uni shadow rejimda yukladi. 11-oy so'rovlariga javobni hali ham v1 beradi, v2 ning bashoratlari faqat logga yoziladi.
  • 12-oy — 11-oy belgilari keldi: v1 AUC 0.6291. Bir xil so'rovlarda v2 bilan taqqoslash — darvoza OK. v2 Production ga, v1 Archived ga, xizmat /admin/yangila bilan yangi versiyani yukladi: 12-oy so'rovlariga v2 javob beryapti. Diqqat: PSI birdan 0.001-0.013 ga tushdi — reference endi v2 ning o'quv taqsimoti (9-10 oy), ya'ni "yangi normal".
  • 13-15 oy boshlari — v2 xizmat qilgan 12, 13 va 14-oylar belgilari keldi: AUC 0.7779, 0.7791, 0.7436 — reference dan ham yuqori, chunki yangi rejimda shikoyat va tarif signali kuchliroq.

3-bo'lim — darvoza tafsiloti (11-oy, bir xil 1500 so'rov): AUC farqi +0.1395 (SE 0.0164), log loss -0.1097 (SE 0.0125), uchala tarifda ham yaxshilanish, shadow xatolari yo'q. Eng katta yaxshilanish A tarifida (-0.1483) — aynan u yerda dunyo o'zgargan edi.

4-bo'lim — foyda halol o'lchandi: agar v1 qolganida 12-14 oylarda AUC 0.62-0.63 bo'lardi; v2 0.74-0.78 berdi.

5-bo'lim — rollback: ikki registr o'tishi va /admin/yangila — /version darhol 1 ni ko'rsatdi; sinovdan keyin v2 qaytarildi. Audit jadvalida butun tarix bor: kim, qachon (oy), qaysi versiya, qaysi bosqichdan qaysi bosqichga va nima sababdan.

Halol cheklovlar: ma'lumot sintetik va drift keskin (to'satdan); haqiqiy hayotda drift aralash va shovqinli, trigger chegaralari tarixiy ma'lumotda kalibrlanadi 27.12-bob. Kechikish zanjiri tufayli drift boshlanganidan yangi model ishga tushguncha 3 oy o'tdi (9 → 12) — bu tizimning xatosi emas, kechikkan belgilarning tabiiy narxi.


5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Sxema to'g'ri bo'lsa — ma'lumot to'g'ri" Birlik xatosini faqat diapazon ushlaydi
"Buzuq partiyani o'chirib yuboramiz" Karantin: sabab bilan saqlanadi, tuzatilgach qayta yuklanadi
"Quvurni bir marta ishga tushiramiz" Qayta ishga tushirish odatiy hol — idempotentlik shart
"Tasodifiy bo'lish yetarli" Vaqtli ma'lumotda vaqt bo'yicha bo'lish
"Model faylini saqlasak yetadi" Pipeline + sxema + metadata + profil + nazorat
"Kalitni konfig faylga yozamiz" Faqat muhit o'zgaruvchisi / secret; loglarda niqob
"Validatsiya AUC — ishlab chiqarish sifati" Haqiqiy sifat — log + kechikkan belgilar
"Drift ogohlantirishi — darhol qayta o'qitish" Ogohlantirish; qaror — oldindan yozilgan trigger
"Yangi model yaxshiroq — darhol almashtiramiz" Shadow, bir xil so'rovlar, darvoza
"Model almashsa monitoring o'zgarmaydi" Reference profil ham yangi modelniki bo'ladi

6. Keng tarqalgan xatolar va yechimlari

1. Validatsiyasiz yuklash

python
df.to_sql("mijozlar", db, if_exists="append")                  # ⚠️
if not validatsiya(df): yukla(db, oy, df)                       # ✅

2. Idempotent bo'lmagan yuklash

python
db.executemany("INSERT INTO mijozlar ...", qatorlar)            # ⚠️ har safar qo'shadi
if xesh != eski_xesh: yukla(db, oy, df)     # DELETE + INSERT tranzaksiyada  # ✅

3. Tasodifiy bo'lish

python
train_test_split(df, test_size=0.2)                             # ⚠️ kelajak o'quvga tushadi
tr, va = df[df.oy <= 5], df[df.oy == 6]                         # ✅

4. Kalit kodda

python
KALIT = "abc123"                                                # ⚠️
kalit = os.environ["CHURN_API_KEY"]                             # ✅

5. Bashoratni loglamaslik

python
javob = {"p": p}                                                # ⚠️ sifatni o'lchab bo'lmaydi
log_yoz(oy, mijoz_id, versiya, p); javob = {"p": p}             # ✅

6. Eski reference

python
prof = v1_paket["profil"]            # butun hayot davomida       # ⚠️
prof = xizmat.paket["profil"]        # joriy Production           # ✅

7. Bitta past oyda qayta o'qitish

python
if auc < chegara: qayta_orgat()                                 # ⚠️ shovqin
if past_ketma_ket >= 2: qayta_orgat()                           # ✅ oldindan kelishilgan

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 27.1-27.13-darslar (o'tilgan): Butun qism
  • 19-qism (o'tilgan): Pipeline, ColumnTransformer, joblib
  • 18-qism (o'tilgan): Juftlashgan taqqoslash va vaqt bo'yicha baholash
  • 11-qism (o'tilgan): Gipoteza testlari va A/B — onlayn tajriba uchun
  • Maxsus mavzular qismi: Yangi turdagi ma'lumot va vazifalarda xuddi shu skelet — baholash dizayni, paket, xizmat va monitoring
  • Loyihalar va karyera qismi: Portfolio loyihasi sifatida — to'liq ishlaydigan ML xizmati

8. Eng yaxshi amaliyotlar

  1. Validatsiya — darvoza; buzuq partiya — karantin.

  2. Yuklash idempotent va atomik.

  3. Vaqt bo'yicha bo'lish, juftlashgan qaror, eng sodda munosib model.

  4. Paket o'zini tekshiradi; registrda audit.

  5. Sir faqat muhitdan; loglarda niqob.

  6. Har bashorat versiya bilan loglanadi.

  7. Qaror qoidalari oldindan, kodda (yoki versiyalangan konfigda).

  8. Shadow, darvoza, rollback — avtomatik va sinalgan.


9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # birlik xatosini qaysi tekshiruv ushlaydi?
2.  # quvur ikki marta ishga tushsa, qatorlar soni?
3.  # yuklash o'rtasida uzilsa?
4.  # nega vaqt bo'yicha bo'lish?
5.  # yurish_id nimadan hisoblanadi?
6.  # paketda nechta qism va nima uchun nazorat namunalari?
7.  # kalit o'rnatilmagan bo'lsa xizmat?
8.  # extra="forbid" nima beradi?
9.  # drift 9-oyda - trigger eng erta qachon?
10. # shadow oyida foydalanuvchi qaysi versiya javobini oladi?
11. # model almashgach PSI reference?
12. # rollback qanday bajariladi?
Javoblar
  1. Diapazon (sxema va tur emas)
  2. O'zgarmaydi — idempotent
  3. Tranzaksiya qaytariladi, eski ma'lumot butun
  4. Ishlab chiqarishda kelajakni bashorat qilamiz; tasodifiy bo'lish optimistik
  5. Model, parametrlar, ma'lumot xeshi, kod versiyasi
  6. Beshta; yuklashda aynan shu chiqish qayta olinishini tekshirish uchun
  7. Ishga tushmaydi
  8. Noma'lum maydon — 422
  9. 11-oy (belgi 1 oy kechikadi + 2 oy qoidasi)
  10. Production (v1)
  11. Yangi modelning o'quv taqsimoti
  12. Registrda Archived -> Production + /admin/yangila, auditda

Vazifa 2: Xatolarni tuzating

python
1.  df.to_sql("mijozlar", db, if_exists="append")

2.  tr, va = train_test_split(df, test_size=0.2)

3.  KALIT = "abc123"

4.  prof = v1_paket["profil"]   # hamma oylar uchun

5.  if auc < chegara: registr.production = orgat(oxirgi_oy)
Javoblar
python
1.  if not validatsiya(df): yukla(db, oy, df)   # xesh + tranzaksiya

2.  tr, va = df[df.oy <= 5], df[df.oy == 6]

3.  kalit = os.environ["CHURN_API_KEY"]

4.  prof = xizmat.paket["profil"]   # joriy Production

5.  if past_ketma_ket >= 2: registr.otkaz(oy, 2, "Staging", "trigger")  # -> shadow -> darvoza

Vazifa 3: Quvur

Modellang (1-misol asosida):

  1. Yangi tekshiruv: har oyda ketish ulushi 0.05-0.60 oralig'ida bo'lsin
  2. Karantin sababini alohida karantin jadvaliga yozing
  3. Partiya kechikib kelsa (6-oydan keyin 4-oy) — quvur to'g'ri ishlaydimi?
  4. Xeshni ustun tartibiga bog'liq bo'lmaydigan qiling

Vazifa 4: O'qitish

Modellang (2-misol asosida):

  1. Vaqt bo'yicha 3 ta fold (1-3→4, 1-4→5, 1-5→6) bilan taqqoslang
  2. HistGB giperparametrlarini 3 xil qilib, jurnalda yurishlarni solishtiring
  3. Kalibrovkani darvozaga qo'shing
  4. Model kartasini JSON sifatida paketga qo'shing

Vazifa 5: Xizmat

Modellang (3-misol asosida):

  1. Kalit tekshiruvini Depends bilan bog'liqlikka chiqaring
  2. /predict javobiga so_rov_id qo'shing va logga yozing
  3. So'rov hajmi limiti (5000) oshsa nima bo'lishini sinang
  4. Ikki kalitni (eski va yangi) bir vaqtda qabul qilib, kalit almashtirishni (rotation) modellang

Vazifa 6: Hayot sikli

Modellang (4-misol asosida):

  1. Trigger qoidasini "bir oy" qiling — 7-8 oylarda yolg'on trigger bo'ladimi?
  2. Challenger oynasini 4 oy (7-10) qiling — darvoza natijasi qanday o'zgaradi?
  3. Challenger ni ataylab buzing (bir tarif ma'lumotini tashlab) — darvoza qaysi qoida bilan rad etadi?
  4. Proksi belgi qo'shing: 15 kunlik faollik — trigger necha oy erta yonadi?

Vazifa 7: O'ylash

Rahbar so'radi: "Tizim drift 9-oyda boshlanganini o'sha oyda ko'rgan ekan (PSI ogohlantirishi). Nega yangi model faqat 12-oyda ishga tushdi? Uch oy yo'qotdik. Keyingi safar ogohlantirish yonishi bilan darhol qayta o'qitib, almashtiraylik."

Javob

Qisqa javob: ogohlantirish bilan darhol almashtirish mumkin emas — o'sha paytda yangi rejim haqida belgili ma'lumot yo'q. Lekin tayyorgarlikni ertaroq boshlash va kechikishni qisqartirish mumkin.

1. 9-oyda nima ma'lum edi. Faqat kirishlar va bashoratlar: to'lov, tarif va bashorat taqsimoti siljigan (PSI 0.145, 0.131, 0.270). Model sifati tushdimi — noma'lum: 27.12 da ko'rdikki, kirish drifti ba'zan sifatga umuman ta'sir qilmaydi. Yangi modelni o'qitish uchun esa yangi rejimdagi belgilar kerak — 9-oy belgilari faqat 10-oy boshida keladi.

2. Agar 9-oyda darhol qayta o'qitsak. Yangi model 1-8 oylarda (hammasi eski rejim) o'qitilgan bo'lardi — u v1 dan deyarli farq qilmaydi. "Qayta o'qitildi" degan belgi qo'yiladi, lekin muammo hal bo'lmaydi.

3. Kechikish zanjiri.

python
# 9-oy:  drift boshlandi, PSI ogohlantirishi       <- tayyorgarlik boshlanishi mumkin
# 10-oy: 9-oy belgilari -> 1-past oy
# 11-oy: 10-oy belgilari -> 2-past oy -> TRIGGER, v2 (9-10 oy) -> shadow
# 12-oy: 11-oy belgilari -> darvoza OK -> v2 Production

4. Qanday qisqartirish mumkin (halol yo'llar).

  • Bir oylik qoida: trigger 10-oyda yonadi. Narxi — yolg'on triggerlar ko'payadi (bitta shovqinli oy); buni tarixiy ma'lumotda o'lchab qaror qilish kerak (Vazifa 6.1).
  • Proksi belgi: 30 kunlik ketish o'rniga, masalan, 10 kunlik faollik pasayishi — sifat signali 2-3 hafta erta keladi.
  • Shadow ni qisqartirish: shadow uchun to'liq oy emas, belgilari tez keladigan kichik namuna.
  • Vaqtinchalik himoya: ogohlantirish paytida saqlash bo'limiga "A tarifi mijozlariga ham e'tibor bering" kabi qo'lda qoida — model almashguncha.

5. Nima qilmaslik kerak. Shadow va darvozani o'tkazib yuborish. 27.13-darsda ko'rdikki, quvur xatosi bilan o'qitilgan model umumiy metrikada yaxshi ko'rinib, bir segmentda halokatli bo'lishi mumkin; xizmat kodi xatosi esa faqat nazorat namunalari bilan ko'rinadi.

Rahbarga javob: "9-oyda biz faqat 'nimadir o'zgardi' degan signalni ko'rdik; yangi modelni o'qitish uchun yangi dunyodan belgilar kerak edi, ular 10-oyda kela boshladi. Keyingi safar uchun uchta yaxshilanish taklif qilaman: proksi belgi, tarixda kalibrlangan bir oylik trigger va ogohlantirish paytidagi vaqtinchalik biznes qoidasi. Bu kechikishni qisqartiradi, xavfsizlik tekshiruvlarini esa saqlab qoladi; qanchaga qisqarishini tarixiy ma'lumotda xuddi shu simulyatsiya bilan o'lchab, keyingi hisobotda raqam bilan beraman."

Nimani mustahkamlaydi: 2.1, 2.2, 2.5-bo'limlar.


Xulosa

Bu darsda to'liq ML xizmatini qurdik: xom oylik partiyadan tortib o'zini kuzatadigan va xavfsiz yangilanadigan xizmatgacha.

Eng muhim uch fikr:

  1. Ma'lumot va model — shartnomalar bilan himoyalangan zanjir. 1-misolda validatsiya birlik xatosini (maksimum 316510) va tushib qolgan ustunni ushlab, karantinga yubordi; quvur ikki marta ishga tushganda qatorlar takrorlanmadi (6000 -> 6000), yuklash uzilganda tranzaksiya eski ma'lumotni saqlab qoldi. 2-misolda vaqt bo'yicha bo'lish va juftlashgan bootstrap LogReg ni tanladi (HistGB farqi -0.0066, SE 0.0090 — sezilarli emas), paket esa Pipeline, sxema, metadata, reference profil va nazorat namunalarini birga olib yuradi va yuklanganda o'zini tekshiradi.

  2. Xizmat — himoyalangan va kuzatiladigan. 3-misolda kalit faqat muhitdan olindi (kalitsiz xizmat ishga tushmadi), loglarda niqoblandi; Pydantic to'rt xil buzuq kirishni aniq xabar bilan 422 qildi; har bashorat versiya bilan loglandi — bu log keyin haqiqiy sifatni o'lchashning yagona manbai bo'ldi.

  3. Hayot sikli oldindan yozilgan qoidalar bilan boshqarildi. 4-misolda 9-oyda PSI erta ogohlantirish berdi, kechikkan belgilar 10 va 11-oyda AUC tushishini (0.6285, 0.6212) ko'rsatdi, trigger v2 ni shadow ga chiqardi, bir xil so'rovlarda darvoza (+0.1395, SE 0.0164) uni tasdiqladi va registr orqali xizmat yangi versiyani yukladi (/version → 2). v2 xizmat qilgan oylarda AUC 0.74-0.78, v1 qolganida 0.62-0.63 bo'lardi. Rollback — ikki registr o'tishi va bitta chaqiruv.

27-qism xulosasi. Bu qismda model "ishlaydigan noutbuk" dan "ishlaydigan xizmat" ga aylandi. Reproduksiya va muhit natijani qayta olishni kafolatladi; ma'lumot quvuri — validatsiya, karantin, idempotent va atomik yuklashni; eksperiment kuzatuvi va registr — har qaror va har versiyaning tarixini; paketlash — tayyorlash va modelni bitta, o'zini tekshiradigan artefaktga birlashtirishni. FastAPI xizmati, uni mustahkamlash va testlash, Docker va deploy strategiyalari modelni foydalanuvchiga xavfsiz yetkazdi. Monitoring, drift aniqlash, qayta o'qitish va A/B test esa eng muhim haqiqatni ko'rsatdi: deploy — oxiri emas, boshlanish; dunyo o'zgaradi va tizim buni sezishi, o'lchashi va oldindan yozilgan qoidalar bilan xavfsiz javob berishi kerak. Butun qism davomida bitta tamoyil takrorlandi: qaror raqam bilan, oldindan belgilangan qoida bilan va halol — bu oldingi qismlardagi juftlashgan taqqoslash va "eng sodda munosib model" g'oyasining ishlab chiqarishdagi davomi.

Keyingi qism — Maxsus mavzular: kursning asosiy yo'lidan tashqarida qolgan, lekin amalda tez-tez uchraydigan vazifa va ma'lumot turlari. Ularning har birida shu kursda qurilgan skelet — baholash dizayni, juftlashgan taqqoslash, "eng sodda munosib model" qoidasi, paket, xizmat va monitoring — yangi sharoitga qo'llanadi.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
27.14-dars: Amaliyot — mijoz ketishini bashorat qilish xizmati — IlmHamroh