IlmHamroh
Data Science va sun'iy intellekt/ML asoslari10/10-dars23 daqiqa
Mundarija (22)

12.10-dars: Amaliyot — to'liq ML loyihasi

12-QISM — MACHINE LEARNING ASOSLARI · 10-dars

1. Kirish va motivatsiya

Bu dars — 12-qismning yakuni: butun blokni bitta loyihada birlashtiramiz. Vazifa real: onlayn ta'lim platformasi obunachilarining ketishini (churn) bashorat qilish va ushlab qolish kampaniyasini kimga yo'naltirishni hal qilish.

Loyiha oqimi: vazifani qo'yish va vaqt kesimi 12.2-bob, guruh va vaqt bo'yicha ajratish 12.3-bob, pipeline bilan tayyorlash 12.9-bob, baza va model nomzodlari 12.6-bob, overfitting nazorati (12.4-12.5), klassifikatsiya metrikalari va chegara 12.7-bob, kutilgan foyda hisobi va yakuniy hisobot. Har qadamda 12-qismdagi qoidalar qo'llanadi va har qaror asoslanadi.

Real vaziyat. Platformada oyiga 3.4% obunachi ketadi. Marketing byudjeti cheklangan: oyiga 400 ta mijozga chegirma taklif qilish mumkin. Savol: kimga? Tasodifiy tanlov bilan 400 tadan ~14 tasi baribir ketayotgan bo'ladi; model bilan bu son bir necha barobar oshishi kerak. Yakuniy qaror ko'rsatkichi — kutilgan sof foyda, accuracy emas.

Bu darsda to'liq ML loyihasini quramiz.

Bu darsda:

  • Vazifa va vaqt kesimi
  • To'g'ri ajratish
  • Pipeline va baza
  • Model tanlash va sozlash
  • Chegara va byudjet
  • Kutilgan foyda
  • Xatolar tahlili
  • Yakuniy hisobot

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Vazifa va vaqt kesimi

text
Biznes savoli: kimga ushlab qolish taklifini yuborish?
ML vazifasi:   binar klassifikatsiya — mijoz keyingi 30 kunda ketadimi?

Vaqt kesimi 12.2-bob:
  BELGILAR:  kesim sanasigacha bo'lgan xatti-harakat (oxirgi 30/90 kun)
  MAQSAD:    kesimdan keyingi 30 kun ichida ketish
  TAQIQ:     kesimdan keyingi hech qanday ma'lumot belgi bo'lolmaydi

Har ML loyihasi vazifani aniq qo'yishdan boshlanadi: kim, qachon, qanday qaror qabul qiladi. Vaqt kesimi — eng muhim dizayn qarori: belgilar faqat kesimgacha, maqsad — kesimdan keyin. Bu bitta qoida belgi leakage'ining ko'pchiligini oldini oladi 12.9-bob.

2.2. To'g'ri ajratish

text
Bu loyihada ikki xavf bor:
  1. Bir mijozning bir necha kesimi bor → GURUH leakage
  2. Kelajak o'tmishni bashorat qiladi   → VAQT leakage

Yechim: VAQT bo'yicha ajratish (eng realistik)
  o'quv:        2024-01 .. 2024-09 kesimlari
  validatsiya:  2024-10 kesimi
  test:         2024-11 kesimi   (ishlab chiqarishga eng yaqin)

Vaqt bo'yicha ajratish ishlab chiqarish sharoitini takrorlaydi: model o'tmishda o'qitiladi, kelajakda ishlatiladi 12.3-bob. Bu odatda tasodifiy ajratishdan pastroq natija beradi — va bu to'g'ri natija. Agar vaqt bo'yicha ajratish tasodifiydan keskin past bo'lsa — ma'lumot vaqt bo'yicha o'zgarmoqda (drift), bu ham muhim topilma.

2.3. Pipeline va baza

text
Pipeline 12.9-bob: imputer → scaler / one-hot → model
Bazalar (12.1, 12.6):
  · tasodifiy tanlov (byudjet ichida)
  · bitta qoida: "oxirgi 30 kunda kirmaganlar"
  · DummyClassifier(strategy="stratified")

Yaxshilanish shu bazalarga nisbatan o'lchanadi

Baza — loyihaning eng muhim va eng ko'p unutiladigan qismi. Bu vazifada eng kuchli baza — oddiy qoida ("oxirgi 30 kunda faol bo'lmaganlar"): ko'p hollarda u ancha yaxshi ishlaydi va model uni yengishi kerak, aks holda model kerak emas.

2.4. Model tanlash va sozlash

text
Nomzodlar: LogisticRegression (talqin), RandomForest, GradientBoosting
Bir xil CV bo'linishi, bir xil metrika (PR AUC — nomutanosib sinf, 12.7)
Sozlash: RandomizedSearchCV, faqat o'quv ma'lumotida 12.6-bob
Overfitting nazorati: o'quv va CV ballari farqi (12.4)

Model tanlashda metrika avvaldan belgilanadi: bu yerda musbat sinf ~3-4% bo'lgani uchun PR AUC 12.7-bob. Model murakkabligini oshirishda o'quv va CV ballari farqini kuzating 12.4-bob: farq katta bo'lsa — regularizatsiya yoki kamroq chuqurlik.

2.5. Chegara va byudjet

text
Byudjet: oyiga 400 ta taklif  →  chegara emas, TOP-400 tanlash

Baholash:
  precision@400 = top-400 ichida haqiqatan ketganlar ulushi
  lift@400      = precision@400 / baza ulushi
  recall@400    = ushlangan ketuvchilar ulushi

Byudjet cheklangan bo'lsa, chegara avtomatik aniqlanadi: top-k ni tanlaysiz. Lift — biznesga eng tushunarli ko'rsatkich: "tasodifiy tanlovdan necha barobar yaxshi". precision@k, recall@k va lift@k — bunday loyihalarning standart hisoboti.

2.6. Kutilgan foyda

text
Har taklif uchun:
  · taklif narxi (chegirma)              C = 15 000
  · saqlangan mijoz qiymati (LTV)        V = 1 200 000
  · taklifning samaradorligi (uplift)    u = 0.25   (ketuvchilarning 25% qoladi)

Sof foyda = (TP × u × V) - (TP + FP) × C
Bu — A/B test bilan tasdiqlanishi kerak (11.10)

Kutilgan foyda — modelning biznes qiymati: u taklif narxi, mijoz qiymati va uplift ga bog'liq. Muhim nozik jihat: ketishni bashorat qilish uplift bilan bir xil emas — ba'zi mijozlar taklifsiz ham qoladi, ba'zilari taklif bilan ham ketadi. Yakuniy tasdiq — A/B test 11.10-bob.

2.7. Xatolar tahlili va hisobot

text
Tahlil: qaysi segmentda model yomon? (yangi mijozlar, kam faollik, tarif)
Hisobot tuzilishi 8.9-bob:
  1. Savol va qaror
  2. Ma'lumot va kesim dizayni
  3. Baza va model natijalari (noaniqlik bilan)
  4. Byudjet ssenariylari va kutilgan foyda
  5. Cheklovlar, xavflar, monitoring
  6. Keyingi qadam: A/B test

Hisobot — loyihaning mahsuloti. U qaror qabul qiluvchiga yo'naltirilgan: savol → natija → tavsiya → cheklovlar. Model kodi emas, qaror yetkaziladi.

2.8. To'liq oqim — bir loyihada

ML loyihasi: vazifa va vaqt kesimi 12.2-bob → vaqt/guruh bo'yicha ajratish 12.3-bob → pipeline 12.9-bob → baza 12.1-bob → model nomzodlari va CV 12.6-bob → overfitting nazorati (12.4-12.5) → metrika va chegara/byudjet 12.7-bob → kutilgan foyda → xatolar tahlili va hisobot. 12-qism shu oqimning har bo'g'inini berdi; keyingi qismlarda modellarning o'zi chuqurlashadi, lekin oqim o'zgarmaydi.

3. Tez ma'lumotnoma

python
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

# vaqt bo'yicha ajratish
tr = df[df.kesim <= "2024-09"]; val = df[df.kesim == "2024-10"]; te = df[df.kesim == "2024-11"]

# top-k baholash
def top_k(y, p, k):
    idx = np.argsort(p)[::-1][:k]
    return y[idx].mean(), y[idx].sum() / y.sum()      # precision@k, recall@k

# kutilgan foyda
foyda = tp * uplift * LTV - (tp + fp) * narx
QOIDA: vaqt kesimi · vaqt bo'yicha ajratish · pipeline · baza · PR AUC · byudjet · foyda

Loyiha xulosasi

1. Vazifa va kesim  2. Ajratish  3. Pipeline  4. Baza
5. Modellar va CV   6. Sozlash   7. Byudjet va chegara
8. Kutilgan foyda   9. Xatolar tahlili  10. Hisobot va A/B test

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14). To'rt misol bitta loyihaning ketma-ket bosqichlari.

Misol 1 — Ma'lumot, kesim dizayni va ajratish

python
"""1-bosqich: vaqt kesimli ma'lumot va vaqt bo'yicha ajratish (real pandas/sklearn)."""

import numpy as np
import pandas as pd


def yarat(seed: int = 42) -> pd.DataFrame:
    """Har oy uchun kesim: mijoz xatti-harakati + keyingi 30 kunda ketish."""
    rng = np.random.default_rng(seed)
    oylar = [f"2024-{o:02d}" for o in range(1, 12)]
    n_mijoz = 6000
    tarif = rng.choice(["start", "standart", "premium"], n_mijoz, p=[0.5, 0.35, 0.15])
    hudud = rng.choice(["Toshkent", "viloyat", "chet"], n_mijoz, p=[0.55, 0.4, 0.05])
    sifat = rng.normal(0, 1, n_mijoz)                    # kuzatilmaydigan sodiqlik

    qatorlar = []
    for i, oy in enumerate(oylar):
        faol = rng.random(n_mijoz) < 0.9                 # shu oyda hali obunada
        kun = np.clip(rng.normal(12 + 3 * sifat, 6, n_mijoz), 0, 30).round()
        dars = np.clip(rng.poisson(np.maximum(0.5, 4 + 1.5 * sifat)), 0, None)
        qollab = rng.poisson(0.3, n_mijoz)
        obuna_oyi = i + rng.integers(1, 18, n_mijoz)
        ball = (-2.6 - 0.09 * kun - 0.10 * dars + 0.30 * qollab
                - 0.02 * obuna_oyi + 0.55 * (tarif == "start")
                - 0.45 * sifat + 0.15 * i / 10)
        ketdi = (rng.random(n_mijoz) < 1 / (1 + np.exp(-ball))).astype(int)
        qatorlar.append(pd.DataFrame({
            "kesim": oy, "mijoz": np.arange(n_mijoz), "tarif": tarif, "hudud": hudud,
            "faol_kun": kun, "darslar": dars.astype(float),
            "qollab_murojaat": qollab.astype(float), "obuna_oyi": obuna_oyi.astype(float),
            "ketdi": ketdi,
        })[faol])
    df = pd.concat(qatorlar, ignore_index=True)
    df.loc[rng.choice(len(df), 900, replace=False), "darslar"] = np.nan
    return df


def main() -> None:
    df = yarat()

    print("=== 1. Ma'lumot ===")
    print(f"  {len(df)} kesim-qator, {df['mijoz'].nunique()} noyob mijoz, "
          f"{df['kesim'].nunique()} oy")
    print(f"  ketish ulushi: {df['ketdi'].mean():.2%}")
    print(f"  yetishmovchilik: {df.isna().sum()[df.isna().sum() > 0].to_dict()}")

    print("\n=== 2. Kesim dizayni ===")
    print("  BELGILAR: kesim oyigacha bo'lgan xatti-harakat")
    print("  MAQSAD:   kesimdan keyingi 30 kunda ketish")
    print("  har mijoz bir necha kesimda uchraydi → guruh leakage xavfi")

    print("\n=== 3. Ketish oylar bo'yicha ===")
    oylik = df.groupby("kesim")["ketdi"].agg(["mean", "size"])
    for oy, q in oylik.iterrows():
        print(f"  {oy}: {q['mean']:.2%} ({int(q['size'])} qator)")

    print("\n=== 4. Vaqt bo'yicha ajratish ===")
    tr = df[df["kesim"] <= "2024-09"]
    val = df[df["kesim"] == "2024-10"]
    te = df[df["kesim"] == "2024-11"]
    for nom, qism in [("o'quv", tr), ("validatsiya", val), ("test", te)]:
        print(f"  {nom:<12}: {len(qism):>6} qator, ketish {qism['ketdi'].mean():.2%}")
    print("  ⭐ Test — eng oxirgi oy: ishlab chiqarishga eng yaqin")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ma'lumot ===
  59371 kesim-qator, 6000 noyob mijoz, 11 oy
  ketish ulushi: 3.20%
  yetishmovchilik: {'darslar': 900}

=== 2. Kesim dizayni ===
  BELGILAR: kesim oyigacha bo'lgan xatti-harakat
  MAQSAD:   kesimdan keyingi 30 kunda ketish
  har mijoz bir necha kesimda uchraydi → guruh leakage xavfi

=== 3. Ketish oylar bo'yicha ===
  2024-01: 3.23% (5420 qator)
  2024-02: 3.54% (5399 qator)
  2024-03: 2.95% (5397 qator)
  2024-04: 3.27% (5411 qator)
  2024-05: 3.04% (5392 qator)
  2024-06: 3.09% (5437 qator)
  2024-07: 3.46% (5428 qator)
  2024-08: 2.89% (5371 qator)
  2024-09: 3.35% (5381 qator)
  2024-10: 2.87% (5392 qator)
  2024-11: 3.50% (5343 qator)

=== 4. Vaqt bo'yicha ajratish ===
  o'quv       :  48636 qator, ketish 3.20%
  validatsiya :   5392 qator, ketish 2.87%
  test        :   5343 qator, ketish 3.50%
  ⭐ Test — eng oxirgi oy: ishlab chiqarishga eng yaqin

Nima ko'rsatdi: 2.1, 2.2-bo'limlar.

Misol 2 — Pipeline, baza va model nomzodlari

python
"""2-bosqich: baza va modellarni solishtirish (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def yarat(seed: int = 42) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    oylar = [f"2024-{o:02d}" for o in range(1, 12)]
    n_mijoz = 6000
    tarif = rng.choice(["start", "standart", "premium"], n_mijoz, p=[0.5, 0.35, 0.15])
    hudud = rng.choice(["Toshkent", "viloyat", "chet"], n_mijoz, p=[0.55, 0.4, 0.05])
    sifat = rng.normal(0, 1, n_mijoz)
    qatorlar = []
    for i, oy in enumerate(oylar):
        faol = rng.random(n_mijoz) < 0.9
        kun = np.clip(rng.normal(12 + 3 * sifat, 6, n_mijoz), 0, 30).round()
        dars = np.clip(rng.poisson(np.maximum(0.5, 4 + 1.5 * sifat)), 0, None)
        qollab = rng.poisson(0.3, n_mijoz)
        obuna_oyi = i + rng.integers(1, 18, n_mijoz)
        ball = (-2.6 - 0.09 * kun - 0.10 * dars + 0.30 * qollab
                - 0.02 * obuna_oyi + 0.55 * (tarif == "start")
                - 0.45 * sifat + 0.15 * i / 10)
        ketdi = (rng.random(n_mijoz) < 1 / (1 + np.exp(-ball))).astype(int)
        qatorlar.append(pd.DataFrame({
            "kesim": oy, "mijoz": np.arange(n_mijoz), "tarif": tarif, "hudud": hudud,
            "faol_kun": kun, "darslar": dars.astype(float),
            "qollab_murojaat": qollab.astype(float), "obuna_oyi": obuna_oyi.astype(float),
            "ketdi": ketdi,
        })[faol])
    df = pd.concat(qatorlar, ignore_index=True)
    df.loc[rng.choice(len(df), 900, replace=False), "darslar"] = np.nan
    return df


SON = ["faol_kun", "darslar", "qollab_murojaat", "obuna_oyi"]
KAT = ["tarif", "hudud"]


def quvur(model) -> Pipeline:
    tayyor = ColumnTransformer([
        ("son", Pipeline([("imp", SimpleImputer(strategy="median")),
                          ("sc", StandardScaler())]), SON),
        ("kat", OneHotEncoder(handle_unknown="ignore"), KAT),
    ])
    return Pipeline([("t", tayyor), ("m", model)])


def main() -> None:
    df = yarat()
    tr = df[df["kesim"] <= "2024-09"]
    val = df[df["kesim"] == "2024-10"]
    Xtr, ytr = tr[SON + KAT], tr["ketdi"].to_numpy()
    Xval, yval = val[SON + KAT], val["ketdi"].to_numpy()

    print("=== 1. Bazalar (validatsiya oyida) ===")
    baza_ulush = yval.mean()
    print(f"  tasodifiy tanlov precision = {baza_ulush:.2%}")
    qoida = -val["faol_kun"].to_numpy()               # kam faol — ko'proq xavf
    print(f"  qoida (kam faollik): PR AUC = {average_precision_score(yval, qoida):.3f}, "
          f"ROC AUC = {roc_auc_score(yval, qoida):.3f}")

    print("\n=== 2. Model nomzodlari ===")
    nomzodlar = {
        "LogisticRegression": LogisticRegression(max_iter=1000),
        "RandomForest": RandomForestClassifier(n_estimators=300, min_samples_leaf=20,
                                               random_state=0, n_jobs=1),
        "GradientBoosting": GradientBoostingClassifier(random_state=0),
    }
    natija = {}
    for nom, m in nomzodlar.items():
        pipe = quvur(m).fit(Xtr, ytr)
        p_tr = pipe.predict_proba(Xtr)[:, 1]
        p_val = pipe.predict_proba(Xval)[:, 1]
        natija[nom] = p_val
        print(f"  {nom:<20}: PR AUC {average_precision_score(yval, p_val):.3f}, "
              f"ROC AUC {roc_auc_score(yval, p_val):.3f} "
              f"(o'quv PR AUC {average_precision_score(ytr, p_tr):.3f})")

    print("\n=== 3. Overfitting nazorati ===")
    print("  o'quv va validatsiya PR AUC farqi katta bo'lsa — soddalashtirish kerak")

    print("\n=== 4. Tanlov ===")
    eng = max(natija, key=lambda k: average_precision_score(yval, natija[k]))
    print(f"  eng yaxshi: {eng}")
    print(f"  baza (tasodifiy) {baza_ulush:.3f} → PR AUC "
          f"{average_precision_score(yval, natija[eng]):.3f}")
    print("  ⭐ Metrika oldindan tanlangan: PR AUC (nomutanosib sinf)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Bazalar (validatsiya oyida) ===
  tasodifiy tanlov precision = 2.87%
  qoida (kam faollik): PR AUC = 0.066, ROC AUC = 0.709

=== 2. Model nomzodlari ===
  LogisticRegression  : PR AUC 0.106, ROC AUC 0.746 (o'quv PR AUC 0.112)
  RandomForest        : PR AUC 0.095, ROC AUC 0.713 (o'quv PR AUC 0.196)
  GradientBoosting    : PR AUC 0.114, ROC AUC 0.732 (o'quv PR AUC 0.133)

=== 3. Overfitting nazorati ===
  o'quv va validatsiya PR AUC farqi katta bo'lsa — soddalashtirish kerak

=== 4. Tanlov ===
  eng yaxshi: GradientBoosting
  baza (tasodifiy) 0.029 → PR AUC 0.114
  ⭐ Metrika oldindan tanlangan: PR AUC (nomutanosib sinf)

Nima ko'rsatdi: 2.3, 2.4-bo'limlar.

Misol 3 — Byudjet, chegara va kutilgan foyda

python
"""3-bosqich: top-k tanlash va biznes qiymati (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def yarat(seed: int = 42) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    oylar = [f"2024-{o:02d}" for o in range(1, 12)]
    n_mijoz = 6000
    tarif = rng.choice(["start", "standart", "premium"], n_mijoz, p=[0.5, 0.35, 0.15])
    hudud = rng.choice(["Toshkent", "viloyat", "chet"], n_mijoz, p=[0.55, 0.4, 0.05])
    sifat = rng.normal(0, 1, n_mijoz)
    qatorlar = []
    for i, oy in enumerate(oylar):
        faol = rng.random(n_mijoz) < 0.9
        kun = np.clip(rng.normal(12 + 3 * sifat, 6, n_mijoz), 0, 30).round()
        dars = np.clip(rng.poisson(np.maximum(0.5, 4 + 1.5 * sifat)), 0, None)
        qollab = rng.poisson(0.3, n_mijoz)
        obuna_oyi = i + rng.integers(1, 18, n_mijoz)
        ball = (-2.6 - 0.09 * kun - 0.10 * dars + 0.30 * qollab
                - 0.02 * obuna_oyi + 0.55 * (tarif == "start")
                - 0.45 * sifat + 0.15 * i / 10)
        ketdi = (rng.random(n_mijoz) < 1 / (1 + np.exp(-ball))).astype(int)
        qatorlar.append(pd.DataFrame({
            "kesim": oy, "mijoz": np.arange(n_mijoz), "tarif": tarif, "hudud": hudud,
            "faol_kun": kun, "darslar": dars.astype(float),
            "qollab_murojaat": qollab.astype(float), "obuna_oyi": obuna_oyi.astype(float),
            "ketdi": ketdi,
        })[faol])
    df = pd.concat(qatorlar, ignore_index=True)
    df.loc[rng.choice(len(df), 900, replace=False), "darslar"] = np.nan
    return df


SON = ["faol_kun", "darslar", "qollab_murojaat", "obuna_oyi"]
KAT = ["tarif", "hudud"]
NARX, LTV, UPLIFT = 15_000, 1_200_000, 0.25


def top_k(y: np.ndarray, p: np.ndarray, k: int):
    idx = np.argsort(p)[::-1][:k]
    return float(y[idx].mean()), float(y[idx].sum() / y.sum()), int(y[idx].sum())


def main() -> None:
    df = yarat()
    tr = df[df["kesim"] <= "2024-10"]                 # o'quv + validatsiya
    te = df[df["kesim"] == "2024-11"]
    tayyor = ColumnTransformer([
        ("son", Pipeline([("imp", SimpleImputer(strategy="median")),
                          ("sc", StandardScaler())]), SON),
        ("kat", OneHotEncoder(handle_unknown="ignore"), KAT),
    ])
    pipe = Pipeline([("t", tayyor), ("m", GradientBoostingClassifier(random_state=0))])
    pipe.fit(tr[SON + KAT], tr["ketdi"])
    y = te["ketdi"].to_numpy()
    p = pipe.predict_proba(te[SON + KAT])[:, 1]

    print("=== 1. Test oyi (2024-11) ===")
    print(f"  {len(te)} mijoz, haqiqatan ketgan {y.sum()} ta ({y.mean():.2%})")

    print("\n=== 2. Byudjet ssenariylari ===")
    for k in [200, 400, 800, 1500]:
        prec, rec, tp = top_k(y, p, k)
        foyda = tp * UPLIFT * LTV - k * NARX
        print(f"  top-{k:<5}: precision {prec:.2%}, recall {rec:.2%}, "
              f"lift {prec / y.mean():4.1f}x, foyda {foyda / 1e6:7.1f} mln")
    print("  (foyda eng yuqori bo'lgan k — byudjetni kengaytirish uchun asos)")

    print("\n=== 3. Tasodifiy tanlov bilan solishtirish (400 ta) ===")
    rng = np.random.default_rng(0)
    tasodif = [y[rng.choice(len(y), 400, replace=False)].sum() for _ in range(2000)]
    prec400, rec400, tp400 = top_k(y, p, 400)
    print(f"  tasodifiy: o'rtacha {np.mean(tasodif):.1f} ketuvchi "
          f"(95% oraliq {np.percentile(tasodif, 2.5):.0f}..{np.percentile(tasodif, 97.5):.0f})")
    print(f"  model    : {tp400} ketuvchi  → lift {prec400 / y.mean():.1f}x")
    print(f"  qo'shimcha foyda: "
          f"{(tp400 - np.mean(tasodif)) * UPLIFT * LTV / 1e6:.1f} mln")

    print("\n=== 4. Sezgirlik: uplift noaniq ===")
    _, _, tp = top_k(y, p, 400)
    for u in [0.10, 0.25, 0.40]:
        print(f"  uplift {u:.0%}: foyda {(tp * u * LTV - 400 * NARX) / 1e6:6.1f} mln")
    print("  ⭐ Uplift — taxmin; yakuniy tasdiq A/B test 11.10-bob")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Test oyi (2024-11) ===
  5343 mijoz, haqiqatan ketgan 187 ta (3.50%)

=== 2. Byudjet ssenariylari ===
  top-200  : precision 15.50%, recall 16.58%, lift  4.4x, foyda     6.3 mln
  top-400  : precision 12.75%, recall 27.27%, lift  3.6x, foyda     9.3 mln
  top-800  : precision 10.75%, recall 45.99%, lift  3.1x, foyda    13.8 mln
  top-1500 : precision 7.53%, recall 60.43%, lift  2.2x, foyda    11.4 mln
  (foyda eng yuqori bo'lgan k — byudjetni kengaytirish uchun asos)

=== 3. Tasodifiy tanlov bilan solishtirish (400 ta) ===
  tasodifiy: o'rtacha 14.0 ketuvchi (95% oraliq 8..21)
  model    : 51 ketuvchi  → lift 3.6x
  qo'shimcha foyda: 11.1 mln

=== 4. Sezgirlik: uplift noaniq ===
  uplift 10%: foyda    0.1 mln
  uplift 25%: foyda    9.3 mln
  uplift 40%: foyda   18.5 mln
  ⭐ Uplift — taxmin; yakuniy tasdiq A/B test (11.10)

Nima ko'rsatdi: 2.5, 2.6-bo'limlar.

Misol 4 — Xatolar tahlili va yakuniy hisobot

python
"""4-bosqich: segmentlar, cheklovlar va hisobot (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def yarat(seed: int = 42) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    oylar = [f"2024-{o:02d}" for o in range(1, 12)]
    n_mijoz = 6000
    tarif = rng.choice(["start", "standart", "premium"], n_mijoz, p=[0.5, 0.35, 0.15])
    hudud = rng.choice(["Toshkent", "viloyat", "chet"], n_mijoz, p=[0.55, 0.4, 0.05])
    sifat = rng.normal(0, 1, n_mijoz)
    qatorlar = []
    for i, oy in enumerate(oylar):
        faol = rng.random(n_mijoz) < 0.9
        kun = np.clip(rng.normal(12 + 3 * sifat, 6, n_mijoz), 0, 30).round()
        dars = np.clip(rng.poisson(np.maximum(0.5, 4 + 1.5 * sifat)), 0, None)
        qollab = rng.poisson(0.3, n_mijoz)
        obuna_oyi = i + rng.integers(1, 18, n_mijoz)
        ball = (-2.6 - 0.09 * kun - 0.10 * dars + 0.30 * qollab
                - 0.02 * obuna_oyi + 0.55 * (tarif == "start")
                - 0.45 * sifat + 0.15 * i / 10)
        ketdi = (rng.random(n_mijoz) < 1 / (1 + np.exp(-ball))).astype(int)
        qatorlar.append(pd.DataFrame({
            "kesim": oy, "mijoz": np.arange(n_mijoz), "tarif": tarif, "hudud": hudud,
            "faol_kun": kun, "darslar": dars.astype(float),
            "qollab_murojaat": qollab.astype(float), "obuna_oyi": obuna_oyi.astype(float),
            "ketdi": ketdi,
        })[faol])
    df = pd.concat(qatorlar, ignore_index=True)
    df.loc[rng.choice(len(df), 900, replace=False), "darslar"] = np.nan
    return df


SON = ["faol_kun", "darslar", "qollab_murojaat", "obuna_oyi"]
KAT = ["tarif", "hudud"]
NARX, LTV, UPLIFT, BYUDJET = 15_000, 1_200_000, 0.25, 400


def main() -> None:
    df = yarat()
    tr = df[df["kesim"] <= "2024-10"]
    te = df[df["kesim"] == "2024-11"].copy()
    tayyor = ColumnTransformer([
        ("son", Pipeline([("imp", SimpleImputer(strategy="median")),
                          ("sc", StandardScaler())]), SON),
        ("kat", OneHotEncoder(handle_unknown="ignore"), KAT),
    ])
    pipe = Pipeline([("t", tayyor), ("m", GradientBoostingClassifier(random_state=0))])
    pipe.fit(tr[SON + KAT], tr["ketdi"])
    te["ball"] = pipe.predict_proba(te[SON + KAT])[:, 1]
    y = te["ketdi"].to_numpy()

    print("=== 1. Segmentlar bo'yicha sifat (PR AUC) ===")
    for ustun in ["tarif", "hudud"]:
        for qiymat, qism in te.groupby(ustun):
            if len(qism) >= 100 and qism["ketdi"].nunique() == 2:
                print(f"  {ustun}={qiymat:<10}: n={len(qism):>4}, "
                      f"ketish {qism['ketdi'].mean():.2%}, "
                      f"PR AUC {average_precision_score(qism['ketdi'], qism['ball']):.3f}")

    print("\n=== 2. Tanlangan 400 ta kim? ===")
    top = te.nlargest(BYUDJET, "ball")
    print(f"  tarif: {top['tarif'].value_counts(normalize=True).round(2).to_dict()}")
    print(f"  o'rtacha faol kun: {top['faol_kun'].mean():.1f} "
          f"(hammasi bo'yicha {te['faol_kun'].mean():.1f})")
    print(f"  o'rtacha obuna oyi: {top['obuna_oyi'].mean():.1f} "
          f"(hammasi bo'yicha {te['obuna_oyi'].mean():.1f})")

    print("\n=== 3. O'tkazib yuborilganlar ===")
    ketgan = te[te["ketdi"] == 1]
    otkazib = ketgan[~ketgan.index.isin(top.index)]
    print(f"  {len(otkazib)} / {len(ketgan)} ketuvchi byudjetdan tashqarida")
    print(f"  ularning o'rtacha faol kuni: {otkazib['faol_kun'].mean():.1f} "
          f"(tanlanganlarniki {top['faol_kun'].mean():.1f})")
    print("  → faol, lekin baribir ketganlar: model ularni ko'rmaydi")

    print("\n=== 4. Yakuniy hisobot ===")
    tp = int(top["ketdi"].sum())
    print("  1) Savol: oyiga 400 ta taklifni kimga yuborish")
    print(f"  2) Ma'lumot: {df['kesim'].nunique()} oylik kesim, vaqt bo'yicha ajratish")
    print(f"  3) Natija (test oyi): PR AUC "
          f"{average_precision_score(y, te['ball']):.3f}, "
          f"ROC AUC {roc_auc_score(y, te['ball']):.3f}")
    print(f"  4) Byudjet: top-{BYUDJET} da {tp} ketuvchi "
          f"(lift {top['ketdi'].mean() / y.mean():.1f}x), "
          f"kutilgan foyda {(tp * UPLIFT * LTV - BYUDJET * NARX) / 1e6:.1f} mln")
    print("  5) Cheklovlar: uplift taxminiy, drift xavfi, ketish sababi o'rganilmagan")
    print("  6) Keyingi qadam: A/B test 11.10-bob va oylik monitoring")
    print("  ⭐ Loyiha mahsuloti — model emas, QAROR")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Segmentlar bo'yicha sifat (PR AUC) ===
  tarif=premium   : n= 782, ketish 2.69%, PR AUC 0.151
  tarif=standart  : n=1876, ketish 2.61%, PR AUC 0.082
  tarif=start     : n=2685, ketish 4.36%, PR AUC 0.101
  hudud=Toshkent  : n=2926, ketish 3.59%, PR AUC 0.093
  hudud=chet      : n= 280, ketish 5.71%, PR AUC 0.136
  hudud=viloyat   : n=2137, ketish 3.09%, PR AUC 0.101

=== 2. Tanlangan 400 ta kim? ===
  tarif: {'start': 0.78, 'standart': 0.16, 'premium': 0.06}
  o'rtacha faol kun: 2.5 (hammasi bo'yicha 12.1)
  o'rtacha obuna oyi: 17.8 (hammasi bo'yicha 19.0)

=== 3. O'tkazib yuborilganlar ===
  136 / 187 ketuvchi byudjetdan tashqarida
  ularning o'rtacha faol kuni: 8.9 (tanlanganlarniki 2.5)
  → faol, lekin baribir ketganlar: model ularni ko'rmaydi

=== 4. Yakuniy hisobot ===
  1) Savol: oyiga 400 ta taklifni kimga yuborish
  2) Ma'lumot: 11 oylik kesim, vaqt bo'yicha ajratish
  3) Natija (test oyi): PR AUC 0.094, ROC AUC 0.744
  4) Byudjet: top-400 da 51 ketuvchi (lift 3.6x), kutilgan foyda 9.3 mln
  5) Cheklovlar: uplift taxminiy, drift xavfi, ketish sababi o'rganilmagan
  6) Keyingi qadam: A/B test 11.10-bob va oylik monitoring
  ⭐ Loyiha mahsuloti — model emas, QAROR

Nima ko'rsatdi: 2.5, 2.7-bo'limlar.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Loyiha — modelni tanlash" Oqimning bir qadami
"Tasodifiy ajratish yetarli" Vaqt/guruh bo'yicha
"Baza kerak emas" Eng muhim o'lchov
"Accuracy — natija" PR AUC, lift, foyda
"Chegara 0.5" Byudjet belgilaydi
"Ketishni bashorat = uplift" Har xil narsa
"Model tayyor — ish tugadi" A/B test va monitoring
"Hisobot — grafiklar" Qaror va cheklovlar

6. Keng tarqalgan xatolar va yechimlari

1. Kesim dizaynisiz belgilar

python
X = df.drop(columns=["ketdi"])         # kelajak ustunlari bor         # ⚠️
# faqat kesimgacha mavjud ustunlar                                     # ✅

2. Tasodifiy ajratish

python
train_test_split(df, test_size=0.2)                                    # ⚠️
tr = df[df.kesim <= "2024-09"]; te = df[df.kesim == "2024-11"]         # ✅

3. Bazasiz hisobot

python
print("PR AUC 0.31")                                                   # ⚠️
print(f"PR AUC 0.31 (baza {y.mean():.3f}, qoida 0.19)")                # ✅

4. Pipeline'siz tayyorlash

python
X = StandardScaler().fit_transform(X)                                  # ⚠️
Pipeline([("t", tayyor), ("m", model)])                                # ✅

5. Chegara 0.5

python
tanlangan = te[p >= 0.5]               # byudjetga sig'maydi           # ⚠️
tanlangan = te.nlargest(400, "ball")                                   # ✅

6. Uplift'ni e'tiborsiz qoldirish

python
foyda = tp * LTV - 400 * narx                                          # ⚠️
foyda = tp * uplift * LTV - 400 * narx                                 # ✅

7. A/B testsiz e'lon qilish

python
# "model 120 mln tejadi"                                               # ⚠️
# "kutilgan foyda 120 mln; A/B test bilan tasdiqlanadi"                # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 8-qism (o'tilgan): EDA va hisobot
  • 11.10-dars (o'tilgan): A/B test
  • 12.1-12.9 (o'tilgan): Butun oqim
  • Keyingi qismlar: Regressiya, klassifikatsiya, ansambllar
  • MLOps qismi: Ishlab chiqarish va monitoring

8. Eng yaxshi amaliyotlar

  1. Vazifani va kesimni yozib qo'ying.

  2. Vaqt bo'yicha ajrating.

  3. Bazani hisoblang.

  4. Hamma tayyorlash pipeline ichida.

  5. Metrikani oldindan tanlang.

  6. Byudjetdan chiqing.

  7. Foydani sezgirlik bilan bering.

  8. A/B test bilan tasdiqlang.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # vaqt kesimi nima?
2.  # nega vaqt bo'yicha ajratish?
3.  # bu vazifada baza nima?
4.  # qaysi metrika?
5.  # chegarani nima belgilaydi?
6.  # lift nima?
7.  # precision@k?
8.  # uplift nima?
9.  # sof foyda formulasi?
10. # sezgirlik tahlili nima uchun?
11. # xatolar tahlilida nimaga qaraladi?
12. # yakuniy tasdiq?
Javoblar
  1. Belgilar — kesimgacha, maqsad — keyin
  2. Ishlab chiqarishni takrorlaydi
  3. Tasodifiy tanlov va oddiy qoida
  4. PR AUC (nomutanosib)
  5. Byudjet
  6. Bazaga nisbatan necha barobar
  7. Top-k ichida musbatlar ulushi
  8. Taklifning haqiqiy ta'siri
  9. TP × uplift × LTV - k × narx
  10. Uplift noaniq
  11. Segmentlar va o'tkazib yuborilganlar
  12. A/B test

Vazifa 2: Xatolarni tuzating

python
1.  train_test_split(df, test_size=0.2, random_state=0)   # kesimli ma'lumot

2.  print("ROC AUC 0.82 — model yaxshi")   # bazasiz

3.  tanlangan = te[te.ball >= 0.5]   # byudjet 400

4.  foyda = tp * LTV - 400 * narx

5.  # "model yiliga 1.4 mlrd tejaydi" (A/B testsiz)
Javoblar
python
1.  tr = df[df.kesim <= "2024-09"]; te = df[df.kesim == "2024-11"]

2.  print(f"PR AUC {ap:.3f} (baza {y.mean():.3f})")

3.  tanlangan = te.nlargest(400, "ball")

4.  foyda = tp * uplift * LTV - 400 * narx

5.  # "kutilgan foyda ...; A/B test bilan tasdiqlanadi"

Vazifa 3: Loyihani takrorlang

Modellang:

  1. Kesimli ma'lumot
  2. Vaqt bo'yicha ajratish
  3. Baza va modellar
  4. Byudjet va foyda

Vazifa 4: Byudjet ssenariylari

Modellang:

  1. 100..2000 oralig'ida k
  2. Foyda egri chizig'i
  3. Optimal k
  4. Sezgirlik

Vazifa 5: Drift

Modellang:

  1. Oylar bo'yicha PR AUC
  2. Tushish bormi
  3. Qayta o'qitish jadvali
  4. Monitoring rejasi

Vazifa 6: Hisobot

Modellang:

  1. Olti bo'limli hisobot
  2. Noaniqlik (11.8)
  3. Cheklovlar
  4. Tavsiya

Vazifa 7: O'ylash

Ketishni bashorat qiluvchi model eng yuqori xavfli mijozlarni topadi — lekin ushlab qolish taklifi aynan shularga eng kam ta'sir qilishi mumkin: ular baribir ketadi. Bu muammo (uplift modellashtirish) nima va uni qanday hal qilish kerak?

Javob

Qisqa javob: ketish ehtimoli va taklifning ta'siri — turli narsalar. Kerak bo'lgani — taklif fikrini o'zgartiradigan mijozlar (persuadables). Buni topish uchun uplift modellashtirish kerak, uning ma'lumoti esa faqat tajribadan (A/B test) keladi.

1. To'rt segment

Segment Taklifsiz Taklif bilan Nima qilish
Ishontirsa bo'ladigan ketadi qoladi MAQSAD
Yo'qotilgan ketadi ketadi byudjetni sarflamang
Ishonchli qoladi qoladi taklif keraksiz (pul isrofi)
Bezovta bo'ladigan qoladi ketadi taklif ZARAR

2. Nega oddiy churn modeli yetarli emas

  • U "ketish ehtimoli" ni reytinglaydi, "ta'sir" ni emas
  • Eng xavfli mijozlar ko'pincha "yo'qotilgan" segmentda
  • Natijada byudjet samarasiz sarflanadi

3. Uplift qanday o'lchanadi

  • A/B test: tasodifiy yarmi taklif oladi (11.10)
  • uplift(x) = P(qoladi | taklif, x) - P(qoladi | taklifsiz, x)
  • Ikki model yondashuvi (T-learner), sinf o'zgartirish, maxsus daraxtlar
  • Baholash: Qini egri chizig'i, uplift@k

4. Amaliy tartib

  1. Churn modeli bilan boshlang (bu dars)
  2. Tasodifiy taklif tajribasini o'tkazing (ma'lumot yig'ish)
  3. Tajriba ma'lumotida uplift modelini quring
  4. Byudjetni uplift bo'yicha taqsimlang

5. Xulosa

  1. Bashorat ≠ ta'sir
  2. Uplift ma'lumoti faqat tajribadan
  3. "Bezovta bo'ladigan" segment zarar keltiradi
  4. Churn modeli — birinchi qadam, oxirgisi emas

Nimani mustahkamlaydi: 2.6, 2.7-bo'limlar.

Xulosa

Bu darsda to'liq ML loyihasini qurdik va 12-qismni yakunladik.

Eng muhim uch fikr:

  1. Loyiha — oqim, model emas. Vazifa va vaqt kesimi 12.2-bob → vaqt/guruh bo'yicha ajratish 12.3-bob → pipeline 12.9-bob → baza 12.1-bob → model nomzodlari va CV 12.6-bob → overfitting nazorati (12.4-12.5) → metrika va chegara (12.7-12.8) → biznes qiymati → hisobot. Modelni almashtirish oson, oqimni to'g'ri qurish qiyin — va natijani aynan oqim belgilaydi.

  2. Chegara emas, byudjet. Cheklangan resurs bo'lsa (oyiga 400 taklif) qaror top-k tanlash bilan qabul qilinadi; hisobot tili — precision@k, recall@k va lift@k. Yakuniy ko'rsatkich — kutilgan sof foyda (TP × uplift × LTV - k × narx), uning eng noaniq qismi uplift, shuning uchun sezgirlik tahlili beriladi.

  3. Model — qarorning bir qismi. Hisobot savol, ma'lumot dizayni, baza, natija, byudjet ssenariylari, cheklovlar va keyingi qadamni (A/B test, monitoring) o'z ichiga oladi. Ketish ehtimoli taklif ta'siri emas — yakuniy tasdiq faqat tajribadan keladi 11.10-bob.

Keyingi qismda chiziqli regressiyani chuqur o'rganamiz: eng kichik kvadratlar, regularizatsiya (Ridge, Lasso, ElasticNet), belgi muhandisligi va modelni talqin qilish.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
12.10-dars: Amaliyot — to'liq ML loyihasi — IlmHamroh