16.12-dars: Amaliyot — to'liq nazoratsiz loyiha

1. MA'LUMOT: hajm, yo'qolgan qiymatlar, taqsimotlar, qiyshiqlik
2. TAYYORLASH: log1p, masshtablash, kategoriyalarni kodlash
3. ANOMALIYA: chetlanishlarni topish va qaror qabul qilish
4. TUZILMA BORMI: gap statistikasi, barqarorlik 16.3-bob
5. O'LCHAM: PCA (kerak bo'lsa), vizualizatsiya uchun t-SNE
6. KLASTERLASH: bir necha algoritm, bir necha k
7. BAHOLASH: jadval (silhouette, DB, barqarorlik, o'lchamlar)
8. TALQIN: markazlar asl birliklarda, segment nomlari
9. TEKSHIRUV: domen mutaxassisi, bashorat testi
10. CHIQARISH: Pipeline saqlash, yangi obyektni biriktirish

Har qadamda yozib boriladigan qarorlar:

  transformatsiya  - nega log1p? (qiyshiqlik koeffitsiyenti)
  masshtablash     - nega StandardScaler? (chetlanishlar bormi -> Robust)
  anomaliya        - olib tashlandimi yoki qoldirildimi? nega?
  o'lcham          - PCA ishlatildimi? nechta komponent? nega?
  algoritm         - nega K-means? (shakl, tezlik, predict kerakmi)
  k                - qaysi mezonlar bo'yicha? (jadval)
  nomlash          - kim tasdiqladi?

Bu qarorlar TAKRORLANUVCHANLIKning asosi

UCH TEKSHIRUV:

1. BARQARORLIK 16.3-bob
   bootstrap/yarim bo'lish -> ARI
   > 0.75 yaxshi, < 0.5 shubhali

2. BASHORAT TESTI
   klasterlash X1 belgilarida bajariladi
   klasterlar X2 (ishlatilmagan) belgilarni bashorat qila oladimi?
   -> klasterlar haqiqiy tuzilmani aks ettiradi

3. VAQT BO'YICHA
   oldingi davr ma'lumotida klasterlab, yangi davrga qo'llash
   segmentlar ulushlari barqaror qoldimi?

Har segment uchun:
  1. MARKAZ asl birliklarda (inverse_transform + expm1)
  2. UMUMIY o'rtachadan farq (foizda) - eng farq qiladigan 3 belgi
  3. O'LCHAM (n va ulush)
  4. NOM - qisqa, marketing tilida, harakatga ishora qiladigan

YOMON NOM: "Klaster 2", "Yuqori RFM guruhi"
YAXSHI NOM: "Tez-tez, kam summa", "Yo'qolayotgan qimmatli"

Nomni DOMEN mutaxassisi tasdiqlashi kerak

# 1. Butun quvurni saqlash (15.13 kabi)
joblib.dump({"quvur": quvur, "belgilar": nomlar, "nomlar": segment_nomlari,
             "versiyalar": {...}, "sana": "2026-09-21"}, "segment.joblib")

# 2. Yangi mijozni biriktirish
segment = quvur.predict(yangi_df)      # K-means/GMM da bor

# 3. Ierarxik/DBSCAN da predict yo'q -> markazlarni saqlang
markazlar = np.array([Xs[yorliq == k].mean(axis=0) for k in ...])
segment = np.argmin(((Xs_yangi[:, None] - markazlar) ** 2).sum(axis=2), axis=1)

# 4. Kuzatuv
#    segment ulushlari vaqt bo'yicha o'zgaryaptimi (PSI - 15.13)
#    yangi mijozlarning markazgacha masofasi oshyaptimi

TOPSHIRISHDA BO'LISHI KERAK:
  [ ] Ma'lumot ta'rifi (davr, filtrlar, belgilar)
  [ ] Tayyorlash qadamlari va ularning sababi
  [ ] Anomaliyalar bo'yicha qaror
  [ ] Algoritm va k tanlovi (jadval bilan)
  [ ] Barqarorlik natijasi
  [ ] Segment profillari (markazlar asl birliklarda)
  [ ] Segment nomlari va kim tasdiqlagani
  [ ] Yangi obyektni biriktirish kodi
  [ ] Qayta klasterlash siyosati
  [ ] Cheklovlar va ma'lum muammolar

Bu - "segmentatsiya kartasi" (model card ning analogi)

import joblib
import numpy as np
from sklearn.cluster import KMeans
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import IsolationForest
from sklearn.metrics import adjusted_rand_score, silhouette_samples
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler

quvur = Pipeline([
    ("log", FunctionTransformer(np.log1p, inverse_func=np.expm1)),
    ("sc", StandardScaler()),
    ("km", KMeans(n_clusters=5, n_init=10, random_state=0))])
yorliq = quvur.fit_predict(X)

# markazlar asl birliklarda
markaz = quvur[:-1].inverse_transform(quvur["km"].cluster_centers_)

# yangi mijoz
quvur.predict(yangi_X)
joblib.dump({"quvur": quvur, "nomlar": segment_nomlari}, "segment.joblib")
QOIDA: anomaliyani tekshir · tuzilmani tasdiqla · barqarorlikni o'lcha ·
       nom ber va hujjatlashtir

1 ma'lumot -> 2 tayyorlash -> 3 anomaliya -> 4 tuzilma -> 5 o'lcham
-> 6 klasterlash -> 7 baholash -> 8 talqin -> 9 tekshiruv -> 10 chiqarish
Uch tekshiruv: barqarorlik, bashorat testi, domen tasdig'i
Topshirishda: segmentatsiya kartasi

"""1-3 qadamlar (real numpy/pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 42, n: int = 5000) -> pd.DataFrame:
    """Onlayn do'kon mijozlari: 5 ta tabiiy segment + anomaliyalar."""
    rng = np.random.default_rng(seed)
    segment = rng.choice(5, n, p=[0.30, 0.25, 0.20, 0.15, 0.10])
    profil = np.array([
        [25.0, 4.0, 180_000.0, 2.0],      # oddiy
        [8.0, 14.0, 95_000.0, 5.0],       # tez-tez, kichik chek
        [75.0, 2.0, 420_000.0, 1.5],      # kamdan-kam, katta chek
        [120.0, 1.0, 150_000.0, 1.0],     # uyquda
        [15.0, 9.0, 850_000.0, 6.0],      # qimmatli
    ])
    X = profil[segment] * rng.lognormal(0, 0.28, (n, 4))
    df = pd.DataFrame(X, columns=["oxirgi_kun", "xaridlar", "ortacha_chek",
                                  "toifalar"])
    df["segment"] = segment
    # anomaliyalar: ulgurji xaridorlar
    anomaliya = rng.random(n) < 0.012
    df.loc[anomaliya, "ortacha_chek"] *= rng.uniform(15, 40, anomaliya.sum())
    df.loc[anomaliya, "xaridlar"] *= rng.uniform(3, 8, anomaliya.sum())
    df["anomaliya"] = anomaliya.astype(int)
    return df


def main() -> None:
    df = yarat()
    nomlar = ["oxirgi_kun", "xaridlar", "ortacha_chek", "toifalar"]

    print("=== 1. Ma'lumot ===")
    print(f"  {len(df)} mijoz, {len(nomlar)} belgi")
    print(f"  yo'qolgan qiymatlar: {int(df[nomlar].isna().sum().sum())}")
    print(f"  {'belgi':<15} {'mediana':>12} {'o_rtacha':>12} {'max':>14} "
          f"{'qiyshiqlik':>11}")
    for nom in nomlar:
        print(f"  {nom:<15} {df[nom].median():>12,.1f} {df[nom].mean():>12,.1f} "
              f"{df[nom].max():>14,.0f} {df[nom].skew():>11.2f}")

    print("\n=== 2. Transformatsiya qarori ===")
    X = df[nomlar].to_numpy()
    Xlog = np.log1p(X)
    print(f"  {'belgi':<15} {'xom qiyshiqlik':>16} {'log1p dan keyin':>18}")
    for i, nom in enumerate(nomlar):
        xom = float(pd.Series(X[:, i]).skew())
        yangi = float(pd.Series(Xlog[:, i]).skew())
        print(f"  {nom:<15} {xom:>16.2f} {yangi:>18.2f}")
    print("  QAROR: log1p ishlatiladi (qiyshiqlik > 1)")

    print("\n=== 3. Anomaliyalarni aniqlash ===")
    izo = IsolationForest(contamination=0.015, n_estimators=200,
                          random_state=0, n_jobs=1).fit(X)
    ball = -izo.score_samples(X)
    belgilangan = izo.predict(X) == -1
    haqiqiy = df["anomaliya"].to_numpy().astype(bool)
    print(f"  belgilangan: {int(belgilangan.sum())} "
          f"({belgilangan.mean():.2%})")
    print(f"  haqiqiy anomaliyalar: {int(haqiqiy.sum())}")
    print(f"  to'g'ri topildi: {int((belgilangan & haqiqiy).sum())} "
          f"({(belgilangan & haqiqiy).sum() / max(haqiqiy.sum(), 1):.1%})")
    print(f"  {'holat':<22} {'ortacha_chek':>15} {'xaridlar':>11}")
    for nom, m in [("normal", ~belgilangan), ("belgilangan", belgilangan)]:
        print(f"  {nom:<22} {df.loc[m, 'ortacha_chek'].mean():>15,.0f} "
              f"{df.loc[m, 'xaridlar'].mean():>11.1f}")

    print("\n=== 4. Anomaliyalar bo'yicha qaror ===")
    Xs_hammasi = StandardScaler().fit_transform(Xlog)
    Xs_toza = StandardScaler().fit_transform(Xlog[~belgilangan])
    from sklearn.cluster import KMeans
    from sklearn.metrics import silhouette_score
    print(f"  {'variant':<22} {'n':>7} {'silhouette (k=5)':>18} "
          f"{'eng kichik %':>14}")
    for nom, Xa in [("hammasi bilan", Xs_hammasi), ("anomaliyasiz", Xs_toza)]:
        yorliq = KMeans(5, n_init=10, random_state=0).fit_predict(Xa)
        print(f"  {nom:<22} {len(Xa):>7} {silhouette_score(Xa, yorliq):>18.4f} "
              f"{np.bincount(yorliq).min() / len(Xa):>13.1%}")
    print("  QAROR: anomaliyalar alohida segment sifatida ajratiladi")
    print("  ⭐ Anomaliyalarni o'chirmang - alohida ko'rib chiqing")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  5000 mijoz, 4 belgi
  yo'qolgan qiymatlar: 0
  belgi                mediana     o_rtacha            max  qiyshiqlik
  oxirgi_kun              25.1         45.5            280        1.42
  xaridlar                 4.1          6.8            200        7.21
  ortacha_chek       179,332.2    387,161.9     45,521,055       18.16
  toifalar                 2.1          3.0             16        1.18

=== 2. Transformatsiya qarori ===
  belgi             xom qiyshiqlik    log1p dan keyin
  oxirgi_kun                  1.42               0.18
  xaridlar                    7.21               0.32
  ortacha_chek               18.16               1.27
  toifalar                    1.18               0.46
  QAROR: log1p ishlatiladi (qiyshiqlik > 1)

=== 3. Anomaliyalarni aniqlash ===
  belgilangan: 75 (1.50%)
  haqiqiy anomaliyalar: 63
  to'g'ri topildi: 61 (96.8%)
  holat                     ortacha_chek    xaridlar
  normal                         278,785         6.4
  belgilangan                  7,503,923        31.5

=== 4. Anomaliyalar bo'yicha qaror ===
  variant                      n   silhouette (k=5)   eng kichik %
  hammasi bilan             5000             0.5385         10.7%
  anomaliyasiz              4925             0.5560          9.6%
  QAROR: anomaliyalar alohida segment sifatida ajratiladi
  ⭐ Anomaliyalarni o'chirmang - alohida ko'rib chiqing

"""4-7 qadamlar (real numpy/pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.cluster import DBSCAN, AgglomerativeClustering, KMeans
from sklearn.ensemble import IsolationForest
from sklearn.metrics import (adjusted_rand_score, calinski_harabasz_score,
                             davies_bouldin_score, silhouette_score)
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 42, n: int = 5000) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    segment = rng.choice(5, n, p=[0.30, 0.25, 0.20, 0.15, 0.10])
    profil = np.array([[25.0, 4.0, 180_000.0, 2.0], [8.0, 14.0, 95_000.0, 5.0],
                       [75.0, 2.0, 420_000.0, 1.5], [120.0, 1.0, 150_000.0, 1.0],
                       [15.0, 9.0, 850_000.0, 6.0]])
    X = profil[segment] * rng.lognormal(0, 0.28, (n, 4))
    df = pd.DataFrame(X, columns=["oxirgi_kun", "xaridlar", "ortacha_chek",
                                  "toifalar"])
    df["segment"] = segment
    anomaliya = rng.random(n) < 0.012
    df.loc[anomaliya, "ortacha_chek"] *= rng.uniform(15, 40, anomaliya.sum())
    df.loc[anomaliya, "xaridlar"] *= rng.uniform(3, 8, anomaliya.sum())
    return df


def gap_statistika(X, kk, B: int = 6, seed: int = 0):
    rng = np.random.default_rng(seed)
    past, yuqori = X.min(axis=0), X.max(axis=0)
    gaplar, sklar = [], []
    for k in kk:
        J = max(KMeans(k, n_init=5, random_state=0).fit(X).inertia_, 1e-12)
        tasodifiy = []
        for b in range(B):
            Xr = rng.uniform(past, yuqori, X.shape)
            tasodifiy.append(np.log(max(KMeans(k, n_init=3,
                                               random_state=b).fit(Xr).inertia_,
                                        1e-12)))
        tasodifiy = np.array(tasodifiy)
        gaplar.append(tasodifiy.mean() - np.log(J))
        sklar.append(tasodifiy.std() * np.sqrt(1 + 1 / B))
    return np.array(gaplar), np.array(sklar)


def barqarorlik(X, yaratuvchi, B: int = 10, seed: int = 0) -> float:
    rng = np.random.default_rng(seed)
    ballar = []
    for _ in range(B):
        a = rng.choice(len(X), int(len(X) * 0.7), replace=False)
        b = rng.choice(len(X), int(len(X) * 0.7), replace=False)
        umumiy = np.intersect1d(a, b)
        ia = {v: i for i, v in enumerate(a)}
        ib = {v: i for i, v in enumerate(b)}
        ya = yaratuvchi().fit_predict(X[a])
        yb = yaratuvchi().fit_predict(X[b])
        ballar.append(adjusted_rand_score([ya[ia[v]] for v in umumiy],
                                          [yb[ib[v]] for v in umumiy]))
    return float(np.mean(ballar))


def main() -> None:
    df = yarat()
    nomlar = ["oxirgi_kun", "xaridlar", "ortacha_chek", "toifalar"]
    X = df[nomlar].to_numpy()
    izo = IsolationForest(contamination=0.015, n_estimators=200,
                          random_state=0, n_jobs=1).fit(X)
    toza = izo.predict(X) == 1
    Xs = StandardScaler().fit_transform(np.log1p(X[toza]))
    haqiqiy = df.loc[toza, "segment"].to_numpy()

    print("=== 1. Tuzilma bormi (gap statistikasi) ===")
    kk = list(range(1, 9))
    g, s = gap_statistika(Xs, kk)
    tanlangan = kk[-1]
    for i in range(len(kk) - 1):
        if g[i] >= g[i + 1] - s[i + 1]:
            tanlangan = kk[i]
            break
    print(f"  {'k':>3} {'gap':>9} {'s_k':>8}")
    for i, k in enumerate(kk[:6]):
        print(f"  {k:>3} {g[i]:>9.4f} {s[i]:>8.4f}")
    print(f"  gap tanlagan k = {tanlangan} (1 bo'lsa tuzilma yo'q)")

    print("\n=== 2. k bo'yicha jadval (K-means) ===")
    print(f"  {'k':>3} {'silh':>8} {'DB':>7} {'CH':>9} {'barqaror':>10} "
          f"{'eng kichik %':>14}")
    natijalar = {}
    for k in range(2, 9):
        yorliq = KMeans(k, n_init=10, random_state=0).fit_predict(Xs)
        b = barqarorlik(Xs, lambda k=k: KMeans(k, n_init=10, random_state=0),
                        B=8)
        kichik = np.bincount(yorliq).min() / len(Xs)
        natijalar[k] = (silhouette_score(Xs, yorliq), b, kichik)
        print(f"  {k:>3} {silhouette_score(Xs, yorliq):>8.4f} "
              f"{davies_bouldin_score(Xs, yorliq):>7.4f} "
              f"{calinski_harabasz_score(Xs, yorliq):>9.1f} {b:>10.4f} "
              f"{kichik:>13.1%}")

    print("\n=== 3. Algoritmlarni taqqoslash (k = 5) ===")
    nomzodlar = {
        "KMeans": lambda: KMeans(5, n_init=10, random_state=0),
        "GMM(full)": lambda: GaussianMixture(5, covariance_type="full",
                                             n_init=5, random_state=0),
        "Ward": lambda: AgglomerativeClustering(5, linkage="ward"),
        "DBSCAN": lambda: DBSCAN(eps=0.45, min_samples=20),
    }
    print(f"  {'algoritm':<12} {'qamrov':>8} {'silh':>8} {'barqaror':>10} "
          f"{'ARI (haqiqiy)':>15}")
    for nom, yaratuvchi in nomzodlar.items():
        yorliq = yaratuvchi().fit_predict(Xs)
        m = yorliq != -1
        k = len(set(yorliq[m]))
        if k < 2:
            print(f"  {nom:<12} (bitta klaster)")
            continue
        b = barqarorlik(Xs, yaratuvchi, B=8) if nom != "DBSCAN" else float("nan")
        print(f"  {nom:<12} {m.mean():>7.1%} "
              f"{silhouette_score(Xs[m], yorliq[m]):>8.4f} {b:>10.4f} "
              f"{adjusted_rand_score(haqiqiy, yorliq):>15.4f}")

    print("\n=== 4. Yakuniy tanlov ===")
    nomzod_k = [k for k in range(3, 8) if natijalar[k][2] >= 0.05]
    eng = max(nomzod_k, key=lambda k: natijalar[k][1])
    print(f"  nomzodlar (eng kichik klaster >= 5%): {nomzod_k}")
    print(f"  barqarorligi eng yuqorisi: k = {eng} "
          f"({natijalar[eng][1]:.4f})")
    print(f"  silhouette: {natijalar[eng][0]:.4f}")
    print(f"  QAROR: KMeans(k={eng}) - barqaror, tez, predict bor")
    print("  ⭐ Jadval qarorni asoslaydi")


if __name__ == "__main__":
    main()

=== 1. Tuzilma bormi (gap statistikasi) ===
    k       gap      s_k
    1    0.7819   0.0070
    2    1.1422   0.0074
    3    1.4492   0.0061
    4    1.8810   0.0038
    5    2.1758   0.0061
    6    2.1668   0.0078
  gap tanlagan k = 5 (1 bo'lsa tuzilma yo'q)

=== 2. k bo'yicha jadval (K-means) ===
    k     silh      DB        CH   barqaror   eng kichik %
    2   0.4950  0.8670    5549.9     0.9969         35.6%
    3   0.5483  0.5715    5617.1     0.9995          9.7%
    4   0.5191  0.7022    7710.3     0.9987          9.6%
    5   0.5560  0.6170    9189.2     0.9980          9.6%
    6   0.4593  0.9393    8080.0     0.9604          9.6%
    7   0.3591  1.2166    7319.2     0.8239          9.6%
    8   0.3512  1.1777    6657.2     0.8676          9.6%

=== 3. Algoritmlarni taqqoslash (k = 5) ===
  algoritm       qamrov     silh   barqaror   ARI (haqiqiy)
  KMeans        100.0%   0.5560     0.9980          0.9877
  GMM(full)     100.0%   0.5553     0.9989          0.9881
  Ward          100.0%   0.5545     0.9792          0.9810
  DBSCAN         97.8%   0.5541        nan          0.4466

=== 4. Yakuniy tanlov ===
  nomzodlar (eng kichik klaster >= 5%): [3, 4, 5, 6, 7]
  barqarorligi eng yuqorisi: k = 3 0.9995-bob
  silhouette: 0.5483
  QAROR: KMeans(k=3) - barqaror, tez, predict bor
  ⭐ Jadval qarorni asoslaydi

"""8-9 qadamlar (real numpy/pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.ensemble import HistGradientBoostingRegressor, IsolationForest
from sklearn.metrics import r2_score, silhouette_samples
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler


def yarat(seed: int = 42, n: int = 5000) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    segment = rng.choice(5, n, p=[0.30, 0.25, 0.20, 0.15, 0.10])
    profil = np.array([[25.0, 4.0, 180_000.0, 2.0], [8.0, 14.0, 95_000.0, 5.0],
                       [75.0, 2.0, 420_000.0, 1.5], [120.0, 1.0, 150_000.0, 1.0],
                       [15.0, 9.0, 850_000.0, 6.0]])
    X = profil[segment] * rng.lognormal(0, 0.28, (n, 4))
    df = pd.DataFrame(X, columns=["oxirgi_kun", "xaridlar", "ortacha_chek",
                                  "toifalar"])
    # klasterlashda ISHLATILMAYDIGAN o'zgaruvchi - tekshiruv uchun
    kelgusi = (profil[segment, 1] * profil[segment, 2] / 30.0
               * rng.lognormal(0, 0.4, n))
    df["kelgusi_oy_xarid"] = kelgusi
    df["segment"] = segment
    anomaliya = rng.random(n) < 0.012
    df.loc[anomaliya, "ortacha_chek"] *= rng.uniform(15, 40, anomaliya.sum())
    df.loc[anomaliya, "xaridlar"] *= rng.uniform(3, 8, anomaliya.sum())
    return df


def main() -> None:
    df = yarat()
    nomlar = ["oxirgi_kun", "xaridlar", "ortacha_chek", "toifalar"]
    X = df[nomlar].to_numpy()
    izo = IsolationForest(contamination=0.015, n_estimators=200,
                          random_state=0, n_jobs=1).fit(X)
    toza = izo.predict(X) == 1
    sc = StandardScaler().fit(np.log1p(X[toza]))
    Xs = sc.transform(np.log1p(X[toza]))
    km = KMeans(5, n_init=10, random_state=0).fit(Xs)
    yorliq = km.labels_
    dft = df.loc[toza].copy()
    dft["klaster"] = yorliq

    print("=== 1. Segment profillari (asl birliklarda) ===")
    markazlar = np.expm1(sc.inverse_transform(km.cluster_centers_))
    umumiy = dft[nomlar].median().to_numpy()
    print(f"  {'k':>3} {'n':>6} {'ulush':>7} " + "".join(f"{n:>15}" for n in nomlar))
    for k in range(5):
        n = int((yorliq == k).sum())
        print(f"  {k:>3} {n:>6} {n / len(yorliq):>6.1%} "
              + "".join(f"{markazlar[k, i]:>15,.0f}" for i in range(len(nomlar))))
    print(f"  {'umumiy mediana':<17}" + "".join(f"{v:>15,.0f}" for v in umumiy))

    print("\n=== 2. Eng farq qiladigan belgilar ===")
    for k in range(5):
        farq = (markazlar[k] - umumiy) / umumiy
        tartib = np.argsort(-np.abs(farq))[:3]
        tavsif = ", ".join(f"{nomlar[i]} {farq[i]:+.0%}" for i in tartib)
        print(f"  klaster {k}: {tavsif}")

    print("\n=== 3. Segment nomlari ===")
    segment_nomlari = {}
    for k in range(5):
        m = markazlar[k]
        if m[0] > umumiy[0] * 2:
            nom = "Uyquda"
        elif m[2] > umumiy[2] * 2.5:
            nom = "Qimmatli"
        elif m[1] > umumiy[1] * 1.8:
            nom = "Tez-tez, kichik chek"
        elif m[0] > umumiy[0] * 1.3:
            nom = "Kamdan-kam, katta chek"
        else:
            nom = "Oddiy faol"
        segment_nomlari[k] = nom
    for k in range(5):
        print(f"  klaster {k} -> \"{segment_nomlari[k]}\" "
              f"({(yorliq == k).mean():.1%})")

    print("\n=== 4. Bashorat testi (klasterlashda ishlatilmagan o'zgaruvchi) ===")
    y = dft["kelgusi_oy_xarid"].to_numpy()
    # faqat segment yorlig'i bilan bashorat
    bir_issiq = np.eye(5)[yorliq]
    r2_segment = cross_val_score(HistGradientBoostingRegressor(max_iter=100,
                                                               random_state=0),
                                 bir_issiq, y, cv=3, scoring="r2").mean()
    r2_belgilar = cross_val_score(HistGradientBoostingRegressor(max_iter=100,
                                                                random_state=0),
                                  Xs, y, cv=3, scoring="r2").mean()
    print(f"  faqat segment yorlig'i bilan: R^2 {r2_segment:.4f}")
    print(f"  to'liq belgilar bilan:        R^2 {r2_belgilar:.4f}")
    print(f"  segment yorlig'i ma'lumotning "
          f"{r2_segment / max(r2_belgilar, 1e-9):.0%} ini saqlaydi")
    print(f"  {'segment':<24} {'kelgusi oy o_rtacha':>22}")
    for k in range(5):
        print(f"  {segment_nomlari[k]:<24} "
              f"{y[yorliq == k].mean():>22,.0f}")
    print("  ⭐ Segmentlar ishlatilmagan o'zgaruvchini bashorat qila oladi")


if __name__ == "__main__":
    main()

=== 1. Segment profillari (asl birliklarda) ===
    k      n   ulush      oxirgi_kun       xaridlar   ortacha_chek       toifalar
    0    745  15.1%             120              1        152,323              1
    1   1234  25.1%               8             14         96,583              5
    2   1518  30.8%              25              4        181,149              2
    3    480   9.7%              15              9        855,055              6
    4    948  19.2%              74              2        427,919              2
  umumiy mediana                25              4        177,589              2

=== 2. Eng farq qiladigan belgilar ===
  klaster 0: oxirgi_kun +378%, xaridlar -74%, toifalar -52%
  klaster 1: xaridlar +247%, toifalar +137%, oxirgi_kun -68%
  klaster 2: toifalar -5%, ortacha_chek +2%, oxirgi_kun +0%
  klaster 3: ortacha_chek +381%, toifalar +182%, xaridlar +123%
  klaster 4: oxirgi_kun +195%, ortacha_chek +141%, xaridlar -49%

=== 3. Segment nomlari ===
  klaster 0 -> "Uyquda" (15.1%)
  klaster 1 -> "Tez-tez, kichik chek" (25.1%)
  klaster 2 -> "Oddiy faol" (30.8%)
  klaster 3 -> "Qimmatli" (9.7%)
  klaster 4 -> "Uyquda" (19.2%)

=== 4. Bashorat testi (klasterlashda ishlatilmagan o'zgaruvchi) ===
  faqat segment yorlig'i bilan: R^2 0.7757
  to'liq belgilar bilan:        R^2 0.7363
  segment yorlig'i ma'lumotning 105% ini saqlaydi
  segment                     kelgusi oy o_rtacha
  Uyquda                                    5,883
  Tez-tez, kichik chek                     47,106
  Oddiy faol                               26,357
  Qimmatli                                284,829
  Uyquda                                   30,579
  ⭐ Segmentlar ishlatilmagan o'zgaruvchini bashorat qila oladi

"""10-qadam: Pipeline, biriktirish va hujjat (real pandas/sklearn/joblib)."""

import io
import pickle

import numpy as np
import pandas as pd
import sklearn
from sklearn.cluster import KMeans
from sklearn.ensemble import IsolationForest
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler


def yarat(seed: int = 42, n: int = 5000) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    segment = rng.choice(5, n, p=[0.30, 0.25, 0.20, 0.15, 0.10])
    profil = np.array([[25.0, 4.0, 180_000.0, 2.0], [8.0, 14.0, 95_000.0, 5.0],
                       [75.0, 2.0, 420_000.0, 1.5], [120.0, 1.0, 150_000.0, 1.0],
                       [15.0, 9.0, 850_000.0, 6.0]])
    X = profil[segment] * rng.lognormal(0, 0.28, (n, 4))
    return pd.DataFrame(X, columns=["oxirgi_kun", "xaridlar", "ortacha_chek",
                                    "toifalar"])


def hajm_kb(obyekt) -> float:
    b = io.BytesIO()
    pickle.dump(obyekt, b, protocol=pickle.HIGHEST_PROTOCOL)
    return b.tell() / 1024


def main() -> None:
    df = yarat()
    nomlar = list(df.columns)
    X = df.to_numpy()

    print("=== 1. Yakuniy quvur ===")
    quvur = Pipeline([
        ("log", FunctionTransformer(np.log1p, inverse_func=np.expm1,
                                    validate=True)),
        ("sc", StandardScaler()),
        ("km", KMeans(n_clusters=5, n_init=10, random_state=0))])
    yorliq = quvur.fit_predict(X)
    print(f"  bosqichlar: {[n for n, _ in quvur.steps]}")
    print(f"  klaster o'lchamlari: {np.bincount(yorliq).tolist()}")
    print(f"  inersiya: {quvur['km'].inertia_:.2f}")

    print("\n=== 2. Yangi mijozni biriktirish ===")
    yangi = pd.DataFrame({
        "oxirgi_kun": [5.0, 140.0, 20.0],
        "xaridlar": [18.0, 1.0, 5.0],
        "ortacha_chek": [80_000.0, 160_000.0, 900_000.0],
        "toifalar": [6.0, 1.0, 5.0]})
    segment_nomlari = {0: "Oddiy faol", 1: "Tez-tez, kichik chek",
                       2: "Kamdan-kam, katta chek", 3: "Uyquda",
                       4: "Qimmatli"}
    bashorat = quvur.predict(yangi.to_numpy())
    markazlar = np.expm1(quvur["sc"].inverse_transform(
        quvur["km"].cluster_centers_))
    Xs_yangi = quvur[:-1].transform(yangi.to_numpy())
    masofa = quvur["km"].transform(Xs_yangi).min(axis=1)
    print(f"  {'mijoz':>6} {'segment':>9} {'markazgacha masofa':>21} "
          f"{'ishonch':>10}")
    for i, (k, d) in enumerate(zip(bashorat, masofa)):
        ishonch = "yuqori" if d < 1.5 else ("o'rta" if d < 2.5 else "past")
        print(f"  {i:>6} {int(k):>9} {d:>21.3f} {ishonch:>10}")

    print("\n=== 3. Model paketi ===")
    paket = {
        "quvur": quvur,
        "belgilar": nomlar,
        "segment_nomlari": segment_nomlari,
        "markazlar_asl": markazlar.round(1).tolist(),
        "ulushlar": (np.bincount(yorliq) / len(yorliq)).round(4).tolist(),
        "versiyalar": {"sklearn": sklearn.__version__,
                       "numpy": np.__version__, "pandas": pd.__version__},
        "sana": "2026-09-21",
        "o_quv_hajmi": len(df),
    }
    print(f"  kalitlar: {sorted(paket)}")
    print(f"  paket hajmi: {hajm_kb(paket):.1f} KB")
    print(f"  ulushlar: {paket['ulushlar']}")

    print("\n=== 4. Segmentatsiya kartasi ===")
    karta = [
        ("Ma'lumot", f"{len(df)} mijoz, 4 belgi, 2026-09 holati"),
        ("Tayyorlash", "log1p (qiyshiqlik > 1) + StandardScaler"),
        ("Anomaliya", "IsolationForest(0.015) - alohida ko'rib chiqiladi"),
        ("Algoritm", "KMeans(k=5, n_init=10, random_state=0)"),
        ("k tanlovi", "gap + silhouette + barqarorlik jadvali"),
        ("Barqarorlik", "bootstrap ARI (16.3 usuli)"),
        ("Tekshiruv", "bashorat testi: kelgusi oy xaridi"),
        ("Biriktirish", "quvur.predict(yangi_df) - K-means da mavjud"),
        ("Qayta klasterlash", "har chorakda yoki ulush PSI > 0.25 bo'lsa"),
        ("Cheklovlar", "faqat >= 1 xaridi bor mijozlar uchun"),
    ]
    for kalit, qiymat in karta:
        print(f"  {kalit:<20}: {qiymat}")
    print("  ⭐ Karta + paket = topshirishga tayyor natija")


if __name__ == "__main__":
    main()

=== 1. Yakuniy quvur ===
  bosqichlar: ['log', 'sc', 'km']
  klaster o'lchamlari: [1253, 960, 493, 757, 1537]
  inersiya: 2345.05

=== 2. Yangi mijozni biriktirish ===
   mijoz   segment    markazgacha masofa    ishonch
       0         0                 0.655     yuqori
       1         3                 0.172     yuqori
       2         2                 0.825     yuqori

=== 3. Model paketi ===
  kalitlar: ['belgilar', 'markazlar_asl', 'o_quv_hajmi', 'quvur', 'sana', 'segment_nomlari', 'ulushlar', 'versiyalar']
  paket hajmi: 21.5 KB
  ulushlar: [0.2506, 0.192, 0.0986, 0.1514, 0.3074]

=== 4. Segmentatsiya kartasi ===
  Ma'lumot            : 5000 mijoz, 4 belgi, 2026-09 holati
  Tayyorlash          : log1p (qiyshiqlik > 1) + StandardScaler
  Anomaliya           : IsolationForest(0.015) - alohida ko'rib chiqiladi
  Algoritm            : KMeans(k=5, n_init=10, random_state=0)
  k tanlovi           : gap + silhouette + barqarorlik jadvali
  Barqarorlik         : bootstrap ARI (16.3 usuli)
  Tekshiruv           : bashorat testi: kelgusi oy xaridi
  Biriktirish         : quvur.predict(yangi_df) - K-means da mavjud
  Qayta klasterlash   : har chorakda yoki ulush PSI > 0.25 bo'lsa
  Cheklovlar          : faqat >= 1 xaridi bor mijozlar uchun
  ⭐ Karta + paket = topshirishga tayyor natija

KMeans(5).fit(Xs)                # 60 ta ulgurji xaridor bor     # ⚠️
# avval IsolationForest, keyin qaror                             # ✅

KMeans(5).fit(Xs)                # tuzilma bormi?                # ⚠️
# gap statistikasi + barqarorlik                                 # ✅

StandardScaler().fit_transform(df[["monetary"]])   # qiyshiqlik 8 # ⚠️
StandardScaler().fit_transform(np.log1p(df[["monetary"]]))       # ✅

# "Klaster 0", "Klaster 1", ...                                  # ⚠️
# "Uyquda", "Qimmatli", "Tez-tez, kichik chek"                   # ✅

# faqat yorliqlar ro'yxatini berish                              # ⚠️
joblib.dump({"quvur": quvur, "nomlar": nomlar}, "segment.joblib") # ✅

print(km.cluster_centers_)       # -0.42, 1.17                   # ⚠️
print(np.expm1(sc.inverse_transform(km.cluster_centers_)))       # ✅

# "5 ta segment topdik"                                          # ⚠️
# barqarorlik + bashorat testi + domen tasdig'i                  # ✅

1.  # oqimning birinchi uch qadami?
2.  # anomaliya qachon tekshiriladi?
3.  # tuzilma borligini nima ko'rsatadi?
4.  # qiyshiq summalar uchun?
5.  # uch tekshiruv?
6.  # bashorat testi nima?
7.  # yaxshi segment nomi qanday?
8.  # markazlar qanday ko'rsatiladi?
9.  # predict qaysi algoritmlarda bor?
10. # DBSCAN da nima qilish kerak?
11. # segmentatsiya kartasida nima bo'ladi?
12. # qayta klasterlash qachon?

1.  KMeans(5).fit(Xs)   # anomaliya tekshirilmagan

2.  StandardScaler().fit_transform(df[["monetary"]])   # qiyshiqlik 8

3.  # "Klaster 0", "Klaster 1"

4.  print(km.cluster_centers_)   # hisobot uchun

5.  # faqat yorliqlar ro'yxatini topshirish