17.4-dars: Masshtablash va normallashtirish

MAJBURIY:
  KNN, K-means, DBSCAN, PCA        - masofaga asoslangan
  SVM (ayniqsa RBF)                - yadro masofaga bog'liq
  Neyron tarmoqlar                 - gradient tushish barqarorligi
  Ridge/Lasso/ElasticNet           - jarima koeffitsiyent kattaligiga bog'liq

FOYDALI, LEKIN MAJBURIY EMAS:
  Logistik/chiziqli regressiya     - yaqinlashish tezlashadi
                                     (koeffitsiyentlarni taqqoslash uchun kerak)

KERAK EMAS:
  Qaror daraxti, RandomForest, ExtraTrees
  Gradient boosting (XGBoost, LightGBM, HistGB)
  Naive Bayes (ba'zi variantlari)

StandardScaler:   (x - mean) / std        -> o'rtacha 0, std 1
  + eng keng tarqalgan, ko'p usulga mos
  - chetlanishlarga SEZGIR (mean va std ularga bog'liq)
  - diapazon cheklanmagan

MinMaxScaler:     (x - min) / (max - min) -> [0, 1]
  + qat'iy diapazon (neyron tarmoqlar, tasvir)
  - chetlanishlarga JUDA sezgir (bitta chetlanish hammani siqadi)

RobustScaler:     (x - mediana) / IQR
  + chetlanishlarga CHIDAMLI
  - diapazon cheklanmagan

MaxAbsScaler:     x / max(|x|)            -> [-1, 1]
  + siyrak matritsani buzmaydi (markazlashtirmaydi)

Normalizer:       har QATORni birlik uzunlikka keltiradi
  + matn/kosinus o'xshashlik uchun
  - ustun emas, QATOR bo'yicha ishlaydi (boshqa g'oya)

Bitta katta chetlanish nima qiladi:

  StandardScaler: std oshadi -> qolgan qiymatlar 0 atrofiga SIQILADI
  MinMaxScaler:   max oshadi -> qolganlar [0, 0.01] ga siqiladi
  RobustScaler:   mediana va IQR o'zgarmaydi -> ta'sir MINIMAL

Misol: [1, 2, 3, 4, 1000]
  StandardScaler -> [-0.5, -0.5, -0.5, -0.5, 2.0]
  MinMaxScaler   -> [0.000, 0.001, 0.002, 0.003, 1.0]
  RobustScaler   -> [-1.0, -0.5, 0.0, 0.5, 498.5]

QAROR: chetlanishlarni AVVAL hal qiling (kesish, o'chirish, alohida belgi),
       keyin masshtablang

from sklearn.preprocessing import PowerTransformer, QuantileTransformer

PowerTransformer(method="yeo-johnson")        # manfiy qiymatlar ham
PowerTransformer(method="box-cox")            # faqat x > 0
QuantileTransformer(output_distribution="normal", n_quantiles=1000)
QuantileTransformer(output_distribution="uniform")

PowerTransformer: optimal daraja lambda ni O'ZI topadi
  + qiyshiqlikni kamaytiradi va masshtablaydi
  - monoton EMAS deb o'ylash xato - u monoton
  - talqin qiyinlashadi

QuantileTransformer: taqsimotni majburan normal/tekis qiladi
  + chetlanishlarni butunlay yo'qotadi
  - ma'lumotni KUCHLI o'zgartiradi (masofalar buziladi)
  - n_quantiles < n bo'lsin

NOTO'G'RI:
  Xs = StandardScaler().fit_transform(X)      # BUTUN ma'lumot
  cross_val_score(model, Xs, y)               # test statistikani ko'rgan

TO'G'RI:
  Pipeline([("sc", StandardScaler()), ("m", model)])
  cross_val_score(quvur, X, y)                # har foldda qayta fit

TA'SIRI: odatda kichik (0.001-0.01), lekin:
  - kichik ma'lumotda sezilarli
  - chetlanishlar bo'lsa katta
  - vaqt qatorlarida jiddiy

QOIDA: har qanday fit qiluvchi transformatsiya Pipeline ichida

Siyrak matritsada (TF-IDF, one-hot) MARKAZLASHTIRISH xavfli:
  (x - mean) barcha nol qiymatlarni NOLGA TENG BO'LMAGAN qiladi
  -> siyraklik yo'qoladi -> xotira portlaydi

YECHIM:
  StandardScaler(with_mean=False)    - faqat std ga bo'ladi
  MaxAbsScaler()                     - markazlashtirmaydi
  Normalizer()                       - qator bo'yicha

sklearn siyrak kirishda with_mean=True bo'lsa XATO beradi

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import (MaxAbsScaler, MinMaxScaler, Normalizer,
                                   PowerTransformer, QuantileTransformer,
                                   RobustScaler, StandardScaler)

Pipeline([("sc", StandardScaler()), ("m", model)])       # DOIM Pipeline ichida

StandardScaler()                      # odatiy
RobustScaler(quantile_range=(25, 75))  # chetlanishlar bo'lsa
MinMaxScaler(feature_range=(0, 1))     # qat'iy diapazon
MaxAbsScaler()                         # siyrak matritsa
StandardScaler(with_mean=False)        # siyrak matritsa
PowerTransformer(method="yeo-johnson") # qiyshiqlik + masshtab

sc.inverse_transform(Z)                # talqin uchun
QOIDA: modelga qarab tanla · chetlanishni avval hal qil ·
       Pipeline ichida · siyrakda markazlashtirma

Majburiy: KNN, K-means, PCA, SVM, NN, Ridge/Lasso
Keraksiz: daraxtlar va boosting
Standard (odatiy) / Robust (chetlanish) / MinMax (diapazon) / MaxAbs (siyrak)
fit faqat o'quvda -> Pipeline MAJBURIY

"""Masshtablashning modelga qarab ta'siri (real numpy/sklearn)."""

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression, RidgeClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.calibration import CalibratedClassifierCV
from sklearn.svm import SVC


def yarat(seed: int = 7, n: int = 3000):
    """Uch belgi, birliklari juda har xil."""
    rng = np.random.default_rng(seed)
    daromad = rng.lognormal(14.5, 0.5, n)          # millionlar
    yosh = rng.integers(18, 70, n).astype(float)   # o'nlar
    ball = rng.uniform(0, 1, n)                    # 0-1
    kuch = (1.2 * (np.log(daromad) - 14.5) - 0.04 * (yosh - 44)
            + 2.0 * (ball - 0.5))
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    return np.column_stack([daromad, yosh, ball]), y


def main() -> None:
    X, y = yarat()
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Belgilarning masshtabi ===")
    nomlar = ["daromad", "yosh", "ball"]
    for i, nom in enumerate(nomlar):
        print(f"  {nom:<10}: o'rtacha {X[:, i].mean():>14,.2f}, "
              f"std {X[:, i].std():>14,.2f}")

    print("\n=== 2. Masshtablash bilan va bilansiz ===")
    modellar = {
        "KNN(15)": lambda: KNeighborsClassifier(15),
        "SVC(rbf)": lambda: CalibratedClassifierCV(SVC(random_state=0),
                                                   ensemble=False, cv=3),
        "LogReg": lambda: LogisticRegression(max_iter=5000),
        "Ridge": lambda: RidgeClassifier(alpha=1.0),
        "RandomForest": lambda: RandomForestClassifier(n_estimators=150,
                                                       random_state=0,
                                                       n_jobs=1),
        "HistGB": lambda: HistGradientBoostingClassifier(learning_rate=0.1,
                                                         max_iter=150,
                                                         random_state=0),
    }
    print(f"  {'model':<14} {'masshtabsiz':>13} {'masshtab bilan':>16} "
          f"{'farq':>9}")
    for nom, yaratuvchi in modellar.items():
        ball_nomi = "roc_auc" if nom != "Ridge" else "accuracy"
        xom = cross_val_score(yaratuvchi(), X, y, cv=cv,
                              scoring=ball_nomi).mean()
        sc = cross_val_score(Pipeline([("sc", StandardScaler()),
                                       ("m", yaratuvchi())]),
                             X, y, cv=cv, scoring=ball_nomi).mean()
        print(f"  {nom:<14} {xom:>13.4f} {sc:>16.4f} {sc - xom:>+9.4f}")

    print("\n=== 3. KNN da qaysi belgi hukmron ===")
    from sklearn.neighbors import NearestNeighbors
    for nom, Xa in [("xom", X), ("masshtablangan",
                                 StandardScaler().fit_transform(X))]:
        nn = NearestNeighbors(n_neighbors=11).fit(Xa)
        _, idx = nn.kneighbors(Xa[:300])
        # qo'shnilar har belgi bo'yicha qanchalik yaqin
        farqlar = []
        for i in range(3):
            asl = X[:300, i]
            qoshni = X[idx[:, 1:], i].mean(axis=1)
            farqlar.append(np.abs(asl - qoshni).mean() / X[:, i].std())
        print(f"  {nom:<16}: normallashgan farqlar "
              f"{[round(v, 3) for v in farqlar]}")
    print("  (xom holatda daromad bo'yicha juda yaqin, boshqalarda uzoq)")

    print("\n=== 4. Ridge jarimasining adolatsizligi ===")
    from sklearn.linear_model import Ridge
    for nom, Xa in [("xom", X), ("masshtablangan",
                                 StandardScaler().fit_transform(X))]:
        m = Ridge(alpha=100.0).fit(Xa, y)
        print(f"  {nom:<16}: koeffitsiyentlar "
              f"{np.round(m.coef_, 6).tolist()}")
    print("  (xom holatda daromad koeffitsiyenti juda kichik -> kam jarima)")
    print("  ⭐ Regulyarizatsiyada masshtablash majburiy")


if __name__ == "__main__":
    main()

=== 1. Belgilarning masshtabi ===
  daromad   : o'rtacha   2,210,099.91, std   1,174,979.27
  yosh      : o'rtacha          42.90, std          15.02
  ball      : o'rtacha           0.51, std           0.29

=== 2. Masshtablash bilan va bilansiz ===
  model            masshtabsiz   masshtab bilan      farq
  KNN(15)               0.5839           0.7000   +0.1161
  SVC(rbf)              0.6120           0.7173   +0.1053
  LogReg                0.7019           0.7316   +0.0297
  Ridge                 0.6647           0.6647   +0.0000
  RandomForest          0.6815           0.6812   -0.0002
  HistGB                0.6827           0.6827   +0.0000

=== 3. KNN da qaysi belgi hukmron ===
  xom             : normallashgan farqlar [np.float64(0.001), np.float64(0.903), np.float64(0.902)]
  masshtablangan  : normallashgan farqlar [np.float64(0.051), np.float64(0.042), np.float64(0.047)]
  (xom holatda daromad bo'yicha juda yaqin, boshqalarda uzoq)

=== 4. Ridge jarimasining adolatsizligi ===
  xom             : koeffitsiyentlar [0.0, -0.008665, 0.293175]
  masshtablangan  : koeffitsiyentlar [0.100677, -0.125962, 0.114845]
  (xom holatda daromad koeffitsiyenti juda kichik -> kam jarima)
  ⭐ Regulyarizatsiyada masshtablash majburiy

"""Beshta scaler, bitta chetlanish (real numpy/sklearn)."""

import numpy as np
from sklearn.preprocessing import (MaxAbsScaler, MinMaxScaler, PowerTransformer,
                                   QuantileTransformer, RobustScaler,
                                   StandardScaler)


def main() -> None:
    print("=== 1. Sodda misol: [1, 2, 3, 4, 1000] ===")
    x = np.array([[1.0], [2.0], [3.0], [4.0], [1000.0]])
    scalerlar = {
        "StandardScaler": StandardScaler(),
        "MinMaxScaler": MinMaxScaler(),
        "RobustScaler": RobustScaler(),
        "MaxAbsScaler": MaxAbsScaler(),
    }
    for nom, sc in scalerlar.items():
        z = sc.fit_transform(x).ravel()
        print(f"  {nom:<16}: {np.round(z, 4).tolist()}")

    print("\n=== 2. Real ma'lumotda chetlanish ta'siri ===")
    rng = np.random.default_rng(0)
    toza = rng.normal(50, 10, 1000)
    chetlangan = np.concatenate([toza, [5000.0, 6000.0]])
    print(f"  {'scaler':<16} {'toza std':>10} {'chetlangan std':>16} "
          f"{'99% diapazoni':>18}")
    for nom, sc in scalerlar.items():
        z1 = sc.fit_transform(toza.reshape(-1, 1)).ravel()
        z2 = sc.fit_transform(chetlangan.reshape(-1, 1)).ravel()
        past, yuqori = np.quantile(z2[:1000], [0.005, 0.995])
        print(f"  {nom:<16} {z1.std():>10.4f} {z2.std():>16.4f} "
              f"{yuqori - past:>18.4f}")
    print("  (RobustScaler chetlanishdan eng kam ta'sirlanadi)")

    print("\n=== 3. Taqsimotni o'zgartiruvchilar ===")
    qiyshiq = rng.lognormal(3, 1.2, 2000).reshape(-1, 1)
    import pandas as pd
    variantlar = {
        "xom": qiyshiq,
        "log1p": np.log1p(qiyshiq),
        "StandardScaler": StandardScaler().fit_transform(qiyshiq),
        "PowerTransformer": PowerTransformer().fit_transform(qiyshiq),
        "Quantile(normal)": QuantileTransformer(
            output_distribution="normal", n_quantiles=500,
            random_state=0).fit_transform(qiyshiq),
    }
    print(f"  {'usul':<20} {'qiyshiqlik':>12} {'ekssess':>10} "
          f"{'min':>9} {'max':>9}")
    for nom, z in variantlar.items():
        s = pd.Series(z.ravel())
        print(f"  {nom:<20} {s.skew():>12.3f} {s.kurtosis():>10.3f} "
              f"{s.min():>9.3f} {s.max():>9.3f}")

    print("\n=== 4. QuantileTransformer masofalarni buzadi ===")
    nuqtalar = np.array([[1.0], [2.0], [3.0], [100.0], [101.0]])
    fon = rng.lognormal(1, 1.5, 500).reshape(-1, 1)
    hammasi = np.vstack([fon, nuqtalar])
    qt = QuantileTransformer(output_distribution="uniform", n_quantiles=200,
                             random_state=0).fit(hammasi)
    z = qt.transform(nuqtalar).ravel()
    print(f"  asl nuqtalar: {nuqtalar.ravel().tolist()}")
    print(f"  asl masofalar: |1-2| = 1, |100-101| = 1")
    print(f"  Quantile dan keyin: {np.round(z, 4).tolist()}")
    print(f"  yangi masofalar: |z1-z2| = {abs(z[0] - z[1]):.4f}, "
          f"|z4-z5| = {abs(z[3] - z[4]):.4f}")
    print("  ⭐ QuantileTransformer - kuchli, lekin masofalarni o'zgartiradi")


if __name__ == "__main__":
    main()

=== 1. Sodda misol: [1, 2, 3, 4, 1000] ===
  StandardScaler  : [-0.5038, -0.5013, -0.4987, -0.4962, 2.0]
  MinMaxScaler    : [0.0, 0.001, 0.002, 0.003, 1.0]
  RobustScaler    : [-1.0, -0.5, 0.0, 0.5, 498.5]
  MaxAbsScaler    : [0.001, 0.002, 0.003, 0.004, 1.0]

=== 2. Real ma'lumotda chetlanish ta'siri ===
  scaler             toza std   chetlangan std      99% diapazoni
  StandardScaler       1.0000           1.0000             0.2176
  MinMaxScaler         0.1402           0.0408             0.0089
  RobustScaler         0.7481          18.6356             4.0553
  MaxAbsScaler         0.1211           0.0407             0.0089
  (RobustScaler chetlanishdan eng kam ta'sirlanadi)

=== 3. Taqsimotni o'zgartiruvchilar ===
  usul                   qiyshiqlik    ekssess       min       max
  xom                         4.400     28.610     0.433   711.296
  log1p                       0.243     -0.351     0.360     6.568
  StandardScaler              4.400     28.610    -0.644    11.045
  PowerTransformer            0.015     -0.460    -2.706     2.783
  Quantile(normal)            0.000      0.490    -5.199     5.199

=== 4. QuantileTransformer masofalarni buzadi ===
  asl nuqtalar: [1.0, 2.0, 3.0, 100.0, 101.0]
  asl masofalar: |1-2| = 1, |100-101| = 1
  Quantile dan keyin: [0.2462, 0.4092, 0.4979, 0.9794, 0.9799]
  yangi masofalar: |z1-z2| = 0.1631, |z4-z5| = 0.0005
  ⭐ QuantileTransformer - kuchli, lekin masofalarni o'zgartiradi

"""fit faqat o'quvda (real numpy/sklearn)."""

import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold, cross_val_score, train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.metrics import roc_auc_score


def yarat(seed: int = 3, n: int = 600, chetlanish: int = 0):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 6))
    kuch = 1.2 * X[:, 0] - 0.9 * X[:, 1] + 0.7 * X[:, 2]
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    if chetlanish:
        idx = rng.choice(n, chetlanish, replace=False)
        X[idx] *= rng.uniform(20, 60, (chetlanish, 1))
    return X, y


def main() -> None:
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Kichik ma'lumotda leakage ta'siri ===")
    print(f"  {'n':>6} {'leakage bilan':>15} {'Pipeline bilan':>16} "
          f"{'farq':>9}")
    for n in [150, 300, 600, 2000]:
        X, y = yarat(n=n)
        Xs = StandardScaler().fit_transform(X)          # NOTO'G'RI
        leak = cross_val_score(KNeighborsClassifier(11), Xs, y, cv=cv,
                               scoring="roc_auc").mean()
        togri = cross_val_score(Pipeline([("sc", StandardScaler()),
                                          ("m", KNeighborsClassifier(11))]),
                                X, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {n:>6} {leak:>15.4f} {togri:>16.4f} {leak - togri:>+9.4f}")

    print("\n=== 2. Chetlanishlar leakage ni kuchaytiradi ===")
    print(f"  {'chetlanishlar':>14} {'leakage bilan':>15} "
          f"{'Pipeline bilan':>16} {'farq':>9}")
    for nechta in [0, 5, 20, 50]:
        X, y = yarat(n=600, chetlanish=nechta)
        Xs = MinMaxScaler().fit_transform(X)
        leak = cross_val_score(KNeighborsClassifier(11), Xs, y, cv=cv,
                               scoring="roc_auc").mean()
        togri = cross_val_score(Pipeline([("sc", MinMaxScaler()),
                                          ("m", KNeighborsClassifier(11))]),
                                X, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {nechta:>14} {leak:>15.4f} {togri:>16.4f} "
              f"{leak - togri:>+9.4f}")

    print("\n=== 3. Test to'plamini alohida masshtablash (yana bir xato) ===")
    X, y = yarat(n=2000)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    # NOTO'G'RI: har to'plam o'z statistikasi bilan
    sc1, sc2 = StandardScaler().fit(Xtr), StandardScaler().fit(Xte)
    m1 = KNeighborsClassifier(11).fit(sc1.transform(Xtr), ytr)
    a_notogri = roc_auc_score(yte, m1.predict_proba(sc2.transform(Xte))[:, 1])
    # TO'G'RI: o'quv statistikasi test uchun ham
    m2 = KNeighborsClassifier(11).fit(sc1.transform(Xtr), ytr)
    a_togri = roc_auc_score(yte, m2.predict_proba(sc1.transform(Xte))[:, 1])
    print(f"  test alohida masshtablangan: AUC {a_notogri:.4f}")
    print(f"  o'quv statistikasi bilan:    AUC {a_togri:.4f}")
    print(f"  farq: {a_notogri - a_togri:+.4f}")

    print("\n=== 4. Pipeline har foldda qayta fit qiladi ===")
    from sklearn.base import BaseEstimator, TransformerMixin

    class KuzatuvchiScaler(StandardScaler):
        """fit necha marta chaqirilganini sanaydi."""
        sanoq = 0

        def fit(self, X, y=None, sample_weight=None):
            KuzatuvchiScaler.sanoq += 1
            return super().fit(X, y, sample_weight)

    KuzatuvchiScaler.sanoq = 0
    cross_val_score(Pipeline([("sc", KuzatuvchiScaler()),
                              ("m", KNeighborsClassifier(11))]),
                    X, y, cv=cv, scoring="roc_auc")
    print(f"  5-karra CV da scaler.fit chaqirildi: "
          f"{KuzatuvchiScaler.sanoq} marta")
    print(f"  o'quv qismining o'rtachasi har foldda BOSHQA")
    for nechanchi, (tr, _) in enumerate(cv.split(X, y)):
        if nechanchi < 3:
            print(f"    fold {nechanchi}: o'rtacha[0] = "
                  f"{X[tr, 0].mean():+.4f}, std[0] = {X[tr, 0].std():.4f}")
    print("  ⭐ Pipeline leakage ni tuzilma darajasida oldini oladi")


if __name__ == "__main__":
    main()

=== 1. Kichik ma'lumotda leakage ta'siri ===
       n   leakage bilan   Pipeline bilan      farq
     150          0.7464           0.7370   +0.0093
     300          0.8034           0.7964   +0.0069
     600          0.7686           0.7678   +0.0008
    2000          0.8024           0.8041   -0.0017

=== 2. Chetlanishlar leakage ni kuchaytiradi ===
   chetlanishlar   leakage bilan   Pipeline bilan      farq
               0          0.7752           0.7761   -0.0009
               5          0.7658           0.7624   +0.0033
              20          0.7765           0.7661   +0.0104
              50          0.7649           0.7649   -0.0000

=== 3. Test to'plamini alohida masshtablash (yana bir xato) ===
  test alohida masshtablangan: AUC 0.8040
  o'quv statistikasi bilan:    AUC 0.8073
  farq: -0.0033

=== 4. Pipeline har foldda qayta fit qiladi ===
  5-karra CV da scaler.fit chaqirildi: 5 marta
  o'quv qismining o'rtachasi har foldda BOSHQA
    fold 0: o'rtacha[0] = +0.0087, std[0] = 1.0351
    fold 1: o'rtacha[0] = +0.0127, std[0] = 1.0113
    fold 2: o'rtacha[0] = +0.0297, std[0] = 1.0051
  ⭐ Pipeline leakage ni tuzilma darajasida oldini oladi

"""Matn belgilarida masshtablash (real numpy/scipy/sklearn)."""

import numpy as np
from scipy import sparse
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import MaxAbsScaler, Normalizer, StandardScaler


def matn_yarat(seed: int = 4, n: int = 1200):
    rng = np.random.default_rng(seed)
    sinf = rng.integers(0, 2, n)
    lugat = {0: ["arzon", "chegirma", "aksiya", "tezkor", "bepul"],
             1: ["sifat", "kafolat", "asl", "sertifikat", "brend"]}
    umumiy = ["mahsulot", "buyurtma", "yetkazish", "narx", "xizmat"]
    matnlar = []
    for s in sinf:
        uzunlik = int(rng.integers(20, 120))
        sozlar = list(rng.choice(lugat[s], int(uzunlik * 0.4)))
        sozlar += list(rng.choice(umumiy, int(uzunlik * 0.4)))
        sozlar += list(rng.choice(lugat[1 - s], int(uzunlik * 0.2)))
        rng.shuffle(sozlar)
        matnlar.append(" ".join(sozlar))
    return matnlar, sinf


def main() -> None:
    matnlar, y = matn_yarat()
    X = TfidfVectorizer(min_df=2).fit_transform(matnlar)
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Siyrak matritsa ===")
    print(f"  o'lcham: {X.shape}")
    print(f"  nolga teng bo'lmagan elementlar: {X.nnz:,} "
          f"({X.nnz / (X.shape[0] * X.shape[1]):.2%})")
    print(f"  siyrak holda xotira: {X.data.nbytes / 1024:.1f} KB")
    print(f"  zich holda bo'lardi: "
          f"{X.shape[0] * X.shape[1] * 8 / 1024:.1f} KB")

    print("\n=== 2. Markazlashtirish siyraklikni buzadi ===")
    try:
        StandardScaler().fit_transform(X)
        print("  StandardScaler() ishladi (kutilmagan)")
    except (TypeError, ValueError) as xato:
        print(f"  StandardScaler() -> {type(xato).__name__}: "
              f"{str(xato)[:52]}...")
    zich = StandardScaler().fit_transform(X.toarray())
    print(f"  zichga aylantirib markazlashtirsak: "
          f"{(zich != 0).mean():.1%} element nolga teng emas")
    print(f"  xotira: {zich.nbytes / 1024:.1f} KB "
          f"({zich.nbytes / X.data.nbytes:.0f}x ko'proq)")

    print("\n=== 3. Siyrak uchun mos scaler lar ===")
    variantlar = {
        "masshtabsiz": None,
        "MaxAbsScaler": MaxAbsScaler(),
        "StandardScaler(with_mean=False)": StandardScaler(with_mean=False),
        "Normalizer(l2)": Normalizer(norm="l2"),
    }
    print(f"  {'scaler':<32} {'siyraklik saqlandi':>20} {'CV AUC':>9}")
    for nom, sc in variantlar.items():
        if sc is None:
            Xa, quvur = X, LogisticRegression(max_iter=2000)
        else:
            Xa = sc.fit_transform(X)
            quvur = Pipeline([("sc", sc),
                              ("m", LogisticRegression(max_iter=2000))])
        siyrak = "ha" if sparse.issparse(Xa) else "yo'q"
        b = cross_val_score(quvur, X, y, cv=cv, scoring="roc_auc").mean()
        print(f"  {nom:<32} {siyrak:>20} {b:>9.4f}")

    print("\n=== 4. TF-IDF allaqachon normallashtirilgan ===")
    normalar = np.sqrt(np.asarray(X.multiply(X).sum(axis=1))).ravel()
    print(f"  qator normalari: min {normalar.min():.4f}, "
          f"max {normalar.max():.4f}, std {normalar.std():.6f}")
    print(f"  (TfidfVectorizer standart holda norm='l2' qo'llaydi)")
    xom = TfidfVectorizer(min_df=2, norm=None).fit_transform(matnlar)
    xom_normalar = np.sqrt(np.asarray(xom.multiply(xom).sum(axis=1))).ravel()
    print(f"  norm=None bilan: min {xom_normalar.min():.2f}, "
          f"max {xom_normalar.max():.2f}")
    b_xom = cross_val_score(LogisticRegression(max_iter=2000), xom, y, cv=cv,
                            scoring="roc_auc").mean()
    b_norm = cross_val_score(LogisticRegression(max_iter=2000), X, y, cv=cv,
                             scoring="roc_auc").mean()
    print(f"  CV AUC: norm=None {b_xom:.4f}, norm='l2' {b_norm:.4f}")
    print("  ⭐ Siyrak ma'lumotda markazlashtirmang")


if __name__ == "__main__":
    main()

=== 1. Siyrak matritsa ===
  o'lcham: (1200, 15)
  nolga teng bo'lmagan elementlar: 17,121 (95.12%)
  siyrak holda xotira: 133.8 KB
  zich holda bo'lardi: 140.6 KB

=== 2. Markazlashtirish siyraklikni buzadi ===
  StandardScaler() -> ValueError: Cannot center sparse matrices: pass `with_mean=False...
  zichga aylantirib markazlashtirsak: 100.0% element nolga teng emas
  xotira: 140.6 KB (1x ko'proq)

=== 3. Siyrak uchun mos scaler lar ===
  scaler                             siyraklik saqlandi    CV AUC
  masshtabsiz                                        ha    1.0000
  MaxAbsScaler                                       ha    1.0000
  StandardScaler(with_mean=False)                    ha    1.0000
  Normalizer(l2)                                     ha    1.0000

=== 4. TF-IDF allaqachon normallashtirilgan ===
  qator normalari: min 1.0000, max 1.0000, std 0.000000
  (TfidfVectorizer standart holda norm='l2' qo'llaydi)
  norm=None bilan: min 6.26, max 35.97
  CV AUC: norm=None 1.0000, norm='l2' 1.0000
  ⭐ Siyrak ma'lumotda markazlashtirmang

Xs = StandardScaler().fit_transform(X); cross_val_score(m, Xs, y)  # ⚠️
cross_val_score(Pipeline([("sc", StandardScaler()), ("m", m)]), X, y) # ✅

Xte_s = StandardScaler().fit_transform(Xte)                        # ⚠️
Xte_s = sc.transform(Xte)        # o'quvda fit qilingan sc         # ✅

MinMaxScaler().fit(X)            # bitta 5000 lik qiymat bor       # ⚠️
RobustScaler()                   # yoki avval chetlanishni hal qiling # ✅

StandardScaler().fit(tfidf)      # TypeError                       # ⚠️
MaxAbsScaler()  yoki  StandardScaler(with_mean=False)              # ✅

Pipeline([("sc", StandardScaler()), ("m", RandomForestClassifier())]) # ⚠️
RandomForestClassifier()         # keraksiz murakkablik            # ✅

Normalizer().fit_transform(X)    # ustunlar masshtablanmadi        # ⚠️
StandardScaler()                 # ustun bo'yicha                  # ✅

print(km.cluster_centers_)       # masshtablangan qiymatlar        # ⚠️
print(sc.inverse_transform(km.cluster_centers_))                   # ✅

1.  # qaysi modellarga majburiy?
2.  # qaysilariga keraksiz?
3.  # StandardScaler formulasi?
4.  # RobustScaler formulasi?
5.  # MinMaxScaler ning zaifligi?
6.  # Normalizer nimani normallashtiradi?
7.  # siyrak uchun qaysi scaler?
8.  # nega markazlashtirish xavfli?
9.  # fit qayerda bajariladi?
10. # test qanday masshtablanadi?
11. # PowerTransformer nima qiladi?
12. # QuantileTransformer xavfi?

1.  Xs = StandardScaler().fit_transform(X); cross_val_score(m, Xs, y)

2.  Xte_s = StandardScaler().fit_transform(Xte)

3.  MinMaxScaler().fit(X)   # chetlanishlar bor

4.  StandardScaler().fit(tfidf_matritsa)

5.  Pipeline([("sc", StandardScaler()), ("m", RandomForestClassifier())])