6.6-dars: Standartlashtirish

6-QISM — MA'LUMOTNI TOZALASH · 6-dars

1. Kirish va motivatsiya

6.5-darsda normalizatsiya (Min-Max, 0-1) — ustunlarni bir masshtabga keltirish. Standartlashtirish (standardization) — masshtablashning boshqa keng tarqalgan usuli: z-score bilan ((x - o'rtacha) / std — 4.6) ustunni o'rtacha 0, standart og'ish 1 ga keltirish. Natijada har qiymat "o'rtachadan necha std uzoq" degan ma'noni oladi. Standartlashtirish normalizatsiyadan farq qiladi: chegara yo'q (0-1 emas — istalgan qiymat, odatda -3..+3), va outlierga chidamliroq (Min-Max max outlierga sezgir; z-score o'rtacha/std). Ko'p ML modellar (regressiya, SVM, neyron, PCA) standartlashtirishni afzal ko'radi. Bu darsda z-score standartlashtirish, normalizatsiyadan farqi, qachon qaysi, va StandardScaler. Nega muhim? (1) O'rtacha 0 — markazlashtirilgan; (2) Outlier — chidamliroq (Min-Max'dan); (3) Model — regressiya, SVM, PCA afzal. Bu dars standartlashtirishni o'rgatadi — z-score bilan masshtablash.

Standartlashtirish (standardization) — z-score bilan: formula ((x - o'rtacha) / std — 4.6; o'rtacha 0, std 1), sklearn (StandardScaler), normalizatsiyadan farq (6.5 — Min-Max 0-1 chegara; z-score chegarasiz, outlierga chidamliroq), qachon (regressiya, SVM, neyron, PCA), z-score ma'nosi (o'rtachadan necha std). Foydalanish: markazlashtirish, model uchun (outlierga chidamliroq). Bu 6.5 (normalizatsiya), 4.6 (z-score/normal), 20-qism (ML) bilan bog'liq. Standartlashtirish — z-score (o'rtacha 0, std 1). Standart. z-score.

Real vaziyat. Data Scientist regressiya modeli qurmoqchi (yosh, daromad — turli masshtab; daromadda bir necha yuqori outlier). Min-Max (6.5): max outlier → diapazon ulkan → qolgan 0'ga siqiladi (buziladi). Standartlashtirish (z-score): (x - o'rtacha) / std — o'rtacha 0, std 1; outlier — katta z (3, 4) lekin qolgan qiymatlar siqilmaydi (o'rtacha atrofida -2..+2; outlier chetda, lekin markaz buzilmaydi). Data Scientist standartlashtirdi (z-score — o'rtacha 0, std 1), model yaxshi (outlierga chidamliroq; regressiya afzal). Standartlashtirish — z-score bilan masshtablash.

Bu darsda standartlashtirishni o'rganamiz.

Bu darsda:

• z-score standartlashtirish (o'rtacha 0, std 1)
• Normalizatsiyadan farqi
• sklearn StandardScaler
• Qachon qaysi (normalizatsiya vs standartlashtirish)
• RobustScaler (outlier)
• Standartlashtirish amaliyoti
• Standartlashtirish tuzoqlari
• Amaliy: standartlashtirish

ℹ Misollar real pandas/numpy/sklearn bilan (Python 3.14) ishlaydi.

2. Nazariya — chuqur tushuntirish

2.1. z-score standartlashtirish (o'rtacha 0, std 1)

Formulasi:

# Standartlashtirish (z-score): (x - o'rtacha) / std
standardized = (x - x.mean()) / x.std()

# natija: o'rtacha 0, standart og'ish 1
# har qiymat "o'rtachadan necha std uzoq" (4.6)
# odatda -3..+3 (chegara yo'q)

z-score standartlashtirish (o'rtacha 0, std 1) — formula: (x - o'rtacha) / std (z-score — 4.6; har qiymatdan o'rtacha ayirib, std'ga bo'lish); natija o'rtacha 0, std 1 (markazlashtirilgan; har qiymat "o'rtachadan necha std uzoq"); odatda -3..+3 (chegara yo'q). Sabab: turli masshtab → standart (o'rtacha 0, std 1 — teng); z-score (4.6 — normal; "necha std"); Min-Max (0-1 chegara) o'rniga (chegarasiz — outlier siqmaydi). (x-mean)/std (z-score), o'rtacha 0 (markaz), std 1 (masshtab), chegarasiz (-3..+3). z-score standartlashtirish — (x-mean)/std (o'rtacha 0, std 1). z-score. Standart.

2.2. Normalizatsiyadan farqi

Normalizatsiyadan farqi — Min-Max vs z-score: Min-Max (6.5 — (x-min)/(max-min); 0-1 chegara; min/max ga bog'liq — outlierga sezgir), z-score ((x-mean)/std; chegarasiz — o'rtacha 0, -3..+3; o'rtacha/std ga bog'liq — outlierga chidamliroq). Sabab: chegara (Min-Max — 0-1 aniq; z-score — chegarasiz), outlier (Min-Max — max outlier diapazonni buzadi, qolgan siqiladi; z-score — outlier katta z, lekin markaz siqmaydi), taqsimot (Min-Max — shakl saqlanadi; z-score — markazlashtirilgan). Min-Max (0-1, outlier sezgir), z-score (chegarasiz, chidamliroq). Normalizatsiyadan farq — Min-Max (0-1) vs z-score (chegarasiz, chidamliroq). Farq. Chegara.

2.3. sklearn StandardScaler

sklearn StandardScaler — kutubxona usuli: from sklearn.preprocessing import StandardScaler; scaler.fit_transform(X_train) (o'quv — o'rgan+qo'lla; o'rtacha/std), scaler.transform(X_test) (test — qo'lla; o'quv o'rtacha/std; sizish yo'q — 6.5). Sabab: sklearn — standart (ML quvuri — 20-qism; fit/transform); fit faqat o'quv (o'rtacha/std o'rgan; test — transform; ma'lumot sizishi oldini ol — 6.5); scaler.mean_/scaler.scale_ (o'rgangan o'rtacha/std). fit_transform (o'quv), transform (test — o'quv o'rtacha/std), sizish (test fit — xato). StandardScaler — fit_transform (o'quv), transform (test). Scaler. StandardScaler.

2.4. Qachon qaysi (normalizatsiya vs standartlashtirish)

Qachon qaysi (normalizatsiya vs standartlashtirish) — tanlov: Standartlashtirish (z-score) — normal taqsimot (4.6; o'rtacha/std ma'noli), outlier bor (chidamliroq), regressiya/SVM/neyron/PCA (markazlashtirish afzal; PCA — o'rtacha 0 shart); Normalizatsiya (Min-Max) — chegara kerak (0-1 — masalan rasm piksel 0-255→0-1; neyron kirish), taqsimot normal emas (chegaralangan), outlier yo'q. Sabab: taqsimot (normal — z-score; chegaralangan — Min-Max), outlier (bor — z-score chidamliroq; RobustScaler yanada), model (PCA/regressiya — z-score; rasm/neyron chegara — Min-Max). Amaliyotda z-score ko'proq (umumiy — ML). Standartlashtirish (normal, outlier, regressiya/PCA), normalizatsiya (chegara, rasm). Qachon qaysi — taqsimot/outlier/model (z-score umumiy). Tanlov. z-score.

2.5. RobustScaler (outlier)

RobustScaler (outlier) — median/IQR: from sklearn.preprocessing import RobustScaler; (x - median) / IQR (o'rtacha/std o'rniga median/IQR — 4.2, 4.3; outlierga eng chidamli). Sabab: outlier ko'p bo'lsa — z-score ham biroz sezgir (o'rtacha/std — outlier ta'sir); RobustScaler — median/IQR (outlierga chidamli — 4.2; markaz median, masshtab IQR; outlier ta'sir qilmaydi). Min-Max (outlier sezgir), StandardScaler (z-score — o'rtacha/std; biroz), RobustScaler (median/IQR — eng chidamli). Outlier ko'p → RobustScaler. RobustScaler — median/IQR (outlier — eng chidamli). Robust. median/IQR.

2.6. Standartlashtirish amaliyoti

Standartlashtirish amaliyoti: model tekshir (regressiya/SVM/neyron/PCA — z-score; daraxt — kerak emas 6.5); taqsimot (normal — z-score; chegaralangan — Min-Max); outlier (bor — z-score/RobustScaler; Min-Max sezgir); z-score ((x-mean)/std yoki StandardScaler); fit/transform (o'quv fit_transform, test transform — o'quv o'rtacha/std; sizish yo'q); son ustun (kategoriya emas); tekshir (o'rtacha ~0, std ~1). Tuzoqlar: test fit (sizish), kategoriya (son emas), taqsimot (chegara kerak — Min-Max), outlier ko'p (RobustScaler), z-score normal faraz (juda qiyshiq — transformatsiya oldin). Amaliyot — model, taqsimot, z-score, fit/transform. Standart. z-score.

2.7. Standartlashtirish tuzoqlari

Standartlashtirish asosiy tuzoqlari: test fit (fit faqat o'quv to'plamida — o'rtacha/std; test'ni fit → ma'lumot sizishi (test o'rtacha/std — o'quvga; noto'g'ri baholash; 6.5, 20-qism); test — transform); kategoriya standartlashtirish (son ustun; kategoriya (0/1 kodlangan) standartlashtirish odatda ma'nosiz — faqat son); taqsimot e'tiborsiz (z-score — normal faraz yaxshi (o'rtacha/std ma'noli); juda qiyshiq (o'ngga cho'zilgan — daromad; 5.5) — o'rtacha/std vakillik emas; avval log transformatsiya 6.5-bob yoki RobustScaler); outlier ko'p (z-score biroz sezgir (o'rtacha/std — outlier ta'sir); ko'p outlier → RobustScaler (median/IQR — chidamli)); normalizatsiya bilan aralash (Min-Max (0-1 chegara) vs z-score (chegarasiz) — maqsadga qarab; ikkalasini birga qo'llama — bitta usul); chegara kutish (z-score — chegarasiz (-∞..+∞; odatda -3..+3); "0-1" kutish (Min-Max) — z-score emas; maqsad (0-1 kerak — Min-Max)); target standartlashtirish (odatda X; target (y) — regressiyada ba'zan, keyin inverse_transform); std=0 ustun (bir xil qiymat — std=0 → bo'lish 0'ga (NaN/inf); o'zgarmas ustun — tashla yoki e'tibor). Sabab: standartlashtirish sizish/taqsimot/outlier nozik (test fit, qiyshiq, outlier — noto'g'ri). Yechim: o'quv fit, normal tekshir (log), outlier RobustScaler, son ustun. Tuzoqlar — test fit, taqsimot, outlier, kategoriya.

2.8. Standartlashtirish — z-score bilan masshtablash

Standartlashtirish asosiy g'oyasi — z-score bilan masshtablash: masshtablashning boshqa usuli (normalizatsiya 6.5 — Min-Max 0-1 yonida); z-score ((x - o'rtacha) / std — 4.6; o'rtacha 0, std 1; har qiymat "o'rtachadan necha std"; chegarasiz -3..+3). Normalizatsiyadan farq (Min-Max — 0-1 chegara, outlierga sezgir; z-score — chegarasiz, outlierga chidamliroq), sklearn (StandardScaler — fit_transform o'quv, transform test; sizish yo'q), qachon (normal taqsimot, outlier, regressiya/SVM/neyron/PCA — z-score; chegara/rasm — Min-Max; z-score umumiy), RobustScaler (median/IQR — outlier ko'p; eng chidamli). Foydalanish: markazlashtirish (o'rtacha 0 — PCA, regressiya), model uchun (outlierga chidamliroq — Min-Max'dan). Tuzoqlar: test fit (sizish), kategoriya (son emas), taqsimot qiyshiq (log oldin), outlier ko'p (RobustScaler), chegara kutish (Min-Max), std=0 (bo'lish 0). Bu 6.5 (normalizatsiya), 4.6 (z-score/normal), 4.2/4.3 (median/IQR — RobustScaler), 20-qism (ML) bilan. Standartlashtirish — z-score bilan masshtablash (o'rtacha 0, std 1; outlierga chidamliroq). Standart. z-score. Markaz.

3. Tez ma'lumotnoma

import numpy as np
from sklearn.preprocessing import StandardScaler, RobustScaler

# Z-SCORE (qo'lda):
z = (x - x.mean()) / x.std()   # o'rtacha 0, std 1 (chegarasiz, -3..+3)

# SKLEARN StandardScaler:
scaler = StandardScaler()
X_train_std = scaler.fit_transform(X_train)   # o'quv: fit + transform
X_test_std = scaler.transform(X_test)         # test: transform (o'quv mean/std)
scaler.mean_, scaler.scale_   # o'rgangan o'rtacha/std

# OUTLIER KO'P → RobustScaler (median/IQR):
robust = RobustScaler()
X_robust = robust.fit_transform(X_train)      # outlierga eng chidamli

# NORMALIZATSIYA vs STANDARTLASHTIRISH:
#   Min-Max (0-1; chegara; outlier sezgir; rasm/chegara)
#   z-score (o'rtacha 0; chegarasiz; chidamliroq; regressiya/SVM/PCA)
#   RobustScaler (median/IQR; outlier ko'p — eng chidamli)

QOIDA: model turi · o'quv fit · normal (qiyshiq→log) · outlier→Robust

Standartlashtirish xulosasi

Standartlashtirish — z-score bilan (o'rtacha 0, std 1)
z-score — (x-mean)/std (chegarasiz, -3..+3; "necha std")
Farqi — Min-Max (0-1, sezgir) vs z-score (chegarasiz, chidamliroq)
sklearn — StandardScaler (o'quv fit, test transform)
Outlier ko'p — RobustScaler (median/IQR — eng chidamli)

4. Batafsil misollar

Misollar real pandas/numpy/sklearn bilan (Python 3.14) ishlaydi.

Misol 1 — z-score standartlashtirish

"""z-score standartlashtirish (real pandas)."""

import pandas as pd


def main() -> None:
    daromad = pd.Series([300, 500, 700, 900, 1100])

    print("=== 1. Asl (o'rtacha, std) ===")
    print(f"  o'rtacha: {daromad.mean()}, std: {daromad.std().round(1)}")

    print("\n=== 2. z-score standartlashtirish ===")
    z = (daromad - daromad.mean()) / daromad.std()
    print(f"  {z.round(2).tolist()}")

    print("\n=== 3. Natija (o'rtacha ~0, std ~1) ===")
    print(f"  o'rtacha: {z.mean().round(2)}, std: {z.std().round(2)}")

    print("\n=== 4. Ma'no (necha std) ===")
    print("  har qiymat 'o'rtachadan necha std uzoq' 4.6-bob")
    print(f"  700 (o'rtacha) → {z[2].round(2)} (markaz)")
    print("  ⭐ z-score — o'rtacha 0, std 1")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

=== 1. Asl (o'rtacha, std) ===
  o'rtacha: 700.0, std: 316.2

=== 2. z-score standartlashtirish ===
  [-1.26, -0.63, 0.0, 0.63, 1.26]

=== 3. Natija (o'rtacha ~0, std ~1) ===
  o'rtacha: 0.0, std: 1.0

=== 4. Ma'no (necha std) ===
  har qiymat 'o'rtachadan necha std uzoq' 4.6-bob
  700 (o'rtacha) → 0.0 (markaz)
  ⭐ z-score — o'rtacha 0, std 1

Nima ko'rsatdi: 2.1-bo'lim.

Misol 2 — Normalizatsiya vs standartlashtirish (outlier)

"""Min-Max vs z-score (outlier) (real pandas)."""

import pandas as pd


def main() -> None:
    data = pd.Series([400, 450, 500, 550, 50000])   # outlier

    print("=== 1. Min-Max (outlier sezgir) ===")
    minmax = (data - data.min()) / (data.max() - data.min())
    print(f"  {minmax.round(4).tolist()}")
    print("  qolganlar 0'ga siqildi (outlier buzdi)")

    print("\n=== 2. z-score (chidamliroq) ===")
    z = (data - data.mean()) / data.std()
    print(f"  {z.round(2).tolist()}")
    print("  qolganlar siqilmadi (outlier katta z)")

    print("\n=== 3. Farq ===")
    print(f"  Min-Max normal qiymatlar farqi: {(minmax[3] - minmax[0]).round(4)}")
    print(f"  z-score normal qiymatlar farqi: {(z[3] - z[0]).round(2)}")

    print("\n=== 4. Xulosa ===")
    print("  z-score outlierga chidamliroq (markaz siqilmaydi)")
    print("  ⭐ Min-Max sezgir, z-score chidamliroq")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

=== 1. Min-Max (outlier sezgir) ===
  [0.0, 0.001, 0.002, 0.003, 1.0]
  qolganlar 0'ga siqildi (outlier buzdi)

=== 2. z-score (chidamliroq) ===
  [-0.45, -0.45, -0.45, -0.44, 1.79]
  qolganlar siqilmadi (outlier katta z)

=== 3. Farq ===
  Min-Max normal qiymatlar farqi: 0.003
  z-score normal qiymatlar farqi: 0.01

=== 4. Xulosa ===
  z-score outlierga chidamliroq (markaz siqilmaydi)
  ⭐ Min-Max sezgir, z-score chidamliroq

Nima ko'rsatdi: 2.2-bo'lim.

Misol 3 — StandardScaler (sklearn)

"""StandardScaler (real sklearn)."""

import numpy as np
from sklearn.preprocessing import StandardScaler


def main() -> None:
    X_train = np.array([[18, 300], [30, 500], [45, 700], [70, 1100]])
    X_test = np.array([[25, 400], [55, 900]])

    print("=== 1. StandardScaler (o'quv — fit_transform) ===")
    scaler = StandardScaler()
    X_train_std = scaler.fit_transform(X_train)
    print(f"  o'quv:\n{X_train_std.round(2)}")

    print("\n=== 2. O'rgangan o'rtacha/std ===")
    print(f"  o'rtacha: {scaler.mean_}")
    print(f"  std (scale): {scaler.scale_.round(1)}")

    print("\n=== 3. Test (transform — o'quv mean/std) ===")
    X_test_std = scaler.transform(X_test)
    print(f"  test:\n{X_test_std.round(2)}")

    print("\n=== 4. O'quv o'rtacha ~0 ===")
    print(f"  o'quv o'rtacha (ustun): {X_train_std.mean(axis=0).round(2)}")
    print("  ⭐ StandardScaler — o'quv fit, test transform")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

=== 1. StandardScaler (o'quv — fit_transform) ===
  o'quv:
[[-1.17 -1.18]
 [-0.55 -0.51]
 [ 0.22  0.17]
 [ 1.51  1.52]]

=== 2. O'rgangan o'rtacha/std ===
  o'rtacha: [ 40.75 650.  ]
  std (scale): [ 19.4 295.8]

=== 3. Test (transform — o'quv mean/std) ===
  test:
[[-0.81 -0.85]
 [ 0.73  0.85]]

=== 4. O'quv o'rtacha ~0 ===
  o'quv o'rtacha (ustun): [0. 0.]
  ⭐ StandardScaler — o'quv fit, test transform

Nima ko'rsatdi: 2.3-bo'lim.

Misol 4 — RobustScaler (outlier)

"""RobustScaler: outlier (real sklearn/numpy)."""

import numpy as np
from sklearn.preprocessing import StandardScaler, RobustScaler


def main() -> None:
    X = np.array([[400], [450], [500], [550], [50000]])   # outlier

    print("=== 1. StandardScaler (o'rtacha/std) ===")
    std_scaled = StandardScaler().fit_transform(X)
    print(f"  {std_scaled.round(2).flatten().tolist()}")

    print("\n=== 2. RobustScaler (median/IQR) ===")
    robust_scaled = RobustScaler().fit_transform(X)
    print(f"  {robust_scaled.round(2).flatten().tolist()}")

    print("\n=== 3. Farq (normal qiymatlar) ===")
    print("  RobustScaler — median/IQR (outlier ta'sir qilmaydi)")
    print("  normal qiymatlar yaxshi ajratilgan")

    print("\n=== 4. Qachon RobustScaler ===")
    print("  outlier ko'p → RobustScaler (eng chidamli)")
    print("  ⭐ RobustScaler — median/IQR (outlier)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

=== 1. StandardScaler (o'rtacha/std) ===
  [-0.5, -0.5, -0.5, -0.5, 2.0]

=== 2. RobustScaler (median/IQR) ===
  [-1.0, -0.5, 0.0, 0.5, 495.0]

=== 3. Farq (normal qiymatlar) ===
  RobustScaler — median/IQR (outlier ta'sir qilmaydi)
  normal qiymatlar yaxshi ajratilgan

=== 4. Qachon RobustScaler ===
  outlier ko'p → RobustScaler (eng chidamli)
  ⭐ RobustScaler — median/IQR (outlier)

Nima ko'rsatdi: 2.5-bo'lim.

5. To'g'ri va noto'g'ri tushunishlar

6. Keng tarqalgan xatolar va yechimlari

1. Test'ni fit

scaler.fit_transform(X_test)   # ma'lumot sizishi                # ⚠️
scaler.transform(X_test)   # o'quv mean/std                      # ✅

2. Outlier ko'p (StandardScaler)

StandardScaler().fit_transform(X)   # o'rtacha/std biroz sezgir  # ⚠️
RobustScaler().fit_transform(X)   # median/IQR (chidamli)        # ✅

3. Qiyshiq taqsimot

scaler.fit_transform(daromad)   # o'ngga cho'zilgan (vakillik emas) # ⚠️
np.log1p(daromad)   # avval log 6.5-bob, keyin standartlash        # ✅

4. Kategoriya

scaler.fit_transform(df[["shahar_kod"]])   # kategoriya           # ⚠️
scaler.fit_transform(df[["yosh", "daromad"]])   # son            # ✅

5. Chegara kutish

# z-score → 0-1 kutish (chegarasiz — noto'g'ri)                   # ⚠️
MinMaxScaler()   # 0-1 kerak bo'lsa                              # ✅

6. std=0 ustun

(x - x.mean()) / x.std()   # bir xil qiymat → std=0 (NaN)        # ⚠️
# o'zgarmas ustun — tashla (ma'lumot yo'q)                       # ✅

7. Target standartlashtirish (inverse unut)

# target standartlashtirildi, bashorat z-score (asl emas)        # ⚠️
scaler.inverse_transform(bashorat)   # asl birlik                # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

• 6.5-dars (o'tilgan): Normalizatsiya (Min-Max)
• 4.6-dars (o'tilgan): z-score (normal)
• 4.2/4.3-dars (o'tilgan): Median/IQR (RobustScaler)
• 6.11-dars: Tozalash quvuri
• 20-qism (reja): ML (StandardScaler — PCA, regressiya)

8. Eng yaxshi amaliyotlar

1. z-score — (x-mean)/std (o'rtacha 0, std 1).

2. Model — regressiya/SVM/PCA (z-score).

3. O'quv fit, test transform (sizish yo'q).

4. Outlier ko'p — RobustScaler.

5. Qiyshiq — log oldin 6.5-bob.

6. Son ustun — kategoriya emas.

7. Chegara kerak — Min-Max (z-score emas).

8. std=0 — o'zgarmas ustun tashla.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

1.  # standartlashtirish nima?
2.  # z-score formulasi?
3.  # natija (o'rtacha, std)?
4.  # normalizatsiyadan farqi?
5.  # z-score chegarasi?
6.  # outlier ta'siri?
7.  # StandardScaler?
8.  # test fit mi?
9.  # RobustScaler qachon?
10. # qiyshiq taqsimot?
11. # qachon z-score?
12. # nega standartlashtirish?

Vazifa 2: Xatolarni tuzating

1.  scaler.fit_transform(X_test)

2.  StandardScaler().fit_transform(X)   # outlier ko'p

3.  scaler.fit_transform(daromad)   # qiyshiq

4.  scaler.fit_transform(df[["shahar_kod"]])

5.  (x-x.mean())/x.std()   # bir xil qiymat

1.  scaler.transform(X_test)

2.  RobustScaler (median/IQR)

3.  np.log1p avval (6.5)

4.  son ustun (yosh, daromad)

5.  o'zgarmas ustun tashla

Vazifa 3: z-score

Modellang:

1. Formula
2. O'rtacha 0
3. Std 1
4. Chegarasiz

Vazifa 4: Farqi

Modellang:

1. Min-Max (0-1)
2. z-score (chegarasiz)
3. Outlier
4. Taqsimot

Vazifa 5: sklearn va outlier

Modellang:

1. StandardScaler
2. fit/transform
3. RobustScaler
4. median/IQR

Vazifa 6: Integratsiya

Modellang:

1. Normalizatsiya (6.5)
2. z-score (4.6)
3. Quvur (6.11)
4. ML (20)

Vazifa 7: O'ylash

Normalizatsiya (Min-Max) va standartlashtirish (z-score) ikkalasi ham masshtablash — lekin turli xususiyatga ega: Min-Max chegara beradi (0-1) lekin outlierga sezgir; z-score chegarasiz lekin chidamliroq. RobustScaler yanada chidamli. Nima uchun bu uch usul mavjudligi "eng yaxshi usul yo'q, faqat kontekstga mos usul bor" tamoyilini ko'rsatadi, va Data Scientist masshtablash usulini qanday tanlaydi?

Nimani mustahkamlaydi: 2.4, 2.5-bo'limlar.

Xulosa

Bu darsda standartlashtirishni o'rgandik.

Eng muhim uch fikr:

1. z-score va farqi. Standartlashtirish — (x - o'rtacha) / std (z-score — 4.6; o'rtacha 0, std 1; "necha std"; chegarasiz -3..+3). Normalizatsiyadan farq — Min-Max ((x-min)/(max-min); 0-1 chegara; outlierga sezgir) vs z-score (chegarasiz; outlierga chidamliroq; o'rtacha/std).

2. sklearn va RobustScaler. StandardScaler — fit_transform(X_train) (o'quv), transform(X_test) (test — o'quv o'rtacha/std; test'ni fit qilma — sizish). RobustScaler — (x-median)/IQR (median/IQR — 4.2/4.3; outlier ko'p bo'lsa; eng chidamli). Qachon qaysi: normal/outlier/regressiya/PCA — z-score (umumiy); chegara/rasm — Min-Max; outlier ko'p — RobustScaler.

3. z-score bilan masshtablash. Standartlashtirish — masshtablashning z-score usuli (Min-Max yonida; markazlashtirish — o'rtacha 0; outlierga chidamliroq). Eng yaxshi usul yo'q — kontekstga mos (ma'lumot/model/maqsad; universal yo'q — almashuv). Tuzoqlar: test fit (sizish), kategoriya (son emas), taqsimot qiyshiq (o'ngga cho'zilgan — log oldin 6.5), outlier ko'p (RobustScaler), chegara kutish (Min-Max), std=0 (o'zgarmas ustun — tashla), target (inverse).

Keyingi darsda matn tozalash (string cleaning)ni o'rganamiz: iflos matn (bo'shliq, katta-kichik harf, maxsus belgi, xato yozuv) — tozalash (str.strip, str.lower, str.replace, regex); matn ustunlari dublikat/guruh uchun normallashtirish (6.2 — "Ali" vs "ali ").