Python kursi25.3-dars: Ma'lumotni tayyorlash
Python kursi · ML va AI

25.3-dars: Ma'lumotni tayyorlash

16 daqiqa3/12-dars

25-QISM — ML VA AI · 3-dars


1. Kirish va motivatsiya

25.2 da model qurdik — lekin toza, tayyor ma'lumot (iris) bilan. Real ma'lumot tayyor emas: kategoriya matn ("kurs", "kitob" — model son kutadi), turli masshtab (maydon 50–200, narx 1000–500000 — katta son modelni "tortadi"), bo'sh qiymatlar. Model son va birxil masshtab kutadi. Ma'lumotni modelga tayyorlash — ML ning muhim (va ko'p vaqt oladigan) qismi.

Ma'lumot tayyorlash (preprocessing) — xom ma'lumotni modelga mos shaklga keltirish: kodlash (kategoriya → son — LabelEncoder, OneHotEncoder), masshtablash (turli oraliqni birxil — StandardScaler, MinMaxScaler), tozalash (bo'sh, chetlangan — 24.8), feature tanlovi. "Garbage in, garbage out" 24.6-bob — iflos ma'lumot → yomon model. Pipeline — bu qadamlarni model bilan birlashtirish. Ma'lumot sifati model sifatini belgilaydi.

Real vaziyat. Bir model yomon ishlardi (60%). Sabab: xususiyatlar turli masshtabda edi (yosh 20–60, daromad 1000–100000) — model daromadni "muhimroq" deb qabul qildi (katta son). StandardScaler qo'llandi (barcha birxil masshtab) — aniqlik 85%'ga oshdi. Masshtablash — model uchun muhim. Ma'lumot tayyorlash ko'pincha modeldan muhimroq.

Bu darsda ML uchun ma'lumot tayyorlashni o'rganamiz.

Bu darsda:

  • Nega tayyorlash kerak
  • Kategoriya kodlash (LabelEncoder, OneHot)
  • Masshtablash (StandardScaler)
  • MinMax masshtablash
  • Bo'sh qiymatlar (ML uchun)
  • Pipeline (qadamlarni birlashtirish)
  • Train/test to'g'ri tayyorlash
  • Amaliy: to'liq tayyorlash

ℹ Misollarda scikit-learn bilan sinaladi.


2. Nazariya — chuqur tushuntirish

2.1. Nega tayyorlash kerak

Model son va birxil masshtab kutadi:

Xom ma'lumot:
  kategoriya: "kurs", "kitob"  (matn — model tushunmaydi)
  masshtab: yosh 20-60, daromad 1000-100000  (turli)

Tayyorlangan:
  kategoriya → 0, 1 (son)
  masshtab → birxil (StandardScaler)

Model son kutadi (matn emas) va birxil masshtab yaxshi ishlaydi (turli oraliq — katta son modelni "tortadi"). Xom ma'lumot: kategoriya (matn), turli masshtab, bo'sh. Tayyorlash: kodlash (matn → son), masshtablash (birxil), tozalash (bo'sh). Bu ML ning muhim qismi — ko'pincha modeldan ko'proq ta'sir. "Ma'lumot sifati — model sifati".

2.2. Kategoriya kodlash

Matn kategoriyani songa:

python
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
# LabelEncoder: har kategoriya → son (0, 1, 2)
le = LabelEncoder()
le.fit_transform(["kurs", "kitob", "kurs"])   # [1, 0, 1]

# OneHotEncoder: har kategoriya → alohida ustun (0/1)
# "A" → [1,0,0], "B" → [0,1,0]

Kategoriya kodlash: LabelEncoder (har kategoriya → son: "kurs"→1 — tartibli kategoriyaga, masalan baho: past/o'rta/yuqori), OneHotEncoder (har kategoriya → alohida ustun 0/1 — tartibsiz kategoriyaga: rang, shahar — model tartibni noto'g'ri tushunmasin). Muhim: tartibsiz kategoriyada LabelEncoder xato (model 2 > 1 deb o'ylaydi) — OneHot to'g'ri. Kategoriya turiga qarab.

2.3. Masshtablash (StandardScaler)

Turli oraliqni birxil:

python
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_scaled = sc.fit_transform(X)
# har ustun: o'rtacha 0, std 1 (birxil masshtab)

StandardScaler — har xususiyatni standartlash: o'rtacha 0, standart chetlanish 1 (birxil masshtab). Sabab: turli masshtab (yosh 20–60, daromad 1000–100000) — model katta sonli xususiyatni "muhimroq" deb qabul qiladi (masofaga asoslangan model, chiziqli). Standartlashdan keyin barcha birxil (adolatli). Bu masofaga asoslangan (KNN), chiziqli (regressiya) modellar uchun muhim. Daraxt modellarga (RandomForest) kam ta'sir.

2.4. MinMax masshtablash

0–1 oraliqga:

python
from sklearn.preprocessing import MinMaxScaler
mm = MinMaxScaler()
mm.fit_transform(X)     # har ustun 0-1 oraliqda
# min → 0, max → 1

MinMaxScaler — har xususiyatni 0–1 oraliqga (min→0, max→1). StandardScaler'dan farq: qat'iy oraliq (0–1), lekin chetlangan qiymatga sezgir. Qachan: neyron tarmoq (0–1 kirish), aniq oraliq kerak. StandardScaler — umumiy (normal taqsimot), MinMax — chegaralangan oraliq. Ikkalasi masshtablash — tanlov ma'lumot va modelga qarab. Birxil masshtab — asosiy maqsad.

2.5. Bo'sh qiymatlar (ML uchun)

Model bo'sh qiymat bilan ishlamaydi:

python
from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy="mean")   # o'rtacha bilan to'ldir
X_filled = imp.fit_transform(X)
# yoki pandas fillna (24.8)

Bo'sh qiymatlar — ko'p model bo'sh (NaN) bilan ishlamaydi: SimpleImputer(strategy="mean") (o'rtacha bilan to'ldir), "median", "most_frequent" (kategoriya). Yoki pandas fillna 24.8-bob. Muhim: train'da o'rgan (o'rtachani train'dan), test'ga qo'lla (test o'rtachasini ishlatma — ma'lumot sizishi). Bo'sh qiymatlar — tozalashning 24.8-bob ML versiyasi. Model tayyorlashda birinchi qadam.

2.6. Pipeline (qadamlarni birlashtirish)

Tayyorlash + model bir joyda:

python
from sklearn.pipeline import Pipeline
pipe = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression()),
])
pipe.fit(X_train, y_train)     # tayyorlash + o'qitish
pipe.predict(X_test)           # tayyorlash + bashorat (avtomatik)

Pipeline — tayyorlash qadamlari + modelni birlashtirish: Pipeline([("scale", StandardScaler()), ("model", ...)]). fit — barcha qadam (scale → model), predict — avtomatik (test'ni ham scale). Afzallik: ma'lumot sizishi yo'q (scale train'dan o'rgan, test'ga qo'lla — avtomatik to'g'ri), toza kod (bir obyekt). Bu ML ning eng yaxshi amaliyoti. Tayyorlash va model — bir butun.

2.7. Train/test to'g'ri tayyorlash

Muhim: train'dan o'rgan, test'ga qo'lla:

python
sc = StandardScaler()
X_train_s = sc.fit_transform(X_train)   # train'dan o'rgan + qo'lla
X_test_s = sc.transform(X_test)          # test'ga qo'lla (fit EMAS)
# fit_transform(X_test) — XATO (ma'lumot sizishi)

To'g'ri tayyorlash: fit_transform faqat train'da (o'rtacha, std — train'dan), transform test'da (train sozlamalarini qo'lla — fit emas!). Sabab: fit_transform(X_test) — test ma'lumotini "ko'rish" (ma'lumot sizishi — data leakage — soxta yuqori aniqlik). Pipeline buni avtomatik to'g'ri qiladi. Bu ML ning nozik, lekin muhim qoidasi. Test — faqat baholash 25.2-bob, tayyorlashda ham.

2.8. Ma'lumot sifati — model sifati

Ma'lumot tayyorlash — ML ning eng muhim (va ko'p vaqt) qismi: ko'pincha yaxshi tayyorlash + oddiy model > yomon tayyorlash + murakkab model. "Garbage in, garbage out" 24.6-bob — iflos ma'lumot hech qanday model tuzatolmaydi. Tayyorlash: tozalash (bo'sh, xato), kodlash (kategoriya), masshtablash (birxil), feature yaratish 25.5-bob. Muhim qoida: train'dan o'rgan, test'ga qo'lla (ma'lumot sizishi yo'q). "Ma'lumot sifati model sifatini belgilaydi" — ML ning asosiy haqiqati.


3. Tez ma'lumotnoma

python
from sklearn.preprocessing import StandardScaler, MinMaxScaler, LabelEncoder, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

# kategoriya kodlash:
LabelEncoder().fit_transform(["a", "b", "a"])   # tartibli
OneHotEncoder(sparse_output=False).fit_transform(X)  # tartibsiz

# masshtablash:
StandardScaler().fit_transform(X)   # o'rtacha 0, std 1
MinMaxScaler().fit_transform(X)     # 0-1

# bo'sh qiymat:
SimpleImputer(strategy="mean").fit_transform(X)

# to'g'ri train/test:
sc.fit_transform(X_train)   # train'dan o'rgan
sc.transform(X_test)        # test'ga qo'lla (fit EMAS)

# pipeline (birlashtirish):
pipe = Pipeline([("scale", StandardScaler()), ("model", ...)])
pipe.fit(X_train, y_train)

Tayyorlash xulosasi

Model son + birxil masshtab kutadi · kodlash: Label(tartibli)/OneHot(tartibsiz)
Masshtab: StandardScaler/MinMax · bo'sh: SimpleImputer
Train'dan o'rgan, test'ga qo'lla (sizish yo'q) · Pipeline: birlashtir

4. Batafsil misollar

Misollarda scikit-learn bilan sinaladi.

Misol 1 — Kategoriya kodlash

python
"""LabelEncoder (tartibli → son); OneHotEncoder (tartibsiz → alohida ustun)."""

import warnings
warnings.filterwarnings("ignore")

from sklearn.preprocessing import LabelEncoder, OneHotEncoder


def main() -> None:
    print("=== 1. LabelEncoder (kategoriya → son) ===")
    le = LabelEncoder()
    kategoriyalar = ["kurs", "kitob", "kurs", "video", "kitob"]
    enc = le.fit_transform(kategoriyalar)
    print(f"  {kategoriyalar}")
    print(f"  → {enc.tolist()}")
    print(f"  sinflar: {[str(c) for c in le.classes_]}")

    print("\n=== 2. OneHotEncoder (alohida ustun) ===")
    oh = OneHotEncoder(sparse_output=False)
    X = [["A"], ["B"], ["A"], ["C"]]
    ohe = oh.fit_transform(X)
    print(f"  A/B/C → ustunlar:")
    for orig, row in zip(X, ohe):
        print(f"    {orig[0]} → {[int(v) for v in row]}")

    print("\n=== 3. Qachon qaysi ===")
    print("  tartibli (past/o'rta/yuqori) → LabelEncoder")
    print("  tartibsiz (rang, shahar) → OneHot")

    print("\n=== 4. Nega OneHot tartibsizga ===")
    print("  LabelEncoder: model 2 > 1 deb o'ylaydi (xato)")
    print("  OneHot: tartib yo'q (to'g'ri)")
    print("  ⭐ kategoriya kodlash — tur (tartibli/tartibsiz) bo'yicha")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. LabelEncoder (kategoriya → son) ===
  ['kurs', 'kitob', 'kurs', 'video', 'kitob']
  → [1, 0, 1, 2, 0]
  sinflar: ['kitob', 'kurs', 'video']

=== 2. OneHotEncoder (alohida ustun) ===
  A/B/C → ustunlar:
    A → [1, 0, 0]
    B → [0, 1, 0]
    A → [1, 0, 0]
    C → [0, 0, 1]

=== 3. Qachon qaysi ===
  tartibli (past/o'rta/yuqori) → LabelEncoder
  tartibsiz (rang, shahar) → OneHot

=== 4. Nega OneHot tartibsizga ===
  LabelEncoder: model 2 > 1 deb o'ylaydi (xato)
  OneHot: tartib yo'q (to'g'ri)
  ⭐ kategoriya kodlash — tur (tartibli/tartibsiz) bo'yicha

Nima ko'rsatdi: 2.2-bo'lim.

Misol 2 — Masshtablash

python
"""StandardScaler (o'rtacha 0, std 1); MinMaxScaler (0-1); turli masshtabni birxil."""

import warnings
warnings.filterwarnings("ignore")

import numpy as np
from sklearn.preprocessing import MinMaxScaler, StandardScaler


def main() -> None:
    # turli masshtab: yosh (20-50), daromad (1000-4000)
    X = np.array([[20, 1000], [30, 2000], [40, 3000], [50, 4000]])

    print("=== 1. Xom ma'lumot (turli masshtab) ===")
    print(f"  yosh: {X[:, 0].tolist()}, daromad: {X[:, 1].tolist()}")

    print("\n=== 2. StandardScaler (o'rtacha 0, std 1) ===")
    sc = StandardScaler()
    Xs = sc.fit_transform(X)
    print(f"  o'rtacha: {np.round(Xs.mean(axis=0), 1).tolist()}")
    print(f"  std: {np.round(Xs.std(axis=0), 1).tolist()}")

    print("\n=== 3. MinMaxScaler (0-1) ===")
    mm = MinMaxScaler()
    Xm = mm.fit_transform(X)
    print(f"  {np.round(Xm, 2).tolist()}")

    print("\n=== 4. Nega masshtablash ===")
    print("  turli masshtab → model kattani muhimroq deb qabul")
    print("  birxil → adolatli (barcha teng)")
    print("  ⭐ masshtablash — birxil oraliq (chiziqli, KNN uchun)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Xom ma'lumot (turli masshtab) ===
  yosh: [20, 30, 40, 50], daromad: [1000, 2000, 3000, 4000]

=== 2. StandardScaler (o'rtacha 0, std 1) ===
  o'rtacha: [0.0, 0.0]
  std: [1.0, 1.0]

=== 3. MinMaxScaler (0-1) ===
  [[0.0, 0.0], [0.33, 0.33], [0.67, 0.67], [1.0, 1.0]]

=== 4. Nega masshtablash ===
  turli masshtab → model kattani muhimroq deb qabul
  birxil → adolatli (barcha teng)
  ⭐ masshtablash — birxil oraliq (chiziqli, KNN uchun)

Nima ko'rsatdi: 2.3, 2.4-bo'limlar.

Misol 3 — Train/test to'g'ri tayyorlash

python
"""fit_transform (train'dan o'rgan); transform (test'ga qo'lla); ma'lumot sizishi."""

import warnings
warnings.filterwarnings("ignore")

import numpy as np
from sklearn.preprocessing import StandardScaler


def main() -> None:
    X_train = np.array([[10.0], [20.0], [30.0], [40.0]])
    X_test = np.array([[25.0], [35.0]])

    print("=== 1. fit_transform (TRAIN'da) ===")
    sc = StandardScaler()
    X_train_s = sc.fit_transform(X_train)
    print(f"  train o'rgatildi: o'rtacha={round(float(sc.mean_[0]), 1)}")
    print(f"  train scaled: {np.round(X_train_s.flatten(), 2).tolist()}")

    print("\n=== 2. transform (TEST'ga — fit EMAS) ===")
    X_test_s = sc.transform(X_test)
    print(f"  test scaled (train sozlamasi bilan): {np.round(X_test_s.flatten(), 2).tolist()}")

    print("\n=== 3. Nega fit_transform(test) XATO ===")
    print("  test'ni fit qilish → test ma'lumotini 'ko'rish'")
    print("  bu ma'lumot sizishi (data leakage) — soxta aniqlik")

    print("\n=== 4. To'g'ri qoida ===")
    print("  train: fit_transform (o'rgan + qo'lla)")
    print("  test: transform (faqat qo'lla)")
    print("  ⭐ train'dan o'rgan, test'ga qo'lla (sizish yo'q)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. fit_transform (TRAIN'da) ===
  train o'rgatildi: o'rtacha=25.0
  train scaled: [-1.34, -0.45, 0.45, 1.34]

=== 2. transform (TEST'ga — fit EMAS) ===
  test scaled (train sozlamasi bilan): [0.0, 0.89]

=== 3. Nega fit_transform(test) XATO ===
  test'ni fit qilish → test ma'lumotini 'ko'rish'
  bu ma'lumot sizishi (data leakage) — soxta aniqlik

=== 4. To'g'ri qoida ===
  train: fit_transform (o'rgan + qo'lla)
  test: transform (faqat qo'lla)
  ⭐ train'dan o'rgan, test'ga qo'lla (sizish yo'q)

Nima ko'rsatdi: 2.5, 2.7-bo'limlar.

Misol 4 — Amaliy: Pipeline bilan to'liq tayyorlash

Real ML: Pipeline bilan tayyorlash + model — bir butun, ma'lumot sizishisiz. Bu — to'g'ri ML ish oqimining namunasi.

python
"""to'liq Pipeline: StandardScaler + model, train/test — sizishsiz tayyorlash + baholash."""

import warnings
warnings.filterwarnings("ignore")

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    X, y = load_iris(return_X_y=True)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

    print("=== 1. Pipeline yaratish (scale + model) ===")
    pipe = Pipeline([
        ("scale", StandardScaler()),
        ("model", LogisticRegression(max_iter=1000, random_state=42)),
    ])
    print("  qadamlar: scale → model")

    print("\n=== 2. fit (tayyorlash + o'qitish) ===")
    pipe.fit(X_train, y_train)
    print("  scale train'dan o'rgandi, model o'qitildi")

    print("\n=== 3. Baholash (avtomatik scale) ===")
    acc = pipe.score(X_test, y_test)
    print(f"  test aniqlik: {round(acc, 3)}")

    print("\n=== 4. Bashorat (avtomatik tayyorlash) ===")
    yangi = [[5.1, 3.5, 1.4, 0.2]]
    print(f"  bashorat: sinf {int(pipe.predict(yangi)[0])}")
    print("  test avtomatik scale qilindi (sizish yo'q)")
    print("  ⭐ Pipeline — tayyorlash + model bir butun (sizishsiz)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Pipeline yaratish (scale + model) ===
  qadamlar: scale → model

=== 2. fit (tayyorlash + o'qitish) ===
  scale train'dan o'rgandi, model o'qitildi

=== 3. Baholash (avtomatik scale) ===
  test aniqlik: 1.0

=== 4. Bashorat (avtomatik tayyorlash) ===
  bashorat: sinf 0
  test avtomatik scale qilindi (sizish yo'q)
  ⭐ Pipeline — tayyorlash + model bir butun (sizishsiz)

Nima ko'rsatdi: 2.1–2.8-bo'limlar.


5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Model matn bilan ishlaydi" Son kerak (kodlash)
"Masshtab muhim emas" Chiziqli/KNN uchun muhim
"Tartibsizga LabelEncoder" OneHot (tartib xato)
"test'ni fit_transform" transform (sizish)
"Tayyorlash — ozroq ish" ML ning katta qismi
"Murakkab model tuzatadi" Ma'lumot sifati muhimroq
"Pipeline keraksiz" Sizishni oldini oladi
"Daraxtga masshtab kerak" Kam ta'sir (chiziqliga muhim)

6. Keng tarqalgan xatolar va yechimlari

1. test'ni fit_transform

python
sc.fit_transform(X_test)   # ma'lumot sizishi          # ⚠️
sc.transform(X_test)        # faqat qo'lla               # ✅

2. Tartibsiz kategoriyaga LabelEncoder

python
LabelEncoder().fit(["qizil", "ko'k"])   # 0,1 (tartib xato)  # ⚠️
OneHotEncoder().fit(...)                 # alohida ustun       # ✅

3. Masshtablashni unutish (chiziqli model)

python
LogisticRegression().fit(turli_masshtab, y)   # ⚠️ yomon
# StandardScaler + model (Pipeline)             # ✅

4. Bo'sh qiymat bilan o'qitish

python
model.fit(bo_sh_X, y)   # xato (NaN)                     # ⚠️
SimpleImputer().fit_transform(X)   # to'ldir             # ✅

5. Pipeline'siz qo'lda (sizish xavfi)

python
# scale, model alohida (tartib xato mumkin)              # ⚠️
Pipeline([...])   # avtomatik to'g'ri                     # ✅

6. Kodlash test'da mos emas

python
# train'da "A,B", test'da "C" (yangi)                    # ⚠️
# handle_unknown="ignore" yoki to'liq kategoriya          # ✅

7. Barcha ustunni masshtablash (kategoriya ham)

python
StandardScaler().fit(kategoriya_ustun)   # ⚠️ kodlangan
# faqat son ustunlarni (ColumnTransformer)                # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 24.8-dars (o'tilgan): Tozalash — tayyorlash asosi
  • 25.2-dars (o'tilgan): Model — tayyorlangan ma'lumot
  • 25.4-dars: Model baholash
  • 25.5-dars: Feature engineering — feature yaratish
  • 25.8-dars: PyTorch — masshtablash (0-1)

8. Eng yaxshi amaliyotlar

  1. Kategoriya → son (Label/OneHot).

  2. Tartibsiz → OneHot (Label xato).

  3. Masshtabla (chiziqli, KNN).

  4. train'dan o'rgan, test'ga qo'lla.

  5. Pipeline (sizishni oldini oladi).

  6. Bo'sh qiymat — to'ldir (imputer/fillna).

  7. Faqat son ustunni masshtabla.

  8. Ma'lumot sifatiga e'tibor (model sifati).


9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # nega tayyorlash kerak?
2.  # LabelEncoder nima?
3.  # OneHotEncoder nima?
4.  # qachon OneHot?
5.  # StandardScaler nima?
6.  # MinMaxScaler nima?
7.  # nega masshtablash?
8.  # fit_transform vs transform?
9.  # ma'lumot sizishi nima?
10. # Pipeline nima?
11. # bo'sh qiymat ML'da?
12. # ma'lumot sifati muhimmi?
Javoblar
  1. Model son + birxil masshtab kutadi
  2. Kategoriya → son (tartibli)
  3. Kategoriya → alohida ustun (tartibsiz)
  4. Tartibsiz kategoriyada (rang, shahar)
  5. O'rtacha 0, std 1 (masshtab)
  6. 0-1 oraliq (masshtab)
  7. Turli masshtab modelni tortadi
  8. fit_transform o'rgan+qo'lla, transform qo'lla
  9. Test ma'lumotini ko'rish (soxta aniqlik)
  10. Tayyorlash + model birlashtirish
  11. To'ldir (imputer/fillna)
  12. Ha (model sifatini belgilaydi)

Vazifa 2: Xatolarni tuzating

python
1.  sc.fit_transform(X_test)               # transform

2.  LabelEncoder().fit(["qizil","ko'k"])   # OneHot

3.  LogisticRegression().fit(turli_mas, y)  # scale

4.  model.fit(bo_sh_X, y)                   # imputer

5.  # scale, model alohida                  # Pipeline
Javoblar
python
1.  sc.transform(X_test)

2.  OneHotEncoder().fit(...)

3.  Pipeline([("scale", StandardScaler()), ...])

4.  SimpleImputer().fit_transform(X)

5.  Pipeline([...])

Vazifa 3: Kodlash

Kategoriya:

  1. LabelEncoder
  2. OneHotEncoder
  3. Qachon qaysi
  4. Tartib xatosi

Vazifa 4: Masshtablash

Birxil:

  1. StandardScaler
  2. MinMaxScaler
  3. Nega kerak
  4. Qaysi model uchun

Vazifa 5: Train/test

To'g'ri:

  1. fit_transform (train)
  2. transform (test)
  3. Sizish
  4. Qoida

Vazifa 6: Pipeline

Birlashtirish:

  1. Pipeline
  2. fit
  3. Bahola
  4. Sizishsiz

Vazifa 7: O'ylash

Ma'lumot tayyorlash — ML ning eng muhim qismi: ko'pincha yaxshi tayyorlash + oddiy model > yomon tayyorlash + murakkab model. Muhim qoida — train'dan o'rgan, test'ga qo'lla (ma'lumot sizishi yo'q). Nima uchun "ma'lumot sifati model sifatini belgilaydi", va nega "ma'lumot sizishi" (data leakage — test ma'lumotini ko'rish) xavfli — bu 25.2 dagi train/test bo'lish tamoyilining davomi qanday?

Javob

Qisqa javob: "Ma'lumot sifati model sifatini belgilaydi", chunki model ma'lumotdan o'rganadi — iflos, noto'g'ri tayyorlangan ma'lumot → yomon model ("garbage in, garbage out", 24.6). Yaxshi tayyorlash (kodlash, masshtablash, tozalash) ko'pincha modeldan muhimroq. "Ma'lumot sizishi" (data leakage) — model test ma'lumotini (kelajakni) "ko'rsa" (masalan, fit_transform(X_test) — test o'rtachasini ishlatsa) — soxta yuqori aniqlik (test'ni bilgani uchun), lekin real ishda yomon (haqiqiy yangi ma'lumotda). Bu 25.2 (train/test bo'lish) davomi: test ko'rmagan bo'lishi kerak — nafaqat model o'qitishda 25.2-bob, balki tayyorlashda ham (scale, imputer train'dan o'rgan). Sizish — test'ni "ko'rish"ning yashirin shakli. Qoida: hamma narsa train'dan o'rganiladi, test'ga qo'llanadi. Pipeline buni avtomatik to'g'ri qiladi.

1. Ma'lumot sifati — model sifati

Model ma'lumotdan o'rganadi. Iflos/noto'g'ri tayyorlangan → yomon model. Yaxshi tayyorlash + oddiy model > yomon tayyorlash + murakkab.

2. Ma'lumot sizishi (data leakage)

Model test (kelajak) ma'lumotini "ko'rsa":

  • fit_transform(X_test) — test o'rtachasini ishlatadi
  • Soxta yuqori aniqlik (test'ni bilgan)
  • Real ishda yomon (yangi ma'lumot)

3. 25.2 davomi

25.2: test — model ko'rmagan (halol baholash). Bu tayyorlashda ham: scale, imputer train'dan o'rgan (test'ni emas). Sizish — test'ni ko'rishning yashirin shakli (o'rtacha orqali).

4. To'g'ri qoida

Amal Train Test
fit_transform Ha Yo'q
transform — Ha

Hamma narsa train'dan o'rganiladi. Pipeline avtomatik to'g'ri.

5. Muhandislik saboqlari

  1. Ma'lumot sifati — model sifati
  2. Sizish — test'ni ko'rish (soxta aniqlik)
  3. Train'dan o'rgan, test'ga qo'lla
  4. Pipeline — sizishni oldini oladi

6. Xulosa

  1. Ma'lumot sifati model sifatini belgilaydi
  2. Sizish — soxta aniqlik (real yomon)
  3. 25.2 davomi (test ko'rmagan)
  4. Train'dan o'rgan (tayyorlashda ham)

Nimani mustahkamlaydi: 2.1–2.8-bo'limlar.


Xulosa

Bu darsda ML uchun ma'lumot tayyorlashni o'rgandik.

Eng muhim uch fikr:

  1. Kategoriya kodlash va masshtablash. Model son kutadi (matn emas) va birxil masshtab yaxshi ishlaydi. Kodlash: LabelEncoder (kategoriya → son — tartibli kategoriyaga: past/o'rta/yuqori), OneHotEncoder (kategoriya → alohida ustun 0/1 — tartibsiz: rang, shahar — model tartibni noto'g'ri tushunmasin). Masshtablash: StandardScaler (o'rtacha 0, std 1 — turli masshtab modelni "tortadi"), MinMaxScaler (0–1). Masofaga asoslangan (KNN), chiziqli modellar uchun muhim (daraxtga kam ta'sir).

  2. Bo'sh qiymatlar va Pipeline. Bo'sh qiymatlar — ko'p model NaN bilan ishlamaydi: SimpleImputer(strategy="mean") (yoki pandas fillna, 24.8). Pipeline — tayyorlash qadamlari + model birlashtirish (Pipeline([("scale", ...), ("model", ...)])): fit barcha qadam, predict avtomatik. Afzallik: ma'lumot sizishi yo'q (scale train'dan o'rgan, test'ga qo'lla avtomatik), toza kod. ML ning eng yaxshi amaliyoti.

  3. Train'dan o'rgan, test'ga qo'lla. Muhim qoida: fit_transform faqat train'da (o'rtacha, std — train'dan), transform test'da (train sozlamasi — fit emas!). fit_transform(X_test) — test ma'lumotini "ko'rish" (ma'lumot sizishi — data leakage — soxta yuqori aniqlik, real ishda yomon). Bu 25.2 (train/test bo'lish) davomi: test ko'rmagan bo'lishi kerak — model o'qitishda va tayyorlashda. Ma'lumot tayyorlash — ML ning eng muhim (va ko'p vaqt) qismi: yaxshi tayyorlash + oddiy model > yomon tayyorlash + murakkab model. "Ma'lumot sifati model sifatini belgilaydi" — "garbage in, garbage out". Pipeline sizishni avtomatik oldini oladi.

Keyingi darsda modelni baholash ni o'rganamiz: aniqlikdan tashqari o'lchovlar (precision, recall, F1), chalkashlik matritsasi, cross-validation — model qancha yaxshi ishlashini to'g'ri o'lchash.

ML va AI · 3-dars yakunlandi

25.3-dars: Ma'lumotni tayyorlash

16 daqvaqt
0belgi
Keyingi dars: 25.4-dars: Modelni baholash
Telegram'da
Darsda xato koʻrdingizmi?

Izohlar · 0