IlmHamroh
Data Science va sun'iy intellekt/Pandas11/14-dars18 daqiqa
Mundarija (22)

3.11-dars: Apply va transformatsiya

3-QISM — PANDAS · 11-dars


1. Kirish va motivatsiya

3.6-darsda applyni qisqacha ko'rdik. Endi uni va boshqa transformatsiya usullarini chuqurroq o'rganamiz: apply (Series va DataFrame'da, qator/ustun bo'yicha), map (qiymat almashtirish), va eng muhimi — groupby bilan transform (guruh statistikasini har qatorga qaytarish). Bu usullar ma'lumotni kuchli o'zgartirish imkonini beradi: har qatorga o'z guruhining o'rtachasidan farqini hisoblash, murakkab shartli hisob, standartlashtirish. Nega muhim? (1) Murakkab o'zgartirish — vektorlashtirib bo'lmaydigan mantiq; (2) Guruh transform — har qatorga guruh statistikasi (feature engineering); (3) Moslashuvchanlik — istalgan funksiya. Lekin doim eslash kerak: iloji bo'lsa vektorlashtir (apply sekin). Bu dars apply va transformatsiyani chuqur o'rgatadi — ma'lumotni kuchli o'zgartirish.

Apply va transformatsiya — funksiya bilan o'zgartirish: apply (Series — element, DataFrame — qator/ustun axis), map (Series — qiymat almashtirish), transform (guruh — har qatorga guruh statistikasi), groupby.transform (guruh o'rtachasi har qatorga — feature engineering), lambda (nomsiz funksiya), vektorlashtirish (afzal — apply sekin). Foydalanish: murakkab o'zgartirish, guruh transform, feature engineering. Bu 3.6 (ustunlar), 3.8 (groupby) bilan bog'liq. apply — funksiya. transform — guruh. O'zgartirish.

Real vaziyat. Data Scientist talaba baholari bilan ishlar edi (sinf, ball). Guruh transform kerak edi: har talaba balli o'z sinfi o'rtachasidan qancha farq qiladi — df["farq"] = df["ball"] - df.groupby("sinf")["ball"].transform("mean") (har qatorga o'z sinfi o'rtachasi). apply (murakkab baho) — df["baho"] = df["ball"].apply(lambda x: "a'lo" if x >= 90 else "yaxshi" if x >= 70 else "qoniqarli") (uch daraja). map (kod) — df["sinf_kod"] = df["sinf"].map({"A": 1, "B": 2}). groupby.transform har qatorga guruh statistikasi qaytardi (feature engineering — sinf o'rtachasiga nisbatan). Apply/transform ma'lumotni kuchli o'zgartirdi (guruh, murakkab mantiq). transform — Data Science feature engineering asosi.

Bu darsda apply va transformatsiyani o'rganamiz.

Bu darsda:

  • apply (Series va DataFrame)
  • groupby.transform (guruh statistikasi)
  • map va almashtirish
  • lambda funksiyalar
  • Vektorlashtirish afzal
  • Apply amaliyoti
  • Apply tuzoqlari
  • Amaliy: transform modeli

ℹ Misollar real pandas bilan (deterministik) ishlaydi.


2. Nazariya — chuqur tushuntirish

2.1. apply (Series va DataFrame)

Funksiya qo'llash:

python
import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3], "b": [10, 20, 30]})

# Series apply (har element)
df["a"].apply(lambda x: x ** 2)          # [1, 4, 9]

# DataFrame apply (ustun bo'yicha — axis=0)
df.apply(lambda ustun: ustun.max())      # har ustun max

# DataFrame apply (qator bo'yicha — axis=1)
df.apply(lambda qator: qator["a"] + qator["b"], axis=1)   # a+b

apply (Series va DataFrame) — funksiya qo'llash: Series (df["a"].apply(func) — har element), DataFrame (df.apply(func) — ustun bo'yicha axis=0; axis=1 — qator bo'yicha). Sabab: murakkab o'zgartirish (vektorlashtirib bo'lmaydigan — shart, o'z funksiya); apply moslashuvchan (istalgan funksiya). axis muhim (axis=0 — har ustun; axis=1 — har qator, bir nechta ustundan hisob). Series (element), DataFrame (qator/ustun — axis). apply — funksiya (Series element, DataFrame axis). Moslashuvchan. axis.

2.2. groupby.transform (guruh statistikasi)

Guruh statistikasi har qatorga:

python
df = pd.DataFrame({
    "sinf": ["A", "A", "B", "B"],
    "ball": [80, 90, 60, 70],
})

# transform — guruh statistikasi HAR QATORGA
df["sinf_ort"] = df.groupby("sinf")["ball"].transform("mean")
# A: 85, A: 85, B: 65, B: 65 (har qatorga o'z sinf o'rtachasi)

# farq (o'z o'rtachasidan)
df["farq"] = df["ball"] - df["sinf_ort"]

groupby.transform (guruh statistikasi) — guruh statistikasini har qatorga qaytaradi: df.groupby("sinf")["ball"].transform("mean") (har qator uchun o'z sinfi o'rtachasi — A qatorlar 85, B qatorlar 65). Sabab: ba'zan har qatorga guruh statistikasi kerak (o'z sinf o'rtachasidan farq — standartlashtirish; guruh bo'yicha normallashtirish); groupby.agg (guruh — kichik natija, 3.8) vs groupby.transform (har qatorga — asl o'lcham). Farq: agg (guruh soni — 2 sinf); transform (qator soni — 4 talaba, har biriga guruh qiymati). Feature engineering (guruh xususiyat). transform — guruh statistikasi har qatorga (asl o'lcham). agg emas. Feature.

2.3. map va almashtirish

Qiymat almashtirish:

python
df = pd.DataFrame({"sinf": ["A", "B", "A", "C"]})

# map — dict bilan almashtirish
df["kod"] = df["sinf"].map({"A": 1, "B": 2, "C": 3})     # [1,2,1,3]

# map — funksiya bilan
df["katta"] = df["sinf"].map(lambda x: x.upper())

# replace — qiymatlarni almashtirish (DataFrame)
df.replace({"A": "Alfa", "B": "Beta"})

map va almashtirish — qiymat almashtirish: map (Series — dict {"A": 1} yoki funksiya; qiymat → yangi qiymat), replace (DataFrame — qiymatlarni almashtirish; {"eski": "yangi"}). Sabab: kategoriya → kod (matn → son — ML; map dict), qiymat tuzatish (xato → to'g'ri; replace). map (Series — bir ustun), replace (DataFrame — hamma). map (dict — tez, aniq; yo'q kalit → NaN — 3.6). map/replace — qiymat almashtirish (dict). Kategoriya→kod. Tuzatish.

2.4. lambda funksiyalar

Nomsiz funksiya:

python
# lambda — qisqa nomsiz funksiya
kvadrat = lambda x: x ** 2          # oddiy funksiya kabi
df["a"].apply(lambda x: x ** 2)     # apply ichida

# shartli lambda
df["baho"] = df["ball"].apply(
    lambda x: "a'lo" if x >= 90 else "yaxshi" if x >= 70 else "past"
)

# murakkab — oddiy funksiya afzal (o'qilishli)
def bahole(x):
    if x >= 90: return "a'lo"
    return "past"

lambda funksiyalar — nomsiz funksiya: lambda x: x ** 2 (qisqa — bir ifoda; apply ichida ko'p ishlatiladi). Sabab: apply/map funksiya talab qiladi (har elementga); oddiy amal uchun lambda (qisqa — alohida funksiya yozmasdan); shartli (lambda x: "a" if x > 0 else "b"). Murakkab bo'lsa — oddiy funksiya afzal (def — o'qilishli; lambda bir qatorda — chalkash). lambda (oddiy — bir ifoda), def (murakkab — o'qilishli). lambda — nomsiz funksiya (qisqa, apply ichida). Oddiy. Murakkabda def.

2.5. Vektorlashtirish afzal

Vektorlashtirish afzal — apply oxirgi chora: iloji bo'lsa vektorlashtir (df["a"] * 2 — tez, C; apply(lambda x: x*2) — sekin, Python sikl — 2.12/3.6); apply faqat vektorlashtirib bo'lmaganda (murakkab mantiq, matn regex, tashqi funksiya). Vektorlashtirilgan mumkin amallar: arifmetik (* + - /), taqqoslash (> ==), np.where (shartli — 2.9), matn (.str — 3.5), map (dict almashtirish). apply — oxirgi chora (moslashuvchan lekin sekin). Sabab: apply Python sikl (har element — sekin; million qator — farq katta 2.12); vektorlashtirilgan C (tez). To'g'ri tartib: avval vektorlashtir (* , np.where, .str, map), keyin apply (iloji bo'lmasa). Vektorlashtirish afzal — apply oxirgi chora (sekin). Tez. Iloji bo'lsa.

2.6. Apply amaliyoti

Apply amaliyoti: apply (Series element, DataFrame axis — murakkab mantiq); groupby.transform (guruh statistikasi har qatorga — feature engineering); map (dict almashtirish — kategoriya→kod); replace (qiymat tuzatish); lambda (oddiy — apply ichida; murakkab — def); vektorlashtir (avval — *, np.where, .str, map); axis (apply — 0 ustun, 1 qator). Tuzoqlar: apply sekin (vektorlashtir), axis (0 vs 1), transform vs agg (har qator vs guruh), map yo'q kalit (NaN), lambda murakkab (def afzal). Amaliyot — apply, transform, map, lambda. Feature. Vektorlashtir.

2.7. Apply tuzoqlari

Apply asosiy tuzoqlari: apply sekin (Python sikl — vektorlashtir afzal; 2.12); axis chalkash (apply(func) — ustun bo'yicha axis=0 (standart); qator uchun axis=1; noto'g'ri axis — xato natija); transform vs agg (transform — har qatorga (asl o'lcham); agg — guruh (kichik); chalkash — transform feature uchun, agg xulosa uchun); map yo'q kalit (map({"A": 1}) — "B" bo'lsa → NaN; to'liq dict yoki fillna); lambda murakkab (uzun lambda — o'qib bo'lmaydi; def afzal); transform funksiya (transform("mean") — string yoki funksiya; asl o'lcham qaytarishi kerak — sum/mean/std); apply natija turi (Series/skalyar — turli natija shakl); inplace yo'q (apply/map — yangi, df["a"] = ... saqlash). Sabab: apply axis/tur/tezlik nozik (axis, transform/agg, sekin — jim xato yoki sekin). Yechim: vektorlashtir, axis to'g'ri, transform (har qator), def (murakkab). Tuzoqlar — sekin, axis, transform/agg, map.

2.8. Apply — ma'lumotni kuchli o'zgartirish

Apply asosiy g'oyasi — ma'lumotni kuchli o'zgartirish: vektorlashtirib bo'lmaganda (murakkab mantiq, guruh statistikasi) apply/transform moslashuvchan o'zgartirish beradi. apply (Series element, DataFrame axis — murakkab funksiya), groupby.transform (guruh statistikasi har qatorga — feature engineering, standartlashtirish; agg dan farq — asl o'lcham), map (qiymat almashtirish — kategoriya→kod), lambda (oddiy funksiya). Eng muhim: transform (guruh xususiyat — har qatorga o'z guruh statistikasi; ML feature). Lekin vektorlashtirish afzal (apply sekin — Python sikl; avval */np.where/.str/map, keyin apply). Data Science'da (murakkab o'zgartirish — apply; guruh feature — transform; ML tayyorgarlik — 20-qism). Bu 3.6 (ustunlar — apply) va 3.8 (groupby — transform) davomi, feature engineering uchun. Apply — ma'lumotni kuchli o'zgartirish (apply, transform, map). Feature. Moslashuvchan.


3. Tez ma'lumotnoma

python
import pandas as pd

# apply (Series — element):
df["a"].apply(lambda x: x ** 2)

# apply (DataFrame — axis):
df.apply(lambda ustun: ustun.max())        # ustun (axis=0)
df.apply(lambda qator: qator["a"]+qator["b"], axis=1)  # qator

# groupby.transform (guruh statistikasi HAR QATORGA):
df["sinf_ort"] = df.groupby("sinf")["ball"].transform("mean")
df["farq"] = df["ball"] - df["sinf_ort"]   # o'z guruhidan farq
#   agg — guruh (kichik) · transform — har qator (asl o'lcham)

# map (qiymat almashtirish):
df["kod"] = df["sinf"].map({"A": 1, "B": 2})   # dict
df.replace({"A": "Alfa"})                       # DataFrame

# lambda (oddiy) / def (murakkab):
df["a"].apply(lambda x: x*2)   # oddiy

# VEKTORLASHTIR AFZAL (apply sekin):
df["a"] * 2   # tez (apply emas)

QOIDA: vektorlashtir avval · transform har qator (agg guruh) · axis · def murakkab

Apply xulosasi

Apply — ma'lumotni kuchli o'zgartirish (murakkab, guruh)
apply — Series (element), DataFrame (axis=0 ustun, axis=1 qator)
groupby.transform — guruh statistikasi HAR QATORGA (feature)
  agg (guruh, kichik) vs transform (har qator, asl o'lcham)
map — qiymat almashtirish (dict) · vektorlashtir afzal (apply sekin)

4. Batafsil misollar

Misollar real pandas bilan (deterministik) ishlaydi.

Misol 1 — apply

python
"""apply: Series va DataFrame (real pandas)."""

import pandas as pd


def main() -> None:
    df = pd.DataFrame({"a": [1, 2, 3], "b": [10, 20, 30]})

    print("=== 1. Series apply (element) ===")
    print(f"  kvadrat: {list(df['a'].apply(lambda x: x ** 2))}")

    print("\n=== 2. DataFrame apply (ustun, axis=0) ===")
    print(f"  har ustun max: {df.apply(lambda u: u.max()).to_dict()}")

    print("\n=== 3. DataFrame apply (qator, axis=1) ===")
    yigindi = df.apply(lambda q: q["a"] + q["b"], axis=1)
    print(f"  a+b: {list(yigindi)}")

    print("\n=== 4. Shartli apply ===")
    df["kat"] = df["a"].apply(lambda x: "katta" if x > 1 else "kichik")
    print(f"  kategoriya: {list(df['kat'])}")
    print("  ⭐ apply — funksiya (Series element, DataFrame axis)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Series apply (element) ===
  kvadrat: [1, 4, 9]

=== 2. DataFrame apply (ustun, axis=0) ===
  har ustun max: {'a': 3, 'b': 30}

=== 3. DataFrame apply (qator, axis=1) ===
  a+b: [11, 22, 33]

=== 4. Shartli apply ===
  kategoriya: ['kichik', 'katta', 'katta']
  ⭐ apply — funksiya (Series element, DataFrame axis)

Nima ko'rsatdi: 2.1-bo'lim.

Misol 2 — groupby.transform

python
"""groupby.transform: guruh statistikasi (real pandas)."""

import pandas as pd


def main() -> None:
    df = pd.DataFrame({
        "sinf": ["A", "A", "B", "B", "A"],
        "ball": [80, 90, 60, 70, 100],
    })

    print("=== 1. transform (har qatorga) ===")
    df["sinf_ort"] = df.groupby("sinf")["ball"].transform("mean")
    print(f"  sinf o'rtachalari: {list(df['sinf_ort'])}")

    print("\n=== 2. Farq (o'z guruhidan) ===")
    df["farq"] = df["ball"] - df["sinf_ort"]
    print(f"  farqlar: {list(df['farq'])}")

    print("\n=== 3. agg vs transform ===")
    agg = df.groupby("sinf")["ball"].mean()
    print(f"  agg (guruh): {len(agg)} qator · transform (har qator): {len(df)}")

    print("\n=== 4. Guruh max ===")
    df["sinf_max"] = df.groupby("sinf")["ball"].transform("max")
    print(f"  sinf max: {list(df['sinf_max'])}")
    print("  ⭐ transform — guruh statistikasi har qatorga (feature)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. transform (har qatorga) ===
  sinf o'rtachalari: [90.0, 90.0, 65.0, 65.0, 90.0]

=== 2. Farq (o'z guruhidan) ===
  farqlar: [-10.0, 0.0, -5.0, 5.0, 10.0]

=== 3. agg vs transform ===
  agg (guruh): 2 qator · transform (har qator): 5

=== 4. Guruh max ===
  sinf max: [100, 100, 70, 70, 100]
  ⭐ transform — guruh statistikasi har qatorga (feature)

Nima ko'rsatdi: 2.2-bo'lim.

Misol 3 — map va replace

python
"""map va replace (real pandas)."""

import pandas as pd


def main() -> None:
    df = pd.DataFrame({"sinf": ["A", "B", "A", "C"]})

    print("=== 1. map (dict) ===")
    df["kod"] = df["sinf"].map({"A": 1, "B": 2, "C": 3})
    print(f"  kodlar: {list(df['kod'])}")

    print("\n=== 2. map (funksiya) ===")
    df["kichik"] = df["sinf"].map(lambda x: x.lower())
    print(f"  kichik: {list(df['kichik'])}")

    print("\n=== 3. replace ===")
    d2 = df.replace({"A": "Alfa", "B": "Beta"})
    print(f"  almashtirilgan: {list(d2['sinf'])}")

    print("\n=== 4. map yo'q kalit → NaN ===")
    natija = df["sinf"].map({"A": 1})
    print(f"  NaN soni: {natija.isna().sum()} (B, C yo'q)")
    print("  ⭐ map — qiymat almashtirish (dict)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. map (dict) ===
  kodlar: [1, 2, 1, 3]

=== 2. map (funksiya) ===
  kichik: ['a', 'b', 'a', 'c']

=== 3. replace ===
  almashtirilgan: ['Alfa', 'Beta', 'Alfa', 'C']

=== 4. map yo'q kalit → NaN ===
  NaN soni: 2 (B, C yo'q)
  ⭐ map — qiymat almashtirish (dict)

Nima ko'rsatdi: 2.3-bo'lim.

Misol 4 — Vektorlashtirish vs apply

python
"""Vektorlashtirish vs apply (real pandas)."""

import numpy as np
import pandas as pd


def main() -> None:
    df = pd.DataFrame({"narx": [100, 200, 150, 300]})

    print("=== 1. Vektorlashtirilgan (afzal) ===")
    df["chegirma"] = df["narx"] * 0.9
    print(f"  chegirma: {list(df['chegirma'])}")

    print("\n=== 2. np.where (shartli — vektor) ===")
    df["kat"] = np.where(df["narx"] > 150, "qimmat", "arzon")
    print(f"  kategoriya: {list(df['kat'])}")

    print("\n=== 3. apply (murakkab mantiq) ===")
    df["daraja"] = df["narx"].apply(
        lambda x: "yuqori" if x > 250 else "o'rta" if x > 120 else "past"
    )
    print(f"  daraja: {list(df['daraja'])}")

    print("\n=== 4. Xulosa ===")
    print("  oddiy → vektorlashtir (tez) · murakkab → apply")
    print("  ⭐ Vektorlashtirish afzal (apply sekin — oxirgi chora)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Vektorlashtirilgan (afzal) ===
  chegirma: [90.0, 180.0, 135.0, 270.0]

=== 2. np.where (shartli — vektor) ===
  kategoriya: ['arzon', 'qimmat', 'arzon', 'qimmat']

=== 3. apply (murakkab mantiq) ===
  daraja: ['past', "o'rta", "o'rta", 'yuqori']

=== 4. Xulosa ===
  oddiy → vektorlashtir (tez) · murakkab → apply
  ⭐ Vektorlashtirish afzal (apply sekin — oxirgi chora)

Nima ko'rsatdi: 2.5-bo'lim.


5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"apply doim kerak" Vektorlashtir afzal (sekin)
"apply axis muhim emas" axis=0 ustun, axis=1 qator
"transform = agg" transform har qator, agg guruh
"map hamma kalitni" Yo'q kalit → NaN
"lambda murakkab uchun" def afzal (o'qilishli)
"transform xulosa" Feature (har qator)
"apply tez" Sekin (Python sikl)
"map = replace" map Series, replace DataFrame

6. Keng tarqalgan xatolar va yechimlari

1. apply sekin

python
df["a"].apply(lambda x: x * 2)   # sekin                     # ⚠️
df["a"] * 2   # vektorlashtir (tez)                           # ✅

2. axis

python
df.apply(func)   # ustun bo'yicha (axis=0)                   # ⚠️
df.apply(func, axis=1)   # qator bo'yicha                     # ✅

3. transform vs agg

python
df.groupby("g")["b"].mean()   # guruh (kichik)               # ⚠️
df.groupby("g")["b"].transform("mean")   # har qator          # ✅

4. map yo'q kalit

python
df["s"].map({"A": 1})   # "B" → NaN                          # ⚠️
df["s"].map({"A": 1, "B": 2})   # to'liq dict                 # ✅

5. lambda murakkab

python
df["a"].apply(lambda x: ... uzun mantiq ...)   # o'qib bo'lmaydi # ⚠️
def func(x): ...; df["a"].apply(func)   # def (o'qilishli)     # ✅

6. Natijani saqlamaslik

python
df["a"].apply(func)   # asl o'zgarmaydi                       # ⚠️
df["a"] = df["a"].apply(func)   # saqla                        # ✅

7. transform funksiya

python
df.groupby("g")["b"].transform(lambda x: x.head())   # o'lcham xato # ⚠️
df.groupby("g")["b"].transform("mean")   # asl o'lcham         # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 3.6-dars (o'tilgan): Ustunlar (apply, map)
  • 3.8-dars (o'tilgan): groupby (transform)
  • 2.9-dars (o'tilgan): np.where (vektorlashtirilgan shart)
  • 6-qism: Tozalash (murakkab o'zgartirish)
  • 20-qism: ML (feature engineering — transform)

8. Eng yaxshi amaliyotlar

  1. Vektorlashtir avval (*, np.where, .str, map).

  2. apply — faqat murakkab (sekin).

  3. axis — 0 ustun, 1 qator (apply).

  4. transform — guruh feature (har qator).

  5. agg — guruh xulosa (kichik).

  6. map — to'liq dict (yo'q → NaN).

  7. def — murakkab lambda (o'qilishli).

  8. Apply — ma'lumotni kuchli o'zgartirish.


9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # apply nima?
2.  # Series vs DataFrame apply?
3.  # axis nima?
4.  # transform nima?
5.  # transform vs agg?
6.  # map nima?
7.  # map yo'q kalit?
8.  # lambda nima?
9.  # lambda vs def?
10. # apply tez?
11. # vektorlashtir yoki apply?
12. # nega transform muhim?
Javoblar
  1. Funksiya qo'llash
  2. Series element, DataFrame axis
  3. axis=0 ustun, axis=1 qator
  4. Guruh statistikasi har qatorga
  5. transform har qator, agg guruh
  6. Qiymat almashtirish (dict)
  7. NaN
  8. Nomsiz funksiya
  9. lambda oddiy, def murakkab
  10. Sekin (Python sikl)
  11. Vektorlashtir (apply oxirgi)
  12. Feature engineering (guruh xususiyat)

Vazifa 2: Xatolarni tuzating

python
1.  df["a"].apply(lambda x: x*2)   # tez

2.  df.apply(func)   # qator bo'yicha

3.  df.groupby("g")["b"].mean()   # har qatorga

4.  df["s"].map({"A": 1})   # "B" bor

5.  df["a"].apply(func)   # saqlash
Javoblar
python
1.  df["a"] * 2

2.  df.apply(func, axis=1)

3.  ....transform("mean")

4.  {"A": 1, "B": 2}

5.  df["a"] = df["a"].apply(func)

Vazifa 3: apply

Modellang:

  1. Series
  2. DataFrame
  3. axis
  4. Murakkab

Vazifa 4: transform

Modellang:

  1. Guruh
  2. Har qator
  3. agg farqi
  4. Feature

Vazifa 5: map

Modellang:

  1. Dict
  2. Almashtirish
  3. Yo'q kalit
  4. Kod

Vazifa 6: Vektorlashtirish

Modellang:

  1. Afzal
  2. Tez
  3. apply sekin
  4. Oxirgi chora

Vazifa 7: O'ylash

groupby.transform guruh statistikasini (masalan o'rtacha) har qatorga qaytaradi, groupby.agg esa har guruh uchun bitta qiymat. Nima uchun bu farq feature engineering (ML uchun xususiyat yaratish) da shunchalik muhim, va nega "har qatorga o'z guruhining statistikasi" kuchli xususiyat bo'lishi mumkin?

Javob

Qisqa javob: transform guruh statistikasini har qatorga (asl o'lcham), agg guruh (kichik); farq feature engineering'da muhim, chunki: (1) feature — har qatorga — ML har namuna (qator) uchun xususiyat talab qiladi (model qatorlar bilan ishlaydi); transform har qatorga qiymat beradi (yangi ustun — model uchun); agg kichik (guruh soni — qatorga mos kelmaydi, model uchun to'g'ridan ishlamaydi); (2) kontekst xususiyat — "har qatorga o'z guruhining statistikasi" — kuchli (talaba balli o'z sinfiga nisbatan — 90 ball A sinfda o'rtacha, B sinfda a'lo; kontekst muhim); transform bu kontekstni qo'shadi (guruh o'rtachasi — har qator); (3) nisbiy xususiyat — mutloq qiymat (ball 90) vs nisbiy (o'z guruhidan farq — +5); nisbiy ko'pincha kuchliroq (kontekst — model yaxshi o'rganadi). "Nega kuchli xususiyat": (a) kontekst — qiymat o'z guruhida ma'no (90 ball — qaysi sinfda?; o'z sinf o'rtachasidan farq — aniq); model kontekstni ko'radi (guruh bilan solishtirish); (b) normallashtirish — guruh bo'yicha (har sinf o'z o'lchovida — z-score guruhda; 2.6); (c) naqsh — guruh ichidagi holat (o'rtachadan yuqori/past — model uchun signal); (d) feature engineering — yangi ma'noli ustun (o'z guruhidan farq — model kirishi). Misol: mijoz xaridi (1000) vs shahri o'rtachasi (transform — shahar o'rtachasi); farq (mijoz o'z shahrida ko'p/kam xarid — kontekst; VIP aniqlash). "Nega transform (agg emas)": agg kichik (guruh — 5 shahar; qatorga (10000 mijoz) mos kelmaydi; qo'lda merge kerak); transform har qatorga (10000 — to'g'ridan ustun; oson). transform avtomatik (guruh → har qator — merge kerak emas). Saboqlar: transform har qatorga (feature — model uchun); agg guruh (xulosa — kichik); kontekst xususiyat (o'z guruhidan — kuchli); nisbiy (mutloqdan yaxshi — ko'pincha). To'g'ri: feature — transform (har qator); xulosa — agg (guruh). Muvozanat: xulosa (agg — hisobot) + feature (transform — ML) — turli maqsad. Bu feature engineering asosi (guruh xususiyat — transform; ML — 20-qism); "o'z guruhidan farq" — kuchli signal (kontekst). transform — ML feature yaratish vositasi (guruh statistikasi har qatorga).

1. Nega transform har qatorga

  • ML har namuna (qator) — xususiyat
  • transform har qatorga (yangi ustun — model)
  • agg kichik (guruh — qatorga mos emas)

2. Nega kuchli xususiyat

  • Kontekst (o'z guruhida ma'no)
  • Normallashtirish (guruh bo'yicha)
  • Naqsh (guruh ichidagi holat — signal)
  • Nisbiy (mutloqdan yaxshi)

3. transform vs agg (feature)

transform agg
Har qator (asl o'lcham) Guruh (kichik)
Feature (model) Xulosa (hisobot)
Merge kerak emas Merge kerak

4. Misol

  • Mijoz xarid (1000) vs shahar o'rtacha (transform)
  • Farq (o'z shahrida ko'p/kam — VIP signal)

5. Saboqlar

  1. transform har qatorga (feature)
  2. agg guruh (xulosa)
  3. Kontekst xususiyat (kuchli)
  4. Nisbiy (mutloqdan yaxshi)

6. Xulosa

  1. transform har qatorga (feature — ML)
  2. agg guruh (xulosa — kichik)
  3. "O'z guruhidan farq" (kontekst — kuchli)
  4. transform — feature engineering vositasi

Nimani mustahkamlaydi: 2.2, 2.8-bo'limlar.


Xulosa

Bu darsda apply va transformatsiyani o'rgandik.

Eng muhim uch fikr:

  1. apply va transform. apply — funksiya qo'llash: Series (df["a"].apply(func) — har element), DataFrame (df.apply(func) — ustun axis=0; axis=1 — qator). groupby.transform — guruh statistikasini har qatorga qaytaradi (df.groupby("sinf")["ball"].transform("mean") — har qator o'z sinfi o'rtachasi); agg dan farq (agg — guruh, kichik; transform — har qator, asl o'lcham; feature engineering).

  2. map va lambda. map — qiymat almashtirish (Series — dict {"A": 1} yoki funksiya; kategoriya→kod; yo'q kalit → NaN), replace (DataFrame — qiymat tuzatish). lambda — nomsiz funksiya (lambda x: x**2 — qisqa, apply ichida; murakkab bo'lsa def afzal — o'qilishli).

  3. Vektorlashtirish afzal. Vektorlashtirish afzal — apply oxirgi chora (Python sikl — sekin 2.12; avval */np.where/.str/map — tez); apply faqat murakkab mantiq/tashqi funksiya uchun. Apply — ma'lumotni kuchli o'zgartirish (murakkab, guruh); transform — feature engineering (o'z guruhidan farq — kontekst xususiyat, ML kuchli signal; 20-qism). Data Science'da (murakkab o'zgartirish — apply; guruh feature — transform). Tuzoqlar: apply sekin (vektorlashtir), axis (0/1), transform vs agg (har qator vs guruh), map yo'q kalit (NaN), lambda murakkab (def).

Keyingi darsda vaqt qatorlari (datetime)ni o'rganamiz: sana/vaqt bilan ishlash, vaqt bo'yicha guruhlash (resample), trend — vaqtga bog'liq ma'lumot tahlili.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
3.11-dars: Apply va transformatsiya — IlmHamroh