IlmHamroh
Data Science va sun'iy intellekt/Regressiya6/12-dars18 daqiqa
Mundarija (22)

13.6-dars: Gradient tushish

13-QISM — REGRESSIYA · 6-dars

1. Kirish va motivatsiya

Chiziqli regressiyada koeffitsiyentlarni analitik topish mumkin (w = (XTX)^(-1)XTy). Lekin bu yo'l ikki holatda ishlamaydi: ma'lumot juda katta bo'lsa (matritsa xotiraga sig'maydi) yoki yo'qotish funksiyasi analitik yechimga ega bo'lmasa (logistik regressiya — 13.10, neyron tarmoqlar). Shunda gradient tushish ishlatiladi: yechimga qadam-baqadam yaqinlashish.

Bu darsda: gradient nima va nega u eng tik yo'nalishni ko'rsatadi, o'rganish tezligi (learning rate) va uning tanlovi, standartlashtirish nega hal qiluvchi, stoxastik (SGD) va mini-batch variantlar, konvergensiya diagnostikasi va sklearn dagi SGDRegressor.

Real vaziyat. Jamoa 40 million qatorli ma'lumotda regressiya qurmoqchi: XTX matritsasi hisoblash uchun butun ma'lumotni xotiraga yuklash kerak — imkonsiz. SGDRegressor bilan ma'lumot oqim sifatida qismlarga bo'lib o'qitildi: 6 daqiqada OLS ga deyarli teng natija. Lekin birinchi urinish tarqab ketgan edi — sabab: belgilar standartlashtirilmagan (biri 0-1, boshqasi 0-500 000).

Bu darsda gradient tushishni o'rganamiz.

Bu darsda:

  • Gradient va qadam
  • O'rganish tezligi
  • Standartlashtirish nega shart
  • Stoxastik va mini-batch
  • Konvergensiya diagnostikasi
  • SGDRegressor
  • Tuzoqlar
  • Amaliy: qo'lda va sklearn bilan

ℹ Misollar real numpy/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Gradient va qadam

text
Yo'qotish: J(w) = (1/n) · sum (y_i - x_i·w)^2

Gradient (har koeffitsiyent bo'yicha hosila):
  grad = -(2/n) · XT (y - Xw)

Qadam:  w_yangi = w - lr · grad         lr — o'rganish tezligi

Gradient — yo'qotish eng tez O'SADIGAN yo'nalish;
shuning uchun uning TESKARISIGA qadam tashlaymiz

Gradient tushish — optimallashtirishning eng oddiy va eng keng tarqalgan usuli: joriy nuqtada eng tik pastga tushish yo'nalishini hisoblab, shu tomonga kichik qadam tashlash va takrorlash. Kvadratik yo'qotish qavariq (convex) bo'lgani uchun chiziqli regressiyada yagona minimum bor — gradient tushish har doim unga yaqinlashadi (to'g'ri lr bilan).

2.2. O'rganish tezligi

text
lr juda KICHIK  → sekin konvergensiya (minglab iteratsiya)
lr juda KATTA   → sakrab o'tadi, yo'qotish o'sadi, tarqab ketadi (NaN)
lr to'g'ri      → yo'qotish tez va barqaror kamayadi

Tanlash: 0.001, 0.01, 0.1, 0.3 ni sinang va yo'qotish egri chizig'iga qarang
Zamonaviy usul: moslashuvchan lr (Adam, RMSProp — 22-qism) yoki jadval (schedule)

O'rganish tezligi — gradient tushishdagi eng muhim giperparametr. Diagnostika oddiy: yo'qotish egri chizig'i — u silliq kamayishi kerak. Sakrab turgan egri chiziq — lr katta; deyarli gorizontal — lr kichik. sklearn da learning_rate="invscaling" yoki "adaptive" avtomatik kamaytiradi.

2.3. Standartlashtirish nega shart

text
Belgilar masshtabi juda farq qilsa, yo'qotish sirti CHO'ZILGAN vodiy bo'ladi:
  gradient vodiy devoriga perpendikulyar yo'naladi → zigzag → sekin konvergensiya

Standartlashtirishdan keyin sirt DUMALOQ → to'g'ri yo'nalish → tez konvergensiya

Analitik yechim (lstsq) uchun standartlashtirish SHART EMAS
Gradient tushish uchun — deyarli MAJBURIY

Bu — gradient usullaridagi eng ko'p uchraydigan amaliy xato. StandardScaler shart emas (OLS uchun), lekin majburiy (SGD uchun) — farqni tushunish muhim. Shartlilik soni (condition number — 10.5) katta bo'lsa, konvergensiya bir necha tartibga sekinlashadi.

2.4. Stoxastik va mini-batch

text
TO'LIQ (batch):  har qadamda BARCHA n namuna → aniq gradient, qimmat
STOXASTIK (SGD): har qadamda BITTA namuna   → shovqinli, juda tez, xotira talab qilmaydi
MINI-BATCH:      har qadamda 32-256 namuna  → amaliy muvozanat (standart)

Epoxa (epoch) — butun ma'lumotdan bir marta o'tish
SGD shovqini foydali ham: lokal minimumlardan chiqishga yordam beradi (DL da)

Mini-batch — zamonaviy standart: u to'liq gradientning barqarorligi bilan SGD ning tezligini birlashtiradi va GPU da samarali. Chiziqli regressiyada SGD oqim rejimida ishlash imkonini beradi: ma'lumot xotiraga sig'masa ham model o'qitiladi (partial_fit).

2.5. Konvergensiya diagnostikasi

text
1. Yo'qotish egri chizig'i — silliq kamayadimi?
2. Koeffitsiyentlar o'zgarishi — kichrayyaptimi?
3. Analitik yechim bilan solishtirish (kichik ma'lumotda)
4. To'xtash mezoni: |J_yangi - J_eski| < tol yoki maksimal iteratsiya

Belgilar:
  yo'qotish o'sadi/NaN   → lr katta
  juda sekin kamayadi    → lr kichik yoki standartlashtirilmagan
  tebranib turadi        → SGD normasi (o'rtachalab qarang)

Yo'qotish egri chizig'i — gradient usullarining asosiy diagnostikasi (22-qismda ham xuddi shunday). Chiziqli regressiyada natijani analitik yechim bilan solishtirish mumkin — bu o'rganish uchun eng yaxshi tekshiruv: gradient tushish to'g'ri ishlasa, koeffitsiyentlar lstsq bilan mos tushadi.

2.6. SGDRegressor

python
from sklearn.linear_model import SGDRegressor

model = SGDRegressor(loss="squared_error", penalty="l2", alpha=1e-4,
                     learning_rate="invscaling", eta0=0.01,
                     max_iter=1000, tol=1e-4, random_state=0)
# MAJBURIY: StandardScaler bilan pipeline ichida

model.partial_fit(X_batch, y_batch)      # oqim rejimi (katta ma'lumot)

SGDRegressor — katta ma'lumot uchun chiziqli modellar oilasi: loss ni o'zgartirib OLS (squared_error), Huber (huber — 13.11) yoki epsilon-insensitive (SVR) olish mumkin; penalty bilan Ridge/Lasso/ElasticNet (13.7-13.8). Kichik ma'lumotda esa oddiy LinearRegression/Ridge aniqroq va qulayroq — SGD ni faqat kerak bo'lganda ishlating.

2.7. Tuzoqlar

Asosiy tuzoqlar: standartlashtirmaslik (eng ko'p uchraydigan); lr ni sinamasdan tanlash; yo'qotish egri chizig'iga qaramaslik; kichik ma'lumotda SGD ishlatish (keraksiz); max_iter ni juda kichik qo'yish; SGD natijasini analitik yechim bilan solishtirmaslik; partial_fit da ma'lumot tartibini aralashtirmaslik; test to'plamida fit qilish 12.9-bob.

2.8. Iterativ yechim — universal vosita

Gradient tushish yo'qotishning eng tik pasayish yo'nalishiga kichik qadamlar tashlaydi: w ← w - lr·grad. Chiziqli regressiyada yo'qotish qavariq — yagona minimumga yaqinlashadi. Ikki hal qiluvchi tafsilot: o'rganish tezligi (katta bo'lsa tarqab ketadi, kichik bo'lsa sekin) va standartlashtirish (bo'lmasa zigzag va sekin konvergensiya). Mini-batch — amaliy standart; SGDRegressor katta yoki oqim ma'lumotida ishlaydi. Bu usul chiziqli regressiyadan ko'ra ko'proq Deep Learning uchun muhim (22-qism). Keyingi dars — Ridge regularizatsiyasi.

3. Tez ma'lumotnoma

python
import numpy as np
from sklearn.linear_model import SGDRegressor
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# qo'lda
def gradient_tushish(X, y, lr=0.1, qadam=500):
    w = np.zeros(X.shape[1])
    tarix = []
    for _ in range(qadam):
        xato = X @ w - y
        w -= lr * (2 / len(y)) * (X.T @ xato)
        tarix.append(np.mean(xato ** 2))
    return w, tarix

# sklearn
Pipeline([("sc", StandardScaler()),
          ("m", SGDRegressor(eta0=0.01, max_iter=2000, tol=1e-5, random_state=0))])

# oqim rejimi
for X_b, y_b in bo_laklar:
    model.partial_fit(sc.transform(X_b), y_b)
QOIDA: standartlashtir · lr ni sina · yo'qotish egrisiga qara · lstsq bilan solishtir

Gradient xulosasi

grad = -(2/n)·XT(y - Xw) · w ← w - lr·grad
lr katta — tarqaydi · lr kichik — sekin · standartlashtirish majburiy
Batch (aniq) / SGD (tez) / mini-batch (standart) · epoxa — bir to'liq o'tish
SGDRegressor — katta va oqim ma'lumot uchun

4. Batafsil misollar

Misollar real numpy/sklearn bilan (Python 3.14).

Misol 1 — Qo'lda gradient tushish

python
"""Gradient tushish analitik yechimga yaqinlashadi (real numpy)."""

import numpy as np


def yarat(seed: int = 3, n: int = 2000):
    rng = np.random.default_rng(seed)
    x1 = rng.normal(0, 1, n)
    x2 = rng.normal(0, 1, n)
    y = 5 + 2.0 * x1 - 1.5 * x2 + rng.normal(0, 1, n)
    X = np.column_stack([np.ones(n), x1, x2])
    return X, y


def tushish(X, y, lr: float, qadam: int = 300):
    w = np.zeros(X.shape[1])
    tarix = []
    for _ in range(qadam):
        xato = X @ w - y
        tarix.append(float(np.mean(xato ** 2)))
        w = w - lr * (2 / len(y)) * (X.T @ xato)
    return w, tarix


def main() -> None:
    X, y = yarat()

    print("=== 1. Analitik yechim ===")
    w_aniq = np.linalg.lstsq(X, y, rcond=None)[0]
    print(f"  w = [{', '.join(f'{v:.4f}' for v in w_aniq)}]  (haqiqiy: 5, 2, -1.5)")
    print(f"  MSE = {np.mean((X @ w_aniq - y) ** 2):.6f}")

    print("\n=== 2. Gradient tushish, lr = 0.1 ===")
    w, tarix = tushish(X, y, 0.1)
    for q in [0, 1, 5, 20, 100, 299]:
        print(f"  qadam {q:>3}: MSE = {tarix[q]:10.4f}")
    print(f"  yakuniy w = [{', '.join(f'{v:.4f}' for v in w)}]")
    print(f"  analitikdan farq = {np.abs(w - w_aniq).max():.2e}")

    print("\n=== 3. Turli o'rganish tezligi ===")
    for lr in [0.001, 0.01, 0.1, 0.5, 1.1]:
        w, tarix = tushish(X, y, lr, 200)
        holat = "tarqadi" if not np.isfinite(tarix[-1]) or tarix[-1] > tarix[0] \
            else f"MSE {tarix[-1]:.4f}"
        print(f"  lr = {lr:<6}: {holat}")

    print("\n=== 4. Nechta qadam kerak ===")
    for lr in [0.01, 0.05, 0.1, 0.3]:
        w, tarix = tushish(X, y, lr, 1000)
        maqsad = np.mean((X @ w_aniq - y) ** 2) * 1.001
        kerak = next((i for i, v in enumerate(tarix) if v <= maqsad), None)
        print(f"  lr = {lr:<5}: {kerak} qadam (analitikdan 0.1% ichida)")
    print("  ⭐ Qavariq masalada gradient tushish har doim yechimga keladi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Analitik yechim ===
  w = [4.9893, 1.9810, -1.4671]  (haqiqiy: 5, 2, -1.5)
  MSE = 1.020647

=== 2. Gradient tushish, lr = 0.1 ===
  qadam   0: MSE =    32.6300
  qadam   1: MSE =    21.0392
  qadam   5: MSE =     4.2437
  qadam  20: MSE =     1.0241
  qadam 100: MSE =     1.0206
  qadam 299: MSE =     1.0206
  yakuniy w = [4.9893, 1.9810, -1.4671]
  analitikdan farq = 8.88e-16

=== 3. Turli o'rganish tezligi ===
  lr = 0.001 : MSE 15.0340
  lr = 0.01  : MSE 1.0293
  lr = 0.1   : MSE 1.0206
  lr = 0.5   : MSE 1.0206
  lr = 1.1   : tarqadi

=== 4. Nechta qadam kerak ===
  lr = 0.01 : 252 qadam (analitikdan 0.1% ichida)
  lr = 0.05 : 49 qadam (analitikdan 0.1% ichida)
  lr = 0.1  : 23 qadam (analitikdan 0.1% ichida)
  lr = 0.3  : 6 qadam (analitikdan 0.1% ichida)
  ⭐ Qavariq masalada gradient tushish har doim yechimga keladi

Nima ko'rsatdi: 2.1, 2.2-bo'limlar.

Misol 2 — Standartlashtirish nega hal qiluvchi

python
"""Masshtab farqi konvergensiyani qanday buzadi (real numpy)."""

import numpy as np


def tushish(X, y, lr: float, qadam: int = 2000):
    """Tarqab ketishi mumkin — shuning uchun hisob errstate ichida."""
    w = np.zeros(X.shape[1])
    tarix = []
    with np.errstate(all="ignore"):
        for _ in range(qadam):
            xato = X @ w - y
            j = float(np.mean(xato ** 2))
            if not np.isfinite(j):
                return w, tarix + [np.inf]
            tarix.append(j)
            w = w - lr * (2 / len(y)) * (X.T @ xato)
    return w, tarix


def main() -> None:
    rng = np.random.default_rng(7)
    n = 3000
    kichik = rng.uniform(0, 1, n)                 # 0..1
    katta = rng.uniform(0, 500_000, n)            # 0..500000
    y = 10 + 30 * kichik + 0.00004 * katta + rng.normal(0, 1, n)

    Xom = np.column_stack([np.ones(n), kichik, katta])
    ort = Xom[:, 1:].mean(axis=0)
    sd = Xom[:, 1:].std(axis=0)
    Std = np.column_stack([np.ones(n), (Xom[:, 1:] - ort) / sd])

    print("=== 1. Shartlilik soni ===")
    print(f"  xom belgilar:        {np.linalg.cond(Xom):.3e}")
    print(f"  standartlashtirilgan: {np.linalg.cond(Std):.3e}")

    print("\n=== 2. Xom belgilar bilan gradient tushish ===")
    for lr in [1e-12, 1e-10, 1e-8]:
        w, tarix = tushish(Xom, y, lr, 2000)
        holat = "tarqadi" if not np.isfinite(tarix[-1]) else f"MSE {tarix[-1]:.4f}"
        print(f"  lr = {lr:.0e}: {holat}")

    print("\n=== 3. Standartlashtirilgan belgilar bilan ===")
    for lr in [0.01, 0.1, 0.3]:
        w, tarix = tushish(Std, y, lr, 2000)
        holat = "tarqadi" if not np.isfinite(tarix[-1]) else f"MSE {tarix[-1]:.4f}"
        print(f"  lr = {lr:<5}: {holat}")

    print("\n=== 4. Analitik yechim bilan solishtirish ===")
    w_aniq = np.linalg.lstsq(Std, y, rcond=None)[0]
    w_grad, tarix = tushish(Std, y, 0.3, 2000)
    print(f"  analitik MSE = {np.mean((Std @ w_aniq - y) ** 2):.6f}")
    print(f"  gradient MSE = {tarix[-1]:.6f}")
    print(f"  koeffitsiyent farqi = {np.abs(w_grad - w_aniq).max():.2e}")
    print("  ⭐ Gradient tushishda standartlashtirish — majburiy")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Shartlilik soni ===
  xom belgilar:        1.149e+06
  standartlashtirilgan: 1.030e+00

=== 2. Xom belgilar bilan gradient tushish ===
  lr = 1e-12: MSE 244.9026
  lr = 1e-10: tarqadi
  lr = 1e-08: tarqadi

=== 3. Standartlashtirilgan belgilar bilan ===
  lr = 0.01 : MSE 0.9555
  lr = 0.1  : MSE 0.9555
  lr = 0.3  : MSE 0.9555

=== 4. Analitik yechim bilan solishtirish ===
  analitik MSE = 0.955492
  gradient MSE = 0.955492
  koeffitsiyent farqi = 3.55e-14
  ⭐ Gradient tushishda standartlashtirish — majburiy

Nima ko'rsatdi: 2.3-bo'lim.

Misol 3 — Batch, stoxastik va mini-batch

python
"""Uch variantning tezligi va barqarorligi (real numpy)."""

import numpy as np


def main() -> None:
    rng = np.random.default_rng(11)
    n, p = 20_000, 10
    X = np.column_stack([np.ones(n), rng.normal(0, 1, (n, p))])
    haqiqiy = np.concatenate([[3.0], rng.normal(0, 1.5, p)])
    y = X @ haqiqiy + rng.normal(0, 1, n)
    w_aniq = np.linalg.lstsq(X, y, rcond=None)[0]
    mse_aniq = float(np.mean((X @ w_aniq - y) ** 2))

    def ishga_tushir(hajm: int, epoxa: int, lr: float):
        w = np.zeros(X.shape[1])
        yangilanish = 0
        tarix = []
        for _ in range(epoxa):
            tartib = rng.permutation(n)
            for boshi in range(0, n, hajm):
                idx = tartib[boshi:boshi + hajm]
                xato = X[idx] @ w - y[idx]
                w = w - lr * (2 / len(idx)) * (X[idx].T @ xato)
                yangilanish += 1
            tarix.append(float(np.mean((X @ w - y) ** 2)))
        return w, tarix, yangilanish

    print(f"=== 1. Analitik yechim: MSE = {mse_aniq:.4f} ===")

    print("\n=== 2. Uch variant (5 epoxa) ===")
    for nom, hajm, lr in [("to'liq (batch)", n, 0.5), ("mini-batch 64", 64, 0.05),
                          ("stoxastik (1)", 1, 0.01)]:
        w, tarix, yang = ishga_tushir(hajm, 5, lr)
        print(f"  {nom:<15}: MSE {tarix[-1]:.4f}, "
              f"{yang:>6} yangilanish, farq {abs(tarix[-1] - mse_aniq):.5f}")

    print("\n=== 3. Epoxalar bo'yicha (mini-batch 64) ===")
    w, tarix, _ = ishga_tushir(64, 8, 0.05)
    for i, v in enumerate(tarix, 1):
        print(f"  epoxa {i}: MSE = {v:.4f}")

    print("\n=== 4. To'liq gradient bir xil natijaga sekinroq keladi ===")
    for epoxa in [1, 5, 20, 100]:
        _, t_batch, _ = ishga_tushir(n, epoxa, 0.5)
        _, t_mini, _ = ishga_tushir(64, epoxa, 0.05)
        print(f"  {epoxa:>3} epoxa: to'liq {t_batch[-1]:7.4f}, "
              f"mini-batch {t_mini[-1]:7.4f}")
    print("  ⭐ Mini-batch bir epoxada ko'p yangilanish qiladi — tezroq")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Analitik yechim: MSE = 0.9904 ===

=== 2. Uch variant (5 epoxa) ===
  to'liq (batch) : MSE 0.9904,      5 yangilanish, farq 0.00000
  mini-batch 64  : MSE 1.0073,   1565 yangilanish, farq 0.01694
  stoxastik (1)  : MSE 1.0820, 100000 yangilanish, farq 0.09159

=== 3. Epoxalar bo'yicha (mini-batch 64) ===
  epoxa 1: MSE = 0.9975
  epoxa 2: MSE = 0.9962
  epoxa 3: MSE = 0.9982
  epoxa 4: MSE = 0.9979
  epoxa 5: MSE = 0.9982
  epoxa 6: MSE = 0.9966
  epoxa 7: MSE = 0.9969
  epoxa 8: MSE = 0.9995

=== 4. To'liq gradient bir xil natijaga sekinroq keladi ===
    1 epoxa: to'liq  1.0015, mini-batch  0.9968
    5 epoxa: to'liq  0.9904, mini-batch  0.9940
   20 epoxa: to'liq  0.9904, mini-batch  0.9994
  100 epoxa: to'liq  0.9904, mini-batch  0.9991
  ⭐ Mini-batch bir epoxada ko'p yangilanish qiladi — tezroq

Nima ko'rsatdi: 2.4, 2.5-bo'limlar.

Misol 4 — SGDRegressor va oqim rejimi

python
"""sklearn bilan katta ma'lumot (real numpy/sklearn)."""

import warnings

import numpy as np
from sklearn.linear_model import LinearRegression, SGDRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(19)
    n, p = 200_000, 12
    X = rng.normal(0, 1, (n, p)) * rng.uniform(0.5, 3000, p)
    haqiqiy = rng.normal(0, 2, p)
    y = 7 + X @ haqiqiy + rng.normal(0, 3, n)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=0)

    print("=== 1. Analitik yechim (LinearRegression) ===")
    m = LinearRegression().fit(Xtr, ytr)
    print(f"  test MAE = {mean_absolute_error(yte, m.predict(Xte)):.4f}")

    print("\n=== 2. SGD standartlashtirishSIZ ===")
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        with np.errstate(all="ignore"):
            sgd = SGDRegressor(max_iter=50, tol=None, eta0=0.001, random_state=0)
            sgd.fit(Xtr, ytr)
            bashorat = sgd.predict(Xte)
    chekli = bool(np.isfinite(sgd.coef_).all())
    print(f"  koeffitsiyentlar chekli: {chekli}")
    if chekli and np.isfinite(bashorat).all():
        print(f"  test MAE = {mean_absolute_error(yte, bashorat):.4f}")
    else:
        print("  natija: TARQAB KETDI (cheksizlik) — masshtab farqi juda katta")

    print("\n=== 3. SGD standartlashtirish bilan ===")
    quvur = Pipeline([("sc", StandardScaler()),
                      ("m", SGDRegressor(max_iter=200, tol=1e-5, eta0=0.01,
                                         learning_rate="invscaling",
                                         random_state=0))]).fit(Xtr, ytr)
    print(f"  test MAE = {mean_absolute_error(yte, quvur.predict(Xte)):.4f}")
    print(f"  iteratsiyalar: {quvur.named_steps['m'].n_iter_}")

    print("\n=== 4. Oqim rejimi (partial_fit) ===")
    sc = StandardScaler().fit(Xtr[:5000])            # birinchi bo'lakda fit
    oqim = SGDRegressor(eta0=0.01, learning_rate="invscaling", random_state=0)
    hajm = 10_000
    for epoxa in range(3):
        tartib = rng.permutation(len(Xtr))
        for boshi in range(0, len(Xtr), hajm):
            idx = tartib[boshi:boshi + hajm]
            oqim.partial_fit(sc.transform(Xtr[idx]), ytr[idx])
        print(f"  epoxa {epoxa + 1}: test MAE = "
              f"{mean_absolute_error(yte, oqim.predict(sc.transform(Xte))):.4f}")
    print("  ⭐ Ma'lumot xotiraga sig'masa ham model o'qitiladi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Analitik yechim (LinearRegression) ===
  test MAE = 2.3960

=== 2. SGD standartlashtirishSIZ ===
  koeffitsiyentlar chekli: True
  test MAE = 328165686330862.9375

=== 3. SGD standartlashtirish bilan ===
  test MAE = 2.5234
  iteratsiyalar: 23

=== 4. Oqim rejimi (partial_fit) ===
  epoxa 1: test MAE = 2.5170
  epoxa 2: test MAE = 2.5387
  epoxa 3: test MAE = 2.5070
  ⭐ Ma'lumot xotiraga sig'masa ham model o'qitiladi

Nima ko'rsatdi: 2.6-bo'lim.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Gradient tushish — regressiya uchun" Asosan DL uchun
"Standartlashtirish ixtiyoriy" SGD uchun majburiy
"Katta lr — tez natija" Tarqab ketadi
"SGD aniqroq" Shovqinli, lekin tez
"Batch har doim yaxshi" Mini-batch amaliy
"Yo'qotish egri chizig'i keraksiz" Asosiy diagnostika
"SGD kichik ma'lumotda ham kerak" Kerak emas
"Gradient tushish lokal minimumga tushadi" Qavariq masalada yo'q

6. Keng tarqalgan xatolar va yechimlari

1. Standartlashtirmaslik

python
SGDRegressor().fit(X, y)                                          # ⚠️
Pipeline([("sc", StandardScaler()), ("m", SGDRegressor())])       # ✅

2. lr ni sinamaslik

python
lr = 0.5                               # tarqadi                  # ⚠️
# 0.001, 0.01, 0.1 ni sinab, egri chiziqqa qarang                 # ✅

3. Yo'qotishni kuzatmaslik

python
for _ in range(1000): w -= lr * grad                              # ⚠️
tarix.append(mse); # keyin egri chiziqni ko'ring                  # ✅

4. Kichik ma'lumotda SGD

python
SGDRegressor().fit(X, y)               # n = 500                  # ⚠️
LinearRegression().fit(X, y)                                      # ✅

5. max_iter kichik

python
SGDRegressor(max_iter=5)                                          # ⚠️
SGDRegressor(max_iter=1000, tol=1e-4)                             # ✅

6. partial_fit da aralashtirmaslik

python
for b in boblar: model.partial_fit(b.X, b.y)   # tartiblangan     # ⚠️
# har epoxada permutation bilan aralashtiring                     # ✅

7. Scaler ni har bo'lakda qayta fit qilish

python
for b in boblar: sc.fit(b.X)                                      # ⚠️
sc = StandardScaler().fit(X_birinchi)   # yoki partial_fit        # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 10.5-dars (o'tilgan): Shartlilik soni
  • 13.1-dars (o'tilgan): Analitik yechim
  • 13.7-dars: Regularizatsiya bilan gradient
  • 13.10-dars: Logistik regressiya — analitik yechim yo'q
  • 22-qism: Neyron tarmoqlar va orqaga tarqalish

8. Eng yaxshi amaliyotlar

  1. Har doim standartlashtiring.

  2. lr ni sinab tanlang.

  3. Yo'qotish egri chizig'ini chizing.

  4. Mini-batch dan boshlang.

  5. Analitik yechim bilan solishtiring.

  6. Kichik ma'lumotda SGD ishlatmang.

  7. Har epoxada aralashtiring.

  8. tol va max_iter ni qo'ying.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # gradient nima?
2.  # qadam formulasi?
3.  # lr katta bo'lsa?
4.  # lr kichik bo'lsa?
5.  # nega standartlashtirish?
6.  # epoxa nima?
7.  # mini-batch hajmi?
8.  # SGD afzalligi?
9.  # asosiy diagnostika?
10. # qavariq masalada lokal minimum bormi?
11. # partial_fit nima uchun?
12. # kichik ma'lumotda nima ishlatiladi?
Javoblar
  1. Eng tik o'sish yo'nalishi
  2. w ← w - lr·grad
  3. Tarqab ketadi
  4. Sekin
  5. Vodiy cho'zilib zigzag bo'ladi
  6. Butun ma'lumotdan bir o'tish
  7. 32-256
  8. Tezlik va xotira
  9. Yo'qotish egri chizig'i
  10. Yo'q
  11. Oqim rejimi
  12. LinearRegression / Ridge

Vazifa 2: Xatolarni tuzating

python
1.  SGDRegressor().fit(X, y)   # masshtablanmagan

2.  lr = 1.5

3.  for _ in range(5000): w -= lr * grad   # kuzatuvsiz

4.  SGDRegressor().fit(X, y)   # n = 400

5.  for b in boblar: sc.fit(b.X); model.partial_fit(sc.transform(b.X), b.y)
Javoblar
python
1.  Pipeline([("sc", StandardScaler()), ("m", SGDRegressor())])

2.  # 0.001..0.3 oralig'ida sinang

3.  tarix.append(mse)   # va egri chiziqni tekshiring

4.  LinearRegression().fit(X, y)

5.  sc = StandardScaler().fit(X_birinchi)   # bir marta

Vazifa 3: Qo'lda gradient

Modellang:

  1. Yo'qotish va gradient
  2. Turli lr
  3. Analitik bilan solishtirish
  4. Qadamlar soni

Vazifa 4: Masshtab

Modellang:

  1. Turli masshtabli belgilar
  2. Shartlilik soni
  3. Konvergensiya
  4. Standartlashtirish ta'siri

Vazifa 5: Uch variant

Modellang:

  1. Batch, SGD, mini-batch
  2. Yangilanishlar soni
  3. Epoxalar
  4. Xulosa

Vazifa 6: SGDRegressor

Modellang:

  1. Pipeline
  2. Oqim rejimi
  3. Analitik bilan solishtirish
  4. Tavsiya

Vazifa 7: O'ylash

Chiziqli regressiyada aniq analitik yechim bor, shunga qaramay zamonaviy ML deyarli butunlay gradient usullariga qurilgan. Nima uchun?

Javob

Qisqa javob: analitik yechim faqat tor holatlar uchun mavjud (kvadratik yo'qotish, chiziqli model, kichik p). Gradient tushish esa universal: har qanday differensiallanadigan yo'qotish va model uchun ishlaydi, xotira talabi kichik va parallellashtiriladi.

1. Analitik yechimning chegaralari

Chegara Izoh
Faqat kvadratik yo'qotish Logistik, Huber, kvantil uchun yo'q
(XTX)^(-1) — O(p^3) p katta bo'lsa qimmat
Butun X xotirada Milliard qator sig'maydi
Faqat chiziqli model Neyron tarmoq uchun mumkin emas

2. Gradientning universalligi

  • Har qanday differensiallanadigan yo'qotish
  • Har qanday model (avtomatik differensiallash bilan)
  • Oqim va taqsimlangan o'qitish
  • GPU da samarali (matritsa amallari)

3. Nega bu DL uchun hal qiluvchi

  • Neyron tarmoqda parametr soni millionlab
  • Yo'qotish qavariq emas — analitik yechim tushunchasi yo'q
  • Orqaga tarqalish (22-qism) — gradientni samarali hisoblash usuli

4. Chiziqli regressiyada qachon gradient

  • n juda katta yoki oqim ma'lumot
  • Onlayn o'qitish (model doimiy yangilanadi)
  • Maxsus yo'qotish (kvantil — 12.8, Huber — 13.11)

5. Xulosa

  1. Analitik yechim — tor, lekin aniq
  2. Gradient — sekinroq, lekin universal
  3. Kichik ma'lumotda analitikni oling
  4. Gradient bilimlari DL uchun asos

Nimani mustahkamlaydi: 2.1, 2.6-bo'limlar.

Xulosa

Bu darsda gradient tushishni o'rgandik.

Eng muhim uch fikr:

  1. Iterativ yaqinlashish. w ← w - lr·grad, bunda grad = -(2/n)·XT(y - Xw). Kvadratik yo'qotish qavariq bo'lgani uchun chiziqli regressiyada lokal minimum yo'q — to'g'ri lr bilan gradient tushish har doim analitik yechimga keladi (buni lstsq bilan solishtirib tekshirish mumkin).

  2. Ikki hal qiluvchi tafsilot. O'rganish tezligi: katta bo'lsa tarqab ketadi (yo'qotish o'sadi, NaN), kichik bo'lsa minglab qadam kerak — diagnostika yo'qotish egri chizig'i. Standartlashtirish: masshtab farqi yo'qotish sirtini cho'zilgan vodiyga aylantiradi va gradient zigzag qiladi; OLS uchun shart emas, SGD uchun majburiy.

  3. Mini-batch — amaliy standart. To'liq gradient aniq, lekin qimmat; stoxastik (1 namuna) tez, lekin shovqinli; 32-256 lik bo'laklar ikkalasining afzalligini birlashtiradi va bir epoxada ko'p yangilanish beradi. SGDRegressor + partial_fit ma'lumot xotiraga sig'masa ham o'qitish imkonini beradi; kichik ma'lumotda esa oddiy LinearRegression aniqroq va qulayroq.

Keyingi darsda Ridge regularizatsiyasini o'rganamiz: L2 jarima, alpha ni tanlash, multikollinearlikni davolash va bias-variance muvozanati.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
13.6-dars: Gradient tushish — IlmHamroh