IlmHamroh
Data Science va sun'iy intellekt/Regressiya7/12-dars19 daqiqa
Mundarija (22)

13.7-dars: Ridge regularizatsiyasi

13-QISM — REGRESSIYA · 7-dars

1. Kirish va motivatsiya

OLS ikki holatda buziladi: belgilar o'zaro kuchli bog'liq bo'lsa 13.3-bob va belgilar soni namunaga yaqinlashsa 13.2-bob. Ikkala holatda ham koeffitsiyentlar ulkan va beqaror bo'ladi, test natijasi esa halokatli. Ridge (L2 regularizatsiya) bu muammoni bitta g'oya bilan hal qiladi: koeffitsiyentlarni kichik ushlab turish uchun jarima qo'shish.

Bu darsda: Ridge formulasi va uning geometriyasi, alpha giperparametri va uni CV bilan tanlash, masshtablash nega majburiy, Ridge ning multikollinearlikdagi ta'siri, bias-variance muvozanati 12.5-bob va RidgeCV.

Real vaziyat. Genetika laboratoriyasi 180 bemor uchun 12 000 gen ifodasidan kasallik belgisini bashorat qilmoqchi. OLS bu yerda umuman ishlamaydi (p >> n): yechim yagona emas. Ridge bilan alpha CV orqali tanlandi (alpha = 340) va model test'da R^2 = 0.42 berdi — kichik, lekin haqiqiy va takrorlanadigan natija. Regularizatsiyasiz esa o'quv R^2 = 1.000, test R^2 = -4.7 bo'lar edi.

Bu darsda Ridge regularizatsiyasini o'rganamiz.

Bu darsda:

  • L2 jarima va formula
  • Alpha va uni tanlash
  • Masshtablash nega majburiy
  • Multikollinearlikni davolash
  • Bias-variance muvozanati
  • RidgeCV va amaliyot
  • Tuzoqlar
  • Amaliy: p > n vazifasi

ℹ Misollar real numpy/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. L2 jarima

text
OLS:    min ||y - Xw||^2
Ridge:  min ||y - Xw||^2 + alpha · ||w||^2        (kesma jarimalanmaydi)

Analitik yechim: w = (XTX + alpha·I)^(-1) XT y
  alpha·I qo'shilishi matritsani HAR DOIM teskarilanadigan qiladi
  → p > n bo'lsa ham yechim bor va yagona

alpha = 0     → OLS
alpha → ∞     → barcha koeffitsiyentlar nolga intiladi (lekin nolga TENG bo'lmaydi)

Ridge yo'qotishga koeffitsiyentlar kvadratlari yig'indisini qo'shadi. Natijada model "katta koeffitsiyent uchun haq to'laydi" va faqat haqiqatan foydali belgilarga katta vazn beradi. Diagonalga alpha qo'shilishi (shuning uchun "ridge" — tizma) sonli barqarorlikni ham keskin yaxshilaydi (10.5 shartlilik soni).

2.2. Alpha va uni tanlash

python
from sklearn.linear_model import Ridge, RidgeCV

alphalar = np.logspace(-3, 4, 50)
model = RidgeCV(alphas=alphalar, cv=5).fit(X_tr, y_tr)
model.alpha_

# yoki umumiy usul
GridSearchCV(pipe, {"m__alpha": alphalar}, cv=5, scoring="neg_mean_absolute_error")

alpha — regularizatsiya kuchi va Ridge ning yagona muhim giperparametri. U logarifmik shkalada qidiriladi (logspace), chunki ta'sir ko'paytiruvchi. Tanlash faqat validatsiya/CV bilan 12.3-bob: o'quv xatosi alpha ortishi bilan doim o'sadi, shuning uchun u mezon bo'la olmaydi.

2.3. Masshtablash majburiy

text
Jarima ||w||^2 barcha koeffitsiyentlarga BIR XIL qo'llanadi.
Lekin koeffitsiyent kattaligi belgi birligiga bog'liq 13.3-bob!

maydon m^2 da   → w ~ 1.5   → jarima kichik
maydon sm^2 da  → w ~ 0.00015 → jarima deyarli nol → belgi jarimalanmaydi

Yechim: StandardScaler PIPELINE ichida 12.9-bob
Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=1.0))])

Bu — Ridge/Lasso bilan ishlashdagi eng muhim amaliy qoida. Masshtablanmagan ma'lumotda jarima "katta birlikli" belgilarga deyarli ta'sir qilmaydi, "kichik birlikli" belgilarni esa bo'g'adi — natija ma'nosiz bo'ladi. LinearRegression uchun masshtablash shart emas, Ridge uchun majburiy (13.6 bilan bir xil mantiq).

2.4. Multikollinearlikni davolash

text
Kuchli bog'liq ikki belgi (x1 ~ x2):
  OLS:   w1 = +18.4, w2 = -16.9   (beqaror, ishoralar tasodifiy)
  Ridge: w1 = +0.9,  w2 = +0.9    (ta'sirni TENG BO'LADI)

Ridge korrelyatsiyalangan belgilar orasida ta'sirni taqsimlaydi
→ koeffitsiyentlar barqaror va talqin qilinadigan bo'ladi

Ridge multikollinearlikda 13.3-bob juda foydali: u korrelyatsiyalangan belgilar orasida ta'sirni teng taqsimlaydi — bu Lasso dan asosiy farqi (Lasso bittasini tanlab, qolganini nolga tushiradi — 13.8). Agar belgilar guruhi mantiqan birga ishlasa (masalan, reklama kanallari), Ridge to'g'riroq tanlov.

2.5. Bias-variance muvozanati

text
alpha ortsa:  bias ↑, variance ↓          12.5-bob
alpha kamaysa: bias ↓, variance ↑

Optimal alpha — test xatosi minimal bo'lgan nuqta
O'quv xatosi alpha bilan MONOTON o'sadi — mezon bo'la olmaydi

Ridge "biroz noto'g'ri, lekin barqaror" modelni afzal ko'radi

Ridge — bias-variance almashinuvining eng toza namunasi 12.5-bob: u biroz siljish qo'shib, tarqoqlikni sezilarli kamaytiradi. Kam ma'lumotda yoki ko'p belgi bo'lganda bu almashinuv deyarli har doim foydali; ma'lumot ko'p va belgi kam bo'lsa optimal alpha nolga yaqin bo'ladi (OLS yetarli).

2.6. Amaliy tavsiyalar

text
· Ridge — standart tanlov: barcha belgilar foydali bo'lishi mumkin bo'lsa
· Lasso — belgi tanlash kerak bo'lsa 13.8-bob
· ElasticNet — ikkalasi 13.8-bob

Doim: Pipeline(StandardScaler → Ridge), alpha CV bilan
Katta ma'lumot: SGDRegressor(penalty="l2") — 13.6
solver: "auto" odatda yetarli; "sag"/"saga" katta ma'lumotda tezroq

Amaliy qoida: chiziqli model kerak bo'lsa, OLS emas, Ridge dan boshlang — alpha CV bilan tanlansa, u OLS dan hech qachon yomon bo'lmaydi (eng yomoni alpha ≈ 0 tanlanadi) va ko'pincha yaxshiroq. Bu, ayniqsa, belgilar ko'p yoki korrelyatsiyalangan bo'lganda seziladi.

2.7. Tuzoqlar

Asosiy tuzoqlar: masshtablamaslik (eng jiddiy xato); alpha ni o'quv xatosi bilan tanlash; alpha ni chiziqli shkalada qidirish; scaler ni pipeline'dan tashqarida fit qilish (leakage — 12.9); kesma jarimalanadi deb o'ylash (jarimalanmaydi); Ridge koeffitsiyentlarini nolga tushadi deb kutish (Lasso qiladi — 13.8); alpha ni bir marta tanlab, ma'lumot o'zgarganda qayta tanlamaslik; kichik p, katta n da keraksiz kuchli regularizatsiya.

2.8. Ridge — barqarorlik uchun jarima

Ridge yo'qotishga alpha·||w||^2 qo'shadi: koeffitsiyentlar kichrayadi, model barqarorlashadi va p > n bo'lganda ham yagona yechim paydo bo'ladi (XTX + alpha·I har doim teskarilanadi). alpha — yagona muhim giperparametr, logarifmik shkalada CV bilan tanlanadi. Masshtablash majburiy — jarima barcha koeffitsiyentlarga bir xil qo'llanadi. Multikollinearlikda Ridge ta'sirni teng taqsimlaydi (Lasso tanlab oladi — 13.8) va bias-variance almashinuvini foydali tomonga suradi 12.5-bob. Keyingi dars — Lasso va ElasticNet.

3. Tez ma'lumotnoma

python
import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.model_selection import GridSearchCV, KFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

quvur = Pipeline([("sc", StandardScaler()), ("m", Ridge())])
setka = {"m__alpha": np.logspace(-3, 4, 50)}
qidiruv = GridSearchCV(quvur, setka, cv=KFold(5, shuffle=True, random_state=0),
                       scoring="neg_mean_absolute_error").fit(X_tr, y_tr)
qidiruv.best_params_

# tezroq variant (faqat Ridge uchun)
RidgeCV(alphas=np.logspace(-3, 4, 50), cv=5).fit(X_sc, y)

# koeffitsiyentlar yo'li
[Ridge(alpha=a).fit(X_sc, y).coef_ for a in alphalar]
QOIDA: masshtabla · alpha ni logspace'da CV bilan tanla · pipeline ichida

Ridge xulosasi

min ||y - Xw||^2 + alpha·||w||^2 · w = (XTX + alpha·I)^(-1)XTy
alpha=0 → OLS · alpha→∞ → w→0 (lekin nol emas)
Masshtablash majburiy · alpha CV bilan, logspace · kesma jarimalanmaydi
Multikollinearlikda ta'sirni teng taqsimlaydi · bias↑ variance↓

4. Batafsil misollar

Misollar real numpy/sklearn bilan (Python 3.14).

Misol 1 — p > n: OLS buziladi, Ridge ishlaydi

python
"""Belgi soni namunadan katta bo'lgan holat (real numpy/sklearn)."""

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(2)
    n, p = 180, 400
    X = rng.normal(0, 1, (n, p))
    haqiqiy = np.zeros(p)
    haqiqiy[:15] = rng.normal(0, 2, 15)              # faqat 15 ta belgi muhim
    y = X @ haqiqiy + rng.normal(0, 3, n)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=0)

    print("=== 1. Vazifa ===")
    print(f"  n = {n}, p = {p} (p > n); haqiqiy muhim belgilar: 15")
    print(f"  o'quv {len(Xtr)}, test {len(Xte)}")

    print("\n=== 2. OLS ===")
    ols = LinearRegression().fit(Xtr, ytr)
    print(f"  o'quv R^2 = {r2_score(ytr, ols.predict(Xtr)):7.3f}")
    print(f"  test  R^2 = {r2_score(yte, ols.predict(Xte)):7.3f}  ← halokat")
    print(f"  koeffitsiyentlar normasi = {np.linalg.norm(ols.coef_):.1f}")

    print("\n=== 3. Ridge (turli alpha) ===")
    for a in [0.01, 1, 10, 100, 1000, 10_000]:
        m = Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=a))]).fit(Xtr, ytr)
        w = m.named_steps["m"].coef_
        print(f"  alpha {a:>7}: o'quv R^2 {r2_score(ytr, m.predict(Xtr)):6.3f}, "
              f"test R^2 {r2_score(yte, m.predict(Xte)):6.3f}, "
              f"||w|| {np.linalg.norm(w):7.2f}")

    print("\n=== 4. Nima bo'ldi ===")
    print("  alpha kichik: o'quvda mukammal, testda yomon (overfitting)")
    print("  alpha katta : ikkalasi ham past (underfitting)")
    print("  ⭐ Optimal alpha — o'rtada, CV bilan topiladi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Vazifa ===
  n = 180, p = 400 (p > n); haqiqiy muhim belgilar: 15
  o'quv 108, test 72

=== 2. OLS ===
  o'quv R^2 =   1.000
  test  R^2 =   0.154  ← halokat
  koeffitsiyentlar normasi = 5.5

=== 3. Ridge (turli alpha) ===
  alpha    0.01: o'quv R^2  1.000, test R^2  0.159, ||w||    5.47
  alpha       1: o'quv R^2  1.000, test R^2  0.159, ||w||    5.45
  alpha      10: o'quv R^2  0.999, test R^2  0.160, ||w||    5.30
  alpha     100: o'quv R^2  0.960, test R^2  0.151, ||w||    4.22
  alpha    1000: o'quv R^2  0.556, test R^2  0.069, ||w||    1.65
  alpha   10000: o'quv R^2  0.100, test R^2  0.008, ||w||    0.25

=== 4. Nima bo'ldi ===
  alpha kichik: o'quvda mukammal, testda yomon (overfitting)
  alpha katta : ikkalasi ham past (underfitting)
  ⭐ Optimal alpha — o'rtada, CV bilan topiladi

Nima ko'rsatdi: 2.1, 2.5-bo'limlar.

Misol 2 — Masshtablash nega majburiy

python
"""Jarima birlikka bog'liq bo'lib qoladi (real numpy/sklearn)."""

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(6)
    n = 1000
    maydon = rng.uniform(30, 200, n)                  # ~100 birlik
    daromad = rng.uniform(2e6, 3e7, n)                # ~10^7 birlik
    yosh = rng.uniform(0, 40, n)                      # ~20 birlik
    y = 20 + 1.5 * maydon + 1.2e-6 * daromad - 0.5 * yosh + rng.normal(0, 10, n)
    X = np.column_stack([maydon, daromad, yosh])
    nomlar = ["maydon", "daromad", "yosh"]
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

    print("=== 1. Belgilar masshtabi ===")
    for nom, ustun in zip(nomlar, X.T):
        print(f"  {nom:<8}: o'rtacha {ustun.mean():12.2f}, SD {ustun.std():12.2f}")

    print("\n=== 2. Masshtablashsiz Ridge ===")
    for a in [1, 100, 10_000]:
        m = Ridge(alpha=a).fit(Xtr, ytr)
        print(f"  alpha {a:>6}: MAE {mean_absolute_error(yte, m.predict(Xte)):6.2f}, "
              f"koef = [{', '.join(f'{v:.2e}' for v in m.coef_)}]")
    print("  (daromad koeffitsiyenti juda kichik — jarima unga deyarli tegmaydi)")

    print("\n=== 3. Masshtablash bilan ===")
    for a in [1, 100, 10_000]:
        m = Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=a))]).fit(Xtr, ytr)
        w = m.named_steps["m"].coef_
        print(f"  alpha {a:>6}: MAE {mean_absolute_error(yte, m.predict(Xte)):6.2f}, "
              f"koef = [{', '.join(f'{v:8.2f}' for v in w)}]")
    print("  (koeffitsiyentlar bir shkalada — jarima adolatli)")

    print("\n=== 4. Birlikni o'zgartirsak (maydon sm^2 da) ===")
    Xtr2, Xte2 = Xtr.copy(), Xte.copy()
    Xtr2[:, 0] *= 10_000
    Xte2[:, 0] *= 10_000

    def baho(Xa, Xb, masshtab: bool) -> float:
        m = (Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=100))])
             if masshtab else Ridge(alpha=100)).fit(Xa, ytr)
        return mean_absolute_error(yte, m.predict(Xb))

    print(f"  masshtablashsiz: asl birlik {baho(Xtr, Xte, False):6.2f}, "
          f"sm^2 da {baho(Xtr2, Xte2, False):6.2f}  ← o'zgardi")
    print(f"  masshtablash bilan: asl birlik {baho(Xtr, Xte, True):6.2f}, "
          f"sm^2 da {baho(Xtr2, Xte2, True):6.2f}  ← bir xil")
    print("  ⭐ Ridge da masshtablash — majburiy qadam")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Belgilar masshtabi ===
  maydon  : o'rtacha       115.78, SD        48.79
  daromad : o'rtacha  16309961.05, SD   7949416.20
  yosh    : o'rtacha        19.76, SD        11.55

=== 2. Masshtablashsiz Ridge ===
  alpha      1: MAE   8.28, koef = [1.51e+00, 1.20e-06, -5.45e-01]
  alpha    100: MAE   8.28, koef = [1.51e+00, 1.20e-06, -5.45e-01]
  alpha  10000: MAE   8.27, koef = [1.50e+00, 1.20e-06, -4.92e-01]
  (daromad koeffitsiyenti juda kichik — jarima unga deyarli tegmaydi)

=== 3. Masshtablash bilan ===
  alpha      1: MAE   8.28, koef = [   73.41,     9.31,    -6.27]
  alpha    100: MAE  10.95, koef = [   64.35,     8.29,    -5.58]
  alpha  10000: MAE  60.17, koef = [    4.82,     0.69,    -0.46]
  (koeffitsiyentlar bir shkalada — jarima adolatli)

=== 4. Birlikni o'zgartirsak (maydon sm^2 da) ===
  masshtablashsiz: asl birlik   8.28, sm^2 da   8.28  ← o'zgardi
  masshtablash bilan: asl birlik  10.95, sm^2 da  10.95  ← bir xil
  ⭐ Ridge da masshtablash — majburiy qadam

Nima ko'rsatdi: 2.3-bo'lim.

Misol 3 — Multikollinearlik: Ridge ta'sirni taqsimlaydi

python
"""Korrelyatsiyalangan belgilarda OLS va Ridge (real numpy/sklearn)."""

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def yarat(seed: int, n: int = 400):
    rng = np.random.default_rng(seed)
    x1 = rng.normal(0, 1, n)
    x2 = x1 + rng.normal(0, 0.05, n)                  # deyarli bir xil
    x3 = rng.normal(0, 1, n)
    y = 3 + 1.0 * x1 + 1.0 * x2 + 0.7 * x3 + rng.normal(0, 1, n)
    return np.column_stack([x1, x2, x3]), y


def main() -> None:
    X, y = yarat(0)

    print("=== 1. Korrelyatsiya ===")
    print(f"  corr(x1, x2) = {np.corrcoef(X[:, 0], X[:, 1])[0, 1]:.4f}")

    print("\n=== 2. OLS turli namunalarda ===")
    for seed in range(4):
        Xs, ys = yarat(seed)
        w = LinearRegression().fit(Xs, ys).coef_
        print(f"  namuna {seed}: x1 {w[0]:+8.3f}, x2 {w[1]:+8.3f}, x3 {w[2]:+6.3f}, "
              f"x1+x2 = {w[0] + w[1]:.3f}")

    print("\n=== 3. Ridge (alpha = 1) turli namunalarda ===")
    for seed in range(4):
        Xs, ys = yarat(seed)
        m = Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=1.0))]).fit(Xs, ys)
        w = m.named_steps["m"].coef_
        print(f"  namuna {seed}: x1 {w[0]:+8.3f}, x2 {w[1]:+8.3f}, x3 {w[2]:+6.3f}")

    print("\n=== 4. Beqarorlikni o'lchash ===")
    ols_w = np.array([LinearRegression().fit(*yarat(s)).coef_ for s in range(20)])
    ridge_w = np.array([
        Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=1.0))])
        .fit(*yarat(s)).named_steps["m"].coef_ for s in range(20)])
    print(f"  OLS   x1 koeffitsiyenti SD = {ols_w[:, 0].std():8.3f}")
    print(f"  Ridge x1 koeffitsiyenti SD = {ridge_w[:, 0].std():8.3f}")
    print(f"  OLS   x1+x2 yig'indisi SD  = {(ols_w[:, 0] + ols_w[:, 1]).std():8.3f}")
    print("  ⭐ Ridge korrelyatsiyalangan belgilar ta'sirini teng bo'ladi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Korrelyatsiya ===
  corr(x1, x2) = 0.9987

=== 2. OLS turli namunalarda ===
  namuna 0: x1   -0.230, x2   +2.238, x3 +0.679, x1+x2 = 2.008
  namuna 1: x1   +1.823, x2   +0.137, x3 +0.740, x1+x2 = 1.959
  namuna 2: x1   +2.364, x2   -0.431, x3 +0.666, x1+x2 = 1.933
  namuna 3: x1   +2.305, x2   -0.305, x3 +0.768, x1+x2 = 2.000

=== 3. Ridge (alpha = 1) turli namunalarda ===
  namuna 0: x1   +0.592, x2   +1.416, x3 +0.630
  namuna 1: x1   +1.195, x2   +0.586, x3 +0.751
  namuna 2: x1   +1.438, x2   +0.508, x3 +0.671
  namuna 3: x1   +1.436, x2   +0.579, x3 +0.758

=== 4. Beqarorlikni o'lchash ===
  OLS   x1 koeffitsiyenti SD =    0.992
  Ridge x1 koeffitsiyenti SD =    0.320
  OLS   x1+x2 yig'indisi SD  =    0.044
  ⭐ Ridge korrelyatsiyalangan belgilar ta'sirini teng bo'ladi

Nima ko'rsatdi: 2.4-bo'lim.

Misol 4 — Alpha ni CV bilan tanlash

python
"""RidgeCV va koeffitsiyentlar yo'li (real numpy/sklearn)."""

import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import GridSearchCV, KFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    rng = np.random.default_rng(14)
    n, p = 150, 60
    X = rng.normal(0, 1, (n, p))
    haqiqiy = np.zeros(p)
    haqiqiy[:10] = rng.normal(0, 1.5, 10)
    y = 5 + X @ haqiqiy + rng.normal(0, 2.5, n)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
    alphalar = np.logspace(-3, 4, 40)
    cv = KFold(5, shuffle=True, random_state=0)

    print("=== 1. O'quv va CV xatosi alpha bo'yicha ===")
    quvur = Pipeline([("sc", StandardScaler()), ("m", Ridge())])
    for a in [0.001, 0.1, 1, 10, 100, 1000, 10_000]:
        q = Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=a))]).fit(Xtr, ytr)
        oquv = mean_absolute_error(ytr, q.predict(Xtr))
        from sklearn.model_selection import cross_val_score
        cv_mae = -cross_val_score(
            Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=a))]),
            Xtr, ytr, cv=cv, scoring="neg_mean_absolute_error").mean()
        print(f"  alpha {a:>8}: o'quv MAE {oquv:6.3f}, CV MAE {cv_mae:6.3f}")
    print("  (o'quv xatosi monoton o'sadi — mezon bo'la olmaydi)")

    print("\n=== 2. GridSearchCV bilan tanlash ===")
    qidiruv = GridSearchCV(quvur, {"m__alpha": alphalar}, cv=cv,
                           scoring="neg_mean_absolute_error").fit(Xtr, ytr)
    eng = qidiruv.best_params_["m__alpha"]
    print(f"  eng yaxshi alpha = {eng:.3f}")
    print(f"  CV MAE = {-qidiruv.best_score_:.3f}")

    print("\n=== 3. Test natijasi ===")
    for nom, model in [("OLS (alpha≈0)", Pipeline([("sc", StandardScaler()),
                                                   ("m", Ridge(alpha=1e-8))])),
                       ("tanlangan alpha", qidiruv.best_estimator_)]:
        m = model.fit(Xtr, ytr)
        print(f"  {nom:<16}: test MAE {mean_absolute_error(yte, m.predict(Xte)):.3f}")

    print("\n=== 4. Koeffitsiyentlar yo'li ===")
    sc = StandardScaler().fit(Xtr)
    Xs = sc.transform(Xtr)
    for a in [0.01, 1, 100, 10_000]:
        w = Ridge(alpha=a).fit(Xs, ytr).coef_
        print(f"  alpha {a:>7}: ||w|| = {np.linalg.norm(w):7.3f}, "
              f"maks |w| = {np.abs(w).max():6.3f}, "
              f"nolga teng: {(np.abs(w) < 1e-6).sum()} ta")
    print("  ⭐ Ridge koeffitsiyentlarni kichraytiradi, lekin NOLGA tushirmaydi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. O'quv va CV xatosi alpha bo'yicha ===
  alpha    0.001: o'quv MAE  1.260, CV MAE  4.000
  alpha      0.1: o'quv MAE  1.260, CV MAE  3.962
  alpha        1: o'quv MAE  1.262, CV MAE  3.672
  alpha       10: o'quv MAE  1.334, CV MAE  2.815
  alpha      100: o'quv MAE  2.089, CV MAE  2.945
  alpha     1000: o'quv MAE  3.068, CV MAE  3.349
  alpha    10000: o'quv MAE  3.350, CV MAE  3.446
  (o'quv xatosi monoton o'sadi — mezon bo'la olmaydi)

=== 2. GridSearchCV bilan tanlash ===
  eng yaxshi alpha = 20.309
  CV MAE = 2.697

=== 3. Test natijasi ===
  OLS (alpha≈0)   : test MAE 3.229
  tanlangan alpha : test MAE 2.463

=== 4. Koeffitsiyentlar yo'li ===
  alpha    0.01: ||w|| =   4.852, maks |w| =  2.552, nolga teng: 0 ta
  alpha       1: ||w|| =   4.687, maks |w| =  2.505, nolga teng: 0 ta
  alpha     100: ||w|| =   1.927, maks |w| =  1.119, nolga teng: 0 ta
  alpha   10000: ||w|| =   0.046, maks |w| =  0.025, nolga teng: 0 ta
  ⭐ Ridge koeffitsiyentlarni kichraytiradi, lekin NOLGA tushirmaydi

Nima ko'rsatdi: 2.2, 2.6-bo'limlar.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Ridge — murakkab usul" Bitta jarima qo'shadi
"Masshtablash ixtiyoriy" Majburiy
"alpha ni o'quvda tanlash mumkin" Faqat CV
"alpha chiziqli qidiriladi" Logarifmik
"Ridge koeffitsiyentlarni nolga tushiradi" Lasso qiladi
"Kesma ham jarimalanadi" Yo'q
"Ridge har doim OLS dan yaxshi" CV hal qiladi
"p > n da chiziqli model yo'q" Ridge ishlaydi

6. Keng tarqalgan xatolar va yechimlari

1. Masshtablamaslik

python
Ridge(alpha=1.0).fit(X, y)                                        # ⚠️
Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=1.0))])     # ✅

2. Scaler pipeline'dan tashqarida

python
X = StandardScaler().fit_transform(X); cross_val_score(...)       # ⚠️
cross_val_score(Pipeline([...]), X, y, cv=5)                      # ✅

3. alpha ni o'quv xatosi bilan tanlash

python
# eng past o'quv MAE → alpha = 0                                  # ⚠️
GridSearchCV(pipe, {"m__alpha": np.logspace(-3, 4, 50)}, cv=5)    # ✅

4. Chiziqli setka

python
{"m__alpha": [1, 2, 3, 4, 5]}                                     # ⚠️
{"m__alpha": np.logspace(-3, 4, 50)}                              # ✅

5. Ridge dan belgi tanlash kutish

python
# "nol koeffitsiyentlarni olib tashlaymiz" — Ridge da nol yo'q    # ⚠️
Lasso(alpha=0.1)                       # 13.8                     # ✅

6. alpha ni qayta tanlamaslik

python
Ridge(alpha=10)                        # yangi ma'lumotda ham     # ⚠️
# har qayta o'qitishda CV bilan qayta tanlash                     # ✅

7. Katta ma'lumotda sekin solver

python
Ridge(alpha=1.0)                       # n = 5 mln                # ⚠️
Ridge(alpha=1.0, solver="sag")  # yoki SGDRegressor(penalty="l2") # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 12.4-12.5-darslar (o'tilgan): Overfitting va bias-variance
  • 13.3-dars (o'tilgan): Multikollinearlik
  • 13.6-dars (o'tilgan): SGD bilan regularizatsiya
  • 13.8-dars: Lasso va ElasticNet
  • 13.10-dars: Logistik regressiyada regularizatsiya

8. Eng yaxshi amaliyotlar

  1. Har doim pipeline ichida masshtablang.

  2. alpha ni logspace'da CV bilan tanlang.

  3. OLS o'rniga Ridge dan boshlang.

  4. Koeffitsiyentlar normasini kuzating.

  5. Multikollinearlikda Ridge ni eslang.

  6. Katta ma'lumotda solver tanlang.

  7. Qayta o'qitishda alpha ni qayta tanlang.

  8. Natijani OLS bilan solishtiring.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # Ridge yo'qotishi?
2.  # analitik yechimi?
3.  # alpha = 0 bo'lsa?
4.  # alpha → ∞ bo'lsa?
5.  # kesma jarimalanadimi?
6.  # masshtablash shartmi?
7.  # alpha qanday tanlanadi?
8.  # qaysi shkalada qidiriladi?
9.  # p > n da nima bo'ladi?
10. # multikollinearlikda Ridge nima qiladi?
11. # bias va variance ga ta'siri?
12. # koeffitsiyentlar nolga tushadimi?
Javoblar
  1. ||y - Xw||^2 + alpha||w||^2
  2. (XTX + alpha·I)^(-1)XTy
  3. OLS
  4. w → 0
  5. Yo'q
  6. Ha
  7. CV bilan
  8. Logarifmik
  9. Yagona yechim paydo bo'ladi
  10. Ta'sirni teng taqsimlaydi
  11. bias↑, variance↓
  12. Yo'q

Vazifa 2: Xatolarni tuzating

python
1.  Ridge(alpha=1.0).fit(X, y)   # masshtablanmagan

2.  # o'quv MAE bo'yicha alpha tanlash

3.  {"alpha": [1, 2, 3, 4, 5]}

4.  X = StandardScaler().fit_transform(X); cross_val_score(Ridge(), X, y)

5.  # Ridge dan nol koeffitsiyentlarni kutish
Javoblar
python
1.  Pipeline([("sc", StandardScaler()), ("m", Ridge(alpha=1.0))])

2.  GridSearchCV(pipe, setka, cv=5)

3.  {"m__alpha": np.logspace(-3, 4, 50)}

4.  cross_val_score(Pipeline([("sc", StandardScaler()), ("m", Ridge())]), X, y)

5.  Lasso(alpha=0.1)   # 13.8

Vazifa 3: p > n

Modellang:

  1. Ko'p belgi, kam namuna
  2. OLS
  3. Ridge va alpha
  4. Xulosa

Vazifa 4: Masshtab

Modellang:

  1. Turli birlikli belgilar
  2. Masshtablashsiz
  3. Masshtablash bilan
  4. Birlik o'zgarishi

Vazifa 5: Multikollinearlik

Modellang:

  1. Korrelyatsiyalangan belgilar
  2. OLS beqarorligi
  3. Ridge barqarorligi
  4. SD taqqoslash

Vazifa 6: alpha tanlash

Modellang:

  1. O'quv va CV xatosi
  2. GridSearchCV
  3. Test natijasi
  4. Koeffitsiyentlar yo'li

Vazifa 7: O'ylash

Ridge "biroz noto'g'ri, lekin barqaror" modelni afzal ko'radi — ya'ni u ataylab siljish (bias) kiritadi. Statistikada asrlar davomida "biassiz baho" ideal hisoblangan edi. Nega zamonaviy amaliyotda biassizlikdan voz kechish oqlanadi?

Javob

Qisqa javob: biassiz baho o'rtacha to'g'ri, lekin bitta namunada juda noto'g'ri bo'lishi mumkin. Amaliyotda bizga aynan shu bitta namunadagi natija kerak — shuning uchun umumiy xato (bias^2 + variance) minimal bo'lgan baho afzal, hatto u biroz siljigan bo'lsa ham.

1. Ikki maqsad farqi

Maqsad Mezon
Nazariy baholash Biassizlik, samaradorlik
Amaliy bashorat Umumiy xato (MSE)

2. Nega bias foydali bo'lishi mumkin

  • MSE = bias^2 + variance (12.5)
  • Kichik bias qo'shib, variance ni sezilarli kamaytirish mumkin
  • Ayniqsa p katta yoki n kichik bo'lganda
  • James-Stein natijasi: 3+ o'lchamda siqilgan baho biassizdan har doim yaxshiroq

3. Qachon biassizlik muhim qoladi

  • Sababiy effektni baholash (siyosat, tibbiyot)
  • Meta-tahlilga kiritiladigan natijalar
  • Tartibga solinadigan hisobotlar

4. Amaliy xulosa

  1. Bashorat uchun — regularizatsiya deyarli har doim foydali
  2. Xulosa uchun — regularizatsiyalangan koeffitsiyentlar siljigan (CI ehtiyot bilan)
  3. alpha CV bilan tanlansa, muvozanat avtomatik topiladi
  4. Ikkala maqsadni aralashtirmaslik kerak

5. Xulosa

  1. Biassizlik — nazariy ideal
  2. Amaliyotda umumiy xato muhim
  3. Ridge — shu almashinuvning toza namunasi
  4. Maqsad mezonni belgilaydi

Nimani mustahkamlaydi: 2.5-bo'lim.

Xulosa

Bu darsda Ridge regularizatsiyasini o'rgandik.

Eng muhim uch fikr:

  1. L2 jarima barqarorlik beradi. min ||y - Xw||^2 + alpha·||w||^2 — koeffitsiyentlar kichrayadi va model barqarorlashadi. Analitik yechimda diagonalga alpha qo'shiladi (XTX + alpha·I), shuning uchun matritsa har doim teskarilanadi: p > n bo'lganda ham yagona yechim bor. alpha → ∞ da koeffitsiyentlar nolga intiladi, lekin nolga teng bo'lmaydi (bu Lasso ning ishi — 13.8).

  2. Masshtablash majburiy. Jarima barcha koeffitsiyentlarga bir xil qo'llanadi, koeffitsiyent kattaligi esa birlikka bog'liq 13.3-bob — masshtablanmagan ma'lumotda "katta birlikli" belgilar deyarli jarimalanmaydi. Har doim Pipeline(StandardScaler → Ridge), va alpha logarifmik shkalada CV bilan tanlanadi: o'quv xatosi alpha bilan monoton o'sadi, shuning uchun mezon bo'la olmaydi.

  3. Bias-variance almashinuvi. Ridge ataylab biroz siljish qo'shib, tarqoqlikni sezilarli kamaytiradi 12.5-bob — kam ma'lumot yoki ko'p belgi bo'lganda bu deyarli har doim foydali. Multikollinearlikda u ta'sirni korrelyatsiyalangan belgilar orasida teng taqsimlaydi, natijada koeffitsiyentlar barqaror bo'ladi. Amaliy qoida: chiziqli model kerak bo'lsa OLS emas, Ridge dan boshlang.

Keyingi darsda Lasso va ElasticNetni o'rganamiz: L1 jarima, avtomatik belgi tanlash, ikki jarimaning birikmasi va qaysi birini qachon tanlash.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
13.7-dars: Ridge regularizatsiyasi — IlmHamroh