Data Science va sun'iy intellekt20.10-dars: Regularizatsiya
Data Science va sun'iy intellekt · Neyron tarmoqlar

20.10-dars: Regularizatsiya

23 daqiqa10/12-dars

20-QISM — NEYRON TARMOQLAR · 10-dars


1. Kirish va motivatsiya

Tarmoq o'rganadigan bo'ldi. Endi teskari muammo paydo bo'ladi: u haddan tashqari yaxshi o'rganadi.

Neyron tarmoq yetarlicha katta bo'lsa, tasodifiy yorliqlarni ham yod ola oladi — bu 2017-yilda Zhang va hamkasblar tomonidan tajribada ko'rsatilgan. Ya'ni o'quv to'plamidagi 100% aniqlik hech narsani anglatmaydi: model naqsh topgan bo'lishi ham, shunchaki jadvalni yodlagan bo'lishi ham mumkin.

Regularizatsiya — modelni yodlashdan qaytarib, umumlashtirishga majburlaydigan usullar to'plami. Bu darsda uchta asosiysini ko'ramiz: weight decay (og'irliklarni kichik ushlash), dropout (o'rgatishda neyronlarni tasodifiy o'chirish) va erta to'xtash (validatsiya yomonlashganda to'xtash).

Muhim nuans: bu uchtasi bir xil narsani qilmaydi. weight decay og'irliklar fazosini cheklaydi, dropout modelni ortiqcha bog'liqlikdan qaytaradi, erta to'xtash esa optimallashtirish vaqtini cheklaydi. Shuning uchun ular ko'pincha birga ishlatiladi.

Real vaziyat. Jamoa modelni 500 davr o'rgatdi: o'quv aniqligi 0.999, test 0.71. Erta to'xtash qo'shildi — model 40-davrda to'xtadi, test 0.78 ga ko'tarildi. Bir qator kod, +0.07.

Bu darsda yodlashni to'xtatishni o'rganamiz.

Bu darsda:

  • Weight decay va L2
  • Dropout
  • Erta to'xtash
  • Ma'lumotni kengaytirish
  • Birgalikda ishlatish
  • Tuzoqlar
  • Amaliy: regularizatsiya tanlash

ℹ Misollar real numpy/sklearn/torch bilan (Python 3.14).


2. Nazariya — chuqur tushuntirish

2.1. Nima uchun tarmoq yodlaydi

text
SIG'IM: yetarlicha katta tarmoq ISTALGAN yorliqlarni yod oladi
  (Zhang va b., 2017: tasodifiy yorliqlarda ham 100% o'quv aniqligi)

DEMAK: o'quv balli model sifati haqida HECH NARSA demaydi

YODLASH BELGILARI:
  o'quv loss ~0, validatsiya loss O'SMOQDA
  o'quv aniqlik 0.99+, test 0.70
  og'irliklar normasi doimiy O'SIB boradi
  bashoratlar juda ishonchli (p ~ 0.999)

REGULARIZATSIYA UCHTA YO'L BILAN ISHLAYDI:
  1. parametrlar fazosini cheklash  (weight decay)
  2. modelga shovqin qo'shish       (dropout, augmentatsiya)
  3. optimallashtirishni cheklash   (erta to'xtash)

O'quv balli hech narsani isbotlamaydi — faqat validatsiya balli ma'lumot beradi.

2.2. Weight decay va L2

text
L2 JARIMASI:
  L_umumiy = L_ma'lumot + (lambda/2) * sum(w^2)
  gradient: dL/dw + lambda * w

WEIGHT DECAY (to'g'ridan-to'g'ri):
  w <- w - lr*dL/dw - lr*lambda*w
     = (1 - lr*lambda) * w - lr*dL/dw

  har qadamda og'irlik (1 - lr*lambda) ga QISQARADI

SGD DA: L2 va weight decay AYNAN bir xil
ADAM DA: BIR XIL EMAS!
  L2 jarimasi adaptiv masshtablashdan o'tadi va buziladi
  -> AdamW (Loshchilov & Hutter 2017) decay ni ALOHIDA qo'llaydi

BIAS GA ODATDA QO'LLANMAYDI:
  bias modelning sig'imini oshirmaydi, faqat siljitadi

TIPIK QIYMATLAR: 1e-5 .. 1e-2

Adam bilan L2 emas, AdamW ishlating — bu nazariy nozik emas, amaliy farq.

2.3. Dropout

text
O'RGATISHDA:
  har neyron p ehtimollik bilan NOLGA aylantiriladi
  qolganlari 1/(1-p) ga ko'paytiriladi   <- "inverted dropout"

INFERENCE DA:
  hech narsa o'chirilmaydi, masshtablash ham yo'q
  (chunki o'rgatishda allaqachon masshtablangan)

NIMA UCHUN ISHLAYDI (ikki qarash):
  1. ANSAMBL: har qadamda BOSHQA tarmoq o'rgatiladi,
     inference da ularning o'rtachasi olinadi
  2. HAMKORLIKNI BUZISH: neyron qo'shnisiga TAYANA OLMAYDI,
     chunki u o'chib qolishi mumkin -> mustaqil xususiyat o'rganadi

TIPIK p:
  yashirin qatlam: 0.2 - 0.5
  kirish qatlami:  0.1 - 0.2
  chiqish:         QO'YILMAYDI

DIQQAT: BatchNorm bilan birga ishlatish muammoli
        (o'rgatish/inference statistikasi farq qiladi)

Dropout faqat o'rgatishda faol — model.eval() ni unutish klassik xato.

2.4. Erta to'xtash

text
ALGORITM:
  har davr oxirida validatsiya ballini o'lchang
  eng yaxshi ball va uning og'irliklarini SAQLANG
  N davr davomida yaxshilanmasa - TO'XTANG
  ENG YAXShI og'irliklarni tiklang

PARAMETRLAR:
  patience (sabr): 5-20 davr
  min_delta: sezilarli yaxshilanish chegarasi

NIMA UCHUN ISHLAYDI:
  gradient tushish avval UMUMIY naqshlarni o'rganadi,
  keyin detallarni (shovqinni) yodlaydi
  -> vaqtni cheklash = sig'imni cheklash

AFZALLIGI: BEPUL (qo'shimcha hisob yo'q)
KAMCHILIGI: validatsiya to'plami kerak, "sabr" ni tanlash kerak

⭐ Eng yaxshi og'irliklarni TIKLASHNI unutmang -
   aks holda siz eng yomon nuqtada to'xtaysiz

Erta to'xtash — eng arzon regularizatsiya; deyarli har doim qo'shish kerak.

2.5. Boshqa usullar

text
MA'LUMOTNI KENGAYTIRISH (augmentation):
  rasm: burish, kesish, rang o'zgartirish (24-qism)
  tabular: shovqin qo'shish, SMOTE, mixup
  eng kuchli regularizatsiya - CHUNKI U MA'LUMOT QO'SHADI

LABEL SMOOTHING:
  y = 1 o'rniga y = 0.9, y = 0 o'rniga 0.1/(K-1)
  modelni haddan ishonchli bo'lishdan qaytaradi

BATCHNORM: yon ta'siri sifatida regularizatsiya qiladi (batch shovqini)

ANSAMBL: bir necha modelni o'rtachalash
  eng ishonchli, lekin eng qimmat

MODEL HAJMINI KAMAYTIRISH:
  eng oddiy va ko'pincha eng samarali usul

Eng yaxshi regularizatsiya — ko'proq ma'lumot; qolganlari uning o'rnini qisman bosadi.

2.6. Birgalikda ishlatish

text
TIPIK KOMBINATSIYA:
  erta to'xtash          (har doim)
  + weight decay 1e-4    (deyarli har doim)
  + dropout 0.2-0.5      (katta modelda)
  + augmentatsiya        (rasm/matnda)

TANLASH TARTIBI:
  1. erta to'xtash qo'shing (bepul)
  2. model hajmini tekshiring (kichraytirish arzon)
  3. weight decay ni sozlang
  4. hali yodlayotgan bo'lsa - dropout
  5. augmentatsiya (agar domen imkon bersa)

DIQQAT: HAMMASINI birdan qo'shmang
  har birining ta'sirini ALOHIDA o'lchang
  aks holda qaysi biri ishlaganini bilmaysiz

Bir qadamda bitta regularizatsiya — aks holda ta'sirini ajratib bo'lmaydi.

2.7. Tuzoqlar

Asosiy tuzoqlar: model.eval() ni unutish (dropout inference da faol qoladi); Adam ga L2 qo'shish (AdamW kerak); erta to'xtashda eng yaxshi og'irliklarni tiklamaslik; bias ga weight decay qo'llash; dropout ni chiqish qatlamiga qo'yish; hamma regularizatsiyani birdan yoqish; dropout va BatchNorm ni ketma-ket qo'yish; validatsiya to'plamini test bilan chalkashtirish.


3. Tez ma'lumotnoma

python
import numpy as np

# dropout (inverted, o'rgatishda)
maska = (rng.random(A.shape) > p) / (1 - p)
A = A * maska
# inference da: A o'zgarishsiz

# weight decay (SGD)
W -= lr * (dW + lam * W)
# yoki: W *= (1 - lr * lam); W -= lr * dW

# erta to'xtash
eng_yaxshi, sabr_hisobi = -np.inf, 0
for davr in range(davrlar):
    ...
    ball = baholash(X_val, y_val)
    if ball > eng_yaxshi + 1e-4:
        eng_yaxshi, eng_params, sabr_hisobi = ball, nusxa(), 0
    else:
        sabr_hisobi += 1
        if sabr_hisobi >= sabr:
            break
tikla(eng_params)

# torch
torch.optim.AdamW(params, lr=1e-3, weight_decay=1e-4)
torch.nn.Dropout(p=0.3)
model.train() / model.eval()

Regularizatsiya xulosasi

erta to'xtash  - bepul, har doim
weight decay   - AdamW bilan
dropout        - katta modelda, 0.2-0.5
augmentatsiya  - eng kuchli, domenga bog'liq
bir qadamda BITTA o'zgarish

4. Batafsil misollar

Misollar real numpy/sklearn/torch bilan (Python 3.14).

Misol 1 — Yodlash qanday ko'rinadi

python
"""Model tasodifiy yorliqlarni ham yod oladi (real sklearn)."""

import warnings

import numpy as np
from sklearn.datasets import make_classification
from sklearn.exceptions import ConvergenceWarning
from sklearn.metrics import accuracy_score, log_loss
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler


def orgat(Xtr, ytr, **kw):
    with warnings.catch_warnings():
        warnings.simplefilter("ignore", ConvergenceWarning)
        return MLPClassifier(random_state=0, early_stopping=False,
                             **kw).fit(Xtr, ytr)


def main() -> None:
    X, y = make_classification(n_samples=800, n_features=20,
                               n_informative=8, n_redundant=4,
                               n_classes=2, flip_y=0.15, class_sep=0.8,
                               random_state=0)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.35,
                                          random_state=0, stratify=y)
    sc = StandardScaler().fit(Xtr)
    Xtr, Xte = sc.transform(Xtr), sc.transform(Xte)
    print(f"  o'quv: {len(ytr)}, test: {len(yte)}")

    print("\n=== 1. Tasodifiy yorliqlarni yod olish ===")
    rng = np.random.default_rng(0)
    y_tasodif = rng.integers(0, 2, len(ytr))
    m = orgat(Xtr, y_tasodif, hidden_layer_sizes=(256, 256),
              max_iter=2000, alpha=1e-8, learning_rate_init=0.01)
    print(f"  yorliqlar TASODIFIY (naqsh YO'Q)")
    print(f"  o'quv aniqligi: "
          f"{accuracy_score(y_tasodif, m.predict(Xtr)):.4f}")
    print(f"  test aniqligi:  "
          f"{accuracy_score(yte, m.predict(Xte)):.4f}")
    print("  ⭐ O'quv aniqligi 1.0 - lekin model HECH NARSA bilmaydi")

    print("\n=== 2. Haqiqiy yorliqlarda ===")
    print(f"  {'max_iter':>9} {'o_quv':>8} {'test':>8} {'farq':>8} "
          f"{'o_quv loss':>11}")
    for iters in [10, 50, 200, 1000, 3000]:
        m = orgat(Xtr, ytr, hidden_layer_sizes=(256, 256),
                  max_iter=iters, alpha=1e-8, learning_rate_init=0.01)
        a_tr = accuracy_score(ytr, m.predict(Xtr))
        a_te = accuracy_score(yte, m.predict(Xte))
        print(f"  {iters:>9} {a_tr:>8.4f} {a_te:>8.4f} "
              f"{a_tr - a_te:>8.4f} {m.loss_:>11.6f}")
    print("  o'quv o'sadi, test to'xtaydi - YODLASH boshlandi")

    print("\n=== 3. Og'irliklar normasi o'sadi ===")
    print(f"  {'max_iter':>9} {'||W|| jami':>12} {'maksimal |w|':>14}")
    for iters in [10, 50, 200, 1000, 3000]:
        m = orgat(Xtr, ytr, hidden_layer_sizes=(256, 256),
                  max_iter=iters, alpha=1e-8, learning_rate_init=0.01)
        norma = float(np.sqrt(sum((W ** 2).sum() for W in m.coefs_)))
        eng = float(max(np.abs(W).max() for W in m.coefs_))
        print(f"  {iters:>9} {norma:>12.4f} {eng:>14.4f}")
    print("  yodlash og'irliklarni O'STIRADI")

    print("\n=== 4. Bashoratlar haddan ishonchli ===")
    print(f"  {'max_iter':>9} {'o_rt max p':>12} {'p>0.99 ulushi':>15} "
          f"{'test log_loss':>15}")
    for iters in [10, 200, 3000]:
        m = orgat(Xtr, ytr, hidden_layer_sizes=(256, 256),
                  max_iter=iters, alpha=1e-8, learning_rate_init=0.01)
        p = m.predict_proba(Xte)
        eng = p.max(axis=1)
        print(f"  {iters:>9} {eng.mean():>12.4f} "
              f"{(eng > 0.99).mean():>15.1%} "
              f"{log_loss(yte, p):>15.4f}")
    print("  log_loss aniqlikdan OLDIN yomonlashadi")

    print("\n=== 5. Model hajmi va yodlash ===")
    print(f"  {'shakl':<16} {'o_quv':>8} {'test':>8} {'farq':>8}")
    for yashirin in [(8,), (32,), (128,), (256, 256)]:
        m = orgat(Xtr, ytr, hidden_layer_sizes=yashirin, max_iter=2000,
                  alpha=1e-8, learning_rate_init=0.01)
        a_tr = accuracy_score(ytr, m.predict(Xtr))
        a_te = accuracy_score(yte, m.predict(Xte))
        print(f"  {str(yashirin):<16} {a_tr:>8.4f} {a_te:>8.4f} "
              f"{a_tr - a_te:>8.4f}")
    print("  kichik model yodlay OLMAYDI - bu ham regularizatsiya")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
  o'quv: 520, test: 280

=== 1. Tasodifiy yorliqlarni yod olish ===
  yorliqlar TASODIFIY (naqsh YO'Q)
  o'quv aniqligi: 1.0000
  test aniqligi:  0.5214
  ⭐ O'quv aniqligi 1.0 - lekin model HECH NARSA bilmaydi

=== 2. Haqiqiy yorliqlarda ===
   max_iter    o_quv     test     farq  o_quv loss
         10   0.9558   0.7679   0.1879    0.135607
         50   1.0000   0.7679   0.2321    0.000298
        200   1.0000   0.7679   0.2321    0.000298
       1000   1.0000   0.7679   0.2321    0.000298
       3000   1.0000   0.7679   0.2321    0.000298
  o'quv o'sadi, test to'xtaydi - YODLASH boshlandi

=== 3. Og'irliklar normasi o'sadi ===
   max_iter   ||W|| jami   maksimal |w|
         10      22.8154         0.3108
         50      28.7659         0.6206
        200      28.7659         0.6206
       1000      28.7659         0.6206
       3000      28.7659         0.6206
  yodlash og'irliklarni O'STIRADI

=== 4. Bashoratlar haddan ishonchli ===
   max_iter   o_rt max p   p>0.99 ulushi   test log_loss
         10       0.9003           41.8%          0.6829
        200       0.9617           74.3%          1.6258
       3000       0.9617           74.3%          1.6258
  log_loss aniqlikdan OLDIN yomonlashadi

=== 5. Model hajmi va yodlash ===
  shakl               o_quv     test     farq
  (8,)               0.9404   0.7714   0.1690
  (32,)              1.0000   0.7357   0.2643
  (128,)             1.0000   0.7429   0.2571
  (256, 256)         1.0000   0.7679   0.2321
  kichik model yodlay OLMAYDI - bu ham regularizatsiya

Nima ko'rsatdi: 2.1-bo'lim.

Misol 2 — Weight decay

python
"""L2 jarimasi va weight decay: qanday ishlaydi (real numpy)."""

import numpy as np


def main() -> None:
    print("=== 1. Ikki ko'rinish bir xilmi (SGD da) ===")
    lr, lam = 0.1, 0.05
    w1 = np.array([1.0, -2.0, 0.5])
    w2 = w1.copy()
    dW = np.array([0.3, 0.1, -0.2])
    w1 = w1 - lr * (dW + lam * w1)              # L2 gradienti
    w2 = (1 - lr * lam) * w2 - lr * dW          # weight decay
    print(f"  L2 gradienti orqali:  {w1.round(8)}")
    print(f"  weight decay orqali:  {w2.round(8)}")
    print(f"  bir xilmi: {np.allclose(w1, w2)}")

    print("\n=== 2. Qisqarish tezligi ===")
    print(f"  gradient BO'LMASA og'irlik qanday so'nadi:")
    print(f"  {'lambda':>9} {'har qadam koef':>16} "
          f"{'100 qadamdan keyin':>20}")
    for lam in [0.0, 1e-4, 1e-3, 1e-2, 1e-1]:
        koef = 1 - 0.1 * lam
        print(f"  {lam:>9.0e} {koef:>16.6f} {koef ** 100:>20.6f}")

    print("\n=== 3. Regressiyada ta'siri ===")
    rng = np.random.default_rng(0)
    n, d = 60, 40                      # ataylab n < d ga yaqin
    X = rng.normal(0, 1, (n, d))
    w_haq = np.zeros(d)
    w_haq[:5] = [3.0, -2.0, 1.5, 2.5, -1.0]
    y = X @ w_haq + rng.normal(0, 0.5, n)
    Xte = rng.normal(0, 1, (400, d))
    yte = Xte @ w_haq + rng.normal(0, 0.5, 400)
    print(f"  n = {n}, d = {d} (yodlash uchun qulay)")
    print(f"  {'lambda':>9} {'o_quv MSE':>11} {'test MSE':>11} "
          f"{'||w||':>10}")
    for lam in [0.0, 1e-3, 1e-2, 1e-1, 1.0]:
        w = np.zeros(d)
        for _ in range(3000):
            g = 2 * X.T @ (X @ w - y) / n + lam * w
            w = w - 0.01 * g
        print(f"  {lam:>9.0e} {np.mean((X @ w - y) ** 2):>11.4f} "
              f"{np.mean((Xte @ w - yte) ** 2):>11.4f} "
              f"{np.linalg.norm(w):>10.4f}")
    print("  lambda o'sganda o'quv yomonlashadi, test YAXSHILANADI")

    print("\n=== 4. Bias ga qo'llanmaydi ===")
    print("  bias modelning SIG'IMINI oshirmaydi, faqat siljitadi")
    y_siljigan = y + 50.0
    print(f"  {'holat':<26} {'test MSE':>11} {'b':>9}")
    for nom, bias_decay in [("bias ga decay YO'Q", False),
                            ("bias ga decay BOR", True)]:
        w, b = np.zeros(d), 0.0
        for _ in range(3000):
            xato = X @ w + b - y_siljigan
            w = w - 0.01 * (2 * X.T @ xato / n + 0.1 * w)
            gb = 2 * xato.mean()
            b = b - 0.01 * (gb + (0.1 * b if bias_decay else 0.0))
        print(f"  {nom:<26} "
              f"{np.mean((Xte @ w + b - (yte + 50.0)) ** 2):>11.4f} "
              f"{b:>9.4f}")
    print("  decay bias ni 50 dan pastga tortadi - ZARAR")

    print("\n=== 5. Adam bilan L2 nima uchun buziladi ===")
    print("  Adam qadami: lr * m_hat / (sqrt(s_hat) + eps)")
    print("  L2 gradienti (lam*w) shu BO'LINISHDAN o'tadi")
    print("  -> katta gradientli parametrda decay KUCHSIZ bo'ladi")
    print(f"  {'|gradient|':>12} {'L2 ning haqiqiy ta_siri':>26}")
    for g_kat in [0.001, 0.01, 0.1, 1.0]:
        tasir = 0.1 * 1.0 / (np.sqrt(g_kat ** 2) + 1e-8)
        print(f"  {g_kat:>12.3f} {tasir:>26.4f}")
    print("  AdamW decay ni bo'linishdan TASHQARIDA qo'llaydi")
    print("  ⭐ Adam ishlatsangiz AdamW ni tanlang")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ikki ko'rinish bir xilmi (SGD da) ===
  L2 gradienti orqali:  [ 0.965  -2.      0.5175]
  weight decay orqali:  [ 0.965  -2.      0.5175]
  bir xilmi: True

=== 2. Qisqarish tezligi ===
  gradient BO'LMASA og'irlik qanday so'nadi:
     lambda   har qadam koef   100 qadamdan keyin
      0e+00         1.000000             1.000000
      1e-04         0.999990             0.999000
      1e-03         0.999900             0.990049
      1e-02         0.999000             0.904792
      1e-01         0.990000             0.366032

=== 3. Regressiyada ta'siri ===
  n = 60, d = 40 (yodlash uchun qulay)
     lambda   o_quv MSE    test MSE      ||w||
      0e+00      0.0742      1.3719     4.4725
      1e-03      0.0742      1.3813     4.4628
      1e-02      0.0765      1.4744     4.3795
      1e-01      0.1960      2.7372     3.7902
      1e+00      1.9860     10.1018     2.2205
  lambda o'sganda o'quv yomonlashadi, test YAXSHILANADI

=== 4. Bias ga qo'llanmaydi ===
  bias modelning SIG'IMINI oshirmaydi, faqat siljitadi
  holat                         test MSE         b
  bias ga decay YO'Q              2.7944   50.0827
  bias ga decay BOR              62.5278   44.2951
  decay bias ni 50 dan pastga tortadi - ZARAR

=== 5. Adam bilan L2 nima uchun buziladi ===
  Adam qadami: lr * m_hat / (sqrt(s_hat) + eps)
  L2 gradienti (lam*w) shu BO'LINISHDAN o'tadi
  -> katta gradientli parametrda decay KUCHSIZ bo'ladi
    |gradient|    L2 ning haqiqiy ta_siri
         0.001                    99.9990
         0.010                    10.0000
         0.100                     1.0000
         1.000                     0.1000
  AdamW decay ni bo'linishdan TASHQARIDA qo'llaydi
  ⭐ Adam ishlatsangiz AdamW ni tanlang

Nima ko'rsatdi: 2.2-bo'lim.

Misol 3 — Dropout

python
"""Dropout ning mexanizmi va ta'siri (real numpy/torch)."""

import numpy as np
import torch


def dropout_oquv(A, p, rng):
    maska = (rng.random(A.shape) > p) / (1 - p)
    return A * maska


def main() -> None:
    rng = np.random.default_rng(0)

    print("=== 1. Inverted dropout ===")
    A = np.ones((4, 8))
    p = 0.5
    D = dropout_oquv(A, p, rng)
    print(f"  kirish: hammasi 1.0, shakl {A.shape}, p = {p}")
    print(f"  chiqish (birinchi qator): {D[0]}")
    print(f"  nol ulushi: {(D == 0).mean():.2%} (kutilgan {p:.0%})")
    print(f"  nolmas qiymat: {D[D > 0][0]:.1f} (= 1/(1-p))")

    print("\n=== 2. O'rtacha saqlanadi ===")
    A = rng.normal(5.0, 2.0, (2000, 50))
    print(f"  {'p':>6} {'o_rtacha':>10} {'std':>9} {'nol %':>8}")
    print(f"  {'yo_q':>6} {A.mean():>10.4f} {A.std():>9.4f} "
          f"{0.0:>8.1%}")
    for p in [0.1, 0.3, 0.5, 0.8]:
        D = dropout_oquv(A, p, rng)
        print(f"  {p:>6.1f} {D.mean():>10.4f} {D.std():>9.4f} "
              f"{(D == 0).mean():>8.1%}")
    print("  o'rtacha saqlanadi, DISPERSIYA oshadi (bu shovqin)")

    print("\n=== 3. O'rgatish va inference farqi ===")
    torch.manual_seed(0)
    qatlam = torch.nn.Dropout(p=0.5)
    x = torch.ones(2, 6)
    qatlam.train()
    print(f"  train() rejimida: {qatlam(x)[0].tolist()}")
    qatlam.eval()
    print(f"  eval()  rejimida: {qatlam(x)[0].tolist()}")
    print("  ⭐ eval() ni unutsangiz bashoratlar TASODIFIY bo'ladi")

    print("\n=== 4. eval() ni unutishning narxi ===")
    torch.manual_seed(0)
    model = torch.nn.Sequential(
        torch.nn.Linear(10, 64), torch.nn.ReLU(),
        torch.nn.Dropout(0.5),
        torch.nn.Linear(64, 3))
    x = torch.randn(8, 10)
    model.train()
    yurishlar = torch.stack([model(x) for _ in range(20)])
    model.eval()
    with torch.no_grad():
        barqaror = model(x)
    print(f"  train() rejimida 20 yurish:")
    print(f"    chiqish std (yurishlar bo'ylab): "
          f"{yurishlar.std(dim=0).mean().item():.4f}")
    print(f"  eval() rejimida 2 yurish bir xilmi: "
          f"{torch.allclose(barqaror, model(x))}")

    print("\n=== 5. Dropout ansambl sifatida ===")
    torch.manual_seed(0)
    model.train()
    with torch.no_grad():
        namunalar = torch.stack([model(x) for _ in range(500)])
        ortacha = namunalar.mean(dim=0)
    model.eval()
    with torch.no_grad():
        eval_chiqish = model(x)
    farq = (ortacha - eval_chiqish).abs().mean().item()
    print(f"  500 ta dropout yurishining o'rtachasi va eval()")
    print(f"  o'rtacha absolyut farq: {farq:.4f}")
    print(f"  chiqish kattaligi: {eval_chiqish.abs().mean().item():.4f}")
    print("  eval() rejimi ansambl o'rtachasiga YAQIN")

    print("\n=== 6. Neyronlar hamkorligi buziladi ===")
    print("  dropout siz neyronlar bir-biriga TAYANADI")
    A_toza = rng.normal(0, 1, (1000, 6))
    A_toza[:, 1] = A_toza[:, 0] + rng.normal(0, 0.05, 1000)
    print(f"  {'holat':<16} {'0 va 1 neyron korrelyatsiyasi':>32}")
    print(f"  {'dropout siz':<16} "
          f"{np.corrcoef(A_toza[:, 0], A_toza[:, 1])[0, 1]:>32.4f}")
    D = dropout_oquv(A_toza, 0.5, rng)
    print(f"  {'dropout 0.5':<16} "
          f"{np.corrcoef(D[:, 0], D[:, 1])[0, 1]:>32.4f}")
    print("  dropout bog'liqlikni buzadi - har neyron mustaqil bo'ladi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Inverted dropout ===
  kirish: hammasi 1.0, shakl (4, 8), p = 0.5
  chiqish (birinchi qator): [2. 0. 0. 0. 2. 2. 2. 2.]
  nol ulushi: 37.50% (kutilgan 50%)
  nolmas qiymat: 2.0 (= 1/(1-p))

=== 2. O'rtacha saqlanadi ===
       p   o_rtacha       std    nol %
    yo_q     4.9981    2.0003     0.0%
     0.1     4.9995    2.6837    10.0%
     0.3     4.9907    4.0493    30.0%
     0.5     5.0207    5.7504    49.8%
     0.8     5.0250   10.9983    79.9%
  o'rtacha saqlanadi, DISPERSIYA oshadi (bu shovqin)

=== 3. O'rgatish va inference farqi ===
  train() rejimida: [0.0, 0.0, 2.0, 0.0, 0.0, 0.0]
  eval()  rejimida: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
  ⭐ eval() ni unutsangiz bashoratlar TASODIFIY bo'ladi

=== 4. eval() ni unutishning narxi ===
  train() rejimida 20 yurish:
    chiqish std (yurishlar bo'ylab): 0.2526
  eval() rejimida 2 yurish bir xilmi: True

=== 5. Dropout ansambl sifatida ===
  500 ta dropout yurishining o'rtachasi va eval()
  o'rtacha absolyut farq: 0.0087
  chiqish kattaligi: 0.2374
  eval() rejimi ansambl o'rtachasiga YAQIN

=== 6. Neyronlar hamkorligi buziladi ===
  dropout siz neyronlar bir-biriga TAYANADI
  holat               0 va 1 neyron korrelyatsiyasi
  dropout siz                                0.9987
  dropout 0.5                                0.4858
  dropout bog'liqlikni buzadi - har neyron mustaqil bo'ladi

Nima ko'rsatdi: 2.3-bo'lim.

Misol 4 — Erta to'xtash va kombinatsiya

python
"""Erta to'xtash, weight decay va dropout birga (real torch)."""

import numpy as np
import torch
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler


def tarmoq_yasa(dropout=0.0, seed=0):
    torch.manual_seed(seed)
    qatlamlar = [torch.nn.Linear(20, 256), torch.nn.ReLU()]
    if dropout > 0:
        qatlamlar.append(torch.nn.Dropout(dropout))
    qatlamlar += [torch.nn.Linear(256, 128), torch.nn.ReLU()]
    if dropout > 0:
        qatlamlar.append(torch.nn.Dropout(dropout))
    qatlamlar.append(torch.nn.Linear(128, 2))
    return torch.nn.Sequential(*qatlamlar)


def baho(model, X, y):
    model.eval()
    with torch.no_grad():
        chiqish = model(X)
        loss = torch.nn.functional.cross_entropy(chiqish, y).item()
        aniq = (chiqish.argmax(1) == y).float().mean().item()
    return loss, aniq


def orgat(model, Xtr, ytr, Xva, yva, davrlar=300, wd=0.0,
          erta=False, sabr=15):
    opt = torch.optim.AdamW(model.parameters(), lr=0.003,
                            weight_decay=wd)
    eng_ball, eng_holat, eng_davr, hisob = -1.0, None, 0, 0
    for davr in range(1, davrlar + 1):
        model.train()
        opt.zero_grad()
        torch.nn.functional.cross_entropy(model(Xtr), ytr).backward()
        opt.step()
        if erta:
            _, ball = baho(model, Xva, yva)
            if ball > eng_ball + 1e-4:
                eng_ball, eng_davr, hisob = ball, davr, 0
                eng_holat = {k: v.clone()
                             for k, v in model.state_dict().items()}
            else:
                hisob += 1
                if hisob >= sabr:
                    break
    if erta and eng_holat is not None:
        model.load_state_dict(eng_holat)
        return davr, eng_davr
    return davrlar, davrlar


def main() -> None:
    X, y = make_classification(n_samples=1200, n_features=20,
                               n_informative=8, n_redundant=4,
                               n_classes=2, flip_y=0.15, class_sep=0.8,
                               random_state=0)
    Xtr, Xrest, ytr, yrest = train_test_split(X, y, test_size=0.5,
                                              random_state=0, stratify=y)
    Xva, Xte, yva, yte = train_test_split(Xrest, yrest, test_size=0.5,
                                          random_state=0, stratify=yrest)
    sc = StandardScaler().fit(Xtr)
    T = lambda M: torch.tensor(sc.transform(M), dtype=torch.float32)
    Xtr_t, Xva_t, Xte_t = T(Xtr), T(Xva), T(Xte)
    ytr_t = torch.tensor(ytr, dtype=torch.int64)
    yva_t = torch.tensor(yva, dtype=torch.int64)
    yte_t = torch.tensor(yte, dtype=torch.int64)
    print(f"  o'quv {len(ytr)}, validatsiya {len(yva)}, test {len(yte)}")

    print("\n=== 1. Regularizatsiyasiz ===")
    m = tarmoq_yasa()
    orgat(m, Xtr_t, ytr_t, Xva_t, yva_t)
    l_tr, a_tr = baho(m, Xtr_t, ytr_t)
    l_te, a_te = baho(m, Xte_t, yte_t)
    print(f"  {'to_plam':<12} {'loss':>9} {'aniqlik':>9}")
    print(f"  {'o_quv':<12} {l_tr:>9.4f} {a_tr:>9.4f}")
    print(f"  {'test':<12} {l_te:>9.4f} {a_te:>9.4f}")
    print(f"  farq: {a_tr - a_te:+.4f}")
    asos = a_te

    print("\n=== 2. Erta to'xtash ===")
    m = tarmoq_yasa()
    tugagan, eng_davr = orgat(m, Xtr_t, ytr_t, Xva_t, yva_t, erta=True)
    l_tr, a_tr = baho(m, Xtr_t, ytr_t)
    l_te, a_te = baho(m, Xte_t, yte_t)
    print(f"  {tugagan}-davrda to'xtadi, eng yaxshisi {eng_davr}-davr")
    print(f"  o'quv {a_tr:.4f}, test {a_te:.4f}, "
          f"farq {a_tr - a_te:+.4f}")
    print(f"  asosdan: {a_te - asos:+.4f}")

    print("\n=== 3. Weight decay ===")
    print(f"  {'wd':>9} {'o_quv':>8} {'test':>8} {'farq':>8} "
          f"{'asosdan':>9}")
    for wd in [0.0, 1e-4, 1e-2, 1e-1, 1.0]:
        m = tarmoq_yasa()
        orgat(m, Xtr_t, ytr_t, Xva_t, yva_t, wd=wd)
        _, a_tr = baho(m, Xtr_t, ytr_t)
        _, a_te = baho(m, Xte_t, yte_t)
        print(f"  {wd:>9.0e} {a_tr:>8.4f} {a_te:>8.4f} "
              f"{a_tr - a_te:>8.4f} {a_te - asos:>+9.4f}")

    print("\n=== 4. Dropout ===")
    print(f"  {'p':>6} {'o_quv':>8} {'test':>8} {'farq':>8} "
          f"{'asosdan':>9}")
    for p in [0.0, 0.2, 0.4, 0.6]:
        m = tarmoq_yasa(dropout=p)
        orgat(m, Xtr_t, ytr_t, Xva_t, yva_t)
        _, a_tr = baho(m, Xtr_t, ytr_t)
        _, a_te = baho(m, Xte_t, yte_t)
        print(f"  {p:>6.1f} {a_tr:>8.4f} {a_te:>8.4f} "
              f"{a_tr - a_te:>8.4f} {a_te - asos:>+9.4f}")

    print("\n=== 5. Kombinatsiya ===")
    variantlar = [
        ("hech narsa", 0.0, 0.0, False),
        ("erta to'xtash", 0.0, 0.0, True),
        ("wd 1e-2", 1e-2, 0.0, False),
        ("dropout 0.3", 0.0, 0.3, False),
        ("erta + wd", 1e-2, 0.0, True),
        ("erta + wd + drop", 1e-2, 0.3, True),
    ]
    print(f"  {'variant':<20} {'o_quv':>8} {'test':>8} {'farq':>8} "
          f"{'asosdan':>9}")
    for nom, wd, p, erta in variantlar:
        m = tarmoq_yasa(dropout=p)
        orgat(m, Xtr_t, ytr_t, Xva_t, yva_t, wd=wd, erta=erta)
        _, a_tr = baho(m, Xtr_t, ytr_t)
        _, a_te = baho(m, Xte_t, yte_t)
        print(f"  {nom:<20} {a_tr:>8.4f} {a_te:>8.4f} "
              f"{a_tr - a_te:>8.4f} {a_te - asos:>+9.4f}")
    print("  ⭐ Har birini ALOHIDA o'lchang, keyin birlashtiring")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
  o'quv 600, validatsiya 300, test 300

=== 1. Regularizatsiyasiz ===
  to_plam           loss   aniqlik
  o_quv           0.0001    1.0000
  test            1.4752    0.8100
  farq: +0.1900

=== 2. Erta to'xtash ===
  39-davrda to'xtadi, eng yaxshisi 24-davr
  o'quv 0.9267, test 0.8300, farq +0.0967
  asosdan: +0.0200

=== 3. Weight decay ===
         wd    o_quv     test     farq   asosdan
      0e+00   1.0000   0.8100   0.1900   +0.0000
      1e-04   1.0000   0.8067   0.1933   -0.0033
      1e-02   1.0000   0.8133   0.1867   +0.0033
      1e-01   1.0000   0.8067   0.1933   -0.0033
      1e+00   1.0000   0.8200   0.1800   +0.0100

=== 4. Dropout ===
       p    o_quv     test     farq   asosdan
     0.0   1.0000   0.8100   0.1900   +0.0000
     0.2   1.0000   0.8367   0.1633   +0.0267
     0.4   1.0000   0.8633   0.1367   +0.0533
     0.6   0.9967   0.8767   0.1200   +0.0667

=== 5. Kombinatsiya ===
  variant                 o_quv     test     farq   asosdan
  hech narsa             1.0000   0.8100   0.1900   +0.0000
  erta to'xtash          0.9267   0.8300   0.0967   +0.0200
  wd 1e-2                1.0000   0.8133   0.1867   +0.0033
  dropout 0.3            1.0000   0.8467   0.1533   +0.0367
  erta + wd              0.9267   0.8267   0.1000   +0.0167
  erta + wd + drop       0.9033   0.8400   0.0633   +0.0300
  ⭐ Har birini ALOHIDA o'lchang, keyin birlashtiring

Nima ko'rsatdi: 2.4, 2.6-bo'limlar.


5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"O'quv aniqligi 1.0 — yaxshi" Tasodifiy yorliqlarda ham shunday
"Dropout inference da ham ishlaydi" eval() da o'chadi
"Adam + L2 = AdamW" Yo'q, natija boshqa
"weight decay hamma parametrga" bias ga odatda yo'q
"Erta to'xtash — to'xtash" Eng yaxshi og'irliklarni tiklash
"Dropout har doim yordam beradi" Kichik modelda zarar qilishi mumkin
"Hammasini birga yoqish yaxshi" Ta'sirini ajratib bo'lmaydi
"Regularizatsiya ma'lumot o'rnini bosadi" Faqat qisman

6. Keng tarqalgan xatolar va yechimlari

1. eval() ni unutish

python
bashorat = model(X_test)                       # ⚠️ dropout faol
model.eval(); bashorat = model(X_test)         # ✅

2. Adam ga L2

python
Adam(params, weight_decay=0.01)                # ⚠️
AdamW(params, weight_decay=0.01)               # ✅

3. Eng yaxshi og'irliklarni tiklamaslik

python
if sabr_hisobi >= sabr: break                  # ⚠️ yomon nuqtada
model.load_state_dict(eng_holat)               # ✅

4. bias ga decay

python
optim.AdamW(model.parameters(), weight_decay=0.1)     # ⚠️
# bias larni alohida guruhga ajrating, wd=0            # ✅

5. Chiqishga dropout

python
nn.Sequential(..., nn.Dropout(0.5), nn.Linear(64, K)) # ✅
nn.Sequential(..., nn.Linear(64, K), nn.Dropout(0.5)) # ⚠️

6. Test to'plamida erta to'xtash

python
ball = baho(model, X_test, y_test)             # ⚠️ leakage
ball = baho(model, X_val, y_val)               # ✅

7. Hammasini birdan yoqish

python
# wd + dropout + label smoothing + augment birga   # ⚠️
# bittadan qo'shib, har birini o'lchang            # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 18-qism (o'tilgan): Validatsiya dizayni
  • 20.3-dars (o'tilgan): Model hajmi
  • 20.11-dars: To'liq o'rgatish sikli
  • 21-qism: nn.Dropout, AdamW
  • 24-qism: Rasm augmentatsiyasi

8. Eng yaxshi amaliyotlar

  1. Erta to'xtashni har doim qo'ying.

  2. Eng yaxshi og'irliklarni tiklang.

  3. AdamW ishlating.

  4. model.eval() ni unutmang.

  5. Bir qadamda bitta o'zgarish.

  6. Validatsiya va testni ajrating.

  7. Avval model hajmini tekshiring.

  8. O'quv/test farqini kuzating.


9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # tarmoq tasodifiy yorliqni yod oladimi?
2.  # yodlash belgilari?
3.  # weight decay formulasi?
4.  # SGD da L2 va wd bir xilmi?
5.  # Adam da-chi?
6.  # dropout o'rgatishda nima qiladi?
7.  # inference da-chi?
8.  # inverted dropout koeffitsiyenti?
9.  # erta to'xtashda nima qilish shart?
10. # bias ga decay kerakmi?
11. # tipik dropout p?
12. # eng kuchli regularizatsiya?
Javoblar
  1. Ha
  2. O'quv loss ~0, validatsiya o'sadi
  3. w *= (1 - lr*lambda)
  4. Ha
  5. Yo'q — AdamW kerak
  6. Neyronlarni tasodifiy o'chiradi
  7. Hech narsa
  8. 1/(1-p)
  9. Eng yaxshi og'irliklarni tiklash
  10. Odatda yo'q
  11. 0.2–0.5
  12. Ko'proq ma'lumot

Vazifa 2: Xatolarni tuzating

python
1.  bashorat = model(X_test)

2.  Adam(params, weight_decay=0.01)

3.  if sabr_hisobi >= sabr: break

4.  ball = baho(model, X_test, y_test)   # erta to'xtash uchun

5.  nn.Sequential(..., nn.Linear(64, K), nn.Dropout(0.5))
Javoblar
python
1.  model.eval(); bashorat = model(X_test)

2.  AdamW(params, weight_decay=0.01)

3.  model.load_state_dict(eng_holat)

4.  ball = baho(model, X_val, y_val)

5.  nn.Sequential(..., nn.Dropout(0.5), nn.Linear(64, K))

Vazifa 3: Yodlash

Modellang:

  1. Tasodifiy yorliq
  2. Davrlar
  3. Normalar
  4. Ishonch

Vazifa 4: Weight decay

Modellang:

  1. Tenglik
  2. Qisqarish
  3. Regressiya
  4. Bias

Vazifa 5: Dropout

Modellang:

  1. Mexanizm
  2. O'rtacha
  3. Rejimlar
  4. Ansambl

Vazifa 6: Kombinatsiya

Modellang:

  1. Bazaviy
  2. Erta to'xtash
  3. wd
  4. Birga

Vazifa 7: O'ylash

Regularizatsiyani kuchaytirdingiz: dropout=0.5, wd=0.1, erta to'xtash. Endi o'quv aniqligi ham, test aniqligi ham past. Nima bo'lgan?

Javob

Nima bo'lgan: siz yodlashdan to'liq qarama-qarshi tomonga o'tib ketdingiz. Model endi yetarlicha o'rgana olmayapti — bu underfitting.

Ikki holatni farqlash:

Belgi Yodlash Yetarsiz o'rganish
O'quv aniqligi Juda yuqori Past
Test aniqligi Past Past
Farq Katta Kichik
Yechim Regularizatsiya Regularizatsiyani kamaytirish

O'quv va test ikkalasi ham past va farq kichik bo'lsa — regularizatsiya haddan oshgan.

Har birining hissasini alohida tekshiring:

python
variantlar = [
    ("hammasi", 0.5, 0.1, True),
    ("dropout yo'q", 0.0, 0.1, True),
    ("wd yo'q", 0.5, 0.0, True),
    ("erta to'xtash yo'q", 0.5, 0.1, False),
]
for nom, p, wd, erta in variantlar:
    print(nom, orgat_va_bahola(p, wd, erta))

Qaysi birini o'chirganda o'quv aniqligi sakrab ko'tarilsa — aybdor o'sha.

Eng ehtimolli aybdor: wd = 0.1

Bu juda katta qiymat. Tipik oraliq 1e-5 dan 1e-2 gacha; 0.1 og'irliklarni nolga siqib, modelni deyarli chiziqli qilib qo'yadi.

python
print([W.norm().item() for W in model.parameters()])
# hammasi ~0 bo'lsa - wd juda katta

Ikkinchi aybdor: erta to'xtash juda erta ishlagan

python
print("to'xtagan davr:", tugagan, "eng yaxshi davr:", eng_davr)

eng_davr juda kichik bo'lsa (masalan 5), sabr yetarli emas yoki lr juda kichik.

Uchinchi: dropout=0.5 kichik model uchun

(256, 128) kabi tarmoqda 0.5 ko'p bo'lmasligi mumkin, lekin (32, 16) da u modelning yarmini yo'q qiladi.

Tuzatish tartibi:

  1. Hamma regularizatsiyani o'chiring — o'quv aniqligi yuqori bo'lishi kerak. Bo'lmasa muammo boshqa joyda (lr, arxitektura, ma'lumot)
  2. Erta to'xtashni qaytaring (u deyarli zararsiz)
  3. wd ni 1e-4 dan boshlab oshiring, har qadamda o'lchang
  4. Hali yodlayotgan bo'lsa dropout=0.2 qo'shing
  5. Har qadamda o'quv va test ballini yozing

Asosiy fikr: regularizatsiya — nozik sozlash, kalit emas. Uning maqsadi o'quv va test orasidagi farqni yopish, ikkalasini ham pastga tushirish emas.

Nimani mustahkamlaydi: 2.1, 2.6-bo'limlar.


Xulosa

Bu darsda regularizatsiyani ko'rdik.

Eng muhim uch fikr:

  1. O'quv balli hech narsani isbotlamaydi. 1-misolda tarmoq tasodifiy yorliqlarda ham o'quv to'plamini to'liq yod oldi, test aniqligi esa tangadek qoldi. Yodlashning belgilari aniq: o'quv loss nolga tushadi, validatsiya loss o'sadi, og'irliklar normasi kattalashadi va bashoratlar haddan ishonchli bo'lib qoladi.

  2. Uch usul — uch xil mexanizm. weight decay og'irliklarni har qadamda (1 - lr*lambda) ga qisqartiradi; dropout neyronlarni tasodifiy o'chirib, ularning bir-biriga tayanishini buzadi; erta to'xtash esa optimallashtirish vaqtini cheklaydi. Ular bir-birining o'rnini bosmaydi, shuning uchun ko'pincha birga ishlatiladi.

  3. Ikkita xato eng qimmatga tushadi. Birinchisi — model.eval() ni unutish: dropout inference da faol qolsa, bashoratlar tasodifiy bo'ladi. Ikkinchisi — Adam ga weight_decay berish: L2 jarimasi adaptiv bo'linishdan o'tib buziladi, shuning uchun AdamW kerak. Erta to'xtashda esa eng yaxshi og'irliklarni tiklashni unutmang, aks holda siz eng yomon nuqtada to'xtaysiz.

Keyingi darsda birinchi to'liq tarmoq: o'rgatish siklini boshdan-oxir quramiz — batch, davr, validatsiya, jadval, erta to'xtash va loglash bir joyda.

Neyron tarmoqlar · 10-dars yakunlandi

20.10-dars: Regularizatsiya

23 daqvaqt
0belgi
Keyingi dars: 20.11-dars: Birinchi to'liq tarmoq
Telegram'da
Darsda xato koʻrdingizmi?

Izohlar · 0