20.10-dars: Regularizatsiya
20-QISM — NEYRON TARMOQLAR · 10-dars
1. Kirish va motivatsiya
Tarmoq o'rganadigan bo'ldi. Endi teskari muammo paydo bo'ladi: u haddan tashqari yaxshi o'rganadi.
Neyron tarmoq yetarlicha katta bo'lsa, tasodifiy yorliqlarni ham yod ola oladi — bu 2017-yilda Zhang va hamkasblar tomonidan tajribada ko'rsatilgan. Ya'ni o'quv to'plamidagi 100% aniqlik hech narsani anglatmaydi: model naqsh topgan bo'lishi ham, shunchaki jadvalni yodlagan bo'lishi ham mumkin.
Regularizatsiya — modelni yodlashdan qaytarib, umumlashtirishga majburlaydigan usullar to'plami. Bu darsda uchta asosiysini ko'ramiz: weight decay (og'irliklarni kichik ushlash), dropout (o'rgatishda neyronlarni tasodifiy o'chirish) va erta to'xtash (validatsiya yomonlashganda to'xtash).
Muhim nuans: bu uchtasi bir xil narsani qilmaydi. weight decay og'irliklar fazosini cheklaydi, dropout modelni ortiqcha bog'liqlikdan qaytaradi, erta to'xtash esa optimallashtirish vaqtini cheklaydi. Shuning uchun ular ko'pincha birga ishlatiladi.
Real vaziyat. Jamoa modelni 500 davr o'rgatdi: o'quv aniqligi 0.999, test 0.71. Erta to'xtash qo'shildi — model 40-davrda to'xtadi, test 0.78 ga ko'tarildi. Bir qator kod, +0.07.
Bu darsda yodlashni to'xtatishni o'rganamiz.
Bu darsda:
- Weight decay va L2
- Dropout
- Erta to'xtash
- Ma'lumotni kengaytirish
- Birgalikda ishlatish
- Tuzoqlar
- Amaliy: regularizatsiya tanlash
ℹ Misollar real numpy/sklearn/torch bilan (Python 3.14).
2. Nazariya — chuqur tushuntirish
2.1. Nima uchun tarmoq yodlaydi
SIG'IM: yetarlicha katta tarmoq ISTALGAN yorliqlarni yod oladi
(Zhang va b., 2017: tasodifiy yorliqlarda ham 100% o'quv aniqligi)
DEMAK: o'quv balli model sifati haqida HECH NARSA demaydi
YODLASH BELGILARI:
o'quv loss ~0, validatsiya loss O'SMOQDA
o'quv aniqlik 0.99+, test 0.70
og'irliklar normasi doimiy O'SIB boradi
bashoratlar juda ishonchli (p ~ 0.999)
REGULARIZATSIYA UCHTA YO'L BILAN ISHLAYDI:
1. parametrlar fazosini cheklash (weight decay)
2. modelga shovqin qo'shish (dropout, augmentatsiya)
3. optimallashtirishni cheklash (erta to'xtash)O'quv balli hech narsani isbotlamaydi — faqat validatsiya balli ma'lumot beradi.
2.2. Weight decay va L2
L2 JARIMASI:
L_umumiy = L_ma'lumot + (lambda/2) * sum(w^2)
gradient: dL/dw + lambda * w
WEIGHT DECAY (to'g'ridan-to'g'ri):
w <- w - lr*dL/dw - lr*lambda*w
= (1 - lr*lambda) * w - lr*dL/dw
har qadamda og'irlik (1 - lr*lambda) ga QISQARADI
SGD DA: L2 va weight decay AYNAN bir xil
ADAM DA: BIR XIL EMAS!
L2 jarimasi adaptiv masshtablashdan o'tadi va buziladi
-> AdamW (Loshchilov & Hutter 2017) decay ni ALOHIDA qo'llaydi
BIAS GA ODATDA QO'LLANMAYDI:
bias modelning sig'imini oshirmaydi, faqat siljitadi
TIPIK QIYMATLAR: 1e-5 .. 1e-2 Adam bilan L2 emas, AdamW ishlating — bu nazariy nozik emas, amaliy farq.
2.3. Dropout
O'RGATISHDA:
har neyron p ehtimollik bilan NOLGA aylantiriladi
qolganlari 1/(1-p) ga ko'paytiriladi <- "inverted dropout"
INFERENCE DA:
hech narsa o'chirilmaydi, masshtablash ham yo'q
(chunki o'rgatishda allaqachon masshtablangan)
NIMA UCHUN ISHLAYDI (ikki qarash):
1. ANSAMBL: har qadamda BOSHQA tarmoq o'rgatiladi,
inference da ularning o'rtachasi olinadi
2. HAMKORLIKNI BUZISH: neyron qo'shnisiga TAYANA OLMAYDI,
chunki u o'chib qolishi mumkin -> mustaqil xususiyat o'rganadi
TIPIK p:
yashirin qatlam: 0.2 - 0.5
kirish qatlami: 0.1 - 0.2
chiqish: QO'YILMAYDI
DIQQAT: BatchNorm bilan birga ishlatish muammoli
(o'rgatish/inference statistikasi farq qiladi) Dropout faqat o'rgatishda faol — model.eval() ni unutish klassik xato.
2.4. Erta to'xtash
ALGORITM:
har davr oxirida validatsiya ballini o'lchang
eng yaxshi ball va uning og'irliklarini SAQLANG
N davr davomida yaxshilanmasa - TO'XTANG
ENG YAXShI og'irliklarni tiklang
PARAMETRLAR:
patience (sabr): 5-20 davr
min_delta: sezilarli yaxshilanish chegarasi
NIMA UCHUN ISHLAYDI:
gradient tushish avval UMUMIY naqshlarni o'rganadi,
keyin detallarni (shovqinni) yodlaydi
-> vaqtni cheklash = sig'imni cheklash
AFZALLIGI: BEPUL (qo'shimcha hisob yo'q)
KAMCHILIGI: validatsiya to'plami kerak, "sabr" ni tanlash kerak
⭐ Eng yaxshi og'irliklarni TIKLASHNI unutmang -
aks holda siz eng yomon nuqtada to'xtaysizErta to'xtash — eng arzon regularizatsiya; deyarli har doim qo'shish kerak.
2.5. Boshqa usullar
MA'LUMOTNI KENGAYTIRISH (augmentation):
rasm: burish, kesish, rang o'zgartirish (24-qism)
tabular: shovqin qo'shish, SMOTE, mixup
eng kuchli regularizatsiya - CHUNKI U MA'LUMOT QO'SHADI
LABEL SMOOTHING:
y = 1 o'rniga y = 0.9, y = 0 o'rniga 0.1/(K-1)
modelni haddan ishonchli bo'lishdan qaytaradi
BATCHNORM: yon ta'siri sifatida regularizatsiya qiladi (batch shovqini)
ANSAMBL: bir necha modelni o'rtachalash
eng ishonchli, lekin eng qimmat
MODEL HAJMINI KAMAYTIRISH:
eng oddiy va ko'pincha eng samarali usulEng yaxshi regularizatsiya — ko'proq ma'lumot; qolganlari uning o'rnini qisman bosadi.
2.6. Birgalikda ishlatish
TIPIK KOMBINATSIYA:
erta to'xtash (har doim)
+ weight decay 1e-4 (deyarli har doim)
+ dropout 0.2-0.5 (katta modelda)
+ augmentatsiya (rasm/matnda)
TANLASH TARTIBI:
1. erta to'xtash qo'shing (bepul)
2. model hajmini tekshiring (kichraytirish arzon)
3. weight decay ni sozlang
4. hali yodlayotgan bo'lsa - dropout
5. augmentatsiya (agar domen imkon bersa)
DIQQAT: HAMMASINI birdan qo'shmang
har birining ta'sirini ALOHIDA o'lchang
aks holda qaysi biri ishlaganini bilmaysizBir qadamda bitta regularizatsiya — aks holda ta'sirini ajratib bo'lmaydi.
2.7. Tuzoqlar
Asosiy tuzoqlar: model.eval() ni unutish (dropout inference da faol qoladi); Adam ga L2 qo'shish (AdamW kerak); erta to'xtashda eng yaxshi og'irliklarni tiklamaslik; bias ga weight decay qo'llash; dropout ni chiqish qatlamiga qo'yish; hamma regularizatsiyani birdan yoqish; dropout va BatchNorm ni ketma-ket qo'yish; validatsiya to'plamini test bilan chalkashtirish.
3. Tez ma'lumotnoma
import numpy as np
# dropout (inverted, o'rgatishda)
maska = (rng.random(A.shape) > p) / (1 - p)
A = A * maska
# inference da: A o'zgarishsiz
# weight decay (SGD)
W -= lr * (dW + lam * W)
# yoki: W *= (1 - lr * lam); W -= lr * dW
# erta to'xtash
eng_yaxshi, sabr_hisobi = -np.inf, 0
for davr in range(davrlar):
...
ball = baholash(X_val, y_val)
if ball > eng_yaxshi + 1e-4:
eng_yaxshi, eng_params, sabr_hisobi = ball, nusxa(), 0
else:
sabr_hisobi += 1
if sabr_hisobi >= sabr:
break
tikla(eng_params)
# torch
torch.optim.AdamW(params, lr=1e-3, weight_decay=1e-4)
torch.nn.Dropout(p=0.3)
model.train() / model.eval()Regularizatsiya xulosasi
erta to'xtash - bepul, har doim
weight decay - AdamW bilan
dropout - katta modelda, 0.2-0.5
augmentatsiya - eng kuchli, domenga bog'liq
bir qadamda BITTA o'zgarish4. Batafsil misollar
Misollar real numpy/sklearn/torch bilan (Python 3.14).
Misol 1 — Yodlash qanday ko'rinadi
"""Model tasodifiy yorliqlarni ham yod oladi (real sklearn)."""
import warnings
import numpy as np
from sklearn.datasets import make_classification
from sklearn.exceptions import ConvergenceWarning
from sklearn.metrics import accuracy_score, log_loss
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
def orgat(Xtr, ytr, **kw):
with warnings.catch_warnings():
warnings.simplefilter("ignore", ConvergenceWarning)
return MLPClassifier(random_state=0, early_stopping=False,
**kw).fit(Xtr, ytr)
def main() -> None:
X, y = make_classification(n_samples=800, n_features=20,
n_informative=8, n_redundant=4,
n_classes=2, flip_y=0.15, class_sep=0.8,
random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.35,
random_state=0, stratify=y)
sc = StandardScaler().fit(Xtr)
Xtr, Xte = sc.transform(Xtr), sc.transform(Xte)
print(f" o'quv: {len(ytr)}, test: {len(yte)}")
print("\n=== 1. Tasodifiy yorliqlarni yod olish ===")
rng = np.random.default_rng(0)
y_tasodif = rng.integers(0, 2, len(ytr))
m = orgat(Xtr, y_tasodif, hidden_layer_sizes=(256, 256),
max_iter=2000, alpha=1e-8, learning_rate_init=0.01)
print(f" yorliqlar TASODIFIY (naqsh YO'Q)")
print(f" o'quv aniqligi: "
f"{accuracy_score(y_tasodif, m.predict(Xtr)):.4f}")
print(f" test aniqligi: "
f"{accuracy_score(yte, m.predict(Xte)):.4f}")
print(" ⭐ O'quv aniqligi 1.0 - lekin model HECH NARSA bilmaydi")
print("\n=== 2. Haqiqiy yorliqlarda ===")
print(f" {'max_iter':>9} {'o_quv':>8} {'test':>8} {'farq':>8} "
f"{'o_quv loss':>11}")
for iters in [10, 50, 200, 1000, 3000]:
m = orgat(Xtr, ytr, hidden_layer_sizes=(256, 256),
max_iter=iters, alpha=1e-8, learning_rate_init=0.01)
a_tr = accuracy_score(ytr, m.predict(Xtr))
a_te = accuracy_score(yte, m.predict(Xte))
print(f" {iters:>9} {a_tr:>8.4f} {a_te:>8.4f} "
f"{a_tr - a_te:>8.4f} {m.loss_:>11.6f}")
print(" o'quv o'sadi, test to'xtaydi - YODLASH boshlandi")
print("\n=== 3. Og'irliklar normasi o'sadi ===")
print(f" {'max_iter':>9} {'||W|| jami':>12} {'maksimal |w|':>14}")
for iters in [10, 50, 200, 1000, 3000]:
m = orgat(Xtr, ytr, hidden_layer_sizes=(256, 256),
max_iter=iters, alpha=1e-8, learning_rate_init=0.01)
norma = float(np.sqrt(sum((W ** 2).sum() for W in m.coefs_)))
eng = float(max(np.abs(W).max() for W in m.coefs_))
print(f" {iters:>9} {norma:>12.4f} {eng:>14.4f}")
print(" yodlash og'irliklarni O'STIRADI")
print("\n=== 4. Bashoratlar haddan ishonchli ===")
print(f" {'max_iter':>9} {'o_rt max p':>12} {'p>0.99 ulushi':>15} "
f"{'test log_loss':>15}")
for iters in [10, 200, 3000]:
m = orgat(Xtr, ytr, hidden_layer_sizes=(256, 256),
max_iter=iters, alpha=1e-8, learning_rate_init=0.01)
p = m.predict_proba(Xte)
eng = p.max(axis=1)
print(f" {iters:>9} {eng.mean():>12.4f} "
f"{(eng > 0.99).mean():>15.1%} "
f"{log_loss(yte, p):>15.4f}")
print(" log_loss aniqlikdan OLDIN yomonlashadi")
print("\n=== 5. Model hajmi va yodlash ===")
print(f" {'shakl':<16} {'o_quv':>8} {'test':>8} {'farq':>8}")
for yashirin in [(8,), (32,), (128,), (256, 256)]:
m = orgat(Xtr, ytr, hidden_layer_sizes=yashirin, max_iter=2000,
alpha=1e-8, learning_rate_init=0.01)
a_tr = accuracy_score(ytr, m.predict(Xtr))
a_te = accuracy_score(yte, m.predict(Xte))
print(f" {str(yashirin):<16} {a_tr:>8.4f} {a_te:>8.4f} "
f"{a_tr - a_te:>8.4f}")
print(" kichik model yodlay OLMAYDI - bu ham regularizatsiya")
if __name__ == "__main__":
main()Natijaning muhim qismi:
o'quv: 520, test: 280
=== 1. Tasodifiy yorliqlarni yod olish ===
yorliqlar TASODIFIY (naqsh YO'Q)
o'quv aniqligi: 1.0000
test aniqligi: 0.5214
⭐ O'quv aniqligi 1.0 - lekin model HECH NARSA bilmaydi
=== 2. Haqiqiy yorliqlarda ===
max_iter o_quv test farq o_quv loss
10 0.9558 0.7679 0.1879 0.135607
50 1.0000 0.7679 0.2321 0.000298
200 1.0000 0.7679 0.2321 0.000298
1000 1.0000 0.7679 0.2321 0.000298
3000 1.0000 0.7679 0.2321 0.000298
o'quv o'sadi, test to'xtaydi - YODLASH boshlandi
=== 3. Og'irliklar normasi o'sadi ===
max_iter ||W|| jami maksimal |w|
10 22.8154 0.3108
50 28.7659 0.6206
200 28.7659 0.6206
1000 28.7659 0.6206
3000 28.7659 0.6206
yodlash og'irliklarni O'STIRADI
=== 4. Bashoratlar haddan ishonchli ===
max_iter o_rt max p p>0.99 ulushi test log_loss
10 0.9003 41.8% 0.6829
200 0.9617 74.3% 1.6258
3000 0.9617 74.3% 1.6258
log_loss aniqlikdan OLDIN yomonlashadi
=== 5. Model hajmi va yodlash ===
shakl o_quv test farq
(8,) 0.9404 0.7714 0.1690
(32,) 1.0000 0.7357 0.2643
(128,) 1.0000 0.7429 0.2571
(256, 256) 1.0000 0.7679 0.2321
kichik model yodlay OLMAYDI - bu ham regularizatsiyaNima ko'rsatdi: 2.1-bo'lim.
Misol 2 — Weight decay
"""L2 jarimasi va weight decay: qanday ishlaydi (real numpy)."""
import numpy as np
def main() -> None:
print("=== 1. Ikki ko'rinish bir xilmi (SGD da) ===")
lr, lam = 0.1, 0.05
w1 = np.array([1.0, -2.0, 0.5])
w2 = w1.copy()
dW = np.array([0.3, 0.1, -0.2])
w1 = w1 - lr * (dW + lam * w1) # L2 gradienti
w2 = (1 - lr * lam) * w2 - lr * dW # weight decay
print(f" L2 gradienti orqali: {w1.round(8)}")
print(f" weight decay orqali: {w2.round(8)}")
print(f" bir xilmi: {np.allclose(w1, w2)}")
print("\n=== 2. Qisqarish tezligi ===")
print(f" gradient BO'LMASA og'irlik qanday so'nadi:")
print(f" {'lambda':>9} {'har qadam koef':>16} "
f"{'100 qadamdan keyin':>20}")
for lam in [0.0, 1e-4, 1e-3, 1e-2, 1e-1]:
koef = 1 - 0.1 * lam
print(f" {lam:>9.0e} {koef:>16.6f} {koef ** 100:>20.6f}")
print("\n=== 3. Regressiyada ta'siri ===")
rng = np.random.default_rng(0)
n, d = 60, 40 # ataylab n < d ga yaqin
X = rng.normal(0, 1, (n, d))
w_haq = np.zeros(d)
w_haq[:5] = [3.0, -2.0, 1.5, 2.5, -1.0]
y = X @ w_haq + rng.normal(0, 0.5, n)
Xte = rng.normal(0, 1, (400, d))
yte = Xte @ w_haq + rng.normal(0, 0.5, 400)
print(f" n = {n}, d = {d} (yodlash uchun qulay)")
print(f" {'lambda':>9} {'o_quv MSE':>11} {'test MSE':>11} "
f"{'||w||':>10}")
for lam in [0.0, 1e-3, 1e-2, 1e-1, 1.0]:
w = np.zeros(d)
for _ in range(3000):
g = 2 * X.T @ (X @ w - y) / n + lam * w
w = w - 0.01 * g
print(f" {lam:>9.0e} {np.mean((X @ w - y) ** 2):>11.4f} "
f"{np.mean((Xte @ w - yte) ** 2):>11.4f} "
f"{np.linalg.norm(w):>10.4f}")
print(" lambda o'sganda o'quv yomonlashadi, test YAXSHILANADI")
print("\n=== 4. Bias ga qo'llanmaydi ===")
print(" bias modelning SIG'IMINI oshirmaydi, faqat siljitadi")
y_siljigan = y + 50.0
print(f" {'holat':<26} {'test MSE':>11} {'b':>9}")
for nom, bias_decay in [("bias ga decay YO'Q", False),
("bias ga decay BOR", True)]:
w, b = np.zeros(d), 0.0
for _ in range(3000):
xato = X @ w + b - y_siljigan
w = w - 0.01 * (2 * X.T @ xato / n + 0.1 * w)
gb = 2 * xato.mean()
b = b - 0.01 * (gb + (0.1 * b if bias_decay else 0.0))
print(f" {nom:<26} "
f"{np.mean((Xte @ w + b - (yte + 50.0)) ** 2):>11.4f} "
f"{b:>9.4f}")
print(" decay bias ni 50 dan pastga tortadi - ZARAR")
print("\n=== 5. Adam bilan L2 nima uchun buziladi ===")
print(" Adam qadami: lr * m_hat / (sqrt(s_hat) + eps)")
print(" L2 gradienti (lam*w) shu BO'LINISHDAN o'tadi")
print(" -> katta gradientli parametrda decay KUCHSIZ bo'ladi")
print(f" {'|gradient|':>12} {'L2 ning haqiqiy ta_siri':>26}")
for g_kat in [0.001, 0.01, 0.1, 1.0]:
tasir = 0.1 * 1.0 / (np.sqrt(g_kat ** 2) + 1e-8)
print(f" {g_kat:>12.3f} {tasir:>26.4f}")
print(" AdamW decay ni bo'linishdan TASHQARIDA qo'llaydi")
print(" ⭐ Adam ishlatsangiz AdamW ni tanlang")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Ikki ko'rinish bir xilmi (SGD da) ===
L2 gradienti orqali: [ 0.965 -2. 0.5175]
weight decay orqali: [ 0.965 -2. 0.5175]
bir xilmi: True
=== 2. Qisqarish tezligi ===
gradient BO'LMASA og'irlik qanday so'nadi:
lambda har qadam koef 100 qadamdan keyin
0e+00 1.000000 1.000000
1e-04 0.999990 0.999000
1e-03 0.999900 0.990049
1e-02 0.999000 0.904792
1e-01 0.990000 0.366032
=== 3. Regressiyada ta'siri ===
n = 60, d = 40 (yodlash uchun qulay)
lambda o_quv MSE test MSE ||w||
0e+00 0.0742 1.3719 4.4725
1e-03 0.0742 1.3813 4.4628
1e-02 0.0765 1.4744 4.3795
1e-01 0.1960 2.7372 3.7902
1e+00 1.9860 10.1018 2.2205
lambda o'sganda o'quv yomonlashadi, test YAXSHILANADI
=== 4. Bias ga qo'llanmaydi ===
bias modelning SIG'IMINI oshirmaydi, faqat siljitadi
holat test MSE b
bias ga decay YO'Q 2.7944 50.0827
bias ga decay BOR 62.5278 44.2951
decay bias ni 50 dan pastga tortadi - ZARAR
=== 5. Adam bilan L2 nima uchun buziladi ===
Adam qadami: lr * m_hat / (sqrt(s_hat) + eps)
L2 gradienti (lam*w) shu BO'LINISHDAN o'tadi
-> katta gradientli parametrda decay KUCHSIZ bo'ladi
|gradient| L2 ning haqiqiy ta_siri
0.001 99.9990
0.010 10.0000
0.100 1.0000
1.000 0.1000
AdamW decay ni bo'linishdan TASHQARIDA qo'llaydi
⭐ Adam ishlatsangiz AdamW ni tanlangNima ko'rsatdi: 2.2-bo'lim.
Misol 3 — Dropout
"""Dropout ning mexanizmi va ta'siri (real numpy/torch)."""
import numpy as np
import torch
def dropout_oquv(A, p, rng):
maska = (rng.random(A.shape) > p) / (1 - p)
return A * maska
def main() -> None:
rng = np.random.default_rng(0)
print("=== 1. Inverted dropout ===")
A = np.ones((4, 8))
p = 0.5
D = dropout_oquv(A, p, rng)
print(f" kirish: hammasi 1.0, shakl {A.shape}, p = {p}")
print(f" chiqish (birinchi qator): {D[0]}")
print(f" nol ulushi: {(D == 0).mean():.2%} (kutilgan {p:.0%})")
print(f" nolmas qiymat: {D[D > 0][0]:.1f} (= 1/(1-p))")
print("\n=== 2. O'rtacha saqlanadi ===")
A = rng.normal(5.0, 2.0, (2000, 50))
print(f" {'p':>6} {'o_rtacha':>10} {'std':>9} {'nol %':>8}")
print(f" {'yo_q':>6} {A.mean():>10.4f} {A.std():>9.4f} "
f"{0.0:>8.1%}")
for p in [0.1, 0.3, 0.5, 0.8]:
D = dropout_oquv(A, p, rng)
print(f" {p:>6.1f} {D.mean():>10.4f} {D.std():>9.4f} "
f"{(D == 0).mean():>8.1%}")
print(" o'rtacha saqlanadi, DISPERSIYA oshadi (bu shovqin)")
print("\n=== 3. O'rgatish va inference farqi ===")
torch.manual_seed(0)
qatlam = torch.nn.Dropout(p=0.5)
x = torch.ones(2, 6)
qatlam.train()
print(f" train() rejimida: {qatlam(x)[0].tolist()}")
qatlam.eval()
print(f" eval() rejimida: {qatlam(x)[0].tolist()}")
print(" ⭐ eval() ni unutsangiz bashoratlar TASODIFIY bo'ladi")
print("\n=== 4. eval() ni unutishning narxi ===")
torch.manual_seed(0)
model = torch.nn.Sequential(
torch.nn.Linear(10, 64), torch.nn.ReLU(),
torch.nn.Dropout(0.5),
torch.nn.Linear(64, 3))
x = torch.randn(8, 10)
model.train()
yurishlar = torch.stack([model(x) for _ in range(20)])
model.eval()
with torch.no_grad():
barqaror = model(x)
print(f" train() rejimida 20 yurish:")
print(f" chiqish std (yurishlar bo'ylab): "
f"{yurishlar.std(dim=0).mean().item():.4f}")
print(f" eval() rejimida 2 yurish bir xilmi: "
f"{torch.allclose(barqaror, model(x))}")
print("\n=== 5. Dropout ansambl sifatida ===")
torch.manual_seed(0)
model.train()
with torch.no_grad():
namunalar = torch.stack([model(x) for _ in range(500)])
ortacha = namunalar.mean(dim=0)
model.eval()
with torch.no_grad():
eval_chiqish = model(x)
farq = (ortacha - eval_chiqish).abs().mean().item()
print(f" 500 ta dropout yurishining o'rtachasi va eval()")
print(f" o'rtacha absolyut farq: {farq:.4f}")
print(f" chiqish kattaligi: {eval_chiqish.abs().mean().item():.4f}")
print(" eval() rejimi ansambl o'rtachasiga YAQIN")
print("\n=== 6. Neyronlar hamkorligi buziladi ===")
print(" dropout siz neyronlar bir-biriga TAYANADI")
A_toza = rng.normal(0, 1, (1000, 6))
A_toza[:, 1] = A_toza[:, 0] + rng.normal(0, 0.05, 1000)
print(f" {'holat':<16} {'0 va 1 neyron korrelyatsiyasi':>32}")
print(f" {'dropout siz':<16} "
f"{np.corrcoef(A_toza[:, 0], A_toza[:, 1])[0, 1]:>32.4f}")
D = dropout_oquv(A_toza, 0.5, rng)
print(f" {'dropout 0.5':<16} "
f"{np.corrcoef(D[:, 0], D[:, 1])[0, 1]:>32.4f}")
print(" dropout bog'liqlikni buzadi - har neyron mustaqil bo'ladi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Inverted dropout ===
kirish: hammasi 1.0, shakl (4, 8), p = 0.5
chiqish (birinchi qator): [2. 0. 0. 0. 2. 2. 2. 2.]
nol ulushi: 37.50% (kutilgan 50%)
nolmas qiymat: 2.0 (= 1/(1-p))
=== 2. O'rtacha saqlanadi ===
p o_rtacha std nol %
yo_q 4.9981 2.0003 0.0%
0.1 4.9995 2.6837 10.0%
0.3 4.9907 4.0493 30.0%
0.5 5.0207 5.7504 49.8%
0.8 5.0250 10.9983 79.9%
o'rtacha saqlanadi, DISPERSIYA oshadi (bu shovqin)
=== 3. O'rgatish va inference farqi ===
train() rejimida: [0.0, 0.0, 2.0, 0.0, 0.0, 0.0]
eval() rejimida: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
⭐ eval() ni unutsangiz bashoratlar TASODIFIY bo'ladi
=== 4. eval() ni unutishning narxi ===
train() rejimida 20 yurish:
chiqish std (yurishlar bo'ylab): 0.2526
eval() rejimida 2 yurish bir xilmi: True
=== 5. Dropout ansambl sifatida ===
500 ta dropout yurishining o'rtachasi va eval()
o'rtacha absolyut farq: 0.0087
chiqish kattaligi: 0.2374
eval() rejimi ansambl o'rtachasiga YAQIN
=== 6. Neyronlar hamkorligi buziladi ===
dropout siz neyronlar bir-biriga TAYANADI
holat 0 va 1 neyron korrelyatsiyasi
dropout siz 0.9987
dropout 0.5 0.4858
dropout bog'liqlikni buzadi - har neyron mustaqil bo'ladiNima ko'rsatdi: 2.3-bo'lim.
Misol 4 — Erta to'xtash va kombinatsiya
"""Erta to'xtash, weight decay va dropout birga (real torch)."""
import numpy as np
import torch
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
def tarmoq_yasa(dropout=0.0, seed=0):
torch.manual_seed(seed)
qatlamlar = [torch.nn.Linear(20, 256), torch.nn.ReLU()]
if dropout > 0:
qatlamlar.append(torch.nn.Dropout(dropout))
qatlamlar += [torch.nn.Linear(256, 128), torch.nn.ReLU()]
if dropout > 0:
qatlamlar.append(torch.nn.Dropout(dropout))
qatlamlar.append(torch.nn.Linear(128, 2))
return torch.nn.Sequential(*qatlamlar)
def baho(model, X, y):
model.eval()
with torch.no_grad():
chiqish = model(X)
loss = torch.nn.functional.cross_entropy(chiqish, y).item()
aniq = (chiqish.argmax(1) == y).float().mean().item()
return loss, aniq
def orgat(model, Xtr, ytr, Xva, yva, davrlar=300, wd=0.0,
erta=False, sabr=15):
opt = torch.optim.AdamW(model.parameters(), lr=0.003,
weight_decay=wd)
eng_ball, eng_holat, eng_davr, hisob = -1.0, None, 0, 0
for davr in range(1, davrlar + 1):
model.train()
opt.zero_grad()
torch.nn.functional.cross_entropy(model(Xtr), ytr).backward()
opt.step()
if erta:
_, ball = baho(model, Xva, yva)
if ball > eng_ball + 1e-4:
eng_ball, eng_davr, hisob = ball, davr, 0
eng_holat = {k: v.clone()
for k, v in model.state_dict().items()}
else:
hisob += 1
if hisob >= sabr:
break
if erta and eng_holat is not None:
model.load_state_dict(eng_holat)
return davr, eng_davr
return davrlar, davrlar
def main() -> None:
X, y = make_classification(n_samples=1200, n_features=20,
n_informative=8, n_redundant=4,
n_classes=2, flip_y=0.15, class_sep=0.8,
random_state=0)
Xtr, Xrest, ytr, yrest = train_test_split(X, y, test_size=0.5,
random_state=0, stratify=y)
Xva, Xte, yva, yte = train_test_split(Xrest, yrest, test_size=0.5,
random_state=0, stratify=yrest)
sc = StandardScaler().fit(Xtr)
T = lambda M: torch.tensor(sc.transform(M), dtype=torch.float32)
Xtr_t, Xva_t, Xte_t = T(Xtr), T(Xva), T(Xte)
ytr_t = torch.tensor(ytr, dtype=torch.int64)
yva_t = torch.tensor(yva, dtype=torch.int64)
yte_t = torch.tensor(yte, dtype=torch.int64)
print(f" o'quv {len(ytr)}, validatsiya {len(yva)}, test {len(yte)}")
print("\n=== 1. Regularizatsiyasiz ===")
m = tarmoq_yasa()
orgat(m, Xtr_t, ytr_t, Xva_t, yva_t)
l_tr, a_tr = baho(m, Xtr_t, ytr_t)
l_te, a_te = baho(m, Xte_t, yte_t)
print(f" {'to_plam':<12} {'loss':>9} {'aniqlik':>9}")
print(f" {'o_quv':<12} {l_tr:>9.4f} {a_tr:>9.4f}")
print(f" {'test':<12} {l_te:>9.4f} {a_te:>9.4f}")
print(f" farq: {a_tr - a_te:+.4f}")
asos = a_te
print("\n=== 2. Erta to'xtash ===")
m = tarmoq_yasa()
tugagan, eng_davr = orgat(m, Xtr_t, ytr_t, Xva_t, yva_t, erta=True)
l_tr, a_tr = baho(m, Xtr_t, ytr_t)
l_te, a_te = baho(m, Xte_t, yte_t)
print(f" {tugagan}-davrda to'xtadi, eng yaxshisi {eng_davr}-davr")
print(f" o'quv {a_tr:.4f}, test {a_te:.4f}, "
f"farq {a_tr - a_te:+.4f}")
print(f" asosdan: {a_te - asos:+.4f}")
print("\n=== 3. Weight decay ===")
print(f" {'wd':>9} {'o_quv':>8} {'test':>8} {'farq':>8} "
f"{'asosdan':>9}")
for wd in [0.0, 1e-4, 1e-2, 1e-1, 1.0]:
m = tarmoq_yasa()
orgat(m, Xtr_t, ytr_t, Xva_t, yva_t, wd=wd)
_, a_tr = baho(m, Xtr_t, ytr_t)
_, a_te = baho(m, Xte_t, yte_t)
print(f" {wd:>9.0e} {a_tr:>8.4f} {a_te:>8.4f} "
f"{a_tr - a_te:>8.4f} {a_te - asos:>+9.4f}")
print("\n=== 4. Dropout ===")
print(f" {'p':>6} {'o_quv':>8} {'test':>8} {'farq':>8} "
f"{'asosdan':>9}")
for p in [0.0, 0.2, 0.4, 0.6]:
m = tarmoq_yasa(dropout=p)
orgat(m, Xtr_t, ytr_t, Xva_t, yva_t)
_, a_tr = baho(m, Xtr_t, ytr_t)
_, a_te = baho(m, Xte_t, yte_t)
print(f" {p:>6.1f} {a_tr:>8.4f} {a_te:>8.4f} "
f"{a_tr - a_te:>8.4f} {a_te - asos:>+9.4f}")
print("\n=== 5. Kombinatsiya ===")
variantlar = [
("hech narsa", 0.0, 0.0, False),
("erta to'xtash", 0.0, 0.0, True),
("wd 1e-2", 1e-2, 0.0, False),
("dropout 0.3", 0.0, 0.3, False),
("erta + wd", 1e-2, 0.0, True),
("erta + wd + drop", 1e-2, 0.3, True),
]
print(f" {'variant':<20} {'o_quv':>8} {'test':>8} {'farq':>8} "
f"{'asosdan':>9}")
for nom, wd, p, erta in variantlar:
m = tarmoq_yasa(dropout=p)
orgat(m, Xtr_t, ytr_t, Xva_t, yva_t, wd=wd, erta=erta)
_, a_tr = baho(m, Xtr_t, ytr_t)
_, a_te = baho(m, Xte_t, yte_t)
print(f" {nom:<20} {a_tr:>8.4f} {a_te:>8.4f} "
f"{a_tr - a_te:>8.4f} {a_te - asos:>+9.4f}")
print(" ⭐ Har birini ALOHIDA o'lchang, keyin birlashtiring")
if __name__ == "__main__":
main()Natijaning muhim qismi:
o'quv 600, validatsiya 300, test 300
=== 1. Regularizatsiyasiz ===
to_plam loss aniqlik
o_quv 0.0001 1.0000
test 1.4752 0.8100
farq: +0.1900
=== 2. Erta to'xtash ===
39-davrda to'xtadi, eng yaxshisi 24-davr
o'quv 0.9267, test 0.8300, farq +0.0967
asosdan: +0.0200
=== 3. Weight decay ===
wd o_quv test farq asosdan
0e+00 1.0000 0.8100 0.1900 +0.0000
1e-04 1.0000 0.8067 0.1933 -0.0033
1e-02 1.0000 0.8133 0.1867 +0.0033
1e-01 1.0000 0.8067 0.1933 -0.0033
1e+00 1.0000 0.8200 0.1800 +0.0100
=== 4. Dropout ===
p o_quv test farq asosdan
0.0 1.0000 0.8100 0.1900 +0.0000
0.2 1.0000 0.8367 0.1633 +0.0267
0.4 1.0000 0.8633 0.1367 +0.0533
0.6 0.9967 0.8767 0.1200 +0.0667
=== 5. Kombinatsiya ===
variant o_quv test farq asosdan
hech narsa 1.0000 0.8100 0.1900 +0.0000
erta to'xtash 0.9267 0.8300 0.0967 +0.0200
wd 1e-2 1.0000 0.8133 0.1867 +0.0033
dropout 0.3 1.0000 0.8467 0.1533 +0.0367
erta + wd 0.9267 0.8267 0.1000 +0.0167
erta + wd + drop 0.9033 0.8400 0.0633 +0.0300
⭐ Har birini ALOHIDA o'lchang, keyin birlashtiringNima ko'rsatdi: 2.4, 2.6-bo'limlar.
5. To'g'ri va noto'g'ri tushunishlar
| Noto'g'ri fikr | To'g'risi |
|---|---|
| "O'quv aniqligi 1.0 — yaxshi" | Tasodifiy yorliqlarda ham shunday |
| "Dropout inference da ham ishlaydi" | eval() da o'chadi |
"Adam + L2 = AdamW" |
Yo'q, natija boshqa |
"weight decay hamma parametrga" |
bias ga odatda yo'q |
| "Erta to'xtash — to'xtash" | Eng yaxshi og'irliklarni tiklash |
| "Dropout har doim yordam beradi" | Kichik modelda zarar qilishi mumkin |
| "Hammasini birga yoqish yaxshi" | Ta'sirini ajratib bo'lmaydi |
| "Regularizatsiya ma'lumot o'rnini bosadi" | Faqat qisman |
6. Keng tarqalgan xatolar va yechimlari
1. eval() ni unutish
bashorat = model(X_test) # ⚠️ dropout faol
model.eval(); bashorat = model(X_test) # ✅2. Adam ga L2
Adam(params, weight_decay=0.01) # ⚠️
AdamW(params, weight_decay=0.01) # ✅3. Eng yaxshi og'irliklarni tiklamaslik
if sabr_hisobi >= sabr: break # ⚠️ yomon nuqtada
model.load_state_dict(eng_holat) # ✅4. bias ga decay
optim.AdamW(model.parameters(), weight_decay=0.1) # ⚠️
# bias larni alohida guruhga ajrating, wd=0 # ✅5. Chiqishga dropout
nn.Sequential(..., nn.Dropout(0.5), nn.Linear(64, K)) # ✅
nn.Sequential(..., nn.Linear(64, K), nn.Dropout(0.5)) # ⚠️6. Test to'plamida erta to'xtash
ball = baho(model, X_test, y_test) # ⚠️ leakage
ball = baho(model, X_val, y_val) # ✅7. Hammasini birdan yoqish
# wd + dropout + label smoothing + augment birga # ⚠️
# bittadan qo'shib, har birini o'lchang # ✅7. Integratsiya — bu bilim qayerda kerak bo'ladi
- 18-qism (o'tilgan): Validatsiya dizayni
- 20.3-dars (o'tilgan): Model hajmi
- 20.11-dars: To'liq o'rgatish sikli
- 21-qism:
nn.Dropout,AdamW - 24-qism: Rasm augmentatsiyasi
8. Eng yaxshi amaliyotlar
Erta to'xtashni har doim qo'ying.
Eng yaxshi og'irliklarni tiklang.
AdamWishlating.model.eval()ni unutmang.Bir qadamda bitta o'zgarish.
Validatsiya va testni ajrating.
Avval model hajmini tekshiring.
O'quv/test farqini kuzating.
9. Amaliy topshiriq
Vazifa 1: Bashorat qiling
1. # tarmoq tasodifiy yorliqni yod oladimi?
2. # yodlash belgilari?
3. # weight decay formulasi?
4. # SGD da L2 va wd bir xilmi?
5. # Adam da-chi?
6. # dropout o'rgatishda nima qiladi?
7. # inference da-chi?
8. # inverted dropout koeffitsiyenti?
9. # erta to'xtashda nima qilish shart?
10. # bias ga decay kerakmi?
11. # tipik dropout p?
12. # eng kuchli regularizatsiya?Javoblar
- Ha
- O'quv loss ~0, validatsiya o'sadi
w *= (1 - lr*lambda)- Ha
- Yo'q —
AdamWkerak - Neyronlarni tasodifiy o'chiradi
- Hech narsa
1/(1-p)- Eng yaxshi og'irliklarni tiklash
- Odatda yo'q
- 0.2–0.5
- Ko'proq ma'lumot
Vazifa 2: Xatolarni tuzating
1. bashorat = model(X_test)
2. Adam(params, weight_decay=0.01)
3. if sabr_hisobi >= sabr: break
4. ball = baho(model, X_test, y_test) # erta to'xtash uchun
5. nn.Sequential(..., nn.Linear(64, K), nn.Dropout(0.5))Javoblar
1. model.eval(); bashorat = model(X_test)
2. AdamW(params, weight_decay=0.01)
3. model.load_state_dict(eng_holat)
4. ball = baho(model, X_val, y_val)
5. nn.Sequential(..., nn.Dropout(0.5), nn.Linear(64, K))Vazifa 3: Yodlash
Modellang:
- Tasodifiy yorliq
- Davrlar
- Normalar
- Ishonch
Vazifa 4: Weight decay
Modellang:
- Tenglik
- Qisqarish
- Regressiya
- Bias
Vazifa 5: Dropout
Modellang:
- Mexanizm
- O'rtacha
- Rejimlar
- Ansambl
Vazifa 6: Kombinatsiya
Modellang:
- Bazaviy
- Erta to'xtash
- wd
- Birga
Vazifa 7: O'ylash
Regularizatsiyani kuchaytirdingiz: dropout=0.5, wd=0.1, erta to'xtash. Endi o'quv aniqligi ham, test aniqligi ham past. Nima bo'lgan?
Javob
Nima bo'lgan: siz yodlashdan to'liq qarama-qarshi tomonga o'tib ketdingiz. Model endi yetarlicha o'rgana olmayapti — bu underfitting.
Ikki holatni farqlash:
| Belgi | Yodlash | Yetarsiz o'rganish |
|---|---|---|
| O'quv aniqligi | Juda yuqori | Past |
| Test aniqligi | Past | Past |
| Farq | Katta | Kichik |
| Yechim | Regularizatsiya | Regularizatsiyani kamaytirish |
O'quv va test ikkalasi ham past va farq kichik bo'lsa — regularizatsiya haddan oshgan.
Har birining hissasini alohida tekshiring:
variantlar = [
("hammasi", 0.5, 0.1, True),
("dropout yo'q", 0.0, 0.1, True),
("wd yo'q", 0.5, 0.0, True),
("erta to'xtash yo'q", 0.5, 0.1, False),
]
for nom, p, wd, erta in variantlar:
print(nom, orgat_va_bahola(p, wd, erta))Qaysi birini o'chirganda o'quv aniqligi sakrab ko'tarilsa — aybdor o'sha.
Eng ehtimolli aybdor: wd = 0.1
Bu juda katta qiymat. Tipik oraliq 1e-5 dan 1e-2 gacha; 0.1 og'irliklarni nolga siqib, modelni deyarli chiziqli qilib qo'yadi.
print([W.norm().item() for W in model.parameters()])
# hammasi ~0 bo'lsa - wd juda kattaIkkinchi aybdor: erta to'xtash juda erta ishlagan
print("to'xtagan davr:", tugagan, "eng yaxshi davr:", eng_davr)eng_davr juda kichik bo'lsa (masalan 5), sabr yetarli emas yoki lr juda kichik.
Uchinchi: dropout=0.5 kichik model uchun
(256, 128) kabi tarmoqda 0.5 ko'p bo'lmasligi mumkin, lekin (32, 16) da u modelning yarmini yo'q qiladi.
Tuzatish tartibi:
- Hamma regularizatsiyani o'chiring — o'quv aniqligi yuqori bo'lishi kerak. Bo'lmasa muammo boshqa joyda (
lr, arxitektura, ma'lumot) - Erta to'xtashni qaytaring (u deyarli zararsiz)
wdni1e-4dan boshlab oshiring, har qadamda o'lchang- Hali yodlayotgan bo'lsa
dropout=0.2qo'shing - Har qadamda o'quv va test ballini yozing
Asosiy fikr: regularizatsiya — nozik sozlash, kalit emas. Uning maqsadi o'quv va test orasidagi farqni yopish, ikkalasini ham pastga tushirish emas.
Nimani mustahkamlaydi: 2.1, 2.6-bo'limlar.
Xulosa
Bu darsda regularizatsiyani ko'rdik.
Eng muhim uch fikr:
O'quv balli hech narsani isbotlamaydi. 1-misolda tarmoq tasodifiy yorliqlarda ham o'quv to'plamini to'liq yod oldi, test aniqligi esa tangadek qoldi. Yodlashning belgilari aniq: o'quv loss nolga tushadi, validatsiya loss o'sadi, og'irliklar normasi kattalashadi va bashoratlar haddan ishonchli bo'lib qoladi.
Uch usul — uch xil mexanizm.
weight decayog'irliklarni har qadamda(1 - lr*lambda)ga qisqartiradi;dropoutneyronlarni tasodifiy o'chirib, ularning bir-biriga tayanishini buzadi; erta to'xtash esa optimallashtirish vaqtini cheklaydi. Ular bir-birining o'rnini bosmaydi, shuning uchun ko'pincha birga ishlatiladi.Ikkita xato eng qimmatga tushadi. Birinchisi —
model.eval()ni unutish:dropoutinference da faol qolsa, bashoratlar tasodifiy bo'ladi. Ikkinchisi —Adamgaweight_decayberish:L2jarimasi adaptiv bo'linishdan o'tib buziladi, shuning uchunAdamWkerak. Erta to'xtashda esa eng yaxshi og'irliklarni tiklashni unutmang, aks holda siz eng yomon nuqtada to'xtaysiz.
Keyingi darsda birinchi to'liq tarmoq: o'rgatish siklini boshdan-oxir quramiz — batch, davr, validatsiya, jadval, erta to'xtash va loglash bir joyda.
Izohlar · 0
Izoh yozish uchun kiring.