23.8-dars: RNN — rekurrent neyron tarmoqlar

FORMULA (har qadam t = 1..T):
  h_t = tanh(W x_t + U h_{t-1} + b)
  h_0 = 0 (odatda)

  x_t: (d_kir,)       - t-so'zning embeddingi
  h_t: (d_yash,)      - "shu paytgacha o'qilganlarning xulosasi"
  W:   (d_yash, d_kir)
  U:   (d_yash, d_yash)  - o'tmishdan kelgan holatni aralashtiradi
  b:   (d_yash,)

OG'IRLIK ULASHISH:
  bir xil W, U, b HAR qadamda ishlatiladi
  -> parametrlar soni ketma-ketlik uzunligiga bog'liq EMAS
  -> istalgan uzunlikdagi matnni o'qiy oladi

YOYILGAN KO'RINISH:
  x_1     x_2     x_3           x_T
   |       |       |             |
  [RNN]->[RNN]->[RNN]-> ... ->[RNN] -> h_T -> Linear -> sinf
  h_1     h_2     h_3           h_T

NIMA UCHUN tanh:
  h_t ni (-1, 1) oralig'ida ushlaydi - qiymatlar portlamaydi
  lekin to'yinganda (|h| ~ 1) hosilasi ~ 0 - bu gradient uchun muammo (2.4)

rnn = nn.RNN(input_size=d_kir, hidden_size=d_yash, batch_first=True)

KIRISH:  x (B, T, d_kir)       batch_first=True bo'lsa
         h0 (1, B, d_yash)     ixtiyoriy, sukut - nollar
CHIQISH: out (B, T, d_yash)    HAR qadamdagi h_t
         h_n (1, B, d_yash)    faqat OXIRGI h_T
         (1 - qatlamlar soni * yo'nalishlar soni)

PARAMETRLAR:
  weight_ih_l0 (d_yash, d_kir)    - W
  weight_hh_l0 (d_yash, d_yash)   - U
  bias_ih_l0, bias_hh_l0          - ikkita bias (yig'indisi = b)
  jami: d_yash * (d_kir + d_yash) + 2 * d_yash

TENGLIK: out[:, -1] == h_n[0]   (padding BO'LMASA)

SUKUT batch_first=False:
  x (T, B, d_kir) - ko'p xatolarning manbai

BPTT (backpropagation through time):
  yoyilgan tarmoq - T qatlamli chuqur tarmoq, og'irliklar ulashilgan
  oddiy orqaga tarqalish (20-qism) shu yoyilgan grafda ishlaydi
  U ning gradienti = HAR qadamdagi hissalar yig'indisi

ZANJIR QOIDASI:
  dL/dh_t = dL/dh_T * (dh_T/dh_{T-1}) * ... * (dh_{t+1}/dh_t)

  dh_{k+1}/dh_k = diag(1 - h_{k+1}^2) * U      (tanh hosilasi * U)

  -> dL/dh_t da (T - t) ta matritsa KO'PAYTMASI

TORCH DA:
  autograd buni o'zi qiladi - loss.backward()
  xotira: T qadamning hamma oraliq qiymatlari saqlanadi
  -> juda uzun ketma-ketlikda "kesilgan BPTT" (truncated):
     har k qadamda h.detach() - gradient faqat k qadam orqaga boradi

KO'PAYTMA XULQI (soddalashtirilgan):
  || dL/dh_t || ~ (rho(U) * tanh')^(T - t) * || dL/dh_T ||
  rho(U) - U ning spektral radiusi (eng katta xos qiymat moduli)

  rho * tanh' < 1  -> eksponensial SO'NISH
     50 qadamda 0.6^50 ~ 1e-11 - boshidagi so'z gradienti amalda nol
  rho * tanh' > 1  -> eksponensial PORTLASH
     loss NaN, og'irliklar sakraydi

AMALIY OQIBAT:
  so'nish: model ketma-ketlik BOSHIDAGI ma'lumotni o'rgana olmaydi
    ("uzoq bog'liqlik" muammosi)
  portlash: o'rgatish beqaror - clipping bilan davolanadi 2.5-bob

SUKUT INIT (nn.RNN):
  og'irliklar U(-1/sqrt(h), 1/sqrt(h)) -> rho(U) odatda < 1
  -> sukut bo'yicha so'nish tomonida

YECHIMLAR:
  portlash -> gradient clipping
  so'nish  -> arxitektura: LSTM va GRU (keyingi dars) - "darvozalar"
              va qo'shish orqali o'tadigan xotira yo'li

21.4-DARSDAN:
  norma = sqrt(sum ||g_p||^2)          (barcha parametrlar bo'yicha)
  agar norma > max_norm:  g <- g * max_norm / norma

  torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
  loss.backward() dan KEYIN, opt.step() dan OLDIN
  qaytaradi: clipping dan OLDINGI norma - uni qayd qiling

XOSSALARI:
  gradient YO'NALISHI saqlanadi, faqat uzunligi cheklanadi
  portlashni davolaydi, SO'NISHNI DAVOLAMAYDI
  (kichik gradientni kattalashtirmaydi)

max_norm NI TANLASH:
  bir necha yuz qadam norma tarixini qarang
  odatiy normalardan biroz yuqori (masalan 90-foizil)
  RNN larda 0.5 - 5 oralig'i keng tarqalgan

KLASSIFIKATSIYA:
  sharh -> RNN -> h_T -> Linear -> 3 logit
  h_T - butun sharhni o'qib bo'lgan "xulosa"

PADDING TUZOG'I (o'ngdan padding):
  sharh:  [57, 31, 102,  0,  0,  0]     uzunlik 3
  out[:, -1] -> h_6: model 3 ta PAD ni ham o'qib bo'lgan holat
  kerakli holat -> h_3

TO'G'RI YO'LLAR:
  1. gather: out[arange(B), uzunlik - 1]
  2. pack_padded_sequence(x, uzunlik, batch_first=True,
                          enforce_sorted=False)
     -> RNN padding ni umuman ko'rmaydi, h_n - har sharhning o'z oxiri
  ikkalasi bir xil natija beradi (4-misolda tekshiramiz)

ALTERNATIVA:
  barcha h_t larning maskali o'rtachasi yoki maksimumi (23.7 dagi kabi)
  - gradient boshidagi so'zlarga qisqa yo'l bilan boradi (o'lchab tanlang)

import torch
import torch.nn as nn

rnn = nn.RNN(input_size=32, hidden_size=64, batch_first=True)
out, h_n = rnn(x)                      # x (B, T, 32)
# out (B, T, 64) - har qadam, h_n (1, B, 64) - oxirgi

# qo'lda bitta qadam
W, U = rnn.weight_ih_l0, rnn.weight_hh_l0
b = rnn.bias_ih_l0 + rnn.bias_hh_l0
h = torch.tanh(x[:, t] @ W.T + h @ U.T + b)

# padding bilan to'g'ri oxirgi holat
h_oxirgi = out[torch.arange(B), uzunlik - 1]
paket = nn.utils.rnn.pack_padded_sequence(
    emb(ids), uzunlik, batch_first=True, enforce_sorted=False)
_, h_n = rnn(paket)                    # h_n[0] == h_oxirgi

# o'rgatish qadami
loss.backward()
norma = nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
opt.step()

h_t = tanh(W x_t + U h_{t-1} + b), og'irliklar hamma qadamda bir xil
out - hamma h_t, h_n - oxirgisi; batch_first=True
BPTT: dL/dh_t da (T - t) ta matritsa ko'paytmasi
rho(U) * tanh' < 1 -> so'nish, > 1 -> portlash
clipping: backward va step orasida, faqat portlashga qarshi
padding: gather (uzunlik - 1) yoki pack_padded_sequence

"""h_t = tanh(W x_t + U h_{t-1} + b) ni qo'lda hisoblab, nn.RNN bilan solishtirish."""

import torch
import torch.nn as nn


def main() -> None:
    torch.manual_seed(0)
    B, T, d_kir, d_yash = 2, 5, 4, 3
    rnn = nn.RNN(d_kir, d_yash, batch_first=True)
    x = torch.randn(B, T, d_kir)

    print("=== 1. Parametrlar ===")
    for nom, p in rnn.named_parameters():
        print(f"  {nom:<14} {str(tuple(p.shape)):>8}")
    jami = sum(p.numel() for p in rnn.parameters())
    formula = d_yash * (d_kir + d_yash) + 2 * d_yash
    print(f"  jami: {jami}, formula d_yash*(d_kir+d_yash) + 2*d_yash = "
          f"{formula}")

    print("\n=== 2. nn.RNN chiqishlari ===")
    out, h_n = rnn(x)
    print(f"  x {tuple(x.shape)} -> out {tuple(out.shape)}, "
          f"h_n {tuple(h_n.shape)}")
    print(f"  out[:, -1] == h_n[0]: {torch.allclose(out[:, -1], h_n[0])}")

    print("\n=== 3. Qo'lda hisoblash ===")
    W, U = rnn.weight_ih_l0, rnn.weight_hh_l0
    b = rnn.bias_ih_l0 + rnn.bias_hh_l0
    h = torch.zeros(B, d_yash)
    qolda = []
    with torch.no_grad():
        for t in range(T):
            h = torch.tanh(x[:, t] @ W.T + h @ U.T + b)
            qolda.append(h)
            print(f"  t={t + 1}: h[0] = {h[0].numpy().round(4)}")
    qolda = torch.stack(qolda, 1)
    farq = (qolda - out).abs().max().item()
    print(f"  qo'lda va nn.RNN maksimal farqi: {farq:.2e}")

    print("\n=== 4. Og'irlik ulashish: uzunlikka bog'liq emas ===")
    for T2 in (5, 50, 500):
        o, _ = rnn(torch.randn(1, T2, d_kir))
        print(f"  T = {T2:>3}: out {tuple(o.shape)}, parametrlar {jami}")

    print("\n=== 5. batch_first tuzog'i ===")
    rnn_sukut = nn.RNN(d_kir, d_yash)            # batch_first=False
    rnn_sukut.load_state_dict(rnn.state_dict())
    o_xato, _ = rnn_sukut(x)                     # x (B, T, d) - lekin
    print(f"  batch_first=False, x {tuple(x.shape)} -> out "
          f"{tuple(o_xato.shape)}")               # (T, B, d) deb o'qiydi
    print(f"  xato jim: shakl mos, lekin T={B} qadam, B={T} 'sharh'")
    o_togri, _ = rnn_sukut(x.transpose(0, 1))
    print(f"  transpose bilan natija bir xil: "
          f"{torch.allclose(o_togri.transpose(0, 1), out)}")
    print("  ⭐ nn.RNN - qo'lda yozilgan bitta formula, T marta takrorlangan")


if __name__ == "__main__":
    main()

=== 1. Parametrlar ===
  weight_ih_l0     (3, 4)
  weight_hh_l0     (3, 3)
  bias_ih_l0         (3,)
  bias_hh_l0         (3,)
  jami: 27, formula d_yash*(d_kir+d_yash) + 2*d_yash = 27

=== 2. nn.RNN chiqishlari ===
  x (2, 5, 4) -> out (2, 5, 3), h_n (1, 2, 3)
  out[:, -1] == h_n[0]: True

=== 3. Qo'lda hisoblash ===
  t=1: h[0] = [ 0.0821 -0.1653  0.6807]
  t=2: h[0] = [ 0.2422 -0.1528  0.7892]
  t=3: h[0] = [ 0.7004 -0.1101  0.8675]
  t=4: h[0] = [-0.4132  0.53    0.7104]
  t=5: h[0] = [0.111  0.5738 0.802 ]
  qo'lda va nn.RNN maksimal farqi: 5.96e-08

=== 4. Og'irlik ulashish: uzunlikka bog'liq emas ===
  T =   5: out (1, 5, 3), parametrlar 27
  T =  50: out (1, 50, 3), parametrlar 27
  T = 500: out (1, 500, 3), parametrlar 27

=== 5. batch_first tuzog'i ===
  batch_first=False, x (2, 5, 4) -> out (2, 5, 3)
  xato jim: shakl mos, lekin T=2 qadam, B=5 'sharh'
  transpose bilan natija bir xil: True
  ⭐ nn.RNN - qo'lda yozilgan bitta formula, T marta takrorlangan

"""dL/dh_t normasi vaqt bo'ylab: U ning spektral radiusiga bog'liqlik."""

import numpy as np
import torch
import torch.nn as nn


def yoy(rnn, x):
    """Qo'lda yoyish - har h_t uchun gradientni saqlash."""
    W, U = rnn.weight_ih_l0, rnn.weight_hh_l0
    b = rnn.bias_ih_l0 + rnn.bias_hh_l0
    h = torch.zeros(x.shape[0], U.shape[0])
    holatlar = []
    for t in range(x.shape[1]):
        h = torch.tanh(x[:, t] @ W.T + h @ U.T + b)
        h.retain_grad()
        holatlar.append(h)
    return holatlar


def main() -> None:
    H, T, B = 32, 50, 16
    torch.manual_seed(0)
    x = torch.randn(B, T, 8)
    nuqtalar = [49, 40, 30, 20, 10, 0]

    print("=== 1. ||dL/dh_t|| (T = 50, loss faqat h_50 dan) ===")
    print(f"  {'U':<8} {'rho(U)':>6} " + " ".join(f"{'t=' + str(t + 1):>8}"
                                              for t in nuqtalar)
          + f" {'to_yingan':>9}")
    natija = {}
    for nom, r in [("sukut", None), ("r=0.5", 0.5), ("r=1.0", 1.0),
                   ("r=1.5", 1.5), ("r=3.0", 3.0)]:
        torch.manual_seed(1)
        rnn = nn.RNN(8, H, batch_first=True)
        if r is not None:
            q, _ = torch.linalg.qr(torch.randn(H, H))
            rnn.weight_hh_l0.data = r * q          # xos qiymatlar moduli = r
        holatlar = yoy(rnn, x)
        loss = (holatlar[-1] ** 2).sum() / B
        loss.backward()
        n = [holatlar[t].grad.norm().item() for t in range(T)]
        rho = torch.linalg.eigvals(rnn.weight_hh_l0).abs().max().item()
        toy = torch.stack(holatlar).abs().gt(0.99).float().mean().item()
        natija[nom] = (n, rnn)
        print(f"  {nom:<8} {rho:>6.2f} " + " ".join(f"{n[t]:>8.1e}"
                                                    for t in nuqtalar)
              + f" {toy:>9.1%}")

    print("\n=== 2. Bir qadamdagi o'rtacha ko'paytuvchi ===")
    for nom, (n, _) in natija.items():
        k = (n[0] / n[-1]) ** (1 / (T - 1))
        holat = "so'nish" if k < 1 else "portlash"
        print(f"  {nom:<8} har qadam orqaga x{k:.3f}  -> 49 qadamda "
              f"x{k ** 49:.1e}  ({holat})")

    print("\n=== 3. Parametr gradienti va clipping (r=3.0) ===")
    rnn = natija["r=3.0"][1]
    rnn.zero_grad()
    holatlar = yoy(rnn, x)
    ((holatlar[-1] ** 2).sum() / B).backward()
    oldin = torch.cat([p.grad.flatten() for p in rnn.parameters()]).clone()
    norma = nn.utils.clip_grad_norm_(rnn.parameters(), max_norm=1.0)
    keyin = torch.cat([p.grad.flatten() for p in rnn.parameters()])
    cos = torch.dot(oldin, keyin) / (oldin.norm() * keyin.norm())
    print(f"  clip_grad_norm_ qaytardi (oldingi norma): {norma.item():.1f}")
    print(f"  clipping dan keyingi norma: {keyin.norm().item():.4f}")
    print(f"  yo'nalish saqlandimi: cos = {cos.item():.6f}")

    print("\n=== 4. Clipping so'nishni davolamaydi (sukut init) ===")
    n0 = natija["sukut"][0]
    print(f"  ||dL/dh_1|| = {n0[0]:.1e}; clipping faqat KATTA normani "
          f"kichraytiradi")
    print(f"  bu gradient 1.0 dan {1.0 / n0[0]:.0e} marta kichik - "
          f"clipping unga tegmaydi")
    print("  ⭐ So'nish ham, portlash ham - (T - t) ta matritsa ko'paytmasi")


if __name__ == "__main__":
    main()

=== 1. ||dL/dh_t|| (T = 50, loss faqat h_50 dan) ===
  U        rho(U)     t=50     t=41     t=31     t=21     t=11      t=1 to_yingan
  sukut      0.61  9.2e-01  2.2e-03  5.2e-06  8.5e-09  1.6e-11  3.8e-14      0.0%
  r=0.5      0.50  8.9e-01  6.0e-04  2.1e-07  7.8e-11  3.0e-14  1.1e-17      0.0%
  r=1.0      1.00  1.3e+00  1.4e-01  2.1e-02  3.4e-03  5.4e-04  8.6e-05      0.0%
  r=1.5      1.50  1.8e+00  1.2e+00  1.6e+00  2.9e+00  3.6e+00  1.4e+01      0.8%
  r=3.0      3.00  2.4e+00  1.7e+01  4.5e+02  4.5e+03  6.1e+04  2.4e+06     29.1%

=== 2. Bir qadamdagi o'rtacha ko'paytuvchi ===
  sukut    har qadam orqaga x0.533  -> 49 qadamda x4.2e-14  (so'nish)
  r=0.5    har qadam orqaga x0.452  -> 49 qadamda x1.2e-17  (so'nish)
  r=1.0    har qadam orqaga x0.822  -> 49 qadamda x6.8e-05  (so'nish)
  r=1.5    har qadam orqaga x1.042  -> 49 qadamda x7.6e+00  (portlash)
  r=3.0    har qadam orqaga x1.326  -> 49 qadamda x1.0e+06  (portlash)

=== 3. Parametr gradienti va clipping (r=3.0) ===
  clip_grad_norm_ qaytardi (oldingi norma): 6484182.5
  clipping dan keyingi norma: 1.0000
  yo'nalish saqlandimi: cos = 1.000000

=== 4. Clipping so'nishni davolamaydi (sukut init) ===
  ||dL/dh_1|| = 3.8e-14; clipping faqat KATTA normani kichraytiradi
  bu gradient 1.0 dan 3e+13 marta kichik - clipping unga tegmaydi
  ⭐ So'nish ham, portlash ham - (T - t) ta matritsa ko'paytmasi

"""Yorliq = ketma-ketlikning BIRINCHI belgisi; qolgani shovqin."""

import numpy as np
import torch
import torch.nn as nn


def toplam(n, L, g):
    """1-belgi: 0 yoki 1 (yorliq). Qolgan L-1 belgi: 2..9 tasodifiy."""
    x = torch.randint(2, 10, (n, L), generator=g)
    y = torch.randint(0, 2, (n,), generator=g)
    x[:, 0] = y
    return x, y


class Eslovchi(nn.Module):
    def __init__(self, yashirin=32):
        super().__init__()
        self.emb = nn.Embedding(10, 8)
        self.rnn = nn.RNN(8, yashirin, batch_first=True)
        self.chiq = nn.Linear(yashirin, 2)

    def forward(self, x):
        _, h_n = self.rnn(self.emb(x))
        return self.chiq(h_n[0])                 # faqat oxirgi holat


def orgat_bahola(L, seed, qadamlar=300):
    torch.manual_seed(seed)
    g = torch.Generator().manual_seed(seed)
    model = Eslovchi()
    opt = torch.optim.Adam(model.parameters(), lr=0.005)
    for _ in range(qadamlar):
        x, y = toplam(64, L, g)
        loss = nn.functional.cross_entropy(model(x), y)
        opt.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
    x, y = toplam(1000, L, torch.Generator().manual_seed(999))
    with torch.no_grad():
        return (model(x).argmax(1) == y).float().mean().item()


def main() -> None:
    uzunliklar = [5, 10, 20, 40, 80]
    seedlar = [0, 1, 2]
    x, y = toplam(4, 8, torch.Generator().manual_seed(3))
    print("=== 1. Vazifa ===")
    for i in range(4):
        print(f"  {x[i].tolist()} -> yorliq {int(y[i])}")
    print("  tasodifiy taxmin aniqligi: 0.5")

    print("\n=== 2. Test aniqligi (300 qadam, batch 64, clipping 1.0) ===")
    print(f"  {'L':>4} " + " ".join(f"{'seed ' + str(s):>8}" for s in seedlar)
          + f" {'o_rtacha':>9} {'>0.9':>5}")
    natija = {}
    for L in uzunliklar:
        a = np.array([orgat_bahola(L, s) for s in seedlar])
        natija[L] = a
        print(f"  {L:>4} " + " ".join(f"{v:>8.3f}" for v in a)
              + f" {a.mean():>9.3f} {int((a > 0.9).sum()):>3}/3")

    print("\n=== 3. L = 5 ga nisbatan juftlashgan farq (seed bo'yicha) ===")
    for L in uzunliklar[1:]:
        d = natija[L] - natija[5]
        se = d.std(ddof=1) / np.sqrt(len(d))
        x_ = "sezilarli" if abs(d.mean()) > 2 * se else "sezilarli emas"
        print(f"  L={L:>3}: {d.mean():+.3f}  SE {se:.3f}  {x_}")
    tasodifiy = [L for L in uzunliklar if natija[L].max() < 0.6]
    if tasodifiy:
        print(f"  L >= {min(tasodifiy)} da hech bir seed tasodifiy "
              f"taxmindan chiqmadi")
    print("  ⭐ Uzunlik oshsa - boshidagi signalning gradienti so'nadi")


if __name__ == "__main__":
    main()

=== 1. Vazifa ===
  [0, 2, 3, 5, 2, 2, 2, 7] -> yorliq 0
  [1, 5, 4, 5, 3, 3, 4, 7] -> yorliq 1
  [1, 8, 2, 6, 6, 7, 9, 8] -> yorliq 1
  [0, 4, 3, 4, 8, 4, 4, 9] -> yorliq 0
  tasodifiy taxmin aniqligi: 0.5

=== 2. Test aniqligi (300 qadam, batch 64, clipping 1.0) ===
     L   seed 0   seed 1   seed 2  o_rtacha  >0.9
     5    1.000    1.000    1.000     1.000   3/3
    10    1.000    1.000    1.000     1.000   3/3
    20    1.000    1.000    0.483     0.828   2/3
    40    1.000    0.489    0.503     0.664   1/3
    80    0.505    0.504    0.503     0.504   0/3

=== 3. L = 5 ga nisbatan juftlashgan farq (seed bo'yicha) ===
  L= 10: +0.000  SE 0.000  sezilarli emas
  L= 20: -0.172  SE 0.172  sezilarli emas
  L= 40: -0.336  SE 0.168  sezilarli emas
  L= 80: -0.496  SE 0.001  sezilarli
  L >= 80 da hech bir seed tasodifiy taxmindan chiqmadi
  ⭐ Uzunlik oshsa - boshidagi signalning gradienti so'nadi

"""RNN bilan sharh klassifikatsiyasi: out[:, -1] (xato) va to'g'ri oxirgi holat."""

from collections import Counter

import numpy as np
import torch
import torch.nn as nn
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

IJOBIY = ["yaxshi", "zo'r", "a'lo", "ajoyib", "sifatli", "qulay",
          "chiroyli", "mustahkam"]
SALBIY = ["yomon", "sifatsiz", "noqulay", "xunuk", "nosoz", "mo'rt",
          "sust", "chatoq"]
NEYTRAL = ["o'rtacha", "oddiy", "normal", "odatiy", "kutilgandek"]
JIHAT = ["sifati", "narxi", "ekrani", "batareyasi", "ovozi", "qadoqlash",
         "yetkazib berish", "sotuvchi", "dizayni", "materiali"]
MAHSULOT = ["telefon", "noutbuk", "quloqchin", "soat", "choynak", "kurtka",
            "krossovka", "sumka", "planshet", "kolonka"]
BOSH = ["kecha", "o'tgan hafta", "bir oy oldin", "do'kondan", "internetdan",
        "sovg'a uchun", "akam uchun", ""]
FEL = ["oldim", "buyurtma qildim", "sotib oldim", "ishlatib ko'rdim"]
YAKUN = {1: ["tavsiya qilaman", "yana olaman", "hammaga maslahat beraman"],
         -1: ["tavsiya qilmayman", "boshqa olmayman", "pulimga achinaman"],
         0: ["hozircha shu", "vaqt ko'rsatadi", "yana ishlatib ko'ramiz"]}
ULUSH = {1: [0.8, 0.1, 0.1], -1: [0.1, 0.8, 0.1], 0: [0.15, 0.15, 0.7]}


def gap(qutb, rng):
    j = rng.choice(JIHAT)
    juda = "juda " if rng.random() < 0.3 else ""
    r = rng.random()
    if qutb == 1:
        if r < 0.2:
            return f"{j} {rng.choice(SALBIY)} emas"
        if r < 0.3:
            return f"{j} bilan muammo yo'q"
        return f"{j} {juda}{rng.choice(IJOBIY)}"
    if qutb == -1:
        if r < 0.25:
            return f"{j} {rng.choice(IJOBIY)} emas"
        if r < 0.35:
            return f"{j} bilan muammo bor"
        return f"{j} {juda}{rng.choice(SALBIY)}"
    return f"{j} {rng.choice(NEYTRAL)}"


def sharh(rng):
    kayfiyat = int(rng.integers(-1, 2))
    qismlar = [f"{rng.choice(BOSH)} {rng.choice(MAHSULOT)} "
               f"{rng.choice(FEL)}".strip()]
    ballar = []
    for t in range(int(rng.integers(1, 5))):
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        bog = rng.choice(["lekin", "va", "ammo", "ham", ""]) if t else ""
        qismlar.append(f"{bog} {gap(q, rng)}".strip())
        ballar.append(q)
    if rng.random() < 0.5:
        q = int(rng.choice([1, -1, 0], p=ULUSH[kayfiyat]))
        qismlar.append(rng.choice(YAKUN[q]))
        ballar.append(q)
    ball = np.mean(ballar) + rng.normal(0, 0.1)
    return " ".join(qismlar), 2 if ball > 0.35 else (0 if ball < -0.35 else 1)


def imlo_xato(matn, ulush, rng):
    chiqish = []
    for s in matn.split():
        if len(s) >= 4 and rng.random() < ulush:
            i = int(rng.integers(1, len(s) - 1))
            tur = int(rng.integers(0, 3))
            if tur == 0:
                s = s[:i] + s[i + 1:]
            elif tur == 1:
                s = s[:i] + s[i] + s[i:]
            else:
                s = s[:i - 1] + s[i] + s[i - 1] + s[i + 1:]
        chiqish.append(s)
    return " ".join(chiqish)


def korpus(n, seed=0, xato=0.07):
    rng = np.random.default_rng(seed)
    toza, y = zip(*[sharh(rng) for _ in range(n)])
    rng2 = np.random.default_rng(seed + 1000)
    return [imlo_xato(t, xato, rng2) for t in toza], np.array(y), list(toza)


def toldir(ketmalar):
    L = max(len(k) for k in ketmalar)
    ids = torch.zeros(len(ketmalar), L, dtype=torch.long)
    for i, k in enumerate(ketmalar):
        ids[i, :len(k)] = torch.tensor(k)
    return ids, torch.tensor([len(k) for k in ketmalar])


class SharhRNN(nn.Module):
    def __init__(self, V, togri):
        super().__init__()
        self.togri = togri
        self.emb = nn.Embedding(V, 32, padding_idx=0)
        self.rnn = nn.RNN(32, 64, batch_first=True)
        self.chiq = nn.Linear(64, 3)

    def forward(self, ids, uzunlik):
        out, _ = self.rnn(self.emb(ids))
        if self.togri:
            h = out[torch.arange(len(uzunlik)), uzunlik - 1]
        else:
            h = out[:, -1]                       # padding ni ham o'qigan
        return self.chiq(h)


def orgat(k_tr, y_tr, V, togri, seed, davrlar=8):
    torch.manual_seed(seed)
    g = torch.Generator().manual_seed(seed)
    model = SharhRNN(V, togri)
    opt = torch.optim.Adam(model.parameters(), lr=0.003)
    yt = torch.tensor(y_tr)
    normlar = []
    for _ in range(davrlar):
        tartib = torch.randperm(len(k_tr), generator=g).tolist()
        for i in range(0, len(tartib), 64):
            b = tartib[i:i + 64]
            opt.zero_grad()
            nn.functional.cross_entropy(
                model(*toldir([k_tr[j] for j in b])), yt[b]).backward()
            normlar.append(
                nn.utils.clip_grad_norm_(model.parameters(), 1.0).item())
            opt.step()
    model.eval()
    return model, np.array(normlar)


def main() -> None:
    matnlar, y, _ = korpus(3000)
    Xtr, Xte, ytr, yte = matnlar[:2400], matnlar[2400:], y[:2400], y[2400:]
    c = Counter(s for m in Xtr for s in m.split())
    lugat = {s: i + 2 for i, s in enumerate(sorted(s for s, k in c.items()
                                                   if k >= 2))}
    V = len(lugat) + 2
    kod = lambda ms: [[lugat.get(s, 1) for s in m.split()] for m in ms]
    k_tr, k_te = kod(Xtr), kod(Xte)
    ids_te, uz_te = toldir(k_te)

    print("=== 1. Padding va oxirgi holat ===")
    torch.manual_seed(0)
    m = SharhRNN(V, togri=True)
    ids, uz = toldir(k_te[:3])
    with torch.no_grad():
        out, _ = m.rnn(m.emb(ids))
        paket = nn.utils.rnn.pack_padded_sequence(
            m.emb(ids), uz, batch_first=True, enforce_sorted=False)
        _, h_n = m.rnn(paket)
    togri = out[torch.arange(3), uz - 1]
    print(f"  uzunliklar: {uz.tolist()}, padding bilan L = {ids.shape[1]}")
    print(f"  gather va pack_padded_sequence farqi: "
          f"{(togri - h_n[0]).abs().max().item():.2e}")
    for i in range(3):
        f = (out[i, -1] - togri[i]).norm().item()
        print(f"  sharh {i}: ||out[:, -1] - h_(uzunlik)|| = {f:.3f}")
    print("  eng uzun sharhda farq 0, qisqalarida - padding ni o'qigan holat")

    print("\n=== 2. O'rgatish (8 davr, clipping 1.0, 3 seed) ===")
    natija, normlar, modellar = {}, {}, {}
    for nom, t in [("out[:, -1]", False), ("to'g'ri oxirgi", True)]:
        natija[nom], normlar[nom] = [], []
        for s in range(3):
            model, n = orgat(k_tr, ytr, V, t, seed=s)
            with torch.no_grad():
                p = model(ids_te, uz_te).argmax(1).numpy()
            natija[nom].append((p == yte).mean())
            normlar[nom].append(n)
            modellar[nom] = model
        a = np.array(natija[nom])
        print(f"  {nom:<15} {a.mean():.4f} ± {a.std(ddof=1):.4f}  "
              f"(seedlar: {', '.join(f'{v:.4f}' for v in a)})")
    d = np.array(natija["to'g'ri oxirgi"]) - np.array(natija["out[:, -1]"])
    se = d.std(ddof=1) / np.sqrt(len(d))
    print(f"  farq: {d.mean():+.4f}, SE {se:.4f}, "
          f"{'sezilarli' if abs(d.mean()) > 2 * se else 'sezilarli emas'}")

    print("\n=== 3. Gradient normalari (clipping dan oldin) ===")
    for nom, ns in normlar.items():
        n = np.concatenate(ns)
        print(f"  {nom:<15} mediana {np.median(n):.2f}, 90-foizil "
              f"{np.percentile(n, 90):.2f}, max {n.max():.2f}, "
              f"kesilgan qadamlar {np.mean(n > 1.0):.1%}")

    print("\n=== 4. Ma'lumot uchun: TF-IDF + bigram (xuddi shu bo'linish) ===")
    tf = make_pipeline(TfidfVectorizer(token_pattern=r"[\w']+",
                                       ngram_range=(1, 2)),
                       LogisticRegression(C=10, max_iter=3000))
    a_tf = tf.fit(Xtr, ytr).score(Xte, yte)
    rnn_a = np.array(natija["to'g'ri oxirgi"])
    print(f"  TF-IDF: {a_tf:.4f}, RNN (to'g'ri): {rnn_a.mean():.4f}")
    print(f"  RNN - TF-IDF: {rnn_a.mean() - a_tf:+.4f} (bitta bo'linish; "
          f"SE faqat seed farqini o'lchaydi: "
          f"{rnn_a.std(ddof=1) / np.sqrt(3):.4f})")

    print("\n=== 5. Tartibni ko'radimi? ===")
    SINF = ["salbiy", "neytral", "ijobiy"]
    model = modellar["to'g'ri oxirgi"]
    for m_ in ["telefon oldim sifati yaxshi",
               "telefon oldim sifati yaxshi emas",
               "telefon oldim sifati yomon emas",
               "telefon oldim emas sifati yomon"]:
        with torch.no_grad():
            p = model(*toldir(kod([m_]))).argmax(1).item()
        print(f"  {m_:<34} -> {SINF[p]}")
    print("  ⭐ RNN tartibni ko'radi - lekin padding ni to'g'ri hisobga oling")


if __name__ == "__main__":
    main()

=== 1. Padding va oxirgi holat ===
  uzunliklar: [20, 16, 7], padding bilan L = 20
  gather va pack_padded_sequence farqi: 2.68e-07
  sharh 0: ||out[:, -1] - h_(uzunlik)|| = 0.000
  sharh 1: ||out[:, -1] - h_(uzunlik)|| = 2.606
  sharh 2: ||out[:, -1] - h_(uzunlik)|| = 3.222
  eng uzun sharhda farq 0, qisqalarida - padding ni o'qigan holat

=== 2. O'rgatish (8 davr, clipping 1.0, 3 seed) ===
  out[:, -1]      0.7833 ± 0.0203  (seedlar: 0.7600, 0.7933, 0.7967)
  to'g'ri oxirgi  0.8272 ± 0.0135  (seedlar: 0.8150, 0.8250, 0.8417)
  farq: +0.0439, SE 0.0068, sezilarli

=== 3. Gradient normalari (clipping dan oldin) ===
  out[:, -1]      mediana 1.69, 90-foizil 4.45, max 23.72, kesilgan qadamlar 74.5%
  to'g'ri oxirgi  mediana 0.91, 90-foizil 1.56, max 3.59, kesilgan qadamlar 41.4%

=== 4. Ma'lumot uchun: TF-IDF + bigram (xuddi shu bo'linish) ===
  TF-IDF: 0.8083, RNN (to'g'ri): 0.8272
  RNN - TF-IDF: +0.0189 (bitta bo'linish; SE faqat seed farqini o'lchaydi: 0.0078)

=== 5. Tartibni ko'radimi? ===
  telefon oldim sifati yaxshi        -> ijobiy
  telefon oldim sifati yaxshi emas   -> salbiy
  telefon oldim sifati yomon emas    -> ijobiy
  telefon oldim emas sifati yomon    -> salbiy
  ⭐ RNN tartibni ko'radi - lekin padding ni to'g'ri hisobga oling

rnn = nn.RNN(32, 64); rnn(x)            # x (B, T, 32)              # ⚠️
rnn = nn.RNN(32, 64, batch_first=True)                               # ✅

h = out[:, -1]                                                       # ⚠️
h = out[torch.arange(B), uzunlik - 1]    # yoki pack_padded_sequence  # ✅

logit = chiq(h_n)                        # (1, B, 64) -> (1, B, 3)   # ⚠️
logit = chiq(h_n[-1])                    # (B, 64)                   # ✅

opt.step(); clip_grad_norm_(model.parameters(), 1.0)                 # ⚠️
loss.backward(); clip_grad_norm_(model.parameters(), 1.0); opt.step()  # ✅

clip_grad_norm_(model.parameters(), 1.0)                             # ⚠️
normlar.append(clip_grad_norm_(model.parameters(), 1.0).item())      # ✅

pack_padded_sequence(x, uz, batch_first=True)   # saralanmagan -> xato  # ⚠️
pack_padded_sequence(x, uz, batch_first=True, enforce_sorted=False)     # ✅

out, h = rnn(x_10000_qadam)              # xotira, so'nish           # ⚠️
for bolak in bolaklar: out, h = rnn(bolak, h.detach())  # kesilgan BPTT  # ✅

1.  # RNN yacheykasi formulasi?
2.  # nn.RNN(4, 3) parametrlari soni?
3.  # batch_first=True da out va h_n shakli (B=2, T=5, h=3)?
4.  # padding siz out[:, -1] va h_n[0]?
5.  # T uzunlik oshsa parametrlar soni?
6.  # dh_{k+1}/dh_k formulasi?
7.  # rho(U) = 0.6 bo'lsa 50 qadamda gradient taxminan necha marta kichrayadi?
8.  # clipping gradient yo'nalishini o'zgartiradimi?
9.  # clip_grad_norm_ nimani qaytaradi?
10. # clipping so'nishni davolaydimi?
11. # padding bilan to'g'ri oxirgi holat qanday olinadi?
12. # kesilgan BPTT nima?

1.  rnn = nn.RNN(32, 64)
    out, h_n = rnn(emb(ids))                 # ids (B, T)

2.  logit = self.chiq(out[:, -1])            # o'ngdan padding bor

3.  opt.step()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)

4.  logit = self.chiq(h_n)                   # h_n (1, B, 64)

5.  paket = pack_padded_sequence(x, uzunlik, batch_first=True)   # saralanmagan