23.9-dars: LSTM va GRU

IKKI HOLAT:
  h_t - yashirin holat (chiqish, "qisqa muddatli")
  c_t - hujayra holati (cell state, "uzoq muddatli xotira")

BITTA QADAM (x_t - kirish, h_{t-1}, c_{t-1} - oldingi holat):
  i_t = sigmoid(W_i x_t + U_i h_{t-1} + b_i)    kirish darvozasi
  f_t = sigmoid(W_f x_t + U_f h_{t-1} + b_f)    unutish darvozasi
  g_t = tanh   (W_g x_t + U_g h_{t-1} + b_g)    nomzod qiymat
  o_t = sigmoid(W_o x_t + U_o h_{t-1} + b_o)    chiqish darvozasi

  c_t = f_t * c_{t-1} + i_t * g_t               (* - elementma-element)
  h_t = o_t * tanh(c_t)

MA'NOSI:
  f_t ~ 1  -> eski xotira saqlanadi
  f_t ~ 0  -> eski xotira o'chiriladi
  i_t ~ 1  -> yangi nomzod g_t yoziladi
  o_t      -> xotiraning qaysi qismi tashqariga ko'rinadi

nn.LSTM DA:
  weight_ih_l0: (4h, d_kir)  - [W_i; W_f; W_g; W_o] ustma-ust
  weight_hh_l0: (4h, h)      - [U_i; U_f; U_g; U_o]
  bias_ih_l0, bias_hh_l0     - ikkita bias (yig'indisi ishlatiladi)
  tartib: i, f, g, o  ->  z.chunk(4, dim=1)

GRU (bitta holat h_t, uchta blok):
  r_t = sigmoid(W_r x_t + b_ir + U_r h_{t-1} + b_hr)     qayta o'rnatish
  z_t = sigmoid(W_z x_t + b_iz + U_z h_{t-1} + b_hz)     yangilash
  n_t = tanh(W_n x_t + b_in + r_t * (U_n h_{t-1} + b_hn)) nomzod
  h_t = (1 - z_t) * n_t + z_t * h_{t-1}

LSTM BILAN FARQI:
  alohida c_t yo'q - xotira h_t ning o'zida
  z_t bir vaqtda "unutish" va "yozish" ni boshqaradi (z va 1 - z)
  chiqish darvozasi yo'q
  3 blok (LSTM da 4) -> parametr 25% kam

nn.GRU DA:
  tartib: r, z, n
  DIQQAT: r_t faqat U_n h + b_hn ga ko'paytiriladi (b_hn ham ichkarida)
  shuning uchun ikki bias ni oldindan qo'shib bo'lmaydi

QAYSI BIRI YAXSHI?
  ko'p vazifada natijalar yaqin; GRU tezroq va kichikroq
  aniq javob - o'z ma'lumotingizda taqqoslash (2-misol)

BITTA "BLOK" (oddiy RNN):
  W: h * d,  U: h * h,  ikki bias: 2h
  jami: h * (d + h) + 2h

RNN  = 1 blok
GRU  = 3 blok  (r, z, n)
LSTM = 4 blok  (i, f, g, o)

MISOL: d = 8, h = 16
  blok = 16 * (8 + 16) + 2 * 16 = 416
  RNN 416,  GRU 1248,  LSTM 1664

AMALIY XULOSA:
  bir xil h da LSTM 4 barobar ko'p parametr va hisob
  halol taqqoslashda buni hisobga oling:
    yoki h ni tenglashtiring (bu dars),
    yoki parametr sonini tenglashtiring (RNN ga kattaroq h)
  ko'p qatlam: 2-qatlam kirishi d emas, h (yoki 2h ikki tomonlamada)

ODDIY RNN 23.8-bob:
  h_t = tanh(W x_t + U h_{t-1} + b)
  d h_t / d h_{t-1} = diag(1 - h_t^2) * U
  m qadam orqaga: m ta shunday matritsa ko'paytmasi
  normasi < 1 bo'lsa -> eksponent so'nish

LSTM HUJAYRA HOLATI:
  c_t = f_t * c_{t-1} + i_t * g_t
  d c_t / d c_{t-1} = diag(f_t)     (asosiy yo'l, matritsa ko'paytmasi YO'Q)
  m qadam orqaga: f ning ko'paytmasi
  f ~ 1 bo'lsa -> gradient deyarli o'zgarmay o'tadi ("doimiy xato karuseli")

LEKIN - AVTOMATIK EMAS:
  boshlang'ich holatda bias ~ 0 -> f ~ sigmoid(0) = 0.5
  0.5^39 ~ 2e-12 -> boshlanishda LSTM ham so'nadi (1-misol)
  tarmoq f ni 1 ga yaqinlashtirishni O'RGANA OLADI - RNN esa bunday yo'lga ega emas
  amaliy hiyla: unutish darvozasi bias ini musbat (1..3) qilib boshlash
  (2-misoldagi LSTM aynan shunday: bias + 1)

GRU:
  h_t = (1 - z) * n + z * h_{t-1}
  z ~ 1 -> h deyarli nusxalanadi, gradient ham shunday

MUAMMO: batch - to'rtburchak tensor (B, T), jumlalar esa turli uzunlikda
  [5, 3, 8, 2, 7]
  [4, 9, 1, 0, 0]    <- 0 = PAD
  [6, 2, 0, 0, 0]

LSTM PAD NI HAM "O'QIYDI":
  chiq[:, -1] - qisqa jumlalar uchun PAD lar ustidan o'tgan holat
  padding_idx=0 embeddingni nol qiladi, lekin LSTM holati baribir o'zgaradi
  (bias va rekurrent qism ishlaydi)

TO'G'RI USULLAR:
  1. gather:  chiq[torch.arange(B), uz - 1]    - haqiqiy oxirgi pozitsiya
  2. pack:    p = pack_padded_sequence(e, uz, batch_first=True,
                                       enforce_sorted=False)
              chiq_p, (h_n, c_n) = lstm(p)
              h_n[-1] - har namunaning HAQIQIY oxirgi holati
              pad_packed_sequence(chiq_p, batch_first=True) - PAD joylari 0

PACK QANDAY ISHLAYDI:
  har qadamda faqat hali tugamagan namunalar hisoblanadi
  PAD umuman LSTM ga kirmaydi -> natija yolg'iz o'tkazish bilan AYNAN bir xil
  ikki tomonlama LSTM da pack MAJBURIY: orqaga yo'nalish PAD dan boshlanmasin

NIMA UCHUN XATO YASHIRIN QOLADI:
  o'rgatish batchlari uzunlik bo'yicha guruhlangan bo'lsa - PAD deyarli yo'q
  chiq[:, -1] o'rgatishda to'g'ri ishlaydi, bashoratda esa - yo'q

IKKI TOMONLAMA (bidirectional=True):
  oldinga LSTM: x_1 -> x_T
  orqaga LSTM:  x_T -> x_1 (alohida og'irliklar)
  chiq[:, t] = [oldinga_t ; orqaga_t]   -> oxirgi o'lcham 2h
  har pozitsiya O'TMISH va KELAJAKNI ko'radi

h_n TARTIBI:
  h_n shakli: (qatlam * yo'nalish, B, h)
  h_n.view(qatlam, 2, B, h)[-1, 0]  - oldinga, oxirgi qatlam = chiq[:, -1, :h]
  h_n.view(qatlam, 2, B, h)[-1, 1]  - orqaga,  oxirgi qatlam = chiq[:, 0, h:]
  orqaga yo'nalishning "oxiri" - BIRINCHI pozitsiya!

KO'P QATLAM (num_layers=L):
  1-qatlam chiqishlari 2-qatlamga kirish bo'ladi
  dropout=p qatlamlar ORASIGA qo'yiladi (L=1 da ta'sirsiz va ogohlantirish)
  chuqurlik murakkabroq funksiyalar beradi, lekin KELAJAKNI ko'rsatmaydi

QACHON NIMA:
  butun matn oldindan ma'lum (klassifikatsiya, belgilash) -> ikki tomonlama
  keyingi belgini bashorat qilish, oqimli vazifa -> faqat bir tomonlama
  (til modeli kelajakni "ko'rsa" - bu ma'lumot sizishi)

import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

lstm = nn.LSTM(input_size=16, hidden_size=32, num_layers=1,
               bidirectional=False, batch_first=True)
chiq, (h_n, c_n) = lstm(x)              # x: (B, T, 16)
# chiq: (B, T, 32), h_n va c_n: (1, B, 32)

gru = nn.GRU(16, 32, batch_first=True)
chiq, h_n = gru(x)                      # GRU da c_n yo'q

# qo'lda bitta qadam: tartib [i, f, g, o]
z = x_t @ lstm.weight_ih_l0.T + lstm.bias_ih_l0 \
    + h @ lstm.weight_hh_l0.T + lstm.bias_hh_l0
zi, zf, zg, zo = z.chunk(4, dim=1)

# turli uzunlik
e = emb(X)                              # X: (B, T), 0 = PAD
p = pack_padded_sequence(e, uz, batch_first=True, enforce_sorted=False)
chiq_p, (h_n, _) = lstm(p)
oxirgi = h_n[-1]                        # haqiqiy oxirgi holat
chiq, _ = pad_packed_sequence(chiq_p, batch_first=True)
# yoki: chiq[torch.arange(B), uz - 1]

# ikki tomonlama, 2 qatlam
bi = nn.LSTM(16, 32, num_layers=2, bidirectional=True, batch_first=True)
chiq, (h_n, _) = bi(x)                  # chiq: (B, T, 64)
h4 = h_n.view(2, 2, B, 32)              # (qatlam, yo'nalish, B, h)
jumla = torch.cat([h4[-1, 0], h4[-1, 1]], dim=1)

LSTM: i, f, g, o darvozalari; c = f*c + i*g; h = o*tanh(c)
GRU: r, z, n; h = (1-z)*n + z*h
parametr: RNN 1x, GRU 3x, LSTM 4x
gradient c orqali qo'shish yo'li bilan oqadi (f ~ 1 bo'lsa)
padding: pack_padded_sequence yoki gather(uz - 1), chiq[:, -1] EMAS
ikki tomonlama - kelajak konteksti; til modelida ishlatilmaydi

"""LSTM va GRU ning bitta qadamini qo'lda hisoblash va nn.LSTM bilan solishtirish."""

import torch
import torch.nn as nn


def lstm_qadam(x, h, c, lstm):
    """nn.LSTM og'irliklari bilan bitta qadam: tartib [i, f, g, o]."""
    z = (x @ lstm.weight_ih_l0.T + lstm.bias_ih_l0
         + h @ lstm.weight_hh_l0.T + lstm.bias_hh_l0)
    zi, zf, zg, zo = z.chunk(4, dim=1)
    i = torch.sigmoid(zi)          # kirish darvozasi
    f = torch.sigmoid(zf)          # unutish darvozasi
    g = torch.tanh(zg)             # nomzod qiymat
    o = torch.sigmoid(zo)          # chiqish darvozasi
    c_yangi = f * c + i * g
    h_yangi = o * torch.tanh(c_yangi)
    return h_yangi, c_yangi, (i, f, g, o)


def gru_qadam(x, h, gru):
    """nn.GRU og'irliklari bilan bitta qadam: tartib [r, z, n]."""
    a = x @ gru.weight_ih_l0.T + gru.bias_ih_l0
    b = h @ gru.weight_hh_l0.T + gru.bias_hh_l0
    ar, az, an = a.chunk(3, dim=1)
    br, bz, bn = b.chunk(3, dim=1)
    r = torch.sigmoid(ar + br)     # qayta o'rnatish darvozasi
    z = torch.sigmoid(az + bz)     # yangilash darvozasi
    n = torch.tanh(an + r * bn)    # nomzod holat
    return (1 - z) * n + z * h


def param_soni(m):
    return sum(p.numel() for p in m.parameters())


def main() -> None:
    torch.manual_seed(0)
    d_kir, d_yash, B = 8, 16, 3
    lstm = nn.LSTM(d_kir, d_yash, batch_first=True)
    x = torch.randn(B, d_kir)
    h0 = torch.randn(B, d_yash) * 0.5
    c0 = torch.randn(B, d_yash) * 0.5

    print("=== 1. LSTM: bitta qadam qo'lda ===")
    print(f"  weight_ih_l0: {tuple(lstm.weight_ih_l0.shape)}  (4*{d_yash}, {d_kir})")
    print(f"  weight_hh_l0: {tuple(lstm.weight_hh_l0.shape)}  (4*{d_yash}, {d_yash})")
    h1, c1, (i, f, g, o) = lstm_qadam(x, h0, c0, lstm)
    with torch.no_grad():
        chiq, (hn, cn) = lstm(x.unsqueeze(1), (h0.unsqueeze(0), c0.unsqueeze(0)))
    print(f"  h farqi (qo'lda - nn.LSTM): {(h1 - hn[0]).abs().max().item():.2e}")
    print(f"  c farqi (qo'lda - nn.LSTM): {(c1 - cn[0]).abs().max().item():.2e}")
    print(f"  chiqish = oxirgi h: {torch.equal(chiq[:, -1], hn[0])}")
    print("  darvozalar (1-namuna, birinchi 5 neyron):")
    for nom, t in [("i kirish", i), ("f unutish", f), ("g nomzod", g),
                   ("o chiqish", o)]:
        print(f"    {nom:<10} " + " ".join(f"{v:+.3f}" for v in t[0, :5].tolist()))
    print(f"  sigmoid darvozalar oralig'i: "
          f"{min(i.min(), f.min(), o.min()).item():.3f}.."
          f"{max(i.max(), f.max(), o.max()).item():.3f}  (0..1)")

    print("\n=== 2. GRU: bitta qadam qo'lda ===")
    gru = nn.GRU(d_kir, d_yash, batch_first=True)
    h_qolda = gru_qadam(x, h0, gru)
    with torch.no_grad():
        _, hg = gru(x.unsqueeze(1), h0.unsqueeze(0))
    print(f"  h farqi (qo'lda - nn.GRU): {(h_qolda - hg[0]).abs().max().item():.2e}")

    print("\n=== 3. Parametrlar soni ===")
    rnn = nn.RNN(d_kir, d_yash)
    asos = d_yash * (d_kir + d_yash) + 2 * d_yash
    print(f"  bitta 'blok': {d_yash}*({d_kir}+{d_yash}) + 2*{d_yash} = {asos}")
    for nom, m, k in [("RNN", rnn, 1), ("GRU", gru, 3), ("LSTM", lstm, 4)]:
        print(f"  {nom:<5} {param_soni(m):>6}  = {k} x {asos}  "
              f"-> {param_soni(m) == k * asos}")

    print("\n=== 4. Gradient masofa bo'yicha: d h_T / d x_t (T=40) ===")
    T = 40
    torch.manual_seed(1)
    xs = torch.randn(1, T, d_kir)
    modellar = {"RNN": nn.RNN(d_kir, d_yash, batch_first=True),
                "LSTM": nn.LSTM(d_kir, d_yash, batch_first=True)}
    lstm_f = nn.LSTM(d_kir, d_yash, batch_first=True)
    lstm_f.load_state_dict(modellar["LSTM"].state_dict())
    with torch.no_grad():
        lstm_f.bias_ih_l0[d_yash:2 * d_yash] += 3.0   # f ~ sigmoid(3) ~ 0.95
    modellar["LSTM f+3"] = lstm_f
    masofalar = [1, 5, 10, 20, 39]
    print(f"  {'model':<9} " + " ".join(f"{'m=' + str(m):>9}" for m in masofalar))
    nisbat = {}
    for nom, m in modellar.items():
        xg = xs.clone().requires_grad_(True)
        chiq, _ = m(xg)
        chiq[0, -1].sum().backward()
        norma = xg.grad[0].norm(dim=1)
        nisbat[nom] = (norma[0] / norma[T - 2]).item()
        print(f"  {nom:<9} " + " ".join(f"{norma[T - 1 - k].item():>9.1e}"
                                         for k in masofalar))
    print("  m - x_t dan chiqishgacha bo'lgan qadamlar soni")
    for nom, v in nisbat.items():
        holat = "saqlandi" if v > 0.1 else "so'ndi"
        print(f"  {nom:<9} gradient(m=39) / gradient(m=1) = {v:.1e} -> {holat}")
    print("  ⭐ Qo'lda hisob nn.LSTM bilan mos: darvozalar - oddiy sigmoid va tanh")


if __name__ == "__main__":
    main()

=== 1. LSTM: bitta qadam qo'lda ===
  weight_ih_l0: (64, 8)  (4*16, 8)
  weight_hh_l0: (64, 16)  (4*16, 16)
  h farqi (qo'lda - nn.LSTM): 5.96e-08
  c farqi (qo'lda - nn.LSTM): 1.19e-07
  chiqish = oxirgi h: True
  darvozalar (1-namuna, birinchi 5 neyron):
    i kirish   +0.447 +0.388 +0.452 +0.447 +0.563
    f unutish  +0.638 +0.724 +0.435 +0.655 +0.549
    g nomzod   +0.324 +0.513 +0.310 -0.579 +0.016
    o chiqish  +0.284 +0.553 +0.548 +0.543 +0.477
  sigmoid darvozalar oralig'i: 0.125..0.788  (0..1)

=== 2. GRU: bitta qadam qo'lda ===
  h farqi (qo'lda - nn.GRU): 5.96e-08

=== 3. Parametrlar soni ===
  bitta 'blok': 16*(8+16) + 2*16 = 416
  RNN      416  = 1 x 416  -> True
  GRU     1248  = 3 x 416  -> True
  LSTM    1664  = 4 x 416  -> True

=== 4. Gradient masofa bo'yicha: d h_T / d x_t (T=40) ===
  model           m=1       m=5      m=10      m=20      m=39
  RNN         4.0e-01   5.2e-02   3.6e-03   5.1e-06   1.2e-10
  LSTM        1.5e-01   2.5e-02   2.0e-03   2.7e-06   6.9e-11
  LSTM f+3    1.3e-01   1.4e-01   2.0e-01   1.7e-01   1.9e-01
  m - x_t dan chiqishgacha bo'lgan qadamlar soni
  RNN       gradient(m=39) / gradient(m=1) = 3.0e-10 -> so'ndi
  LSTM      gradient(m=39) / gradient(m=1) = 4.6e-10 -> so'ndi
  LSTM f+3  gradient(m=39) / gradient(m=1) = 1.4e+00 -> saqlandi
  ⭐ Qo'lda hisob nn.LSTM bilan mos: darvozalar - oddiy sigmoid va tanh

"""Qo'shish vazifasi: RNN, LSTM, GRU uzunlik bo'yicha (3 seed)."""

import numpy as np
import torch
import torch.nn as nn


class Model(nn.Module):
    def __init__(self, tur, h=32):
        super().__init__()
        sinf = {"RNN": nn.RNN, "LSTM": nn.LSTM, "GRU": nn.GRU}[tur]
        self.rnn = sinf(2, h, batch_first=True)
        if tur == "LSTM":                  # unutish darvozasi bias i = 1 (2.4)
            with torch.no_grad():
                self.rnn.bias_ih_l0[h:2 * h] += 1.0
        self.chiqish = nn.Linear(h, 1)

    def forward(self, x):
        o, _ = self.rnn(x)
        return self.chiqish(o[:, -1]).squeeze(1)


def malumot(n, T, g):
    """Kirish: (qiymat, belgi). Nishon: belgilangan ikki qiymat yig'indisi."""
    qiymat = torch.rand(n, T, generator=g)
    belgi = torch.zeros(n, T)
    i = torch.randint(0, T // 2, (n,), generator=g)
    j = torch.randint(T // 2, T, (n,), generator=g)
    belgi[torch.arange(n), i] = 1.0
    belgi[torch.arange(n), j] = 1.0
    y = (qiymat * belgi).sum(1)
    return torch.stack([qiymat, belgi], 2), y


def orgat(tur, T, seed, qadamlar=300):
    torch.manual_seed(seed)
    model = Model(tur)
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    g = torch.Generator().manual_seed(seed)
    tarix = []
    Xv, yv = malumot(1000, T, torch.Generator().manual_seed(999))
    for q in range(1, qadamlar + 1):
        x, y = malumot(32, T, g)
        opt.zero_grad()
        loss = nn.functional.mse_loss(model(x), y)
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step()
        if q % 100 == 0:
            with torch.no_grad():
                tarix.append(nn.functional.mse_loss(model(Xv), yv).item())
    return tarix


def main() -> None:
    torch.set_num_threads(1)
    print("=== 1. Vazifa ===")
    x, y = malumot(1, 8, torch.Generator().manual_seed(3))
    print("  qiymat: " + " ".join(f"{v:.2f}" for v in x[0, :, 0].tolist()))
    print("  belgi:  " + " ".join(f"{int(v):>4}" for v in x[0, :, 1].tolist()))
    print(f"  nishon: {y.item():.2f}  (belgilangan ikki qiymat yig'indisi)")
    _, yv = malumot(100000, 10, torch.Generator().manual_seed(1))
    print(f"  bazaviy MSE (doim 1.0 deb aytish): "
          f"{((yv - 1.0) ** 2).mean().item():.4f}  (nazariy 1/6 = 0.1667)")

    uzunliklar = [5, 15]
    turlar = ["RNN", "LSTM", "GRU"]
    seedlar = [0, 1, 2]
    natija = {}
    for T in uzunliklar:
        for tur in turlar:
            for s in seedlar:
                natija[(T, tur, s)] = orgat(tur, T, s)

    print("\n=== 2. Yakuniy validatsiya MSE (300 qadam, batch 32, 3 seed o'rtachasi) ===")
    print(f"  {'T':>4} " + " ".join(f"{t:>8}" for t in turlar)
          + f" {'yechildi (MSE<0.02)':>22}")
    for T in uzunliklar:
        qator = []
        yechildi = []
        for tur in turlar:
            oxirgi = [natija[(T, tur, s)][-1] for s in seedlar]
            qator.append(np.mean(oxirgi))
            yechildi.append(f"{sum(v < 0.02 for v in oxirgi)}/3")
        print(f"  {T:>4} " + " ".join(f"{v:>8.4f}" for v in qator)
              + f" {'  '.join(yechildi):>22}")

    print("\n=== 3. O'rgatish egri chizig'i, T=15, seed 0 (har 100 qadam) ===")
    for tur in turlar:
        print(f"  {tur:<5} " + " ".join(f"{v:.4f}"
                                      for v in natija[(15, tur, 0)]))

    print("\n=== 4. Juftlashgan farq: RNN MSE - model MSE ===")
    print(f"  {'T':>4} {'model':>6} {'o_rt farq':>10} {'SE':>8} {'xulosa':>26}")
    for T in uzunliklar:
        for tur in ["LSTM", "GRU"]:
            farq = np.array([natija[(T, "RNN", s)][-1] - natija[(T, tur, s)][-1]
                             for s in seedlar])
            se = farq.std(ddof=1) / np.sqrt(len(farq))
            if abs(farq.mean()) <= 2 * se:
                xulosa = "sezilarli farq yo'q"
            elif farq.mean() > 0:
                xulosa = f"{tur} sezilarli yaxshi"
            else:
                xulosa = "RNN sezilarli yaxshi"
            print(f"  {T:>4} {tur:>6} {farq.mean():>+10.4f} {se:>8.4f} "
                  f"{xulosa:>26}")
    rnn_uzun = np.mean([natija[(15, "RNN", s)][-1] for s in seedlar])
    if rnn_uzun > 0.1:
        print(f"  T=15 da RNN MSE {rnn_uzun:.3f} - bazaviy 0.167 ga yaqin: "
              f"u belgilangan qiymatlarni 'eslay' olmadi")
    print("  ⭐ Darvozali modellar uzoq bog'liqlikni o'rganadi, oddiy RNN - yo'q")


if __name__ == "__main__":
    main()

=== 1. Vazifa ===
  qiymat: 0.00 0.11 0.29 0.03 0.47 0.06 0.77 0.74
  belgi:     0    1    0    0    0    0    0    1
  nishon: 0.85  (belgilangan ikki qiymat yig'indisi)
  bazaviy MSE (doim 1.0 deb aytish): 0.1671  (nazariy 1/6 = 0.1667)

=== 2. Yakuniy validatsiya MSE (300 qadam, batch 32, 3 seed o'rtachasi) ===
     T      RNN     LSTM      GRU    yechildi (MSE<0.02)
     5   0.0245   0.0019   0.0006          1/3  3/3  3/3
    15   0.1755   0.0055   0.0009          0/3  3/3  3/3

=== 3. O'rgatish egri chizig'i, T=15, seed 0 (har 100 qadam) ===
  RNN   0.1796 0.1767 0.1732
  LSTM  0.1838 0.0259 0.0072
  GRU   0.0456 0.0025 0.0012

=== 4. Juftlashgan farq: RNN MSE - model MSE ===
     T  model  o_rt farq       SE                     xulosa
     5   LSTM    +0.0226   0.0060      LSTM sezilarli yaxshi
     5    GRU    +0.0240   0.0059       GRU sezilarli yaxshi
    15   LSTM    +0.1700   0.0023      LSTM sezilarli yaxshi
    15    GRU    +0.1746   0.0024       GRU sezilarli yaxshi
  T=15 da RNN MSE 0.176 - bazaviy 0.167 ga yaqin: u belgilangan qiymatlarni 'eslay' olmadi
  ⭐ Darvozali modellar uzoq bog'liqlikni o'rganadi, oddiy RNN - yo'q

"""Turli uzunlikdagi ketma-ketliklar: padding, pack_padded_sequence, oxirgi haqiqiy holat."""

import numpy as np
import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

PAD = 0


def toplam_yarat(ketliklar, uzunlik=None):
    """Ketma-ketliklarni PAD bilan bir xil uzunlikka to'ldiradi."""
    uz = torch.tensor([len(k) for k in ketliklar])
    T = uzunlik or int(uz.max())
    X = torch.full((len(ketliklar), T), PAD, dtype=torch.long)
    for b, k in enumerate(ketliklar):
        X[b, :len(k)] = torch.tensor(k)
    return X, uz


class Klassifikator(nn.Module):
    def __init__(self, V=12, d=16, h=32):
        super().__init__()
        self.emb = nn.Embedding(V, d, padding_idx=PAD)
        self.lstm = nn.LSTM(d, h, batch_first=True)
        self.chiqish = nn.Linear(h, 2)

    def forward(self, X, uz, usul="pack"):
        e = self.emb(X)
        if usul == "pack":
            p = pack_padded_sequence(e, uz, batch_first=True,
                                     enforce_sorted=False)
            _, (hn, _) = self.lstm(p)
            oxirgi = hn[-1]
        elif usul == "gather":
            chiq, _ = self.lstm(e)
            oxirgi = chiq[torch.arange(len(uz)), uz - 1]
        else:                                  # xato: padding dagi holat
            chiq, _ = self.lstm(e)
            oxirgi = chiq[:, -1]
        return self.chiqish(oxirgi)


def ketliklar_yarat(uzunliklar, g):
    """Nishon: oxirgi ikki HAQIQIY token yig'indisi juftmi."""
    ketliklar = [torch.randint(1, 12, (int(n),), generator=g).tolist()
                 for n in uzunliklar]
    y = torch.tensor([(k[-1] + k[-2]) % 2 for k in ketliklar])
    return ketliklar, y


def orgat(seed, qadamlar=300):
    """Uzunlik bo'yicha guruhlangan batchlar: har batch ichida PAD yo'q."""
    torch.manual_seed(seed)
    model = Klassifikator()
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        n = int(torch.randint(3, 31, (1,), generator=g))
        ketliklar, y = ketliklar_yarat([n] * 64, g)
        X, uz = toplam_yarat(ketliklar)
        opt.zero_grad()
        nn.functional.cross_entropy(model(X, uz), y).backward()
        opt.step()
    model.eval()
    return model


def main() -> None:
    torch.set_num_threads(1)
    print("=== 1. Padding ===")
    ketliklar = [[5, 3, 8, 2, 7], [4, 9, 1], [6, 2]]
    X, uz = toplam_yarat(ketliklar)
    print(f"  uzunliklar: {uz.tolist()}")
    for qator in X.tolist():
        print("   ", qator)

    print("\n=== 2. Oxirgi holat: uch usul ===")
    torch.manual_seed(0)
    emb = nn.Embedding(12, 4, padding_idx=PAD)
    lstm = nn.LSTM(4, 6, batch_first=True)
    with torch.no_grad():
        chiq, _ = lstm(emb(X))
        p = pack_padded_sequence(emb(X), uz, batch_first=True,
                                 enforce_sorted=False)
        chiq_p, (hn, _) = lstm(p)
        ochilgan, _ = pad_packed_sequence(chiq_p, batch_first=True)
        haqiqiy = chiq[torch.arange(3), uz - 1]
        yakka = torch.stack([lstm(emb(torch.tensor([k])))[0][0, -1]
                             for k in ketliklar])
    print(f"  {'namuna':>7} {'uz':>3} {'|chiq[:,-1] - yakka|':>21} "
          f"{'|gather - yakka|':>17} {'|pack h_n - yakka|':>19}")
    for b in range(3):
        print(f"  {b:>7} {int(uz[b]):>3} "
              f"{(chiq[b, -1] - yakka[b]).abs().max().item():>21.4f} "
              f"{(haqiqiy[b] - yakka[b]).abs().max().item():>17.2e} "
              f"{(hn[0, b] - yakka[b]).abs().max().item():>19.2e}")
    print(f"  pad_packed_sequence: padding joylari nolmi? "
          f"{bool((ochilgan[1, 3:] == 0).all() and (ochilgan[2, 2:] == 0).all())}")
    print("  yakka - ketma-ketlikni yolg'iz (PAD siz) o'tkazgandagi holat")

    print("\n=== 3. O'rgatish PAD siz, bashorat aralash batchda ===")
    g = torch.Generator().manual_seed(123)
    uzt = torch.randint(3, 31, (2000,), generator=g)
    kt, yt = ketliklar_yarat(uzt.tolist(), g)
    Xt, uzt = toplam_yarat(kt)
    Xt_uzun, _ = toplam_yarat(kt, uzunlik=80)
    print(f"  test: {tuple(Xt.shape)}, o'rtacha PAD ulushi "
          f"{(Xt == PAD).float().mean().item():.1%}; "
          f"80 gacha to'ldirilgan: {tuple(Xt_uzun.shape)}")
    usullar = ["pack", "gather", "chiq[:,-1]"]
    natija = {(u, T): [] for u in usullar for T in (30, 80)}
    qisqa = uzt < 30
    guruh = {u: [] for u in usullar}
    for s in range(3):
        m = orgat(s)
        with torch.no_grad():
            for u in usullar:
                for T, XX in [(30, Xt), (80, Xt_uzun)]:
                    togri = (m(XX, uzt, u).argmax(1) == yt).float()
                    natija[(u, T)].append(togri.mean().item())
                    if T == 30:
                        guruh[u].append((togri[~qisqa].mean().item(),
                                         togri[qisqa].mean().item()))
    print(f"  {'usul':<11} {'T=30 (3 seed)':>23} {'o_rt':>7} {'T=80 o_rt':>10} "
          f"{'PAD yo_q':>9} {'PAD bor':>8}")
    for u in usullar:
        uzun = np.mean([a for a, _ in guruh[u]])
        qis = np.mean([b for _, b in guruh[u]])
        print(f"  {u:<11} " + " ".join(f"{v:>7.4f}" for v in natija[(u, 30)])
              + f" {np.mean(natija[(u, 30)]):>7.4f}"
              f" {np.mean(natija[(u, 80)]):>10.4f} {uzun:>9.4f} {qis:>8.4f}")
    print(f"  PAD yo_q - uz=30 bo'lgan {int((~qisqa).sum())} ta namuna (T=30 da)")
    print()
    for T in (30, 80):
        farq = np.array(natija[("pack", T)]) - np.array(natija[("chiq[:,-1]", T)])
        se = farq.std(ddof=1) / np.sqrt(len(farq))
        xulosa = ("chiq[:,-1] sezilarli yomon" if farq.mean() > 2 * se
                  else "sezilarli farq yo'q")
        print(f"  T={T}: pack - chiq[:,-1] = {farq.mean():+.4f}, "
              f"SE {se:.4f} -> {xulosa}")
    print("  ⭐ Oxirgi HAQIQIY holatni oling: pack (h_n) yoki uz-1 bo'yicha gather")


if __name__ == "__main__":
    main()

=== 1. Padding ===
  uzunliklar: [5, 3, 2]
    [5, 3, 8, 2, 7]
    [4, 9, 1, 0, 0]
    [6, 2, 0, 0, 0]

=== 2. Oxirgi holat: uch usul ===
   namuna  uz  |chiq[:,-1] - yakka|  |gather - yakka|  |pack h_n - yakka|
        0   5                0.0000          0.00e+00            1.49e-08
        1   3                0.1057          0.00e+00            2.98e-08
        2   2                0.1679          0.00e+00            2.98e-08
  pad_packed_sequence: padding joylari nolmi? True
  yakka - ketma-ketlikni yolg'iz (PAD siz) o'tkazgandagi holat

=== 3. O'rgatish PAD siz, bashorat aralash batchda ===
  test: (2000, 30), o'rtacha PAD ulushi 45.3%; 80 gacha to'ldirilgan: (2000, 80)
  usul                  T=30 (3 seed)    o_rt  T=80 o_rt  PAD yo_q  PAD bor
  pack         1.0000  1.0000  1.0000  1.0000     1.0000    1.0000   1.0000
  gather       1.0000  1.0000  1.0000  1.0000     1.0000    1.0000   1.0000
  chiq[:,-1]   0.4905  0.5240  0.5060  0.5068     0.4932    1.0000   0.4876
  PAD yo_q - uz=30 bo'lgan 75 ta namuna (T=30 da)

  T=30: pack - chiq[:,-1] = +0.4932, SE 0.0097 -> chiq[:,-1] sezilarli yomon
  T=80: pack - chiq[:,-1] = +0.5068, SE 0.0137 -> chiq[:,-1] sezilarli yomon
  ⭐ Oxirgi HAQIQIY holatni oling: pack (h_n) yoki uz-1 bo'yicha gather

"""Ikki tomonlama va ko'p qatlamli LSTM: shakllar, h_n tartibi, kelajakka qarash."""

import numpy as np
import torch
import torch.nn as nn


class Belgilovchi(nn.Module):
    """Har pozitsiya uchun yorliq beradi (ketma-ketlikni belgilash)."""

    def __init__(self, qatlam=1, ikki=False, h=24):
        super().__init__()
        self.emb = nn.Embedding(10, 12)
        self.lstm = nn.LSTM(12, h, num_layers=qatlam, bidirectional=ikki,
                            batch_first=True)
        self.chiqish = nn.Linear(h * (2 if ikki else 1), 2)

    def forward(self, x):
        o, _ = self.lstm(self.emb(x))
        return self.chiqish(o)


def malumot(n, g, T=16):
    """Ikki yorliq: kelajak (x[t+1] > x[t]) va o'tmish (x[t-1] > x[t])."""
    x = torch.randint(0, 10, (n, T), generator=g)
    kelajak = torch.zeros(n, T, dtype=torch.long)
    otmish = torch.zeros(n, T, dtype=torch.long)
    kelajak[:, :-1] = (x[:, 1:] > x[:, :-1]).long()
    otmish[:, 1:] = (x[:, :-1] > x[:, 1:]).long()
    return x, kelajak, otmish


def orgat(qatlam, ikki, vazifa, seed, qadamlar=100):
    torch.manual_seed(seed)
    model = Belgilovchi(qatlam, ikki)
    opt = torch.optim.Adam(model.parameters(), lr=0.01)
    g = torch.Generator().manual_seed(seed)
    for _ in range(qadamlar):
        x, kel, ot = malumot(64, g)
        y = kel if vazifa == "kelajak" else ot
        opt.zero_grad()
        nn.functional.cross_entropy(model(x).reshape(-1, 2),
                                    y.reshape(-1)).backward()
        opt.step()
    model.eval()
    return model


def main() -> None:
    torch.set_num_threads(1)
    print("=== 1. Shakllar ===")
    B, T, d, h = 3, 7, 5, 4
    torch.manual_seed(0)
    x = torch.randn(B, T, d)
    print(f"  {'sozlama':<22} {'chiq':>12} {'h_n':>12} {'parametr':>9}")
    for L, ikki in [(1, False), (1, True), (2, False), (2, True)]:
        m = nn.LSTM(d, h, num_layers=L, bidirectional=ikki, batch_first=True)
        with torch.no_grad():
            o, (hn, cn) = m(x)
        n = sum(p.numel() for p in m.parameters())
        nom = f"qatlam={L}, ikki={ikki}"
        print(f"  {nom:<22} {str(tuple(o.shape)):>12} "
              f"{str(tuple(hn.shape)):>12} {n:>9}")
    print("  chiq: (B, T, yo'nalish*h), h_n: (qatlam*yo'nalish, B, h)")

    print("\n=== 2. h_n tartibi (2 qatlam, ikki tomonlama) ===")
    m = nn.LSTM(d, h, num_layers=2, bidirectional=True, batch_first=True)
    with torch.no_grad():
        o, (hn, _) = m(x)
    hn4 = hn.view(2, 2, B, h)            # (qatlam, yo'nalish, B, h)
    oldinga = (hn4[-1, 0] - o[:, -1, :h]).abs().max().item()
    orqaga = (hn4[-1, 1] - o[:, 0, h:]).abs().max().item()
    xato = (hn4[-1, 1] - o[:, -1, h:]).abs().max().item()
    print(f"  oldinga yo'nalish: h_n[-1, 0] == chiq[:, -1, :h]  farq {oldinga:.1e}")
    print(f"  orqaga yo'nalish:  h_n[-1, 1] == chiq[:, 0, h:]   farq {orqaga:.1e}")
    print(f"  xato taxmin:       h_n[-1, 1] vs chiq[:, -1, h:]  farq {xato:.3f}")
    print("  orqaga LSTM ning yakuniy holati - BIRINCHI pozitsiyada")

    print("\n=== 3. Belgilash: kelajakka va o'tmishga bog'liq yorliq ===")
    Xv, kel_v, ot_v = malumot(1000, torch.Generator().manual_seed(77))
    ichki = slice(1, -1)                 # chetdagi pozitsiyalarsiz
    sozlamalar = {"1 qatlam": (1, False), "2 qatlam": (2, False),
                  "ikki tomonlama": (1, True)}
    natija = {}
    for vazifa, yv in [("kelajak", kel_v), ("o'tmish", ot_v)]:
        for nom, (L, ikki) in sozlamalar.items():
            if vazifa == "o'tmish" and nom != "1 qatlam":
                continue                 # nazorat: bir tomonlama yetarlimi
            aniq = []
            for s in range(3):
                m = orgat(L, ikki, vazifa, s)
                with torch.no_grad():
                    b = m(Xv).argmax(2)
                aniq.append((b[:, ichki] == yv[:, ichki]).float().mean().item())
            natija[(vazifa, nom)] = np.array(aniq)
    print(f"  {'vazifa':<9} " + " ".join(f"{n:>15}" for n in sozlamalar))
    for vazifa in ["kelajak", "o'tmish"]:
        print(f"  {vazifa:<9} " + " ".join(
            f"{natija[(vazifa, n)].mean():>15.4f}" if (vazifa, n) in natija
            else f"{'-':>15}" for n in sozlamalar))

    xs = Xv[:, ichki].flatten()
    optimal = np.mean([(9 - v) / 10 if v <= 4 else (v + 1) / 10
                       for v in xs.tolist()])
    print(f"\n  faqat x[t] ni bilgan eng yaxshi taxmin (kelajak): {optimal:.4f}")

    print("\n=== 4. Juftlashgan farq (kelajak): ikki tomonlama - 1 qatlam ===")
    for vazifa in ["kelajak"]:
        farq = natija[(vazifa, "ikki tomonlama")] - natija[(vazifa, "1 qatlam")]
        se = farq.std(ddof=1) / np.sqrt(len(farq))
        if abs(farq.mean()) <= 2 * se:
            xulosa = "sezilarli farq yo'q"
        elif farq.mean() > 0:
            xulosa = "ikki tomonlama sezilarli yaxshi"
        else:
            xulosa = "bir tomonlama sezilarli yaxshi"
        print(f"  {vazifa:<9} {farq.mean():>+8.4f}  SE {se:.4f}  -> {xulosa}")
    print("  ⭐ Kelajak kerak bo'lsa - ikki tomonlama; chuqurlik uni o'rnini bosmaydi")


if __name__ == "__main__":
    main()

=== 1. Shakllar ===
  sozlama                        chiq          h_n  parametr
  qatlam=1, ikki=False      (3, 7, 4)    (1, 3, 4)       176
  qatlam=1, ikki=True       (3, 7, 8)    (2, 3, 4)       352
  qatlam=2, ikki=False      (3, 7, 4)    (2, 3, 4)       336
  qatlam=2, ikki=True       (3, 7, 8)    (4, 3, 4)       800
  chiq: (B, T, yo'nalish*h), h_n: (qatlam*yo'nalish, B, h)

=== 2. h_n tartibi (2 qatlam, ikki tomonlama) ===
  oldinga yo'nalish: h_n[-1, 0] == chiq[:, -1, :h]  farq 0.0e+00
  orqaga yo'nalish:  h_n[-1, 1] == chiq[:, 0, h:]   farq 0.0e+00
  xato taxmin:       h_n[-1, 1] vs chiq[:, -1, h:]  farq 0.149
  orqaga LSTM ning yakuniy holati - BIRINCHI pozitsiyada

=== 3. Belgilash: kelajakka va o'tmishga bog'liq yorliq ===
  vazifa           1 qatlam        2 qatlam  ikki tomonlama
  kelajak            0.7403          0.7450          1.0000
  o'tmish            1.0000               -               -

  faqat x[t] ni bilgan eng yaxshi taxmin (kelajak): 0.7490

=== 4. Juftlashgan farq (kelajak): ikki tomonlama - 1 qatlam ===
  kelajak    +0.2597  SE 0.0098  -> ikki tomonlama sezilarli yaxshi
  ⭐ Kelajak kerak bo'lsa - ikki tomonlama; chuqurlik uni o'rnini bosmaydi

chiq, h_n = lstm(x)            # h_n aslida (h_n, c_n) juftligi   # ⚠️
chiq, (h_n, c_n) = lstm(x)                                        # ✅

lstm = nn.LSTM(16, 32); lstm(x)          # x: (B, T, 16)          # ⚠️
lstm = nn.LSTM(16, 32, batch_first=True)                          # ✅

oxirgi = chiq[:, -1]                                              # ⚠️
oxirgi = chiq[torch.arange(len(uz)), uz - 1]                      # ✅

pack_padded_sequence(e, uz, batch_first=True)   # uz saralanmagan # ⚠️
pack_padded_sequence(e, uz, batch_first=True, enforce_sorted=False)  # ✅

orqaga = chiq[:, -1, h:]       # orqaga LSTM ning BIRINCHI qadami  # ⚠️
orqaga = chiq[:, 0, h:]        # yoki h_n.view(L, 2, B, h)[-1, 1]  # ✅

nn.LSTM(16, 32, num_layers=1, dropout=0.3)   # UserWarning       # ⚠️
nn.LSTM(16, 32, num_layers=2, dropout=0.3)   # yoki alohida nn.Dropout  # ✅

nn.LSTM(d, h, bidirectional=True)   # keyingi belgini "ko'rib" oladi  # ⚠️
nn.LSTM(d, h)                       # faqat o'tmish                 # ✅

1.  # LSTM ning to'rtta darvozasi/bloki va nn.LSTM dagi tartibi?
2.  # c_t formulasi?
3.  # h_t formulasi?
4.  # nn.LSTM(8, 16) parametrlari soni?
5.  # nn.GRU(8, 16) parametrlari soni?
6.  # boshlang'ich f qiymati taxminan qancha va nima uchun?
7.  # d c_t / d c_{t-1} asosiy yo'lda nimaga teng?
8.  # nn.LSTM(5, 4, num_layers=2, bidirectional=True) da h_n shakli (B=3)?
9.  # shu modelda chiq shakli (T=7)?
10. # ikki tomonlamada orqaga yo'nalishning yakuniy holati qaysi pozitsiyada?
11. # padding bor batchda chiq[:, -1] qaysi namuna uchun to'g'ri?
12. # pack_padded_sequence dan keyin h_n[-1] nimani beradi?

1.  chiq, h = nn.LSTM(16, 32, batch_first=True)(x)
    oxirgi = h[-1]

2.  lstm = nn.LSTM(16, 32)
    chiq, _ = lstm(x)          # x: (B, T, 16)

3.  p = pack_padded_sequence(e, uz, batch_first=True)   # uz = [3, 5, 2]

4.  bi = nn.LSTM(16, 32, bidirectional=True, batch_first=True)
    chiq, _ = bi(x)
    jumla = torch.cat([chiq[:, -1, :32], chiq[:, -1, 32:]], dim=1)

5.  chiq, _ = lstm(emb(X))    # X da PAD bor
    logit = linear(chiq[:, -1])