28.8-dars: Chuqur RL va policy gradient

DISKRETLASH: har o'lchovni n ta bo'lakka -> n^d katak
  tayoq (d = 4):   n = 10 -> 10 000 katak
  robot (d = 10):  n = 10 -> 10^10 katak
  tasvir (84x84):  sanab bo'lmaydi

IKKI MUAMMO:
  1. DAG'AL bo'laklar: farq qilishi kerak bo'lgan holatlar bitta
     katakka tushadi -> siyosat ularni ajrata olmaydi (aliasing)
  2. MAYDA bo'laklar: har katak ALOHIDA o'rganiladi -> umumlashtirish
     yo'q; kataklarning ko'pi hech ko'rilmaydi, ko'rilganlari kam
     tajribaga ega

FUNKSIYA APPROKSIMATSIYASI:
  Q(s, a) ~ Q(s, a; w)       w - og'irliklar (chiziqli yoki neyron tarmoq)
  o'xshash holatlar o'xshash qiymat oladi -> UMUMLASHTIRISH
  parametrlar soni holatlar soniga bog'liq emas

MAQSAD:  y = r + gamma * max_a' Q(s', a'; w_nishon) * (1 - tugadi)
YO'QOTISH: L(w) = E[ huber( Q(s, a; w) - y ) ]
GRADIENT: faqat Q(s, a; w) orqali (y - o'zgarmas deb olinadi,
          "semi-gradient")

NAIV VARIANT (har qadamda oxirgi o'tish bilan o'qitish) ikki sababdan buziladi:
  1. KORRELYATSIYA: ketma-ket o'tishlar deyarli bir xil
     (0.02 s dan keyin tayoq deyarli o'sha joyda) -> gradient bir
     yo'nalishga "suriladi", SGD ning i.i.d. taxmini buziladi
  2. HARAKATLANUVCHI NISHON: y ham xuddi o'sha w ga bog'liq ->
     Q(s) ni oshirsak, Q(s') ham oshadi (s' ~ s), y ham oshadi ->
     "o'z dumini quvish", qiymatlar portlashi mumkin

DQN (Mnih va boshq., 2015) ikki yechimi:
  EXPERIENCE REPLAY: o'tishlar (s, a, r, s', tugadi) xotiraga yoziladi;
    o'qitish xotiradan TASODIFIY partiya bilan -> korrelyatsiya
    buziladi, har o'tish ko'p marta ishlatiladi (namuna samaradorligi)
  TARGET NETWORK: y ni hisoblash uchun alohida nusxa w_nishon,
    har C qadamda w_nishon <- w -> nishon C qadam davomida qimirlamaydi

MAQSAD:  J(theta) = E_pi[ G_0 ]  - kutilgan qaytish; uni OSHIRAMIZ

POLICY GRADIENT TEOREMASI:
  grad J = E_pi[ sum_t grad log pi(a_t | s_t) * G_t ]
  G_t = r_{t+1} + gamma r_{t+2} + ...   (reward-to-go: t dan KEYINgi mukofot)

REINFORCE (Williams, 1992):
  1. siyosat bilan M ta epizod yig'ish
  2. har qadam uchun G_t
  3. yo'qotish: L = - (1/M) sum_epizod sum_t log pi(a_t|s_t) * G_t
  4. theta <- theta - lr * grad L   (gradient KO'TARILISHI)

INTUITSIYA: katta qaytishga olib kelgan harakatlar ehtimoli oshadi,
            kichigiga olib kelganlar - kamayadi

grad J = E[ sum_t grad log pi(a_t|s_t) * (G_t - b(s_t)) ]

b(s) HARAKATGA bog'liq bo'lmasa, kutilma O'ZGARMAYDI:
  E_a[ grad log pi(a|s) * b(s) ] = b(s) * grad sum_a pi(a|s) = b(s) * grad 1 = 0

LEKIN DISPERSIYA KAMAYADI:
  hamma G_t musbat (tayoq: +1 har qadam) -> baseline siz HAR harakat
  ehtimoli oshiriladi, faqat har xil kuch bilan; signal = kichik farq
  katta shovqin ichida
  b(s) ~ V(s): "bu holatda odatdagidan yaxshi/yomon bo'ldimi?"
  G_t - V(s_t) = USTUNLIK (advantage) bahosi

VARIANTLAR:
  b = o'rtacha qaytish (bitta son)         - oddiy, ancha yordam beradi
  b = V(s; phi) - o'rganiladigan critic    - yaxshiroq, lekin critic
                                              sifati muhim

ACTOR  - siyosat pi(a|s; theta)
CRITIC - qiymat V(s; phi)

1-QADAMLI USTUNLIK (TD):   A_t = r_{t+1} + gamma V(s_{t+1}) - V(s_t) = delta_t
  + dispersiya juda kichik (bitta qadam tasodifiyligi)
  - SILJISH: critic noto'g'ri bo'lsa, ustunlik ham noto'g'ri
  - epizod tugashini kutmaydi

GAE (generalized advantage estimation, lambda):
  A_t = delta_t + (gamma lambda) delta_{t+1} + (gamma lambda)^2 delta_{t+2} + ...
  lambda = 0 -> 1-qadamli TD (kam dispersiya, ko'p siljish)
  lambda = 1 -> G_t - V(s_t)  (siljishsiz, ko'p dispersiya)
  amalda lambda = 0.9-0.97

A2C (advantage actor-critic): ko'p parallel muhit, n-qadamli yoki GAE
  ustunligi, actor va critic birga yangilanadi

nisbat:  rho_t(theta) = pi_theta(a_t|s_t) / pi_eski(a_t|s_t)

PPO-CLIP maqsadi:
  L = E[ min( rho_t * A_t,  clip(rho_t, 1 - eps, 1 + eps) * A_t ) ]
  eps ~ 0.2: yangi siyosat eskisidan har harakatda ~20% dan ko'p
  farq qila olmaydi -> bitta partiyada BIR NECHA epoch o'qitish xavfsiz

AMALDA: actor + critic (GAE) + entropiya bonusi + clip
        -> hozirgi eng keng tarqalgan "standart" algoritm

# PPO-CLIP yo'qotishining asosi (faqat g'oya; to'liq PPO - stable-baselines3 kabi
# kutubxonalarda: PPO("MlpPolicy", env).learn(total_timesteps=100_000))
import torch


def ppo_yoqotish(logp_yangi, logp_eski, ustunlik, eps=0.2):
    nisbat = torch.exp(logp_yangi - logp_eski)
    a = nisbat * ustunlik
    b = torch.clamp(nisbat, 1 - eps, 1 + eps) * ustunlik
    return -torch.min(a, b).mean()

NAMUNA SAMARASIZLIGI:
  tayoq - 4 ta son, 2 ta harakat; baribir:
    jadvalli Q-learning:   ~60-100 ming qadam  -> ~96-138 (1-misol)
    DQN:                   10 ming qadam       -> ~183    (2-misol)
    REINFORCE:             ~200 ming qadam     -> ~198    (3-misol)
  Atari o'yinlari: 10-200 million kadr; robotlar: simulyatorda yillar
  sabab: mukofot - kuchsiz signal ("yaxshi/yomon", "qaysi qadam" emas)

URUG'GA SEZGIRLIK:
  bir xil kod, boshqa urug' -> butunlay boshqa natija
  2-misol, "replay yo'q": [63, 9, 11, 55]
  3-misol, GAE:           [140.9, 169.6, 186.2]
  QOIDA: kamida 3-5 urug', o'rtacha + SE (yoki ishonch oralig'i),
         juftlashgan farq; "eng yaxshi urug'" ni e'lon qilmang

AGENT MUKOFOTNI OPTIMALLASHTIRADI, NIYATINGIZNI EMAS.

MUKOFOTNI ALDASH (reward hacking / specification gaming):
  mukofot "proksi" - biz xohlagan narsaning taxminiy o'lchovi;
  agent proksi va haqiqiy maqsad orasidagi bo'shliqni topadi
  misollar: qayiq poygasida bitta joyda aylanib bonus yig'ish;
  "yaqinlashish" uchun mukofot -> maqsad yonida aylanish (4-misol)

XAVFSIZ SHAKLLASH (potential-based shaping, Ng va boshq., 1999):
  F(s, s') = gamma * phi(s') - phi(s)
  sikl bo'ylab yig'indi ~ 0 -> aylanib mukofot yig'ib bo'lmaydi
  OPTIMAL SIYOSATNI O'ZGARTIRMAYDI (isbotlangan), faqat yo'naltiradi

AMALIY QOIDALAR:
  1. haqiqiy maqsadni ALOHIDA o'lchang (proksi mukofotdan tashqari)
  2. agent xulqini KO'RING (traektoriyalar), faqat egri chiziqni emas
  3. qo'shimcha mukofot kerak bo'lsa - potensial shaklida

SAVOL                                         JAVOB
harakat keyingi holatni o'zgartiradimi?  yo'q -> kontekstli bandit 27.13-bob
to'g'ri javob (belgi) ma'lummi?          ha  -> supervised learning
tarixiy tasodifiy tajriba logi bormi?    ha  -> avval supervised/oflayn baholash
model (fizika, qoidalar) ma'lummi?       ha  -> rejalashtirish, DP, optimallashtirish
xato qimmat, simulyator yo'q?            ha  -> RL dan ehtiyot bo'ling

import torch
from torch import nn

# DQN o'qitish qadami (replay + target)
sb, ab, rb, s2b, tb = xotira.namuna(B=128)                     # tasodifiy partiya
q_sa = onlayn(sb).gather(1, ab[:, None])[:, 0]
with torch.no_grad():
    y = rb + gamma * nishon(s2b).max(1).values * (1 - tb)        # tugadi (kesildi emas!)
loss = nn.functional.smooth_l1_loss(q_sa, y)
opt.zero_grad(); loss.backward(); opt.step()
if qadam % C == 0:
    nishon.load_state_dict(onlayn.state_dict())

# REINFORCE + baseline
logp = torch.log_softmax(siyosat(S), -1).gather(-1, A[..., None])[..., 0]
ustunlik = G - V(S).detach()                                    # b(s) = V(s)
loss_pi = -(logp * ustunlik * niqob).sum() / n_epizod
loss_v = (((V(S) - G) ** 2) * niqob).sum() / niqob.sum()

# GAE
delta = r + gamma * V_keyin * (1 - tugadi) - V
# A_t = delta_t + gamma * lam * A_{t+1}   (oxiridan boshiga)

# potensial shaklidagi qo'shimcha mukofot (siyosatni o'zgartirmaydi)
r_yangi = r + gamma * phi(s2) - phi(s)

jadval -> o'lchov la'nati; funksiya -> umumlashtirish
DQN = Q-tarmoq + replay (korrelyatsiya) + target (harakatlanuvchi nishon)
REINFORCE: grad = E[grad log pi * (G - b)]; baseline - dispersiya kamayadi
A2C/GAE: critic - kam dispersiya, lekin siljish; PPO - clip bilan xavfsiz qadam
amaliyot: millionlab qadam, urug'ga sezgir, mukofotni aldaydi
avval so'rang: bandit yoki supervised yetmaydimi?

"""Jadvalli Q-learning uzluksiz holatda: diskretlash va o'lchov la'nati."""

import math
import random

import numpy as np


class Tayoq:
    """"Tayoqni muvozanatda ushlash" (CartPole fizikasi, Euler qadami 0.02 s).

    holat: (x, x_tezlik, burchak, burchak_tezlik); harakat 0 - chapga,
    1 - o'ngga itarish (10 N). Har qadam +1. Burchak > 12 daraja yoki
    |x| > 2.4 - tugadi; 200 qadam - kesildi (muvaffaqiyat).
    """

    def reset(self, seed=None):
        self.rng = random.Random(seed)
        self.s = [self.rng.uniform(-0.05, 0.05) for _ in range(4)]
        self.t = 0
        return tuple(self.s), {}

    def step(self, a):
        x, v, th, w = self.s
        kuch = 10.0 if a == 1 else -10.0
        cos, sin = math.cos(th), math.sin(th)
        tmp = (kuch + 0.05 * w * w * sin) / 1.1
        w_acc = (9.8 * sin - cos * tmp) / (0.5 * (4 / 3 - 0.1 * cos * cos / 1.1))
        x_acc = tmp - 0.05 * w_acc * cos / 1.1
        self.s = [x + 0.02 * v, v + 0.02 * x_acc, th + 0.02 * w, w + 0.02 * w_acc]
        self.t += 1
        tugadi = abs(self.s[0]) > 2.4 or abs(self.s[2]) > 0.2095
        return tuple(self.s), 1.0, tugadi, self.t >= 200 and not tugadi, {}


CHEGARA = [(-2.4, 2.4), (-3.0, 3.0), (-0.21, 0.21), (-3.5, 3.5)]


def katak(s, n):
    """4 o'lchovli holat -> bitta butun son (har o'lchov n ta bo'lakka)."""
    k = 0
    for qiymat, (lo, hi) in zip(s, CHEGARA):
        i = int((qiymat - lo) / (hi - lo) * n)
        k = k * n + min(max(i, 0), n - 1)
    return k


def orgat(n, urug, nuqtalar=(100, 300, 1000), alfa=0.2, gamma=0.99):
    """Q-learning; har nazorat nuqtasida ochko'z siyosat bahosi olinadi."""
    rng = random.Random(urug)
    env = Tayoq()
    Q = [[0.0, 0.0] for _ in range(n ** 4)]
    korilgan = set()
    qadamlar, baholar = 0, []
    for e in range(nuqtalar[-1]):
        eps = max(0.02, 1.0 - e / 300)
        s, _ = env.reset(seed=urug * 100_000 + e)
        k = katak(s, n)
        while True:
            q = Q[k]
            a = rng.randrange(2) if rng.random() < eps else (0 if q[0] >= q[1] else 1)
            s2, r, tugadi, kesildi, _ = env.step(a)
            k2 = katak(s2, n)
            korilgan.add(k)
            maqsad = r if tugadi else r + gamma * max(Q[k2])
            q[a] += alfa * (maqsad - q[a])
            k = k2
            qadamlar += 1
            if tugadi or kesildi:
                break
        if e + 1 in nuqtalar:
            baholar.append(baholash(Q, n, urug))
    return baholar, len(korilgan), qadamlar


def baholash(Q, n, urug, epizodlar=20):
    env = Tayoq()
    natija = []
    for e in range(epizodlar):
        s, _ = env.reset(seed=10_000_000 + urug * 1000 + e)
        jami = 0.0
        while True:
            q = Q[katak(s, n)]
            s, r, tugadi, kesildi, _ = env.step(0 if q[0] >= q[1] else 1)
            jami += r
            if tugadi or kesildi:
                break
        natija.append(jami)
    return float(np.mean(natija))


def main() -> None:
    print("=== 1. Diskretlash: jadval hajmi o'lchov soniga qarab ===")
    print(f"  {'bo_lak':>7} {'1 o_lchov':>10} {'4 o_lchov':>10} {'10 o_lchov':>12}")
    for n in [3, 6, 10, 20]:
        print(f"  {n:>7} {n:>10} {n ** 4:>10} {n ** 10:>12.2e}")
    print("  (robot: 10 ta sensor -> 10 o'lchov; kamera -> minglab o'lchov)")

    env = Tayoq()
    rng = random.Random(0)
    tasodifiy = []
    for e in range(200):
        env.reset(seed=e)
        jami, tugadi, kesildi = 0.0, False, False
        while not (tugadi or kesildi):
            _, r, tugadi, kesildi, _ = env.step(rng.randrange(2))
            jami += r
        tasodifiy.append(jami)
    print(f"\n  tasodifiy siyosat (bazaviy): {np.mean(tasodifiy):.1f} qadam")

    print("\n=== 2. Jadvalli Q-learning, 5 urug' ===")
    bolaklar = [3, 6, 10, 20]
    print(f"  {'bo_lak':>7} {'kataklar':>9} {'ko_rilgan':>10} {'ulush':>7} "
          f"{'qadamlar':>9}   ochko'z baho: 100 / 300 / 1000 epizod")
    natija = {}
    for n in bolaklar:
        baho, kor, qad = [], [], []
        for u in range(5):
            b, k, q = orgat(n, u)
            baho.append(b)
            kor.append(k)
            qad.append(q)
        natija[n] = np.array(baho)                    # [urug', nazorat nuqtasi]
        orta = natija[n].mean(0)
        print(f"  {n:>7} {n ** 4:>9} {np.mean(kor):>10.0f} "
              f"{np.mean(kor) / n ** 4:>7.3f} {np.mean(qad):>9.0f}   "
              f"{orta[0]:>6.1f} {orta[1]:>6.1f} {orta[2]:>6.1f}")
    print("  (qadamlar - 1000 epizoddagi jami muhit qadamlari)")

    print("\n=== 3. 1000 epizoddan keyin: urug'lar bo'yicha ===")
    for n in bolaklar:
        print(f"  {n:>2} bo'lak: {np.round(natija[n][:, -1], 1).tolist()}")
    eng = max(bolaklar, key=lambda n: natija[n][:, -1].mean())
    print(f"  eng yaxshi: {eng} bo'lak ({natija[eng][:, -1].mean():.1f})")
    munosib = [eng]
    for n in bolaklar:
        if n == eng:
            continue
        d = natija[n][:, -1] - natija[eng][:, -1]
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  {n:>2} bo'lak - {eng} bo'lak: {d.mean():+6.1f}, SE {se:5.1f}, "
              f"sezilarli: {abs(d.mean()) > 2 * se}")
        if abs(d.mean()) <= 2 * se:
            munosib.append(n)
    print(f"  eng kichik munosib jadval: {min(munosib)} bo'lak "
          f"({min(munosib) ** 4} katak)")
    print("  dag'al: o'xshash holatlar birlashadi, lekin farqlanishi kerak "
          "bo'lganlar ham")
    print("  mayda: har katak alohida o'rganiladi - umumlashtirish yo'q, "
          "ma'lumot yetmaydi")
    print("  ⭐ yechim: Q(s, a) ni funksiya (chiziqli yoki neyron tarmoq) bilan "
          "yaqinlashtirish")


if __name__ == "__main__":
    main()

=== 1. Diskretlash: jadval hajmi o'lchov soniga qarab ===
   bo_lak  1 o_lchov  4 o_lchov   10 o_lchov
        3          3         81     5.90e+04
        6          6       1296     6.05e+07
       10         10      10000     1.00e+10
       20         20     160000     1.02e+13
  (robot: 10 ta sensor -> 10 o'lchov; kamera -> minglab o'lchov)

  tasodifiy siyosat (bazaviy): 23.0 qadam

=== 2. Jadvalli Q-learning, 5 urug' ===
   bo_lak  kataklar  ko_rilgan   ulush  qadamlar   ochko'z baho: 100 / 300 / 1000 epizod
        3        81         43   0.536     58158     29.5   68.3   96.1
        6      1296        225   0.174    102230     39.0  125.0  136.2
       10     10000        423   0.042     87706     85.7   99.1  137.6
       20    160000       1138   0.007     59188     47.1   62.2   74.3
  (qadamlar - 1000 epizoddagi jami muhit qadamlari)

=== 3. 1000 epizoddan keyin: urug'lar bo'yicha ===
   3 bo'lak: [89.0, 128.0, 92.7, 95.8, 74.8]
   6 bo'lak: [124.5, 133.5, 126.4, 155.4, 141.4]
  10 bo'lak: [98.0, 120.8, 191.6, 173.5, 104.0]
  20 bo'lak: [103.6, 76.9, 45.0, 86.2, 60.0]
  eng yaxshi: 10 bo'lak 137.6-bob
   3 bo'lak - 10 bo'lak:  -41.5, SE  20.2, sezilarli: True
   6 bo'lak - 10 bo'lak:   -1.3, SE  18.5, sezilarli: False
  20 bo'lak - 10 bo'lak:  -63.2, SE  25.5, sezilarli: True
  eng kichik munosib jadval: 6 bo'lak (1296 katak)
  dag'al: o'xshash holatlar birlashadi, lekin farqlanishi kerak bo'lganlar ham
  mayda: har katak alohida o'rganiladi - umumlashtirish yo'q, ma'lumot yetmaydi
  ⭐ yechim: Q(s, a) ni funksiya (chiziqli yoki neyron tarmoq) bilan yaqinlashtirish

"""DQN noldan: experience replay va target network - har biri nima uchun kerak."""

import numpy as np
import torch
from torch import nn

torch.set_num_threads(1)


class Tayoqlar:
    """K ta mustaqil "tayoqni muvozanatda ushlash" muhiti (CartPole fizikasi).

    holat: [x, x_tezlik, burchak, burchak_tezlik]; harakat: 0 chapga, 1 o'ngga
    itarish. Har qadam +1; tayoq 12 darajadan og'sa yoki arava 2.4 dan
    chiqsa - tugadi; 200 qadamda - kesildi.
    """

    def __init__(self, K, rng):
        self.K, self.rng = K, rng
        self.s = np.zeros((K, 4))
        self.t = np.zeros(K, dtype=int)
        for k in range(K):
            self._reset(k)

    def _reset(self, k):
        self.s[k] = self.rng.uniform(-0.05, 0.05, 4)
        self.t[k] = 0

    def step(self, a):
        x, v, th, w = self.s.T
        kuch = np.where(a == 1, 10.0, -10.0)
        cos, sin = np.cos(th), np.sin(th)
        tmp = (kuch + 0.05 * w ** 2 * sin) / 1.1
        w_acc = (9.8 * sin - cos * tmp) / (0.5 * (4 / 3 - 0.1 * cos ** 2 / 1.1))
        x_acc = tmp - 0.05 * w_acc * cos / 1.1
        self.s = np.stack([x + 0.02 * v, v + 0.02 * x_acc,
                           th + 0.02 * w, w + 0.02 * w_acc], axis=1)
        self.t += 1
        tugadi = (np.abs(self.s[:, 0]) > 2.4) | (np.abs(self.s[:, 2]) > 0.2095)
        kesildi = (self.t >= 200) & ~tugadi
        keyingi = self.s.copy()
        for k in np.flatnonzero(tugadi | kesildi):
            self._reset(k)
        return keyingi, np.ones(self.K), tugadi, kesildi


class KopTarmoq(nn.Module):
    """K ta MUSTAQIL MLP (4 -> 32 -> 32 -> 2) bitta tensor ichida.

    Bu faqat hisoblash hiylasi: K ta agent parallel o'rganadi, lekin
    og'irliklari, gradientlari va Adam holati alohida (loss yig'indi).
    """

    def __init__(self, K, kir=4, yash=32, chiq=2):
        super().__init__()

        def qatlam(a, b):
            chegara = 1 / np.sqrt(a)
            W = (torch.rand(K, a, b) * 2 - 1) * chegara
            return nn.Parameter(W), nn.Parameter(torch.zeros(K, 1, b))

        self.W1, self.b1 = qatlam(kir, yash)
        self.W2, self.b2 = qatlam(yash, yash)
        self.W3, self.b3 = qatlam(yash, chiq)

    def forward(self, x):                               # x: [K, B, 4]
        h = torch.relu(torch.baddbmm(self.b1, x, self.W1))
        h = torch.relu(torch.baddbmm(self.b2, h, self.W2))
        return torch.baddbmm(self.b3, h, self.W3)        # [K, B, 2]


def baholash(tarmoq, K, urug, epizodlar=5):
    """Ochko'z siyosat: har agent uchun `epizodlar` ta epizod o'rtachasi."""
    env = Tayoqlar(K * epizodlar, np.random.default_rng(urug))
    jami = np.zeros(K * epizodlar)
    tirik = np.ones(K * epizodlar, dtype=bool)
    with torch.no_grad():
        for _ in range(200):
            x = torch.tensor(env.s, dtype=torch.float32).view(K, epizodlar, 4)
            a = tarmoq(x).argmax(2).view(-1).numpy()
            _, _, tugadi, kesildi = env.step(a)
            jami += tirik
            tirik &= ~(tugadi | kesildi)
    return jami.reshape(K, epizodlar).mean(1)


def dqn(replay, target, urug, qadamlar=10_000, B=128, gamma=0.99):
    """replay, target - [K] bool massivlar (har agent uchun variant)."""
    K = len(replay)
    torch.manual_seed(urug)
    rng = np.random.default_rng(urug)
    onlayn = KopTarmoq(K)
    nishon = KopTarmoq(K)
    nishon.load_state_dict(onlayn.state_dict())
    opt = torch.optim.Adam(onlayn.parameters(), lr=2e-3, foreach=True)
    N = 10_000                                           # xotira (replay buffer)
    buf_s = np.zeros((K, N, 4), dtype=np.float32)
    buf_a = np.zeros((K, N), dtype=np.int64)
    buf_s2 = np.zeros((K, N, 4), dtype=np.float32)
    buf_t = np.zeros((K, N), dtype=np.float32)
    env = Tayoqlar(K, rng)
    kk = np.arange(K)[:, None]
    replay_t = torch.tensor(replay)[:, None]
    target_t = torch.tensor(target)[:, None]
    epizod_jami = np.zeros(K)
    tarix = [[] for _ in range(K)]
    baholar, q_max = [], []
    for t in range(qadamlar):
        eps = max(0.05, 1.0 - t / 4000)
        with torch.no_grad():
            q = onlayn(torch.tensor(env.s, dtype=torch.float32)[:, None, :])[:, 0]
        a = q.argmax(1).numpy()
        tasodif = rng.random(K) < eps
        a[tasodif] = rng.integers(0, 2, tasodif.sum())
        i = t % N
        buf_s[:, i] = env.s
        s2, r, tugadi, kesildi = env.step(a)
        buf_a[:, i], buf_s2[:, i], buf_t[:, i] = a, s2, tugadi
        epizod_jami += r
        for k in np.flatnonzero(tugadi | kesildi):
            tarix[k].append(epizod_jami[k])
            epizod_jami[k] = 0.0
        if t >= 500 and t % 4 == 0:                       # har 4 qadamda o'qitish
            tasodifiy_idx = rng.integers(0, min(t + 1, N), (K, B))   # tasodifiy eski
            ketma_ket = np.broadcast_to((t - np.arange(B)) % N, (K, B))  # oxirgi B ta
            idx = np.where(replay[:, None], tasodifiy_idx, ketma_ket)
            sb = torch.from_numpy(buf_s[kk, idx])
            s2b = torch.from_numpy(buf_s2[kk, idx])
            ab = torch.from_numpy(buf_a[kk, idx])
            tb = torch.from_numpy(buf_t[kk, idx])
            q_hammasi = onlayn(torch.cat([sb, s2b], dim=1))      # bitta chaqiruv
            q_sa = q_hammasi[:, :B].gather(2, ab[:, :, None])[:, :, 0]
            with torch.no_grad():
                q_nishon = nishon(s2b).max(2).values
                q_ozi = q_hammasi[:, B:].max(2).values          # target YO'Q: o'zi
                q_keyin = torch.where(target_t, q_nishon, q_ozi)
                y = 1.0 + gamma * q_keyin * (1 - tb)
            loss = nn.functional.smooth_l1_loss(q_sa, y, reduction="none").mean(1).sum()
            opt.zero_grad()
            loss.backward()
            opt.step()
        if t % 400 == 0:
            nishon.load_state_dict(onlayn.state_dict())
        if (t + 1) % 2000 == 0:
            baholar.append(baholash(onlayn, K, urug=1000 + t))
            with torch.no_grad():
                x0 = torch.tensor(np.random.default_rng(7).uniform(-0.05, 0.05, (1, 64, 4)),
                                  dtype=torch.float32).expand(K, 64, 4)
                q_max.append(onlayn(x0).max(2).values.mean(1).numpy())
    return np.array(baholar), np.array(q_max), tarix


def main() -> None:
    variantlar = [("to'liq DQN", True, True), ("replay yo'q", False, True),
                  ("target yo'q", True, False)]
    urug_soni = 4
    replay = np.repeat([v[1] for v in variantlar], urug_soni)
    target = np.repeat([v[2] for v in variantlar], urug_soni)
    baholar, q_max, tarix = dqn(replay, target, urug=0)
    V = len(variantlar)
    b = baholar.reshape(-1, V, urug_soni)             # [baholash, variant, urug']
    q = q_max.reshape(-1, V, urug_soni)

    print("=== 1. Ochko'z siyosat bahosi har 2000 qadamda (4 urug' o'rtachasi) ===")
    print(f"  {'variant':<15}" + "".join(f"{(i + 1) * 2:>6}k" for i in range(b.shape[0])))
    for j, (nom, _, _) in enumerate(variantlar):
        print(f"  {nom:<15}" + "".join(f"{x:>7.1f}" for x in b[:, j].mean(1)))
    print("  maksimum 200 (tayoq 200 qadam turdi)")
    env = Tayoqlar(400, np.random.default_rng(5))
    jami, tirik = np.zeros(400), np.ones(400, dtype=bool)
    tas_rng = np.random.default_rng(6)
    for _ in range(200):
        _, _, tugadi, kesildi = env.step(tas_rng.integers(0, 2, 400))
        jami += tirik
        tirik &= ~(tugadi | kesildi)
    print(f"  tasodifiy siyosat (bazaviy): {jami.mean():.1f}")

    print("\n=== 2. Barqarorlik: urug'lar bo'yicha ===")
    print(f"  {'variant':<15} {'oxirgi':>20} {'eng yaxshi':>11} {'qulash':>7} "
          f"{'Q(s0)':>10}")
    for j, (nom, _, _) in enumerate(variantlar):
        oxirgi = b[-1, j]
        eng = b[:, j].max(0)
        print(f"  {nom:<15} {str(np.round(oxirgi).astype(int).tolist()):>20} "
              f"{eng.mean():>11.1f} {np.mean(eng - oxirgi):>7.1f} "
              f"{q[-1, j].mean():>10.1f}")
    print("  Q(s0) - boshlang'ich holatlarda max Q (4 urug' o'rtachasi)")
    print(f"  chegaraviy qiymat: 1 / (1 - 0.99) = {1 / (1 - 0.99):.0f} "
          f"(200 qadamda kesilish - terminal emas)")
    n_nishon = 10_000 // 400
    print(f"  target {n_nishon} marta yangilandi: sum_(k<{n_nishon}) 0.99^k = "
          f"{(1 - 0.99 ** n_nishon) / (1 - 0.99):.1f}")

    print("\n=== 3. O'rganish davomidagi epizod qaytishlari (o'rtacha) ===")
    for j, (nom, _, _) in enumerate(variantlar):
        eps_list = [tarix[j * urug_soni + u] for u in range(urug_soni)]
        n_ep = [len(e) for e in eps_list]
        oxirgi20 = [np.mean(e[-20:]) for e in eps_list]
        print(f"  {nom:<15} epizodlar {n_ep}, oxirgi 20 ta: "
              f"{np.round(oxirgi20, 1).tolist()}")

    print("\n=== 4. Xulosa (natijadan) ===")
    oxirgi_orta = b[-1].mean(1)
    j_best = int(np.argmax(oxirgi_orta))
    print(f"  oxirida eng yaxshi: {variantlar[j_best][0]} ({oxirgi_orta[j_best]:.1f})")
    for j, (nom, _, _) in enumerate(variantlar):
        if j == 0:
            continue
        d = b[-1, j] - b[-1, 0]
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  {nom:<15} - to'liq DQN: {d.mean():+7.1f}, SE {se:.1f}, "
              f"sezilarli: {abs(d.mean()) > 2 * se}")


if __name__ == "__main__":
    main()

=== 1. Ochko'z siyosat bahosi har 2000 qadamda (4 urug' o'rtachasi) ===
  variant             2k     4k     6k     8k    10k
  to'liq DQN         9.5   99.6  155.2  145.4  182.8
  replay yo'q        9.1   35.8   23.3   31.8   34.6
  target yo'q        9.2    9.2    9.2    9.6    9.5
  maksimum 200 (tayoq 200 qadam turdi)
  tasodifiy siyosat (bazaviy): 21.8

=== 2. Barqarorlik: urug'lar bo'yicha ===
  variant                       oxirgi  eng yaxshi  qulash      Q(s0)
  to'liq DQN      [185, 148, 200, 199]       184.7     1.9       22.4
  replay yo'q          [63, 9, 11, 55]        50.0    15.3       23.9
  target yo'q           [10, 9, 10, 9]         9.8     0.3   500078.9
  Q(s0) - boshlang'ich holatlarda max Q (4 urug' o'rtachasi)
  chegaraviy qiymat: 1 / (1 - 0.99) = 100 (200 qadamda kesilish - terminal emas)
  target 25 marta yangilandi: sum_(k<25) 0.99^k = 22.2

=== 3. O'rganish davomidagi epizod qaytishlari (o'rtacha) ===
  to'liq DQN      epizodlar [224, 284, 202, 275], oxirgi 20 ta: [147.0, 125.6, 199.9, 136.8]
  replay yo'q     epizodlar [294, 702, 632, 748], oxirgi 20 ta: [75.6, 9.8, 13.7, 59.4]
  target yo'q     epizodlar [919, 917, 908, 909], oxirgi 20 ta: [9.8, 9.4, 9.8, 9.6]

=== 4. Xulosa (natijadan) ===
  oxirida eng yaxshi: to'liq DQN 182.8-bob
  replay yo'q     - to'liq DQN:  -148.2, SE 14.5, sezilarli: True
  target yo'q     - to'liq DQN:  -173.4, SE 12.1, sezilarli: True

"""Policy gradient noldan: REINFORCE, baseline va actor-critic (torch)."""

import numpy as np
import torch
from torch import nn

torch.set_num_threads(1)
T_MAX, GAMMA = 200, 0.99


class Tayoqlar:
    """Ko'p "tayoq" muhiti birga (CartPole fizikasi, 2-misoldagi kabi).
    Epizod tugagach muhit "muzlaydi" (tirik = False) - qayta boshlanmaydi."""

    def __init__(self, n, rng):
        self.s = rng.uniform(-0.05, 0.05, (n, 4))
        self.tirik = np.ones(n, dtype=bool)

    def step(self, a):
        x, v, th, w = self.s.T
        kuch = np.where(a == 1, 10.0, -10.0)
        cos, sin = np.cos(th), np.sin(th)
        tmp = (kuch + 0.05 * w ** 2 * sin) / 1.1
        w_acc = (9.8 * sin - cos * tmp) / (0.5 * (4 / 3 - 0.1 * cos ** 2 / 1.1))
        x_acc = tmp - 0.05 * w_acc * cos / 1.1
        yangi = np.stack([x + 0.02 * v, v + 0.02 * x_acc,
                          th + 0.02 * w, w + 0.02 * w_acc], axis=1)
        self.s = np.where(self.tirik[:, None], yangi, self.s)
        r = self.tirik.astype(float)
        tugadi = self.tirik & ((np.abs(self.s[:, 0]) > 2.4) | (np.abs(self.s[:, 2]) > 0.2095))
        self.tirik = self.tirik & ~tugadi
        return r, tugadi


class KopTarmoq(nn.Module):
    """K ta mustaqil MLP (4 -> 32 -> chiq) bitta tensorda (2-misoldagi hiyla)."""

    def __init__(self, K, chiq, kir=4, yash=32):
        super().__init__()
        self.W1 = nn.Parameter((torch.rand(K, kir, yash) * 2 - 1) / kir ** 0.5)
        self.b1 = nn.Parameter(torch.zeros(K, 1, yash))
        self.W2 = nn.Parameter((torch.rand(K, yash, chiq) * 2 - 1) / yash ** 0.5 * 0.1)
        self.b2 = nn.Parameter(torch.zeros(K, 1, chiq))

    def forward(self, x):                                  # [K, B, 4]
        h = torch.tanh(torch.baddbmm(self.b1, x, self.W1))
        return torch.baddbmm(self.b2, h, self.W2)


def yigish(siyosat, K, M, rng):
    """Har agent M ta epizod: holatlar, harakatlar, mukofotlar, tugash, tiriklik."""
    env = Tayoqlar(K * M, rng)
    S, A, R, D, T = [], [], [], [], []
    with torch.no_grad():
        for _ in range(T_MAX):
            if not env.tirik.any():
                break
            s = torch.tensor(env.s, dtype=torch.float32).view(K, M, 4)
            p = torch.softmax(siyosat(s), dim=2).view(-1, 2).numpy()
            a = (rng.random(K * M) < p[:, 1]).astype(int)   # siyosatdan namuna
            tirik = env.tirik.copy()
            r, tugadi = env.step(a)
            S.append(s)
            A.append(a.reshape(K, M))
            R.append(r.reshape(K, M))
            D.append(tugadi.reshape(K, M))
            T.append(tirik.reshape(K, M))
    S = torch.stack(S, dim=2)                              # [K, M, vaqt, 4]
    A = torch.tensor(np.stack(A, 2))
    R, D, T = (torch.tensor(np.stack(x, 2), dtype=torch.float32) for x in (R, D, T))
    G = torch.zeros_like(R)                                # reward-to-go G_t
    keyingi = torch.zeros(K, M)
    for t in reversed(range(R.shape[2])):
        keyingi = R[:, :, t] + GAMMA * keyingi
        G[:, :, t] = keyingi
    return S, A, R, D, T, G


def yoqotish(siyosat, qiymat, S, A, R, D, T, G, tur, lam):
    """tur 0: G_t;  tur 1: G_t - V(s_t);  tur 2: GAE(lambda) ustunligi (A2C)."""
    K, M, n, _ = S.shape
    logp = torch.log_softmax(siyosat(S.view(K, M * n, 4)), 2).view(K, M, n, 2)
    logp_a = logp.gather(3, A[..., None])[..., 0]
    V = qiymat(S.view(K, M * n, 4)).view(K, M, n)
    Vd = V.detach()
    V_keyin = torch.cat([Vd[:, :, 1:], torch.zeros(K, M, 1)], 2) * (1 - D)
    delta = (R + GAMMA * V_keyin - Vd) * T                  # TD xatosi
    gae = torch.zeros_like(delta)
    keyingi = torch.zeros(K, M)
    for t in reversed(range(n)):
        keyingi = delta[:, :, t] + GAMMA * lam[:, None] * keyingi * (1 - D[:, :, t])
        gae[:, :, t] = keyingi
    tur = tur.view(K, 1, 1)
    ustunlik = torch.where(tur == 0, G, torch.where(tur == 1, G - Vd, gae))
    loss_pi = -(logp_a * ustunlik * T).sum((1, 2)) / M
    maqsad_v = torch.where(tur == 2, gae + Vd, G)           # critic maqsadi
    loss_v = (((V - maqsad_v) ** 2) * T).sum((1, 2)) / T.sum((1, 2))
    return loss_pi.sum() + loss_v.sum()


def main() -> None:
    variantlar = [("REINFORCE", 0, 0.0), ("REINFORCE + V(s)", 1, 0.0),
                  ("A2C, 1 qadam", 2, 0.0), ("A2C, GAE 0.95", 2, 0.95)]
    urug_soni, M, yangilash = 3, 16, 80
    nV = len(variantlar)
    K = nV * urug_soni
    tur = torch.tensor(np.repeat([v[1] for v in variantlar], urug_soni))
    lam = torch.tensor(np.repeat([v[2] for v in variantlar], urug_soni),
                       dtype=torch.float32)
    torch.manual_seed(0)
    rng = np.random.default_rng(0)
    siyosat, qiymat = KopTarmoq(K, 2), KopTarmoq(K, 1)
    opt = torch.optim.Adam(list(siyosat.parameters()) + list(qiymat.parameters()),
                           lr=1e-2, foreach=True)
    egri, snapshot = [], None
    for u in range(yangilash):
        S, A, R, D, T, G = yigish(siyosat, K, M, rng)
        egri.append(T.sum(2).mean(1).double().numpy())     # o'rtacha epizod uzunligi
        if u == 10:                                        # dispersiya o'lchash uchun
            snapshot = {n: p[urug_soni:urug_soni + 1].clone()
                        for n, p in siyosat.state_dict().items()}
        loss = yoqotish(siyosat, qiymat, S, A, R, D, T, G, tur, lam)
        opt.zero_grad()
        loss.backward()
        opt.step()
    egri = np.array(egri).reshape(yangilash, nV, urug_soni)

    print(f"=== 1. O'rganish egri chizig'i: o'rtacha epizod uzunligi "
          f"({M} epizod/yangilash, {urug_soni} urug') ===")
    print(f"  {'yangilash':<10}" + "".join(f"{v[0]:>18}" for v in variantlar))
    for i, j in [(0, 10), (10, 20), (20, 40), (40, 60), (60, 80)]:
        print(f"  {f'{i}-{j}':<10}" + "".join(
            f"{egri[i:j, v].mean():>18.1f}" for v in range(nV)))
    print("  tasodifiy siyosat (bazaviy) ~ 22 qadam; maksimum 200")
    print(f"  jami epizodlar: {yangilash * M} ta, muhit qadamlari: "
          f"~{int(egri[:, 0].mean() * yangilash * M)} (REINFORCE)")

    print("\n=== 2. Urug'lar bo'yicha (oxirgi 20 yangilash) ===")
    oxir = egri[-20:].mean(0)                              # [variant, urug']
    for v in range(nV):
        print(f"  {variantlar[v][0]:<18} {np.round(oxir[v], 1).tolist()}")
    for v in range(1, nV):
        d = oxir[v] - oxir[0]
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  {variantlar[v][0]:<18} - REINFORCE: {d.mean():+7.1f}, "
              f"SE {se:5.1f}, sezilarli: {abs(d.mean()) > 2 * se}")

    print("\n=== 3. Gradient dispersiyasi: bitta siyosat, 40 mustaqil partiya ===")
    pol = KopTarmoq(1, 2)
    pol.load_state_dict(snapshot)                          # 10-yangilashdagi siyosat
    # shu siyosat uchun critic ni alohida ma'lumotda yaxshilab o'rgatamiz
    Sc, _, _, _, Tc, Gc = yigish(pol, 1, 64, np.random.default_rng(2))
    critic = KopTarmoq(1, 1)
    opt_c = torch.optim.Adam(critic.parameters(), lr=1e-2)
    niqob = Tc.view(-1) > 0
    xc, yc = Sc.view(1, -1, 4)[:, niqob], Gc.view(-1)[niqob]
    for _ in range(300):
        opt_c.zero_grad()
        ((critic(xc).view(-1) - yc) ** 2).mean().backward()
        opt_c.step()
    P, Mb = 40, 8
    S, A, R, D, T, G = yigish(pol, 1, P * Mb, np.random.default_rng(1))
    print(f"  siyosatning o'rtacha epizod uzunligi: {T.sum(2).mean():.1f}")
    with torch.no_grad():
        Vc = critic(S.view(1, -1, 4)).view(G.shape)
    r2 = 1 - ((Vc - G)[T > 0] ** 2).mean() / G[T > 0].var()
    print(f"  critic R^2 (yangi epizodlarda): {r2:.3f}")
    baselinelar = {"baseline yo'q": torch.zeros_like(G),
                   "o'rtacha qaytish": G[T > 0].mean() * torch.ones_like(G),
                   "V(s) critic": Vc}
    grads = {n: [] for n in baselinelar}
    for p in range(P):
        sl = slice(p * Mb, (p + 1) * Mb)
        for nom, b in baselinelar.items():
            pol.zero_grad()
            logp = torch.log_softmax(pol(S[:, sl].reshape(1, -1, 4)), 2)
            logp = logp.view(1, Mb, -1, 2).gather(3, A[:, sl][..., None])[..., 0]
            (-(logp * (G[:, sl] - b[:, sl]) * T[:, sl]).sum() / Mb).backward()
            grads[nom].append(torch.cat([q.grad.flatten()
                                         for q in pol.parameters()]).numpy())
    print(f"  {'baseline':<17} {'|o_rtacha grad|':>16} {'dispersiya (iz)':>16} "
          f"{'nisbat':>7} {'kosinus':>8}")
    asos = None
    for nom, g in grads.items():
        g = np.array(g, dtype=float)
        orta = g.mean(0)
        disp = g.var(0, ddof=1).sum()
        asos = disp if asos is None else asos
        kos = np.mean([gi @ orta / (np.linalg.norm(gi) * np.linalg.norm(orta))
                       for gi in g])
        print(f"  {nom:<17} {np.linalg.norm(orta):>16.2f} {disp:>16.1f} "
              f"{disp / asos:>7.3f} {kos:>8.3f}")
    print("  kosinus - bitta partiya gradienti o'rtacha yo'nalishga qanchalik mos")
    ortalar = {n: np.array(g, dtype=float).mean(0) for n, g in grads.items()}
    b = ortalar["V(s) critic"]
    for nom in ["baseline yo'q", "o'rtacha qaytish"]:
        a = ortalar[nom]
        print(f"  o'rtacha gradientlar kosinusi ({nom} vs V(s)): "
              f"{a @ b / (np.linalg.norm(a) * np.linalg.norm(b)):.3f}")
    print("  ⭐ baseline gradient KUTILMASINI o'zgartirmaydi, dispersiyasini kamaytiradi")


if __name__ == "__main__":
    main()

=== 1. O'rganish egri chizig'i: o'rtacha epizod uzunligi (16 epizod/yangilash, 3 urug') ===
  yangilash          REINFORCE  REINFORCE + V(s)      A2C, 1 qadam     A2C, GAE 0.95
  0-10                    34.8              33.1              28.6              32.8
  10-20                   85.1              73.4              19.6              69.3
  20-40                  180.6             175.9              10.0             157.3
  40-60                  196.5             193.5               9.4             191.1
  60-80                  197.9             197.1               9.4             165.6
  tasodifiy siyosat (bazaviy) ~ 22 qadam; maksimum 200
  jami epizodlar: 1280 ta, muhit qadamlari: ~203186 (REINFORCE)

=== 2. Urug'lar bo'yicha (oxirgi 20 yangilash) ===
  REINFORCE          [197.5, 198.1, 198.0]
  REINFORCE + V(s)   [199.6, 196.9, 194.9]
  A2C, 1 qadam       [9.4, 9.3, 9.4]
  A2C, GAE 0.95      [140.9, 169.6, 186.2]
  REINFORCE + V(s)   - REINFORCE:    -0.7, SE   1.5, sezilarli: False
  A2C, 1 qadam       - REINFORCE:  -188.5, SE   0.2, sezilarli: True
  A2C, GAE 0.95      - REINFORCE:   -32.3, SE  13.1, sezilarli: True

=== 3. Gradient dispersiyasi: bitta siyosat, 40 mustaqil partiya ===
  siyosatning o'rtacha epizod uzunligi: 49.2
  critic R^2 (yangi epizodlarda): 0.449
  baseline           |o_rtacha grad|  dispersiya (iz)  nisbat  kosinus
  baseline yo'q                32.27           4317.2   1.000    0.477
  o'rtacha qaytish             41.17           1640.2   0.380    0.730
  V(s) critic                  37.82           1044.7   0.242    0.806
  kosinus - bitta partiya gradienti o'rtacha yo'nalishga qanchalik mos
  o'rtacha gradientlar kosinusi (baseline yo'q vs V(s)): 0.994
  o'rtacha gradientlar kosinusi (o'rtacha qaytish vs V(s)): 0.993
  ⭐ baseline gradient KUTILMASINI o'zgartirmaydi, dispersiyasini kamaytiradi

"""RL ning amaliy qiyinchiliklari: mukofotni aldash va "RL kerak emas" holati."""

import random

import numpy as np

XARITA = [
    "#########",
    "#S      #",
    "#       #",
    "#   C   #",
    "#       #",
    "#      G#",
    "#########",
]
YONALISH = [(-1, 0), (0, 1), (1, 0), (0, -1)]


class Xona:
    """Robot S dan G ga (+10) boradi. Loyihachi "yordam" uchun C nazorat
    nuqtasiga kirishni ham mukofotlaydi. 100 qadamda epizod kesiladi.

    mukofot turi:
      "siyrak"     - faqat G da +10
      "C +1"       - G da +10 va C ga HAR kirganda +1 (xato dizayn)
      "potensial"  - G da +10 va F = gamma * phi(s') - phi(s),
                     phi(s) = -(G gacha Manhattan masofa)
    """

    def __init__(self, tur, gamma):
        self.tur, self.gamma = tur, gamma
        self.kataklar = [(r, c) for r in range(len(XARITA))
                         for c in range(len(XARITA[0])) if XARITA[r][c] != "#"]
        self.idx = {k: i for i, k in enumerate(self.kataklar)}
        self.n_holat = len(self.kataklar)
        self.start = next(i for i, (r, c) in enumerate(self.kataklar)
                          if XARITA[r][c] == "S")
        self.g = next((r, c) for r, c in self.kataklar if XARITA[r][c] == "G")

    def phi(self, s):
        r, c = self.kataklar[s]
        return -(abs(r - self.g[0]) + abs(c - self.g[1]))

    def reset(self, seed=None):
        self.s, self.t, self.c_soni = self.start, 0, 0
        return self.s, {}

    def step(self, a):
        r, c = self.kataklar[self.s]
        nr, nc = r + YONALISH[a][0], c + YONALISH[a][1]
        eski = self.s
        if XARITA[nr][nc] != "#":
            self.s = self.idx[(nr, nc)]
        self.t += 1
        belgi = XARITA[self.kataklar[self.s][0]][self.kataklar[self.s][1]]
        haqiqiy = 10.0 if belgi == "G" else 0.0            # biz ASLIDA xohlagan narsa
        mukofot = haqiqiy
        if self.tur == "C +1" and belgi == "C" and self.s != eski:
            mukofot += 1.0
            self.c_soni += 1
        if self.tur == "potensial":
            keyingi = 0.0 if belgi == "G" else self.phi(self.s)
            mukofot += self.gamma * keyingi - self.phi(eski)
        return self.s, mukofot, belgi == "G", self.t >= 100, {"haqiqiy": haqiqiy}


def q_learning(tur, urug, gamma=0.95, epizodlar=300, alfa=0.5):
    rng = random.Random(urug)
    env = Xona(tur, gamma)
    Q = [[0.0] * 4 for _ in range(env.n_holat)]
    haqiqiy_jami, qadamlar = 0.0, 0
    for e in range(epizodlar):
        eps = max(0.05, 1.0 - e / 150)
        s, _ = env.reset()
        while True:
            if rng.random() < eps:
                a = rng.randrange(4)
            else:
                m = max(Q[s])
                a = rng.choice([i for i in range(4) if Q[s][i] == m])
            s2, r, tugadi, kesildi, info = env.step(a)
            haqiqiy_jami += info["haqiqiy"]
            qadamlar += 1
            maqsad = r if tugadi else r + gamma * max(Q[s2])
            Q[s][a] += alfa * (maqsad - Q[s][a])
            s = s2
            if tugadi or kesildi:
                break
    return Q, haqiqiy_jami / epizodlar, qadamlar


def ochkoz(Q, tur, gamma=0.95):
    env = Xona(tur, gamma)
    s, _ = env.reset()
    proksi, haqiqiy = 0.0, 0.0
    while True:
        s, r, tugadi, kesildi, info = env.step(int(np.argmax(Q[s])))
        proksi += r
        haqiqiy += info["haqiqiy"]
        if tugadi or kesildi:
            return proksi, haqiqiy, env.t, env.c_soni


def bandit_vazifa(usul, urug, n=3000, gamma=0.9):
    """Chegirma taklifi: mijoz segmenti (10 ta) -> taklif (3 ta) -> xarid (0/1).
    Keyingi mijoz taklifga BOG'LIQ EMAS - bu kontekstli bandit.

    usul: "Q-learning" (holat = segment, gamma = 0.9), "bandit" (gamma = 0),
          "supervised" (tarixiy tasodifiy tajriba logidan jadval, keyin ochko'z)
    """
    rng = np.random.default_rng(urug)
    p = np.random.default_rng(99).uniform(0.02, 0.12, (10, 3))  # haqiqiy ehtimollar
    eng_yaxshi = p.argmax(1)
    Q = np.zeros((10, 3))
    N = np.zeros((10, 3))
    if usul == "supervised":                          # 1500 ta tasodifiy log + qolgani
        seg = rng.integers(0, 10, n // 2)
        tak = rng.integers(0, 3, n // 2)
        xarid = rng.random(n // 2) < p[seg, tak]
        np.add.at(N, (seg, tak), 1)
        np.add.at(Q, (seg, tak), xarid)
        Q = Q / np.maximum(N, 1)
        seg2 = rng.integers(0, 10, n - n // 2)
        tanlov = Q[seg2].argmax(1)
        xarid2 = rng.random(len(seg2)) < p[seg2, tanlov]
        jami = xarid.sum() + xarid2.sum()
        return jami / n, float(np.mean(Q.argmax(1) == eng_yaxshi)), Q.max(1).mean()
    s = rng.integers(10)
    jami = 0
    for t in range(n):
        eps = max(0.05, 1.0 - t / 1000)
        a = rng.integers(3) if rng.random() < eps else int(Q[s].argmax())
        r = float(rng.random() < p[s, a])
        s2 = rng.integers(10)                         # keyingi mijoz - tasodifiy
        N[s, a] += 1
        g = gamma if usul == "Q-learning" else 0.0
        Q[s, a] += (r + g * Q[s2].max() - Q[s, a]) / N[s, a]
        jami += r
        s = s2
    return jami / n, float(np.mean(Q.argmax(1) == eng_yaxshi)), Q.max(1).mean()


def main() -> None:
    urug = list(range(10))
    turlar = ["siyrak", "C +1", "potensial"]

    print("=== 1. Mukofotni aldash (reward hacking): 300 epizod, 10 urug' ===")
    print("  xona 5 x 7; S -> G eng qisqa yo'l 10 qadam; C - yo'l o'rtasida")
    print(f"  {'mukofot':<11} {'proksi':>8} {'haqiqiy':>8} {'G ga yetdi':>11} "
          f"{'qadam':>6} {'C ga kirish':>12} {'o_qitishda':>11} {'jami qadam':>11}")
    natija = {}
    for tur in turlar:
        qator = [q_learning(tur, u) for u in urug]
        ball = [ochkoz(Q, tur) for Q, _, _ in qator]
        proksi, haq, qadam, csoni = (np.array(x) for x in zip(*ball))
        natija[tur] = (haq, np.array([o for _, o, _ in qator]),
                       np.array([q for _, _, q in qator]))
        print(f"  {tur:<11} {proksi.mean():>8.1f} {haq.mean():>8.1f} "
              f"{int((haq > 0).sum()):>8}/10 {np.median(qadam):>6.0f} "
              f"{csoni.mean():>12.1f} {natija[tur][1].mean():>11.2f} "
              f"{natija[tur][2].mean():>11.0f}")
    print("  proksi - agent optimallashtirgan mukofot; haqiqiy - faqat G uchun +10")
    print("  o'qitishda - o'qitish davomida epizodga o'rtacha HAQIQIY mukofot")

    print("\n=== 2. Xulosa (natijadan) ===")
    if natija["C +1"][0].mean() < natija["siyrak"][0].mean():
        print("  'C +1' dizayni: proksi mukofot yuqori, lekin maqsadga yetilmaydi -"
              " agent C atrofida aylanadi")
    print(f"  potensial - siyrak, yakuniy haqiqiy: "
          f"{(natija['potensial'][0] - natija['siyrak'][0]).mean():+.1f}")
    for nom, i in [("o'qitishdagi haqiqiy mukofot", 1), ("jami o'qitish qadamlari", 2)]:
        d = natija["potensial"][i] - natija["siyrak"][i]
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  potensial - siyrak, {nom}: {d.mean():+.2f}, SE {se:.2f}, "
              f"sezilarli: {abs(d.mean()) > 2 * se}")

    print("\n=== 3. RL kerak emas: kontekstli bandit (3000 mijoz, 10 urug') ===")
    print(f"  {'usul':<12} {'xarid ulushi':>13} {'to_g_ri taklif':>15} {'max Q o_rt.':>12}")
    kon = {}
    for usul in ["Q-learning", "bandit", "supervised"]:
        r = np.array([bandit_vazifa(usul, u) for u in urug])
        kon[usul] = r
        print(f"  {usul:<12} {r[:, 0].mean():>13.4f} {r[:, 1].mean():>13.2f}/1 "
              f"{r[:, 2].mean():>12.3f}")
    p = np.random.default_rng(99).uniform(0.02, 0.12, (10, 3))
    print(f"  mumkin bo'lgan maksimum (eng yaxshi taklif): {p.max(1).mean():.4f}, "
          f"tasodifiy taklif: {p.mean():.4f}")
    for usul in ["bandit", "supervised"]:
        for nom, i, f in [("xarid", 0, 4), ("to'g'ri taklif", 1, 2)]:
            d = kon[usul][:, i] - kon["Q-learning"][:, i]
            se = d.std(ddof=1) / np.sqrt(len(d))
            print(f"  {nom}: {usul} - Q-learning: {d.mean():+.{f}f}, "
                  f"SE {se:.{f}f}, sezilarli: {abs(d.mean()) > 2 * se}")
    print("  ⭐ harakat keyingi holatni o'zgartirmasa - RL emas, bandit yoki supervised")


if __name__ == "__main__":
    main()

=== 1. Mukofotni aldash (reward hacking): 300 epizod, 10 urug' ===
  xona 5 x 7; S -> G eng qisqa yo'l 10 qadam; C - yo'l o'rtasida
  mukofot       proksi  haqiqiy  G ga yetdi  qadam  C ga kirish  o_qitishda  jami qadam
  siyrak          10.0     10.0       10/10     10          0.0        9.63        6088
  C +1            48.0      0.0        0/10    100         48.0        2.53       26060
  potensial       22.3     10.0       10/10     10          0.0        9.67        6074
  proksi - agent optimallashtirgan mukofot; haqiqiy - faqat G uchun +10
  o'qitishda - o'qitish davomida epizodga o'rtacha HAQIQIY mukofot

=== 2. Xulosa (natijadan) ===
  'C +1' dizayni: proksi mukofot yuqori, lekin maqsadga yetilmaydi - agent C atrofida aylanadi
  potensial - siyrak, yakuniy haqiqiy: +0.0
  potensial - siyrak, o'qitishdagi haqiqiy mukofot: +0.04, SE 0.02, sezilarli: False
  potensial - siyrak, jami o'qitish qadamlari: -13.80, SE 61.06, sezilarli: False

=== 3. RL kerak emas: kontekstli bandit (3000 mijoz, 10 urug') ===
  usul          xarid ulushi  to_g_ri taklif  max Q o_rt.
  Q-learning          0.0779          0.46/1        0.739
  bandit              0.0804          0.47/1        0.090
  supervised          0.0825          0.62/1        0.115
  mumkin bo'lgan maksimum (eng yaxshi taklif): 0.0929, tasodifiy taklif: 0.0735
  xarid: bandit - Q-learning: +0.0025, SE 0.0017, sezilarli: False
  to'g'ri taklif: bandit - Q-learning: +0.01, SE 0.08, sezilarli: False
  xarid: supervised - Q-learning: +0.0046, SE 0.0016, sezilarli: True
  to'g'ri taklif: supervised - Q-learning: +0.16, SE 0.05, sezilarli: True
  ⭐ harakat keyingi holatni o'zgartirmasa - RL emas, bandit yoki supervised

loss = huber(Q(s_t, a_t), r + gamma * Q(s_t1).max())                  # ⚠️ korrelyatsiya
sb, ab, rb, s2b, tb = xotira.namuna(128)                               # ✅ replay

y = r + gamma * onlayn(s2).max(1).values                               # ⚠️ harakatlanuvchi nishon
y = r + gamma * nishon(s2).max(1).values * (1 - tugadi)                # ✅ har C qadamda nusxa

y = r + gamma * nishon(s2).max(1).values                               # ⚠️ grad oqadi
with torch.no_grad():
    y = r + gamma * nishon(s2).max(1).values                           # ✅

loss = -(logp * G_0).sum()                                             # ⚠️ o'tmish mukofoti ham
loss = -(logp * (G_t - V(s).detach())).sum()                           # ✅ reward-to-go + baseline

ustunlik = r + gamma * V(s2) - V(s)                                    # ⚠️ critic noldan
# critic R^2 ni kuzating; GAE lambda ~ 0.95 yoki uzunroq ufq            # ✅

r += 1.0 if s == nazorat_nuqtasi else 0.0                              # ⚠️ aylanib yig'iladi
r += gamma * phi(s2) - phi(s)                                          # ✅ potensial shakl

print(max(baholar))                                                    # ⚠️
print(np.mean(oxirgi_baholar), se)                                     # ✅ 3-5 urug', oxirgi nuqta

1.  # 10 o'lchov, har biri 10 bo'lak - nechta katak?
2.  # 20 bo'lakli jadval nega yomon ishladi?
3.  # experience replay qaysi muammoni hal qiladi?
4.  # target network qaysi muammoni hal qiladi?
5.  # target siz DQN da Q qiymatlari?
6.  # nishon 25 marta yangilansa, Q(s0) taxminan?
7.  # baseline gradient kutilmasini o'zgartiradimi?
8.  # yaxshi critic dispersiyani necha marta kamaytirdi?
9.  # 1-qadamli A2C nega yiqildi?
10. # "C +1" mukofotida agent nima qildi?
11. # potensial shakli optimal siyosatni o'zgartiradimi?
12. # harakat keyingi holatga ta'sir qilmasa - qaysi usul?

1.  y = r + gamma * onlayn(s2).max(1).values

2.  y = r + gamma * nishon(s2).max(1).values * (1 - (tugadi | kesildi))

3.  loss = -(logp * G_0).sum()

4.  mukofot = 10 * (s == G) + 1 * (s == C)

5.  print("DQN ishlaydi:", orgat(urug=0)[-1] > 190)