21.6-dars: Qurilmalar va aniqlik

QURILMA (device):
  torch.device("cpu")
  torch.device("cuda")      NVIDIA GPU  (cuda:0, cuda:1, ...)
  torch.device("mps")       Apple Silicon

TANLASH:
  qurilma = torch.device("cuda" if torch.cuda.is_available() else "cpu")

KO'CHIRISH:
  model.to(qurilma)         JOYIDA (model qaytariladi, lekin o'zi o'zgaradi)
  x = x.to(qurilma)         YANGI tensor qaytaradi - QAYTA TAYINLASH SHART

QOIDA: bitta amaldagi hamma tensor BIR qurilmada bo'lishi shart
  aks holda: RuntimeError: Expected all tensors to be on the same device

x.device       -> tensor qayerda
next(model.parameters()).device -> model qayerda

1. QURILMANI BIR JOYDA TANLANG
   qurilma = torch.device("cuda" if torch.cuda.is_available() else "cpu")

2. MODEL VA BATCH NI KO'CHIRING
   model.to(qurilma)
   for x, y in dl:
       x, y = x.to(qurilma), y.to(qurilma)

3. FORWARD ICHIDA YANGI TENSOR - kirish qurilmasida
   torch.zeros(n, device=x.device)        ✅
   torch.zeros_like(x)                    ✅ (qurilma va dtype meros)
   torch.zeros(n)                         ⚠️ doim CPU da

4. DOIMIY TENSOR - bufer sifatida
   self.register_buffer("maska", ...)     ✅ model bilan ko'chadi
   self.maska = torch.tensor(...)          ⚠️ CPU da qoladi

5. NUMPY GA QAYTISH
   t.detach().cpu().numpy()               ✅
   t.numpy()                              ⚠️ GPU tensorida xato

6. YUKLASH
   torch.load(yol, map_location=qurilma)

TUR         BAYT  EPS (1 dan keyingi qadam)  MAKS         DIAPAZON
float64     8     2.2e-16                   1.8e308      juda keng
float32     4     1.2e-7                    3.4e38       keng
bfloat16    2     7.8e-3                    3.4e38       float32 BILAN BIR XIL
float16     2     9.8e-4                    65504        TOR!

EPS - nisbiy aniqlik: 1.0 + eps/2 == 1.0 (yaxlitlanadi)

bfloat16:  diapazon float32 niki, aniqlik past  -> toshmaydi
float16:   aniqlik bfloat16 dan yaxshi, diapazon tor -> TOSHADI

TOSHISH (overflow):  float16 da 70000 -> inf
YO'QOLISH (underflow): float16 da 1e-8 -> 0  (kichik gradientlar!)

XOTIRA:  float16/bfloat16 - 2 bayt, float32 - 4 bayt
  parametrlar, gradientlar, aktivatsiyalar - hammasi 2x kichik
  -> kattaroq model yoki kattaroq batch sig'adi

TEZLIK (GPU da):
  Tensor Core lar float16/bfloat16 matmul ni float32 dan
  bir necha barobar tez bajaradi
  CPU da odatda TEZLIK yutug'i YO'Q yoki kam

NARXI:
  yig'indilar va loss da aniqlik yo'qoladi
  softmax, log, exp - past aniqlikda xavfli
  -> "hamma narsani float16 ga" - YAXSHI FIKR EMAS

G'OYA: har amalni O'ZIGA MOS aniqlikda bajarish

with torch.autocast(device_type="cuda", dtype=torch.float16):
    chiqish = model(x)          # matmul/conv -> float16
    loss = kriteriy(chiqish, y)  # softmax/loss -> float32 (avtomatik)

autocast QOIDALARI (torch ichida):
  past aniqlikka: Linear, matmul, conv      (tez, xavfsiz)
  float32 da:     softmax, log, loss        (aniqlik kerak)
  DIQQAT: ro'yxat QURILMAGA bog'liq - CUDA da layer_norm float32,
          CPU da esa past aniqlikda ham bo'lishi mumkin

PARAMETRLAR float32 DA QOLADI ("master weights")
  -> yangilash aniq, kichik qadamlar yo'qolmaydi

GRADIENT SCALER (faqat float16 uchun):
  loss ni katta songa ko'paytiradi -> kichik gradientlar 0 bo'lmaydi
  step dan oldin qaytarib bo'ladi, inf/NaN bo'lsa qadamni o'tkazadi

  scaler = torch.amp.GradScaler("cuda")
  scaler.scale(loss).backward()
  scaler.step(opt)
  scaler.update()

bfloat16 da scaler KERAK EMAS (diapazon keng)

CPU:
  float32 - sukut, deyarli har doim to'g'ri
  bfloat16 autocast - xotira kerak bo'lsa, tezlik kutilmasin

ZAMONAVIY GPU (Ampere+ : A100, RTX 30xx/40xx, H100):
  bfloat16 autocast, scaler siz - eng oddiy va barqaror

ESKIROQ GPU (V100, T4, RTX 20xx):
  float16 autocast + GradScaler

INFERENCE:
  float16/bfloat16 ga to'liq o'tkazish ko'pincha xavfsiz
  lekin natijani float32 bilan TAQQOSLANG

HECH QACHON:
  float16 da loss ni hisoblash (autocast buni o'zi hal qiladi)
  parametrlarni float16 ga o'tkazib, shunda o'rgatish

import torch

qurilma = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = Model().to(qurilma)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)

kichik_aniqlik = torch.bfloat16
for x, y in dl:
    x, y = x.to(qurilma), y.to(qurilma)
    opt.zero_grad()
    with torch.autocast(device_type=qurilma.type, dtype=kichik_aniqlik):
        loss = kriteriy(model(x), y)
    loss.backward()
    opt.step()

# forward ichida
tozalik = torch.zeros(x.shape[0], device=x.device, dtype=x.dtype)

# numpy ga
p = chiqish.detach().float().cpu().numpy()

# yuklash
holat = torch.load(yol, map_location=qurilma, weights_only=True)

qurilma bir joyda tanlanadi
model.to() joyida, x = x.to() qayta tayinlash
forward ichida device=x.device
bfloat16: diapazon keng, aniqlik past
float16: diapazon tor (65504), GradScaler kerak
autocast: xavfli amallar float32 da qoladi

"""device, .to() va 'bir qurilma' qoidasi (real torch)."""

import torch
import torch.nn as nn


def main() -> None:
    torch.manual_seed(0)

    print("=== 1. Mavjud qurilmalar ===")
    print(f"  cuda mavjudmi: {torch.cuda.is_available()}")
    mps = (hasattr(torch.backends, "mps")
           and torch.backends.mps.is_available())
    print(f"  mps mavjudmi:  {mps}")
    qurilma = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(f"  tanlangan qurilma: {qurilma}")
    print(f"  qurilma turi: {qurilma.type}")

    print("\n=== 2. model.to() va x.to() farqi ===")
    model = nn.Linear(4, 2)
    natija = model.to(qurilma)
    print(f"  model.to() o'sha obyektni qaytaradimi: {natija is model}")
    x = torch.randn(3, 4)
    y = x.to(torch.float64)
    print(f"  x.to(float64) yangi tensormi: {y is not x}")
    print(f"  x ning turi o'zgarmadi: {x.dtype}")
    print(f"  y ning turi: {y.dtype}")
    x.to(torch.float64)
    print(f"  tayinlashsiz x.to() dan keyin x: {x.dtype}")
    print("  x = x.to(...) - QAYTA TAYINLASH shart")

    print("\n=== 3. Qayerda joylashgan ===")
    print(f"  x.device: {x.device}")
    print(f"  model qurilmasi: {next(model.parameters()).device}")
    print(f"  model dtype:     {next(model.parameters()).dtype}")

    print("\n=== 4. Tur mos kelmasa (qurilma kabi xato) ===")
    try:
        model(x.to(torch.float64))
        print("  xato chiqmadi (kutilmagan)")
    except RuntimeError as xato:
        print(f"  RuntimeError: {str(xato)[:60]}")
    print("  qurilma mos kelmasa ham xuddi shunday xato chiqadi")

    print("\n=== 5. model.to() dtype ni ham o'zgartiradi ===")
    m64 = nn.Linear(4, 2).to(torch.float64)
    print(f"  parametr turi: {m64.weight.dtype}")
    print(f"  float64 kirish bilan: "
          f"{m64(torch.randn(2, 4, dtype=torch.float64)).dtype}")

    print("\n=== 6. Tensor yaratishda qurilma va tur ===")
    namuna = torch.randn(2, 3, dtype=torch.float64)
    variantlar = {
        "torch.zeros(3)": torch.zeros(3),
        "torch.zeros(3, device=, dtype=)": torch.zeros(
            3, device=namuna.device, dtype=namuna.dtype),
        "torch.zeros_like(namuna)": torch.zeros_like(namuna),
        "namuna.new_zeros(3)": namuna.new_zeros(3),
    }
    print(f"  {'ifoda':<34} {'qurilma':>8} {'dtype':>15}")
    for nom, t in variantlar.items():
        print(f"  {nom:<34} {str(t.device):>8} {str(t.dtype):>15}")
    print("  ⭐ _like va new_ - qurilma va turni meros oladi")


if __name__ == "__main__":
    main()

=== 1. Mavjud qurilmalar ===
  cuda mavjudmi: False
  mps mavjudmi:  False
  tanlangan qurilma: cpu
  qurilma turi: cpu

=== 2. model.to() va x.to() farqi ===
  model.to() o'sha obyektni qaytaradimi: True
  x.to(float64) yangi tensormi: True
  x ning turi o'zgarmadi: torch.float32
  y ning turi: torch.float64
  tayinlashsiz x.to() dan keyin x: torch.float32
  x = x.to(...) - QAYTA TAYINLASH shart

=== 3. Qayerda joylashgan ===
  x.device: cpu
  model qurilmasi: cpu
  model dtype:     torch.float32

=== 4. Tur mos kelmasa (qurilma kabi xato) ===
  RuntimeError: mat1 and mat2 must have the same dtype, but got Double and F
  qurilma mos kelmasa ham xuddi shunday xato chiqadi

=== 5. model.to() dtype ni ham o'zgartiradi ===
  parametr turi: torch.float64
  float64 kirish bilan: torch.float64

=== 6. Tensor yaratishda qurilma va tur ===
  ifoda                               qurilma           dtype
  torch.zeros(3)                          cpu   torch.float32
  torch.zeros(3, device=, dtype=)         cpu   torch.float64
  torch.zeros_like(namuna)                cpu   torch.float64
  namuna.new_zeros(3)                     cpu   torch.float64
  ⭐ _like va new_ - qurilma va turni meros oladi

"""forward ichidagi yangi tensorlar va buferlar (real torch)."""

import shutil
import tempfile
from pathlib import Path

import torch
import torch.nn as nn


class Yomon(nn.Module):
    def __init__(self, d):
        super().__init__()
        self.fc = nn.Linear(d, d)
        self.masshtab = torch.full((d,), 0.5)          # ODDIY atribut

    def forward(self, x):
        shovqin = torch.zeros(x.shape[0], x.shape[1])  # device yo'q
        self.ichki = (shovqin.dtype, self.masshtab.dtype)
        return self.fc(x + shovqin) * self.masshtab


class Yaxshi(nn.Module):
    def __init__(self, d):
        super().__init__()
        self.fc = nn.Linear(d, d)
        self.register_buffer("masshtab", torch.full((d,), 0.5))

    def forward(self, x):
        shovqin = torch.zeros_like(x)                  # x dan meros
        self.ichki = (shovqin.dtype, self.masshtab.dtype)
        return self.fc(x + shovqin) * self.masshtab


def main() -> None:
    torch.manual_seed(0)

    print("=== 1. Ikki model .to(float64) dan keyin ===")
    print("  GPU yo'q, shuning uchun .to() ni dtype bilan sinaymiz")
    print(f"  {'model':<8} {'chiqish':>15} {'shovqin':>15} "
          f"{'masshtab':>15}")
    for sinf in [Yomon, Yaxshi]:
        m = sinf(4).to(torch.float64)
        chiqish = m(torch.randn(3, 4, dtype=torch.float64))
        print(f"  {sinf.__name__:<8} {str(chiqish.dtype):>15} "
              f"{str(m.ichki[0]):>15} {str(m.ichki[1]):>15}")
    print("  ikkalasi ham ISHLADI: torch turlarni avtomatik ko'taradi")
    print("  (float32 * float64 -> float64), lekin Yomon ichida")
    print("  ikki tensor float32 da QOLIB KETDI")

    print("\n=== 1b. Qurilmada avtomatik ko'tarish YO'Q ===")
    try:
        torch.ones(2) + torch.ones(2, device="meta")
        print("  xato chiqmadi (kutilmagan)")
    except RuntimeError as xato:
        print(f"  cpu + meta: RuntimeError - {str(xato)[:46]}")
    print("  'meta' - ma'lumotsiz sinov qurilmasi; cuda bilan ham")
    print("  AYNAN shu xato chiqadi - Yomon model GPU da ISHLAMAYDI")

    print("\n=== 2. Nima ko'chdi, nima ko'chmadi ===")
    y = Yomon(4).to(torch.float64)
    z = Yaxshi(4).to(torch.float64)
    print(f"  {'model':<8} {'fc.weight':>15} {'masshtab':>15}")
    print(f"  {'Yomon':<8} {str(y.fc.weight.dtype):>15} "
          f"{str(y.masshtab.dtype):>15}")
    print(f"  {'Yaxshi':<8} {str(z.fc.weight.dtype):>15} "
          f"{str(z.masshtab.dtype):>15}")
    print("  oddiy atribut .to() bilan KO'CHMAYDI")

    print("\n=== 3. state_dict da ham farq ===")
    print(f"  Yomon:  {list(Yomon(4).state_dict())}")
    print(f"  Yaxshi: {list(Yaxshi(4).state_dict())}")

    print("\n=== 4. Qurilmadan mustaqil sikl ===")
    qurilma = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = Yaxshi(4).to(qurilma)
    opt = torch.optim.AdamW(model.parameters(), lr=0.01)
    X = torch.randn(64, 4)
    Y = X * 2.0
    for _ in range(100):
        x, t = X.to(qurilma), Y.to(qurilma)
        opt.zero_grad()
        loss = ((model(x) - t) ** 2).mean()
        loss.backward()
        opt.step()
    print(f"  qurilma: {qurilma}, yakuniy loss: {loss.item():.6f}")
    p = model(X.to(qurilma)).detach().cpu().numpy()
    print(f"  numpy ga: shakl {p.shape}, tur {p.dtype}")
    print("  .detach().cpu().numpy() - har qanday qurilmada ishlaydi")

    print("\n=== 5. map_location bilan yuklash ===")
    papka = Path(tempfile.mkdtemp(prefix="torch_qurilma_"))
    try:
        yol = papka / "m.pt"
        torch.save(model.state_dict(), yol)
        holat = torch.load(yol, map_location="cpu", weights_only=True)
        print(f"  {'kalit':<12} {'qurilma':>8}")
        for k, v in holat.items():
            print(f"  {k:<12} {str(v.device):>8}")
        print("  GPU da saqlangan fayl GPU siz mashinada ham ochiladi")
        print("  ⭐ map_location - har doim bering")
    finally:
        shutil.rmtree(papka, ignore_errors=True)


if __name__ == "__main__":
    main()

=== 1. Ikki model .to(float64) dan keyin ===
  GPU yo'q, shuning uchun .to() ni dtype bilan sinaymiz
  model            chiqish         shovqin        masshtab
  Yomon      torch.float64   torch.float32   torch.float32
  Yaxshi     torch.float64   torch.float64   torch.float64
  ikkalasi ham ISHLADI: torch turlarni avtomatik ko'taradi
  (float32 * float64 -> float64), lekin Yomon ichida
  ikki tensor float32 da QOLIB KETDI

=== 1b. Qurilmada avtomatik ko'tarish YO'Q ===
  cpu + meta: RuntimeError - Tensor on device meta is not on the expected d
  'meta' - ma'lumotsiz sinov qurilmasi; cuda bilan ham
  AYNAN shu xato chiqadi - Yomon model GPU da ISHLAMAYDI

=== 2. Nima ko'chdi, nima ko'chmadi ===
  model          fc.weight        masshtab
  Yomon      torch.float64   torch.float32
  Yaxshi     torch.float64   torch.float64
  oddiy atribut .to() bilan KO'CHMAYDI

=== 3. state_dict da ham farq ===
  Yomon:  ['fc.weight', 'fc.bias']
  Yaxshi: ['masshtab', 'fc.weight', 'fc.bias']

=== 4. Qurilmadan mustaqil sikl ===
  qurilma: cpu, yakuniy loss: 2.578763
  numpy ga: shakl (64, 4), tur float32
  .detach().cpu().numpy() - har qanday qurilmada ishlaydi

=== 5. map_location bilan yuklash ===
  kalit         qurilma
  masshtab          cpu
  fc.weight         cpu
  fc.bias           cpu
  GPU da saqlangan fayl GPU siz mashinada ham ochiladi
  ⭐ map_location - har doim bering

"""float64 / float32 / bfloat16 / float16 ni son bilan (real torch)."""

import torch


def main() -> None:
    turlar = [torch.float64, torch.float32, torch.bfloat16, torch.float16]

    print("=== 1. Asosiy xossalar ===")
    print(f"  {'tur':<16} {'bayt':>5} {'eps':>12} {'maks':>12} "
          f"{'eng kichik':>12}")
    for t in turlar:
        f = torch.finfo(t)
        print(f"  {str(t):<16} {f.bits // 8:>5} {f.eps:>12.2e} "
              f"{f.max:>12.3e} {f.tiny:>12.2e}")

    print("\n=== 2. 1 + kichik son ===")
    print(f"  {'tur':<16}", end="")
    kichiklar = [1e-2, 1e-3, 1e-4, 1e-8]
    for k in kichiklar:
        print(f" {f'1+{k:g}':>10}", end="")
    print()
    for t in turlar:
        print(f"  {str(t):<16}", end="")
        for k in kichiklar:
            q = (torch.tensor(1.0, dtype=t) + torch.tensor(k, dtype=t))
            print(f" {q.item():>10.6f}", end="")
        print()
    print("  bfloat16 da 1 + 0.001 = 1 (aniqlik YO'QOLDI)")

    print("\n=== 3. Toshish (overflow) ===")
    print(f"  {'son':>10}", end="")
    for t in turlar:
        print(f" {str(t).split('.')[1]:>10}", end="")
    print()
    for son in [1000.0, 65000.0, 70000.0, 1e10]:
        print(f"  {son:>10.0e}", end="")
        for t in turlar:
            print(f" {torch.tensor(son).to(t).item():>10.3g}", end="")
        print()
    print("  float16 da 65504 dan kattasi - inf")

    print("\n=== 4. Yo'qolish (underflow) - kichik gradientlar ===")
    gradient = torch.tensor([1e-3, 1e-5, 1e-7, 1e-8])
    print(f"  asl (float32): {gradient.tolist()}")
    for t in [torch.bfloat16, torch.float16]:
        q = gradient.to(t).float()
        nol = int((q == 0).sum())
        print(f"  {str(t):<16}: {[f'{v:.2e}' for v in q.tolist()]}  "
              f"nol: {nol}")
    masshtablangan = (gradient * 1024).to(torch.float16).float() / 1024
    print(f"  float16, 1024 ga ko'paytirilgan: "
          f"{[f'{v:.2e}' for v in masshtablangan.tolist()]}")
    print("  GradScaler aynan shunday qiladi")

    print("\n=== 5. Matritsa ko'paytmasi aniqligi ===")
    torch.manual_seed(0)
    a = torch.randn(128, 128, dtype=torch.float64)
    b = torch.randn(128, 128, dtype=torch.float64)
    etalon = a @ b
    print(f"  {'tur':<16} {'maks xato':>12} {'nisbiy xato':>13}")
    for t in turlar:
        c = (a.to(t) @ b.to(t)).double()
        xato = (c - etalon).abs().max().item()
        nisbiy = xato / etalon.abs().max().item()
        print(f"  {str(t):<16} {xato:>12.3e} {nisbiy:>13.3e}")

    print("\n=== 6. Uzun yig'indida aniqlik ===")
    son = torch.full((100000,), 0.1)
    print(f"  {'tur':<16} {'torch.sum':>11} {'qo_lda 5000':>13} "
          f"{'haqiqiy':>9}")
    for t in turlar:
        s = son.to(t).sum().item()
        jami = torch.tensor(0.0, dtype=t)
        qadam = torch.tensor(0.1, dtype=t)
        for _ in range(5000):
            jami = jami + qadam
        print(f"  {str(t):<16} {s:>11.1f} {jami.item():>13.2f} "
              f"{'10000 / 500':>9}")
    print("  torch.sum ichkarida aniqroq to'playdi, lekin QO'LDA")
    print("  past aniqlikda yig'ish to'xtab qoladi: jami katta")
    print("  bo'lganda 0.1 qo'shilishi yaxlitlashda yo'qoladi")
    print("  ⭐ Yig'indi va loss ni past aniqlikda to'plamang")


if __name__ == "__main__":
    main()

=== 1. Asosiy xossalar ===
  tur               bayt          eps         maks   eng kichik
  torch.float64        8     2.22e-16   1.798e+308    2.23e-308
  torch.float32        4     1.19e-07    3.403e+38     1.18e-38
  torch.bfloat16       2     7.81e-03    3.390e+38     1.18e-38
  torch.float16        2     9.77e-04    6.550e+04     6.10e-05

=== 2. 1 + kichik son ===
  tur                  1+0.01    1+0.001   1+0.0001    1+1e-08
  torch.float64      1.010000   1.001000   1.000100   1.000000
  torch.float32      1.010000   1.001000   1.000100   1.000000
  torch.bfloat16     1.007812   1.000000   1.000000   1.000000
  torch.float16      1.009766   1.000977   1.000000   1.000000
  bfloat16 da 1 + 0.001 = 1 (aniqlik YO'QOLDI)

=== 3. Toshish (overflow) ===
         son    float64    float32   bfloat16    float16
       1e+03      1e+03      1e+03      1e+03      1e+03
       6e+04    6.5e+04    6.5e+04    6.5e+04    6.5e+04
       7e+04      7e+04      7e+04   7.01e+04        inf
       1e+10      1e+10      1e+10      1e+10        inf
  float16 da 65504 dan kattasi - inf

=== 4. Yo'qolish (underflow) - kichik gradientlar ===
  asl (float32): [0.0010000000474974513, 9.999999747378752e-06, 1.0000000116860974e-07, 9.99999993922529e-09]
  torch.bfloat16  : ['9.99e-04', '1.00e-05', '1.00e-07', '1.00e-08']  nol: 0
  torch.float16   : ['1.00e-03', '1.00e-05', '1.19e-07', '0.00e+00']  nol: 1
  float16, 1024 ga ko'paytirilgan: ['1.00e-03', '1.00e-05', '1.00e-07', '1.00e-08']
  GradScaler aynan shunday qiladi

=== 5. Matritsa ko'paytmasi aniqligi ===
  tur                 maks xato   nisbiy xato
  torch.float64       0.000e+00     0.000e+00
  torch.float32       1.643e-05     3.745e-07
  torch.bfloat16      1.524e-01     3.474e-03
  torch.float16       2.267e-02     5.169e-04

=== 6. Uzun yig'indida aniqlik ===
  tur                torch.sum   qo_lda 5000   haqiqiy
  torch.float64        10000.0        500.00 10000 / 500
  torch.float32        10000.0        500.02 10000 / 500
  torch.bfloat16        9984.0         32.00 10000 / 500
  torch.float16        10000.0        256.00 10000 / 500
  torch.sum ichkarida aniqroq to'playdi, lekin QO'LDA
  past aniqlikda yig'ish to'xtab qoladi: jami katta
  bo'lganda 0.1 qo'shilishi yaxlitlashda yo'qoladi
  ⭐ Yig'indi va loss ni past aniqlikda to'plamang

"""autocast qaysi amalni qaysi aniqlikda bajaradi (real torch)."""

import torch
import torch.nn as nn
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler


def main() -> None:
    torch.manual_seed(0)
    qurilma = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    past = torch.bfloat16

    print("=== 1. autocast ichida turlar ===")
    model = nn.Sequential(nn.Linear(16, 32), nn.LayerNorm(32),
                          nn.ReLU(), nn.Linear(32, 4)).to(qurilma)
    x = torch.randn(8, 16, device=qurilma)
    y = torch.randint(0, 4, (8,), device=qurilma)
    turlar = {}
    tutqichlar = [m.register_forward_hook(
        lambda mod, k, c, nom=type(m).__name__ + str(i):
        turlar.__setitem__(nom, c.dtype))
        for i, m in enumerate(model)]
    with torch.autocast(device_type=qurilma.type, dtype=past):
        chiqish = model(x)
        loss = nn.functional.cross_entropy(chiqish, y)
    for t in tutqichlar:
        t.remove()
    print(f"  {'qatlam':<14} {'chiqish turi':>16}")
    for nom, t in turlar.items():
        print(f"  {nom:<14} {str(t):>16}")
    print(f"  {'loss':<14} {str(loss.dtype):>16}")
    print(f"  parametrlar turi: {model[0].weight.dtype}")
    print("  CPU autocast da LayerNorm ham past aniqlikda chiqdi;")
    print("  CUDA ro'yxatida layer_norm float32 da. Loss IKKALASIDA")
    print("  float32 - bu eng muhimi")

    print("\n=== 2. Gradientlar float32 da ===")
    loss.backward()
    print(f"  model[0].weight.grad turi: {model[0].weight.grad.dtype}")
    print("  master weights va ularning gradienti - float32")

    print("\n=== 3. To'liq o'rgatish: float32 va autocast ===")
    X, t = make_classification(n_samples=3000, n_features=16,
                               n_informative=8, n_classes=4,
                               flip_y=0.1, class_sep=1.0, random_state=0)
    Xtr, Xva, ttr, tva = train_test_split(X, t, test_size=0.3,
                                          random_state=0, stratify=t)
    sc = StandardScaler().fit(Xtr)
    Xtr = torch.tensor(sc.transform(Xtr), dtype=torch.float32)
    Xva = torch.tensor(sc.transform(Xva), dtype=torch.float32)
    ttr, tva = torch.tensor(ttr), torch.tensor(tva)

    def orgat(autocast):
        torch.manual_seed(0)
        m = nn.Sequential(nn.Linear(16, 64), nn.ReLU(),
                          nn.Linear(64, 64), nn.ReLU(),
                          nn.Linear(64, 4)).to(qurilma)
        o = torch.optim.AdamW(m.parameters(), lr=0.005)
        g = torch.Generator().manual_seed(0)
        for _ in range(20):
            m.train()
            tartib = torch.randperm(len(ttr), generator=g)
            for i in range(0, len(ttr), 128):
                idx = tartib[i:i + 128]
                xb, yb = Xtr[idx].to(qurilma), ttr[idx].to(qurilma)
                o.zero_grad()
                with torch.autocast(device_type=qurilma.type, dtype=past,
                                    enabled=autocast):
                    l = nn.functional.cross_entropy(m(xb), yb)
                l.backward()
                o.step()
        m.eval()
        with torch.no_grad():
            ch = m(Xva.to(qurilma)).float()
            va_loss = nn.functional.cross_entropy(ch, tva.to(qurilma))
            aniq = (ch.argmax(1) == tva.to(qurilma)).float().mean()
        return va_loss.item(), aniq.item(), m

    print(f"  {'rejim':<18} {'val loss':>10} {'val aniqlik':>13}")
    natijalar = {}
    for nom, a in [("float32", False), ("autocast bf16", True)]:
        vl, va, m = orgat(a)
        natijalar[nom] = m
        print(f"  {nom:<18} {vl:>10.4f} {va:>13.4f}")
    print("  natija deyarli bir xil - aniqlik yo'qotilmadi")

    print("\n=== 4. Model xotirasi ===")
    m = natijalar["float32"]
    print(f"  {'tur':<16} {'bayt':>8}")
    for t in [torch.float32, torch.bfloat16, torch.float16]:
        bayt = sum(p.numel() * torch.finfo(t).bits // 8
                   for p in m.parameters())
        print(f"  {str(t):<16} {bayt:>8}")

    print("\n=== 5. Inference ni to'liq bf16 ga o'tkazish ===")
    m32 = natijalar["float32"]
    m16 = nn.Sequential(nn.Linear(16, 64), nn.ReLU(), nn.Linear(64, 64),
                        nn.ReLU(), nn.Linear(64, 4)).to(qurilma)
    m16.load_state_dict(m32.state_dict())
    m16 = m16.to(torch.bfloat16).eval()
    with torch.no_grad():
        p32 = m32(Xva.to(qurilma)).argmax(1)
        p16 = m16(Xva.to(qurilma).to(torch.bfloat16)).argmax(1)
    print(f"  bashoratlar mos kelishi: "
          f"{(p32 == p16).float().mean().item():.4f}")
    print("  ⭐ Past aniqlikdan keyin natijani float32 bilan solishtiring")

    print("\n=== 6. GradScaler (float16 uchun) ===")
    scaler = torch.amp.GradScaler(qurilma.type, init_scale=1024.0)
    print(f"  boshlang'ich masshtab: {scaler.get_scale()}")
    print("  float16 da:  scaler.scale(loss).backward()")
    print("               scaler.step(opt); scaler.update()")
    print("  bfloat16 da: scaler KERAK EMAS")


if __name__ == "__main__":
    main()

=== 1. autocast ichida turlar ===
  qatlam             chiqish turi
  Linear0          torch.bfloat16
  LayerNorm1       torch.bfloat16
  ReLU2            torch.bfloat16
  Linear3          torch.bfloat16
  loss              torch.float32
  parametrlar turi: torch.float32
  CPU autocast da LayerNorm ham past aniqlikda chiqdi;
  CUDA ro'yxatida layer_norm float32 da. Loss IKKALASIDA
  float32 - bu eng muhimi

=== 2. Gradientlar float32 da ===
  model[0].weight.grad turi: torch.float32
  master weights va ularning gradienti - float32

=== 3. To'liq o'rgatish: float32 va autocast ===
  rejim                val loss   val aniqlik
  float32                0.8186        0.7378
  autocast bf16          0.8169        0.7467
  natija deyarli bir xil - aniqlik yo'qotilmadi

=== 4. Model xotirasi ===
  tur                  bayt
  torch.float32       22032
  torch.bfloat16      11016
  torch.float16       11016

=== 5. Inference ni to'liq bf16 ga o'tkazish ===
  bashoratlar mos kelishi: 1.0000
  ⭐ Past aniqlikdan keyin natijani float32 bilan solishtiring

=== 6. GradScaler (float16 uchun) ===
  boshlang'ich masshtab: 1024.0
  float16 da:  scaler.scale(loss).backward()
               scaler.step(opt); scaler.update()
  bfloat16 da: scaler KERAK EMAS

x.to(qurilma)                                    # ⚠️ x o'zgarmaydi
x = x.to(qurilma)                                # ✅

maska = torch.zeros(n)                           # ⚠️
maska = torch.zeros(n, device=x.device)          # ✅

self.k = torch.tensor([...])                     # ⚠️ ko'chmaydi
self.register_buffer("k", torch.tensor([...]))   # ✅

chiqish.numpy()                                  # ⚠️
chiqish.detach().cpu().numpy()                   # ✅

torch.load("gpu_model.pt")                       # ⚠️ GPU siz mashinada
torch.load("gpu_model.pt", map_location="cpu")   # ✅

model.half(); orgat(...)                     # ⚠️ beqaror
with torch.autocast(..., dtype=torch.float16):   # ✅ + GradScaler

jami = jami + loss.to(torch.float16)             # ⚠️ aniqlik yo'qoladi
jami += loss.item()                              # ✅

1.  # model.to() nima qaytaradi?
2.  # x.to() nima qaytaradi?
3.  # torch.zeros(n) qayerda yaratiladi?
4.  # oddiy atribut .to() bilan ko'chadimi?
5.  # float16 maksimumi?
6.  # bfloat16 ning afzalligi?
7.  # underflow nima?
8.  # GradScaler nima qiladi?
9.  # autocast da softmax qaysi turda?
10. # parametrlar qaysi turda qoladi?
11. # CPU da past aniqlik tez-mi?
12. # GPU fayl CPU da qanday ochiladi?

1.  x.to(qurilma)

2.  maska = torch.zeros(n)            # forward ichida

3.  self.k = torch.tensor([1.0, 2.0])

4.  chiqish.numpy()                   # GPU tensor

5.  model.half(); orgat(...)