21.3-dars: Dataset va DataLoader

MAP-STYLE DATASET (eng keng tarqalgan):
  class MeningDataset(Dataset):
      def __len__(self):            # nechta namuna
          return self.n
      def __getitem__(self, i):     # i-chi namuna
          return x_i, y_i

  SHARTNOMA: dataset[i] har chaqiruvda BIR XIL narsa qaytarsin
             (augmentatsiyadan tashqari)

ITERABLE DATASET:
  __iter__ ni yozasiz, indeks yo'q
  oqimli ma'lumot uchun (log, tarmoqdan kelayotgan)

TAYYOR SINFLAR:
  TensorDataset(X, y)        tensorlar xotirada bo'lsa
  Subset(ds, indekslar)      qism olish
  ConcatDataset([ds1, ds2])  birlashtirish
  random_split(ds, [0.8, 0.2], generator=g)

MUHIM: __getitem__ BITTA namunani qaytaradi
       batch yig'ish - DataLoader ishi

DataLoader(dataset,
           batch_size=64,
           shuffle=True,          # har davrda aralashtirish
           drop_last=False,       # oxirgi kichik batchni tashlash
           num_workers=0,         # parallel o'qish jarayonlari
           collate_fn=None,       # namunalarni batchga yig'ish
           sampler=None,          # indeks tartibi
           generator=g,           # takrorlanuvchanlik
           pin_memory=False)      # GPU ga tez ko'chirish

NIMA QILADI:
  1. sampler dan indekslar oladi
  2. har indeks uchun dataset[i] chaqiradi
  3. collate_fn bilan batchga yig'adi
  4. (num_workers > 0 bo'lsa) buni parallel qiladi

O'QUV:       shuffle=True,  drop_last=True (BatchNorm bo'lsa)
VALIDATSIYA: shuffle=False, drop_last=False

SUKUT collate:
  [(x0, y0), (x1, y1), ...] -> (stack(x), stack(y))
  SHART: hamma x BIR XIL shaklda

TURLI UZUNLIK (matn, ketma-ketlik):
  stack ishlamaydi -> o'z collate_fn

  def toldir(batch):
      xlar, ylar = zip(*batch)
      uzunliklar = [len(x) for x in xlar]
      L = max(uzunliklar)
      X = torch.zeros(len(xlar), L)
      for i, x in enumerate(xlar):
          X[i, :len(x)] = x
      maska = torch.arange(L) < torch.tensor(uzunliklar)[:, None]
      return X, torch.tensor(ylar), maska

LUG'AT QAYTARISH ham mumkin:
  return {"x": X, "y": y, "maska": maska}

SAMPLER - qaysi indekslar qaysi tartibda

RandomSampler         shuffle=True bilan bir xil
SequentialSampler     shuffle=False bilan bir xil
SubsetRandomSampler   faqat berilgan indekslardan
WeightedRandomSampler og'irlik bo'yicha, qaytarib (replacement)

NOMUTANOSIB SINFLAR:
  og'irlik_i = 1 / sinf_chastotasi[y_i]
  sampler = WeightedRandomSampler(og'irliklar, num_samples=N,
                                  replacement=True, generator=g)
  -> har batchda sinflar TAXMINAN teng

DIQQAT:
  sampler va shuffle=True BIRGA ISHLATILMAYDI (xato)
  WeightedRandomSampler kam sinfni TAKRORLAYDI -> yodlash xavfi
  loss og'irligi 20.5-bob bilan BIRGA qo'llamang - ikki marta

BO'LISH:
  g = torch.Generator().manual_seed(42)
  oquv, val = random_split(ds, [0.8, 0.2], generator=g)

  DIQQAT: guruh/vaqt tuzilmasi bo'lsa random_split YAROQSIZ
          -> indekslarni sklearn bilan hisoblab, Subset ishlating
          (GroupKFold, TimeSeriesSplit - 18-qism)

TAKRORLANUVCHANLIK:
  DataLoader(..., shuffle=True, generator=g)
  g ni har yurishda QAYTA seed qiling

  num_workers > 0 bo'lsa har ishchi o'z seed ini oladi:
    worker_init_fn=lambda i: np.random.seed(asos + i)

AUGMENTATSIYA TASODIFIYLIGI:
  __getitem__ ichidagi tasodif ham seed ga bog'lanishi kerak

XOTIRAGA SIG'SA:
  TensorDataset - eng tez, eng oddiy

SIG'MASA (lazy loading):
  __init__ da faqat YO'LLAR/indekslarni saqlang
  __getitem__ da faylni O'QING
  -> xotira ~ bitta batch hajmi

num_workers:
  0  - asosiy jarayonda (debug uchun eng yaxshi)
  >0 - parallel jarayonlar (I/O sekin bo'lsa foydali)
  Windows/macOS da spawn - Dataset pickle qilinishi SHART
  -> lambda, ochiq fayl tutqichlari muammo beradi

pin_memory=True: CPU->GPU ko'chirishni tezlashtiradi
persistent_workers=True: ishchilarni davrlar orasida saqlaydi

import torch
from torch.utils.data import (DataLoader, Dataset, Subset, TensorDataset,
                              WeightedRandomSampler, random_split)


class JadvalDataset(Dataset):
    def __init__(self, X, y):
        self.X = torch.as_tensor(X, dtype=torch.float32)
        self.y = torch.as_tensor(y, dtype=torch.int64)

    def __len__(self):
        return len(self.y)

    def __getitem__(self, i):
        return self.X[i], self.y[i]


g = torch.Generator().manual_seed(42)
oquv_dl = DataLoader(JadvalDataset(Xtr, ytr), batch_size=64,
                     shuffle=True, drop_last=True, generator=g)
val_dl = DataLoader(JadvalDataset(Xva, yva), batch_size=256,
                    shuffle=False)

# nomutanosib sinflar
chastota = torch.bincount(ytr_t)
w = (1.0 / chastota)[ytr_t]
sampler = WeightedRandomSampler(w, num_samples=len(w),
                                replacement=True, generator=g)
DataLoader(ds, batch_size=64, sampler=sampler)

Dataset:     __len__ + __getitem__ (bitta namuna)
DataLoader:  batch, tartib, parallellik
collate_fn:  turli uzunlik -> padding + maska
sampler:     tartib strategiyasi
generator:   takrorlanuvchanlik

"""Dataset shartnomasi va TensorDataset (real torch)."""

import numpy as np
import torch
from torch.utils.data import ConcatDataset, Dataset, Subset, TensorDataset


class JadvalDataset(Dataset):
    """Son belgilar + kategoriya + maqsad."""

    def __init__(self, X_son, X_kat, y):
        self.X_son = torch.as_tensor(X_son, dtype=torch.float32)
        self.X_kat = torch.as_tensor(X_kat, dtype=torch.int64)
        self.y = torch.as_tensor(y, dtype=torch.int64)
        assert len(self.X_son) == len(self.X_kat) == len(self.y)

    def __len__(self):
        return len(self.y)

    def __getitem__(self, i):
        return {"son": self.X_son[i], "kat": self.X_kat[i],
                "y": self.y[i]}


def main() -> None:
    rng = np.random.default_rng(0)
    n = 100
    X_son = rng.normal(0, 1, (n, 4))
    X_kat = rng.integers(0, 5, (n, 2))
    y = rng.integers(0, 3, n)

    print("=== 1. Shartnoma ===")
    ds = JadvalDataset(X_son, X_kat, y)
    print(f"  len(ds): {len(ds)}")
    namuna = ds[7]
    print(f"  ds[7] turi: {type(namuna).__name__}")
    print(f"  {'kalit':<6} {'shakl':>8} {'dtype':>14}")
    for k, v in namuna.items():
        print(f"  {k:<6} {str(tuple(v.shape)):>8} {str(v.dtype):>14}")

    print("\n=== 2. Bir xil indeks - bir xil natija ===")
    a, b = ds[7], ds[7]
    print(f"  ds[7]['son'] ikki marta bir xilmi: "
          f"{torch.equal(a['son'], b['son'])}")
    print("  Dataset DETERMINISTIK bo'lishi kerak (augmentatsiyasiz)")

    print("\n=== 3. TensorDataset ===")
    td = TensorDataset(torch.tensor(X_son, dtype=torch.float32),
                       torch.tensor(y))
    x0, y0 = td[0]
    print(f"  len: {len(td)}, td[0] = (shakl {tuple(x0.shape)}, "
          f"y={int(y0)})")
    print(f"  tuple qaytaradi: {type(td[0]).__name__}")

    print("\n=== 4. Subset va ConcatDataset ===")
    birinchi = Subset(ds, range(0, 30))
    ikkinchi = Subset(ds, range(70, 100))
    birga = ConcatDataset([birinchi, ikkinchi])
    print(f"  Subset(0..30): {len(birinchi)}")
    print(f"  Subset(70..100): {len(ikkinchi)}")
    print(f"  Concat: {len(birga)}")
    print(f"  birga[30] == ds[70]: "
          f"{torch.equal(birga[30]['son'], ds[70]['son'])}")

    print("\n=== 5. Salbiy va chegara indekslari ===")
    print(f"  ds[-1]['y'] == ds[99]['y']: "
          f"{int(ds[-1]['y']) == int(ds[99]['y'])}")
    try:
        ds[100]
        print("  ds[100] xato bermadi (kutilmagan)")
    except IndexError as xato:
        print(f"  ds[100]: IndexError - {str(xato)[:40]}")

    print("\n=== 6. Uzunlik mos kelmasa ===")
    try:
        JadvalDataset(X_son, X_kat[:50], y)
        print("  xato chiqmadi (kutilmagan)")
    except AssertionError:
        print("  AssertionError: uzunliklar mos emas")
    print("  ⭐ __init__ da tekshiruv - keyingi xatolarni oldini oladi")


if __name__ == "__main__":
    main()

=== 1. Shartnoma ===
  len(ds): 100
  ds[7] turi: dict
  kalit     shakl          dtype
  son        (4,)  torch.float32
  kat        (2,)    torch.int64
  y            ()    torch.int64

=== 2. Bir xil indeks - bir xil natija ===
  ds[7]['son'] ikki marta bir xilmi: True
  Dataset DETERMINISTIK bo'lishi kerak (augmentatsiyasiz)

=== 3. TensorDataset ===
  len: 100, td[0] = (shakl (4,), y=1)
  tuple qaytaradi: tuple

=== 4. Subset va ConcatDataset ===
  Subset(0..30): 30
  Subset(70..100): 30
  Concat: 60
  birga[30] == ds[70]: True

=== 5. Salbiy va chegara indekslari ===
  ds[-1]['y'] == ds[99]['y']: True
  ds[100]: IndexError - index 100 is out of bounds for dimension

=== 6. Uzunlik mos kelmasa ===
  AssertionError: uzunliklar mos emas
  ⭐ __init__ da tekshiruv - keyingi xatolarni oldini oladi

"""batch_size, shuffle, drop_last, generator (real torch)."""

import torch
from torch.utils.data import DataLoader, TensorDataset


def main() -> None:
    X = torch.arange(10, dtype=torch.float32).unsqueeze(1)
    y = torch.arange(10)
    ds = TensorDataset(X, y)

    print("=== 1. Tartibli (validatsiya) ===")
    dl = DataLoader(ds, batch_size=4, shuffle=False)
    for i, (xb, yb) in enumerate(dl):
        print(f"  batch {i}: y = {yb.tolist()}")
    print(f"  batchlar soni: {len(dl)}")

    print("\n=== 2. drop_last ===")
    for dl_ in [DataLoader(ds, batch_size=4, drop_last=False),
                DataLoader(ds, batch_size=4, drop_last=True)]:
        hajmlar = [len(yb) for _, yb in dl_]
        print(f"  drop_last={dl_.drop_last}: hajmlar {hajmlar}, "
              f"jami {sum(hajmlar)}")
    print("  drop_last=True oxirgi kichik batchni tashlaydi")

    print("\n=== 3. shuffle har davrda boshqa ===")
    g = torch.Generator().manual_seed(0)
    dl = DataLoader(ds, batch_size=10, shuffle=True, generator=g)
    for davr in range(3):
        _, yb = next(iter(dl))
        print(f"  davr {davr}: {yb.tolist()}")

    print("\n=== 4. generator bilan takrorlanuvchanlik ===")
    def tartib(seed):
        g = torch.Generator().manual_seed(seed)
        dl = DataLoader(ds, batch_size=10, shuffle=True, generator=g)
        return [next(iter(dl))[1].tolist() for _ in range(2)]

    a, b = tartib(42), tartib(42)
    print(f"  seed 42, 1-yurish: {a[0]}")
    print(f"  seed 42, 2-yurish: {b[0]}")
    print(f"  bir xilmi: {a == b}")
    print(f"  seed 7 bilan farqlimi: {tartib(7) != a}")

    print("\n=== 5. Har davrda har namuna bir marta ===")
    g = torch.Generator().manual_seed(1)
    dl = DataLoader(ds, batch_size=3, shuffle=True, generator=g)
    korilgan = sorted(v for _, yb in dl for v in yb.tolist())
    print(f"  bir davrda ko'rilgan: {korilgan}")
    print(f"  hammasi bir martadan: {korilgan == list(range(10))}")

    print("\n=== 6. sampler va shuffle birga ===")
    from torch.utils.data import SequentialSampler
    try:
        DataLoader(ds, batch_size=4, shuffle=True,
                   sampler=SequentialSampler(ds))
        print("  xato chiqmadi (kutilmagan)")
    except ValueError as xato:
        print(f"  ValueError: {str(xato)[:58]}")
    print("  ⭐ sampler berilsa shuffle ni bermang")

    print("\n=== 7. O'quv va validatsiya sozlamalari ===")
    jadval = [
        ("batch_size", "32-256", "kattaroq (512+)"),
        ("shuffle", "True", "False"),
        ("drop_last", "True (BN bo'lsa)", "False"),
        ("generator", "seed bilan", "kerak emas"),
    ]
    print(f"  {'parametr':<12} {'o_quv':<18} {'validatsiya'}")
    for a, b, c in jadval:
        print(f"  {a:<12} {b:<18} {c}")


if __name__ == "__main__":
    main()

=== 1. Tartibli (validatsiya) ===
  batch 0: y = [0, 1, 2, 3]
  batch 1: y = [4, 5, 6, 7]
  batch 2: y = [8, 9]
  batchlar soni: 3

=== 2. drop_last ===
  drop_last=False: hajmlar [4, 4, 2], jami 10
  drop_last=True: hajmlar [4, 4], jami 8
  drop_last=True oxirgi kichik batchni tashlaydi

=== 3. shuffle har davrda boshqa ===
  davr 0: [3, 7, 5, 2, 0, 8, 1, 6, 9, 4]
  davr 1: [9, 2, 1, 7, 4, 0, 3, 5, 6, 8]
  davr 2: [4, 6, 7, 1, 2, 5, 0, 3, 9, 8]

=== 4. generator bilan takrorlanuvchanlik ===
  seed 42, 1-yurish: [6, 5, 4, 0, 8, 9, 2, 1, 3, 7]
  seed 42, 2-yurish: [6, 5, 4, 0, 8, 9, 2, 1, 3, 7]
  bir xilmi: True
  seed 7 bilan farqlimi: True

=== 5. Har davrda har namuna bir marta ===
  bir davrda ko'rilgan: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  hammasi bir martadan: True

=== 6. sampler va shuffle birga ===
  ValueError: sampler option is mutually exclusive with shuffle
  ⭐ sampler berilsa shuffle ni bermang

=== 7. O'quv va validatsiya sozlamalari ===
  parametr     o_quv              validatsiya
  batch_size   32-256             kattaroq (512+)
  shuffle      True               False
  drop_last    True (BN bo'lsa)   False
  generator    seed bilan         kerak emas

"""Turli uzunlikdagi ketma-ketliklarni batchga yig'ish (real torch)."""

import torch
from torch.utils.data import DataLoader, Dataset


class KetmaKetlik(Dataset):
    """Har namuna - turli uzunlikdagi sonlar ketma-ketligi."""

    def __init__(self, n=40, seed=0):
        g = torch.Generator().manual_seed(seed)
        self.uzunliklar = torch.randint(3, 12, (n,), generator=g)
        self.xlar = [torch.randn(int(L), generator=g)
                     for L in self.uzunliklar]
        self.ylar = [int(x.sum() > 0) for x in self.xlar]

    def __len__(self):
        return len(self.xlar)

    def __getitem__(self, i):
        return self.xlar[i], self.ylar[i]


def toldir(batch):
    xlar, ylar = zip(*batch)
    uzunliklar = torch.tensor([len(x) for x in xlar])
    L = int(uzunliklar.max())
    X = torch.zeros(len(xlar), L)
    for i, x in enumerate(xlar):
        X[i, :len(x)] = x
    maska = torch.arange(L)[None, :] < uzunliklar[:, None]
    return {"x": X, "maska": maska, "uzunlik": uzunliklar,
            "y": torch.tensor(ylar)}


def main() -> None:
    ds = KetmaKetlik()

    print("=== 1. Namunalar uzunligi ===")
    print(f"  birinchi 8 ta: {[len(ds[i][0]) for i in range(8)]}")

    print("\n=== 2. Sukut collate ishlamaydi ===")
    dl = DataLoader(ds, batch_size=4)
    try:
        next(iter(dl))
        print("  xato chiqmadi (kutilmagan)")
    except RuntimeError as xato:
        print(f"  RuntimeError: {str(xato).splitlines()[0][:56]}")

    print("\n=== 3. O'z collate_fn ===")
    dl = DataLoader(ds, batch_size=4, collate_fn=toldir)
    batch = next(iter(dl))
    print(f"  {'kalit':<9} {'shakl':>10} {'dtype':>14}")
    for k, v in batch.items():
        print(f"  {k:<9} {str(tuple(v.shape)):>10} {str(v.dtype):>14}")
    print(f"  uzunliklar: {batch['uzunlik'].tolist()}")

    print("\n=== 4. Maska ===")
    for i in range(4):
        m = batch["maska"][i].int().tolist()
        print(f"  namuna {i}: {''.join(map(str, m))}")
    print("  1 - haqiqiy qiymat, 0 - to'ldirilgan (padding)")

    print("\n=== 5. Maskasiz o'rtacha xato beradi ===")
    x = batch["x"]
    maska = batch["maska"]
    notogri = x.mean(dim=1)
    togri = (x * maska).sum(dim=1) / maska.sum(dim=1)
    haqiqiy = torch.tensor([ds[i][0].mean().item() for i in range(4)])
    print(f"  {'namuna':>7} {'maskasiz':>10} {'maska bilan':>12} "
          f"{'haqiqiy':>10}")
    for i in range(4):
        print(f"  {i:>7} {notogri[i].item():>10.4f} "
              f"{togri[i].item():>12.4f} {haqiqiy[i].item():>10.4f}")
    print("  padding nollari o'rtachani BUZADI")

    print("\n=== 6. Uzunlik bo'yicha saralash (bucketing) ===")
    tartib = sorted(range(len(ds)), key=lambda i: len(ds[i][0]))
    oddiy = DataLoader(ds, batch_size=8, collate_fn=toldir)
    saralangan = DataLoader(torch.utils.data.Subset(ds, tartib),
                            batch_size=8, collate_fn=toldir)
    for nom, dl_ in [("oddiy", oddiy), ("saralangan", saralangan)]:
        jami, haqiqiy_son = 0, 0
        for b in dl_:
            jami += b["maska"].numel()
            haqiqiy_son += int(b["maska"].sum())
        print(f"  {nom:<12} padding ulushi: "
              f"{1 - haqiqiy_son / jami:.1%}")
    print("  ⭐ O'xshash uzunliklarni bir batchga - kam padding")


if __name__ == "__main__":
    main()

=== 1. Namunalar uzunligi ===
  birinchi 8 ta: [11, 3, 5, 9, 10, 9, 10, 4]

=== 2. Sukut collate ishlamaydi ===
  RuntimeError: stack expects each tensor to be equal size, but got [11]

=== 3. O'z collate_fn ===
  kalit          shakl          dtype
  x            (4, 11)  torch.float32
  maska        (4, 11)     torch.bool
  uzunlik         (4,)    torch.int64
  y               (4,)    torch.int64
  uzunliklar: [11, 3, 5, 9]

=== 4. Maska ===
  namuna 0: 11111111111
  namuna 1: 11100000000
  namuna 2: 11111000000
  namuna 3: 11111111100
  1 - haqiqiy qiymat, 0 - to'ldirilgan (padding)

=== 5. Maskasiz o'rtacha xato beradi ===
   namuna   maskasiz  maska bilan    haqiqiy
        0     0.2450       0.2450     0.2450
        1    -0.0525      -0.1926    -0.1926
        2    -0.2924      -0.6434    -0.6434
        3    -0.2707      -0.3308    -0.3308
  padding nollari o'rtachani BUZADI

=== 6. Uzunlik bo'yicha saralash (bucketing) ===
  oddiy        padding ulushi: 34.5%
  saralangan   padding ulushi: 10.0%
  ⭐ O'xshash uzunliklarni bir batchga - kam padding

"""WeightedRandomSampler, guruhli bo'lish va o'rgatish (real torch)."""

import numpy as np
import torch
import torch.nn as nn
from sklearn.metrics import average_precision_score
from sklearn.model_selection import GroupShuffleSplit
from torch.utils.data import (DataLoader, Subset, TensorDataset,
                              WeightedRandomSampler)


def orgat(dl, Xva, yva, davrlar=25, seed=0):
    torch.manual_seed(seed)
    model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(),
                          nn.Linear(32, 2))
    opt = torch.optim.AdamW(model.parameters(), lr=0.01)
    kriteriy = nn.CrossEntropyLoss()
    for _ in range(davrlar):
        model.train()
        for xb, yb in dl:
            opt.zero_grad()
            kriteriy(model(xb), yb).backward()
            opt.step()
    model.eval()
    with torch.no_grad():
        p = torch.softmax(model(Xva), dim=1)[:, 1].numpy()
    return p


def main() -> None:
    rng = np.random.default_rng(0)
    n = 4000
    guruh = rng.integers(0, 400, n)
    X = rng.normal(0, 1, (n, 10)).astype(np.float32)
    kuch = X[:, 0] * 1.5 + X[:, 1] - 3.8 + rng.normal(0, 0.8, n)
    y = (kuch > 0).astype(np.int64)
    print(f"  {n} namuna, musbat ulush: {y.mean():.2%}")

    print("\n=== 1. Guruhli bo'lish (random_split EMAS) ===")
    tr, va = next(GroupShuffleSplit(1, test_size=0.25, random_state=0)
                  .split(X, y, groups=guruh))
    ds = TensorDataset(torch.tensor(X), torch.tensor(y))
    oquv, val = Subset(ds, tr), Subset(ds, va)
    print(f"  o'quv: {len(oquv)}, validatsiya: {len(val)}")
    print(f"  guruh kesishishi: "
          f"{len(np.intersect1d(guruh[tr], guruh[va]))}")
    Xva = torch.tensor(X[va])
    yva = y[va]

    print("\n=== 2. Oddiy DataLoader da batch tarkibi ===")
    g = torch.Generator().manual_seed(0)
    dl_oddiy = DataLoader(oquv, batch_size=64, shuffle=True,
                          generator=g)
    ulushlar = [yb.float().mean().item() for _, yb in dl_oddiy]
    print(f"  batchdagi musbat ulushi: o'rtacha {np.mean(ulushlar):.3f}, "
          f"min {min(ulushlar):.3f}, max {max(ulushlar):.3f}")
    nol = sum(1 for u in ulushlar if u == 0)
    print(f"  birorta ham musbat yo'q batchlar: {nol}/{len(ulushlar)}")

    print("\n=== 3. WeightedRandomSampler ===")
    ytr = torch.tensor(y[tr])
    chastota = torch.bincount(ytr).float()
    ogirlik = (1.0 / chastota)[ytr]
    g = torch.Generator().manual_seed(0)
    sampler = WeightedRandomSampler(ogirlik, num_samples=len(ogirlik),
                                    replacement=True, generator=g)
    dl_w = DataLoader(oquv, batch_size=64, sampler=sampler)
    ulushlar = [yb.float().mean().item() for _, yb in dl_w]
    print(f"  batchdagi musbat ulushi: o'rtacha {np.mean(ulushlar):.3f}, "
          f"min {min(ulushlar):.3f}, max {max(ulushlar):.3f}")
    idx_royxat = list(iter(sampler))
    noyob = len(set(idx_royxat))
    print(f"  bir davrda noyob indekslar: {noyob}/{len(idx_royxat)}")
    musbat_idx = set(np.where(y[tr] == 1)[0].tolist())
    takror = sum(1 for i in idx_royxat if i in musbat_idx)
    print(f"  musbat namunalar {len(musbat_idx)} ta, "
          f"davrda {takror} marta tanlandi")
    print("  kam sinf TAKRORLANADI - yodlash xavfi")

    print("\n=== 4. O'rgatishga ta'siri ===")
    print(f"  {'variant':<22} {'val AP':>9} {'o_rt bashorat':>14}")
    g1 = torch.Generator().manual_seed(0)
    p1 = orgat(DataLoader(oquv, batch_size=64, shuffle=True,
                          generator=g1), Xva, yva)
    g2 = torch.Generator().manual_seed(0)
    s2 = WeightedRandomSampler(ogirlik, num_samples=len(ogirlik),
                               replacement=True, generator=g2)
    p2 = orgat(DataLoader(oquv, batch_size=64, sampler=s2), Xva, yva)
    for nom, p in [("oddiy", p1), ("WeightedRandomSampler", p2)]:
        print(f"  {nom:<22} {average_precision_score(yva, p):>9.4f} "
              f"{p.mean():>14.4f}")
    print(f"  haqiqiy musbat ulush: {yva.mean():.4f}")
    print("  sampler tartiblashni SAQLAYDI, lekin ehtimollikni SILJITADI")

    print("\n=== 5. Validatsiya uchun ===")
    dl_val = DataLoader(val, batch_size=512, shuffle=False)
    hajmlar = [len(yb) for _, yb in dl_val]
    print(f"  batchlar: {hajmlar}")
    print("  validatsiyada sampler ham, shuffle ham YO'Q")
    print("  ⭐ Muvozanatlash faqat O'QUV DataLoader ida")


if __name__ == "__main__":
    main()

  4000 namuna, musbat ulush: 2.53%

=== 1. Guruhli bo'lish (random_split EMAS) ===
  o'quv: 2958, validatsiya: 1042
  guruh kesishishi: 0

=== 2. Oddiy DataLoader da batch tarkibi ===
  batchdagi musbat ulushi: o'rtacha 0.026, min 0.000, max 0.143
  birorta ham musbat yo'q batchlar: 13/47

=== 3. WeightedRandomSampler ===
  batchdagi musbat ulushi: o'rtacha 0.492, min 0.359, max 0.578
  bir davrda noyob indekslar: 1222/2958
  musbat namunalar 71 ta, davrda 1511 marta tanlandi
  kam sinf TAKRORLANADI - yodlash xavfi

=== 4. O'rgatishga ta'siri ===
  variant                   val AP  o_rt bashorat
  oddiy                     0.5258         0.0193
  WeightedRandomSampler     0.5618         0.0479
  haqiqiy musbat ulush: 0.0288
  sampler tartiblashni SAQLAYDI, lekin ehtimollikni SILJITADI

=== 5. Validatsiya uchun ===
  batchlar: [512, 512, 18]
  validatsiyada sampler ham, shuffle ham YO'Q
  ⭐ Muvozanatlash faqat O'QUV DataLoader ida

DataLoader(val_ds, shuffle=True)                    # ⚠️
DataLoader(val_ds, shuffle=False)                   # ✅

random_split(ds, [0.8, 0.2])                        # ⚠️ leakage
Subset(ds, tr_idx) # GroupShuffleSplit dan         # ✅

DataLoader(ds, shuffle=True)                        # ⚠️ takrorlanmaydi
DataLoader(ds, shuffle=True, generator=g)           # ✅

x.mean(dim=1)                                       # ⚠️ padding hisobga
(x * maska).sum(1) / maska.sum(1)                   # ✅

DataLoader(ds, sampler=s, shuffle=True)             # ⚠️
DataLoader(ds, sampler=s)                           # ✅

WeightedRandomSampler(...) + CrossEntropyLoss(weight=w)   # ⚠️
# bittasini tanlang                                        # ✅

self.ma_lumot = pd.read_csv("20GB.csv")             # ⚠️
self.yollar = [...]  # __getitem__ da o'qish        # ✅

1.  # Dataset qaysi ikki metodni talab qiladi?
2.  # __getitem__ nima qaytaradi?
3.  # batch kim yig'adi?
4.  # validatsiyada shuffle?
5.  # drop_last nima qiladi?
6.  # generator nima uchun?
7.  # turli uzunlik uchun nima?
8.  # maska nima uchun?
9.  # WeightedRandomSampler nima qiladi?
10. # uning xavfi?
11. # guruhli ma'lumotda bo'lish?
12. # debug da num_workers?

1.  DataLoader(val_ds, shuffle=True)

2.  random_split(ds, [0.8, 0.2])   # mijozlar takrorlanadi

3.  x.mean(dim=1)                   # padding bor

4.  DataLoader(ds, sampler=s, shuffle=True)

5.  DataLoader(ds, shuffle=True)    # takrorlanmaydi