28.4-dars: Tavsiya tizimlari asoslari

1. REYTING BASHORATI (rating prediction)
   savol:   "foydalanuvchi u mahsulot i ga qanday baho qo'yadi?"
   ma'lumot: (u, i, baho) uchliklari, baho 1..5
   metrika: RMSE, MAE - faqat MA'LUM baholarda
   klassik misol: Netflix Prize (2006-2009)

2. TOP-N RANJIRLASH (top-N recommendation)
   savol:   "foydalanuvchi u ga qaysi N ta mahsulotni ko'rsatamiz?"
   ma'lumot: (u, i) juftliklari - ko'rdi/oldi (baho shart emas)
   metrika: precision@k, recall@k, hit rate, MAP, nDCG@k
   amaliyotda deyarli har doim SHU vazifa

NEGA FARQ QILADI:
   RMSE faqat baho qo'yilgan kataklarda o'lchanadi - bu foydalanuvchi
   allaqachon TANLAGAN mahsulotlar. Tavsiya esa KO'RMAGAN mahsulotlar
   orasidan tanlaydi. Baholarni yaxshi bashorat qiladigan model
   "nimani ko'rsatish kerak" savoliga yomon javob berishi mumkin.

EXPLICIT (aniq) feedback: foydalanuvchi o'zi aytgan fikr
   yulduzcha baho, like/dislike, sharh
   + ma'nosi aniq (1 = yoqmadi, 5 = yoqdi)
   - KAM: 1-10% ta'sirlarda bor
   - TANLOV TARAFKASHLIGI: odamlar yoqqan narsani sotib oladi va baholaydi,
     baholar "tasodifiy yo'qolmagan" (missing not at random, MNAR)

IMPLICIT (yashirin) feedback: xulqdan kelib chiqadigan signal
   ko'rish, savatga qo'shish, xarid, tinglash vaqti, qayta xarid
   + KO'P: har harakat yoziladi
   - faqat IJOBIY signal: bo'sh katak = "yoqmadi" YOKI "ko'rmadi"
   - kuchi har xil: 1 ko'rish << 5 xarid  (28.5 da "ishonch" g'oyasi)
   - shovqin: tasodifiy bosish, sovg'a uchun xarid

             mahsulotlar ->
            i1  i2  i3  i4  i5  i6  ...
   u1        1   -   -   1   -   -
   u2        -   1   -   -   -   1          1 = o'zaro ta'sir bor
   u3        1   -   1   -   -   -          - = BO'SH (noma'lum!)
   ...

ZICHLIK = to'ldirilgan kataklar / (foydalanuvchilar x mahsulotlar)
   bizning sintetik do'kon:   ~2.8%
   real onlayn do'kon:        0.001% - 0.1%

SAQLASH: zich massiv o'rniga SIYRAK (CSR - compressed sparse row)
   data    - nol bo'lmagan qiymatlar
   indices - ularning ustun raqamlari
   indptr  - har qator qayerdan boshlanishi
   xotira ~ nnz * 12 bayt (float64 + int32), zich esa n_user * n_item * 8

  ta'sirlar soni
   |#
   |##
   |####
   |#######__
   |          ------------------------------------  <- UZUN DUM
   +---------------------------------------------->  mahsulotlar (mashhurlik tartibida)
     BOSH: oz sonli mashhur       DUM: ko'p sonli kam uchraydigan
     mahsulot, ko'p ta'sir        mahsulot, har biri oz ta'sir

TASODIFIY:    har foydalanuvchiga tasodifiy k ta (ko'rmagan) mahsulot
              pastki chegara - "model umuman biror narsa o'rgandimi?"

MASHHURLIK:   hamma foydalanuvchiga eng ko'p ta'sirli k ta mahsulot
              (u ko'rganlari chiqarib tashlanadi)
              shaxsiylashtirilmagan, lekin KO'PINCHA YENGISH QIYIN

VAQT OYNASIDAGI MASHHURLIK: faqat oxirgi 30 kundagi ta'sirlar
              katalog va moda o'zgarganda muhim
              ⚠️ "oxirgi" - test boshlanishiga nisbatan, kelajakdan emas

1. Har mahsulot -> vektor (tavsif TF-IDF, kategoriya, narx, brend)    [23.4]
2. Foydalanuvchi profili = ko'rgan mahsulotlari vektorlarining yig'indisi
3. ball(u, i) = kosinus(profil_u, vektor_i)

+ yangi mahsulotni birinchi kundanoq tavsiya qila oladi (sovuq start, 28.5)
+ tushuntirish oson: "siz 'qahva' ko'rgansiz"
- "tor": faqat allaqachon ko'rganlariga o'xshashlarni beradi (xilma-xillik past)
- mashhurlik va sifatni bilmaydi - tavsifi o'xshash yomon mahsulot ham chiqadi
- atributlar sifati hal qiladi

KOSINUS O'XSHASHLIK (ikki vektor orasida):
   cos(a, b) = (a . b) / (|a| * |b|)
   binar ma'lumotda: umumiy foydalanuvchilar / sqrt(n_a * n_b)

ITEM-kNN (mahsulotga asoslangan):
   S = mahsulot-mahsulot o'xshashlik:  X ustunlari normallanadi, S = Xn^T Xn
   har mahsulotda faqat eng yaqin K qo'shni qoladi
   ball(u, i) = sum_{j u ko'rgan} S[i, j]
   "siz X ni olgansiz - X ni olganlar Y ni ham olgan"

USER-kNN (foydalanuvchiga asoslangan):
   W = foydalanuvchi-foydalanuvchi o'xshashlik: X qatorlari normallanadi
   har foydalanuvchida eng yaqin K qo'shni
   ball(u, i) = sum_{v qo'shni} W[u, v] * X[v, i]

QAYSI BIRI:
   item-kNN - mahsulotlar kamroq va barqarorroq -> S ni kamdan-kam qayta
              hisoblash yetadi; tushuntirish oson; sanoatda keng tarqalgan
   user-kNN - foydalanuvchi didi tez o'zgaradi, foydalanuvchilar ko'p ->
              W katta va tez eskiradi
   ikkalasi ham: yangi foydalanuvchi/mahsulot uchun ishlamaydi (sovuq start)

TASODIFIY BO'LISH: har foydalanuvchining 20% ta'sirlari tasodifan test
   ⚠️ train ga KELAJAK kiradi: foydalanuvchi keyinroq olganlari train da,
      oldin olganlari test da; mashhurlik keyingi oylardan hisoblanadi
   natija ODATDA oshirib ko'rsatiladi

HAR FOYDALANUVCHINING OXIRGISI (leave-last-out, vaqt bo'yicha):
   har foydalanuvchining OXIRGI 20% ta'sirlari test
   real vazifaga yaqin: "tarixiga qarab keyingisini top"
   ⚠️ boshqa foydalanuvchilarning keyingi ta'sirlari train da qoladi
      (kichik sizish) - kerak bo'lsa, mashhurlikni test boshlanishigacha
      bo'lgan oynadan hisoblang

GLOBAL VAQT CHEGARASI: sana T gacha - train, T dan keyin - test
   eng halol (ishlab chiqarishga aynan mos), lekin test da yangi
   mahsulotlar va yangi foydalanuvchilar ko'proq - raqamlar pastroq

precision@k = (top ichidagi to'g'rilar) / k
recall@k    = (top ichidagi to'g'rilar) / |T|
hit@k       = 1, agar kamida bitta to'g'ri bo'lsa (hit rate - o'rtachasi)

AP@k  = (1 / min(|T|, k)) * sum_{j=1..k} precision@j * rel_j
MAP@k = foydalanuvchilar bo'yicha AP@k o'rtachasi

DCG@k  = sum_{j=1..k} rel_j / log2(j + 1)          (yuqoridagi o'rin qimmatroq)
IDCG@k = eng yaxshi tartibdagi DCG = sum_{j=1..min(|T|,k)} 1 / log2(j + 1)
nDCG@k = DCG@k / IDCG@k                            ([0, 1] oralig'ida)

MISOL: top = [5, 3, 9, 1, 7], T = {3, 7, 8}  ->  rel = [0, 1, 0, 0, 1]
   precision@5 = 2/5 = 0.4        recall@5 = 2/3 = 0.667     hit = 1
   AP@5  = (1/2 + 2/5) / 3 = 0.3
   nDCG@5 = (1/log2(3) + 1/log2(6)) / (1 + 1/log2(3) + 1/log2(4)) = 0.478

QAMROV (catalog coverage):
   kamida bir marta tavsiya qilingan mahsulotlar / katalog hajmi
   mashhurlik ro'yxati: k / n_item  (hammaga bir xil 10 ta)

XILMA-XILLIK (intra-list diversity, ILD):
   ro'yxat ichidagi juftliklarning qanchasi TURLI kategoriyadan
   (yoki 1 - o'rtacha o'xshashlik)

YANGILIK (novelty):
   o'rtacha -log2(pop_i / n_user) - kam uchraydigan mahsulot "yangiroq"

MASHHURLIK TARAFKASHLIGI (popularity bias):
   tavsiyalarda eng mashhur 10% mahsulotlar ulushi
   VS testdagi haqiqiy ta'sirlarda shu ulush
   tavsiya >> test  ->  model mashhurlarga yig'ilyapti

har foydalanuvchi u uchun: d_u = nDCG_A(u) - nDCG_B(u)
o'rtacha farq  = mean(d)
SE             = std(d) / sqrt(n_user)
|mean(d)| > 2 * SE  ->  farq sezilarli

QAROR QOIDASI: eng yaxshisidan sezilarli yomon bo'lmagan ENG SODDA usul
   soddalik tartibi: mashhurlik < kontent < item-kNN < user-kNN

import numpy as np
from scipy import sparse

X = sparse.csr_matrix((np.ones(len(df)), (df["user"], df["item"])),
                      shape=(n_user, n_item))              # foydalanuvchi x mahsulot

# mashhurlik
pop = np.asarray(X.sum(0)).ravel()

# item-kNN: kosinus = normallangan ustunlar ko'paytmasi
Xn = X @ sparse.diags(1 / np.sqrt(np.maximum(pop, 1)))
S = (Xn.T @ Xn).toarray()
np.fill_diagonal(S, 0)
ball = X[users] @ S                                        # (n, n_item)

# kontent: TF-IDF profil (23.4)
V = TfidfVectorizer().fit_transform(items["tavsif"])
ball = cosine_similarity(X[users] @ V, V)

# baholash: ko'rilganlarni chiqarish, top-k, metrikalar
ball[X[users].toarray() > 0] = -np.inf
top = np.argsort(-ball, axis=1, kind="stable")[:, :10]
rel = np.take_along_axis(T, top, axis=1)                   # T - test (bool)
recall = rel.sum(1) / T.sum(1)
ndcg = (rel / np.log2(np.arange(2, 12))).sum(1) / idcg

# juftlashgan taqqoslash
d = ndcg_A - ndcg_B
sezilarli = abs(d.mean()) > 2 * d.std(ddof=1) / np.sqrt(len(d))

vazifa: top-N ranjirlash (reyting bashorati - kamdan-kam)
implicit: bo'sh katak = noma'lum, "yoqmadi" emas
bazaviy: mashhurlik - majburiy va kuchli
bo'lish: vaqt bo'yicha; tasodifiy - oshirib ko'rsatadi
metrikalar: precision/recall/hit/MAP/nDCG @k - har foydalanuvchida
aniqlikdan tashqari: qamrov, xilma-xillik, yangilik, mashhurlik ulushi

"""Onlayn do'kon ma'lumoti: matritsa, siyraklik, uzun dum, explicit va implicit."""

import numpy as np
import pandas as pd
from scipy import sparse

KATEGORIYA = {
    "elektronika": "telefon smartfon quloqchin zaryadlovchi noutbuk planshet "
                   "sichqoncha klaviatura kolonka televizor kamera soat",
    "kiyim": "koylak shim kurtka poyabzal krossovka shapka sharf palto "
             "kostyum futbolka yubka sviter",
    "kitob": "roman darslik detektiv sher tarix fantastika lugat ertak "
             "biografiya psixologiya biznes dasturlash",
    "oziq": "choy qahva asal yongoq shokolad guruch yog un makaron sharbat "
            "pechene ziravor",
    "gozallik": "krem atir shampun labbuyoq boyoq niqob sovun losyon taroq "
                "lak gel upa",
    "uy": "idish qozon choynak gilam parda yostiq chiroq sochiq pichoq tova "
          "vaza javon",
    "sport": "koptok gantel velosiped raketka forma gilamcha skakalka shlem "
             "sumka butsa trenajyor suzish",
    "bolalar": "oyinchoq konstruktor qogirchoq kolyaska taglik mashinka pazl "
               "albom rangli kiyimcha kitobcha samokat",
}
NARX = ["arzon", "orta", "premium"]
UMUMIY = "sifatli yangi chegirma original mashhur qulay kafolat tez yetkazish".split()


def dokon(seed=0, n_user=2000, n_item=800, kunlar=180):
    """Sintetik onlayn do'kon: yashirin didlar, mashhurlik dumi, yangi mahsulotlar."""
    rng = np.random.default_rng(seed)
    katlar = list(KATEGORIYA)
    n_kat, n_sub = len(katlar), 4
    sub = rng.integers(0, n_kat * n_sub, n_item)          # har mahsulot kichik guruhi
    kat = sub // n_sub
    narx = rng.choice(3, n_item, p=[0.5, 0.35, 0.15])
    sifat = rng.lognormal(0, 1.4, n_item)                  # mashhurlik dumi manbai
    chiqish = np.where(rng.random(n_item) < 0.7, 0,
                       rng.integers(0, kunlar - 20, n_item))
    tavsif = []
    for i in range(n_item):
        sozlar = KATEGORIYA[katlar[kat[i]]].split()
        guruh = sozlar[(sub[i] % n_sub) * 3:(sub[i] % n_sub) * 3 + 3]
        tanlov = list(rng.choice(guruh, 2, replace=False))
        tanlov.append(rng.choice([s for s in sozlar if s not in tanlov]))
        tanlov += [katlar[kat[i]], NARX[narx[i]], f"brend{rng.integers(0, 40)}"]
        tanlov += list(rng.choice(UMUMIY, 2, replace=False))
        tavsif.append(" ".join(tanlov))
    # foydalanuvchi didi: kategoriya x kichik guruh x narx
    kat_pref = rng.dirichlet(np.full(n_kat, 0.3), n_user)
    sub_pref = rng.dirichlet(np.full(n_sub, 0.5), (n_user, n_kat))
    narx_pref = rng.dirichlet(np.full(3, 1.0), n_user)
    did = (kat_pref[:, kat] * sub_pref[:, kat, sub % n_sub]
           * narx_pref[:, narx])                           # (user, item)
    did /= did.sum(1, keepdims=True)
    hafta = np.arange(0, kunlar, 7)
    yosh = np.clip((hafta[:, None] - chiqish[None, :]) / 7, 0, None)
    jalb = sifat * (0.6 + 1.2 * np.exp(-yosh / 6)) * (hafta[:, None] >= chiqish)
    qatorlar = []
    for u in range(n_user):
        n_u = int(np.clip(rng.lognormal(2.8, 0.8), 3, 150))
        vaqtlar = np.sort(rng.uniform(0, kunlar, n_u))
        korgan = np.zeros(n_item, dtype=bool)
        for t in vaqtlar:
            a = jalb[int(t) // 7] * ~korgan
            p = 0.75 * did[u] * a / (did[u] @ a) + 0.25 * a / a.sum()
            i = int(np.searchsorted(np.cumsum(p), rng.random() * p.sum()))
            korgan[i] = True
            mos = did[u, i] / did[u].max()
            xarid = rng.random() < 0.15 + 0.35 * mos
            baho = 0
            if xarid and rng.random() < 0.3:
                baho = int(np.clip(round(2.4 + 2.2 * mos + 0.3 * np.log(sifat[i])
                                         + rng.normal(0, 0.7)), 1, 5))
            qatorlar.append((u, i, t, int(xarid), baho))
    df = pd.DataFrame(qatorlar, columns=["user", "item", "vaqt", "xarid", "baho"])
    items = pd.DataFrame({"item": np.arange(n_item), "kategoriya": np.array(katlar)[kat],
                          "sub": sub, "narx": narx, "chiqish": chiqish,
                          "tavsif": tavsif})
    return df, items


def main() -> None:
    df, items = dokon()
    n_user, n_item = df["user"].nunique(), len(items)

    print("=== 1. O'zaro ta'sirlar jurnali ===")
    print(f"  foydalanuvchi {n_user}, mahsulot {n_item}, "
          f"ko'rish {len(df)}, xarid {int(df['xarid'].sum())}, "
          f"baho {int((df['baho'] > 0).sum())}")
    faol = df.groupby("user").size()
    q = np.percentile(faol, [10, 50, 90, 99]).astype(int).tolist()
    print(f"  bir foydalanuvchi: min {faol.min()}, persentillar 10/50/90/99 = {q},"
          f" max {faol.max()}")
    print(df.head(3).round(2).to_string(index=False))

    print("\n=== 2. Foydalanuvchi-mahsulot matritsasi ===")
    X = sparse.csr_matrix((np.ones(len(df)), (df["user"], df["item"])),
                          shape=(n_user, n_item))
    zichlik = X.nnz / (n_user * n_item)
    zich_bayt = n_user * n_item * 8
    csr_bayt = X.data.nbytes + X.indices.nbytes + X.indptr.nbytes
    print(f"  to'ldirilgan kataklar: {X.nnz} ({zichlik:.2%}), bo'sh {1 - zichlik:.2%}")
    print(f"  zich float64: {zich_bayt:,} bayt;  CSR: {csr_bayt:,} bayt "
          f"({zich_bayt / csr_bayt:.0f} marta kam)")
    u_real, i_real, nnz_real = 10_000_000, 1_000_000, 500_000_000
    print(f"  real miqyos (10 mln x 1 mln, 500 mln ta'sir): zich "
          f"{u_real * i_real * 8 / 1e12:.0f} TB, CSR ~{nnz_real * 12 / 1e9:.0f} GB, "
          f"zichlik {nnz_real / (u_real * i_real):.3%}")

    print("\n=== 3. Uzun dum: mashhurlik juda notekis ===")
    pop = np.sort(np.asarray(X.sum(0)).ravel())[::-1]
    for ulush in [0.01, 0.1, 0.2]:
        k = int(n_item * ulush)
        print(f"  eng mashhur {ulush:>4.0%} mahsulot ({k:>3} ta): "
              f"ta'sirlarning {pop[:k].sum() / pop.sum():.1%} i")
    print(f"  5 tadan kam ta'sirli mahsulotlar: {np.mean(pop < 5):.1%}")
    print("  o'nlik guruhlar (mashhur -> kam), ta'sirlar ulushi:")
    for d in range(10):
        s = pop[d * n_item // 10:(d + 1) * n_item // 10].sum() / pop.sum()
        print(f"   {d + 1:>2}  {'#' * int(round(s * 100))} {s:.1%}")

    print("\n=== 4. Explicit va implicit ===")
    baholi = df[df["baho"] > 0]
    print(f"  bahoga ega ta'sirlar: {len(baholi) / len(df):.1%}; "
          f"baho qo'ygan foydalanuvchilar {baholi['user'].nunique() / n_user:.1%}")
    taqs = baholi["baho"].value_counts(normalize=True).sort_index()
    print("  baholar taqsimoti: " + ", ".join(f"{k}: {v:.2f}" for k, v in taqs.items()))
    print(f"  o'rtacha baho {baholi['baho'].mean():.2f} - baho faqat XARID "
          f"qilinganlarga, xarid esa ko'proq yoqqanlarga")

    print("\n=== 5. O'rtacha baho bo'yicha top: kichik namuna tuzog'i ===")
    g = baholi.groupby("item")["baho"].agg(["mean", "count"])
    mu, m = baholi["baho"].mean(), 10
    g["silliq"] = (g["count"] * g["mean"] + m * mu) / (g["count"] + m)
    xom = g.sort_values(["mean", "count"], ascending=False).head(5)
    sil = g.sort_values("silliq", ascending=False).head(5)
    print(f"  xom o'rtacha top-5: baholar soni {xom['count'].tolist()}, "
          f"o'rtacha {xom['mean'].round(2).tolist()}")
    print(f"  silliqlangan (m={m}) top-5: baholar soni {sil['count'].tolist()}, "
          f"o'rtacha {sil['mean'].round(2).tolist()}")

    print("\n=== 6. Vaqt: katalog va mashhurlik o'zgaradi ===")
    yangi = items["chiqish"].to_numpy() > 0
    for a, b in [(0, 30), (75, 105), (150, 180)]:
        oyna = df[(df["vaqt"] >= a) & (df["vaqt"] < b)]
        print(f"  {a:>3}-{b:<3} kun: keyin chiqqan mahsulotlar ulushi "
              f"{yangi[oyna['item']].mean():.1%}")
    top_bosh = df[df["vaqt"] < 60]["item"].value_counts().head(20).index
    top_oxir = df[df["vaqt"] >= 120]["item"].value_counts().head(20).index
    print(f"  birinchi va oxirgi 60 kun top-20 lari kesishmasi: "
          f"{len(set(top_bosh) & set(top_oxir))} ta")
    print("  ⭐ Mashhurlik vaqtga bog'liq - baholashda vaqt tartibini saqlang")


if __name__ == "__main__":
    main()

=== 1. O'zaro ta'sirlar jurnali ===
  foydalanuvchi 2000, mahsulot 800, ko'rish 45019, xarid 11197, baho 3327
  bir foydalanuvchi: min 3, persentillar 10/50/90/99 = [6, 16, 47, 118], max 150
 user  item  vaqt  xarid  baho
    0   718  4.25      1     0
    0   249 15.36      0     0
    0   135 20.71      0     0

=== 2. Foydalanuvchi-mahsulot matritsasi ===
  to'ldirilgan kataklar: 45019 (2.81%), bo'sh 97.19%
  zich float64: 12,800,000 bayt;  CSR: 548,232 bayt (23 marta kam)
  real miqyos (10 mln x 1 mln, 500 mln ta'sir): zich 80 TB, CSR ~6 GB, zichlik 0.005%

=== 3. Uzun dum: mashhurlik juda notekis ===
  eng mashhur   1% mahsulot (  8 ta): ta'sirlarning 7.8% i
  eng mashhur  10% mahsulot ( 80 ta): ta'sirlarning 41.5% i
  eng mashhur  20% mahsulot (160 ta): ta'sirlarning 60.9% i
  5 tadan kam ta'sirli mahsulotlar: 9.6%
  o'nlik guruhlar (mashhur -> kam), ta'sirlar ulushi:
    1  ########################################## 41.5%
    2  ################### 19.4%
    3  ############ 12.5%
    4  ######### 8.9%
    5  ###### 6.4%
    6  ##### 4.6%
    7  ### 3.1%
    8  ## 2.1%
    9  # 1.2%
   10   0.4%

=== 4. Explicit va implicit ===
  bahoga ega ta'sirlar: 7.4%; baho qo'ygan foydalanuvchilar 69.2%
  baholar taqsimoti: 1: 0.01, 2: 0.12, 3: 0.30, 4: 0.32, 5: 0.25
  o'rtacha baho 3.68 - baho faqat XARID qilinganlarga, xarid esa ko'proq yoqqanlarga

=== 5. O'rtacha baho bo'yicha top: kichik namuna tuzog'i ===
  xom o'rtacha top-5: baholar soni [3, 2, 2, 2, 2], o'rtacha [5.0, 5.0, 5.0, 5.0, 5.0]
  silliqlangan (m=10) top-5: baholar soni [7, 30, 17, 33, 10], o'rtacha [4.71, 4.23, 4.29, 4.18, 4.4]

=== 6. Vaqt: katalog va mashhurlik o'zgaradi ===
    0-30  kun: keyin chiqqan mahsulotlar ulushi 3.1%
   75-105 kun: keyin chiqqan mahsulotlar ulushi 27.0%
  150-180 kun: keyin chiqqan mahsulotlar ulushi 39.9%
  birinchi va oxirgi 60 kun top-20 lari kesishmasi: 6 ta
  ⭐ Mashhurlik vaqtga bog'liq - baholashda vaqt tartibini saqlang

"""Ranjirlash metrikalari noldan, bazaviylar va uch xil bo'lish."""

import numpy as np
import pandas as pd
from scipy import sparse
from sklearn.metrics import ndcg_score

KATEGORIYA = {
    "elektronika": "telefon smartfon quloqchin zaryadlovchi noutbuk planshet "
                   "sichqoncha klaviatura kolonka televizor kamera soat",
    "kiyim": "koylak shim kurtka poyabzal krossovka shapka sharf palto "
             "kostyum futbolka yubka sviter",
    "kitob": "roman darslik detektiv sher tarix fantastika lugat ertak "
             "biografiya psixologiya biznes dasturlash",
    "oziq": "choy qahva asal yongoq shokolad guruch yog un makaron sharbat "
            "pechene ziravor",
    "gozallik": "krem atir shampun labbuyoq boyoq niqob sovun losyon taroq "
                "lak gel upa",
    "uy": "idish qozon choynak gilam parda yostiq chiroq sochiq pichoq tova "
          "vaza javon",
    "sport": "koptok gantel velosiped raketka forma gilamcha skakalka shlem "
             "sumka butsa trenajyor suzish",
    "bolalar": "oyinchoq konstruktor qogirchoq kolyaska taglik mashinka pazl "
               "albom rangli kiyimcha kitobcha samokat",
}
NARX = ["arzon", "orta", "premium"]
UMUMIY = "sifatli yangi chegirma original mashhur qulay kafolat tez yetkazish".split()


def dokon(seed=0, n_user=2000, n_item=800, kunlar=180):
    """Sintetik onlayn do'kon: yashirin didlar, mashhurlik dumi, yangi mahsulotlar."""
    rng = np.random.default_rng(seed)
    katlar = list(KATEGORIYA)
    n_kat, n_sub = len(katlar), 4
    sub = rng.integers(0, n_kat * n_sub, n_item)          # har mahsulot kichik guruhi
    kat = sub // n_sub
    narx = rng.choice(3, n_item, p=[0.5, 0.35, 0.15])
    sifat = rng.lognormal(0, 1.4, n_item)                  # mashhurlik dumi manbai
    chiqish = np.where(rng.random(n_item) < 0.7, 0,
                       rng.integers(0, kunlar - 20, n_item))
    tavsif = []
    for i in range(n_item):
        sozlar = KATEGORIYA[katlar[kat[i]]].split()
        guruh = sozlar[(sub[i] % n_sub) * 3:(sub[i] % n_sub) * 3 + 3]
        tanlov = list(rng.choice(guruh, 2, replace=False))
        tanlov.append(rng.choice([s for s in sozlar if s not in tanlov]))
        tanlov += [katlar[kat[i]], NARX[narx[i]], f"brend{rng.integers(0, 40)}"]
        tanlov += list(rng.choice(UMUMIY, 2, replace=False))
        tavsif.append(" ".join(tanlov))
    # foydalanuvchi didi: kategoriya x kichik guruh x narx
    kat_pref = rng.dirichlet(np.full(n_kat, 0.3), n_user)
    sub_pref = rng.dirichlet(np.full(n_sub, 0.5), (n_user, n_kat))
    narx_pref = rng.dirichlet(np.full(3, 1.0), n_user)
    did = (kat_pref[:, kat] * sub_pref[:, kat, sub % n_sub]
           * narx_pref[:, narx])                           # (user, item)
    did /= did.sum(1, keepdims=True)
    hafta = np.arange(0, kunlar, 7)
    yosh = np.clip((hafta[:, None] - chiqish[None, :]) / 7, 0, None)
    jalb = sifat * (0.6 + 1.2 * np.exp(-yosh / 6)) * (hafta[:, None] >= chiqish)
    qatorlar = []
    for u in range(n_user):
        n_u = int(np.clip(rng.lognormal(2.8, 0.8), 3, 150))
        vaqtlar = np.sort(rng.uniform(0, kunlar, n_u))
        korgan = np.zeros(n_item, dtype=bool)
        for t in vaqtlar:
            a = jalb[int(t) // 7] * ~korgan
            p = 0.75 * did[u] * a / (did[u] @ a) + 0.25 * a / a.sum()
            i = int(np.searchsorted(np.cumsum(p), rng.random() * p.sum()))
            korgan[i] = True
            mos = did[u, i] / did[u].max()
            xarid = rng.random() < 0.15 + 0.35 * mos
            baho = 0
            if xarid and rng.random() < 0.3:
                baho = int(np.clip(round(2.4 + 2.2 * mos + 0.3 * np.log(sifat[i])
                                         + rng.normal(0, 0.7)), 1, 5))
            qatorlar.append((u, i, t, int(xarid), baho))
    df = pd.DataFrame(qatorlar, columns=["user", "item", "vaqt", "xarid", "baho"])
    items = pd.DataFrame({"item": np.arange(n_item), "kategoriya": np.array(katlar)[kat],
                          "sub": sub, "narx": narx, "chiqish": chiqish,
                          "tavsif": tavsif})
    return df, items


def bolish(df, usul="vaqt", ulush=0.2, seed=0, chegara=150.0):
    """'vaqt' - har foydalanuvchining oxirgi 20% i test; 'tasodifiy'; 'global'."""
    df = df.sort_values(["user", "vaqt"]).reset_index(drop=True)
    if usul == "global":
        tarix = df[df["vaqt"] < chegara].groupby("user").size()
        yetarli = df["user"].map(tarix).fillna(0).to_numpy() >= 3
        test = (df["vaqt"] >= chegara).to_numpy() & yetarli
        return df[df["vaqt"] < chegara].copy(), df[test].copy()
    n = df.groupby("user")["item"].transform("size").to_numpy()
    if usul == "tasodifiy":
        kalit = np.random.default_rng(seed).random(len(df))
        tartib = (df.assign(k=kalit).groupby("user")["k"]
                  .rank(method="first").to_numpy() - 1)
    else:
        tartib = df.groupby("user").cumcount().to_numpy()
    n_test = np.maximum(1, np.round(ulush * n)).astype(int)
    test = (tartib >= n - n_test) & (n >= 5)
    return df[~test].copy(), df[test].copy()


def matritsa(qism, n_user, n_item):
    return sparse.csr_matrix((np.ones(len(qism)), (qism["user"], qism["item"])),
                             shape=(n_user, n_item))


def metrikalar(top, T):
    """top: (n, k) tavsiya indekslari; T: (n, n_item) test to'plami (bool)."""
    k = top.shape[1]
    rel = np.take_along_axis(T, top, axis=1)
    n_rel = T.sum(1)
    pozitsiya = np.arange(1, k + 1)
    ap = (np.cumsum(rel, 1) / pozitsiya * rel).sum(1) / np.minimum(n_rel, k)
    chegirma = 1 / np.log2(pozitsiya + 1)
    idcg = np.array([chegirma[:min(r, k)].sum() for r in n_rel])
    return {"precision": rel.sum(1) / k, "recall": rel.sum(1) / n_rel,
            "hit": rel.any(1).astype(float), "MAP": ap,
            "nDCG": (rel * chegirma).sum(1) / idcg}


def baholash(ball, X_train, test, users, k=10):
    """Train da ko'rilgan mahsulotlar tavsiya qilinmaydi; har foydalanuvchi metrikasi."""
    ball = np.array(ball, dtype=float)
    ball[X_train[users].toarray() > 0] = -np.inf
    top = np.argsort(-ball, axis=1, kind="stable")[:, :k]
    qator = pd.Series(np.arange(len(users)), index=users)
    tt = test[test["user"].isin(users)]
    T = np.zeros(ball.shape, dtype=bool)
    T[qator[tt["user"]].to_numpy(), tt["item"].to_numpy()] = True
    return metrikalar(top, T), top


def mashhurlik_oynada(train, t_bosh, n_item, oyna=30.0):
    """Har foydalanuvchi uchun: test boshlanishidan OLDINGI 'oyna' kundagi ko'rishlar."""
    vaqt, item = train["vaqt"].to_numpy(), train["item"].to_numpy()
    ball = np.zeros((len(t_bosh), n_item))
    for r, t in enumerate(t_bosh):
        m = (vaqt >= t - oyna) & (vaqt < t)
        ball[r] = np.bincount(item[m], minlength=n_item)
    return ball


def main() -> None:
    print("=== 1. Metrikalar noldan: bitta foydalanuvchi ===")
    top = np.array([[5, 3, 9, 1, 7]])                  # tavsiya tartibi
    T = np.zeros((1, 10), dtype=bool)
    T[0, [3, 7, 8]] = True                             # haqiqatda olganlari
    m = metrikalar(top, T)
    print("  tavsiya [5, 3, 9, 1, 7], haqiqiy {3, 7, 8}; relevantlik [0 1 0 0 1]")
    print("  " + ", ".join(f"{k}@5 {v[0]:.4f}" for k, v in m.items()))
    print(f"  qo'lda: AP = (1/2 + 2/5) / 3 = {(1 / 2 + 2 / 5) / 3:.4f};  "
          f"nDCG = (1/log2(3) + 1/log2(6)) / (1 + 1/log2(3) + 1/log2(4)) = "
          f"{(1 / np.log2(3) + 1 / np.log2(6)) / (1 + 1 / np.log2(3) + 0.5):.4f}")
    ball = np.zeros((1, 10))
    ball[0, top[0]] = [5, 4, 3, 2, 1]
    print(f"  sklearn ndcg_score(k=5): {ndcg_score(T.astype(float), ball, k=5):.4f}")

    df, items = dokon()
    n_user, n_item = 2000, len(items)
    rng = np.random.default_rng(1)

    print("\n=== 2. Uch xil bo'lish ===")
    bolishlar = {u: bolish(df, u) for u in ["tasodifiy", "vaqt", "global"]}
    for nom, (tr, te) in bolishlar.items():
        kechroq = np.mean(te["item"].map(items.set_index("item")["chiqish"]) > 0)
        yangi = ~te["item"].isin(tr["item"].unique())
        print(f"  {nom:<10} train {len(tr):>6}, test {len(te):>5} "
              f"({te['user'].nunique()} foyd.), test o'rtacha kuni "
              f"{te['vaqt'].mean():5.1f}, trainda yo'q mahsulot {yangi.mean():.1%}, "
              f"kech chiqqan {kechroq:.1%}")

    print("\n=== 3. Bazaviylar ('vaqt' bo'lishi, k=10) ===")
    tr, te = bolishlar["vaqt"]
    X = matritsa(tr, n_user, n_item)
    users = np.sort(te["user"].unique())
    t_bosh = te.groupby("user")["vaqt"].min().loc[users].to_numpy()
    pop = np.asarray(X.sum(0)).ravel()
    usullar = {
        "tasodifiy": rng.random((len(users), n_item)),
        "mashhurlik (butun)": np.tile(pop, (len(users), 1)),
        "mashhurlik (30 kun)": mashhurlik_oynada(tr, t_bosh, n_item),
    }
    natija = {}
    print(f"  {'usul':<22}" + "".join(f"{k:>10}" for k in
                                     ["precision", "recall", "hit", "MAP", "nDCG"]))
    for nom, b in usullar.items():
        natija[nom], _ = baholash(b, X, te, users)
        print(f"  {nom:<22}" + "".join(f"{v.mean():>10.4f}"
                                        for v in natija[nom].values()))

    print("\n=== 4. Juftlashgan taqqoslash: foydalanuvchilar bo'yicha nDCG@10 ===")
    for a, b in [("mashhurlik (butun)", "tasodifiy"),
                 ("mashhurlik (30 kun)", "mashhurlik (butun)")]:
        d = natija[a]["nDCG"] - natija[b]["nDCG"]
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  {a} - {b}: {d.mean():+.4f}, SE {se:.4f}, "
              f"sezilarli: {abs(d.mean()) > 2 * se}")

    print("\n=== 5. Bo'lish usuli natijani o'zgartiradi (mashhurlik, nDCG@10) ===")
    for nom, (tr, te) in bolishlar.items():
        X = matritsa(tr, n_user, n_item)
        us = np.sort(te["user"].unique())
        pop = np.asarray(X.sum(0)).ravel()
        r, _ = baholash(np.tile(pop, (len(us), 1)), X, te, us)
        se = r["nDCG"].std(ddof=1) / np.sqrt(len(us))
        print(f"  {nom:<10} nDCG@10 {r['nDCG'].mean():.4f} (SE {se:.4f}), "
              f"recall@10 {r['recall'].mean():.4f}")
    print("  ⭐ Tasodifiy bo'lish kelajakni o'qitishga qo'shadi - raqamlar yolg'on")


if __name__ == "__main__":
    main()

=== 1. Metrikalar noldan: bitta foydalanuvchi ===
  tavsiya [5, 3, 9, 1, 7], haqiqiy {3, 7, 8}; relevantlik [0 1 0 0 1]
  precision@5 0.4000, recall@5 0.6667, hit@5 1.0000, MAP@5 0.3000, nDCG@5 0.4776
  qo'lda: AP = (1/2 + 2/5) / 3 = 0.3000;  nDCG = (1/log2(3) + 1/log2(6)) / (1 + 1/log2(3) + 1/log2(4)) = 0.4776
  sklearn ndcg_score(k=5): 0.4776

=== 2. Uch xil bo'lish ===
  tasodifiy  train  36124, test  8895 (1864 foyd.), test o'rtacha kuni  89.8, trainda yo'q mahsulot 0.1%, kech chiqqan 24.9%
  vaqt       train  36124, test  8895 (1864 foyd.), test o'rtacha kuni 159.0, trainda yo'q mahsulot 1.2%, kech chiqqan 40.0%
  global     train  37610, test  7332 (1713 foyd.), test o'rtacha kuni 165.1, trainda yo'q mahsulot 4.6%, kech chiqqan 39.8%

=== 3. Bazaviylar ('vaqt' bo'lishi, k=10) ===
  usul                   precision    recall       hit       MAP      nDCG
  tasodifiy                 0.0058    0.0120    0.0536    0.0041    0.0094
  mashhurlik (butun)        0.0383    0.0924    0.3203    0.0351    0.0706
  mashhurlik (30 kun)       0.0407    0.0972    0.3353    0.0381    0.0767

=== 4. Juftlashgan taqqoslash: foydalanuvchilar bo'yicha nDCG@10 ===
  mashhurlik (butun) - tasodifiy: +0.0612, SE 0.0033, sezilarli: True
  mashhurlik (30 kun) - mashhurlik (butun): +0.0061, SE 0.0041, sezilarli: False

=== 5. Bo'lish usuli natijani o'zgartiradi (mashhurlik, nDCG@10) ===
  tasodifiy  nDCG@10 0.0959 (SE 0.0035), recall@10 0.1187
  vaqt       nDCG@10 0.0706 (SE 0.0031), recall@10 0.0924
  global     nDCG@10 0.0624 (SE 0.0032), recall@10 0.0823
  ⭐ Tasodifiy bo'lish kelajakni o'qitishga qo'shadi - raqamlar yolg'on

"""Kontent-asosli va kollaborativ filtrlash (user/item kNN) - bazaviylarga qarshi."""

import numpy as np
import pandas as pd
from scipy import sparse
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

KATEGORIYA = {
    "elektronika": "telefon smartfon quloqchin zaryadlovchi noutbuk planshet "
                   "sichqoncha klaviatura kolonka televizor kamera soat",
    "kiyim": "koylak shim kurtka poyabzal krossovka shapka sharf palto "
             "kostyum futbolka yubka sviter",
    "kitob": "roman darslik detektiv sher tarix fantastika lugat ertak "
             "biografiya psixologiya biznes dasturlash",
    "oziq": "choy qahva asal yongoq shokolad guruch yog un makaron sharbat "
            "pechene ziravor",
    "gozallik": "krem atir shampun labbuyoq boyoq niqob sovun losyon taroq "
                "lak gel upa",
    "uy": "idish qozon choynak gilam parda yostiq chiroq sochiq pichoq tova "
          "vaza javon",
    "sport": "koptok gantel velosiped raketka forma gilamcha skakalka shlem "
             "sumka butsa trenajyor suzish",
    "bolalar": "oyinchoq konstruktor qogirchoq kolyaska taglik mashinka pazl "
               "albom rangli kiyimcha kitobcha samokat",
}
NARX = ["arzon", "orta", "premium"]
UMUMIY = "sifatli yangi chegirma original mashhur qulay kafolat tez yetkazish".split()


def dokon(seed=0, n_user=2000, n_item=800, kunlar=180):
    """Sintetik onlayn do'kon: yashirin didlar, mashhurlik dumi, yangi mahsulotlar."""
    rng = np.random.default_rng(seed)
    katlar = list(KATEGORIYA)
    n_kat, n_sub = len(katlar), 4
    sub = rng.integers(0, n_kat * n_sub, n_item)          # har mahsulot kichik guruhi
    kat = sub // n_sub
    narx = rng.choice(3, n_item, p=[0.5, 0.35, 0.15])
    sifat = rng.lognormal(0, 1.4, n_item)                  # mashhurlik dumi manbai
    chiqish = np.where(rng.random(n_item) < 0.7, 0,
                       rng.integers(0, kunlar - 20, n_item))
    tavsif = []
    for i in range(n_item):
        sozlar = KATEGORIYA[katlar[kat[i]]].split()
        guruh = sozlar[(sub[i] % n_sub) * 3:(sub[i] % n_sub) * 3 + 3]
        tanlov = list(rng.choice(guruh, 2, replace=False))
        tanlov.append(rng.choice([s for s in sozlar if s not in tanlov]))
        tanlov += [katlar[kat[i]], NARX[narx[i]], f"brend{rng.integers(0, 40)}"]
        tanlov += list(rng.choice(UMUMIY, 2, replace=False))
        tavsif.append(" ".join(tanlov))
    # foydalanuvchi didi: kategoriya x kichik guruh x narx
    kat_pref = rng.dirichlet(np.full(n_kat, 0.3), n_user)
    sub_pref = rng.dirichlet(np.full(n_sub, 0.5), (n_user, n_kat))
    narx_pref = rng.dirichlet(np.full(3, 1.0), n_user)
    did = (kat_pref[:, kat] * sub_pref[:, kat, sub % n_sub]
           * narx_pref[:, narx])                           # (user, item)
    did /= did.sum(1, keepdims=True)
    hafta = np.arange(0, kunlar, 7)
    yosh = np.clip((hafta[:, None] - chiqish[None, :]) / 7, 0, None)
    jalb = sifat * (0.6 + 1.2 * np.exp(-yosh / 6)) * (hafta[:, None] >= chiqish)
    qatorlar = []
    for u in range(n_user):
        n_u = int(np.clip(rng.lognormal(2.8, 0.8), 3, 150))
        vaqtlar = np.sort(rng.uniform(0, kunlar, n_u))
        korgan = np.zeros(n_item, dtype=bool)
        for t in vaqtlar:
            a = jalb[int(t) // 7] * ~korgan
            p = 0.75 * did[u] * a / (did[u] @ a) + 0.25 * a / a.sum()
            i = int(np.searchsorted(np.cumsum(p), rng.random() * p.sum()))
            korgan[i] = True
            mos = did[u, i] / did[u].max()
            xarid = rng.random() < 0.15 + 0.35 * mos
            baho = 0
            if xarid and rng.random() < 0.3:
                baho = int(np.clip(round(2.4 + 2.2 * mos + 0.3 * np.log(sifat[i])
                                         + rng.normal(0, 0.7)), 1, 5))
            qatorlar.append((u, i, t, int(xarid), baho))
    df = pd.DataFrame(qatorlar, columns=["user", "item", "vaqt", "xarid", "baho"])
    items = pd.DataFrame({"item": np.arange(n_item), "kategoriya": np.array(katlar)[kat],
                          "sub": sub, "narx": narx, "chiqish": chiqish,
                          "tavsif": tavsif})
    return df, items


def bolish(df, usul="vaqt", ulush=0.2, seed=0, chegara=150.0):
    """'vaqt' - har foydalanuvchining oxirgi 20% i test; 'tasodifiy'; 'global'."""
    df = df.sort_values(["user", "vaqt"]).reset_index(drop=True)
    if usul == "global":
        tarix = df[df["vaqt"] < chegara].groupby("user").size()
        yetarli = df["user"].map(tarix).fillna(0).to_numpy() >= 3
        test = (df["vaqt"] >= chegara).to_numpy() & yetarli
        return df[df["vaqt"] < chegara].copy(), df[test].copy()
    n = df.groupby("user")["item"].transform("size").to_numpy()
    if usul == "tasodifiy":
        kalit = np.random.default_rng(seed).random(len(df))
        tartib = (df.assign(k=kalit).groupby("user")["k"]
                  .rank(method="first").to_numpy() - 1)
    else:
        tartib = df.groupby("user").cumcount().to_numpy()
    n_test = np.maximum(1, np.round(ulush * n)).astype(int)
    test = (tartib >= n - n_test) & (n >= 5)
    return df[~test].copy(), df[test].copy()


def matritsa(qism, n_user, n_item):
    return sparse.csr_matrix((np.ones(len(qism)), (qism["user"], qism["item"])),
                             shape=(n_user, n_item))


def metrikalar(top, T):
    """top: (n, k) tavsiya indekslari; T: (n, n_item) test to'plami (bool)."""
    k = top.shape[1]
    rel = np.take_along_axis(T, top, axis=1)
    n_rel = T.sum(1)
    pozitsiya = np.arange(1, k + 1)
    ap = (np.cumsum(rel, 1) / pozitsiya * rel).sum(1) / np.minimum(n_rel, k)
    chegirma = 1 / np.log2(pozitsiya + 1)
    idcg = np.array([chegirma[:min(r, k)].sum() for r in n_rel])
    return {"precision": rel.sum(1) / k, "recall": rel.sum(1) / n_rel,
            "hit": rel.any(1).astype(float), "MAP": ap,
            "nDCG": (rel * chegirma).sum(1) / idcg}


def baholash(ball, X_train, test, users, k=10):
    """Train da ko'rilgan mahsulotlar tavsiya qilinmaydi; har foydalanuvchi metrikasi."""
    ball = np.array(ball, dtype=float)
    ball[X_train[users].toarray() > 0] = -np.inf
    top = np.argsort(-ball, axis=1, kind="stable")[:, :k]
    qator = pd.Series(np.arange(len(users)), index=users)
    tt = test[test["user"].isin(users)]
    T = np.zeros(ball.shape, dtype=bool)
    T[qator[tt["user"]].to_numpy(), tt["item"].to_numpy()] = True
    return metrikalar(top, T), top


def kosinus_ustun(X):
    """Mahsulotlar orasidagi kosinus o'xshashlik: ustunlarni normallab, X^T X."""
    norma = np.sqrt(np.asarray(X.multiply(X).sum(0)).ravel())
    Xn = X @ sparse.diags(1 / np.maximum(norma, 1e-12))
    S = (Xn.T @ Xn).toarray()
    np.fill_diagonal(S, 0)
    return S


def eng_yaqin(S, k):
    """Har qatorda faqat eng katta k ta qiymat qoladi."""
    S = S.copy()
    kesish = np.argpartition(-S, k, axis=1)[:, k:]
    np.put_along_axis(S, kesish, 0, axis=1)
    return S


def main() -> None:
    df, items = dokon()
    n_user, n_item = 2000, len(items)
    tr, te = bolish(df, "vaqt")
    X = matritsa(tr, n_user, n_item)
    users = np.sort(te["user"].unique())
    Xu = X[users]

    print("=== 1. Kosinus o'xshashlik noldan (siyrak matritsada) ===")
    S_item = kosinus_ustun(X)
    tekshir = cosine_similarity(X.T[:5])
    np.fill_diagonal(tekshir, 0)
    print(f"  noldan va sklearn bir xilmi (5x5, 1e-12 aniqlikda): "
          f"{np.allclose(S_item[:5, :5], tekshir, atol=1e-12)}")
    print(f"  mahsulot-mahsulot matritsa {S_item.shape}, nolmas ulushi "
          f"{np.mean(S_item > 0):.1%}")
    j = int(np.argmax(np.asarray(X.sum(0)).ravel()))
    qoshni = np.argsort(-S_item[j])[:4]
    print(f"  eng mashhur mahsulot: '{items.at[j, 'tavsif']}'")
    for q in qoshni:
        print(f"    o'xshash {S_item[j, q]:.3f}: '{items.at[q, 'tavsif']}'")

    print("\n=== 2. Usullar ('vaqt' bo'lishi, k=10) ===")
    pop = np.asarray(X.sum(0)).ravel()
    tfidf = TfidfVectorizer().fit_transform(items["tavsif"])
    profil = Xu @ tfidf
    kontent = cosine_similarity(profil, tfidf)
    item_knn = (Xu @ eng_yaqin(S_item, 50)).astype(float)
    Xr = sparse.diags(1 / np.sqrt(np.asarray(X.sum(1)).ravel() + 1e-12)) @ X
    S_user = (Xr[users] @ Xr.T).toarray()
    S_user[np.arange(len(users)), users] = 0          # o'zi bilan o'xshashlik
    user_knn = eng_yaqin(S_user, 50) @ X.toarray()
    usullar = {"tasodifiy": np.random.default_rng(1).random((len(users), n_item)),
               "mashhurlik": np.tile(pop, (len(users), 1)),
               "kontent (TF-IDF)": kontent, "item-kNN": item_knn,
               "user-kNN": user_knn}
    natija = {}
    print(f"  {'usul':<18}" + "".join(f"{k:>10}" for k in
                                     ["precision", "recall", "hit", "MAP", "nDCG"]))
    for nom, b in usullar.items():
        natija[nom], _ = baholash(b, X, te, users)
        print(f"  {nom:<18}" + "".join(f"{v.mean():>10.4f}"
                                        for v in natija[nom].values()))

    print("\n=== 3. Juftlashgan taqqoslash va qaror (nDCG@10) ===")
    nomzod = ["mashhurlik", "kontent (TF-IDF)", "item-kNN", "user-kNN"]  # soddadan
    eng = max(nomzod, key=lambda n: natija[n]["nDCG"].mean())
    print(f"  eng yaxshi: {eng}")
    tanlov = None
    for n in nomzod:
        if n == eng:
            tanlov = tanlov or n
            continue
        d = natija[n]["nDCG"] - natija[eng]["nDCG"]
        se = d.std(ddof=1) / np.sqrt(len(d))
        yomon = d.mean() < -2 * se
        print(f"  {n:<18} - {eng}: {d.mean():+.4f}, SE {se:.4f}, "
              f"sezilarli yomon: {yomon}")
        if tanlov is None and not yomon:
            tanlov = n
    print(f"  qaror (eng yaxshisidan sezilarli yomon bo'lmagan eng sodda): {tanlov}")

    print("\n=== 4. Faollik bo'yicha: tarix qisqa bo'lsa ===")
    tarix = np.asarray(Xu.sum(1)).ravel()
    guruhlar = [(0, 6, "1-5"), (6, 16, "6-15"), (16, 10_000, "16+")]
    print(f"  {'train tarixi':<14} {'foyd.':>6}" + "".join(
        f"{n:>18}" for n in ["mashhurlik", "kontent (TF-IDF)", "item-kNN"]))
    for a, b, nom in guruhlar:
        m = (tarix >= a) & (tarix < b)
        print(f"  {nom:<14} {m.sum():>6}" + "".join(
            f"{natija[n]['nDCG'][m].mean():>18.4f}"
            for n in ["mashhurlik", "kontent (TF-IDF)", "item-kNN"]))

    print("\n=== 5. Tushuntirish: nega shu tavsiya? (item-kNN) ===")
    u = int(np.argmax(tarix == 8))
    tarixi = Xu[u].indices
    _, top = baholash(item_knn[[u]], X, te, users[[u]], k=3)
    print(f"  foydalanuvchi {users[u]}, tarix kategoriyalari: "
          f"{sorted(items.loc[tarixi, 'kategoriya'].value_counts().to_dict().items())}")
    for i in top[0]:
        sabab = tarixi[np.argmax(S_item[i, tarixi])]
        print(f"   tavsiya '{items.at[i, 'tavsif'][:34]}'  <- "
              f"'{items.at[sabab, 'tavsif'][:34]}' ({S_item[i, sabab]:.2f})")
    print("  ⭐ Item-kNN ni tushuntirish oson: 'siz X ni olgansiz'")


if __name__ == "__main__":
    main()

=== 1. Kosinus o'xshashlik noldan (siyrak matritsada) ===
  noldan va sklearn bir xilmi (5x5, 1e-12 aniqlikda): True
  mahsulot-mahsulot matritsa (800, 800), nolmas ulushi 45.8%
  eng mashhur mahsulot: 'asal qahva ziravor oziq arzon brend34 kafolat sifatli'
    o'xshash 0.385: 'qahva asal ziravor oziq arzon brend20 sifatli original'
    o'xshash 0.337: 'qahva asal yog oziq orta brend24 sifatli mashhur'
    o'xshash 0.336: 'shim kurtka sviter kiyim arzon brend22 original qulay'
    o'xshash 0.330: 'makaron un qahva oziq orta brend30 tez yangi'

=== 2. Usullar ('vaqt' bo'lishi, k=10) ===
  usul               precision    recall       hit       MAP      nDCG
  tasodifiy             0.0058    0.0120    0.0536    0.0041    0.0094
  mashhurlik            0.0383    0.0924    0.3203    0.0351    0.0706
  kontent (TF-IDF)      0.0335    0.0643    0.2591    0.0236    0.0524
  item-kNN              0.0508    0.1234    0.3954    0.0501    0.0961
  user-kNN              0.0569    0.1288    0.4211    0.0565    0.1070

=== 3. Juftlashgan taqqoslash va qaror (nDCG@10) ===
  eng yaxshi: user-kNN
  mashhurlik         - user-kNN: -0.0364, SE 0.0036, sezilarli yomon: True
  kontent (TF-IDF)   - user-kNN: -0.0546, SE 0.0042, sezilarli yomon: True
  item-kNN           - user-kNN: -0.0108, SE 0.0028, sezilarli yomon: True
  qaror (eng yaxshisidan sezilarli yomon bo'lmagan eng sodda): user-kNN

=== 4. Faollik bo'yicha: tarix qisqa bo'lsa ===
  train tarixi    foyd.        mashhurlik  kontent (TF-IDF)          item-kNN
  1-5               149            0.0527            0.0124            0.0782
  6-15              916            0.0678            0.0407            0.0940
  16+               799            0.0771            0.0732            0.1019

=== 5. Tushuntirish: nega shu tavsiya? (item-kNN) ===
  foydalanuvchi 5, tarix kategoriyalari: [('bolalar', 1), ('gozallik', 2), ('kitob', 1), ('kiyim', 1), ('sport', 2), ('uy', 1)]
   tavsiya 'losyon sovun niqob gozallik orta b'  <- 'sovun losyon taroq gozallik orta b' 0.45-bob
   tavsiya 'shim kurtka sviter kiyim arzon bre'  <- 'krossovka shapka palto kiyim premi' 0.29-bob
   tavsiya 'suzish butsa gantel sport arzon br'  <- 'krossovka shapka palto kiyim premi' 0.24-bob
  ⭐ Item-kNN ni tushuntirish oson: 'siz X ni olgansiz'

"""Aniqlikdan tashqari: qamrov, xilma-xillik, yangilik va mashhurlik tarafkashligi."""

import numpy as np
import pandas as pd
from scipy import sparse
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

KATEGORIYA = {
    "elektronika": "telefon smartfon quloqchin zaryadlovchi noutbuk planshet "
                   "sichqoncha klaviatura kolonka televizor kamera soat",
    "kiyim": "koylak shim kurtka poyabzal krossovka shapka sharf palto "
             "kostyum futbolka yubka sviter",
    "kitob": "roman darslik detektiv sher tarix fantastika lugat ertak "
             "biografiya psixologiya biznes dasturlash",
    "oziq": "choy qahva asal yongoq shokolad guruch yog un makaron sharbat "
            "pechene ziravor",
    "gozallik": "krem atir shampun labbuyoq boyoq niqob sovun losyon taroq "
                "lak gel upa",
    "uy": "idish qozon choynak gilam parda yostiq chiroq sochiq pichoq tova "
          "vaza javon",
    "sport": "koptok gantel velosiped raketka forma gilamcha skakalka shlem "
             "sumka butsa trenajyor suzish",
    "bolalar": "oyinchoq konstruktor qogirchoq kolyaska taglik mashinka pazl "
               "albom rangli kiyimcha kitobcha samokat",
}
NARX = ["arzon", "orta", "premium"]
UMUMIY = "sifatli yangi chegirma original mashhur qulay kafolat tez yetkazish".split()


def dokon(seed=0, n_user=2000, n_item=800, kunlar=180):
    """Sintetik onlayn do'kon: yashirin didlar, mashhurlik dumi, yangi mahsulotlar."""
    rng = np.random.default_rng(seed)
    katlar = list(KATEGORIYA)
    n_kat, n_sub = len(katlar), 4
    sub = rng.integers(0, n_kat * n_sub, n_item)          # har mahsulot kichik guruhi
    kat = sub // n_sub
    narx = rng.choice(3, n_item, p=[0.5, 0.35, 0.15])
    sifat = rng.lognormal(0, 1.4, n_item)                  # mashhurlik dumi manbai
    chiqish = np.where(rng.random(n_item) < 0.7, 0,
                       rng.integers(0, kunlar - 20, n_item))
    tavsif = []
    for i in range(n_item):
        sozlar = KATEGORIYA[katlar[kat[i]]].split()
        guruh = sozlar[(sub[i] % n_sub) * 3:(sub[i] % n_sub) * 3 + 3]
        tanlov = list(rng.choice(guruh, 2, replace=False))
        tanlov.append(rng.choice([s for s in sozlar if s not in tanlov]))
        tanlov += [katlar[kat[i]], NARX[narx[i]], f"brend{rng.integers(0, 40)}"]
        tanlov += list(rng.choice(UMUMIY, 2, replace=False))
        tavsif.append(" ".join(tanlov))
    # foydalanuvchi didi: kategoriya x kichik guruh x narx
    kat_pref = rng.dirichlet(np.full(n_kat, 0.3), n_user)
    sub_pref = rng.dirichlet(np.full(n_sub, 0.5), (n_user, n_kat))
    narx_pref = rng.dirichlet(np.full(3, 1.0), n_user)
    did = (kat_pref[:, kat] * sub_pref[:, kat, sub % n_sub]
           * narx_pref[:, narx])                           # (user, item)
    did /= did.sum(1, keepdims=True)
    hafta = np.arange(0, kunlar, 7)
    yosh = np.clip((hafta[:, None] - chiqish[None, :]) / 7, 0, None)
    jalb = sifat * (0.6 + 1.2 * np.exp(-yosh / 6)) * (hafta[:, None] >= chiqish)
    qatorlar = []
    for u in range(n_user):
        n_u = int(np.clip(rng.lognormal(2.8, 0.8), 3, 150))
        vaqtlar = np.sort(rng.uniform(0, kunlar, n_u))
        korgan = np.zeros(n_item, dtype=bool)
        for t in vaqtlar:
            a = jalb[int(t) // 7] * ~korgan
            p = 0.75 * did[u] * a / (did[u] @ a) + 0.25 * a / a.sum()
            i = int(np.searchsorted(np.cumsum(p), rng.random() * p.sum()))
            korgan[i] = True
            mos = did[u, i] / did[u].max()
            xarid = rng.random() < 0.15 + 0.35 * mos
            baho = 0
            if xarid and rng.random() < 0.3:
                baho = int(np.clip(round(2.4 + 2.2 * mos + 0.3 * np.log(sifat[i])
                                         + rng.normal(0, 0.7)), 1, 5))
            qatorlar.append((u, i, t, int(xarid), baho))
    df = pd.DataFrame(qatorlar, columns=["user", "item", "vaqt", "xarid", "baho"])
    items = pd.DataFrame({"item": np.arange(n_item), "kategoriya": np.array(katlar)[kat],
                          "sub": sub, "narx": narx, "chiqish": chiqish,
                          "tavsif": tavsif})
    return df, items


def bolish(df, usul="vaqt", ulush=0.2, seed=0, chegara=150.0):
    """'vaqt' - har foydalanuvchining oxirgi 20% i test; 'tasodifiy'; 'global'."""
    df = df.sort_values(["user", "vaqt"]).reset_index(drop=True)
    if usul == "global":
        tarix = df[df["vaqt"] < chegara].groupby("user").size()
        yetarli = df["user"].map(tarix).fillna(0).to_numpy() >= 3
        test = (df["vaqt"] >= chegara).to_numpy() & yetarli
        return df[df["vaqt"] < chegara].copy(), df[test].copy()
    n = df.groupby("user")["item"].transform("size").to_numpy()
    if usul == "tasodifiy":
        kalit = np.random.default_rng(seed).random(len(df))
        tartib = (df.assign(k=kalit).groupby("user")["k"]
                  .rank(method="first").to_numpy() - 1)
    else:
        tartib = df.groupby("user").cumcount().to_numpy()
    n_test = np.maximum(1, np.round(ulush * n)).astype(int)
    test = (tartib >= n - n_test) & (n >= 5)
    return df[~test].copy(), df[test].copy()


def matritsa(qism, n_user, n_item):
    return sparse.csr_matrix((np.ones(len(qism)), (qism["user"], qism["item"])),
                             shape=(n_user, n_item))


def metrikalar(top, T):
    """top: (n, k) tavsiya indekslari; T: (n, n_item) test to'plami (bool)."""
    k = top.shape[1]
    rel = np.take_along_axis(T, top, axis=1)
    n_rel = T.sum(1)
    pozitsiya = np.arange(1, k + 1)
    ap = (np.cumsum(rel, 1) / pozitsiya * rel).sum(1) / np.minimum(n_rel, k)
    chegirma = 1 / np.log2(pozitsiya + 1)
    idcg = np.array([chegirma[:min(r, k)].sum() for r in n_rel])
    return {"precision": rel.sum(1) / k, "recall": rel.sum(1) / n_rel,
            "hit": rel.any(1).astype(float), "MAP": ap,
            "nDCG": (rel * chegirma).sum(1) / idcg}


def baholash(ball, X_train, test, users, k=10):
    """Train da ko'rilgan mahsulotlar tavsiya qilinmaydi; har foydalanuvchi metrikasi."""
    ball = np.array(ball, dtype=float)
    ball[X_train[users].toarray() > 0] = -np.inf
    top = np.argsort(-ball, axis=1, kind="stable")[:, :k]
    qator = pd.Series(np.arange(len(users)), index=users)
    tt = test[test["user"].isin(users)]
    T = np.zeros(ball.shape, dtype=bool)
    T[qator[tt["user"]].to_numpy(), tt["item"].to_numpy()] = True
    return metrikalar(top, T), top


def user_knn(X, users, k=50):
    Xr = sparse.diags(1 / np.sqrt(np.asarray(X.sum(1)).ravel() + 1e-12)) @ X
    S = (Xr[users] @ Xr.T).toarray()
    S[np.arange(len(users)), users] = 0
    kesish = np.argpartition(-S, k, axis=1)[:, k:]
    np.put_along_axis(S, kesish, 0, axis=1)
    return S @ X.toarray()


def item_knn(X, users, k=50):
    norma = np.sqrt(np.asarray(X.sum(0)).ravel())
    Xn = X @ sparse.diags(1 / np.maximum(norma, 1e-12))
    S = (Xn.T @ Xn).toarray()
    np.fill_diagonal(S, 0)
    kesish = np.argpartition(-S, k, axis=1)[:, k:]
    np.put_along_axis(S, kesish, 0, axis=1)
    return X[users] @ S


def tavsifla(top, pop, kat, n_train_user):
    """Qamrov, ro'yxat ichidagi xilma-xillik, yangilik, mashhurlar ulushi."""
    k = top.shape[1]
    qamrov = len(np.unique(top)) / len(pop)
    turli = (kat[top][:, :, None] != kat[top][:, None, :]).sum((1, 2)) / (k * (k - 1))
    yangilik = -np.log2((pop[top] + 1) / n_train_user).mean()
    bosh = pop >= np.quantile(pop, 0.9)
    return qamrov, turli.mean(), yangilik, bosh[top].mean()


def qayta_ranjir(ball, pop, beta):
    """ball ni foydalanuvchi ichida [0, 1] ga keltirib, mashhurlik uchun jarima."""
    b = ball / np.maximum(ball.max(1, keepdims=True), 1e-12)
    p = np.log1p(pop) / np.log1p(pop).max()
    return b - beta * p


def main() -> None:
    df, items = dokon()
    n_user, n_item = 2000, len(items)
    kat = pd.factorize(items["kategoriya"])[0]
    tr, te = bolish(df, "vaqt")
    X = matritsa(tr, n_user, n_item)
    users = np.sort(te["user"].unique())
    pop = np.asarray(X.sum(0)).ravel()
    bosh = pop >= np.quantile(pop, 0.9)

    print("=== 1. Test ta'sirlarining o'zi qanchalik 'mashhur'? ===")
    print(f"  train: eng mashhur 10% mahsulot ta'sirlarning "
          f"{pop[bosh].sum() / pop.sum():.1%} ini oladi")
    print(f"  test:  shu mahsulotlar test ta'sirlarining "
          f"{bosh[te['item']].mean():.1%} i")

    print("\n=== 2. Aniqlikdan tashqari metrikalar (k=10) ===")
    tfidf = TfidfVectorizer().fit_transform(items["tavsif"])
    usullar = {"tasodifiy": np.random.default_rng(1).random((len(users), n_item)),
               "mashhurlik": np.tile(pop, (len(users), 1)).astype(float),
               "kontent (TF-IDF)": cosine_similarity(X[users] @ tfidf, tfidf),
               "item-kNN": item_knn(X, users), "user-kNN": user_knn(X, users)}
    print(f"  {'usul':<18} {'nDCG':>7} {'qamrov':>7} {'xilma':>6} {'yangilik':>9} "
          f"{'top10% ulushi':>14}")
    natija = {}
    for nom, b in usullar.items():
        natija[nom], top = baholash(b, X, te, users)
        q, x, y, ul = tavsifla(top, pop, kat, n_user)
        print(f"  {nom:<18} {natija[nom]['nDCG'].mean():>7.4f} {q:>7.1%} {x:>6.3f} "
              f"{y:>9.2f} {ul:>14.1%}")
    print(f"  (test ta'sirlarida top10% ulushi: {bosh[te['item']].mean():.1%})")

    print("\n=== 3. Qaysi mahsulotlar topiladi: bosh va dum ===")
    dum_te = te[~bosh[te["item"]]]
    bosh_te = te[bosh[te["item"]]]
    print(f"  {'usul':<18} {'bosh recall':>12} {'dum recall':>11}")
    ub, ud = np.sort(bosh_te["user"].unique()), np.sort(dum_te["user"].unique())
    ib, id_ = np.searchsorted(users, ub), np.searchsorted(users, ud)
    for nom in ["mashhurlik", "kontent (TF-IDF)", "item-kNN", "user-kNN"]:
        r_b, _ = baholash(usullar[nom][ib], X, bosh_te, ub)
        r_d, _ = baholash(usullar[nom][id_], X, dum_te, ud)
        print(f"  {nom:<18} {r_b['recall'].mean():>12.4f} {r_d['recall'].mean():>11.4f}")

    print("\n=== 4. Mashhurlik jarimasi: beta VALIDATSIYADA tanlanadi ===")
    tr2, va = bolish(tr, "vaqt")
    X2 = matritsa(tr2, n_user, n_item)
    u2 = np.sort(va["user"].unique())
    pop2 = np.asarray(X2.sum(0)).ravel()
    b2 = user_knn(X2, u2)
    asos, _ = baholash(qayta_ranjir(b2, pop2, 0.0), X2, va, u2)
    tanlov = 0.0
    print(f"  {'beta':>5} {'val nDCG':>9} {'farq':>8} {'SE':>7} {'qamrov':>7} "
          f"{'top10%':>7}")
    for beta in [0.0, 0.1, 0.2, 0.4, 0.8]:
        r, top = baholash(qayta_ranjir(b2, pop2, beta), X2, va, u2)
        d = r["nDCG"] - asos["nDCG"]
        se = d.std(ddof=1) / np.sqrt(len(d))
        q, _, _, ul = tavsifla(top, pop2, kat, n_user)
        print(f"  {beta:>5.1f} {r['nDCG'].mean():>9.4f} {d.mean():>+8.4f} {se:>7.4f} "
              f"{q:>7.1%} {ul:>7.1%}")
        if d.mean() >= -2 * se:
            tanlov = beta
    print(f"  tanlov: nDCG sezilarli tushmagan eng katta beta = {tanlov}")
    test = {}
    for beta in [0.0, tanlov]:
        r, top = baholash(qayta_ranjir(usullar["user-kNN"], pop, beta), X, te, users)
        q, _, _, ul = tavsifla(top, pop, kat, n_user)
        test[beta] = r["nDCG"]
        print(f"  TEST beta={beta}: nDCG {r['nDCG'].mean():.4f}, qamrov {q:.1%}, "
              f"top10% ulushi {ul:.1%}")
    d = test[tanlov] - test[0.0]
    print(f"  TEST juftlashgan farq: {d.mean():+.4f}, "
          f"SE {d.std(ddof=1) / np.sqrt(len(d)):.4f}")
    print("  ⭐ Aniqlik yagona maqsad emas: qamrov va dum ham o'lchanadi")


if __name__ == "__main__":
    main()

=== 1. Test ta'sirlarining o'zi qanchalik 'mashhur'? ===
  train: eng mashhur 10% mahsulot ta'sirlarning 43.7% ini oladi
  test:  shu mahsulotlar test ta'sirlarining 32.0% i

=== 2. Aniqlikdan tashqari metrikalar (k=10) ===
  usul                  nDCG  qamrov  xilma  yangilik  top10% ulushi
  tasodifiy           0.0094  100.0%  0.875      6.59           9.4%
  mashhurlik          0.0706    3.8%  0.829      2.54         100.0%
  kontent (TF-IDF)    0.0524   99.5%  0.168      6.93           5.7%
  item-kNN            0.0961   31.6%  0.818      2.82          95.8%
  user-kNN            0.1070   31.6%  0.749      3.08          90.4%
  (test ta'sirlarida top10% ulushi: 32.0%)

=== 3. Qaysi mahsulotlar topiladi: bosh va dum ===
  usul                bosh recall  dum recall
  mashhurlik               0.2517      0.0000
  kontent (TF-IDF)         0.0357      0.0791
  item-kNN                 0.3148      0.0130
  user-kNN                 0.3170      0.0263

=== 4. Mashhurlik jarimasi: beta VALIDATSIYADA tanlanadi ===
   beta  val nDCG     farq      SE  qamrov  top10%
    0.0    0.0944  +0.0000  0.0000   31.5%   90.6%
    0.1    0.0955  +0.0011  0.0008   41.5%   86.1%
    0.2    0.0947  +0.0003  0.0012   63.4%   73.5%
    0.4    0.0511  -0.0433  0.0029   45.5%   22.3%
    0.8    0.0019  -0.0925  0.0039    4.5%    0.3%
  tanlov: nDCG sezilarli tushmagan eng katta beta = 0.2
  TEST beta=0.0: nDCG 0.1070, qamrov 31.6%, top10% ulushi 90.4%
  TEST beta=0.2: nDCG 0.1070, qamrov 52.1%, top10% ulushi 79.4%
  TEST juftlashgan farq: +0.0000, SE 0.0010
  ⭐ Aniqlik yagona maqsad emas: qamrov va dum ham o'lchanadi

test = df.sample(frac=0.2, random_state=0)                           # ⚠️
df = df.sort_values(["user", "vaqt"])                                # ✅ vaqt bo'yicha
test = df[df.groupby("user").cumcount() >= n_u - n_test]

top = np.argsort(-ball, axis=1)[:, :10]                              # ⚠️
ball[X_train[users].toarray() > 0] = -np.inf                         # ✅
top = np.argsort(-ball, axis=1, kind="stable")[:, :10]

pop = df["item"].value_counts()                                      # ⚠️ butun jurnal
pop = train["item"].value_counts()                                   # ✅ faqat train

top = g.sort_values("mean").tail(10)                                 # ⚠️
g["silliq"] = (g["count"] * g["mean"] + m * mu) / (g["count"] + m)   # ✅

print(ndcg.mean())                                                   # ⚠️ yolg'iz
print(ndcg.mean(), qamrov, bosh_ulushi)                              # ✅

if ndcg_A.mean() > ndcg_B.mean(): tanla(A)                           # ⚠️
d = ndcg_A - ndcg_B                                                  # ✅ bir xil foydalanuvchilar
sezilarli = abs(d.mean()) > 2 * d.std(ddof=1) / np.sqrt(len(d))

beta = max(betalar, key=lambda b: ndcg_test(b))                      # ⚠️
tr2, va = bolish(train, "vaqt")                                      # ✅ train ichidan validatsiya

1.  # implicit ma'lumotda bo'sh katak nimani bildiradi?
2.  # top = [4, 2, 8], T = {2} - precision@3, recall@3, hit@3?
3.  # xuddi shu misolda nDCG@3?
4.  # T = {4, 8}, top = [4, 2, 8] - AP@3?
5.  # mashhurlik ro'yxatining qamrovi (800 mahsulot, k=10)?
6.  # tasodifiy bo'lish odatda natijani qanday o'zgartiradi?
7.  # item-kNN da ikkita juda mashhur mahsulot nega "o'xshash" chiqadi?
8.  # kontent-asosli usulning xilma-xilligi nega past?
9.  # user-kNN ni yangi foydalanuvchiga qo'llash mumkinmi?
10. # 5 ta baho bilan 4.9 va 300 ta baho bilan 4.6 - qaysi biri yuqori?
11. # SE qaysi birlik bo'yicha olinadi?
12. # mashhurlik jarimasi beta oshsa, qamrov va nDCG?

1.  train, test = train_test_split(df, test_size=0.2)

2.  pop = df["item"].value_counts().index[:10]

3.  top = np.argsort(-ball, axis=1)[:, :10]      # train mahsulotlari ham bor

4.  eng_yaxshi = g.sort_values("mean").tail(5)   # 2-3 ta baho bilan

5.  print("item-kNN yaxshi", ndcg_knn.mean() > ndcg_pop.mean())