28.5-dars: Matritsa faktorizatsiyasi va implicit feedback

      R (n_user x n_item)      ~      P (n_user x k)   x   Q^T (k x n_item)

   i1 i2 i3 i4 i5 i6               f1   f2                 i1  i2  i3  i4  i5  i6
u1  5  -  4  -  -  1         u1  [ 1.2  -0.3 ]       f1 [ 1.0 0.9 1.1 -0.2 0.1 -0.9 ]
u2  -  4  -  -  2  -    ~    u2  [ 0.9   0.2 ]   x   f2 [ 0.1 0.3 -0.1  1.2 0.8  0.2 ]
u3  1  -  -  5  4  -         u3  [-0.5   1.4 ]

r_hat(u, i) = p_u . q_i = sum_f p_uf * q_if

f1, f2 - NOMSIZ o'qlar; ma'lumotdan o'zi topiladi
   masalan f1 ~ "shirinlik va ichimlik", f2 ~ "sport va faol hayot"
k << n_user, n_item   (odatda 10-200)
parametrlar: (n_user + n_item) * k  -  n_user * n_item o'rniga

TO'LIQ SVD:  R = U S V^T  -  faqat TO'LIQ matritsa uchun aniqlangan
   bo'sh kataklarni nima bilan to'ldiramiz?

VARIANT 1: bo'sh = 0 (TruncatedSVD siyrak matritsaga shunday qaraydi)
   explicit baholarda: "baho 0" - 1..5 shkalada mavjud bo'lmagan qiymat
   97% katak nol -> eng yaxshi past rangli yaqinlashish HAM ~0 ga yaqin
   test bashoratlari ~0.5, RMSE ~3 (1-misol)

VARIANT 2: o'rtacha bilan to'ldirish (yoki biaslarni ayirib, qoldiqni 0 deb olish)
   yaxshiroq, lekin baribir to'ldirilgan qiymatlarni HAQIQAT deb o'rganadi
   97% "soxta" ma'lumot 3% haqiqiy ma'lumotni bosib ketadi

TO'G'RI YO'L (explicit): yo'qotishni FAQAT kuzatilgan kataklar ustida
   min sum_{(u,i) kuzatilgan} (r_ui - r_hat_ui)^2 + regularizatsiya
   -> bu endi SVD emas, gradient tushish yoki ALS bilan yechiladigan
      optimallashtirish (Netflix Prize dagi "Funk SVD")

MODEL:
   r_hat(u, i) = mu + b_u + b_i + p_u . q_i
   mu   - global o'rtacha baho
   b_u  - foydalanuvchi biasi (qattiqqo'l -0.4, saxiy +0.5)
   b_i  - mahsulot biasi (sifatli kitob +0.6)
   p_u . q_i - DID bo'yicha moslik (biaslardan tashqari qism)

YO'QOTISH (faqat kuzatilgan kataklar):
   L = sum (r_ui - r_hat_ui)^2 + reg * (b_u^2 + b_i^2 + |p_u|^2 + |q_i|^2)

SGD YANGILASH (har baho yoki mini-partiya uchun):
   e    = r_ui - r_hat_ui
   b_u += lr * (e - reg * b_u)
   b_i += lr * (e - reg * b_i)
   p_u += lr * (e * q_i - reg * p_u)
   q_i += lr * (e * p_u - reg * q_i)       (eski p_u bilan)

BAHOLASH: RMSE = sqrt(mean (r - r_hat)^2)  - test baholarida

AFZALLIK (preference):  p_ui = 1, agar r_ui > 0 (ko'rgan/olgan), aks holda 0
ISHONCH  (confidence):  c_ui = 1 + alfa * r_ui
   r_ui = signal kuchi (bizda: ko'rish = 1, xarid = 3)
   bo'sh katak: p = 0, c = 1   -> "ehtimol yoqmaydi", lekin ISHONCH PAST
   xarid:       p = 1, c = 1 + 3*alfa -> "yoqadi", ishonch yuqori

YO'QOTISH - HAMMA n_user x n_item katak ustida:
   L = sum_{u,i} c_ui (p_ui - x_u . y_i)^2 + lambda (sum |x_u|^2 + sum |y_i|^2)

NEGA HAMMA KATAK:
   faqat kuzatilganlar (hammasi p = 1) -> trivial yechim: hamma
   ko'paytmani 1 qilish; model "hamma narsa yoqadi" deydi
   nollar "kuchsiz salbiy" misol sifatida ranjirlashga yo'nalish beradi

x_u = (Y^T C_u Y + lambda I)^(-1)  Y^T C_u p_u          C_u = diag(c_u1 .. c_uN)

HIYLA (tezlik):   Y^T C_u Y = Y^T Y  +  Y^T (C_u - I) Y
   Y^T Y             - HAMMA foydalanuvchi uchun bitta, bir marta hisoblanadi
                       (nollarni c = 1 bilan qamraydi)
   Y^T (C_u - I) Y   - faqat u ko'rgan n_u ta mahsulot ustida (c - 1 = 0 qolganida)
   Y^T C_u p_u       - ham faqat ko'rilganlar ustida (p = 0 qolganida)

   narx: O(n_u * k^2 + k^3) har foydalanuvchiga  (O(n_item * k^2) o'rniga)
   hamma katak ishtirok etadi, lekin hisob faqat nol bo'lmaganlar ustida

G'OYA (Rendle va boshq., 2009): biz aniq qiymatni emas, TARTIBNI xohlaymiz
   foydalanuvchi u ko'rgan i ni ko'rmagan j dan YUQORI qo'ysin

   har qadamda uchlik (u, i+, j-):  i - ko'rgan, j - tasodifiy (ko'rmagan)
   x_uij = x_ui - x_uj = p_u . (q_i - q_j) + (b_i - b_j)
   L = - sum log sigmoid(x_uij) + reg * |parametrlar|^2

   bu AUC ning silliq yaqinlashuvi: "tasodifiy juftlikda to'g'ri tartib"

AFZALLIGI: to'g'ridan-to'g'ri ranjirlashni optimallashtiradi, SGD/Adam
           bilan har qanday model (neyron tarmoq ham) ga qo'llanadi
MUAMMOSI:  salbiy namuna olish (negative sampling) - tasodifiy j lar
           ko'pincha "oson" (mashhur emas) -> model asosan mashhurlikni
           o'rganadi; sozlash sezgir (lr, reg, epoxalar)

k (faktorlar soni):   kichik -> "qo'pol" didlar;  katta -> yodlash, sekin
lambda (regularizatsiya): kichik -> overfitting;  katta -> hamma mashhurlikka yaqin
alfa (ishonch):       0 -> hamma katak teng;  katta -> ko'rilganlarni yodlash

TANLASH:
   train ICHIDAN validatsiya: xuddi test kabi bo'lish (har foydalanuvchining
   oxirgi 20% i) - train2 / val
   setka yoki navbatma-navbat qidiruv val nDCG@10 bo'yicha
   QAROR: eng yaxshisidan sezilarli yomon bo'lmagan ENG SODDA
          (kichik k, katta lambda) - juftlashgan SE bilan
   tanlangan sozlama BUTUN train da qayta o'qitiladi, test - bir marta

YANGI FOYDALANUVCHI (tarixi yo'q yoki 1-2 ta):
   CF uchun vektor yo'q -> ball 0, tavsiya tasodifiy
   FALLBACK: mashhurlik (vaqt oynasi, kategoriya, mintaqa bo'yicha)
   FOLD-IN: model qayta o'qitilmaydi - Y qotirilgan holda faqat x_u
            yechiladi (bitta ridge yechim, millisekundlar)
   ONBOARDING: "3 ta qiziqishingizni tanlang" - n ni sun'iy oshirish
   QOIDA: n < chegara -> mashhurlik, aks holda CF; chegara O'LCHANADI

YANGI MAHSULOT (hali hech kim ko'rmagan):
   CF da vektori yo'q (y_i = 0) -> hech kimga tavsiya qilinmaydi
   -> ta'sir to'planmaydi -> hech qachon tavsiya qilinmaydi (halqa!)
   KONTENT: tavsif/atributlar bo'yicha foydalanuvchi profiliga yaqinlik
   GIBRID "KO'PRIK": kontent belgilari -> CF faktorlari regressiyasi
            (iliq mahsulotlarda o'qitiladi), yangi mahsulotga taxminiy y_i
   EXPLORATION: yangi mahsulotlarga ataylab kichik trafik (27.13 bandit)

1. NOMZOD GENERATSIYASI (candidate generation, recall bosqichi)
   bir nechta arzon manba, har biri 50-500 nomzod:
     ALS / item-kNN (vektorlar yaqinligi - ANN indeksi: Faiss, ScaNN)
     mashhurlik (umumiy, kategoriya, mintaqa, trend)
     "oxirgi ko'rgan mahsulotiga o'xshashlar", kontent
   maqsad - YAXSHI mahsulotni ro'yxatga KIRITISH (recall@500)

2. QAYTA RANJIRLASH (ranking bosqichi)
   bir necha yuz nomzod x boy belgilar -> gradient boosting (15-qism)
   yoki neyron tarmoq
   belgilar: CF ballari, mashhurlik, narx, foydalanuvchi faolligi,
             kontekst (vaqt, qurilma), mahsulot yoshi, oldingi ko'rsatishlar
   maqsad - ANIQ tartib (nDCG@10), biznes maqsadi (xarid ehtimoli)

3. BIZNES QOIDALARI
   omborda yo'q -> olib tashlash; xilma-xillik; homiylik; takrorlarni cheklash

OFFLINE:  tez, arzon, ko'p variantni solishtirish mumkin
          lekin ekspozitsiya tarafkashligi, pozitsiya tarafkashligi
ONLINE (A/B test, 27.13): yagona ishonchli javob
          asosiy metrika: xarid/sessiya, daromad; guardrail: qaytarish, tezlik
          ko'pincha offline yutuqning faqat bir qismi saqlanadi
          uzoq muddatli ta'sir (feedback loop) - haftalar kerak

import numpy as np
from scipy import sparse

# EXPLICIT: bias li MF, mini-partiyali SGD
e = r - (mu + bu[u] + bi[i] + np.sum(P[u] * Q[i], 1))
np.add.at(bu, u, lr * (e - reg * bu[u]))
np.add.at(bi, i, lr * (e - reg * bi[i]))
dP, dQ = lr * (e[:, None] * Q[i] - reg * P[u]), lr * (e[:, None] * P[u] - reg * Q[i])
np.add.at(P, u, dP)
np.add.at(Q, i, dQ)


# IMPLICIT: ALS qadami (hamma foydalanuvchi bir yo'la)
def als_qadam(R, Y, lam, alfa):
    k = Y.shape[1]
    YtY = Y.T @ Y + lam * np.eye(k)                       # hamma kataklar (c = 1)
    tashqi = (Y[:, :, None] * Y[:, None, :]).reshape(len(Y), k * k)
    W = R.copy()
    W.data = alfa * R.data                                # c - 1, faqat nol bo'lmagan
    A = YtY + (W @ tashqi).reshape(-1, k, k)
    W.data = 1 + alfa * R.data                            # c
    return np.linalg.solve(A, (W @ Y)[:, :, None])[:, :, 0]


V = rng.normal(0, 0.1, (n_item, k))
for _ in range(8):
    U = als_qadam(R, V, lam, alfa)
    V = als_qadam(R.T.tocsr(), U, lam, alfa)
ball = U[users] @ V.T

# yangi foydalanuvchi: fold-in (V qotirilgan)
x_yangi = als_qadam(R_malum, V, lam, alfa)

# yangi mahsulot: kontent -> faktor ko'prigi
v_yangi = Ridge(alpha=1.0).fit(tfidf[iliq], V[iliq]).predict(tfidf[yangi])

import implicit                                     # ALS, BPR - C++/GPU
model = implicit.als.AlternatingLeastSquares(factors=64, regularization=0.05,
                                             alpha=2.0, iterations=15)
model.fit(user_items)                               # CSR: foydalanuvchi x mahsulot
ids, ballar = model.recommend(u, user_items[u], N=10)

from surprise import SVD, Dataset                  # explicit baholar (bias li MF)
algo = SVD(n_factors=50, reg_all=0.05).fit(trainset)

R ~ P Q^T: foydalanuvchi va mahsulot - k o'lchamli did vektorlari
explicit: faqat kuzatilgan kataklar, mu + b_u + b_i + p.q, regularizatsiya
implicit: hamma kataklar, ishonch c = 1 + alfa*r, ALS yopiq yechim
BPR: tartibni to'g'ridan-to'g'ri, lekin negative sampling va sozlash sezgir
sovuq start: mashhurlik / fold-in / kontent ko'prigi - segment alohida o'lchanadi
offline - filtr, A/B test - hakam

"""Explicit baholar: SVD nima uchun ishlamaydi va bias li MF (SGD) noldan."""

import numpy as np
import pandas as pd
from sklearn.decomposition import TruncatedSVD
from sklearn.linear_model import LinearRegression
from scipy import sparse


def baholar(seed=0, n_user=1500, n_item=600, k=4):
    """Sintetik 1-5 baholar: global o'rtacha + biaslar + k ta yashirin faktor."""
    rng = np.random.default_rng(seed)
    bu, bi = rng.normal(0, 0.35, n_user), rng.normal(0, 0.45, n_item)
    P, Q = rng.normal(0, 0.55, (n_user, k)), rng.normal(0, 0.55, (n_item, k))
    faollik, pop = rng.lognormal(0, 0.8, n_user), rng.lognormal(0, 1.0, n_item)
    ehtimol = np.outer(faollik, pop)
    ehtimol *= 60_000 / ehtimol.sum()
    u, i = np.nonzero(rng.random((n_user, n_item)) < np.minimum(ehtimol, 1))
    r = 3.5 + bu[u] + bi[i] + np.sum(P[u] * Q[i], 1) + rng.normal(0, 0.6, len(u))
    df = pd.DataFrame({"user": u, "item": i, "baho": np.clip(np.round(r), 1, 5)})
    return df, Q


def bolish_baho(df, seed=0):
    """Har foydalanuvchining tasodifiy 20% baholari - test (kamida 5 ta bahosi bo'lsa)."""
    kalit = np.random.default_rng(seed).random(len(df))
    n = df.groupby("user")["baho"].transform("size").to_numpy()
    tartib = df.assign(k=kalit).groupby("user")["k"].rank(method="first").to_numpy()
    test = (tartib <= np.maximum(1, np.round(0.2 * n))) & (n >= 5)
    return df[~test].reset_index(drop=True), df[test].reset_index(drop=True)


def biaslar(tr, n_user, n_item, reg=5.0, iters=10):
    """b_u va b_i navbatma-navbat (regularizatsiyalangan o'rtachalar)."""
    mu = tr["baho"].mean()
    u, i, r = tr["user"].to_numpy(), tr["item"].to_numpy(), tr["baho"].to_numpy()
    bu, bi = np.zeros(n_user), np.zeros(n_item)
    for _ in range(iters):
        bu = np.bincount(u, r - mu - bi[i], n_user) / (np.bincount(u, None, n_user) + reg)
        bi = np.bincount(i, r - mu - bu[u], n_item) / (np.bincount(i, None, n_item) + reg)
    return mu, bu, bi


def mf_sgd(tr, n_user, n_item, k=10, lr=0.02, reg=0.05, epoxlar=40, partiya=256,
           seed=0, kuzat=None):
    """r_hat = mu + b_u + b_i + p_u . q_i; mini-partiyali SGD, L2 regularizatsiya."""
    rng = np.random.default_rng(seed)
    u, i, r = tr["user"].to_numpy(), tr["item"].to_numpy(), tr["baho"].to_numpy()
    mu = r.mean()
    bu, bi = np.zeros(n_user), np.zeros(n_item)
    P, Q = rng.normal(0, 0.1, (n_user, k)), rng.normal(0, 0.1, (n_item, k))
    tarix = []
    for ep in range(1, epoxlar + 1):
        for s in np.array_split(rng.permutation(len(r)), len(r) // partiya):
            uu, ii = u[s], i[s]
            e = r[s] - (mu + bu[uu] + bi[ii] + np.sum(P[uu] * Q[ii], 1))
            np.add.at(bu, uu, lr * (e - reg * bu[uu]))
            np.add.at(bi, ii, lr * (e - reg * bi[ii]))
            dP = lr * (e[:, None] * Q[ii] - reg * P[uu])
            dQ = lr * (e[:, None] * P[uu] - reg * Q[ii])
            np.add.at(P, uu, dP)
            np.add.at(Q, ii, dQ)
        if kuzat is not None and ep in kuzat:
            tarix.append((ep, rmse(tr, (mu, bu, bi, P, Q)), rmse(kuzat[ep], (mu, bu, bi, P, Q))))
    return (mu, bu, bi, P, Q), tarix


def bashorat(model, u, i):
    mu, bu, bi, P, Q = model
    return np.clip(mu + bu[u] + bi[i] + np.sum(P[u] * Q[i], 1), 1, 5)


def rmse(qism, model):
    return float(np.sqrt(np.mean((qism["baho"].to_numpy()
                                  - bashorat(model, qism["user"].to_numpy(),
                                             qism["item"].to_numpy())) ** 2)))


def main() -> None:
    df, Q_haqiqiy = baholar()
    n_user, n_item = 1500, 600
    tr, te = bolish_baho(df)
    print("=== 1. Ma'lumot ===")
    print(f"  baholar {len(df)}, zichlik {len(df) / (n_user * n_item):.1%}, "
          f"train {len(tr)}, test {len(te)} ({te['user'].nunique()} foyd.)")
    print(f"  o'rtacha baho {df['baho'].mean():.2f}, std {df['baho'].std():.2f}")

    u_te, i_te, r_te = te["user"].to_numpy(), te["item"].to_numpy(), te["baho"].to_numpy()
    xato = {}                                          # har test bahosi uchun kvadrat xato

    print("\n=== 2. Bazaviylar ===")
    mu = tr["baho"].mean()
    xato["global o'rtacha"] = (r_te - mu) ** 2
    mu, bu, bi = biaslar(tr, n_user, n_item)
    xato["biaslar"] = (r_te - np.clip(mu + bu[u_te] + bi[i_te], 1, 5)) ** 2
    for nom in xato:
        print(f"  {nom:<16} RMSE {np.sqrt(xato[nom].mean()):.4f}")

    print("\n=== 3. TruncatedSVD: bo'sh kataklar nol deb olinadi ===")
    R = sparse.csr_matrix((tr["baho"], (tr["user"], tr["item"])), shape=(n_user, n_item))
    for k in [4, 10]:
        svd = TruncatedSVD(k, random_state=0).fit(R)
        rek = svd.transform(R) @ svd.components_
        print(f"  xom (nol = baho 0), k={k:>2}: test bashorati o'rtachasi "
              f"{rek[u_te, i_te].mean():.2f}, RMSE {np.sqrt(np.mean((r_te - rek[u_te, i_te]) ** 2)):.4f}")
    qoldiq = tr["baho"] - mu - bu[tr["user"]] - bi[tr["item"]]
    Rq = sparse.csr_matrix((qoldiq, (tr["user"], tr["item"])), shape=(n_user, n_item))
    for k in [4, 10]:
        svd = TruncatedSVD(k, random_state=0).fit(Rq)
        rek = svd.transform(Rq) @ svd.components_
        p = np.clip(mu + bu[u_te] + bi[i_te] + rek[u_te, i_te], 1, 5)
        xato[f"biaslar + SVD k={k}"] = (r_te - p) ** 2
        print(f"  biaslar + qoldiq SVD, k={k:>2}: RMSE {np.sqrt(xato[f'biaslar + SVD k={k}'].mean()):.4f}")

    print("\n=== 4. MF (SGD): regularizatsiya ===")
    kuzat = {ep: te for ep in [2, 5, 10, 20, 40]}
    for reg in [0.0, 0.05]:
        model, tarix = mf_sgd(tr, n_user, n_item, reg=reg, kuzat=kuzat)
        print(f"  reg={reg}: " + "  ".join(f"ep{e} {a:.3f}/{b:.3f}" for e, a, b in tarix))
        if reg > 0:
            xato["MF (SGD)"] = (r_te - bashorat(model, u_te, i_te)) ** 2
    print("  (train RMSE / test RMSE)")

    print("\n=== 5. Juftlashgan taqqoslash (SE foydalanuvchilar bo'yicha) ===")
    asos = pd.Series(xato["biaslar"]).groupby(u_te).mean()
    for nom in ["global o'rtacha", "biaslar + SVD k=4", "MF (SGD)"]:
        d = pd.Series(xato[nom]).groupby(u_te).mean() - asos
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  {nom:<18} - biaslar: MSE farqi {d.mean():+.4f}, SE {se:.4f}, "
              f"sezilarli: {abs(d.mean()) > 2 * se}")

    print("\n=== 6. Yashirin faktorlar haqiqiy tuzilmani tiklaydimi? ===")
    Q_mf = model[4]
    pop = np.bincount(tr["item"], minlength=n_item)
    m = pop >= 20
    r2 = LinearRegression().fit(Q_mf[m], Q_haqiqiy[m]).score(Q_mf[m], Q_haqiqiy[m])
    print(f"  {m.sum()} ta mahsulot (>= 20 baho): o'rganilgan 10 faktor haqiqiy "
          f"4 faktorning R^2 = {r2:.3f} qismini chiziqli tushuntiradi")
    r2_kam = LinearRegression().fit(Q_mf[~m], Q_haqiqiy[~m]).score(Q_mf[~m], Q_haqiqiy[~m])
    print(f"  {(~m).sum()} ta kam baholi mahsulot (< 20): R^2 = {r2_kam:.3f}")
    print("  ⭐ Faktorlar nomsiz, lekin ma'lumotdagi yashirin 'did o'qlari'ni topadi")


if __name__ == "__main__":
    main()

=== 1. Ma'lumot ===
  baholar 59207, zichlik 6.6%, train 47377, test 11830 (1469 foyd.)
  o'rtacha baho 3.43, std 1.00

=== 2. Bazaviylar ===
  global o'rtacha  RMSE 0.9994
  biaslar          RMSE 0.8545

=== 3. TruncatedSVD: bo'sh kataklar nol deb olinadi ===
  xom (nol = baho 0), k= 4: test bashorati o'rtachasi 0.58, RMSE 3.0912
  xom (nol = baho 0), k=10: test bashorati o'rtachasi 0.49, RMSE 3.1733
  biaslar + qoldiq SVD, k= 4: RMSE 0.8137
  biaslar + qoldiq SVD, k=10: RMSE 0.8230

=== 4. MF (SGD): regularizatsiya ===
  reg=0.0: ep2 0.827/0.859  ep5 0.792/0.853  ep10 0.646/0.797  ep20 0.523/0.779  ep40 0.472/0.813
  reg=0.05: ep2 0.830/0.859  ep5 0.812/0.856  ep10 0.726/0.810  ep20 0.582/0.738  ep40 0.521/0.739
  (train RMSE / test RMSE)

=== 5. Juftlashgan taqqoslash (SE foydalanuvchilar bo'yicha) ===
  global o'rtacha    - biaslar: MSE farqi +0.2556, SE 0.0125, sezilarli: True
  biaslar + SVD k=4  - biaslar: MSE farqi -0.0442, SE 0.0018, sezilarli: True
  MF (SGD)           - biaslar: MSE farqi -0.1620, SE 0.0103, sezilarli: True

=== 6. Yashirin faktorlar haqiqiy tuzilmani tiklaydimi? ===
  511 ta mahsulot (>= 20 baho): o'rganilgan 10 faktor haqiqiy 4 faktorning R^2 = 0.846 qismini chiziqli tushuntiradi
  89 ta kam baholi mahsulot (< 20): R^2 = 0.505
  ⭐ Faktorlar nomsiz, lekin ma'lumotdagi yashirin 'did o'qlari'ni topadi

"""Implicit feedback uchun ALS noldan: ishonch, hamma kataklar va bazaviylarga qarshi."""

import numpy as np
import pandas as pd
from scipy import sparse

KATEGORIYA = {
    "elektronika": "telefon smartfon quloqchin zaryadlovchi noutbuk planshet "
                   "sichqoncha klaviatura kolonka televizor kamera soat",
    "kiyim": "koylak shim kurtka poyabzal krossovka shapka sharf palto "
             "kostyum futbolka yubka sviter",
    "kitob": "roman darslik detektiv sher tarix fantastika lugat ertak "
             "biografiya psixologiya biznes dasturlash",
    "oziq": "choy qahva asal yongoq shokolad guruch yog un makaron sharbat "
            "pechene ziravor",
    "gozallik": "krem atir shampun labbuyoq boyoq niqob sovun losyon taroq "
                "lak gel upa",
    "uy": "idish qozon choynak gilam parda yostiq chiroq sochiq pichoq tova "
          "vaza javon",
    "sport": "koptok gantel velosiped raketka forma gilamcha skakalka shlem "
             "sumka butsa trenajyor suzish",
    "bolalar": "oyinchoq konstruktor qogirchoq kolyaska taglik mashinka pazl "
               "albom rangli kiyimcha kitobcha samokat",
}
NARX = ["arzon", "orta", "premium"]
UMUMIY = "sifatli yangi chegirma original mashhur qulay kafolat tez yetkazish".split()


def dokon(seed=0, n_user=2000, n_item=800, kunlar=180):
    """Sintetik onlayn do'kon: yashirin didlar, mashhurlik dumi, yangi mahsulotlar."""
    rng = np.random.default_rng(seed)
    katlar = list(KATEGORIYA)
    n_kat, n_sub = len(katlar), 4
    sub = rng.integers(0, n_kat * n_sub, n_item)          # har mahsulot kichik guruhi
    kat = sub // n_sub
    narx = rng.choice(3, n_item, p=[0.5, 0.35, 0.15])
    sifat = rng.lognormal(0, 1.4, n_item)                  # mashhurlik dumi manbai
    chiqish = np.where(rng.random(n_item) < 0.7, 0,
                       rng.integers(0, kunlar - 20, n_item))
    tavsif = []
    for i in range(n_item):
        sozlar = KATEGORIYA[katlar[kat[i]]].split()
        guruh = sozlar[(sub[i] % n_sub) * 3:(sub[i] % n_sub) * 3 + 3]
        tanlov = list(rng.choice(guruh, 2, replace=False))
        tanlov.append(rng.choice([s for s in sozlar if s not in tanlov]))
        tanlov += [katlar[kat[i]], NARX[narx[i]], f"brend{rng.integers(0, 40)}"]
        tanlov += list(rng.choice(UMUMIY, 2, replace=False))
        tavsif.append(" ".join(tanlov))
    # foydalanuvchi didi: kategoriya x kichik guruh x narx
    kat_pref = rng.dirichlet(np.full(n_kat, 0.3), n_user)
    sub_pref = rng.dirichlet(np.full(n_sub, 0.5), (n_user, n_kat))
    narx_pref = rng.dirichlet(np.full(3, 1.0), n_user)
    did = (kat_pref[:, kat] * sub_pref[:, kat, sub % n_sub]
           * narx_pref[:, narx])                           # (user, item)
    did /= did.sum(1, keepdims=True)
    hafta = np.arange(0, kunlar, 7)
    yosh = np.clip((hafta[:, None] - chiqish[None, :]) / 7, 0, None)
    jalb = sifat * (0.6 + 1.2 * np.exp(-yosh / 6)) * (hafta[:, None] >= chiqish)
    qatorlar = []
    for u in range(n_user):
        n_u = int(np.clip(rng.lognormal(2.8, 0.8), 3, 150))
        vaqtlar = np.sort(rng.uniform(0, kunlar, n_u))
        korgan = np.zeros(n_item, dtype=bool)
        for t in vaqtlar:
            a = jalb[int(t) // 7] * ~korgan
            p = 0.75 * did[u] * a / (did[u] @ a) + 0.25 * a / a.sum()
            i = int(np.searchsorted(np.cumsum(p), rng.random() * p.sum()))
            korgan[i] = True
            mos = did[u, i] / did[u].max()
            xarid = rng.random() < 0.15 + 0.35 * mos
            baho = 0
            if xarid and rng.random() < 0.3:
                baho = int(np.clip(round(2.4 + 2.2 * mos + 0.3 * np.log(sifat[i])
                                         + rng.normal(0, 0.7)), 1, 5))
            qatorlar.append((u, i, t, int(xarid), baho))
    df = pd.DataFrame(qatorlar, columns=["user", "item", "vaqt", "xarid", "baho"])
    items = pd.DataFrame({"item": np.arange(n_item), "kategoriya": np.array(katlar)[kat],
                          "sub": sub, "narx": narx, "chiqish": chiqish,
                          "tavsif": tavsif})
    return df, items


def bolish(df, usul="vaqt", ulush=0.2, seed=0, chegara=150.0):
    """'vaqt' - har foydalanuvchining oxirgi 20% i test; 'tasodifiy'; 'global'."""
    df = df.sort_values(["user", "vaqt"]).reset_index(drop=True)
    if usul == "global":
        tarix = df[df["vaqt"] < chegara].groupby("user").size()
        yetarli = df["user"].map(tarix).fillna(0).to_numpy() >= 3
        test = (df["vaqt"] >= chegara).to_numpy() & yetarli
        return df[df["vaqt"] < chegara].copy(), df[test].copy()
    n = df.groupby("user")["item"].transform("size").to_numpy()
    if usul == "tasodifiy":
        kalit = np.random.default_rng(seed).random(len(df))
        tartib = (df.assign(k=kalit).groupby("user")["k"]
                  .rank(method="first").to_numpy() - 1)
    else:
        tartib = df.groupby("user").cumcount().to_numpy()
    n_test = np.maximum(1, np.round(ulush * n)).astype(int)
    test = (tartib >= n - n_test) & (n >= 5)
    return df[~test].copy(), df[test].copy()


def matritsa(qism, n_user, n_item):
    return sparse.csr_matrix((np.ones(len(qism)), (qism["user"], qism["item"])),
                             shape=(n_user, n_item))


def metrikalar(top, T):
    """top: (n, k) tavsiya indekslari; T: (n, n_item) test to'plami (bool)."""
    k = top.shape[1]
    rel = np.take_along_axis(T, top, axis=1)
    n_rel = T.sum(1)
    pozitsiya = np.arange(1, k + 1)
    ap = (np.cumsum(rel, 1) / pozitsiya * rel).sum(1) / np.minimum(n_rel, k)
    chegirma = 1 / np.log2(pozitsiya + 1)
    idcg = np.array([chegirma[:min(r, k)].sum() for r in n_rel])
    return {"precision": rel.sum(1) / k, "recall": rel.sum(1) / n_rel,
            "hit": rel.any(1).astype(float), "MAP": ap,
            "nDCG": (rel * chegirma).sum(1) / idcg}


def baholash(ball, X_train, test, users, k=10):
    """Train da ko'rilgan mahsulotlar tavsiya qilinmaydi; har foydalanuvchi metrikasi."""
    ball = np.array(ball, dtype=float)
    ball[X_train[users].toarray() > 0] = -np.inf
    top = np.argsort(-ball, axis=1, kind="stable")[:, :k]
    qator = pd.Series(np.arange(len(users)), index=users)
    tt = test[test["user"].isin(users)]
    T = np.zeros(ball.shape, dtype=bool)
    T[qator[tt["user"]].to_numpy(), tt["item"].to_numpy()] = True
    return metrikalar(top, T), top


def als_qadam(R, Y, lam, alfa):
    """Hamma qatorlar uchun bir yo'la: (Y^T Y + Y^T (C_u - I) Y + lam*I) x_u = Y^T C_u p_u.

    Y^T Y bitta marta hisoblanadi va HAMMA kataklarni (nollarni ham, c = 1 bilan)
    qamraydi; qo'shimcha had sum_i (c_ui - 1) y_i y_i^T faqat nol bo'lmagan
    kataklar ustida - siyrak matritsa x "tashqi ko'paytmalar" jadvali.
    """
    k = Y.shape[1]
    YtY = Y.T @ Y + lam * np.eye(k)
    tashqi = (Y[:, :, None] * Y[:, None, :]).reshape(len(Y), k * k)
    W = R.copy()
    W.data = alfa * R.data                           # c - 1
    A = YtY + (W @ tashqi).reshape(-1, k, k)
    W.data = 1 + alfa * R.data                       # c (p = 1 kuzatilganda)
    return np.linalg.solve(A, (W @ Y)[:, :, None])[:, :, 0]


def als(R, k=32, lam=10.0, alfa=10.0, iters=10, seed=0):
    """Implicit ALS (Hu, Koren, Volinsky 2008): ishonch c = 1 + alfa * r."""
    rng = np.random.default_rng(seed)
    R = R.tocsr()
    Rt = R.T.tocsr()
    V = rng.normal(0, 0.1, (R.shape[1], k))
    for _ in range(iters):
        U = als_qadam(R, V, lam, alfa)
        V = als_qadam(Rt, U, lam, alfa)
    return U, V


def kuch_matritsa(qism, n_user, n_item):
    """r = 1 (ko'rish) yoki 3 (xarid) - implicit signal kuchi."""
    return sparse.csr_matrix((1.0 + 2.0 * qism["xarid"].to_numpy(),
                              (qism["user"], qism["item"])), shape=(n_user, n_item))


def item_knn(X, users, k=50):
    norma = np.sqrt(np.asarray(X.multiply(X).sum(0)).ravel())
    Xn = X @ sparse.diags(1 / np.maximum(norma, 1e-12))
    S = (Xn.T @ Xn).toarray()
    np.fill_diagonal(S, 0)
    kesish = np.argpartition(-S, k, axis=1)[:, k:]
    np.put_along_axis(S, kesish, 0, axis=1)
    return X[users] @ S


def als_faqat_kuzatilgan(R, k=32, lam=10.0, alfa=1.0, iters=8, seed=0):
    """XATO variant: faqat kuzatilgan kataklar (nollar tashlab ketilgan)."""
    def qadam(R, Y):
        tashqi = (Y[:, :, None] * Y[:, None, :]).reshape(len(Y), k * k)
        W = R.copy()
        W.data = 1 + alfa * R.data
        A = (W @ tashqi).reshape(-1, k, k) + lam * np.eye(k)   # Y^T Y YO'Q
        return np.linalg.solve(A, (W @ Y)[:, :, None])[:, :, 0]
    rng = np.random.default_rng(seed)
    R, Rt = R.tocsr(), R.T.tocsr()
    V = rng.normal(0, 0.1, (R.shape[1], k))
    for _ in range(iters):
        U = qadam(R, V)
        V = qadam(Rt, U)
    return U, V


def yoqotish(R, U, V, lam, alfa):
    """sum_{hamma u,i} c_ui (p_ui - u.v)^2 + lam (|U|^2 + |V|^2)."""
    D = R.toarray()
    P, C = (D > 0).astype(float), 1 + alfa * D
    return float((C * (P - U @ V.T) ** 2).sum() + lam * ((U ** 2).sum() + (V ** 2).sum()))


def main() -> None:
    df, items = dokon()
    n_user, n_item = 2000, len(items)
    tr, te = bolish(df, "vaqt")
    X = matritsa(tr, n_user, n_item)
    R = kuch_matritsa(tr, n_user, n_item)
    users = np.sort(te["user"].unique())
    lam, alfa = 10.0, 1.0                            # boshlang'ich qiymatlar

    print("=== 1. Tezkor formula = to'liq hisob ===")
    V = np.random.default_rng(0).normal(0, 0.3, (n_item, 32))
    tez = als_qadam(R, V, lam, alfa)
    D = R.toarray()
    for u in [0, 1, 2]:
        c = 1 + alfa * D[u]                          # HAMMA 800 katak
        p = (D[u] > 0).astype(float)
        toliq = np.linalg.solve((V.T * c) @ V + lam * np.eye(32), V.T @ (c * p))
        print(f"  foydalanuvchi {u}: kuzatilgan {int(p.sum()):>2} ta, to'liq hisob "
              f"{n_item} ta katak; bir xil (1e-10): {np.allclose(toliq, tez[u], atol=1e-10)}")

    print("\n=== 2. O'qitish: yo'qotish har iteratsiyada kamayadi ===")
    rng = np.random.default_rng(0)
    Rt = R.T.tocsr()
    V = rng.normal(0, 0.1, (n_item, 32))
    for it in range(1, 9):
        U = als_qadam(R, V, lam, alfa)
        V = als_qadam(Rt, U, lam, alfa)
        if it in (1, 2, 3, 5, 8):
            print(f"  iteratsiya {it:>2}: yo'qotish {yoqotish(R, U, V, lam, alfa):,.0f}")

    print("\n=== 3. Taqqoslash ('vaqt' bo'lishi, k=10) ===")
    pop = np.asarray(X.sum(0)).ravel()
    usullar = {"mashhurlik": np.tile(pop, (len(users), 1)).astype(float),
               "item-kNN": item_knn(X, users), "ALS (k=32)": U[users] @ V.T}
    natija = {}
    print(f"  {'usul':<12}" + "".join(f"{k:>10}" for k in
                                     ["precision", "recall", "hit", "MAP", "nDCG"]))
    for nom, b in usullar.items():
        natija[nom], _ = baholash(b, X, te, users)
        print(f"  {nom:<12}" + "".join(f"{v.mean():>10.4f}"
                                        for v in natija[nom].values()))

    print("\n=== 4. Juftlashgan taqqoslash (nDCG@10, foydalanuvchilar bo'yicha) ===")
    for a, b in [("ALS (k=32)", "mashhurlik"), ("ALS (k=32)", "item-kNN")]:
        d = natija[a]["nDCG"] - natija[b]["nDCG"]
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  {a} - {b}: {d.mean():+.4f}, SE {se:.4f}, "
              f"sezilarli: {abs(d.mean()) > 2 * se}")

    print("\n=== 5. Nega nollar kerak: to'rt variant ===")
    bosh = pop >= np.quantile(pop, 0.9)
    variantlar = {"faqat kuzatilgan": als_faqat_kuzatilgan(R),
                  "alfa=0 (hamma teng)": als(R, alfa=0.0, iters=8),
                  "alfa=1": (U, V),
                  "alfa=10": als(R, alfa=10.0, iters=8)}
    for nom, (U2, V2) in variantlar.items():
        r, top = baholash(U2[users] @ V2.T, X, te, users)
        print(f"  {nom:<19} nDCG {r['nDCG'].mean():.4f}, "
              f"tavsiyalarda top10% ulushi {bosh[top].mean():.1%}, "
              f"qamrov {len(np.unique(top)) / n_item:.1%}")
    print("  ⭐ Bo'sh kataklar 'kuchsiz salbiy' sifatida ishtirok etishi shart")


if __name__ == "__main__":
    main()

=== 1. Tezkor formula = to'liq hisob ===
  foydalanuvchi 0: kuzatilgan 10 ta, to'liq hisob 800 ta katak; bir xil (1e-10): True
  foydalanuvchi 1: kuzatilgan  4 ta, to'liq hisob 800 ta katak; bir xil (1e-10): True
  foydalanuvchi 2: kuzatilgan  7 ta, to'liq hisob 800 ta katak; bir xil (1e-10): True

=== 2. O'qitish: yo'qotish har iteratsiyada kamayadi ===
  iteratsiya  1: yo'qotish 72,626
  iteratsiya  2: yo'qotish 61,889
  iteratsiya  3: yo'qotish 60,145
  iteratsiya  5: yo'qotish 59,152
  iteratsiya  8: yo'qotish 58,783

=== 3. Taqqoslash ('vaqt' bo'lishi, k=10) ===
  usul         precision    recall       hit       MAP      nDCG
  mashhurlik      0.0383    0.0924    0.3203    0.0351    0.0706
  item-kNN        0.0508    0.1234    0.3954    0.0501    0.0961
  ALS (k=32)      0.0598    0.1408    0.4378    0.0638    0.1174

=== 4. Juftlashgan taqqoslash (nDCG@10, foydalanuvchilar bo'yicha) ===
  ALS (k=32) - mashhurlik: +0.0468, SE 0.0048, sezilarli: True
  ALS (k=32) - item-kNN: +0.0213, SE 0.0040, sezilarli: True

=== 5. Nega nollar kerak: to'rt variant ===
  faqat kuzatilgan    nDCG 0.0687, tavsiyalarda top10% ulushi 100.0%, qamrov 5.1%
  alfa=0 (hamma teng) nDCG 0.1167, tavsiyalarda top10% ulushi 83.7%, qamrov 26.5%
  alfa=1              nDCG 0.1174, tavsiyalarda top10% ulushi 69.0%, qamrov 36.0%
  alfa=10             nDCG 0.0833, tavsiyalarda top10% ulushi 60.6%, qamrov 50.9%
  ⭐ Bo'sh kataklar 'kuchsiz salbiy' sifatida ishtirok etishi shart

"""Giperparametrlarni validatsiyada tanlash (ALS) va BPR (torch) bilan taqqoslash."""

import numpy as np
import pandas as pd
import torch
from scipy import sparse

KATEGORIYA = {
    "elektronika": "telefon smartfon quloqchin zaryadlovchi noutbuk planshet "
                   "sichqoncha klaviatura kolonka televizor kamera soat",
    "kiyim": "koylak shim kurtka poyabzal krossovka shapka sharf palto "
             "kostyum futbolka yubka sviter",
    "kitob": "roman darslik detektiv sher tarix fantastika lugat ertak "
             "biografiya psixologiya biznes dasturlash",
    "oziq": "choy qahva asal yongoq shokolad guruch yog un makaron sharbat "
            "pechene ziravor",
    "gozallik": "krem atir shampun labbuyoq boyoq niqob sovun losyon taroq "
                "lak gel upa",
    "uy": "idish qozon choynak gilam parda yostiq chiroq sochiq pichoq tova "
          "vaza javon",
    "sport": "koptok gantel velosiped raketka forma gilamcha skakalka shlem "
             "sumka butsa trenajyor suzish",
    "bolalar": "oyinchoq konstruktor qogirchoq kolyaska taglik mashinka pazl "
               "albom rangli kiyimcha kitobcha samokat",
}
NARX = ["arzon", "orta", "premium"]
UMUMIY = "sifatli yangi chegirma original mashhur qulay kafolat tez yetkazish".split()


def dokon(seed=0, n_user=2000, n_item=800, kunlar=180):
    """Sintetik onlayn do'kon: yashirin didlar, mashhurlik dumi, yangi mahsulotlar."""
    rng = np.random.default_rng(seed)
    katlar = list(KATEGORIYA)
    n_kat, n_sub = len(katlar), 4
    sub = rng.integers(0, n_kat * n_sub, n_item)          # har mahsulot kichik guruhi
    kat = sub // n_sub
    narx = rng.choice(3, n_item, p=[0.5, 0.35, 0.15])
    sifat = rng.lognormal(0, 1.4, n_item)                  # mashhurlik dumi manbai
    chiqish = np.where(rng.random(n_item) < 0.7, 0,
                       rng.integers(0, kunlar - 20, n_item))
    tavsif = []
    for i in range(n_item):
        sozlar = KATEGORIYA[katlar[kat[i]]].split()
        guruh = sozlar[(sub[i] % n_sub) * 3:(sub[i] % n_sub) * 3 + 3]
        tanlov = list(rng.choice(guruh, 2, replace=False))
        tanlov.append(rng.choice([s for s in sozlar if s not in tanlov]))
        tanlov += [katlar[kat[i]], NARX[narx[i]], f"brend{rng.integers(0, 40)}"]
        tanlov += list(rng.choice(UMUMIY, 2, replace=False))
        tavsif.append(" ".join(tanlov))
    # foydalanuvchi didi: kategoriya x kichik guruh x narx
    kat_pref = rng.dirichlet(np.full(n_kat, 0.3), n_user)
    sub_pref = rng.dirichlet(np.full(n_sub, 0.5), (n_user, n_kat))
    narx_pref = rng.dirichlet(np.full(3, 1.0), n_user)
    did = (kat_pref[:, kat] * sub_pref[:, kat, sub % n_sub]
           * narx_pref[:, narx])                           # (user, item)
    did /= did.sum(1, keepdims=True)
    hafta = np.arange(0, kunlar, 7)
    yosh = np.clip((hafta[:, None] - chiqish[None, :]) / 7, 0, None)
    jalb = sifat * (0.6 + 1.2 * np.exp(-yosh / 6)) * (hafta[:, None] >= chiqish)
    qatorlar = []
    for u in range(n_user):
        n_u = int(np.clip(rng.lognormal(2.8, 0.8), 3, 150))
        vaqtlar = np.sort(rng.uniform(0, kunlar, n_u))
        korgan = np.zeros(n_item, dtype=bool)
        for t in vaqtlar:
            a = jalb[int(t) // 7] * ~korgan
            p = 0.75 * did[u] * a / (did[u] @ a) + 0.25 * a / a.sum()
            i = int(np.searchsorted(np.cumsum(p), rng.random() * p.sum()))
            korgan[i] = True
            mos = did[u, i] / did[u].max()
            xarid = rng.random() < 0.15 + 0.35 * mos
            baho = 0
            if xarid and rng.random() < 0.3:
                baho = int(np.clip(round(2.4 + 2.2 * mos + 0.3 * np.log(sifat[i])
                                         + rng.normal(0, 0.7)), 1, 5))
            qatorlar.append((u, i, t, int(xarid), baho))
    df = pd.DataFrame(qatorlar, columns=["user", "item", "vaqt", "xarid", "baho"])
    items = pd.DataFrame({"item": np.arange(n_item), "kategoriya": np.array(katlar)[kat],
                          "sub": sub, "narx": narx, "chiqish": chiqish,
                          "tavsif": tavsif})
    return df, items


def bolish(df, usul="vaqt", ulush=0.2, seed=0, chegara=150.0):
    """'vaqt' - har foydalanuvchining oxirgi 20% i test; 'tasodifiy'; 'global'."""
    df = df.sort_values(["user", "vaqt"]).reset_index(drop=True)
    if usul == "global":
        tarix = df[df["vaqt"] < chegara].groupby("user").size()
        yetarli = df["user"].map(tarix).fillna(0).to_numpy() >= 3
        test = (df["vaqt"] >= chegara).to_numpy() & yetarli
        return df[df["vaqt"] < chegara].copy(), df[test].copy()
    n = df.groupby("user")["item"].transform("size").to_numpy()
    if usul == "tasodifiy":
        kalit = np.random.default_rng(seed).random(len(df))
        tartib = (df.assign(k=kalit).groupby("user")["k"]
                  .rank(method="first").to_numpy() - 1)
    else:
        tartib = df.groupby("user").cumcount().to_numpy()
    n_test = np.maximum(1, np.round(ulush * n)).astype(int)
    test = (tartib >= n - n_test) & (n >= 5)
    return df[~test].copy(), df[test].copy()


def matritsa(qism, n_user, n_item):
    return sparse.csr_matrix((np.ones(len(qism)), (qism["user"], qism["item"])),
                             shape=(n_user, n_item))


def metrikalar(top, T):
    """top: (n, k) tavsiya indekslari; T: (n, n_item) test to'plami (bool)."""
    k = top.shape[1]
    rel = np.take_along_axis(T, top, axis=1)
    n_rel = T.sum(1)
    pozitsiya = np.arange(1, k + 1)
    ap = (np.cumsum(rel, 1) / pozitsiya * rel).sum(1) / np.minimum(n_rel, k)
    chegirma = 1 / np.log2(pozitsiya + 1)
    idcg = np.array([chegirma[:min(r, k)].sum() for r in n_rel])
    return {"precision": rel.sum(1) / k, "recall": rel.sum(1) / n_rel,
            "hit": rel.any(1).astype(float), "MAP": ap,
            "nDCG": (rel * chegirma).sum(1) / idcg}


def baholash(ball, X_train, test, users, k=10):
    """Train da ko'rilgan mahsulotlar tavsiya qilinmaydi; har foydalanuvchi metrikasi."""
    ball = np.array(ball, dtype=float)
    ball[X_train[users].toarray() > 0] = -np.inf
    top = np.argsort(-ball, axis=1, kind="stable")[:, :k]
    qator = pd.Series(np.arange(len(users)), index=users)
    tt = test[test["user"].isin(users)]
    T = np.zeros(ball.shape, dtype=bool)
    T[qator[tt["user"]].to_numpy(), tt["item"].to_numpy()] = True
    return metrikalar(top, T), top


def als_qadam(R, Y, lam, alfa):
    """Hamma qatorlar uchun bir yo'la: (Y^T Y + Y^T (C_u - I) Y + lam*I) x_u = Y^T C_u p_u.

    Y^T Y bitta marta hisoblanadi va HAMMA kataklarni (nollarni ham, c = 1 bilan)
    qamraydi; qo'shimcha had sum_i (c_ui - 1) y_i y_i^T faqat nol bo'lmagan
    kataklar ustida - siyrak matritsa x "tashqi ko'paytmalar" jadvali.
    """
    k = Y.shape[1]
    YtY = Y.T @ Y + lam * np.eye(k)
    tashqi = (Y[:, :, None] * Y[:, None, :]).reshape(len(Y), k * k)
    W = R.copy()
    W.data = alfa * R.data                           # c - 1
    A = YtY + (W @ tashqi).reshape(-1, k, k)
    W.data = 1 + alfa * R.data                       # c (p = 1 kuzatilganda)
    return np.linalg.solve(A, (W @ Y)[:, :, None])[:, :, 0]


def als(R, k=32, lam=10.0, alfa=10.0, iters=10, seed=0):
    """Implicit ALS (Hu, Koren, Volinsky 2008): ishonch c = 1 + alfa * r."""
    rng = np.random.default_rng(seed)
    R = R.tocsr()
    Rt = R.T.tocsr()
    V = rng.normal(0, 0.1, (R.shape[1], k))
    for _ in range(iters):
        U = als_qadam(R, V, lam, alfa)
        V = als_qadam(Rt, U, lam, alfa)
    return U, V


def kuch_matritsa(qism, n_user, n_item):
    """r = 1 (ko'rish) yoki 3 (xarid) - implicit signal kuchi."""
    return sparse.csr_matrix((1.0 + 2.0 * qism["xarid"].to_numpy(),
                              (qism["user"], qism["item"])), shape=(n_user, n_item))


def item_knn(X, users, k=50):
    norma = np.sqrt(np.asarray(X.multiply(X).sum(0)).ravel())
    Xn = X @ sparse.diags(1 / np.maximum(norma, 1e-12))
    S = (Xn.T @ Xn).toarray()
    np.fill_diagonal(S, 0)
    kesish = np.argpartition(-S, k, axis=1)[:, k:]
    np.put_along_axis(S, kesish, 0, axis=1)
    return X[users] @ S


def bpr(X, k=32, epoxlar=16, lr=0.05, reg=1e-5, partiya=2048, seed=0, kuzat=None):
    """BPR: har (u, i+) juftligiga tasodifiy j- olinadi; loss = -log sigmoid(x_ui - x_uj)."""
    torch.manual_seed(seed)
    torch.set_flush_denormal(True)       # Adam dagi juda kichik sonlar CPU ni sekinlatmasin
    g = torch.Generator().manual_seed(seed)
    n_user, n_item = X.shape
    U = torch.nn.Embedding(n_user, k)
    V = torch.nn.Embedding(n_item, k)
    b = torch.nn.Embedding(n_item, 1)
    for e in (U, V):
        torch.nn.init.normal_(e.weight, 0, 0.1)
    torch.nn.init.zeros_(b.weight)
    opt = torch.optim.Adam([*U.parameters(), *V.parameters(), *b.parameters()], lr=lr)
    Xc = X.tocoo()
    uu, ii = torch.tensor(Xc.row, dtype=torch.long), torch.tensor(Xc.col, dtype=torch.long)
    tarix = {}
    for ep in range(1, epoxlar + 1):
        tartib = torch.randperm(len(uu), generator=g)
        jj_hammasi = torch.randint(0, n_item, (len(uu),), generator=g)
        for s in range(0, len(uu), partiya):
            idx = tartib[s:s + partiya]
            u, i, j = uu[idx], ii[idx], jj_hammasi[idx]
            pu = U(u)
            x_ui = (pu * V(i)).sum(1) + b(i).squeeze(1)
            x_uj = (pu * V(j)).sum(1) + b(j).squeeze(1)
            loss = (-torch.nn.functional.logsigmoid(x_ui - x_uj).mean()
                    + reg * (pu.pow(2).sum() + V(i).pow(2).sum() + V(j).pow(2).sum()))
            opt.zero_grad()
            loss.backward()
            opt.step()
        if kuzat is not None and ep in kuzat:
            tarix[ep] = kuzat[ep](bpr_ball(U, V, b))
    return (U, V, b), tarix


def bpr_ball(U, V, b):
    with torch.no_grad():
        return (U.weight @ V.weight.T + b.weight.T).numpy()


def main() -> None:
    df, items = dokon()
    n_user, n_item = 2000, len(items)
    tr, te = bolish(df, "vaqt")
    tr2, va = bolish(tr, "vaqt")                   # validatsiya - train ICHIDAN
    X2, R2 = matritsa(tr2, n_user, n_item), kuch_matritsa(tr2, n_user, n_item)
    u_va = np.sort(va["user"].unique())

    def val_ndcg(ball_hammasi):
        r, _ = baholash(ball_hammasi[u_va], X2, va, u_va)
        return r["nDCG"]

    print("=== 1. ALS: k va lambda validatsiyada (alfa = 1) ===")
    natija = {}
    for k in [8, 16, 32]:
        for lam in [3.0, 10.0, 30.0]:
            U, V = als(R2, k=k, lam=lam, alfa=1.0, iters=6)
            natija[(k, lam, 1.0)] = val_ndcg(U @ V.T)
    eng = max(natija, key=lambda c: natija[c].mean())
    print(f"  {'k':>4}" + "".join(f"{f'lambda={l:g}':>12}" for l in [3, 10, 30]))
    for k in [8, 16, 32]:
        print(f"  {k:>4}" + "".join(f"{natija[(k, l, 1.0)].mean():>12.4f}"
                                    for l in [3.0, 10.0, 30.0]))
    print(f"  eng yaxshi: k={eng[0]}, lambda={eng[1]:g}")

    print("\n=== 2. alfa (eng yaxshi k va lambda bilan) ===")
    for alfa in [0.0, 3.0]:
        U, V = als(R2, k=eng[0], lam=eng[1], alfa=alfa, iters=6)
        natija[(eng[0], eng[1], alfa)] = val_ndcg(U @ V.T)
    for alfa in [0.0, 1.0, 3.0]:
        print(f"  alfa={alfa:g}: val nDCG {natija[(eng[0], eng[1], alfa)].mean():.4f}")
    eng = max(natija, key=lambda c: natija[c].mean())

    print("\n=== 3. Qaror: eng yaxshisidan sezilarli yomon bo'lmagan eng sodda ===")
    mos = []
    for c, v in natija.items():
        d = v - natija[eng]
        if c == eng or d.mean() >= -2 * d.std(ddof=1) / np.sqrt(len(d)):
            mos.append(c)
    tanlov = min(mos, key=lambda c: (c[0], -c[1]))    # kichik k, katta lambda
    print(f"  eng yaxshi {eng}, sezilarli yomon bo'lmaganlar: {len(mos)} ta")
    print(f"  tanlov (k, lambda, alfa) = {tanlov}")

    print("\n=== 4. BPR: epoxalar soni validatsiyada ===")
    kuzat = {ep: (lambda b: float(val_ndcg(b).mean())) for ep in [4, 8, 12, 16]}
    _, tarix = bpr(X2, kuzat=kuzat)
    print("  " + ", ".join(f"ep{e}: {v:.4f}" for e, v in tarix.items()))
    ep_eng = max(tarix, key=tarix.get)
    print(f"  tanlov: {ep_eng} epoxa")

    print("\n=== 5. TEST: tanlangan sozlamalar butun train da qayta o'qitiladi ===")
    X, R = matritsa(tr, n_user, n_item), kuch_matritsa(tr, n_user, n_item)
    users = np.sort(te["user"].unique())
    U, V = als(R, k=tanlov[0], lam=tanlov[1], alfa=tanlov[2], iters=6)
    model, _ = bpr(X, epoxlar=ep_eng)
    pop = np.asarray(X.sum(0)).ravel()
    usullar = {"mashhurlik": np.tile(pop, (len(users), 1)).astype(float),
               "item-kNN": item_knn(X, users), "ALS": U[users] @ V.T,
               "BPR": bpr_ball(*model)[users]}
    test = {}
    for nom, b in usullar.items():
        test[nom], _ = baholash(b, X, te, users)
    eng_t = max(test, key=lambda n: test[n]["nDCG"].mean())
    for nom in usullar:
        d = test[nom]["nDCG"] - test[eng_t]["nDCG"]
        se = d.std(ddof=1) / np.sqrt(len(d))
        print(f"  {nom:<11} nDCG {test[nom]['nDCG'].mean():.4f}, recall "
              f"{test[nom]['recall'].mean():.4f}; - {eng_t}: {d.mean():+.4f} (SE {se:.4f})")
    print("  ⭐ Sozlash - validatsiyada; test - faqat bir marta, yakunda")


if __name__ == "__main__":
    main()

=== 1. ALS: k va lambda validatsiyada (alfa = 1) ===
     k    lambda=3   lambda=10   lambda=30
     8      0.0968      0.0961      0.0917
    16      0.1028      0.1039      0.1014
    32      0.1026      0.1087      0.1054
  eng yaxshi: k=32, lambda=10

=== 2. alfa (eng yaxshi k va lambda bilan) ===
  alfa=0: val nDCG 0.1090
  alfa=1: val nDCG 0.1087
  alfa=3: val nDCG 0.1006

=== 3. Qaror: eng yaxshisidan sezilarli yomon bo'lmagan eng sodda ===
  eng yaxshi (32, 10.0, 0.0), sezilarli yomon bo'lmaganlar: 4 ta
  tanlov (k, lambda, alfa) = (16, 10.0, 1.0)

=== 4. BPR: epoxalar soni validatsiyada ===
  ep4: 0.0635, ep8: 0.0699, ep12: 0.0698, ep16: 0.0696
  tanlov: 8 epoxa

=== 5. TEST: tanlangan sozlamalar butun train da qayta o'qitiladi ===
  mashhurlik  nDCG 0.0706, recall 0.0924; - ALS: -0.0412 (SE 0.0044)
  item-kNN    nDCG 0.0961, recall 0.1234; - ALS: -0.0156 (SE 0.0035)
  ALS         nDCG 0.1118, recall 0.1348; - ALS: +0.0000 (SE 0.0000)
  BPR         nDCG 0.0803, recall 0.1035; - ALS: -0.0314 (SE 0.0040)
  ⭐ Sozlash - validatsiyada; test - faqat bir marta, yakunda

"""Sovuq start: yangi foydalanuvchi (fold-in, mashhurlik fallback) va yangi mahsulot (gibrid)."""

import numpy as np
import pandas as pd
from scipy import sparse
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import Ridge
from sklearn.metrics.pairwise import cosine_similarity

KATEGORIYA = {
    "elektronika": "telefon smartfon quloqchin zaryadlovchi noutbuk planshet "
                   "sichqoncha klaviatura kolonka televizor kamera soat",
    "kiyim": "koylak shim kurtka poyabzal krossovka shapka sharf palto "
             "kostyum futbolka yubka sviter",
    "kitob": "roman darslik detektiv sher tarix fantastika lugat ertak "
             "biografiya psixologiya biznes dasturlash",
    "oziq": "choy qahva asal yongoq shokolad guruch yog un makaron sharbat "
            "pechene ziravor",
    "gozallik": "krem atir shampun labbuyoq boyoq niqob sovun losyon taroq "
                "lak gel upa",
    "uy": "idish qozon choynak gilam parda yostiq chiroq sochiq pichoq tova "
          "vaza javon",
    "sport": "koptok gantel velosiped raketka forma gilamcha skakalka shlem "
             "sumka butsa trenajyor suzish",
    "bolalar": "oyinchoq konstruktor qogirchoq kolyaska taglik mashinka pazl "
               "albom rangli kiyimcha kitobcha samokat",
}
NARX = ["arzon", "orta", "premium"]
UMUMIY = "sifatli yangi chegirma original mashhur qulay kafolat tez yetkazish".split()


def dokon(seed=0, n_user=2000, n_item=800, kunlar=180):
    """Sintetik onlayn do'kon: yashirin didlar, mashhurlik dumi, yangi mahsulotlar."""
    rng = np.random.default_rng(seed)
    katlar = list(KATEGORIYA)
    n_kat, n_sub = len(katlar), 4
    sub = rng.integers(0, n_kat * n_sub, n_item)          # har mahsulot kichik guruhi
    kat = sub // n_sub
    narx = rng.choice(3, n_item, p=[0.5, 0.35, 0.15])
    sifat = rng.lognormal(0, 1.4, n_item)                  # mashhurlik dumi manbai
    chiqish = np.where(rng.random(n_item) < 0.7, 0,
                       rng.integers(0, kunlar - 20, n_item))
    tavsif = []
    for i in range(n_item):
        sozlar = KATEGORIYA[katlar[kat[i]]].split()
        guruh = sozlar[(sub[i] % n_sub) * 3:(sub[i] % n_sub) * 3 + 3]
        tanlov = list(rng.choice(guruh, 2, replace=False))
        tanlov.append(rng.choice([s for s in sozlar if s not in tanlov]))
        tanlov += [katlar[kat[i]], NARX[narx[i]], f"brend{rng.integers(0, 40)}"]
        tanlov += list(rng.choice(UMUMIY, 2, replace=False))
        tavsif.append(" ".join(tanlov))
    # foydalanuvchi didi: kategoriya x kichik guruh x narx
    kat_pref = rng.dirichlet(np.full(n_kat, 0.3), n_user)
    sub_pref = rng.dirichlet(np.full(n_sub, 0.5), (n_user, n_kat))
    narx_pref = rng.dirichlet(np.full(3, 1.0), n_user)
    did = (kat_pref[:, kat] * sub_pref[:, kat, sub % n_sub]
           * narx_pref[:, narx])                           # (user, item)
    did /= did.sum(1, keepdims=True)
    hafta = np.arange(0, kunlar, 7)
    yosh = np.clip((hafta[:, None] - chiqish[None, :]) / 7, 0, None)
    jalb = sifat * (0.6 + 1.2 * np.exp(-yosh / 6)) * (hafta[:, None] >= chiqish)
    qatorlar = []
    for u in range(n_user):
        n_u = int(np.clip(rng.lognormal(2.8, 0.8), 3, 150))
        vaqtlar = np.sort(rng.uniform(0, kunlar, n_u))
        korgan = np.zeros(n_item, dtype=bool)
        for t in vaqtlar:
            a = jalb[int(t) // 7] * ~korgan
            p = 0.75 * did[u] * a / (did[u] @ a) + 0.25 * a / a.sum()
            i = int(np.searchsorted(np.cumsum(p), rng.random() * p.sum()))
            korgan[i] = True
            mos = did[u, i] / did[u].max()
            xarid = rng.random() < 0.15 + 0.35 * mos
            baho = 0
            if xarid and rng.random() < 0.3:
                baho = int(np.clip(round(2.4 + 2.2 * mos + 0.3 * np.log(sifat[i])
                                         + rng.normal(0, 0.7)), 1, 5))
            qatorlar.append((u, i, t, int(xarid), baho))
    df = pd.DataFrame(qatorlar, columns=["user", "item", "vaqt", "xarid", "baho"])
    items = pd.DataFrame({"item": np.arange(n_item), "kategoriya": np.array(katlar)[kat],
                          "sub": sub, "narx": narx, "chiqish": chiqish,
                          "tavsif": tavsif})
    return df, items


def bolish(df, usul="vaqt", ulush=0.2, seed=0, chegara=150.0):
    """'vaqt' - har foydalanuvchining oxirgi 20% i test; 'tasodifiy'; 'global'."""
    df = df.sort_values(["user", "vaqt"]).reset_index(drop=True)
    if usul == "global":
        tarix = df[df["vaqt"] < chegara].groupby("user").size()
        yetarli = df["user"].map(tarix).fillna(0).to_numpy() >= 3
        test = (df["vaqt"] >= chegara).to_numpy() & yetarli
        return df[df["vaqt"] < chegara].copy(), df[test].copy()
    n = df.groupby("user")["item"].transform("size").to_numpy()
    if usul == "tasodifiy":
        kalit = np.random.default_rng(seed).random(len(df))
        tartib = (df.assign(k=kalit).groupby("user")["k"]
                  .rank(method="first").to_numpy() - 1)
    else:
        tartib = df.groupby("user").cumcount().to_numpy()
    n_test = np.maximum(1, np.round(ulush * n)).astype(int)
    test = (tartib >= n - n_test) & (n >= 5)
    return df[~test].copy(), df[test].copy()


def matritsa(qism, n_user, n_item):
    return sparse.csr_matrix((np.ones(len(qism)), (qism["user"], qism["item"])),
                             shape=(n_user, n_item))


def metrikalar(top, T):
    """top: (n, k) tavsiya indekslari; T: (n, n_item) test to'plami (bool)."""
    k = top.shape[1]
    rel = np.take_along_axis(T, top, axis=1)
    n_rel = T.sum(1)
    pozitsiya = np.arange(1, k + 1)
    ap = (np.cumsum(rel, 1) / pozitsiya * rel).sum(1) / np.minimum(n_rel, k)
    chegirma = 1 / np.log2(pozitsiya + 1)
    idcg = np.array([chegirma[:min(r, k)].sum() for r in n_rel])
    return {"precision": rel.sum(1) / k, "recall": rel.sum(1) / n_rel,
            "hit": rel.any(1).astype(float), "MAP": ap,
            "nDCG": (rel * chegirma).sum(1) / idcg}


def baholash(ball, X_train, test, users, k=10):
    """Train da ko'rilgan mahsulotlar tavsiya qilinmaydi; har foydalanuvchi metrikasi."""
    ball = np.array(ball, dtype=float)
    ball[X_train[users].toarray() > 0] = -np.inf
    top = np.argsort(-ball, axis=1, kind="stable")[:, :k]
    qator = pd.Series(np.arange(len(users)), index=users)
    tt = test[test["user"].isin(users)]
    T = np.zeros(ball.shape, dtype=bool)
    T[qator[tt["user"]].to_numpy(), tt["item"].to_numpy()] = True
    return metrikalar(top, T), top


def als_qadam(R, Y, lam, alfa):
    """Hamma qatorlar uchun bir yo'la: (Y^T Y + Y^T (C_u - I) Y + lam*I) x_u = Y^T C_u p_u.

    Y^T Y bitta marta hisoblanadi va HAMMA kataklarni (nollarni ham, c = 1 bilan)
    qamraydi; qo'shimcha had sum_i (c_ui - 1) y_i y_i^T faqat nol bo'lmagan
    kataklar ustida - siyrak matritsa x "tashqi ko'paytmalar" jadvali.
    """
    k = Y.shape[1]
    YtY = Y.T @ Y + lam * np.eye(k)
    tashqi = (Y[:, :, None] * Y[:, None, :]).reshape(len(Y), k * k)
    W = R.copy()
    W.data = alfa * R.data                           # c - 1
    A = YtY + (W @ tashqi).reshape(-1, k, k)
    W.data = 1 + alfa * R.data                       # c (p = 1 kuzatilganda)
    return np.linalg.solve(A, (W @ Y)[:, :, None])[:, :, 0]


def als(R, k=32, lam=10.0, alfa=10.0, iters=10, seed=0):
    """Implicit ALS (Hu, Koren, Volinsky 2008): ishonch c = 1 + alfa * r."""
    rng = np.random.default_rng(seed)
    R = R.tocsr()
    Rt = R.T.tocsr()
    V = rng.normal(0, 0.1, (R.shape[1], k))
    for _ in range(iters):
        U = als_qadam(R, V, lam, alfa)
        V = als_qadam(Rt, U, lam, alfa)
    return U, V


def kuch_matritsa(qism, n_user, n_item):
    """r = 1 (ko'rish) yoki 3 (xarid) - implicit signal kuchi."""
    return sparse.csr_matrix((1.0 + 2.0 * qism["xarid"].to_numpy(),
                              (qism["user"], qism["item"])), shape=(n_user, n_item))


def item_knn(X, users, k=50):
    norma = np.sqrt(np.asarray(X.multiply(X).sum(0)).ravel())
    Xn = X @ sparse.diags(1 / np.maximum(norma, 1e-12))
    S = (Xn.T @ Xn).toarray()
    np.fill_diagonal(S, 0)
    kesish = np.argpartition(-S, k, axis=1)[:, k:]
    np.put_along_axis(S, kesish, 0, axis=1)
    return X[users] @ S


def juft(a, b):
    d = a - b
    return d.mean(), d.std(ddof=1) / np.sqrt(len(d))


def main() -> None:
    df, items = dokon()
    n_user, n_item = 2000, len(items)
    k, lam, alfa = 16, 10.0, 1.0                  # 3-misolda validatsiyada tanlangan
    rng = np.random.default_rng(5)

    print("=== 1. Yangi foydalanuvchilar: 400 tasi o'qitishdan butunlay chiqarildi ===")
    faol = df.groupby("user").size()
    yangi_u = np.sort(rng.choice(faol[faol >= 15].index.to_numpy(), 400, replace=False))
    eski = df[~df["user"].isin(yangi_u)]
    R = kuch_matritsa(eski, n_user, n_item)
    X = matritsa(eski, n_user, n_item)
    U, V = als(R, k=k, lam=lam, alfa=alfa, iters=6)
    pop = np.asarray(X.sum(0)).ravel()
    norma = np.sqrt(pop)
    Xn = X @ sparse.diags(1 / np.maximum(norma, 1e-12))
    S = (Xn.T @ Xn).toarray()
    np.fill_diagonal(S, 0)
    yd = df[df["user"].isin(yangi_u)].sort_values(["user", "vaqt"])
    tartib = yd.groupby("user").cumcount().to_numpy()
    print(f"  ALS o'qitildi: {len(eski)} ta'sir; yangi foydalanuvchilarda "
          f"{len(yd)} ta'sir (o'qitishda yo'q)")
    print(f"  {'ma_lum n':>9} {'mashhurlik':>11} {'item-kNN':>9} {'ALS fold-in':>12} "
          f"{'ALS - mashh.':>13} {'SE':>7}")
    natija = {}
    for n in [0, 1, 3, 5, 10]:
        malum, test = yd[tartib < n], yd[tartib >= n]
        Xm = matritsa(malum, n_user, n_item)
        Rm = kuch_matritsa(malum, n_user, n_item)
        fold = als_qadam(Rm[yangi_u], V, lam, alfa)   # V qotirilgan, faqat x_u yechiladi
        r_pop, _ = baholash(np.tile(pop, (400, 1)).astype(float), Xm, test, yangi_u)
        r_knn, _ = baholash(Xm[yangi_u] @ S, Xm, test, yangi_u)
        r_als, _ = baholash(fold @ V.T, Xm, test, yangi_u)
        natija[n] = (r_pop["nDCG"], r_als["nDCG"])
        m, se = juft(r_als["nDCG"], r_pop["nDCG"])
        print(f"  {n:>9} {r_pop['nDCG'].mean():>11.4f} {r_knn['nDCG'].mean():>9.4f} "
              f"{r_als['nDCG'].mean():>12.4f} {m:>+13.4f} {se:>7.4f}")
    chegara = None
    for n in [1, 3, 5, 10]:
        m, se = juft(natija[n][1], natija[n][0])
        if chegara is None and m > 2 * se:
            chegara = n
    print(f"  ALS mashhurlikdan sezilarli yaxshi bo'ladigan eng kichik n: {chegara}")
    print(f"  qoida: n < {chegara} -> mashhurlik, aks holda ALS fold-in")

    print("\n=== 2. Yangi mahsulotlar: 80 tasining barcha ta'sirlari yashirildi ===")
    yangi_i = np.sort(rng.choice(n_item, 80, replace=False))
    iliq = df[~df["item"].isin(yangi_i)]
    sovuq = df[df["item"].isin(yangi_i)]
    R = kuch_matritsa(iliq, n_user, n_item)
    Xw = matritsa(iliq, n_user, n_item)
    U, V = als(R, k=k, lam=lam, alfa=alfa, iters=6)
    tfidf = TfidfVectorizer().fit_transform(items["tavsif"])
    iliq_i = np.setdiff1d(np.arange(n_item), yangi_i)
    ridge = Ridge(alpha=1.0).fit(tfidf[iliq_i], V[iliq_i])
    V_taxmin = ridge.predict(tfidf[yangi_i])
    print(f"  CF uchun yangi mahsulot vektori: nol (ALS da ta'sir yo'q) -> "
          f"|v| = {np.abs(V[yangi_i]).max():.1e}")
    print(f"  Ridge (TF-IDF -> ALS faktorlari) iliq mahsulotlarda R^2 = "
          f"{ridge.score(tfidf[iliq_i], V[iliq_i]):.3f}")
    us = np.sort(sovuq["user"].unique())
    us = us[np.asarray(Xw[us].sum(1)).ravel() >= 3]
    blok = np.full((len(us), n_item), -np.inf)      # faqat yangi mahsulotlar orasida
    usullar = {
        "tasodifiy": rng.random((len(us), len(yangi_i))),
        "mashhurlik (kategoriya)": np.tile(
            iliq.merge(items[["item", "kategoriya"]])["kategoriya"].value_counts()
            .reindex(items.loc[yangi_i, "kategoriya"]).to_numpy(float), (len(us), 1)),
        "kontent (TF-IDF)": cosine_similarity(Xw[us] @ tfidf, tfidf[yangi_i]),
        "gibrid (TF-IDF -> ALS)": U[us] @ V_taxmin.T,
    }
    test = sovuq[sovuq["user"].isin(us)]
    print(f"  {len(us)} foydalanuvchi, {len(test)} ta yangi mahsulot ta'siri; "
          f"har biriga 80 ta ichidan top-10")
    r = {}
    for nom, b in usullar.items():
        ball = blok.copy()
        ball[:, yangi_i] = b
        r[nom], _ = baholash(ball, Xw, test, us)
    eng = max(r, key=lambda n: r[n]["nDCG"].mean())
    for nom in usullar:
        m, se = juft(r[nom]["nDCG"], r[eng]["nDCG"])
        print(f"  {nom:<24} nDCG {r[nom]['nDCG'].mean():.4f}, hit "
              f"{r[nom]['hit'].mean():.3f}; - eng yaxshi: {m:+.4f} (SE {se:.4f})")
    print(f"  ⭐ Yangi mahsulotga CF o'zi yetmaydi - kontent kerak; eng yaxshisi: {eng}")


if __name__ == "__main__":
    main()

=== 1. Yangi foydalanuvchilar: 400 tasi o'qitishdan butunlay chiqarildi ===
  ALS o'qitildi: 30760 ta'sir; yangi foydalanuvchilarda 14259 ta'sir (o'qitishda yo'q)
   ma_lum n  mashhurlik  item-kNN  ALS fold-in  ALS - mashh.      SE
          0      0.3338    0.0554       0.0554       -0.2784  0.0098
          1      0.3199    0.2943       0.3250       +0.0052  0.0126
          3      0.2986    0.3837       0.3533       +0.0547  0.0111
          5      0.2716    0.3668       0.3428       +0.0712  0.0105
         10      0.2270    0.3222       0.3194       +0.0924  0.0096
  ALS mashhurlikdan sezilarli yaxshi bo'ladigan eng kichik n: 3
  qoida: n < 3 -> mashhurlik, aks holda ALS fold-in

=== 2. Yangi mahsulotlar: 80 tasining barcha ta'sirlari yashirildi ===
  CF uchun yangi mahsulot vektori: nol (ALS da ta'sir yo'q) -> |v| = 0.0e+00
  Ridge (TF-IDF -> ALS faktorlari) iliq mahsulotlarda R^2 = 0.495
  1525 foydalanuvchi, 4553 ta yangi mahsulot ta'siri; har biriga 80 ta ichidan top-10
  tasodifiy                nDCG 0.0766, hit 0.301; - eng yaxshi: -0.1860 (SE 0.0075)
  mashhurlik (kategoriya)  nDCG 0.1810, hit 0.378; - eng yaxshi: -0.0816 (SE 0.0095)
  kontent (TF-IDF)         nDCG 0.2626, hit 0.646; - eng yaxshi: +0.0000 (SE 0.0000)
  gibrid (TF-IDF -> ALS)   nDCG 0.2210, hit 0.578; - eng yaxshi: -0.0416 (SE 0.0057)
  ⭐ Yangi mahsulotga CF o'zi yetmaydi - kontent kerak; eng yaxshisi: kontent (TF-IDF)

rek = TruncatedSVD(10).fit_transform(R) @ svd.components_          # ⚠️ nol = baho 0
# faqat kuzatilgan kataklar ustida bias li MF (SGD yoki ALS)          # ✅

A = Yi.T @ (c[:, None] * Yi) + lam * np.eye(k)                     # ⚠️ Y^T Y yo'q
A = Y.T @ Y + Yi.T @ ((c - 1)[:, None] * Yi) + lam * np.eye(k)     # ✅ hamma katak

r_hat = np.sum(P[u] * Q[i], 1)                                     # ⚠️
r_hat = mu + bu[u] + bi[i] + np.sum(P[u] * Q[i], 1)                # ✅

mf_sgd(tr, reg=0.0, epoxlar=40)                                    # ⚠️ yodlaydi
mf_sgd(tr, reg=0.05)            # ✅ reg va epoxalar - validatsiyada

k = max([8, 16, 32], key=lambda k: ndcg_test(k))                   # ⚠️
tr2, va = bolish(tr, "vaqt")                                       # ✅ train ichidan

top = np.argsort(-(x_u @ V.T))[:10]          # ⚠️ x_u = 0 -> tasodifiy tartib
top = mashhurlik_top if n_u < 3 else cf_top   # ✅ chegara o'lchangan

ball = U @ V.T                                   # ⚠️ yangi mahsulot v = 0
ball_yangi = cosine_similarity(profil, tfidf[yangi])   # ✅ kontent / exploration

1.  # 2000 foydalanuvchi, 800 mahsulot, k=16 - MF parametrlari soni?
2.  # siyrak baholar matritsasiga TruncatedSVD - test bashoratlari qanday bo'ladi?
3.  # bias li modelda b_u = -0.4 nimani bildiradi?
4.  # reg = 0 bilan uzoq SGD - train va test RMSE?
5.  # implicit ALS da bo'sh katakning p va c qiymati?
6.  # faqat kuzatilgan kataklar bilan implicit model nimani o'rganadi?
7.  # Y^T Y hiylasida bitta foydalanuvchi uchun hisob narxi nimaga bog'liq?
8.  # alfa juda katta bo'lsa?
9.  # BPR yo'qotishi qaysi metrikaning silliq yaqinlashuvi?
10. # yangi foydalanuvchi (n=0) uchun ALS bali?
11. # yangi mahsulotning ALS vektori (lambda > 0)?
12. # ikki bosqichli tizimda birinchi bosqich qaysi metrika bilan baholanadi?

1.  svd = TruncatedSVD(10).fit(R_baholar); r_hat = svd.transform(R) @ svd.components_

2.  A = (Yi.T * c) @ Yi + lam * np.eye(k)        # implicit ALS

3.  r_hat = mu + np.sum(P[u] * Q[i], 1)

4.  k = max([8, 16, 32], key=lambda k: ndcg(test, als(R, k=k)))

5.  tavsiya = np.argsort(-(U[yangi] @ V.T))[:10]   # yangi foydalanuvchi, tarix yo'q