23.12-dars: Sentiment tahlili

VAZIFA:
  kirish: sharh matni          chiqish: salbiy / neytral / ijobiy
  3 sinf, nomutanosib: ijobiy 44%, salbiy 35%, neytral 21%

SINTETIK KORPUS (6000 sharh, seed bilan):
  jihat + sifat:        "narxi arzon", "kuryer imillagan"
  fe'l:                 "menga yoqdi", "ikki kunda buzildi"
  faktlar (fikrsiz):    "buyurtma juma kuni keldi", "rangi qora"
  shovqin:              apostrof variantlari, 4% so'zda imlo xatosi,
                        3% tasodifiy yorliq xatosi

GURUHLAR (har sharhda bittasi):
  oddiy     "sotuvchi xushmuomala, narxi qulay"          -> ijobiy
  yumshoq   "yomon emas", "biroz kechikdi"               -> ijobiy/salbiy
            (annotatorlar 25% holda "neytral" deb qo'ygan)
  inkor     "sifati a'lo emas", "menga yoqmadi"          -> teskari qutb
  kontrast  "narxi arzon, lekin sifati chala"            -> 2-qism qutbi
  aralash   "sifati chala va bahosi hamyonbop"           -> neytral
  neytral   "hali ishlatib ko'rmadim", faktlar           -> neytral

LUG'AT BAZAVIYSI:
  IJOBIY_ILDIZ = ["yaxshi", "zo'r", "arzon", "yoq", ...]      (12 ta)
  SALBIY_ILDIZ = ["yomon", "qimmat", "kech", "buzil", ...]    (12 ta)
  ball = (ijobiy ildizlar soni) - (salbiy ildizlar soni)
  ball > 0 -> ijobiy,  ball < 0 -> salbiy,  ball = 0 -> neytral

QOIDA 1 - INKOR:
  keyingi so'z "emas"            -> ishora teskari   ("yaxshi emas")
  ildizdan keyin "ma..."         -> ishora teskari   ("yoq-ma-di")
QOIDA 2 - KONTRAST:
  oxirgi "lekin / ammo / biroq" dan keyingi qism hal qiladi

NIMADA YIQILADI:
  "kech-ik-ma-sdan keldi"     -> "-ma-" ildizdan keyin emas, ichkarida
  "tavsiya qil-ma-yman"       -> inkor boshqa so'zda
  "operator e'tiborsiz"       -> so'z lug'atda yo'q (qamrov)
  "sifati chala va narxi arzon" (aralash) -> ball +1, neytral emas

TfidfVectorizer(preprocessor=normallashtir, ngram_range=(1, 2),
                sublinear_tf=True, min_df=2)
LogisticRegression(C=5.0)       # vektorizator FAQAT o'quvda fit qilinadi

UNIGRAM (1,1):
  "sifati yaxshi emas" = {sifati, yaxshi, emas}
  "yaxshi" -> ijobiy vazn, "emas" -> ikkala sinfda uchraydi
  => inkorni faqat qisman ushlaydi

BIGRAM (1,2):
  + {"sifati yaxshi", "yaxshi emas"}
  "yaxshi emas" - alohida belgi, o'z vazni bilan
  => inkor = yangi so'z

KONTRAST - TARTIB MUAMMOSI:
  "narxi arzon lekin sifati yomon"   -> salbiy
  "sifati yomon lekin narxi arzon"   -> ijobiy
  unigram xaltasi IKKALASIDA BIR XIL -> bir xil bashorat
  bigram: "lekin sifati" va "lekin narxi" farqlanadi -> qisman yordam

LUG'AT 23.2-bob: faqat o'quvdan, min_son=2; 0 - <pad>, 1 - <unk>
  kodla: matn -> [indekslar], uzunliklar L

EmbeddingBag(V, 32, mode="mean")  -> Linear(32, 3)
  = neyron so'z xaltasi: embeddinglar o'rtachasi, TARTIB YO'Q
  unigram TF-IDF bilan bir oilada

Embedding(V, 32) -> LSTM(32, 48) -> oxirgi holat h[-1] -> Linear(48, 3)
  pack_padded_sequence(..., enforce_sorted=False)   23.9-bob
  -> h[-1] - HAQIQIY oxirgi tokendan keyingi holat, pad dan emas
  chapdan o'ngga o'qiydi: "lekin" dan keyingi qism oxirgi
  holatga eng yaqin -> kontrast uchun tabiiy

O'RGATISH:
  AdamW(lr=5e-3), batch 128, 10 davr
  eng yaxshi val aniqlik -> copy.deepcopy(state_dict()) (21.5)

GURUH BO'YICHA ANIQLIK (3-misol, test, 1200 sharh):
                 hamma  oddiy yumshoq inkor kontrast aralash neytral
  lug'at+qoida   0.632  0.686  0.574  0.438  0.594   0.421   0.979
  TF-IDF (1,2)   0.845  0.888  0.694  0.934  0.866   0.316   0.937
  EmbeddingBag   0.750  0.814  0.611  0.810  0.708   0.092   0.972
  LSTM           0.906  0.928  0.694  0.942  0.946   0.737   0.972

JUFTLASHGAN FARQ (bir xil test sharhlari):
  d_i = [LSTM to'g'ri] - [TF-IDF to'g'ri]     (har sharh uchun -1, 0 yoki 1)
  farq = mean(d),  SE = std(d) / sqrt(n)
  |farq| > 2 * SE -> sezilarli

NEYTRAL NEGA QIYIN:
  1) "fikrsiz" matn (faktlar) - oson
  2) aralash: ijobiy + salbiy so'zlar, lekin umumiy qutb yo'q - qiyin
  3) yumshoq ifodalar: annotatorlar kelishmaydi - yorliqning o'zi shovqinli
  4) kam sinf (21%) - model uni "chetlab o'tishga" moyil

KALIBRATSIYA - ishonch rostmi?
  ishonch = max softmax ehtimoli
  kalibrlangan model: ishonch 0.9 bo'lgan sharhlarning ~90% i to'g'ri
  ECE = sum_b (n_b / n) * |o'rtacha ishonch_b - aniqlik_b|

HARORAT BILAN KALIBRLASH (temperature scaling):
  p = softmax(logit / T)
  T validatsiyadagi NLL ni minimallashtiradi   (T > 1 -> ishonch pasayadi)
  argmax o'zgarmaydi -> aniqlik o'zgarmaydi, faqat ishonch "rostlanadi"

ISHLATILISHI:
  ishonch < chegara -> odamga (qo'lda ko'rish navbati)

matnlar, y, g = korpus(6000, seed=42)                  # g - guruh belgisi
tr, va, te = bolish(y)                                 # stratifikatsiya, test qulf
p = [lugat_bashorat(m, inkor=True, kontrast=True) for m in matnlar[te]]
v = TfidfVectorizer(preprocessor=normallashtir, ngram_range=(1, 2),
                    sublinear_tf=True, min_df=2)
lr = LogisticRegression(C=5.0, max_iter=3000).fit(v.fit_transform(matnlar[tr]), y[tr])
lug = Lugat(matnlar[tr]); x, L = lug.kodla(matnlar[te])        # faqat o'quvdan
model = LSTMModel(len(lug)); orgat(model, D["tr"], D["va"])     # deepcopy
for gr in GURUHLAR:                                    # guruh bo'yicha aniqlik
    print(gr, np.mean(p[g[te] == gr] == y[te][g[te] == gr]))
d = (p_lstm == y_te).astype(float) - (p_tfidf == y_te)   # juftlashgan farq
T = argmin_T NLL(logit_val / T); ehtimol = softmax(logit_test / T)

normallashtirish -> lug'at bazaviysi (pastki chegara)
guruh belgisi: oddiy / yumshoq / inkor / kontrast / aralash / neytral
TF-IDF bigram: inkorni ushlaydi, kontrastni qisman
EmbeddingBag = neyron so'z xaltasi; LSTM = tartib -> kontrast, aralash
guruh jadvali + juftlashgan farq (2*SE) -> yutuq qayerda va haqiqiymi
neytral eng qiyin; harorat bilan kalibrlash; past ishonch -> odamga
sarkazm o'lchanmaydi - buni ochiq ayting

"""Sintetik sharhlar korpusi, normallashtirish va lug'atga asoslangan bazaviy."""

import re
from collections import Counter

import numpy as np

SINFLAR = ["salbiy", "neytral", "ijobiy"]
GURUHLAR = ["oddiy", "yumshoq", "inkor", "kontrast", "aralash", "neytral"]
ULUSH = [0.36, 0.09, 0.20, 0.17, 0.06, 0.12]
APOSTROFLAR = ["'", "'", "'", "\u02bb", "\u2019", "`"]
ASPEKTLAR = [   # (jihat nomlari, ijobiy sifatlar, salbiy sifatlar)
    (["sifati", "materiali", "ishlanishi"],
     ["yaxshi", "zo'r", "a'lo", "ajoyib", "mustahkam", "puxta"],
     ["yomon", "past", "bo'sh", "nochor", "xom", "chala"]),
    (["narxi", "bahosi"], ["arzon", "hamyonbop", "qulay", "maqbul"],
     ["qimmat", "baland", "osmonda", "haddan tashqari"]),
    (["yetkazib berish", "dostavka", "kuryer"],
     ["tez", "o'z vaqtida", "chaqqon", "tezkor"],
     ["sekin", "kech", "imillagan", "sust"]),
    (["xizmat", "sotuvchi", "operator"],
     ["yaxshi", "xushmuomala", "a'lo", "e'tiborli"],
     ["yomon", "qo'pol", "e'tiborsiz", "beparvo"]),
    (["qadog'i", "o'rami"], ["chiroyli", "butun", "puxta", "ozoda"],
     ["yirtilgan", "ezilgan", "yomon", "iflos"]),
]
KUCHAYTIR = ["", "", "", "juda ", "ancha ", "rosa "]
FEL_IJOBIY = [("menga yoqdi", "menga yoqmadi"),       # (asl, -ma- shakli)
              ("tavsiya qilaman", "tavsiya qilmayman"),
              ("yana buyurtma beraman", "boshqa buyurtma bermayman"),
              ("mamnun qoldim", "mamnun qolmadim"),
              ("pulimga arzidi", "pulimga arzimadi"),
              ("kutganimdek chiqdi", "kutganimdek chiqmadi")]
FEL_SALBIY = [("ikki kunda buzildi", "bir oyda ham buzilmadi"),
              ("pulimga achindim", "pulimga achinmadim"),
              ("afsuslandim", "afsuslanmadim"),
              ("hafsalam pir bo'ldi", "hafsalam pir bo'lmadi"),
              ("kechikib keldi", "kechikmasdan keldi"),
              ("qaytarib berdim", "qaytarishga hojat bo'lmadi")]
YUMSHOQ = {1: ["yomon emas", "chidasa bo'ladi", "umuman olganda normal",
               "narxiga arziydi"],
           -1: ["biroz kechikdi", "kutganimdan sal pastroq", "unchalik emas",
                "ozgina hafsalam pir bo'ldi"]}
FAKTLAR = ["buyurtma {kun} kuni keldi", "rangi {rang}", "o'lchami {olcham}",
           "sovg'a uchun oldim", "qutisida yo'riqnoma bor",
           "narxi {son} ming so'm", "{shahar}ga yetkazishdi",
           "onamga olib berdim", "ikki dona oldim"]
NEYTRAL_FIKR = ["hali ishlatib ko'rmadim", "oddiy mahsulot", "sifati o'rtacha",
                "narxiga yarasha", "na yaxshi, na yomon", "hozircha fikr yo'q",
                "yaxshi ham emas, yomon ham emas", "boshqasidan farqi yo'q",
                "yaxshi yoki yomonligini keyin bilaman"]
TOLDIR = {"kun": ["dushanba", "seshanba", "juma", "shanba"],
          "rang": ["qora", "oq", "ko'k", "kulrang", "qizil"],
          "olcham": ["o'rtacha", "katta", "kichik", "standart"],
          "son": ["120", "85", "240", "56", "310"],
          "shahar": ["Toshkent", "Samarqand", "Buxoro", "Namangan", "Qarshi"]}
BOGLOVCHI = ["lekin", "ammo", "biroq"]


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def fakt(rng):
    s = tanla(rng, FAKTLAR)
    for k, v in TOLDIR.items():
        s = s.replace("{" + k + "}", tanla(rng, v))
    return s


def birlik(rng, qutb, inkor=False, jihat=None):
    """Bitta fikr bo'lagi; qutb (+1/-1) - bo'lakning HOSIL bo'lgan ma'nosi.
    jihat: None - tasodifiy, son - shu jihat sifati, "fel" - fe'lli bo'lak."""
    if jihat is None:
        jihat = int(rng.integers(len(ASPEKTLAR))) if rng.random() < 0.55 else "fel"
    if jihat != "fel":                               # "narxi arzon"
        nomlar, ij, sl = ASPEKTLAR[jihat]
        nom = tanla(rng, nomlar)
        if not inkor:
            return f"{nom} {tanla(rng, KUCHAYTIR)}{tanla(rng, ij if qutb > 0 else sl)}"
        oldi = "unchalik " if qutb < 0 and rng.random() < 0.3 else ""
        return f"{nom} {oldi}{tanla(rng, sl if qutb > 0 else ij)} emas"
    ij, sl = tanla(rng, FEL_IJOBIY), tanla(rng, FEL_SALBIY)
    if not inkor:
        return (ij if qutb > 0 else sl)[0]
    return (sl if qutb > 0 else ij)[1]               # "-ma-": buzilmadi -> ijobiy


def ikki_jihat(rng, q1, q2, inkor2=False):
    """Ikki xil jihat haqida ikki bo'lak (masalan narx va sifat)."""
    i, j = rng.choice(len(ASPEKTLAR), size=2, replace=False)
    ikkinchi = int(j) if rng.random() < 0.6 else "fel"
    return birlik(rng, q1, jihat=int(i)), birlik(rng, q2, inkor2, ikkinchi)


def yozuvga(rng, bolaklar):
    matn = ""
    for i, b in enumerate(bolaklar):
        if i == 0:
            matn = b[0].upper() + b[1:]
        elif b.split()[0] in BOGLOVCHI:
            matn += ", " + b
        else:
            ayir = tanla(rng, [". ", ", ", ". ", " va "])
            matn += ayir + (b[0].upper() + b[1:] if ayir == ". " else b)
    return matn + tanla(rng, [".", ".", "!", "", "!!"])


def buz(rng, matn, p_imlo):
    """Apostrof variantlari va imlo xatolari (tushgan, ikkilangan, almashgan harf)."""
    sozlar = []
    for s in matn.split(" "):
        if "'" in s:
            s = s.replace("'", tanla(rng, APOSTROFLAR))
        if len(s) >= 5 and rng.random() < p_imlo:
            j, amal = int(rng.integers(2, len(s) - 1)), int(rng.integers(3))
            if amal == 0:
                s = s[:j] + s[j + 1:]
            elif amal == 1:
                s = s[:j] + s[j] + s[j:]
            else:
                s = s[:j - 1] + s[j] + s[j - 1] + s[j + 1:]
        sozlar.append(s)
    return " ".join(sozlar)


def sharh(rng):
    g = int(rng.choice(len(GURUHLAR), p=ULUSH))
    qutb = 1 if rng.random() < 0.55 else -1
    faktlar = [fakt(rng) for _ in range(int(rng.integers(0, 3)))]
    yorliq = 2 if qutb > 0 else 0
    if g == 0:                                       # oddiy
        bol = [birlik(rng, qutb) for _ in range(int(rng.integers(1, 3)))]
    elif g == 1:                                     # yumshoq: "yomon emas"
        bol = [tanla(rng, YUMSHOQ[qutb])]
        if rng.random() < 0.25:                      # annotatorlar kelisha olmaydi
            yorliq = 1
    elif g == 2:                                     # inkor: "emas", "-ma-"
        bol = [birlik(rng, qutb, inkor=True)
               for _ in range(int(rng.integers(1, 3)))]
    elif g == 3:                                     # kontrast: 2-qism hal qiladi
        a, b = ikki_jihat(rng, -qutb, qutb, inkor2=rng.random() < 0.25)
        bol, faktlar = [a, tanla(rng, BOGLOVCHI) + " " + b], faktlar[:1]
        return yozuvga(rng, faktlar + bol), yorliq, g
    elif g == 4:                                     # aralash: +/- teng -> neytral
        a, b = ikki_jihat(rng, qutb, -qutb)
        bol, yorliq = [a + tanla(rng, [", ", " va "]) + b], 1
    else:                                            # neytral
        bol = [tanla(rng, NEYTRAL_FIKR)] if rng.random() < 0.7 else []
        faktlar = faktlar or [fakt(rng)]
        yorliq = 1
    bol += faktlar
    rng.shuffle(bol)
    return yozuvga(rng, bol), yorliq, g


def korpus(n, seed, p_imlo=0.04, p_shovqin=0.03):
    """Sintetik o'zbekcha sharhlar: matnlar, yorliqlar, guruhlar."""
    rng = np.random.default_rng(seed)
    matnlar, y, guruh = [], [], []
    for _ in range(n):
        m, yl, g = sharh(rng)
        if rng.random() < p_shovqin:                # tasodifiy yorliq xatosi
            yl = int(rng.integers(3))
        matnlar.append(buz(rng, m, p_imlo))
        y.append(yl)
        guruh.append(GURUHLAR[g])
    return np.array(matnlar, dtype=object), np.array(y), np.array(guruh)


def normallashtir(s):
    s = s.lower()
    for v in ("\u02bb", "\u2019", "\u2018", "\u02bc", "`"):
        s = s.replace(v, "'")
    s = re.sub(r"[^a-z0-9' ]+", " ", s)
    return " ".join(s.split())


IJOBIY_ILDIZ = ["yaxshi", "zo'r", "a'lo", "ajoyib", "arzon", "qulay", "tez",
                "chiroyli", "yoq", "tavsiya", "mamnun", "arzi"]
SALBIY_ILDIZ = ["yomon", "past", "qimmat", "sekin", "kech", "qo'pol", "buzil",
                "achin", "afsuslan", "hafsala", "qaytar", "iflos"]
ILDIZLAR = [(s, 1) for s in IJOBIY_ILDIZ] + [(s, -1) for s in SALBIY_ILDIZ]


def lugat_bashorat(matn, inkor=False, kontrast=False):
    """Lug'at bazaviysi: ildiz mos kelsa +1/-1; ball ishorasi -> sinf."""
    t = normallashtir(matn).split()
    if kontrast:                        # oxirgi "lekin" dan keyingi qism hal qiladi
        for j in range(len(t) - 1, -1, -1):
            if t[j] in BOGLOVCHI:
                t = t[j + 1:]
                break
    ball = 0
    for i, soz in enumerate(t):
        for ildiz, q in ILDIZLAR:
            if soz.startswith(ildiz):
                if inkor and (soz[len(ildiz):].startswith("ma")
                              or t[i + 1:i + 2] == ["emas"]):
                    q = -q              # "yoqmadi", "yaxshi emas"
                ball += q
                break
    return 2 if ball > 0 else 0 if ball < 0 else 1


def main() -> None:
    matnlar, y, g = korpus(6000, seed=42)
    print("=== 1. Korpus ===")
    print(f"  sharhlar: {len(y)}")
    for k, s in enumerate(SINFLAR):
        print(f"    {s:<8} {np.sum(y == k):>5}  ({np.mean(y == k):.1%})")
    print("  guruhlar: " + ", ".join(f"{gr} {np.sum(g == gr)}" for gr in GURUHLAR))

    print("\n=== 2. Har guruhdan namuna (normallashtirilgan) ===")
    for gr in GURUHLAR:
        i = int(np.flatnonzero(g == gr)[0])
        print(f"  [{gr:<8}] {SINFLAR[y[i]]:<7} {normallashtir(matnlar[i])}")

    print("\n=== 3. Normallashtirish ===")
    nostandart = np.mean([bool(re.search("[\u02bb\u2019`]", m)) for m in matnlar])
    print(f"  nostandart apostrofli sharhlar: {nostandart:.1%}")
    xom = Counter(s.strip(".,!").lower() for m in matnlar for s in m.split())
    toza = Counter(s for m in matnlar for s in normallashtir(m).split())
    print(f"  noyob tokenlar: xom {len(xom)} -> normallangan {len(toza)}")
    zor = sorted((n, s) for s, n in xom.items() if normallashtir(s) == "zo'r")
    print("  zo'r so'zining yozilishlari: " + ", ".join(
        f"{s.encode('ascii', 'backslashreplace').decode()} ({n})"
        for n, s in reversed(zor)))

    print("\n=== 4. Lug'at bazaviysi: guruh bo'yicha aniqlik ===")
    print(f"  {'variant':<16}{'hamma':>8}" + "".join(f"{gr:>9}" for gr in GURUHLAR))
    variantlar = [("faqat lug'at", {}), ("+ inkor", {"inkor": True}),
                  ("+ inkor + lekin", {"inkor": True, "kontrast": True})]
    natija = {}
    for nom, kw in variantlar:
        p = np.array([lugat_bashorat(m, **kw) for m in matnlar])
        natija[nom] = p
        q = [np.mean(p == y)] + [np.mean(p[g == gr] == y[g == gr])
                                 for gr in GURUHLAR]
        print(f"  {nom:<16}" + "".join(f"{v:>8.3f} " for v in q).rstrip())

    print("\n=== 5. Qoidalar qayerda yiqiladi (eng to'liq variant) ===")
    p = natija["+ inkor + lekin"]
    for gr in ["inkor", "kontrast", "aralash"]:
        xato = np.flatnonzero((g == gr) & (p != y))
        i = int(xato[0])
        print(f"  [{gr}] {normallashtir(matnlar[i])}")
        print(f"      haqiqiy {SINFLAR[y[i]]}, lug'at {SINFLAR[p[i]]}")

    print("\n=== 6. Lug'at qamrovi ===")
    bos = np.array([lugat_bashorat(m) == 1 for m in matnlar])
    fikrli = y != 1
    print(f"  fikrli sharhlarda lug'at so'zi topilmadi: {np.mean(bos[fikrli]):.1%}")
    print(f"  qoidalar soni: ildiz {len(ILDIZLAR)}, inkor 2, kontrast 1")
    yaxshilanish = np.mean(p == y) - np.mean(natija["faqat lug'at"] == y)
    print(f"  qoidalar qo'shgan aniqlik: {yaxshilanish:+.3f}")
    if np.mean(p == y) < 0.7:
        print("  ⭐ Qoidalar bilan ham aniqlik 0.7 dan past - "
              "o'rganiladigan model kerak")


if __name__ == "__main__":
    main()

=== 1. Korpus ===
  sharhlar: 6000
    salbiy    2110  (35.2%)
    neytral   1229  (20.5%)
    ijobiy    2661  (44.4%)
  guruhlar: oddiy 2140, yumshoq 545, inkor 1190, kontrast 1031, aralash 363, neytral 731

=== 2. Har guruhdan namuna (normallashtirilgan) ===
  [oddiy   ] ijobiy  yetkazib berish chaqqon toshkentga yetkazishdi va dostavka ancha o'z vaqtida
  [yumshoq ] ijobiy  chidasa bo'ladi rangi qora
  [inkor   ] ijobiy  narxi baland emas va kechikmasdan keldi va onamga olib berdim o'lchami kichik
  [kontrast] ijobiy  onamga olib berdim operator e'tiborsiz biroq qadog'i ozoda
  [aralash ] neytral sifati juda chala va bahosi hamyonbop
  [neytral ] neytral qutisida yo'riqnoma bor toshkentga yetkazishdi

=== 3. Normallashtirish ===
  nostandart apostrofli sharhlar: 38.9%
  noyob tokenlar: xom 993 -> normallangan 880
  zo'r so'zining yozilishlari: zo'r (41), zo\u2019r (18), zo\u02bbr (9), zo`r (8)

=== 4. Lug'at bazaviysi: guruh bo'yicha aniqlik ===
  variant            hamma    oddiy  yumshoq    inkor kontrast  aralash  neytral
  faqat lug'at       0.483    0.699    0.444    0.013    0.258    0.482    0.967
  + inkor            0.589    0.699    0.532    0.452    0.320    0.482    0.967
  + inkor + lekin    0.628    0.699    0.532    0.452    0.546    0.482    0.967

=== 5. Qoidalar qayerda yiqiladi (eng to'liq variant) ===
  [inkor] narxi baland emas va kechikmasdan keldi va onamga olib berdim o'lchami kichik
      haqiqiy ijobiy, lug'at salbiy
  [kontrast] onamga olib berdim operator e'tiborsiz biroq qadog'i ozoda
      haqiqiy ijobiy, lug'at neytral
  [aralash] kuryer rosa imillagan va sotuvchi a'lo rangi ko'k va buyurtma seshanba kuni keldi
      haqiqiy neytral, lug'at ijobiy

=== 6. Lug'at qamrovi ===
  fikrli sharhlarda lug'at so'zi topilmadi: 33.1%
  qoidalar soni: ildiz 24, inkor 2, kontrast 1
  qoidalar qo'shgan aniqlik: +0.145
  ⭐ Qoidalar bilan ham aniqlik 0.7 dan past - o'rganiladigan model kerak

"""TF-IDF + LogisticRegression: unigram va bigram, guruh va sinf bo'yicha."""

import re

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split

SINFLAR = ["salbiy", "neytral", "ijobiy"]
GURUHLAR = ["oddiy", "yumshoq", "inkor", "kontrast", "aralash", "neytral"]
ULUSH = [0.36, 0.09, 0.20, 0.17, 0.06, 0.12]
APOSTROFLAR = ["'", "'", "'", "\u02bb", "\u2019", "`"]
ASPEKTLAR = [   # (jihat nomlari, ijobiy sifatlar, salbiy sifatlar)
    (["sifati", "materiali", "ishlanishi"],
     ["yaxshi", "zo'r", "a'lo", "ajoyib", "mustahkam", "puxta"],
     ["yomon", "past", "bo'sh", "nochor", "xom", "chala"]),
    (["narxi", "bahosi"], ["arzon", "hamyonbop", "qulay", "maqbul"],
     ["qimmat", "baland", "osmonda", "haddan tashqari"]),
    (["yetkazib berish", "dostavka", "kuryer"],
     ["tez", "o'z vaqtida", "chaqqon", "tezkor"],
     ["sekin", "kech", "imillagan", "sust"]),
    (["xizmat", "sotuvchi", "operator"],
     ["yaxshi", "xushmuomala", "a'lo", "e'tiborli"],
     ["yomon", "qo'pol", "e'tiborsiz", "beparvo"]),
    (["qadog'i", "o'rami"], ["chiroyli", "butun", "puxta", "ozoda"],
     ["yirtilgan", "ezilgan", "yomon", "iflos"]),
]
KUCHAYTIR = ["", "", "", "juda ", "ancha ", "rosa "]
FEL_IJOBIY = [("menga yoqdi", "menga yoqmadi"),       # (asl, -ma- shakli)
              ("tavsiya qilaman", "tavsiya qilmayman"),
              ("yana buyurtma beraman", "boshqa buyurtma bermayman"),
              ("mamnun qoldim", "mamnun qolmadim"),
              ("pulimga arzidi", "pulimga arzimadi"),
              ("kutganimdek chiqdi", "kutganimdek chiqmadi")]
FEL_SALBIY = [("ikki kunda buzildi", "bir oyda ham buzilmadi"),
              ("pulimga achindim", "pulimga achinmadim"),
              ("afsuslandim", "afsuslanmadim"),
              ("hafsalam pir bo'ldi", "hafsalam pir bo'lmadi"),
              ("kechikib keldi", "kechikmasdan keldi"),
              ("qaytarib berdim", "qaytarishga hojat bo'lmadi")]
YUMSHOQ = {1: ["yomon emas", "chidasa bo'ladi", "umuman olganda normal",
               "narxiga arziydi"],
           -1: ["biroz kechikdi", "kutganimdan sal pastroq", "unchalik emas",
                "ozgina hafsalam pir bo'ldi"]}
FAKTLAR = ["buyurtma {kun} kuni keldi", "rangi {rang}", "o'lchami {olcham}",
           "sovg'a uchun oldim", "qutisida yo'riqnoma bor",
           "narxi {son} ming so'm", "{shahar}ga yetkazishdi",
           "onamga olib berdim", "ikki dona oldim"]
NEYTRAL_FIKR = ["hali ishlatib ko'rmadim", "oddiy mahsulot", "sifati o'rtacha",
                "narxiga yarasha", "na yaxshi, na yomon", "hozircha fikr yo'q",
                "yaxshi ham emas, yomon ham emas", "boshqasidan farqi yo'q",
                "yaxshi yoki yomonligini keyin bilaman"]
TOLDIR = {"kun": ["dushanba", "seshanba", "juma", "shanba"],
          "rang": ["qora", "oq", "ko'k", "kulrang", "qizil"],
          "olcham": ["o'rtacha", "katta", "kichik", "standart"],
          "son": ["120", "85", "240", "56", "310"],
          "shahar": ["Toshkent", "Samarqand", "Buxoro", "Namangan", "Qarshi"]}
BOGLOVCHI = ["lekin", "ammo", "biroq"]


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def fakt(rng):
    s = tanla(rng, FAKTLAR)
    for k, v in TOLDIR.items():
        s = s.replace("{" + k + "}", tanla(rng, v))
    return s


def birlik(rng, qutb, inkor=False, jihat=None):
    """Bitta fikr bo'lagi; qutb (+1/-1) - bo'lakning HOSIL bo'lgan ma'nosi.
    jihat: None - tasodifiy, son - shu jihat sifati, "fel" - fe'lli bo'lak."""
    if jihat is None:
        jihat = int(rng.integers(len(ASPEKTLAR))) if rng.random() < 0.55 else "fel"
    if jihat != "fel":                               # "narxi arzon"
        nomlar, ij, sl = ASPEKTLAR[jihat]
        nom = tanla(rng, nomlar)
        if not inkor:
            return f"{nom} {tanla(rng, KUCHAYTIR)}{tanla(rng, ij if qutb > 0 else sl)}"
        oldi = "unchalik " if qutb < 0 and rng.random() < 0.3 else ""
        return f"{nom} {oldi}{tanla(rng, sl if qutb > 0 else ij)} emas"
    ij, sl = tanla(rng, FEL_IJOBIY), tanla(rng, FEL_SALBIY)
    if not inkor:
        return (ij if qutb > 0 else sl)[0]
    return (sl if qutb > 0 else ij)[1]               # "-ma-": buzilmadi -> ijobiy


def ikki_jihat(rng, q1, q2, inkor2=False):
    """Ikki xil jihat haqida ikki bo'lak (masalan narx va sifat)."""
    i, j = rng.choice(len(ASPEKTLAR), size=2, replace=False)
    ikkinchi = int(j) if rng.random() < 0.6 else "fel"
    return birlik(rng, q1, jihat=int(i)), birlik(rng, q2, inkor2, ikkinchi)


def yozuvga(rng, bolaklar):
    matn = ""
    for i, b in enumerate(bolaklar):
        if i == 0:
            matn = b[0].upper() + b[1:]
        elif b.split()[0] in BOGLOVCHI:
            matn += ", " + b
        else:
            ayir = tanla(rng, [". ", ", ", ". ", " va "])
            matn += ayir + (b[0].upper() + b[1:] if ayir == ". " else b)
    return matn + tanla(rng, [".", ".", "!", "", "!!"])


def buz(rng, matn, p_imlo):
    """Apostrof variantlari va imlo xatolari (tushgan, ikkilangan, almashgan harf)."""
    sozlar = []
    for s in matn.split(" "):
        if "'" in s:
            s = s.replace("'", tanla(rng, APOSTROFLAR))
        if len(s) >= 5 and rng.random() < p_imlo:
            j, amal = int(rng.integers(2, len(s) - 1)), int(rng.integers(3))
            if amal == 0:
                s = s[:j] + s[j + 1:]
            elif amal == 1:
                s = s[:j] + s[j] + s[j:]
            else:
                s = s[:j - 1] + s[j] + s[j - 1] + s[j + 1:]
        sozlar.append(s)
    return " ".join(sozlar)


def sharh(rng):
    g = int(rng.choice(len(GURUHLAR), p=ULUSH))
    qutb = 1 if rng.random() < 0.55 else -1
    faktlar = [fakt(rng) for _ in range(int(rng.integers(0, 3)))]
    yorliq = 2 if qutb > 0 else 0
    if g == 0:                                       # oddiy
        bol = [birlik(rng, qutb) for _ in range(int(rng.integers(1, 3)))]
    elif g == 1:                                     # yumshoq: "yomon emas"
        bol = [tanla(rng, YUMSHOQ[qutb])]
        if rng.random() < 0.25:                      # annotatorlar kelisha olmaydi
            yorliq = 1
    elif g == 2:                                     # inkor: "emas", "-ma-"
        bol = [birlik(rng, qutb, inkor=True)
               for _ in range(int(rng.integers(1, 3)))]
    elif g == 3:                                     # kontrast: 2-qism hal qiladi
        a, b = ikki_jihat(rng, -qutb, qutb, inkor2=rng.random() < 0.25)
        bol, faktlar = [a, tanla(rng, BOGLOVCHI) + " " + b], faktlar[:1]
        return yozuvga(rng, faktlar + bol), yorliq, g
    elif g == 4:                                     # aralash: +/- teng -> neytral
        a, b = ikki_jihat(rng, qutb, -qutb)
        bol, yorliq = [a + tanla(rng, [", ", " va "]) + b], 1
    else:                                            # neytral
        bol = [tanla(rng, NEYTRAL_FIKR)] if rng.random() < 0.7 else []
        faktlar = faktlar or [fakt(rng)]
        yorliq = 1
    bol += faktlar
    rng.shuffle(bol)
    return yozuvga(rng, bol), yorliq, g


def korpus(n, seed, p_imlo=0.04, p_shovqin=0.03):
    """Sintetik o'zbekcha sharhlar: matnlar, yorliqlar, guruhlar."""
    rng = np.random.default_rng(seed)
    matnlar, y, guruh = [], [], []
    for _ in range(n):
        m, yl, g = sharh(rng)
        if rng.random() < p_shovqin:                # tasodifiy yorliq xatosi
            yl = int(rng.integers(3))
        matnlar.append(buz(rng, m, p_imlo))
        y.append(yl)
        guruh.append(GURUHLAR[g])
    return np.array(matnlar, dtype=object), np.array(y), np.array(guruh)


def normallashtir(s):
    s = s.lower()
    for v in ("\u02bb", "\u2019", "\u2018", "\u02bc", "`"):
        s = s.replace(v, "'")
    s = re.sub(r"[^a-z0-9' ]+", " ", s)
    return " ".join(s.split())


def bolish(y, seed=0):
    """60/20/20, sinf bo'yicha stratifikatsiya; test shu yerda qulflanadi."""
    idx = np.arange(len(y))
    ish, te = train_test_split(idx, test_size=0.2, stratify=y, random_state=seed)
    tr, va = train_test_split(ish, test_size=0.25, stratify=y[ish],
                              random_state=seed)
    return tr, va, te


def jadval_sarlavha():
    print(f"  {'model':<16}{'hamma':>8}" + "".join(f"{g:>9}" for g in GURUHLAR))


def jadval_qator(nom, y, p, g):
    """Umumiy aniqlik va har bir guruh bo'yicha aniqlik."""
    qiymat = [np.mean(p == y)] + [np.mean(p[g == gr] == y[g == gr])
                                  for gr in GURUHLAR]
    print(f"  {nom:<16}" + "".join(f"{v:>8.3f} " for v in qiymat).rstrip())
    return qiymat


def tfidf_model(matnlar, y, ngram):
    v = TfidfVectorizer(preprocessor=normallashtir, ngram_range=ngram,
                        sublinear_tf=True, min_df=2)
    lr = LogisticRegression(C=5.0, max_iter=3000)
    lr.fit(v.fit_transform(matnlar), y)          # vektorizator FAQAT o'quvda
    return v, lr


def main() -> None:
    matnlar, y, g = korpus(6000, seed=42)
    tr, va, te = bolish(y)
    print("=== 1. Bo'lish (60/20/20, stratifikatsiya) ===")
    for nom, ii in [("o'quv", tr), ("validatsiya", va), ("test", te)]:
        print(f"  {nom:<12} {len(ii):>5}  neytral ulushi {np.mean(y[ii] == 1):.3f}")

    print("\n=== 2. Unigram va bigram: guruh bo'yicha aniqlik (test) ===")
    jadval_sarlavha()
    modellar, qiymat = {}, {}
    for nom, ng in [("TF-IDF (1,1)", (1, 1)), ("TF-IDF (1,2)", (1, 2))]:
        v, lr = tfidf_model(matnlar[tr], y[tr], ng)
        modellar[nom] = (v, lr)
        p = lr.predict(v.transform(matnlar[te]))
        qiymat[nom] = jadval_qator(nom, y[te], p, g[te])
    farq = np.array(qiymat["TF-IDF (1,2)"]) - np.array(qiymat["TF-IDF (1,1)"])
    eng = int(np.argmax(farq[1:]))
    print(f"  bigramning eng katta foydasi: {GURUHLAR[eng]} ({farq[1 + eng]:+.3f})")

    print("\n=== 3. Sinf bo'yicha (TF-IDF (1,2), test) ===")
    v, lr = modellar["TF-IDF (1,2)"]
    p = lr.predict(v.transform(matnlar[te]))
    hisobot = classification_report(y[te], p, target_names=SINFLAR,
                                    output_dict=True)
    print(f"  {'sinf':<8} {'precision':>9} {'recall':>7} {'F1':>6} {'n':>5}")
    for s in SINFLAR:
        h = hisobot[s]
        print(f"  {s:<8} {h['precision']:>9.3f} {h['recall']:>7.3f} "
              f"{h['f1-score']:>6.3f} {int(h['support']):>5}")
    print(f"  makro-F1 {hisobot['macro avg']['f1-score']:.3f}, "
          f"aniqlik {hisobot['accuracy']:.3f}")
    eng_past = min(SINFLAR, key=lambda s: hisobot[s]["f1-score"])
    print(f"  eng qiyin sinf: {eng_past}")

    print("\n=== 4. Inkor bigramlari: model nimani o'rgandi ===")
    nomlar = v.get_feature_names_out()
    w = lr.coef_[2] - lr.coef_[0]                # ijobiy - salbiy yo'nalishi
    for soz in ["yaxshi", "yomon", "qimmat", "arzon", "yoqdi", "yoqmadi"]:
        juft = [soz, soz + " emas"] if soz not in ("yoqdi", "yoqmadi") else [soz]
        qism = []
        for f in juft:
            j = np.flatnonzero(nomlar == f)
            if len(j):
                qism.append(f"{f!r}: {w[j[0]]:+.2f}")
        print("  " + ",  ".join(qism))

    print("\n=== 5. Tartib muammosi: bir xil so'zlar, teskari ma'no ===")
    juftlar = ["narxi arzon lekin sifati yomon", "sifati yomon lekin narxi arzon",
               "materiali puxta ammo dostavka sekin",
               "dostavka sekin ammo materiali puxta"]
    print(f"  {'matn':<37} {'(1,1) P(ijobiy)':>15} {'(1,2) P(ijobiy)':>15}")
    for m in juftlar:
        pr = [modellar[k][1].predict_proba(modellar[k][0].transform([m]))[0, 2]
              for k in ["TF-IDF (1,1)", "TF-IDF (1,2)"]]
        print(f"  {m:<37} {pr[0]:>15.3f} {pr[1]:>15.3f}")
    print("  (1,1) uchun juftlik ichidagi ikki matn - bitta so'z xaltasi")


if __name__ == "__main__":
    main()

=== 1. Bo'lish (60/20/20, stratifikatsiya) ===
  o'quv         3600  neytral ulushi 0.205
  validatsiya   1200  neytral ulushi 0.205
  test          1200  neytral ulushi 0.205

=== 2. Unigram va bigram: guruh bo'yicha aniqlik (test) ===
  model              hamma    oddiy  yumshoq    inkor kontrast  aralash  neytral
  TF-IDF (1,1)       0.769    0.814    0.704    0.827    0.723    0.158    0.979
  TF-IDF (1,2)       0.845    0.888    0.694    0.934    0.866    0.316    0.937
  bigramning eng katta foydasi: aralash (+0.158)

=== 3. Sinf bo'yicha (TF-IDF (1,2), test) ===
  sinf     precision  recall     F1     n
  salbiy       0.839   0.886  0.862   422
  neytral      0.800   0.650  0.717   246
  ijobiy       0.866   0.902  0.884   532
  makro-F1 0.821, aniqlik 0.845
  eng qiyin sinf: neytral

=== 4. Inkor bigramlari: model nimani o'rgandi ===
  'yaxshi': +2.68,  'yaxshi emas': -8.07
  'yomon': -4.66,  'yomon emas': +8.25
  'qimmat': -2.93,  'qimmat emas': +5.95
  'arzon': +3.36,  'arzon emas': -5.29
  'yoqdi': +5.56
  'yoqmadi': -4.31

=== 5. Tartib muammosi: bir xil so'zlar, teskari ma'no ===
  matn                                  (1,1) P(ijobiy) (1,2) P(ijobiy)
  narxi arzon lekin sifati yomon                  0.658           0.253
  sifati yomon lekin narxi arzon                  0.658           0.939
  materiali puxta ammo dostavka sekin             0.815           0.674
  dostavka sekin ammo materiali puxta             0.815           0.923
  (1,1) uchun juftlik ichidagi ikki matn - bitta so'z xaltasi

"""EmbeddingBag va LSTM: qiyin guruhlar jadvali va juftlashgan taqqoslash."""

import copy
import re
from collections import Counter

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from torch.nn.utils.rnn import pack_padded_sequence

SINFLAR = ["salbiy", "neytral", "ijobiy"]
GURUHLAR = ["oddiy", "yumshoq", "inkor", "kontrast", "aralash", "neytral"]
ULUSH = [0.36, 0.09, 0.20, 0.17, 0.06, 0.12]
APOSTROFLAR = ["'", "'", "'", "\u02bb", "\u2019", "`"]
ASPEKTLAR = [   # (jihat nomlari, ijobiy sifatlar, salbiy sifatlar)
    (["sifati", "materiali", "ishlanishi"],
     ["yaxshi", "zo'r", "a'lo", "ajoyib", "mustahkam", "puxta"],
     ["yomon", "past", "bo'sh", "nochor", "xom", "chala"]),
    (["narxi", "bahosi"], ["arzon", "hamyonbop", "qulay", "maqbul"],
     ["qimmat", "baland", "osmonda", "haddan tashqari"]),
    (["yetkazib berish", "dostavka", "kuryer"],
     ["tez", "o'z vaqtida", "chaqqon", "tezkor"],
     ["sekin", "kech", "imillagan", "sust"]),
    (["xizmat", "sotuvchi", "operator"],
     ["yaxshi", "xushmuomala", "a'lo", "e'tiborli"],
     ["yomon", "qo'pol", "e'tiborsiz", "beparvo"]),
    (["qadog'i", "o'rami"], ["chiroyli", "butun", "puxta", "ozoda"],
     ["yirtilgan", "ezilgan", "yomon", "iflos"]),
]
KUCHAYTIR = ["", "", "", "juda ", "ancha ", "rosa "]
FEL_IJOBIY = [("menga yoqdi", "menga yoqmadi"),       # (asl, -ma- shakli)
              ("tavsiya qilaman", "tavsiya qilmayman"),
              ("yana buyurtma beraman", "boshqa buyurtma bermayman"),
              ("mamnun qoldim", "mamnun qolmadim"),
              ("pulimga arzidi", "pulimga arzimadi"),
              ("kutganimdek chiqdi", "kutganimdek chiqmadi")]
FEL_SALBIY = [("ikki kunda buzildi", "bir oyda ham buzilmadi"),
              ("pulimga achindim", "pulimga achinmadim"),
              ("afsuslandim", "afsuslanmadim"),
              ("hafsalam pir bo'ldi", "hafsalam pir bo'lmadi"),
              ("kechikib keldi", "kechikmasdan keldi"),
              ("qaytarib berdim", "qaytarishga hojat bo'lmadi")]
YUMSHOQ = {1: ["yomon emas", "chidasa bo'ladi", "umuman olganda normal",
               "narxiga arziydi"],
           -1: ["biroz kechikdi", "kutganimdan sal pastroq", "unchalik emas",
                "ozgina hafsalam pir bo'ldi"]}
FAKTLAR = ["buyurtma {kun} kuni keldi", "rangi {rang}", "o'lchami {olcham}",
           "sovg'a uchun oldim", "qutisida yo'riqnoma bor",
           "narxi {son} ming so'm", "{shahar}ga yetkazishdi",
           "onamga olib berdim", "ikki dona oldim"]
NEYTRAL_FIKR = ["hali ishlatib ko'rmadim", "oddiy mahsulot", "sifati o'rtacha",
                "narxiga yarasha", "na yaxshi, na yomon", "hozircha fikr yo'q",
                "yaxshi ham emas, yomon ham emas", "boshqasidan farqi yo'q",
                "yaxshi yoki yomonligini keyin bilaman"]
TOLDIR = {"kun": ["dushanba", "seshanba", "juma", "shanba"],
          "rang": ["qora", "oq", "ko'k", "kulrang", "qizil"],
          "olcham": ["o'rtacha", "katta", "kichik", "standart"],
          "son": ["120", "85", "240", "56", "310"],
          "shahar": ["Toshkent", "Samarqand", "Buxoro", "Namangan", "Qarshi"]}
BOGLOVCHI = ["lekin", "ammo", "biroq"]


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def fakt(rng):
    s = tanla(rng, FAKTLAR)
    for k, v in TOLDIR.items():
        s = s.replace("{" + k + "}", tanla(rng, v))
    return s


def birlik(rng, qutb, inkor=False, jihat=None):
    """Bitta fikr bo'lagi; qutb (+1/-1) - bo'lakning HOSIL bo'lgan ma'nosi.
    jihat: None - tasodifiy, son - shu jihat sifati, "fel" - fe'lli bo'lak."""
    if jihat is None:
        jihat = int(rng.integers(len(ASPEKTLAR))) if rng.random() < 0.55 else "fel"
    if jihat != "fel":                               # "narxi arzon"
        nomlar, ij, sl = ASPEKTLAR[jihat]
        nom = tanla(rng, nomlar)
        if not inkor:
            return f"{nom} {tanla(rng, KUCHAYTIR)}{tanla(rng, ij if qutb > 0 else sl)}"
        oldi = "unchalik " if qutb < 0 and rng.random() < 0.3 else ""
        return f"{nom} {oldi}{tanla(rng, sl if qutb > 0 else ij)} emas"
    ij, sl = tanla(rng, FEL_IJOBIY), tanla(rng, FEL_SALBIY)
    if not inkor:
        return (ij if qutb > 0 else sl)[0]
    return (sl if qutb > 0 else ij)[1]               # "-ma-": buzilmadi -> ijobiy


def ikki_jihat(rng, q1, q2, inkor2=False):
    """Ikki xil jihat haqida ikki bo'lak (masalan narx va sifat)."""
    i, j = rng.choice(len(ASPEKTLAR), size=2, replace=False)
    ikkinchi = int(j) if rng.random() < 0.6 else "fel"
    return birlik(rng, q1, jihat=int(i)), birlik(rng, q2, inkor2, ikkinchi)


def yozuvga(rng, bolaklar):
    matn = ""
    for i, b in enumerate(bolaklar):
        if i == 0:
            matn = b[0].upper() + b[1:]
        elif b.split()[0] in BOGLOVCHI:
            matn += ", " + b
        else:
            ayir = tanla(rng, [". ", ", ", ". ", " va "])
            matn += ayir + (b[0].upper() + b[1:] if ayir == ". " else b)
    return matn + tanla(rng, [".", ".", "!", "", "!!"])


def buz(rng, matn, p_imlo):
    """Apostrof variantlari va imlo xatolari (tushgan, ikkilangan, almashgan harf)."""
    sozlar = []
    for s in matn.split(" "):
        if "'" in s:
            s = s.replace("'", tanla(rng, APOSTROFLAR))
        if len(s) >= 5 and rng.random() < p_imlo:
            j, amal = int(rng.integers(2, len(s) - 1)), int(rng.integers(3))
            if amal == 0:
                s = s[:j] + s[j + 1:]
            elif amal == 1:
                s = s[:j] + s[j] + s[j:]
            else:
                s = s[:j - 1] + s[j] + s[j - 1] + s[j + 1:]
        sozlar.append(s)
    return " ".join(sozlar)


def sharh(rng):
    g = int(rng.choice(len(GURUHLAR), p=ULUSH))
    qutb = 1 if rng.random() < 0.55 else -1
    faktlar = [fakt(rng) for _ in range(int(rng.integers(0, 3)))]
    yorliq = 2 if qutb > 0 else 0
    if g == 0:                                       # oddiy
        bol = [birlik(rng, qutb) for _ in range(int(rng.integers(1, 3)))]
    elif g == 1:                                     # yumshoq: "yomon emas"
        bol = [tanla(rng, YUMSHOQ[qutb])]
        if rng.random() < 0.25:                      # annotatorlar kelisha olmaydi
            yorliq = 1
    elif g == 2:                                     # inkor: "emas", "-ma-"
        bol = [birlik(rng, qutb, inkor=True)
               for _ in range(int(rng.integers(1, 3)))]
    elif g == 3:                                     # kontrast: 2-qism hal qiladi
        a, b = ikki_jihat(rng, -qutb, qutb, inkor2=rng.random() < 0.25)
        bol, faktlar = [a, tanla(rng, BOGLOVCHI) + " " + b], faktlar[:1]
        return yozuvga(rng, faktlar + bol), yorliq, g
    elif g == 4:                                     # aralash: +/- teng -> neytral
        a, b = ikki_jihat(rng, qutb, -qutb)
        bol, yorliq = [a + tanla(rng, [", ", " va "]) + b], 1
    else:                                            # neytral
        bol = [tanla(rng, NEYTRAL_FIKR)] if rng.random() < 0.7 else []
        faktlar = faktlar or [fakt(rng)]
        yorliq = 1
    bol += faktlar
    rng.shuffle(bol)
    return yozuvga(rng, bol), yorliq, g


def korpus(n, seed, p_imlo=0.04, p_shovqin=0.03):
    """Sintetik o'zbekcha sharhlar: matnlar, yorliqlar, guruhlar."""
    rng = np.random.default_rng(seed)
    matnlar, y, guruh = [], [], []
    for _ in range(n):
        m, yl, g = sharh(rng)
        if rng.random() < p_shovqin:                # tasodifiy yorliq xatosi
            yl = int(rng.integers(3))
        matnlar.append(buz(rng, m, p_imlo))
        y.append(yl)
        guruh.append(GURUHLAR[g])
    return np.array(matnlar, dtype=object), np.array(y), np.array(guruh)


def normallashtir(s):
    s = s.lower()
    for v in ("\u02bb", "\u2019", "\u2018", "\u02bc", "`"):
        s = s.replace(v, "'")
    s = re.sub(r"[^a-z0-9' ]+", " ", s)
    return " ".join(s.split())


def bolish(y, seed=0):
    """60/20/20, sinf bo'yicha stratifikatsiya; test shu yerda qulflanadi."""
    idx = np.arange(len(y))
    ish, te = train_test_split(idx, test_size=0.2, stratify=y, random_state=seed)
    tr, va = train_test_split(ish, test_size=0.25, stratify=y[ish],
                              random_state=seed)
    return tr, va, te


def jadval_sarlavha():
    print(f"  {'model':<16}{'hamma':>8}" + "".join(f"{g:>9}" for g in GURUHLAR))


def jadval_qator(nom, y, p, g):
    """Umumiy aniqlik va har bir guruh bo'yicha aniqlik."""
    qiymat = [np.mean(p == y)] + [np.mean(p[g == gr] == y[g == gr])
                                  for gr in GURUHLAR]
    print(f"  {nom:<16}" + "".join(f"{v:>8.3f} " for v in qiymat).rstrip())
    return qiymat


IJOBIY_ILDIZ = ["yaxshi", "zo'r", "a'lo", "ajoyib", "arzon", "qulay", "tez",
                "chiroyli", "yoq", "tavsiya", "mamnun", "arzi"]
SALBIY_ILDIZ = ["yomon", "past", "qimmat", "sekin", "kech", "qo'pol", "buzil",
                "achin", "afsuslan", "hafsala", "qaytar", "iflos"]
ILDIZLAR = [(s, 1) for s in IJOBIY_ILDIZ] + [(s, -1) for s in SALBIY_ILDIZ]


def lugat_bashorat(matn, inkor=False, kontrast=False):
    """Lug'at bazaviysi: ildiz mos kelsa +1/-1; ball ishorasi -> sinf."""
    t = normallashtir(matn).split()
    if kontrast:                        # oxirgi "lekin" dan keyingi qism hal qiladi
        for j in range(len(t) - 1, -1, -1):
            if t[j] in BOGLOVCHI:
                t = t[j + 1:]
                break
    ball = 0
    for i, soz in enumerate(t):
        for ildiz, q in ILDIZLAR:
            if soz.startswith(ildiz):
                if inkor and (soz[len(ildiz):].startswith("ma")
                              or t[i + 1:i + 2] == ["emas"]):
                    q = -q              # "yoqmadi", "yaxshi emas"
                ball += q
                break
    return 2 if ball > 0 else 0 if ball < 0 else 1


class Lugat:
    """Token -> indeks; FAQAT o'quv matnlaridan quriladi (0 - pad, 1 - unk)."""

    def __init__(self, matnlar, min_son=2):
        c = Counter(s for m in matnlar for s in normallashtir(m).split())
        self.itos = ["<pad>", "<unk>"] + sorted(s for s, n in c.items()
                                                if n >= min_son)
        self.stoi = {s: i for i, s in enumerate(self.itos)}

    def __len__(self):
        return len(self.itos)

    def kodla(self, matnlar):
        seqs = [[self.stoi.get(s, 1) for s in normallashtir(m).split()] or [1]
                for m in matnlar]
        L = torch.tensor([len(s) for s in seqs])
        x = torch.zeros(len(seqs), int(L.max()), dtype=torch.long)
        for i, s in enumerate(seqs):
            x[i, :len(s)] = torch.tensor(s)
        return x, L


class BagModel(nn.Module):
    """Neyron so'z xaltasi: o'rtacha embedding -> Linear (tartib yo'q)."""

    def __init__(self, V, d=32):
        super().__init__()
        self.emb = nn.EmbeddingBag(V, d, mode="mean", padding_idx=0)
        self.bosh = nn.Linear(d, 3)

    def forward(self, x, L):
        return self.bosh(self.emb(x))


class LSTMModel(nn.Module):
    """LSTM chapdan o'ngga o'qiydi; pad lar pack_padded_sequence bilan chetlanadi."""

    def __init__(self, V, d=32, h=48):
        super().__init__()
        self.emb = nn.Embedding(V, d, padding_idx=0)
        self.lstm = nn.LSTM(d, h, batch_first=True)
        self.bosh = nn.Linear(h, 3)

    def forward(self, x, L):
        pk = pack_padded_sequence(self.emb(x), L, batch_first=True,
                                  enforce_sorted=False)
        _, (h, _) = self.lstm(pk)
        return self.bosh(h[-1])                   # oxirgi HAQIQIY token holati


def orgat(model, tr, va, davrlar=10, seed=0):
    """tr, va = (x, L, y). Eng yaxshi val aniqlikdagi holat deepcopy bilan."""
    torch.manual_seed(seed)
    opt = torch.optim.AdamW(model.parameters(), lr=5e-3, weight_decay=1e-4)
    g = torch.Generator().manual_seed(seed)
    eng, eng_davr, holat = -1.0, 0, None
    for davr in range(1, davrlar + 1):
        model.train()
        tartib = torch.randperm(len(tr[2]), generator=g)
        for i in range(0, len(tartib), 128):
            j = tartib[i:i + 128]
            opt.zero_grad()
            F.cross_entropy(model(tr[0][j], tr[1][j]), tr[2][j]).backward()
            opt.step()
        aniqlik = float((logitlar(model, va[0], va[1]).argmax(1)
                         == va[2]).float().mean())
        if aniqlik > eng:
            eng, eng_davr = aniqlik, davr
            holat = copy.deepcopy(model.state_dict())
    model.load_state_dict(holat)
    return eng, eng_davr


def logitlar(model, x, L):
    model.eval()
    with torch.no_grad():
        return model(x, L)


def main() -> None:
    matnlar, y, g = korpus(6000, seed=42)
    tr, va, te = bolish(y)
    lug = Lugat(matnlar[tr])
    D = {k: (*lug.kodla(matnlar[ii]), torch.tensor(y[ii]))
         for k, ii in [("tr", tr), ("va", va), ("te", te)]}
    print("=== 1. Lug'at (faqat o'quvdan) ===")
    unk = float((D["te"][0] == 1).sum() / D["te"][1].sum())
    print(f"  hajm {len(lug)}, testdagi <unk> ulushi {unk:.3f}, "
          f"eng uzun sharh {int(D['tr'][1].max())} token")

    print("\n=== 2. O'rgatish (eng yaxshi holat - deepcopy) ===")
    neyron = {}
    for nom, sinf in [("EmbeddingBag", BagModel), ("LSTM", LSTMModel)]:
        torch.manual_seed(0)
        model = sinf(len(lug))
        eng, davr = orgat(model, D["tr"], D["va"])
        neyron[nom] = model
        print(f"  {nom:<13} val aniqlik {eng:.3f} ({davr}-davr), "
              f"parametrlar {sum(p.numel() for p in model.parameters())}")

    print("\n=== 3. Qiyin guruhlar: aniqlik jadvali (test) ===")
    bash = {"lug'at+qoida": np.array([lugat_bashorat(m, True, True)
                                      for m in matnlar[te]])}
    v = TfidfVectorizer(preprocessor=normallashtir, ngram_range=(1, 2),
                        sublinear_tf=True, min_df=2)
    lr = LogisticRegression(C=5.0, max_iter=3000)
    lr.fit(v.fit_transform(matnlar[tr]), y[tr])
    bash["TF-IDF (1,2)"] = lr.predict(v.transform(matnlar[te]))
    for nom, model in neyron.items():
        bash[nom] = logitlar(model, *D["te"][:2]).argmax(1).numpy()
    jadval_sarlavha()
    for nom, p in bash.items():
        jadval_qator(nom, y[te], p, g[te])

    print("\n=== 4. Juftlashgan taqqoslash: LSTM - TF-IDF (1,2) ===")
    d = ((bash["LSTM"] == y[te]).astype(float)
         - (bash["TF-IDF (1,2)"] == y[te]).astype(float))
    print(f"  {'guruh':<9} {'n':>4} {'farq':>7} {'2*SE':>6}  xulosa")
    for gr in ["hamma"] + GURUHLAR:
        m = np.ones(len(d), bool) if gr == "hamma" else g[te] == gr
        o, se = d[m].mean(), d[m].std(ddof=1) / np.sqrt(m.sum())
        xulosa = ("LSTM yaxshi" if o > 2 * se else
                  "TF-IDF yaxshi" if o < -2 * se else "farq sezilmas")
        print(f"  {gr:<9} {int(m.sum()):>4} {o:>+7.3f} {2 * se:>6.3f}  {xulosa}")

    print("\n=== 5. LSTM seedlarga qanchalik sezgir ===")
    natija = []
    for seed in (0, 1, 2):
        if seed == 0:
            model = neyron["LSTM"]                   # 2-bo'limda o'rgatilgan
        else:
            torch.manual_seed(seed)
            model = LSTMModel(len(lug))
            orgat(model, D["tr"], D["va"], seed=seed)
        p = logitlar(model, *D["te"][:2]).argmax(1).numpy()
        natija.append([np.mean(p == y[te]),
                       np.mean(p[g[te] == "kontrast"] == y[te][g[te] == "kontrast"])])
    natija = np.array(natija)
    for nom, ustun in [("hamma", 0), ("kontrast", 1)]:
        print(f"  {nom:<9} seedlar {natija[:, ustun].round(3).tolist()}  "
              f"o'rtacha {natija[:, ustun].mean():.3f} "
              f"+- {natija[:, ustun].std(ddof=1):.3f}")


if __name__ == "__main__":
    main()

=== 1. Lug'at (faqat o'quvdan) ===
  hajm 330, testdagi <unk> ulushi 0.021, eng uzun sharh 16 token

=== 2. O'rgatish (eng yaxshi holat - deepcopy) ===
  EmbeddingBag  val aniqlik 0.752 (10-davr), parametrlar 10659
  LSTM          val aniqlik 0.923 (10-davr), parametrlar 26451

=== 3. Qiyin guruhlar: aniqlik jadvali (test) ===
  model              hamma    oddiy  yumshoq    inkor kontrast  aralash  neytral
  lug'at+qoida       0.632    0.686    0.574    0.438    0.594    0.421    0.979
  TF-IDF (1,2)       0.845    0.888    0.694    0.934    0.866    0.316    0.937
  EmbeddingBag       0.750    0.814    0.611    0.810    0.708    0.092    0.972
  LSTM               0.906    0.928    0.694    0.942    0.946    0.737    0.972

=== 4. Juftlashgan taqqoslash: LSTM - TF-IDF (1,2) ===
  guruh        n    farq   2*SE  xulosa
  hamma     1200  +0.061  0.019  LSTM yaxshi
  oddiy      446  +0.040  0.026  LSTM yaxshi
  yumshoq    108  +0.000  0.079  farq sezilmas
  inkor      226  +0.009  0.025  farq sezilmas
  kontrast   202  +0.079  0.049  LSTM yaxshi
  aralash     76  +0.421  0.136  LSTM yaxshi
  neytral    142  +0.035  0.042  farq sezilmas

=== 5. LSTM seedlarga qanchalik sezgir ===
  hamma     seedlar [0.906, 0.881, 0.908]  o'rtacha 0.898 +- 0.015
  kontrast  seedlar [0.946, 0.96, 0.97]  o'rtacha 0.959 +- 0.012

"""Neytral sinfning qiyinligi va ishonch kalibratsiyasi (ECE, harorat)."""

import copy
import re
from collections import Counter

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from torch.nn.utils.rnn import pack_padded_sequence

SINFLAR = ["salbiy", "neytral", "ijobiy"]
GURUHLAR = ["oddiy", "yumshoq", "inkor", "kontrast", "aralash", "neytral"]
ULUSH = [0.36, 0.09, 0.20, 0.17, 0.06, 0.12]
APOSTROFLAR = ["'", "'", "'", "\u02bb", "\u2019", "`"]
ASPEKTLAR = [   # (jihat nomlari, ijobiy sifatlar, salbiy sifatlar)
    (["sifati", "materiali", "ishlanishi"],
     ["yaxshi", "zo'r", "a'lo", "ajoyib", "mustahkam", "puxta"],
     ["yomon", "past", "bo'sh", "nochor", "xom", "chala"]),
    (["narxi", "bahosi"], ["arzon", "hamyonbop", "qulay", "maqbul"],
     ["qimmat", "baland", "osmonda", "haddan tashqari"]),
    (["yetkazib berish", "dostavka", "kuryer"],
     ["tez", "o'z vaqtida", "chaqqon", "tezkor"],
     ["sekin", "kech", "imillagan", "sust"]),
    (["xizmat", "sotuvchi", "operator"],
     ["yaxshi", "xushmuomala", "a'lo", "e'tiborli"],
     ["yomon", "qo'pol", "e'tiborsiz", "beparvo"]),
    (["qadog'i", "o'rami"], ["chiroyli", "butun", "puxta", "ozoda"],
     ["yirtilgan", "ezilgan", "yomon", "iflos"]),
]
KUCHAYTIR = ["", "", "", "juda ", "ancha ", "rosa "]
FEL_IJOBIY = [("menga yoqdi", "menga yoqmadi"),       # (asl, -ma- shakli)
              ("tavsiya qilaman", "tavsiya qilmayman"),
              ("yana buyurtma beraman", "boshqa buyurtma bermayman"),
              ("mamnun qoldim", "mamnun qolmadim"),
              ("pulimga arzidi", "pulimga arzimadi"),
              ("kutganimdek chiqdi", "kutganimdek chiqmadi")]
FEL_SALBIY = [("ikki kunda buzildi", "bir oyda ham buzilmadi"),
              ("pulimga achindim", "pulimga achinmadim"),
              ("afsuslandim", "afsuslanmadim"),
              ("hafsalam pir bo'ldi", "hafsalam pir bo'lmadi"),
              ("kechikib keldi", "kechikmasdan keldi"),
              ("qaytarib berdim", "qaytarishga hojat bo'lmadi")]
YUMSHOQ = {1: ["yomon emas", "chidasa bo'ladi", "umuman olganda normal",
               "narxiga arziydi"],
           -1: ["biroz kechikdi", "kutganimdan sal pastroq", "unchalik emas",
                "ozgina hafsalam pir bo'ldi"]}
FAKTLAR = ["buyurtma {kun} kuni keldi", "rangi {rang}", "o'lchami {olcham}",
           "sovg'a uchun oldim", "qutisida yo'riqnoma bor",
           "narxi {son} ming so'm", "{shahar}ga yetkazishdi",
           "onamga olib berdim", "ikki dona oldim"]
NEYTRAL_FIKR = ["hali ishlatib ko'rmadim", "oddiy mahsulot", "sifati o'rtacha",
                "narxiga yarasha", "na yaxshi, na yomon", "hozircha fikr yo'q",
                "yaxshi ham emas, yomon ham emas", "boshqasidan farqi yo'q",
                "yaxshi yoki yomonligini keyin bilaman"]
TOLDIR = {"kun": ["dushanba", "seshanba", "juma", "shanba"],
          "rang": ["qora", "oq", "ko'k", "kulrang", "qizil"],
          "olcham": ["o'rtacha", "katta", "kichik", "standart"],
          "son": ["120", "85", "240", "56", "310"],
          "shahar": ["Toshkent", "Samarqand", "Buxoro", "Namangan", "Qarshi"]}
BOGLOVCHI = ["lekin", "ammo", "biroq"]


def tanla(rng, royxat):
    return royxat[int(rng.integers(len(royxat)))]


def fakt(rng):
    s = tanla(rng, FAKTLAR)
    for k, v in TOLDIR.items():
        s = s.replace("{" + k + "}", tanla(rng, v))
    return s


def birlik(rng, qutb, inkor=False, jihat=None):
    """Bitta fikr bo'lagi; qutb (+1/-1) - bo'lakning HOSIL bo'lgan ma'nosi.
    jihat: None - tasodifiy, son - shu jihat sifati, "fel" - fe'lli bo'lak."""
    if jihat is None:
        jihat = int(rng.integers(len(ASPEKTLAR))) if rng.random() < 0.55 else "fel"
    if jihat != "fel":                               # "narxi arzon"
        nomlar, ij, sl = ASPEKTLAR[jihat]
        nom = tanla(rng, nomlar)
        if not inkor:
            return f"{nom} {tanla(rng, KUCHAYTIR)}{tanla(rng, ij if qutb > 0 else sl)}"
        oldi = "unchalik " if qutb < 0 and rng.random() < 0.3 else ""
        return f"{nom} {oldi}{tanla(rng, sl if qutb > 0 else ij)} emas"
    ij, sl = tanla(rng, FEL_IJOBIY), tanla(rng, FEL_SALBIY)
    if not inkor:
        return (ij if qutb > 0 else sl)[0]
    return (sl if qutb > 0 else ij)[1]               # "-ma-": buzilmadi -> ijobiy


def ikki_jihat(rng, q1, q2, inkor2=False):
    """Ikki xil jihat haqida ikki bo'lak (masalan narx va sifat)."""
    i, j = rng.choice(len(ASPEKTLAR), size=2, replace=False)
    ikkinchi = int(j) if rng.random() < 0.6 else "fel"
    return birlik(rng, q1, jihat=int(i)), birlik(rng, q2, inkor2, ikkinchi)


def yozuvga(rng, bolaklar):
    matn = ""
    for i, b in enumerate(bolaklar):
        if i == 0:
            matn = b[0].upper() + b[1:]
        elif b.split()[0] in BOGLOVCHI:
            matn += ", " + b
        else:
            ayir = tanla(rng, [". ", ", ", ". ", " va "])
            matn += ayir + (b[0].upper() + b[1:] if ayir == ". " else b)
    return matn + tanla(rng, [".", ".", "!", "", "!!"])


def buz(rng, matn, p_imlo):
    """Apostrof variantlari va imlo xatolari (tushgan, ikkilangan, almashgan harf)."""
    sozlar = []
    for s in matn.split(" "):
        if "'" in s:
            s = s.replace("'", tanla(rng, APOSTROFLAR))
        if len(s) >= 5 and rng.random() < p_imlo:
            j, amal = int(rng.integers(2, len(s) - 1)), int(rng.integers(3))
            if amal == 0:
                s = s[:j] + s[j + 1:]
            elif amal == 1:
                s = s[:j] + s[j] + s[j:]
            else:
                s = s[:j - 1] + s[j] + s[j - 1] + s[j + 1:]
        sozlar.append(s)
    return " ".join(sozlar)


def sharh(rng):
    g = int(rng.choice(len(GURUHLAR), p=ULUSH))
    qutb = 1 if rng.random() < 0.55 else -1
    faktlar = [fakt(rng) for _ in range(int(rng.integers(0, 3)))]
    yorliq = 2 if qutb > 0 else 0
    if g == 0:                                       # oddiy
        bol = [birlik(rng, qutb) for _ in range(int(rng.integers(1, 3)))]
    elif g == 1:                                     # yumshoq: "yomon emas"
        bol = [tanla(rng, YUMSHOQ[qutb])]
        if rng.random() < 0.25:                      # annotatorlar kelisha olmaydi
            yorliq = 1
    elif g == 2:                                     # inkor: "emas", "-ma-"
        bol = [birlik(rng, qutb, inkor=True)
               for _ in range(int(rng.integers(1, 3)))]
    elif g == 3:                                     # kontrast: 2-qism hal qiladi
        a, b = ikki_jihat(rng, -qutb, qutb, inkor2=rng.random() < 0.25)
        bol, faktlar = [a, tanla(rng, BOGLOVCHI) + " " + b], faktlar[:1]
        return yozuvga(rng, faktlar + bol), yorliq, g
    elif g == 4:                                     # aralash: +/- teng -> neytral
        a, b = ikki_jihat(rng, qutb, -qutb)
        bol, yorliq = [a + tanla(rng, [", ", " va "]) + b], 1
    else:                                            # neytral
        bol = [tanla(rng, NEYTRAL_FIKR)] if rng.random() < 0.7 else []
        faktlar = faktlar or [fakt(rng)]
        yorliq = 1
    bol += faktlar
    rng.shuffle(bol)
    return yozuvga(rng, bol), yorliq, g


def korpus(n, seed, p_imlo=0.04, p_shovqin=0.03):
    """Sintetik o'zbekcha sharhlar: matnlar, yorliqlar, guruhlar."""
    rng = np.random.default_rng(seed)
    matnlar, y, guruh = [], [], []
    for _ in range(n):
        m, yl, g = sharh(rng)
        if rng.random() < p_shovqin:                # tasodifiy yorliq xatosi
            yl = int(rng.integers(3))
        matnlar.append(buz(rng, m, p_imlo))
        y.append(yl)
        guruh.append(GURUHLAR[g])
    return np.array(matnlar, dtype=object), np.array(y), np.array(guruh)


def normallashtir(s):
    s = s.lower()
    for v in ("\u02bb", "\u2019", "\u2018", "\u02bc", "`"):
        s = s.replace(v, "'")
    s = re.sub(r"[^a-z0-9' ]+", " ", s)
    return " ".join(s.split())


def bolish(y, seed=0):
    """60/20/20, sinf bo'yicha stratifikatsiya; test shu yerda qulflanadi."""
    idx = np.arange(len(y))
    ish, te = train_test_split(idx, test_size=0.2, stratify=y, random_state=seed)
    tr, va = train_test_split(ish, test_size=0.25, stratify=y[ish],
                              random_state=seed)
    return tr, va, te


def jadval_sarlavha():
    print(f"  {'model':<16}{'hamma':>8}" + "".join(f"{g:>9}" for g in GURUHLAR))


def jadval_qator(nom, y, p, g):
    """Umumiy aniqlik va har bir guruh bo'yicha aniqlik."""
    qiymat = [np.mean(p == y)] + [np.mean(p[g == gr] == y[g == gr])
                                  for gr in GURUHLAR]
    print(f"  {nom:<16}" + "".join(f"{v:>8.3f} " for v in qiymat).rstrip())
    return qiymat


class Lugat:
    """Token -> indeks; FAQAT o'quv matnlaridan quriladi (0 - pad, 1 - unk)."""

    def __init__(self, matnlar, min_son=2):
        c = Counter(s for m in matnlar for s in normallashtir(m).split())
        self.itos = ["<pad>", "<unk>"] + sorted(s for s, n in c.items()
                                                if n >= min_son)
        self.stoi = {s: i for i, s in enumerate(self.itos)}

    def __len__(self):
        return len(self.itos)

    def kodla(self, matnlar):
        seqs = [[self.stoi.get(s, 1) for s in normallashtir(m).split()] or [1]
                for m in matnlar]
        L = torch.tensor([len(s) for s in seqs])
        x = torch.zeros(len(seqs), int(L.max()), dtype=torch.long)
        for i, s in enumerate(seqs):
            x[i, :len(s)] = torch.tensor(s)
        return x, L


class BagModel(nn.Module):
    """Neyron so'z xaltasi: o'rtacha embedding -> Linear (tartib yo'q)."""

    def __init__(self, V, d=32):
        super().__init__()
        self.emb = nn.EmbeddingBag(V, d, mode="mean", padding_idx=0)
        self.bosh = nn.Linear(d, 3)

    def forward(self, x, L):
        return self.bosh(self.emb(x))


class LSTMModel(nn.Module):
    """LSTM chapdan o'ngga o'qiydi; pad lar pack_padded_sequence bilan chetlanadi."""

    def __init__(self, V, d=32, h=48):
        super().__init__()
        self.emb = nn.Embedding(V, d, padding_idx=0)
        self.lstm = nn.LSTM(d, h, batch_first=True)
        self.bosh = nn.Linear(h, 3)

    def forward(self, x, L):
        pk = pack_padded_sequence(self.emb(x), L, batch_first=True,
                                  enforce_sorted=False)
        _, (h, _) = self.lstm(pk)
        return self.bosh(h[-1])                   # oxirgi HAQIQIY token holati


def orgat(model, tr, va, davrlar=10, seed=0):
    """tr, va = (x, L, y). Eng yaxshi val aniqlikdagi holat deepcopy bilan."""
    torch.manual_seed(seed)
    opt = torch.optim.AdamW(model.parameters(), lr=5e-3, weight_decay=1e-4)
    g = torch.Generator().manual_seed(seed)
    eng, eng_davr, holat = -1.0, 0, None
    for davr in range(1, davrlar + 1):
        model.train()
        tartib = torch.randperm(len(tr[2]), generator=g)
        for i in range(0, len(tartib), 128):
            j = tartib[i:i + 128]
            opt.zero_grad()
            F.cross_entropy(model(tr[0][j], tr[1][j]), tr[2][j]).backward()
            opt.step()
        aniqlik = float((logitlar(model, va[0], va[1]).argmax(1)
                         == va[2]).float().mean())
        if aniqlik > eng:
            eng, eng_davr = aniqlik, davr
            holat = copy.deepcopy(model.state_dict())
    model.load_state_dict(holat)
    return eng, eng_davr


def logitlar(model, x, L):
    model.eval()
    with torch.no_grad():
        return model(x, L)


def ece(ehtimol, y, qutilar=10):
    """Expected Calibration Error: |ishonch - aniqlik| ning vaznli o'rtachasi."""
    ishonch, p = ehtimol.max(1), ehtimol.argmax(1)
    chegaralar = np.linspace(0, 1, qutilar + 1)
    jami = 0.0
    for a, b in zip(chegaralar[:-1], chegaralar[1:]):
        m = (ishonch > a) & (ishonch <= b)
        if m.any():
            jami += m.mean() * abs(ishonch[m].mean() - (p[m] == y[m]).mean())
    return jami


def main() -> None:
    matnlar, y, g = korpus(6000, seed=42)
    tr, va, te = bolish(y)
    lug = Lugat(matnlar[tr])
    D = {k: (*lug.kodla(matnlar[ii]), torch.tensor(y[ii]))
         for k, ii in [("tr", tr), ("va", va), ("te", te)]}
    v = TfidfVectorizer(preprocessor=normallashtir, ngram_range=(1, 2),
                        sublinear_tf=True, min_df=2)
    lr = LogisticRegression(C=5.0, max_iter=3000)
    lr.fit(v.fit_transform(matnlar[tr]), y[tr])
    torch.manual_seed(0)
    lstm = LSTMModel(len(lug))
    orgat(lstm, D["tr"], D["va"])
    ehtimol = {"TF-IDF (1,2)": lr.predict_proba(v.transform(matnlar[te])),
               "LSTM": torch.softmax(logitlar(lstm, *D["te"][:2]), 1).numpy()}

    print("=== 1. Chalkashlik matritsasi (qator - haqiqiy, ustun - bashorat) ===")
    for nom, P in ehtimol.items():
        cm = confusion_matrix(y[te], P.argmax(1))
        print(f"  {nom}:  {'':<8}" + "".join(f"{s:>8}" for s in SINFLAR))
        for k, s in enumerate(SINFLAR):
            print(f"  {'':<{len(nom) + 3}}{s:<8}" + "".join(f"{c:>8}" for c in cm[k]))

    print("\n=== 2. Neytral sharhlar qayerdan keladi va qayerda yo'qoladi ===")
    p = ehtimol["LSTM"].argmax(1)
    neyt = y[te] == 1
    print(f"  {'guruh':<9} {'neytral n':>9} {'LSTM recall':>12}")
    for gr in ["yumshoq", "aralash", "neytral", "boshqa"]:
        m = neyt & (np.isin(g[te], ["oddiy", "inkor", "kontrast"])
                    if gr == "boshqa" else (g[te] == gr))
        print(f"  {gr:<9} {int(m.sum()):>9} {np.mean(p[m] == 1):>12.3f}")
    print(f"  neytral recall: {np.mean(p[neyt] == 1):.3f}, "
          f"salbiy {np.mean(p[y[te] == 0] == 0):.3f}, "
          f"ijobiy {np.mean(p[y[te] == 2] == 2):.3f}")

    print("\n=== 3. Ishonch jadvali (LSTM, test) ===")
    P = ehtimol["LSTM"]
    ishonch, togri = P.max(1), P.argmax(1) == y[te]
    print(f"  {'oraliq':<12} {'n':>5} {'ishonch':>13} {'aniqlik':>8}")
    for a, b in [(0.0, 0.6), (0.6, 0.8), (0.8, 0.95), (0.95, 1.0)]:
        m = (ishonch > a) & (ishonch <= b)
        print(f"  ({a:.2f}, {b:.2f}] {int(m.sum()):>5} {ishonch[m].mean():>13.3f} "
              f"{togri[m].mean():>8.3f}")

    print("\n=== 4. ECE va harorat bilan kalibrlash ===")
    lv, yv = logitlar(lstm, *D["va"][:2]), D["va"][2]
    haroratlar = np.round(np.arange(0.5, 3.01, 0.05), 2)
    nll = [F.cross_entropy(lv / T, yv).item() for T in haroratlar]
    T = float(haroratlar[int(np.argmin(nll))])          # T FAQAT validatsiyadan
    lt = logitlar(lstm, *D["te"][:2])
    kalibr = torch.softmax(lt / T, 1).numpy()
    print(f"  validatsiyada tanlangan harorat T = {T:.2f}")
    print(f"  {'model':<18} {'ECE':>6} {'ishonch':>13} {'aniqlik':>8}")
    for nom, Q in list(ehtimol.items()) + [("LSTM / T", kalibr)]:
        print(f"  {nom:<18} {ece(Q, y[te]):>6.3f} {Q.max(1).mean():>13.3f} "
              f"{np.mean(Q.argmax(1) == y[te]):>8.3f}")
    oldin, keyin = ece(ehtimol["LSTM"], y[te]), ece(kalibr, y[te])
    print("  harorat ECE ni " + ("kamaytirdi" if keyin < oldin else "kamaytirmadi")
          + "; aniqlik o'zgarmaydi (argmax bir xil)")

    print("\n=== 5. Past ishonchni odamga yuborish (kalibrlangan LSTM) ===")
    print(f"  {'chegara':>8} {'avtomatik':>10} {'aniqlik':>8} {'odamga':>7}")
    for c in [0.0, 0.6, 0.8, 0.9]:
        m = kalibr.max(1) >= c
        print(f"  {c:>8.1f} {m.mean():>10.1%} "
              f"{np.mean(kalibr.argmax(1)[m] == y[te][m]):>8.3f} "
              f"{int((~m).sum()):>7}")
    m = kalibr.max(1) < 0.8
    ulush = np.mean(y[te][m] == 1)
    print(f"  ishonch < 0.8 bo'lgan sharhlarning {ulush:.1%} i haqiqatan neytral "
          f"(umumiy ulush {np.mean(y[te] == 1):.1%})")


if __name__ == "__main__":
    main()

=== 1. Chalkashlik matritsasi (qator - haqiqiy, ustun - bashorat) ===
  TF-IDF (1,2):            salbiy neytral  ijobiy
                 salbiy       374      16      32
                 neytral       44     160      42
                 ijobiy        28      24     480
  LSTM:            salbiy neytral  ijobiy
         salbiy       389      16      17
         neytral       18     197      31
         ijobiy        22       9     501

=== 2. Neytral sharhlar qayerdan keladi va qayerda yo'qoladi ===
  guruh     neytral n  LSTM recall
  yumshoq          26        0.115
  aralash          74        0.757
  neytral         140        0.986
  boshqa            6        0.000
  neytral recall: 0.801, salbiy 0.922, ijobiy 0.942

=== 3. Ishonch jadvali (LSTM, test) ===
  oraliq           n       ishonch  aniqlik
  (0.00, 0.60]    36         0.538    0.500
  (0.60, 0.80]    83         0.718    0.771
  (0.80, 0.95]   211         0.895    0.834
  (0.95, 1.00]   870         0.991    0.953

=== 4. ECE va harorat bilan kalibrlash ===
  validatsiyada tanlangan harorat T = 1.45
  model                 ECE       ishonch  aniqlik
  TF-IDF (1,2)        0.058         0.787    0.845
  LSTM                0.044         0.942    0.906
  LSTM / T            0.029         0.898    0.906
  harorat ECE ni kamaytirdi; aniqlik o'zgarmaydi (argmax bir xil)

=== 5. Past ishonchni odamga yuborish (kalibrlangan LSTM) ===
   chegara  avtomatik  aniqlik  odamga
       0.0     100.0%    0.906       0
       0.6      95.1%    0.920      59
       0.8      82.2%    0.945     214
       0.9      68.3%    0.954     380
  ishonch < 0.8 bo'lgan sharhlarning 40.7% i haqiqatan neytral (umumiy ulush 20.5%)

print(np.mean(p == y))                                  # ⚠️ inkordagi sinish ko'rinmaydi
for gr in GURUHLAR:                                     # ✅
    print(gr, np.mean(p[g == gr] == y[g == gr]))

TfidfVectorizer(ngram_range=(1, 2))                     # ⚠️ "zo'r" to'rt xil token
TfidfVectorizer(preprocessor=normallashtir, ngram_range=(1, 2))   # ✅

TfidfVectorizer(preprocessor=normallashtir, ngram_range=(1, 1))   # ⚠️ "yaxshi emas" = "yaxshi" + "emas"
TfidfVectorizer(preprocessor=normallashtir, ngram_range=(1, 2))   # ✅ "yaxshi emas" - alohida belgi

TfidfVectorizer(stop_words=["va", "emas", "lekin"])     # ⚠️ inkor va kontrast o'chadi
TfidfVectorizer(stop_words=None)                        # ✅ sentimentda stop-so'z xavfli

out, _ = model.lstm(model.emb(x)); h = out[:, -1]       # ⚠️ qisqa sharhda - pad holati
_, (h, _) = model.lstm(pack_padded_sequence(e, L, batch_first=True,
                                            enforce_sorted=False))   # ✅ h[-1]

T = min(Ts, key=lambda t: nll(logit_test / t, y_test))  # ⚠️ test sizib ketdi
T = min(Ts, key=lambda t: nll(logit_val / t, y_val))    # ✅ faqat validatsiya

print("LSTM yumshoqda yaxshi:", acc_lstm > acc_tfidf)   # ⚠️ 108 ta sharh, tasodif
print(d.mean(), 2 * d.std(ddof=1) / np.sqrt(len(d)))    # ✅ juftlashgan farq va 2*SE

1.  # "sifati yaxshi emas" unigram xaltasi qanday tokenlardan iborat?
2.  # "yomon emas" bigrami ijobiy yoki salbiy vazn oladi?
3.  # "yoqdi" va "yoqmadi" ni farqlash uchun bigram kerakmi?
4.  # lug'at bazaviysi "kechikmasdan keldi" ni qanday baholaydi va nega?
5.  # "narxi arzon lekin sifati yomon" va "sifati yomon lekin narxi arzon" -
    # unigram modelda ehtimollar qanday?
6.  # EmbeddingBag qaysi klassik modelga eng yaqin?
7.  # LSTM da nega pack_padded_sequence kerak?
8.  # juftlashgan taqqoslashda d_i qanday qiymatlar oladi?
9.  # yumshoq guruhda aniqlik nega ~0.7 dan oshmaydi?
10. # model o'rtacha ishonchi 0.94, aniqligi 0.91 - bu qanday holat?
11. # harorat T > 1 aniqlikni o'zgartiradimi?
12. # sarkazm bo'yicha aniqlikni bu korpusda o'lchash mumkinmi?

1.  v = TfidfVectorizer(stop_words=["emas", "va", "lekin"])

2.  v = TfidfVectorizer(ngram_range=(1, 2)).fit(butun_korpus)

3.  lug = Lugat(matnlar)                               # butun korpus

4.  out, _ = lstm(emb(x)); logit = bosh(out[:, -1])     # x - padlangan

5.  print(f"LSTM kontrastda yaxshiroq: {acc_lstm_k:.3f} > {acc_tf_k:.3f}")