IlmHamroh
Data Science va sun'iy intellekt/Klassifikatsiya6/14-dars21 daqiqa
Mundarija (22)

14.6-dars: Kernel SVM

14-QISM — KLASSIFIKATSIYA · 6-dars

1. Kirish va motivatsiya

Chiziqli SVM faqat to'g'ri chegara chiza oladi. Nochiziqli chegara kerak bo'lsa, 13.5 dagi yo'lni tanlash mumkin: x1^2, x1·x2 kabi belgilar qo'shish. Lekin daraja oshgani sari belgi soni portlaydi: 100 ta belgi va 3-daraja uchun ~170 000 ta ustun.

Kernel hiylasi (kernel trick) shu muammoni chetlab o'tadi: yangi belgilarni hisoblamasdan, ular fazosidagi skalyar ko'paytmani to'g'ridan-to'g'ri hisoblash. Natijada model cheksiz o'lchovli fazoda ishlashi mumkin, hisoblash esa asl o'lchamda qoladi.

Bu darsda: kernel hiylasining mohiyati, asosiy kernellar (RBF, polinomial, sigmoid), gamma va C ning birgalikdagi ta'siri, sozlash strategiyasi, hisoblash narxi va kernel SVM qachon oqlanishi.

Real vaziyat. Ishlab chiqarish liniyasida sensor ma'lumotidan nosozlikni aniqlash kerak: 14 ta belgi, 8 000 namuna, chegara aniq nochiziqli (harorat va bosimning kombinatsiyasi muhim). Chiziqli SVM F1 = 0.61, RBF SVM (C=10, gamma=0.1) F1 = 0.84. Gradient boosting 0.85 berdi — deyarli teng, lekin SVM ni sozlash 2 ta parametr, boosting esa 6 ta parametr talab qildi.

Bu darsda kernel SVM ni o'rganamiz.

Bu darsda:

  • Kernel hiylasi
  • RBF kernel va gamma
  • Polinomial va boshqa kernellar
  • C va gamma birgalikda
  • Sozlash strategiyasi
  • Hisoblash narxi
  • Tuzoqlar
  • Amaliy: nochiziqli chegara

ℹ Misollar real numpy/sklearn bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Kernel hiylasi

text
SVM yechimi faqat SKALYAR KO'PAYTMALARGA bog'liq: x_i · x_j

Agar belgilarni fi(x) bilan boyitsak, kerak bo'lgani: fi(x_i) · fi(x_j)
Kernel — shu ko'paytmani TO'G'RIDAN-TO'G'RI hisoblaydigan funksiya:
  K(x, z) = fi(x) · fi(z)        fi ni HISOBLAMASDAN

Misol (2-daraja polinom, 2 belgi):
  K(x, z) = (x·z + 1)^2
  fi(x) = [1, sqrt(2)x1, sqrt(2)x2, x1^2, sqrt(2)x1x2, x2^2]   — 6 o'lcham
  → K ni hisoblash 2 ta ko'paytma, fi ni hisoblash 6 ta son
  RBF uchun fi CHEKSIZ o'lchovli, K esa bitta eksponenta

Kernel hiylasi — matematik nafislik: model cheksiz o'lchovli fazoda chegara chizadi, lekin hech qachon o'sha fazoga o'tmaydi. Shart: K musbat yarim aniqlangan bo'lishi kerak (Mercer sharti) — shunda u haqiqiy skalyar ko'paytmaga mos keladi.

2.2. RBF kernel va gamma

text
RBF (Gauss):  K(x, z) = exp(-gamma · ||x - z||^2)

gamma — "ta'sir radiusi"ning teskarisi:
  gamma KICHIK → keng ta'sir → silliq, deyarli chiziqli chegara (underfitting)
  gamma KATTA  → tor ta'sir → har nuqta atrofida "orolcha" (overfitting)

sklearn: gamma="scale" (standart) = 1 / (p · X.var())
         gamma="auto" = 1 / p
Masshtablash MAJBURIY — gamma masofaga bog'liq

RBF — standart tanlov: u har nuqta atrofida "qo'ng'iroq" qo'yadi va ularning yig'indisidan chegara hosil qiladi. gamma — moslashuvchanlik boshqaruvchisi: katta gamma da model har namunani alohida "yodlaydi" (k=1 li KNN ga o'xshab). gamma="scale" odatda yaxshi boshlang'ich.

2.3. Boshqa kernellar

text
linear       K = x·z                          — chiziqli SVM 14.5-bob
poly         K = (gamma·x·z + coef0)^degree   — degree 2-3; sozlash qiyin
rbf          K = exp(-gamma·||x-z||^2)        — STANDART tanlov
sigmoid      K = tanh(gamma·x·z + coef0)      — kamdan-kam foydali

Maxsus kernellar: matn uchun string kernel, graf kernellari, kosinus
precomputed  — o'zingiz hisoblagan Gram matritsasi

Amaliyotda RBF deyarli har doim birinchi tanlov; poly faqat domen mantiqan polinomial bog'liqlikni ko'rsatsa (masalan, fizika) va degree, gamma, coef0 uch parametrni sozlashga vaqt bo'lsa. sigmoid Mercer shartini har doim qanoatlantirmaydi va kamdan-kam ishlatiladi.

2.4. C va gamma birgalikda

text
        gamma kichik              gamma katta
C kichik  juda silliq             mahalliy, lekin yumshoq
          (underfitting)          chegara
C katta   silliq, lekin xatolarga  har nuqta atrofida orolcha
          toqatsiz                (kuchli overfitting)

Qidiruv: C in logspace(-2, 4), gamma in logspace(-5, 1)
  ikkalasi BIRGA qidiriladi (GridSearchCV) — ular o'zaro bog'liq

C va gamma — kernel SVM ning ikki giperparametri va ular o'zaro bog'liq: birini alohida sozlash noto'g'ri natija beradi. Odatiy yo'l — ikki o'lchovli GridSearchCV (yoki RandomizedSearchCV), logarifmik setkada. Bu sozlash SVM ning asosiy amaliy narxi.

2.5. Sozlash strategiyasi

python
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

setka = {"m__C": np.logspace(-2, 4, 7), "m__gamma": np.logspace(-5, 1, 7)}
GridSearchCV(quvur, setka, cv=5, scoring="f1_macro", n_jobs=1)

Amaliy tartib:
  1. Katta qadamli qo'pol setka (10 barobar qadam)
  2. Eng yaxshi nuqta atrofida zichroq setka
  3. Katta ma'lumotda: kichik namunada sozlab, keyin to'liqda o'qitish
  4. cache_size=1000 (MB) — tezlashtiradi

Sozlash ikki bosqichli bo'lishi kerak: avval keng, keyin zich. Katta ma'lumotda to'liq setka qidiruvi haftalab davom etishi mumkin — shuning uchun namunada sozlash (masalan, 10 000 qator) amaliy hiyla.

2.6. Hisoblash narxi

text
O'qitish:  O(n^2 · p) .. O(n^3 · p)  — n ga kvadratik/kubik!
Xotira:    Gram matritsasi O(n^2) (cache bilan boshqariladi)
Bashorat:  O(n_TV · p) — tayanch vektorlar soniga bog'liq

Amaliy chegara: n ~ 10 000..50 000 (undan keyin juda sekin)

Alternativalar katta ma'lumotda:
  · Nystroem yoki RBFSampler + LinearSVC (taxminiy kernel)
  · gradient boosting
  · neyron tarmoq

Kvadratik murakkablik — kernel SVM ning asosiy cheklovi: 100 000 qator uchun o'qitish soatlab davom etishi mumkin. Nystroem yoki RBFSampler bilan kernelni taxminiy belgilar sifatida hisoblab, so'ngra LinearSVC ishlatish — amaliy yechim (tezlik O(n) ga tushadi).

2.7. Tuzoqlar

Asosiy tuzoqlar: masshtablamaslik (gamma butunlay buziladi); C va gamma ni alohida sozlash; katta ma'lumotda kernel SVM ishlatish; gamma ni juda katta qo'yib overfitting; probability=True ni ishlatish (sklearn 1.9 da eskirgan — CalibratedClassifierCV); nomutanosib sinfda class_weight ni unutish; kernelni domen mantiqisiz tanlash; cache_size ni oshirmaslik (sekinlik).

2.8. Cheksiz o'lchamda chiziqli

Kernel hiylasi yangi belgilarni hisoblamasdan ular fazosidagi skalyar ko'paytmani beradi: model cheksiz o'lchovli fazoda chiziqli chegara chizadi, natija esa asl fazoda nochiziqli ko'rinadi. RBF — standart kernel; gamma ta'sir radiusini boshqaradi (katta gamma — overfitting), C esa xatolarga toqatni. Ular birga sozlanadi, masshtablash majburiy. Asosiy cheklov — O(n^2..n^3) murakkablik: katta ma'lumotda Nystroem/RBFSampler + LinearSVC ishlatiladi. Keyingi dars — qaror chegaralarini solishtirish.

3. Tez ma'lumotnoma

python
import numpy as np
from sklearn.kernel_approximation import Nystroem, RBFSampler
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

quvur = Pipeline([("sc", StandardScaler()),
                  ("m", SVC(kernel="rbf", cache_size=1000))])
setka = {"m__C": np.logspace(-2, 4, 7), "m__gamma": np.logspace(-5, 1, 7)}
GridSearchCV(quvur, setka, cv=5, scoring="f1_macro").fit(X_tr, y_tr)

# katta ma'lumot uchun taxminiy kernel
Pipeline([("sc", StandardScaler()),
          ("k", Nystroem(gamma=0.1, n_components=500, random_state=0)),
          ("m", LinearSVC(max_iter=10_000))])
QOIDA: masshtabla · C va gamma ni birga qidir · n > 50k bo'lsa taxminiy kernel

Kernel SVM xulosasi

K(x,z) = fi(x)·fi(z) — fi ni hisoblamasdan
RBF: exp(-gamma·||x-z||^2) · gamma katta → overfitting
C va gamma o'zaro bog'liq — birga sozlanadi
O(n^2..n^3) — katta ma'lumotda Nystroem + LinearSVC

4. Batafsil misollar

Misollar real numpy/sklearn bilan (Python 3.14).

Misol 1 — Kernel hiylasi: qo'lda tekshirish

python
"""Kernel = boyitilgan fazodagi skalyar ko'paytma (real numpy/sklearn)."""

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.svm import SVC


def main() -> None:
    rng = np.random.default_rng(3)
    x = rng.normal(0, 1, 2)
    z = rng.normal(0, 1, 2)

    print("=== 1. Polinomial kernel va aniq boyitish ===")
    kernel = (x @ z + 1) ** 2
    print(f"  K(x, z) = (x·z + 1)^2 = {kernel:.6f}")
    # mos keluvchi boyitish
    fi = lambda v: np.array([1.0, np.sqrt(2) * v[0], np.sqrt(2) * v[1],
                             v[0] ** 2, np.sqrt(2) * v[0] * v[1], v[1] ** 2])
    print(f"  fi(x)·fi(z) = {fi(x) @ fi(z):.6f}")
    print(f"  teng: {np.isclose(kernel, fi(x) @ fi(z))}")
    print(f"  kernel: 2 ta ko'paytma; boyitish: {len(fi(x))} o'lcham")

    print("\n=== 2. Belgi soni portlashi ===")
    for p, d in [(2, 2), (10, 2), (10, 3), (100, 3)]:
        from math import comb
        soni = comb(p + d, d)
        print(f"  p = {p:>3}, daraja {d}: {soni:>9} ta belgi")
    print("  (kernel bilan bu sonlar hech qachon hisoblanmaydi)")

    print("\n=== 3. RBF kerneli cheksiz o'lchovli ===")
    for gamma in [0.1, 1.0, 5.0]:
        k = np.exp(-gamma * np.sum((x - z) ** 2))
        print(f"  gamma {gamma:>4}: K(x, z) = {k:.6f}, "
              f"||x - z|| = {np.linalg.norm(x - z):.4f}")
    print("  (RBF ning Teylor yoyilmasi cheksiz hadli — cheksiz o'lchovli fi)")

    print("\n=== 4. Kernel SVM va boyitilgan chiziqli SVM tengligi ===")
    n = 300
    X = rng.normal(0, 1, (n, 2))
    y = (X[:, 0] ** 2 + X[:, 1] ** 2 > 1.2).astype(int)
    kernel_svm = SVC(kernel="poly", degree=2, gamma=1.0, coef0=1.0,
                     C=10.0).fit(X, y)
    Xp = PolynomialFeatures(2, include_bias=False).fit_transform(X)
    chiziqli = SVC(kernel="linear", C=10.0).fit(Xp, y)
    print(f"  kernel SVM (poly, d=2):        aniqlik "
          f"{(kernel_svm.predict(X) == y).mean():.4f}")
    print(f"  boyitilgan belgilar + chiziqli: aniqlik "
          f"{(chiziqli.predict(Xp) == y).mean():.4f}")
    print(f"  tayanch vektorlar: {len(kernel_svm.support_)} va "
          f"{len(chiziqli.support_)}")
    print("  ⭐ Bir xil natija, lekin kernel yo'li o'lcham portlashini chetlab o'tadi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Polinomial kernel va aniq boyitish ===
  K(x, z) = (x·z + 1)^2 = 10.918629
  fi(x)·fi(z) = 10.918629
  teng: True
  kernel: 2 ta ko'paytma; boyitish: 6 o'lcham

=== 2. Belgi soni portlashi ===
  p =   2, daraja 2:         6 ta belgi
  p =  10, daraja 2:        66 ta belgi
  p =  10, daraja 3:       286 ta belgi
  p = 100, daraja 3:    176851 ta belgi
  (kernel bilan bu sonlar hech qachon hisoblanmaydi)

=== 3. RBF kerneli cheksiz o'lchovli ===
  gamma  0.1: K(x, z) = 0.517613, ||x - z|| = 2.5662
  gamma  1.0: K(x, z) = 0.001381, ||x - z|| = 2.5662
  gamma  5.0: K(x, z) = 0.000000, ||x - z|| = 2.5662
  (RBF ning Teylor yoyilmasi cheksiz hadli — cheksiz o'lchovli fi)

=== 4. Kernel SVM va boyitilgan chiziqli SVM tengligi ===
  kernel SVM (poly, d=2):        aniqlik 0.9900
  boyitilgan belgilar + chiziqli: aniqlik 0.9900
  tayanch vektorlar: 20 va 20
  ⭐ Bir xil natija, lekin kernel yo'li o'lcham portlashini chetlab o'tadi

Nima ko'rsatdi: 2.1-bo'lim.

Misol 2 — gamma va C: moslashuvchanlik boshqaruvi

python
"""Ikki parametr chegara shaklini qanday o'zgartiradi (real numpy/sklearn)."""

import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC


def yarat(seed: int = 11, n: int = 1500):
    """Ikki halqa + shovqin."""
    rng = np.random.default_rng(seed)
    y = rng.integers(0, 2, n)
    radius = np.where(y == 1, 2.2, 1.0) + rng.normal(0, 0.35, n)
    burchak = rng.uniform(0, 2 * np.pi, n)
    X = np.column_stack([radius * np.cos(burchak), radius * np.sin(burchak)])
    almash = rng.random(n) < 0.05
    y[almash] = 1 - y[almash]
    return X, y


def main() -> None:
    X, y = yarat()
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    def baho(C: float, gamma) -> tuple[float, float, float]:
        q = Pipeline([("sc", StandardScaler()),
                      ("m", SVC(C=C, gamma=gamma, cache_size=500))])
        q.fit(Xtr, ytr)
        oquv = (q.predict(Xtr) == ytr).mean()
        cvb = cross_val_score(Pipeline([("sc", StandardScaler()),
                                        ("m", SVC(C=C, gamma=gamma))]),
                              Xtr, ytr, cv=cv).mean()
        tv = len(q.named_steps["m"].support_) / len(Xtr)
        return oquv, cvb, tv

    print("=== 1. gamma ning ta'siri (C = 1) ===")
    print(f"  {'gamma':>8} {'o_quv':>8} {'CV':>8} {'TV ulushi':>10}")
    for g in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
        oquv, cvb, tv = baho(1.0, g)
        print(f"  {g:>8} {oquv:>8.4f} {cvb:>8.4f} {tv:>10.1%}")
    print("  (gamma katta → o'quvda mukammal, CV da halokat)")

    print("\n=== 2. C ning ta'siri (gamma = 'scale') ===")
    print(f"  {'C':>8} {'o_quv':>8} {'CV':>8} {'TV ulushi':>10}")
    for C in [0.01, 0.1, 1, 10, 100, 1000]:
        oquv, cvb, tv = baho(C, "scale")
        print(f"  {C:>8} {oquv:>8.4f} {cvb:>8.4f} {tv:>10.1%}")

    print("\n=== 3. Birgalikdagi setka (CV aniqligi) ===")
    gammalar = [0.01, 0.1, 1.0, 10.0]
    Clar = [0.1, 1.0, 10.0, 100.0]
    print("  " + "gamma\\\\C".rjust(8) + "".join(f"{c:>9}" for c in Clar))
    eng = (None, -1.0)
    for g in gammalar:
        qator = []
        for C in Clar:
            b = cross_val_score(Pipeline([("sc", StandardScaler()),
                                          ("m", SVC(C=C, gamma=g))]),
                                Xtr, ytr, cv=cv).mean()
            qator.append(b)
            if b > eng[1]:
                eng = ((C, g), b)
        print(f"  {g:>8}" + "".join(f"{v:>9.4f}" for v in qator))

    print("\n=== 4. Eng yaxshi kombinatsiya ===")
    (C, g), b = eng
    q = Pipeline([("sc", StandardScaler()),
                  ("m", SVC(C=C, gamma=g))]).fit(Xtr, ytr)
    print(f"  C = {C}, gamma = {g}: CV {b:.4f}, "
          f"test {(q.predict(Xte) == yte).mean():.4f}")
    chiziqli = Pipeline([("sc", StandardScaler()),
                         ("m", SVC(kernel="linear", C=1.0))]).fit(Xtr, ytr)
    print(f"  taqqoslash uchun chiziqli SVM: test "
          f"{(chiziqli.predict(Xte) == yte).mean():.4f}")
    print("  ⭐ C va gamma o'zaro bog'liq — birga qidiriladi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. gamma ning ta'siri (C = 1) ===
     gamma    o_quv       CV  TV ulushi
     0.001   0.5162   0.5162      96.8%
      0.01   0.5581   0.5333      96.9%
       0.1   0.9200   0.9181      37.0%
       1.0   0.9200   0.9200      27.8%
      10.0   0.9210   0.9124      36.0%
     100.0   0.9552   0.8819      81.9%
  (gamma katta → o'quvda mukammal, CV da halokat)

=== 2. C ning ta'siri (gamma = 'scale') ===
         C    o_quv       CV  TV ulushi
      0.01   0.9038   0.9162      93.0%
       0.1   0.9181   0.9171      44.3%
         1   0.9219   0.9219      29.3%
        10   0.9210   0.9162      26.2%
       100   0.9190   0.9124      24.2%
      1000   0.9190   0.9114      23.5%

=== 3. Birgalikdagi setka (CV aniqligi) ===
  gamma\\C      0.1      1.0     10.0    100.0
      0.01   0.5162   0.5333   0.9019   0.9190
       0.1   0.9086   0.9181   0.9181   0.9200
       1.0   0.9162   0.9200   0.9162   0.9114
      10.0   0.9095   0.9124   0.8867   0.8619

=== 4. Eng yaxshi kombinatsiya ===
  C = 100.0, gamma = 0.1: CV 0.9200, test 0.8978
  taqqoslash uchun chiziqli SVM: test 0.5156
  ⭐ C va gamma o'zaro bog'liq — birga qidiriladi

Nima ko'rsatdi: 2.2, 2.4-bo'limlar.

Misol 3 — Kernellar taqqoslash

python
"""Turli ma'lumotda turli kernel (real numpy/sklearn)."""

import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC


def yarat(tur: str, seed: int = 7, n: int = 1200):
    rng = np.random.default_rng(seed)
    if tur == "chiziqli":
        X = rng.normal(0, 1, (n, 2))
        y = (1.3 * X[:, 0] + 0.9 * X[:, 1] + rng.normal(0, 0.4, n) > 0).astype(int)
    elif tur == "kvadratik":
        X = rng.normal(0, 1, (n, 2))
        y = (X[:, 0] ** 2 + 0.6 * X[:, 1] ** 2 + rng.normal(0, 0.25, n)
             > 1.2).astype(int)
    else:                                     # "murakkab"
        X = rng.uniform(-3, 3, (n, 2))
        y = (np.sin(1.6 * X[:, 0]) * np.cos(1.6 * X[:, 1])
             + rng.normal(0, 0.15, n) > 0).astype(int)
    return X, y


def main() -> None:
    cv = StratifiedKFold(5, shuffle=True, random_state=0)
    kernellar = {
        "linear": SVC(kernel="linear", C=1.0),
        "poly(2)": SVC(kernel="poly", degree=2, C=1.0, gamma="scale", coef0=1.0),
        "poly(3)": SVC(kernel="poly", degree=3, C=1.0, gamma="scale", coef0=1.0),
        "rbf": SVC(kernel="rbf", C=1.0, gamma="scale"),
    }

    print("=== 1. CV aniqligi (standart parametrlar) ===")
    print(f"  {'malumot':<12} " + "".join(f"{k:>10}" for k in kernellar))
    for tur in ["chiziqli", "kvadratik", "murakkab"]:
        X, y = yarat(tur)
        ballar = [cross_val_score(Pipeline([("sc", StandardScaler()), ("m", m)]),
                                  X, y, cv=cv).mean() for m in kernellar.values()]
        print(f"  {tur:<12} " + "".join(f"{b:>10.4f}" for b in ballar))

    print("\n=== 2. Sozlangan RBF ===")
    for tur in ["chiziqli", "kvadratik", "murakkab"]:
        X, y = yarat(tur)
        eng = -1.0
        eng_par = None
        for C in [0.1, 1.0, 10.0, 100.0]:
            for g in [0.01, 0.1, 1.0, 10.0]:
                b = cross_val_score(Pipeline([("sc", StandardScaler()),
                                              ("m", SVC(C=C, gamma=g))]),
                                    X, y, cv=cv).mean()
                if b > eng:
                    eng, eng_par = b, (C, g)
        print(f"  {tur:<12}: eng yaxshi C={eng_par[0]}, gamma={eng_par[1]}, "
              f"CV {eng:.4f}")

    print("\n=== 3. Tayanch vektorlar ulushi ===")
    for tur in ["chiziqli", "murakkab"]:
        X, y = yarat(tur)
        for nom in ["linear", "rbf"]:
            m = Pipeline([("sc", StandardScaler()),
                          ("m", kernellar[nom])]).fit(X, y)
            tv = len(m.named_steps["m"].support_) / len(X)
            print(f"  {tur:<11} {nom:<7}: TV ulushi {tv:.1%}")

    print("\n=== 4. Qoida ===")
    print("  chegara chiziqli bo'lsa — linear (tez va barqaror)")
    print("  nochiziqlik bor — rbf (sozlash bilan)")
    print("  poly — faqat domen mantiqi ko'rsatsa")
    print("  ⭐ RBF — standart tanlov, lekin sozlashsiz emas")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. CV aniqligi (standart parametrlar) ===
  malumot          linear   poly(2)   poly(3)       rbf
  chiziqli         0.9317    0.9258    0.9225    0.9242
  kvadratik        0.5858    0.9542    0.9508    0.9467
  murakkab         0.4800    0.5500    0.6025    0.8108

=== 2. Sozlangan RBF ===
  chiziqli    : eng yaxshi C=10.0, gamma=0.01, CV 0.9283
  kvadratik   : eng yaxshi C=10.0, gamma=0.1, CV 0.9533
  murakkab    : eng yaxshi C=100.0, gamma=1.0, CV 0.8867

=== 3. Tayanch vektorlar ulushi ===
  chiziqli    linear : TV ulushi 18.8%
  chiziqli    rbf    : TV ulushi 20.7%
  murakkab    linear : TV ulushi 95.6%
  murakkab    rbf    : TV ulushi 77.1%

=== 4. Qoida ===
  chegara chiziqli bo'lsa — linear (tez va barqaror)
  nochiziqlik bor — rbf (sozlash bilan)
  poly — faqat domen mantiqi ko'rsatsa
  ⭐ RBF — standart tanlov, lekin sozlashsiz emas

Nima ko'rsatdi: 2.3-bo'lim.

Misol 4 — Katta ma'lumot: taxminiy kernel

python
"""Nystroem va RBFSampler bilan tezlashtirish (real numpy/sklearn)."""

import warnings

import numpy as np
from sklearn.kernel_approximation import Nystroem, RBFSampler
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC


def yarat(n: int, seed: int = 5, p: int = 8):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    ball = (np.sin(1.4 * X[:, 0]) + X[:, 1] ** 2 - 0.8 * X[:, 2] * X[:, 3]
            + 0.5 * X[:, 4])
    y = (ball + rng.normal(0, 0.3, n) > 0.5).astype(int)
    return X, y


def main() -> None:
    print("=== 1. Ish hajmi n ga qanday bog'liq (RBF SVM) ===")
    for n in [1000, 2000, 4000, 8000]:
        X, y = yarat(n)
        q = Pipeline([("sc", StandardScaler()),
                      ("m", SVC(C=10.0, gamma=0.2, cache_size=500))])
        q.fit(X, y)
        tv = len(q.named_steps["m"].support_)
        print(f"  n = {n:>5}: tayanch vektorlar {tv:>5}, "
              f"Gram matritsasi {n * n * 8 / 1e6:7.1f} MB, "
              f"kernel hisoblari ~{n * n / 1e6:6.2f} mln")
    print("  (ish hajmi n^2 ga proporsional — vaqt ham shunday o'sadi)")

    print("\n=== 2. Aniq va taxminiy kernel ===")
    X, y = yarat(12_000)
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0,
                                          stratify=y)
    natijalar = {}
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        modellar = {
            "RBF SVM (aniq)": Pipeline([("sc", StandardScaler()),
                                        ("m", SVC(C=10.0, gamma=0.2,
                                                  cache_size=1000))]),
            "Nystroem(300)": Pipeline([("sc", StandardScaler()),
                                       ("k", Nystroem(gamma=0.2, n_components=300,
                                                      random_state=0)),
                                       ("m", LinearSVC(C=10.0, max_iter=10_000))]),
            "RBFSampler(600)": Pipeline([("sc", StandardScaler()),
                                         ("k", RBFSampler(gamma=0.2,
                                                          n_components=600,
                                                          random_state=0)),
                                         ("m", LinearSVC(C=10.0,
                                                         max_iter=10_000))]),
            "LinearSVC": Pipeline([("sc", StandardScaler()),
                                   ("m", LinearSVC(C=1.0, max_iter=10_000))]),
        }
        for nom, m in modellar.items():
            m.fit(Xtr, ytr)
            aniqlik = (m.predict(Xte) == yte).mean()
            natijalar[nom] = aniqlik
            if nom == "RBF SVM (aniq)":
                olcham = f"{len(m.named_steps['m'].support_)} tayanch vektor"
            elif nom == "LinearSVC":
                olcham = f"{Xtr.shape[1]} koeffitsiyent"
            else:
                olcham = f"{m.named_steps['k'].n_components} komponent"
            print(f"  {nom:<16}: aniqlik {aniqlik:.4f}, model {olcham}")

    print("\n=== 3. Komponentlar soni ta'siri (Nystroem) ===")
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        for k in [50, 150, 300, 600]:
            m = Pipeline([("sc", StandardScaler()),
                          ("k", Nystroem(gamma=0.2, n_components=k,
                                         random_state=0)),
                          ("m", LinearSVC(C=10.0, max_iter=10_000))]).fit(Xtr, ytr)
            print(f"  n_components = {k:>4}: aniqlik "
                  f"{(m.predict(Xte) == yte).mean():.4f}")

    print("\n=== 4. Tanlov ===")
    aniq = natijalar["RBF SVM (aniq)"]
    nys = natijalar["Nystroem(300)"]
    n_tr = len(Xtr)
    print(f"  aniq RBF: aniqlik {aniq:.4f} — kernel matritsasi "
          f"{n_tr * n_tr / 1e6:.1f} mln element")
    print(f"  Nystroem: aniqlik {nys:.4f} — belgi matritsasi "
          f"{n_tr * 300 / 1e6:.1f} mln element")
    print(f"  aniqlik farqi {aniq - nys:+.4f}, ish hajmi farqi "
          f"{n_tr / 300:.0f}x")
    print("  ⭐ n katta bo'lsa taxminiy kernel — amaliy yechim")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ish hajmi n ga qanday bog'liq (RBF SVM) ===
  n =  1000: tayanch vektorlar   382, Gram matritsasi     8.0 MB, kernel hisoblari ~  1.00 mln
  n =  2000: tayanch vektorlar   634, Gram matritsasi    32.0 MB, kernel hisoblari ~  4.00 mln
  n =  4000: tayanch vektorlar  1066, Gram matritsasi   128.0 MB, kernel hisoblari ~ 16.00 mln
  n =  8000: tayanch vektorlar  1799, Gram matritsasi   512.0 MB, kernel hisoblari ~ 64.00 mln
  (ish hajmi n^2 ga proporsional — vaqt ham shunday o'sadi)

=== 2. Aniq va taxminiy kernel ===
  RBF SVM (aniq)  : aniqlik 0.9150, model 1937 tayanch vektor
  Nystroem(300)   : aniqlik 0.9164, model 300 komponent
  RBFSampler(600) : aniqlik 0.9092, model 600 komponent
  LinearSVC       : aniqlik 0.7056, model 8 koeffitsiyent

=== 3. Komponentlar soni ta'siri (Nystroem) ===
  n_components =   50: aniqlik 0.8244
  n_components =  150: aniqlik 0.9056
  n_components =  300: aniqlik 0.9164
  n_components =  600: aniqlik 0.9253

=== 4. Tanlov ===
  aniq RBF: aniqlik 0.9150 — kernel matritsasi 70.6 mln element
  Nystroem: aniqlik 0.9164 — belgi matritsasi 2.5 mln element
  aniqlik farqi -0.0014, ish hajmi farqi 28x
  ⭐ n katta bo'lsa taxminiy kernel — amaliy yechim

Nima ko'rsatdi: 2.6-bo'lim.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Kernel yangi belgilar yaratadi" Ularni hisoblamaydi
"RBF har doim eng yaxshi" Chiziqli chegarada ortiqcha
"gamma katta — aniqroq" Overfitting
"C va gamma mustaqil" O'zaro bog'liq
"Kernel SVM katta ma'lumot uchun" O(n^2..n^3)
"Masshtablash ixtiyoriy" gamma buziladi
"poly kernel oson" Uch parametr sozlanadi
"Taxminiy kernel yomon" Ko'pincha deyarli teng

6. Keng tarqalgan xatolar va yechimlari

1. Masshtablamaslik

python
SVC(kernel="rbf").fit(X, y)                                       # ⚠️
Pipeline([("sc", StandardScaler()), ("m", SVC())])                # ✅

2. C va gamma ni alohida sozlash

python
# avval C ni, keyin gamma ni qidirish                             # ⚠️
GridSearchCV(pipe, {"m__C": ..., "m__gamma": ...}, cv=5)          # ✅

3. Katta ma'lumotda kernel SVM

python
SVC(kernel="rbf").fit(X_200k, y)                                  # ⚠️
Pipeline([("k", Nystroem(...)), ("m", LinearSVC())])              # ✅

4. gamma ni juda katta qo'yish

python
SVC(gamma=1000)                        # har nuqta orolcha        # ⚠️
# CV bilan tanlang: logspace(-5, 1)                               # ✅

5. probability=True (eskirgan)

python
SVC(probability=True).fit(X, y)        # sklearn 1.9 da eskirgan  # ⚠️
CalibratedClassifierCV(SVC(), ensemble=False, cv=5)  # yoki decision_function # ✅

6. cache_size ni oshirmaslik

python
SVC()                                  # cache 200 MB            # ⚠️
SVC(cache_size=1000)                                              # ✅

7. Nomutanosib sinf

python
SVC().fit(X, y)                        # 2% musbat sinf           # ⚠️
SVC(class_weight="balanced")           # + chegara 12.7-bob         # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 13.5-dars (o'tilgan): Nochiziqli belgilar
  • 14.5-dars (o'tilgan): Chiziqli SVM
  • 14.7-dars: Qaror chegaralarini solishtirish
  • 14.12-dars: Modellarni tanlash
  • 18-qism: Kernel usullari va o'lchamni kamaytirish

8. Eng yaxshi amaliyotlar

  1. Har doim masshtablang.

  2. C va gamma ni birga qidiring.

  3. Qo'pol setkadan zichga o'ting.

  4. Chiziqli SVM bilan solishtiring.

  5. n > 50 000 bo'lsa taxminiy kernel.

  6. cache_size ni oshiring.

  7. probability ni faqat kerak bo'lsa.

  8. Tayanch vektorlar ulushini kuzating.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # kernel hiylasi nima?
2.  # Mercer sharti?
3.  # RBF formulasi?
4.  # gamma kichik bo'lsa?
5.  # gamma katta bo'lsa?
6.  # gamma="scale" nima?
7.  # C va gamma bog'liqmi?
8.  # kernel SVM murakkabligi?
9.  # amaliy n chegarasi?
10. # katta ma'lumotda nima qilinadi?
11. # poly kerneldagi parametrlar?
12. # masshtablash shartmi?
Javoblar
  1. Boyitilgan fazodagi skalyar ko'paytmani to'g'ridan-to'g'ri hisoblash
  2. Kernel musbat yarim aniqlangan bo'lishi
  3. exp(-gamma·||x-z||^2)
  4. Silliq chegara
  5. Overfitting
  6. 1/(p·X.var())
  7. Ha
  8. O(n^2..n^3)
  9. ~10k-50k
  10. Nystroem/RBFSampler + LinearSVC
  11. degree, gamma, coef0
  12. Ha

Vazifa 2: Xatolarni tuzating

python
1.  SVC(kernel="rbf").fit(X, y)   # turli birlikli belgilar

2.  SVC(gamma=500)

3.  SVC(kernel="rbf").fit(X_150k, y)

4.  # avval C, keyin gamma qidirish

5.  SVC(probability=True)   # sklearn 1.9
Javoblar
python
1.  Pipeline([("sc", StandardScaler()), ("m", SVC())])

2.  GridSearchCV(pipe, {"m__gamma": np.logspace(-5, 1, 7)}, cv=5)

3.  Pipeline([("k", Nystroem(...)), ("m", LinearSVC())])

4.  GridSearchCV(pipe, {"m__C": ..., "m__gamma": ...}, cv=5)

5.  CalibratedClassifierCV(SVC(), ensemble=False)  # yoki SVC() + decision_function

Vazifa 3: Kernel hiylasi

Modellang:

  1. Polinomial kernel
  2. Aniq boyitish
  3. Tenglik
  4. O'lcham portlashi

Vazifa 4: gamma va C

Modellang:

  1. gamma setkasi
  2. C setkasi
  3. Birgalikdagi jadval
  4. Eng yaxshi kombinatsiya

Vazifa 5: Kernellar

Modellang:

  1. Uch ma'lumot turi
  2. To'rt kernel
  3. Sozlangan RBF
  4. Tavsiya

Vazifa 6: Katta ma'lumot

Modellang:

  1. Vaqt o'sishi
  2. Taxminiy kernel
  3. Komponentlar soni
  4. Tanlov

Vazifa 7: O'ylash

Kernel usullari 2000-yillarda ML nazariyasining markazida edi va "cheksiz o'lchovli fazoda o'qitish" g'oyasi inqilobiy hisoblangan. Bugun esa neyron tarmoqlar vakillikni o'zi o'rganadi. Bu ikki yondashuv o'rtasidagi tub farq nimada?

Javob

Qisqa javob: kernel usullarida vakillik oldindan belgilangan (kernel tanlovi bilan), neyron tarmoqlarda esa u ma'lumotdan o'rganiladi. Birinchisi — qo'lda tanlangan cheksiz fazo, ikkinchisi — moslashuvchan chekli fazo.

1. Tub farq

Kernel usullari Neyron tarmoqlar
Vakillik = kernel tanlovi Vakillik o'rganiladi
Qavariq optimallashtirish (global minimum) Qavariq emas (lokal minimumlar)
O(n^2) — ma'lumot hajmiga qarshi O(n) — ma'lumot ko'p bo'lsa yaxshiroq
Kam ma'lumotda kuchli Ko'p ma'lumotda kuchli
Nazariy kafolatlar bor Nazariya kamroq

2. Nega tarmoqlar yutdi

  • Ma'lumot hajmi o'sdi (kernel O(n^2) bilan raqobat qila olmaydi)
  • Rasm/matn/ovozda o'rganilgan vakillik qo'lda tanlangandan yaxshiroq
  • GPU hisoblash chiziqli murakkablikni afzal qiladi

3. Kernel g'oyalari qayerda qoldi

  • Gauss jarayonlari (noaniqlik baholash)
  • Neyron Tangent Kernel — tarmoqlarni tahlil qilish vositasi
  • Attention mexanizmi kernel sifatida talqin qilinadi
  • Kichik ma'lumotli vazifalarda hali ham amaliy

4. Amaliy xulosa

  1. Jadval ma'lumotida: boosting yoki chiziqli model
  2. Kam namuna + nochiziqlik: kernel SVM yaxshi tanlov
  3. Rasm/matn: oldindan o'qitilgan tarmoq
  4. Nazariy tushuncha: kernel — vakillik haqida o'ylashning eng toza yo'li

5. Xulosa

  1. Farq — vakillik qayerdan keladi
  2. Har ikkalasi ham "chiziqli model + boyitilgan fazo"
  3. Ma'lumot hajmi tanlovni belgilaydi
  4. Kernel nazariyasi hali ham foydali

Nimani mustahkamlaydi: 2.1, 2.6-bo'limlar.

Xulosa

Bu darsda kernel SVM ni o'rgandik.

Eng muhim uch fikr:

  1. Kernel hiylasi. SVM yechimi faqat skalyar ko'paytmalarga bog'liq, shuning uchun boyitilgan fazodagi ko'paytmani K(x, z) bilan to'g'ridan-to'g'ri hisoblash mumkin — yangi belgilarni hech qachon yaratmasdan. RBF uchun bu fazo cheksiz o'lchovli, hisoblash esa bitta eksponenta.

  2. gamma va C birga sozlanadi. gamma — ta'sir radiusining teskarisi: katta gamma da model har namuna atrofida "orolcha" yasaydi (o'quvda 100%, CV da halokat); C — xatolarga toqat. Ular o'zaro bog'liq, shuning uchun ikki o'lchovli logarifmik setkada birga qidiriladi. Masshtablash majburiy — gamma masofaga bog'liq.

  3. Kvadratik murakkablik — asosiy cheklov. O(n^2..n^3) o'qitish vaqti kernel SVM ni ~10 000-50 000 qator bilan cheklaydi. Kattaroq ma'lumotda Nystroem yoki RBFSampler bilan kernelni taxminiy belgilarga aylantirib, LinearSVC ishlatiladi — tezlik O(n) ga tushadi, aniqlik esa ko'pincha deyarli teng qoladi.

Keyingi darsda qaror chegaralarini solishtirishni o'rganamiz: barcha algoritmlarni bir xil ma'lumotda ko'rib, ularning kuchli va zaif tomonlarini birga baholaymiz.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
14.6-dars: Kernel SVM — IlmHamroh