IlmHamroh
Data Science va sun'iy intellekt/Daraxtlar va ansambllar10/14-dars18 daqiqa
Mundarija (22)

15.10-dars: XGBoost va LightGBM

15-QISM — DARAXTLAR VA ANSAMBLLAR · 10-dars

1. Kirish va motivatsiya

Gradient boosting g'oyasi 2001 yilda paydo bo'lgan, lekin uning sanoat standartiga aylanishi ikki kutubxona bilan bog'liq: XGBoost (2014) va LightGBM (2017). Ular bir xil matematikani ancha tezroq va aniqroq amalga oshiradi.

Farq faqat tezlikda emas: XGBoost ikkinchi tartibli yaqinlashish va aniq L1/L2 regulyarizatsiya qo'shdi, LightGBM esa barg bo'yicha o'sish (leaf-wise) va gistogramma yondashuvini joriy qildi. Ikkalasi ham yo'qolgan qiymatlar va (LightGBM) kategoriyalar bilan to'g'ridan-to'g'ri ishlaydi.

Bu darsda: XGBoost va LightGBM API lari, asosiy giperparametrlar va ularning sklearn dagi ekvivalentlari, early_stopping, kategoriyali belgilar, scale_pos_weight va amaliy tanlov.

Real vaziyat. Bankda 3 mln qatorli, 180 belgili ma'lumotda sklearn GradientBoostingClassifier bir kechada ham tugamadi. LightGBM bir xil vazifani 9 daqiqada bajardi va ROC AUC 0.03 ga yuqori chiqdi. Kutubxona tanlovi shu loyihada modelni umuman mumkin qildi.

Bu darsda XGBoost va LightGBM ni o'rganamiz.

Bu darsda:

  • Asosiy farqlar
  • Giperparametrlar mosligi
  • sklearn API
  • Erta to'xtatish
  • Kategoriyali belgilar
  • Nomutanosib sinflar
  • Tuzoqlar
  • Amaliy: uch kutubxona

ℹ Misollar real numpy/pandas/sklearn/xgboost/lightgbm bilan (Python 3.14).

2. Nazariya — chuqur tushuntirish

2.1. Asosiy farqlar

text
                     sklearn Hist*     XGBoost          LightGBM
o'sish               barg bo'yicha     daraja bo'yicha  barg bo'yicha
                     (max_leaf_nodes)  (max_depth)      (num_leaves)
gistogramma          ha                ha (hist)        ha
L1/L2                l2 only           reg_alpha/lambda lambda_l1/l2
NaN                  avtomatik         avtomatik        avtomatik
kategoriya           from_dtype        enable_categorical  avtomatik
GPU                  yo'q              ha               ha
tezlik               yaxshi            yaxshi           eng yaxshi

Barg bo'yicha o'sish (leaf-wise) — LightGBM ning asosiy g'oyasi: daraxt eng katta foyda beradigan bargdan o'sadi, daraja bo'yicha emas. Bu bir xil barglar sonida kichikroq yo'qotish beradi, lekin overfitting xavfini oshiradi — shuning uchun num_leaves ni ehtiyot bilan tanlash kerak.

2.2. Giperparametrlar mosligi

text
sklearn Hist*           XGBoost              LightGBM
learning_rate           learning_rate (eta)  learning_rate
max_iter                n_estimators         n_estimators
max_leaf_nodes          max_leaves           num_leaves
max_depth               max_depth            max_depth
min_samples_leaf        min_child_weight     min_child_samples
l2_regularization       reg_lambda           lambda_l2
-                       reg_alpha            lambda_l1
max_features            colsample_bytree     feature_fraction
-                       subsample            bagging_fraction
max_bins                max_bin              max_bin

min_child_weight (XGBoost) va min_samples_leaf (sklearn) bir xil emas: birinchisi bargdagi gessian yig'indisini, ikkinchisi namunalar sonini cheklaydi. Klassifikatsiyada min_child_weight=1 juda kam cheklov beradi.

2.3. sklearn API

python
from xgboost import XGBClassifier, XGBRegressor
from lightgbm import LGBMClassifier, LGBMRegressor

x = XGBClassifier(n_estimators=1000, learning_rate=0.05, max_depth=4,
                  subsample=0.8, colsample_bytree=0.8,
                  reg_lambda=1.0, tree_method="hist",
                  early_stopping_rounds=50, eval_metric="auc",
                  random_state=0).fit(Xtr, ytr, eval_set=[(Xval, yval)],
                                      verbose=False)

l = LGBMClassifier(n_estimators=1000, learning_rate=0.05, num_leaves=31,
                   min_child_samples=20, subsample=0.8, colsample_bytree=0.8,
                   reg_lambda=1.0, random_state=0, verbose=-1)

Ikkala kutubxona ham sklearn API ni to'liq qo'llab-quvvatlaydi: Pipeline, GridSearchCV, cross_val_score bilan ishlaydi. Shuning uchun ularni sklearn modellari kabi ishlatish mumkin.

2.4. Erta to'xtatish

python
# XGBoost (2.0+): konstruktorda
XGBClassifier(early_stopping_rounds=50, eval_metric="auc")
  .fit(Xtr, ytr, eval_set=[(Xval, yval)], verbose=False)
x.best_iteration, x.best_score

# LightGBM: callback orqali
import lightgbm as lgb
l.fit(Xtr, ytr, eval_set=[(Xval, yval)], eval_metric="auc",
      callbacks=[lgb.early_stopping(50, verbose=False)])
l.best_iteration_

eval_set ga test to'plamini bermang (15.9 — leakage). Bu ikki kutubxonada validatsiya to'plami qo'lda beriladi, shuning uchun xato qilish osonroq.

2.5. Kategoriyali belgilar

python
# LightGBM: pandas category dtype ni o'zi tanidi
df["hudud"] = df["hudud"].astype("category")
LGBMClassifier().fit(df, y)          # avtomatik

# XGBoost: enable_categorical
XGBClassifier(enable_categorical=True, tree_method="hist").fit(df, y)

# Ichki mexanizm: kategoriyalarni maqsad bo'yicha saralab bo'lish
#   one-hot dan ancha samarali (ayniqsa ko'p darajali kategoriyalarda)

Ko'p darajali kategoriyalarda (100+ qiymat) bu mexanizm one-hot dan ancha ustun: one-hot har darajaga alohida ustun beradi va daraxtni sayozlashtiradi, ichki mexanizm esa darajalarni guruhlarga bo'la oladi.

2.6. Nomutanosib sinflar

text
scale_pos_weight = manfiy / musbat        (XGBoost, LightGBM)
  ehtimolliklarni buzadi 14.10-bob

is_unbalance=True (LightGBM) — avtomatik

Muqobillar 14.9-bob:
  - chegarani sozlash (eng xavfsiz)
  - average_precision bo'yicha sozlash
  - max_delta_step (XGBoost) — juda nomutanosibda barqarorlik

scale_pos_weight ni faqat aniqlik metrikasi tartib bo'yicha (AUC) bo'lmagan hollarda ishlating. Ehtimollik kerak bo'lsa, uni qo'ymang va chegarani sozlang.

2.7. Tuzoqlar

Asosiy tuzoqlar: eval_set ga test berish; LightGBM da num_leaves ni 2^max_depth dan katta qo'yish; min_child_weight ni min_samples_leaf bilan adashtirish; kichik ma'lumotda LightGBM ishlatish (min_child_samples sabab bo'sh model); verbose ni o'chirmaslik (loglar oqadi); scale_pos_weight dan keyin ehtimolliklarga ishonish; kategoriyalarni astype("category") qilmaslik; n_estimators ni erta to'xtatishsiz qo'yish.

2.8. Sanoat standarti

XGBoost va LightGBM — gradient boosting ning sanoat standarti. LightGBM barg bo'yicha o'sadi (num_leaves) va odatda eng tez, XGBoost esa daraja bo'yicha (max_depth) va barqarorroq. Ikkalasi ham NaN va kategoriyalar bilan to'g'ridan-to'g'ri ishlaydi, L1/L2 regulyarizatsiyaga ega va sklearn API ni qo'llab-quvvatlaydi. eval_set ga hech qachon test bermang. Keyingi dars — belgi muhimligi.

3. Tez ma'lumotnoma

python
from lightgbm import LGBMClassifier, early_stopping
from xgboost import XGBClassifier

x = XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=4,
                  subsample=0.8, colsample_bytree=0.8, reg_lambda=1.0,
                  tree_method="hist", early_stopping_rounds=50,
                  eval_metric="auc", random_state=0)
x.fit(Xtr, ytr, eval_set=[(Xval, yval)], verbose=False)
x.best_iteration

l = LGBMClassifier(n_estimators=2000, learning_rate=0.05, num_leaves=31,
                   min_child_samples=20, colsample_bytree=0.8,
                   reg_lambda=1.0, random_state=0, verbose=-1)
l.fit(Xtr, ytr, eval_set=[(Xval, yval)],
      callbacks=[early_stopping(50, verbose=False)])
QOIDA: eval_set = validatsiya (test EMAS) · num_leaves ni chekla ·
       kategoriyani category dtype qil

Kutubxonalar xulosasi

LightGBM: barg bo'yicha (num_leaves), eng tez, kategoriya avtomatik
XGBoost:  daraja bo'yicha (max_depth), barqaror, reg_alpha/lambda
sklearn Hist*: o'rnatish shart emas, sodda, yetarlicha tez
Uchalasi ham NaN ni o'zi boshqaradi

4. Batafsil misollar

Misollar real numpy/pandas/sklearn/xgboost/lightgbm bilan (Python 3.14).

Misol 1 — Uch kutubxona yonma-yon

python
"""sklearn Hist*, XGBoost va LightGBM (real numpy/sklearn/xgboost/lightgbm)."""

import warnings

import numpy as np
from lightgbm import LGBMClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier


def yarat(seed: int = 7, n: int = 12000, p: int = 18, shovqin: float = 0.10):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    qoida = (((X[:, 0] > 0.25) & (X[:, 1] < 0.25))
             | ((X[:, 2] > 0.4) & (X[:, 3] > 0.0) & (X[:, 5] < 0.6))
             | (X[:, 4] < -1.0))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    warnings.simplefilter("ignore")
    X, y = yarat()
    Xtr, Xqol, ytr, yqol = train_test_split(X, y, test_size=0.4,
                                            random_state=0, stratify=y)
    Xval, Xte, yval, yte = train_test_split(Xqol, yqol, test_size=0.5,
                                            random_state=0, stratify=yqol)

    print("=== 1. Ma'lumot ===")
    print(f"  o'quv {len(Xtr)}, validatsiya {len(Xval)}, test {len(Xte)}")
    print(f"  belgilar {X.shape[1]}, musbat sinf {y.mean():.2%}")

    print("\n=== 2. Uch model (bir xil eta, erta to'xtatish) ===")
    natijalar = {}

    h = HistGradientBoostingClassifier(learning_rate=0.05, max_iter=2000,
                                       early_stopping=True,
                                       validation_fraction=0.15,
                                       n_iter_no_change=50,
                                       random_state=0).fit(Xtr, ytr)
    natijalar["sklearn Hist"] = (h.n_iter_,
                                 roc_auc_score(yte, h.predict_proba(Xte)[:, 1]))

    x = XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=4,
                      subsample=0.8, colsample_bytree=0.8, reg_lambda=1.0,
                      tree_method="hist", early_stopping_rounds=50,
                      eval_metric="auc", random_state=0, n_jobs=1)
    x.fit(Xtr, ytr, eval_set=[(Xval, yval)], verbose=False)
    natijalar["XGBoost"] = (x.best_iteration + 1,
                            roc_auc_score(yte, x.predict_proba(Xte)[:, 1]))

    import lightgbm as lgb
    l = LGBMClassifier(n_estimators=2000, learning_rate=0.05, num_leaves=31,
                       min_child_samples=20, colsample_bytree=0.8,
                       reg_lambda=1.0, random_state=0, verbose=-1, n_jobs=1)
    l.fit(Xtr, ytr, eval_set=[(Xval, yval)], eval_metric="auc",
          callbacks=[lgb.early_stopping(50, verbose=False)])
    natijalar["LightGBM"] = (l.best_iteration_,
                             roc_auc_score(yte, l.predict_proba(Xte)[:, 1]))

    print(f"  {'kutubxona':<16} {'qadam':>7} {'test AUC':>10}")
    for nom, (qadam, auc) in natijalar.items():
        print(f"  {nom:<16} {qadam:>7} {auc:>10.4f}")

    print("\n=== 3. Model hajmi ===")
    print(f"  sklearn Hist: {h.n_iter_} qadam, "
          f"max_leaf_nodes={h.max_leaf_nodes}")
    xg = x.get_booster().trees_to_dataframe()
    print(f"  XGBoost: {x.best_iteration + 1} daraxt, "
          f"jami {len(xg):,} tugun")
    print(f"  LightGBM: {l.best_iteration_} daraxt, "
          f"num_leaves={l.num_leaves}")

    print("\n=== 4. Bashoratlarning o'zaro kelishuvi ===")
    ph = h.predict_proba(Xte)[:, 1]
    px = x.predict_proba(Xte)[:, 1]
    pl = l.predict_proba(Xte)[:, 1]
    print(f"  Hist - XGB korrelyatsiya:  {np.corrcoef(ph, px)[0, 1]:.4f}")
    print(f"  Hist - LGBM korrelyatsiya: {np.corrcoef(ph, pl)[0, 1]:.4f}")
    print(f"  XGB - LGBM korrelyatsiya:  {np.corrcoef(px, pl)[0, 1]:.4f}")
    ortacha = (ph + px + pl) / 3
    print(f"  uchtasining o'rtachasi: AUC "
          f"{roc_auc_score(yte, ortacha):.4f}")
    print("  ⭐ Bir xil matematika, yaqin natijalar")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ma'lumot ===
  o'quv 7200, validatsiya 2400, test 2400
  belgilar 18, musbat sinf 45.02%

=== 2. Uch model (bir xil eta, erta to'xtatish) ===
  kutubxona          qadam   test AUC
  sklearn Hist         101     0.8969
  XGBoost                7     0.8993
  LightGBM               9     0.8927

=== 3. Model hajmi ===
  sklearn Hist: 101 qadam, max_leaf_nodes=31
  XGBoost: 7 daraxt, jami 1,637 tugun
  LightGBM: 9 daraxt, num_leaves=31

=== 4. Bashoratlarning o'zaro kelishuvi ===
  Hist - XGB korrelyatsiya:  0.9560
  Hist - LGBM korrelyatsiya: 0.9660
  XGB - LGBM korrelyatsiya:  0.9733
  uchtasining o'rtachasi: AUC 0.8953
  ⭐ Bir xil matematika, yaqin natijalar

Nima ko'rsatdi: 2.1, 2.3, 2.4-bo'limlar.

Misol 2 — num_leaves va max_depth

python
"""Barg bo'yicha va daraja bo'yicha o'sish (real numpy/lightgbm/xgboost)."""

import warnings

import numpy as np
from lightgbm import LGBMClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier


def yarat(seed: int = 14, n: int = 10000, p: int = 15, shovqin: float = 0.12):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, p))
    qoida = (((X[:, 0] > 0.3) & (X[:, 1] < 0.2))
             | ((X[:, 2] > 0.4) & (X[:, 3] > 0.0))
             | (X[:, 4] < -1.1))
    y = qoida.astype(int)
    alm = rng.random(n) < shovqin
    y[alm] = 1 - y[alm]
    return X, y


def main() -> None:
    warnings.simplefilter("ignore")
    X, y = yarat()
    Xtr, Xqol, ytr, yqol = train_test_split(X, y, test_size=0.4,
                                            random_state=0, stratify=y)
    Xval, Xte, yval, yte = train_test_split(Xqol, yqol, test_size=0.5,
                                            random_state=0, stratify=yqol)
    import lightgbm as lgb

    print("=== 1. LightGBM: num_leaves ===")
    print(f"  {'num_leaves':>11} {'qadam':>7} {'o_quv AUC':>11} "
          f"{'test AUC':>10}")
    for nl in [7, 15, 31, 63, 255]:
        m = LGBMClassifier(n_estimators=1500, learning_rate=0.05,
                           num_leaves=nl, min_child_samples=20,
                           random_state=0, verbose=-1, n_jobs=1)
        m.fit(Xtr, ytr, eval_set=[(Xval, yval)], eval_metric="auc",
              callbacks=[lgb.early_stopping(50, verbose=False)])
        a1 = roc_auc_score(ytr, m.predict_proba(Xtr)[:, 1])
        a2 = roc_auc_score(yte, m.predict_proba(Xte)[:, 1])
        print(f"  {nl:>11} {m.best_iteration_:>7} {a1:>11.4f} {a2:>10.4f}")

    print("\n=== 2. XGBoost: max_depth ===")
    print(f"  {'max_depth':>10} {'qadam':>7} {'o_quv AUC':>11} "
          f"{'test AUC':>10}")
    for d in [2, 3, 4, 6, 10]:
        m = XGBClassifier(n_estimators=1500, learning_rate=0.05, max_depth=d,
                          subsample=0.8, colsample_bytree=0.8,
                          tree_method="hist", early_stopping_rounds=50,
                          eval_metric="auc", random_state=0, n_jobs=1)
        m.fit(Xtr, ytr, eval_set=[(Xval, yval)], verbose=False)
        a1 = roc_auc_score(ytr, m.predict_proba(Xtr)[:, 1])
        a2 = roc_auc_score(yte, m.predict_proba(Xte)[:, 1])
        print(f"  {d:>10} {m.best_iteration + 1:>7} {a1:>11.4f} "
              f"{a2:>10.4f}")

    print("\n=== 3. min_child_samples (LightGBM) ===")
    print(f"  {'min_child':>10} {'qadam':>7} {'test AUC':>10}")
    for mcs in [1, 20, 100, 400]:
        m = LGBMClassifier(n_estimators=1500, learning_rate=0.05,
                           num_leaves=31, min_child_samples=mcs,
                           random_state=0, verbose=-1, n_jobs=1)
        m.fit(Xtr, ytr, eval_set=[(Xval, yval)], eval_metric="auc",
              callbacks=[lgb.early_stopping(50, verbose=False)])
        print(f"  {mcs:>10} {m.best_iteration_:>7} "
              f"{roc_auc_score(yte, m.predict_proba(Xte)[:, 1]):>10.4f}")

    print("\n=== 4. Regulyarizatsiya (L1 va L2) ===")
    print(f"  {'lambda_l1':>10} {'lambda_l2':>10} {'test AUC':>10}")
    for l1, l2 in [(0.0, 0.0), (0.0, 5.0), (1.0, 0.0), (1.0, 5.0)]:
        m = LGBMClassifier(n_estimators=1500, learning_rate=0.05,
                           num_leaves=31, reg_alpha=l1, reg_lambda=l2,
                           random_state=0, verbose=-1, n_jobs=1)
        m.fit(Xtr, ytr, eval_set=[(Xval, yval)], eval_metric="auc",
              callbacks=[lgb.early_stopping(50, verbose=False)])
        print(f"  {l1:>10.1f} {l2:>10.1f} "
              f"{roc_auc_score(yte, m.predict_proba(Xte)[:, 1]):>10.4f}")
    print("  ⭐ num_leaves - LightGBM ning eng muhim parametri")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. LightGBM: num_leaves ===
   num_leaves   qadam   o_quv AUC   test AUC
            7      39      0.8813     0.8875
           15      11      0.8980     0.8845
           31      23      0.9392     0.8840
           63       5      0.9345     0.8858
          255      51      0.9921     0.8866

=== 2. XGBoost: max_depth ===
   max_depth   qadam   o_quv AUC   test AUC
           2      50      0.8778     0.8943
           3      56      0.8848     0.8939
           4      51      0.9077     0.8901
           6     101      0.9745     0.8887
          10      41      0.9862     0.8859

=== 3. min_child_samples (LightGBM) ===
   min_child   qadam   test AUC
           1      16     0.8814
          20      23     0.8840
         100      11     0.8863
         400      13     0.8913

=== 4. Regulyarizatsiya (L1 va L2) ===
   lambda_l1  lambda_l2   test AUC
         0.0        0.0     0.8840
         0.0        5.0     0.8893
         1.0        0.0     0.8824
         1.0        5.0     0.8872
  ⭐ num_leaves - LightGBM ning eng muhim parametri

Nima ko'rsatdi: 2.1, 2.2-bo'limlar.

Misol 3 — Kategoriyalar va NaN

python
"""To'g'ridan-to'g'ri qo'llab-quvvatlash (real pandas/lightgbm/xgboost)."""

import warnings

import numpy as np
import pandas as pd
from lightgbm import LGBMClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier


def yarat(seed: int = 3, n: int = 15000, darajalar: int = 60) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    tuman = rng.integers(0, darajalar, n)
    # har tumanning yashirin qiyinligi
    qiyinlik = rng.normal(0, 1, darajalar)
    tur = rng.choice(["oddiy", "tezkor", "yirik"], n, p=[0.6, 0.3, 0.1])
    masofa = rng.gamma(3, 60, n)
    ogirlik = rng.gamma(2, 4, n)
    tt = pd.Series(tur).map({"oddiy": 0.0, "tezkor": -0.9,
                             "yirik": 1.0}).to_numpy()
    kuch = -2.2 + 0.005 * masofa + 0.04 * ogirlik + 1.2 * qiyinlik[tuman] + tt
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    df = pd.DataFrame({"tuman": pd.Categorical(tuman.astype(str)),
                       "tur": pd.Categorical(tur), "masofa": masofa,
                       "ogirlik": ogirlik, "kechikdi": y})
    yoq = rng.random(n) < 0.05
    df.loc[yoq, "ogirlik"] = np.nan
    return df


def main() -> None:
    warnings.simplefilter("ignore")
    import lightgbm as lgb
    df = yarat()
    X = df.drop(columns="kechikdi")
    y = df["kechikdi"]
    Xtr, Xqol, ytr, yqol = train_test_split(X, y, test_size=0.4,
                                            random_state=0, stratify=y)
    Xval, Xte, yval, yte = train_test_split(Xqol, yqol, test_size=0.5,
                                            random_state=0, stratify=yqol)

    print("=== 1. Ma'lumot ===")
    print(f"  {len(df)} qator, tuman darajalari "
          f"{df['tuman'].nunique()}, NaN {int(df['ogirlik'].isna().sum())}")

    print("\n=== 2. LightGBM: kategoriya avtomatik ===")
    l = LGBMClassifier(n_estimators=1500, learning_rate=0.05, num_leaves=31,
                       random_state=0, verbose=-1, n_jobs=1)
    l.fit(Xtr, ytr, eval_set=[(Xval, yval)], eval_metric="auc",
          callbacks=[lgb.early_stopping(50, verbose=False)])
    al = roc_auc_score(yte, l.predict_proba(Xte)[:, 1])
    print(f"  qadam {l.best_iteration_}, test AUC {al:.4f}")

    print("\n=== 3. XGBoost: enable_categorical ===")
    x = XGBClassifier(n_estimators=1500, learning_rate=0.05, max_depth=5,
                      enable_categorical=True, tree_method="hist",
                      early_stopping_rounds=50, eval_metric="auc",
                      random_state=0, n_jobs=1)
    x.fit(Xtr, ytr, eval_set=[(Xval, yval)], verbose=False)
    ax = roc_auc_score(yte, x.predict_proba(Xte)[:, 1])
    print(f"  qadam {x.best_iteration + 1}, test AUC {ax:.4f}")

    print("\n=== 4. One-hot bilan solishtirish ===")
    Xtr_oh = pd.get_dummies(Xtr, columns=["tuman", "tur"])
    Xval_oh = pd.get_dummies(Xval, columns=["tuman", "tur"]).reindex(
        columns=Xtr_oh.columns, fill_value=0)
    Xte_oh = pd.get_dummies(Xte, columns=["tuman", "tur"]).reindex(
        columns=Xtr_oh.columns, fill_value=0)
    l2 = LGBMClassifier(n_estimators=1500, learning_rate=0.05, num_leaves=31,
                        random_state=0, verbose=-1, n_jobs=1)
    l2.fit(Xtr_oh, ytr, eval_set=[(Xval_oh, yval)], eval_metric="auc",
           callbacks=[lgb.early_stopping(50, verbose=False)])
    a2 = roc_auc_score(yte, l2.predict_proba(Xte_oh)[:, 1])
    print(f"  belgilar soni: ichki {Xtr.shape[1]}, one-hot {Xtr_oh.shape[1]}")
    print(f"  ichki mexanizm: AUC {al:.4f}, qadam {l.best_iteration_}")
    print(f"  one-hot:        AUC {a2:.4f}, qadam {l2.best_iteration_}")
    print(f"  farq: {al - a2:+.4f}")
    print("  ⭐ Ko'p darajali kategoriyada ichki mexanizm ustun")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ma'lumot ===
  15000 qator, tuman darajalari 60, NaN 742

=== 2. LightGBM: kategoriya avtomatik ===
  qadam 48, test AUC 0.8234

=== 3. XGBoost: enable_categorical ===
  qadam 54, test AUC 0.8254

=== 4. One-hot bilan solishtirish ===
  belgilar soni: ichki 4, one-hot 65
  ichki mexanizm: AUC 0.8234, qadam 48
  one-hot:        AUC 0.8239, qadam 87
  farq: -0.0005
  ⭐ Ko'p darajali kategoriyada ichki mexanizm ustun

Nima ko'rsatdi: 2.5-bo'lim.

Misol 4 — Nomutanosib sinflar

python
"""scale_pos_weight va muqobillari (real numpy/lightgbm/xgboost)."""

import warnings

import numpy as np
from lightgbm import LGBMClassifier
from sklearn.metrics import (average_precision_score, brier_score_loss,
                             f1_score, precision_recall_curve, roc_auc_score)
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier


def yarat(seed: int = 9, n: int = 20000, ulush: float = 0.02):
    rng = np.random.default_rng(seed)
    X = rng.normal(0, 1, (n, 12))
    kuch = 1.2 * X[:, 0] - 1.0 * X[:, 1] + 0.9 * (X[:, 2] > 0.5) + 0.7 * X[:, 3]
    chegara = np.quantile(kuch, 1 - ulush)
    p = 1 / (1 + np.exp(-(kuch - chegara) * 1.5))
    y = (rng.random(n) < p).astype(int)
    return X, y


def main() -> None:
    warnings.simplefilter("ignore")
    import lightgbm as lgb
    X, y = yarat()
    Xtr, Xqol, ytr, yqol = train_test_split(X, y, test_size=0.4,
                                            random_state=0, stratify=y)
    Xval, Xte, yval, yte = train_test_split(Xqol, yqol, test_size=0.5,
                                            random_state=0, stratify=yqol)

    print("=== 1. Nomutanosiblik ===")
    print(f"  musbat sinf: {y.mean():.2%} ({int(y.sum())} ta)")
    spw = float((ytr == 0).sum() / (ytr == 1).sum())
    print(f"  scale_pos_weight = {spw:.1f}")

    print("\n=== 2. scale_pos_weight ta'siri (LightGBM) ===")
    print(f"  {'spw':>7} {'ROC AUC':>9} {'PR AUC':>9} {'Brier':>9} "
          f"{'o_rt p':>9}")
    for w in [1.0, spw / 4, spw]:
        m = LGBMClassifier(n_estimators=1500, learning_rate=0.05,
                           num_leaves=31, scale_pos_weight=w,
                           random_state=0, verbose=-1, n_jobs=1)
        m.fit(Xtr, ytr, eval_set=[(Xval, yval)], eval_metric="average_precision",
              callbacks=[lgb.early_stopping(50, verbose=False)])
        p = m.predict_proba(Xte)[:, 1]
        print(f"  {w:>7.1f} {roc_auc_score(yte, p):>9.4f} "
              f"{average_precision_score(yte, p):>9.4f} "
              f"{brier_score_loss(yte, p):>9.5f} {p.mean():>9.4f}")
    print(f"  haqiqiy musbat ulush: {yte.mean():.4f}")

    print("\n=== 3. Chegarani sozlash (spw = 1) ===")
    m = LGBMClassifier(n_estimators=1500, learning_rate=0.05, num_leaves=31,
                       random_state=0, verbose=-1, n_jobs=1)
    m.fit(Xtr, ytr, eval_set=[(Xval, yval)], eval_metric="average_precision",
          callbacks=[lgb.early_stopping(50, verbose=False)])
    pval = m.predict_proba(Xval)[:, 1]
    pte = m.predict_proba(Xte)[:, 1]
    pr, rc, ch = precision_recall_curve(yval, pval)
    f1 = 2 * pr[:-1] * rc[:-1] / np.maximum(pr[:-1] + rc[:-1], 1e-12)
    eng = float(ch[int(np.argmax(f1))])
    print(f"  validatsiyada eng yaxshi chegara: {eng:.4f}")
    print(f"  0.5 chegarada test F1:        "
          f"{f1_score(yte, (pte > 0.5).astype(int)):.4f}")
    print(f"  tanlangan chegarada test F1:  "
          f"{f1_score(yte, (pte > eng).astype(int)):.4f}")

    print("\n=== 4. XGBoost bilan ham tekshirish ===")
    for w, nom in [(1.0, "spw=1 + chegara"), (spw, "spw=balanced")]:
        x = XGBClassifier(n_estimators=1500, learning_rate=0.05, max_depth=4,
                          scale_pos_weight=w, tree_method="hist",
                          early_stopping_rounds=50, eval_metric="aucpr",
                          random_state=0, n_jobs=1)
        x.fit(Xtr, ytr, eval_set=[(Xval, yval)], verbose=False)
        px = x.predict_proba(Xte)[:, 1]
        pv = x.predict_proba(Xval)[:, 1]
        pr2, rc2, ch2 = precision_recall_curve(yval, pv)
        f12 = 2 * pr2[:-1] * rc2[:-1] / np.maximum(pr2[:-1] + rc2[:-1], 1e-12)
        t = float(ch2[int(np.argmax(f12))])
        print(f"  {nom:<18}: PR AUC {average_precision_score(yte, px):.4f}, "
              f"F1 {f1_score(yte, (px > t).astype(int)):.4f}, "
              f"Brier {brier_score_loss(yte, px):.5f}")
    print("  ⭐ Chegarani sozlash kalibrlashni saqlaydi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Nomutanosiblik ===
  musbat sinf: 4.51% (902 ta)
  scale_pos_weight = 21.2

=== 2. scale_pos_weight ta'siri (LightGBM) ===
      spw   ROC AUC    PR AUC     Brier    o_rt p
      1.0    0.9201    0.4586   0.03103    0.0407
      5.3    0.9130    0.4589   0.03436    0.0801
     21.2    0.8931    0.3527   0.03792    0.0797
  haqiqiy musbat ulush: 0.0450

=== 3. Chegarani sozlash (spw = 1) ===
  validatsiyada eng yaxshi chegara: 0.2904
  0.5 chegarada test F1:        0.4030
  tanlangan chegarada test F1:  0.5000

=== 4. XGBoost bilan ham tekshirish ===
  spw=1 + chegara   : PR AUC 0.4908, F1 0.5110, Brier 0.03009
  spw=balanced      : PR AUC 0.4998, F1 0.4963, Brier 0.09405
  ⭐ Chegarani sozlash kalibrlashni saqlaydi

Nima ko'rsatdi: 2.6-bo'lim.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"XGBoost va LightGBM juda farq qiladi" Bir xil matematika
"num_leaves = max_depth" Boshqa mexanizm
"min_child_weight = min_samples_leaf" Gessian vs namuna soni
"eval_set ga test bersa bo'ladi" Leakage
"LightGBM har doim yaxshiroq" Kichik ma'lumotda emas
"One-hot majburiy" Ichki mexanizm ustun
"scale_pos_weight zararsiz" Kalibrlashni buzadi
"GPU har doim tezroq" Kichik ma'lumotda emas

6. Keng tarqalgan xatolar va yechimlari

1. eval_set ga test berish

python
x.fit(Xtr, ytr, eval_set=[(Xte, yte)])                            # ⚠️
x.fit(Xtr, ytr, eval_set=[(Xval, yval)])                          # ✅

2. num_leaves ni cheklamaslik

python
LGBMClassifier(num_leaves=1024, max_depth=-1)                     # ⚠️
LGBMClassifier(num_leaves=31, min_child_samples=20)               # ✅

3. Kategoriyani kodlamaslik

python
df["tuman"] = df["tuman"].astype(int)    # tartib paydo bo'ladi   # ⚠️
df["tuman"] = df["tuman"].astype("category")                      # ✅

4. verbose ni o'chirmaslik

python
LGBMClassifier()                          # loglar oqadi          # ⚠️
LGBMClassifier(verbose=-1)                                        # ✅

5. Erta to'xtatishsiz

python
XGBClassifier(n_estimators=5000)                                  # ⚠️
XGBClassifier(n_estimators=5000, early_stopping_rounds=50)        # ✅

6. scale_pos_weight dan keyin ehtimollik

python
p = m.predict_proba(X)[:, 1]   # spw=50 bilan                     # ⚠️
# spw=1 + chegarani sozlash yoki kalibrlash                       # ✅

7. Kichik ma'lumotda LightGBM standartlari

python
LGBMClassifier()   # 300 qator: min_child_samples=20 -> bo'sh     # ⚠️
LGBMClassifier(min_child_samples=5, num_leaves=7)                 # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 15.8-dars (o'tilgan): Gradient boosting
  • 15.9-dars (o'tilgan): Sozlash
  • 15.11-dars: Belgi muhimligi va SHAP
  • 15.12-dars: Ansambllarni solishtirish
  • 15.14-dars: Amaliyot

8. Eng yaxshi amaliyotlar

  1. Validatsiyani alohida ajrating.

  2. num_leaves ni cheklang.

  3. Kategoriyalarni category qiling.

  4. verbose ni o'chiring.

  5. Erta to'xtatish qo'ying.

  6. Ehtimollik kerak bo'lsa spw dan saqlaning.

  7. Kichik ma'lumotda parametrlarni moslang.

  8. Uch kutubxonani taqqoslang.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # LightGBM qanday o'sadi?
2.  # XGBoost qanday?
3.  # LightGBM ning asosiy parametri?
4.  # XGBoost ning?
5.  # min_child_weight nima?
6.  # NaN ni kim boshqaradi?
7.  # kategoriya uchun LightGBM?
8.  # XGBoost uchun?
9.  # eval_set ga nima beriladi?
10. # scale_pos_weight nimani buzadi?
11. # reg_alpha nima?
12. # kichik ma'lumotda nima muammo?
Javoblar
  1. Barg bo'yicha (leaf-wise)
  2. Daraja bo'yicha
  3. num_leaves
  4. max_depth
  5. Bargdagi gessian yig'indisi
  6. Uchalasi ham avtomatik
  7. category dtype
  8. enable_categorical=True
  9. Validatsiya to'plami
  10. Kalibrlashni
  11. L1 regulyarizatsiya
  12. min_child_samples katta

Vazifa 2: Xatolarni tuzating

python
1.  x.fit(Xtr, ytr, eval_set=[(Xte, yte)])

2.  LGBMClassifier(num_leaves=2048)

3.  df["tuman"] = df["tuman"].astype(int)

4.  XGBClassifier(n_estimators=5000)

5.  LGBMClassifier(scale_pos_weight=80)   # ehtimollik kerak
Javoblar
python
1.  x.fit(Xtr, ytr, eval_set=[(Xval, yval)])

2.  LGBMClassifier(num_leaves=31, min_child_samples=20)

3.  df["tuman"] = df["tuman"].astype("category")

4.  XGBClassifier(n_estimators=5000, early_stopping_rounds=50)

5.  # spw=1 + chegarani sozlang

Vazifa 3: Uch kutubxona

Modellang:

  1. Ma'lumot
  2. Uch model
  3. Hajm
  4. Kelishuv

Vazifa 4: Tuzilma

Modellang:

  1. num_leaves
  2. max_depth
  3. min_child_samples
  4. L1/L2

Vazifa 5: Kategoriyalar

Modellang:

  1. Ma'lumot
  2. LightGBM
  3. XGBoost
  4. One-hot

Vazifa 6: Nomutanosiblik

Modellang:

  1. Nomutanosiblik
  2. scale_pos_weight
  3. Chegara
  4. XGBoost

Vazifa 7: O'ylash

Uch kutubxona bir xil matematikani amalga oshiradi va natijalari juda yaqin. Loyihada qaysi birini tanlash kerak?

Javob

Qisqa javob: aniqlik bo'yicha farq odatda ahamiyatsiz (0.001-0.005 AUC), shuning uchun tanlov muhandislik mezonlari bo'yicha qilinadi: o'rnatish, tezlik, jamoaning tajribasi va ishlab chiqarish muhiti.

1. Amaliy mezonlar

Mezon Tanlov
Qo'shimcha kutubxonasiz sklearn Hist*
Eng tez (katta ma'lumot) LightGBM
Barqarorlik, keng qo'llab-quvvatlash XGBoost
Ko'p kategoriyali belgilar LightGBM yoki CatBoost
Kichik ma'lumot (< 2000) sklearn Hist* yoki XGBoost
GPU bor XGBoost yoki LightGBM

2. Nega farq kichik

  1. Uchalasi ham gistogramma asosida bo'linish qidiradi
  2. Uchalasi ham ikkinchi tartibli yaqinlashishni ishlatadi
  3. Regulyarizatsiya mexanizmlari o'xshash
  4. Sozlangandan keyin ular bir xil funksiyani taqriblaydi

3. Qachon farq sezilarli bo'ladi

  • Juda katta ma'lumot: LightGBM tezligi hal qiluvchi
  • Ko'p darajali kategoriyalar: ichki mexanizm muhim
  • Kichik ma'lumot: LightGBM standartlari mos kelmaydi
  • Juda nomutanosib: max_delta_step (XGBoost) yordam beradi

4. Amaliy tavsiya

  1. Prototip: sklearn HistGradientBoosting* (o'rnatish shart emas)
  2. Ishlab chiqarish: LightGBM yoki XGBoost
  3. Ikkalasini ham sinang — 30 daqiqa ish
  4. Bashoratlarni o'rtachalashtirish (blending) ko'pincha ikkalasidan yaxshiroq

5. Xulosa

  1. Aniqlik farqi kichik
  2. Tanlov muhandislik mezonlari bo'yicha
  3. Kichik ma'lumotda LightGBM standartlariga ehtiyot
  4. Blending bepul yaxshilanish beradi

Nimani mustahkamlaydi: 2.1, 2.2-bo'limlar.

Xulosa

Bu darsda XGBoost va LightGBM ni o'rgandik.

Eng muhim uch fikr:

  1. Bir xil matematika, boshqa o'sish strategiyasi. LightGBM barg bo'yicha (leaf-wise) o'sadi — eng katta foyda beradigan bargdan, shuning uchun asosiy parametri num_leaves. XGBoost daraja bo'yicha (level-wise) o'sadi va max_depth bilan boshqariladi. Sozlangandan keyin natijalar odatda 0.001-0.005 AUC ichida farq qiladi.

  2. NaN va kategoriyalar — to'g'ridan-to'g'ri. Ikkala kutubxona ham yo'qolgan qiymatlarni o'zi boshqaradi va kategoriyali belgilarni ichki mexanizm bilan qayta ishlaydi (astype("category") yoki enable_categorical=True). Ko'p darajali kategoriyalarda bu one-hot dan sezilarli ustun.

  3. eval_set ga test bermang. Bu ikki kutubxonada validatsiya to'plami qo'lda beriladi — shuning uchun leakage qilish osonroq. O'quvni ichki o'quv + validatsiyaga ajrating. scale_pos_weight esa kalibrlashni buzadi: ehtimollik kerak bo'lsa, uni 1 da qoldirib chegarani sozlang.

Keyingi darsda belgi muhimligini o'rganamiz: feature_importances_ nega aldaydi va uning o'rniga nima ishlatish kerak.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
15.10-dars: XGBoost va LightGBM — IlmHamroh