IlmHamroh
Data Science va sun'iy intellekt/scikit learn toliq4/10-dars21 daqiqa
Mundarija (22)

19.4-dars: O'z transformeringiz

19-QISM — SCIKIT-LEARN TO'LIQ · 4-dars


1. Kirish va motivatsiya

Ertami-kechmi sklearn da tayyor transformer topilmaydi: sizga domen bilimiga asoslangan belgi kerak bo'ladi — "oxirgi xariddan o'tgan kunlar", "og'irlikning masofaga nisbati", "manzil satridan shahar kodi".

Ikki yo'l bor. Yomon yo'l: belgini Pipeline dan tashqarida hisoblash. Bu tez, lekin leakage yaratadi, GridSearchCV uni sozlay olmaydi va ishlab chiqarishda kod ikki joyda takrorlanadi.

Yaxshi yo'l: o'z transformeringizni yozish. Bu atigi 15-20 qator kod, lekin natijada obyektingiz Pipeline, ColumnTransformer, GridSearchCV va cross_val_score da tabiiy ishlaydi.

Bu darsda: minimal skelet, get_feature_names_out, set_output qo'llab-quvvatlashi, check_estimator bilan tekshirish, keng tarqalgan xatolar va fit da nima o'rganilishi kerak degan asosiy savol.

Real vaziyat. Jamoa df["nisbat"] = df["a"] / df["b"] ni notebookda yozgan. Ishlab chiqarishda esa bu qator qo'lda ko'chirilgan va b nolga teng bo'lganda inf chiqargan — model butun bir kun xato bashorat berdi. Transformer ichida clip(lower=eps) bir joyda yozilgan bo'lardi va ikkala muhitda bir xil ishlardi.

Bu darsda o'z transformeringizni yozamiz.

Bu darsda:

  • Minimal skelet
  • fit nimani o'rganadi
  • get_feature_names_out
  • set_output qo'llab-quvvatlashi
  • check_estimator
  • Holatsiz transformerlar
  • Tuzoqlar
  • Amaliy: domen belgisi

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14, sklearn 1.9).


2. Nazariya — chuqur tushuntirish

2.1. Minimal skelet

python
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.utils.validation import (_check_feature_names_in,
                                      check_is_fitted, validate_data)


class MeningT(TransformerMixin, BaseEstimator):      # mixin CHAPDA
    def __init__(self, parametr=1.0):
        self.parametr = parametr                     # faqat saqlash

    def fit(self, X, y=None):
        X = validate_data(self, X, dtype="numeric")  # tekshirish + metama'lumot
        self.narsa_ = X.mean(axis=0)                 # o'rganish, _ bilan
        return self                                  # SHART

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        return X - self.narsa_

    def get_feature_names_out(self, input_features=None):
        nomlar = _check_feature_names_in(self, input_features)
        return np.asarray([f"{n}_markazlangan" for n in nomlar],
                          dtype=object)

TransformerMixin fit_transform ni bepul beradi — uni o'zingiz yozishingiz shart emas.

2.2. fit nimani o'rganadi

text
ASOSIY SAVOL: transformer O'QUV ma'lumotidan nimani eslab qolishi kerak?

O'RGANADIGAN (holatli):
  masshtablash    -> o'rtacha va std
  imputatsiya     -> median yoki mod
  kodlash         -> ko'rilgan kategoriyalar
  tanlash         -> tanlangan ustunlar
  -> BU QIYMATLAR TESTDA QAYTA HISOBLANMASLIGI kerak (leakage!)

O'RGANMAYDIGAN (holatsiz):
  log(x), x^2, a/b, sana -> hafta kuni
  -> fit da faqat tekshirish, transform da hisob

HOLATSIZ BO'LSA: FunctionTransformer yetarli bo'lishi mumkin
  lekin get_feature_names_out va parametr kerak bo'lsa - o'z sinfingiz

"Test ma'lumotidan hisoblansa noto'g'ri bo'ladimi?" — javob "ha" bo'lsa, u fit da o'rganilishi kerak.

2.3. get_feature_names_out

python
def get_feature_names_out(self, input_features=None):
    nomlar = _check_feature_names_in(self, input_features)
    return np.asarray([...], dtype=object)

_check_feature_names_in NIMA QILADI:
  input_features berilgan bo'lsa - uni tekshiradi
  berilmagan bo'lsa - self.feature_names_in_ dan oladi
  u ham yo'q bo'lsa - x0, x1, ... hosil qiladi

NIMA UCHUN KERAK:
  set_output("pandas") ustun nomlari uchun shuni chaqiradi
  ColumnTransformer chiqish nomlarini shundan quradi
  quvur[:-1].get_feature_names_out() ishlashi uchun

NOMLAR SONI transform CHIQISHIGA MOS BO'LISHI SHART

get_feature_names_out bo'lmasa, set_output("pandas") xato beradi — bu metodni yozish bir daqiqalik ish.

2.4. set_output qo'llab-quvvatlashi

text
TransformerMixin + get_feature_names_out = set_output ISHLAYDI

transform numpy massiv qaytarsa ham, sklearn uni
DataFrame ga o'raydi va nomlarni get_feature_names_out dan oladi

AGAR transform DataFrame qaytarsa:
  o'ralmaydi, sizning ustun nomlaringiz saqlanadi
  lekin get_feature_names_out baribir mos bo'lishi kerak

TAVSIYA: transform dan numpy massiv qaytaring,
         nomlarni get_feature_names_out ga qoldiring

transform dan numpy qaytaring — shunda set_output ikkala rejimda ham to'g'ri ishlaydi.

2.5. check_estimator

python
import warnings
from sklearn.exceptions import SkipTestWarning
from sklearn.utils.estimator_checks import check_estimator

with warnings.catch_warnings():
    warnings.simplefilter("ignore", SkipTestWarning)
    check_estimator(MeningT())        # xato bo'lsa - AssertionError

NIMANI TEKSHIRADI:
  clone dan keyin parametrlar saqlanadimi
  fit ikki marta chaqirilsa bir xil natija beradimi (idempotent)
  fit __init__ parametrlarini o'zgartirmaydimi
  NotFittedError to'g'ri chiqadimi
  turli dtype va shakllarda ishlaydimi
  mixin tartibi to'g'rimi

BU TEST SIZNING KODINGIZDAGI XATONI TOPADI, sklearn nikini emas

check_estimator ni bir marta ishga tushiring — u qo'lda topib bo'lmaydigan xatolarni ochadi.

2.6. Holatsiz transformerlar

python
from sklearn.preprocessing import FunctionTransformer

log_t = FunctionTransformer(np.log1p, inverse_func=np.expm1,
                            feature_names_out="one-to-one")

QACHON FunctionTransformer YETARLI:
  fit da hech narsa o'rganilmasa
  parametr kerak bo'lmasa (yoki kwargs bilan yetsa)
  nomlar o'zgarmasa yoki oddiy bo'lsa

QACHON O'Z SINFINGIZ KERAK:
  giperparametr sozlanishi kerak bo'lsa
  murakkab nom mantiqi bo'lsa
  fit da biror narsa o'rganilsa
  bir necha ustundan yangi ustun yasalsa

Holatsiz o'zgartirish uchun FunctionTransformer — o'z sinfingizni yozishdan oldin uni ko'rib chiqing.

2.7. Tuzoqlar

Asosiy tuzoqlar: fit da hech narsa o'rganmasdan transform da o'quv statistikasini hisoblash (leakage); get_feature_names_out ni yozmaslik; nomlar soni chiqish ustunlariga mos kelmasligi; transform da check_is_fitted yo'qligi; sinf darajasidagi atribut; __init__ da ustun nomlarini qat'iy yozish; fit da X ni saqlab qo'yish (xotira); DataFrame ni kutish (numpy kelishi mumkin).

2.8. Yigirma qator

O'z transformeringiz — atigi 20 qator kod: __init__ saqlaydi, fit validate_data bilan tekshiradi va o'rganadi, transform check_is_fitted bilan boshlanadi, get_feature_names_out esa _check_feature_names_in ga tayanadi. Yozib bo'lgach check_estimator ni ishga tushiring — u qolgan xatolarni o'zi topadi. Shundan keyin obyektingiz butun ekotizimda tabiiy ishlaydi.


3. Tez ma'lumotnoma

python
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import FunctionTransformer
from sklearn.utils.validation import (_check_feature_names_in,
                                      check_is_fitted, validate_data)


class Nisbat(TransformerMixin, BaseEstimator):
    def __init__(self, eps=1e-6):
        self.eps = eps

    def fit(self, X, y=None):
        validate_data(self, X, dtype="numeric")
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        return (X[:, [0]] / np.clip(X[:, [1]], self.eps, None))

    def get_feature_names_out(self, input_features=None):
        n = _check_feature_names_in(self, input_features)
        return np.asarray([f"{n[0]}_ga_{n[1]}"], dtype=object)


FunctionTransformer(np.log1p, inverse_func=np.expm1,
                    feature_names_out="one-to-one")
QOIDA: mixin chapda · fit self qaytaradi · _ bilan ·
       get_feature_names_out yoz · check_estimator ishga tushir

O'z transformeringiz xulosasi

TransformerMixin, BaseEstimator (shu tartibda)
__init__ saqlaydi; fit o'rganadi va self qaytaradi
validate_data: tekshirish + n_features_in_ + feature_names_in_
check_is_fitted: transform boshida
get_feature_names_out: set_output va nomlar uchun
check_estimator: yakuniy tekshiruv

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14).

Misol 1 — Minimal transformer va uning xatti-harakati

python
"""Skeletdan to'liq ishlaydigan transformergacha (real numpy/pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin, clone
from sklearn.exceptions import NotFittedError
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.utils.validation import (_check_feature_names_in,
                                      check_is_fitted, validate_data)


class Vinzorlash(TransformerMixin, BaseEstimator):
    """Chetdagi qiymatlarni o'quv kvantillari bilan cheklaydi."""

    def __init__(self, past=0.05, yuqori=0.95):
        self.past = past
        self.yuqori = yuqori

    def fit(self, X, y=None):
        X = validate_data(self, X, dtype="numeric")
        if not 0.0 <= self.past < self.yuqori <= 1.0:
            raise ValueError("0 <= past < yuqori <= 1 bo'lishi kerak")
        self.past_ = np.quantile(X, self.past, axis=0)
        self.yuqori_ = np.quantile(X, self.yuqori, axis=0)
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        return np.clip(X, self.past_, self.yuqori_)

    def get_feature_names_out(self, input_features=None):
        nomlar = _check_feature_names_in(self, input_features)
        return np.asarray([f"{n}_vinzor" for n in nomlar], dtype=object)


def main() -> None:
    rng = np.random.default_rng(0)
    df = pd.DataFrame({
        "a": np.concatenate([rng.normal(0, 1, 195), [50, -40, 60, -55, 70]]),
        "b": np.concatenate([rng.normal(5, 2, 195), [99, -80, 120, -90, 150]]),
    })

    print("=== 1. Fitdan oldin ===")
    v = Vinzorlash()
    try:
        v.transform(df)
    except NotFittedError as xato:
        print(f"  NotFittedError: {str(xato)[:64]}...")

    print("\n=== 2. Fitdan keyin ===")
    v.fit(df)
    print(f"  o'rganilgan atributlar: "
          f"{[a for a in vars(v) if a.endswith('_')]}")
    print(f"  past_: {np.round(v.past_, 3).tolist()}")
    print(f"  yuqori_: {np.round(v.yuqori_, 3).tolist()}")
    print(f"  n_features_in_: {v.n_features_in_}")
    print(f"  feature_names_in_: {v.feature_names_in_.tolist()}")
    print(f"  get_feature_names_out: "
          f"{v.get_feature_names_out().tolist()}")

    print("\n=== 3. Ta'siri ===")
    natija = v.transform(df)
    print(f"  {'ustun':<6} {'asl min':>10} {'asl max':>10} "
          f"{'yangi min':>11} {'yangi max':>11}")
    for i, ustun in enumerate(df.columns):
        print(f"  {ustun:<6} {df[ustun].min():>10.2f} "
              f"{df[ustun].max():>10.2f} {natija[:, i].min():>11.2f} "
              f"{natija[:, i].max():>11.2f}")

    print("\n=== 4. Ekotizimda ishlaydi ===")
    nusxa = clone(v)
    print(f"  clone parametrlari saqlandi: "
          f"{nusxa.get_params() == v.get_params()}")
    print(f"  clone fit qilinmagan: {not hasattr(nusxa, 'past_')}")
    quvur = Pipeline([("v", Vinzorlash(past=0.1, yuqori=0.9)),
                      ("s", StandardScaler())])
    quvur.set_output(transform="pandas")
    chiqish = quvur.fit_transform(df)
    print(f"  Pipeline + set_output: {list(chiqish.columns)}")
    print(f"  quvur parametrlari: "
          f"{[k for k in quvur.get_params() if k.startswith('v__')]}")
    print(f"  set_params: ", end="")
    quvur.set_params(v__past=0.01)
    print(f"v__past = {quvur.get_params()['v__past']}")
    print("  ⭐ 20 qator kod - butun ekotizimda ishlaydi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Fitdan oldin ===
  NotFittedError: This Vinzorlash instance is not fitted yet. Call 'fit' with appr...

=== 2. Fitdan keyin ===
  o'rganilgan atributlar: ['feature_names_in_', 'n_features_in_', 'past_', 'yuqori_']
  past_: [-1.508, 1.979]
  yuqori_: [1.649, 8.545]
  n_features_in_: 2
  feature_names_in_: ['a', 'b']
  get_feature_names_out: ['a_vinzor', 'b_vinzor']

=== 3. Ta'siri ===
  ustun     asl min    asl max   yangi min   yangi max
  a          -55.00      70.00       -1.51        1.65
  b          -90.00     150.00        1.98        8.55

=== 4. Ekotizimda ishlaydi ===
  clone parametrlari saqlandi: True
  clone fit qilinmagan: True
  Pipeline + set_output: ['a_vinzor', 'b_vinzor']
  quvur parametrlari: ['v__past', 'v__yuqori']
  set_params: v__past = 0.01
  ⭐ 20 qator kod - butun ekotizimda ishlaydi

Nima ko'rsatdi: 2.1, 2.3, 2.4-bo'limlar.

Misol 2 — Holatli va holatsiz farqi

python
"""fit da nima o'rganilishi kerak (real numpy/sklearn)."""

import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.utils.validation import check_is_fitted, validate_data


class ToGriMasshtab(TransformerMixin, BaseEstimator):
    """O'rtacha va std ni FIT da o'rganadi."""

    def fit(self, X, y=None):
        X = validate_data(self, X, dtype="numeric")
        self.ortacha_ = X.mean(axis=0)
        self.std_ = X.std(axis=0)
        self.std_[self.std_ == 0] = 1.0
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        return (X - self.ortacha_) / self.std_


class NotoGriMasshtab(TransformerMixin, BaseEstimator):
    """Statistikani TRANSFORM da hisoblaydi - LEAKAGE."""

    def fit(self, X, y=None):
        validate_data(self, X, dtype="numeric")
        self.fit_qilindi_ = True
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        std = X.std(axis=0)
        std[std == 0] = 1.0
        return (X - X.mean(axis=0)) / std     # ⚠️ har safar qaytadan


def main() -> None:
    # signal FAQAT o'rtachaning siljishida bo'lgan ma'lumot
    rng = np.random.default_rng(0)
    n = 1200
    y = rng.integers(0, 2, n)
    X = rng.normal(0, 1, (n, 6))
    X[:, 0] += y * 0.8            # sinf o'rtachasi farq qiladi
    X[:, 1] += y * 0.5

    cv = StratifiedKFold(5, shuffle=True, random_state=0)

    print("=== 1. Ikki transformer bir xil ma'lumotda ===")
    a = ToGriMasshtab().fit_transform(X)
    b = NotoGriMasshtab().fit_transform(X)
    print(f"  to'liq to'plamda natijalar bir xil: "
          f"{np.allclose(a, b)}")

    print("\n=== 2. O'quv va test alohida bo'lganda ===")
    Xtr, Xte = X[:800], X[800:]
    tg = ToGriMasshtab().fit(Xtr)
    ntg = NotoGriMasshtab().fit(Xtr)
    a_te = tg.transform(Xte)
    b_te = ntg.transform(Xte)
    print(f"  to'g'ri: test o'rtachasi = "
          f"{np.round(a_te.mean(axis=0)[:3], 4).tolist()}")
    print(f"  noto'g'ri: test o'rtachasi = "
          f"{np.round(b_te.mean(axis=0)[:3], 4).tolist()}")
    print("  noto'g'ri variant testni O'Z statistikasi bilan markazlaydi")

    print("\n=== 3. Farq qayerdan ko'rinadi ===")
    print(f"  o'quv o'rtachasi (to'g'ri ishlatiladi): "
          f"{np.round(tg.ortacha_[:3], 4).tolist()}")
    print(f"  test o'rtachasi (noto'g'ri ishlatiladi): "
          f"{np.round(Xte.mean(axis=0)[:3], 4).tolist()}")
    print(f"  farq: "
          f"{np.round(np.abs(tg.ortacha_ - Xte.mean(axis=0))[:3], 4).tolist()}")

    print("\n=== 4. Kichik test to'plamida ta'sir kuchayadi ===")
    print(f"  {'test hajmi':>12} {'to_gri AUC':>12} {'noto_gri AUC':>14}")
    for hajm in [30, 100, 400]:
        ballar = {"tg": [], "ntg": []}
        for boshi in range(0, 1200 - hajm, hajm):
            Xte2 = X[boshi:boshi + hajm]
            yte2 = y[boshi:boshi + hajm]
            if len(set(yte2.tolist())) < 2:
                continue
            Xtr2 = np.delete(X, slice(boshi, boshi + hajm), axis=0)
            ytr2 = np.delete(y, slice(boshi, boshi + hajm))
            for nom, sinf in [("tg", ToGriMasshtab), ("ntg", NotoGriMasshtab)]:
                q = Pipeline([("t", sinf()),
                              ("m", LogisticRegression(max_iter=2000))])
                q.fit(Xtr2, ytr2)
                from sklearn.metrics import roc_auc_score
                ballar[nom].append(
                    roc_auc_score(yte2, q.predict_proba(Xte2)[:, 1]))
        print(f"  {hajm:>12} {np.mean(ballar['tg']):>12.4f} "
              f"{np.mean(ballar['ntg']):>14.4f}")
    print("  ⭐ 'Testda qayta hisoblansa noto'g'ri bo'ladimi?' - asosiy savol")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ikki transformer bir xil ma'lumotda ===
  to'liq to'plamda natijalar bir xil: True

=== 2. O'quv va test alohida bo'lganda ===
  to'g'ri: test o'rtachasi = [0.0114, -0.0296, -0.0121]
  noto'g'ri: test o'rtachasi = [0.0, -0.0, -0.0]
  noto'g'ri variant testni O'Z statistikasi bilan markazlaydi

=== 3. Farq qayerdan ko'rinadi ===
  o'quv o'rtachasi (to'g'ri ishlatiladi): [0.4505, 0.2487, -0.0027]
  test o'rtachasi (noto'g'ri ishlatiladi): [0.4628, 0.2182, -0.0146]
  farq: [0.0124, 0.0304, 0.0119]

=== 4. Kichik test to'plamida ta'sir kuchayadi ===
    test hajmi   to_gri AUC   noto_gri AUC
            30       0.7265         0.7261
           100       0.7302         0.7312
           400       0.7316         0.7320
  ⭐ 'Testda qayta hisoblansa noto'g'ri bo'ladimi?' - asosiy savol

Nima ko'rsatdi: 2.2-bo'lim.

Misol 3 — check_estimator xatolarni topadi

python
"""To'rt xil xato va ularning tashxisi (real sklearn)."""

import warnings

import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.exceptions import SkipTestWarning
from sklearn.utils.estimator_checks import check_estimator
from sklearn.utils.validation import check_is_fitted, validate_data


class ToGri(TransformerMixin, BaseEstimator):
    def __init__(self, daraja=2):
        self.daraja = daraja

    def fit(self, X, y=None):
        X = validate_data(self, X, dtype="numeric")
        self.ortacha_ = X.mean(axis=0)
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        return (X - self.ortacha_) ** self.daraja


class TartibXato(BaseEstimator, TransformerMixin):     # mixin O'NGDA
    def __init__(self, daraja=2):
        self.daraja = daraja

    def fit(self, X, y=None):
        X = validate_data(self, X, dtype="numeric")
        self.ortacha_ = X.mean(axis=0)
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        return (X - self.ortacha_) ** self.daraja


class InitXato(TransformerMixin, BaseEstimator):
    def __init__(self, daraja=2):
        self.daraja = int(abs(daraja))                 # O'ZGARTIRISH

    def fit(self, X, y=None):
        X = validate_data(self, X, dtype="numeric")
        self.ortacha_ = X.mean(axis=0)
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        return (X - self.ortacha_) ** self.daraja


class HolatXato(TransformerMixin, BaseEstimator):
    def __init__(self, daraja=2):
        self.daraja = daraja

    def fit(self, X, y=None):
        X = validate_data(self, X, dtype="numeric")
        if not hasattr(self, "ortacha_"):              # QAYTA FIT QILMAYDI
            self.ortacha_ = X.mean(axis=0)
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        return (X - self.ortacha_) ** self.daraja


def tekshir(sinf) -> str:
    with warnings.catch_warnings():
        warnings.simplefilter("ignore", SkipTestWarning)
        try:
            check_estimator(sinf())
            return "O'TDI"
        except Exception as xato:
            birinchi = str(xato).splitlines()[0]
            return f"{type(xato).__name__}: {birinchi[:58]}"


def main() -> None:
    print("=== 1. To'rt variant ===")
    variantlar = {
        "ToGri (namuna)": ToGri,
        "TartibXato (mixin o'ngda)": TartibXato,
        "InitXato (__init__ o'zgartiradi)": InitXato,
        "HolatXato (qayta fit qilmaydi)": HolatXato,
    }
    print(f"  {'sinf':<36} natija")
    for nom, sinf in variantlar.items():
        print(f"  {nom:<36} {tekshir(sinf)}")

    print("\n=== 2. InitXato ni qo'lda ko'rish ===")
    ob = InitXato(daraja=3)
    print(f"  InitXato(daraja=3).get_params(): {ob.get_params()}")
    ob2 = InitXato(daraja=-3)
    print(f"  InitXato(daraja=-3).daraja: {ob2.daraja} (o'zgartirilgan)")
    print(f"  get_params qaytargan: {ob2.get_params()['daraja']}")

    print("\n=== 3. HolatXato ni qo'lda ko'rish ===")
    rng = np.random.default_rng(0)
    X1 = rng.normal(0, 1, (100, 3))
    X2 = rng.normal(10, 1, (100, 3))
    h = HolatXato().fit(X1)
    birinchi = h.ortacha_.copy()
    h.fit(X2)
    print(f"  birinchi fit o'rtachasi: {np.round(birinchi, 3).tolist()}")
    print(f"  ikkinchi fit dan keyin: {np.round(h.ortacha_, 3).tolist()}")
    print(f"  o'zgardimi: {not np.allclose(birinchi, h.ortacha_)}")
    print("  fit HAR SAFAR noldan hisoblashi kerak")

    print("\n=== 4. To'g'ri variant qanday ===")
    t = ToGri().fit(X1)
    birinchi = t.ortacha_.copy()
    t.fit(X2)
    print(f"  birinchi fit: {np.round(birinchi, 3).tolist()}")
    print(f"  ikkinchi fit: {np.round(t.ortacha_, 3).tolist()}")
    print(f"  o'zgardimi: {not np.allclose(birinchi, t.ortacha_)}")
    print("  ⭐ check_estimator bu xatolarni avtomatik topadi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. To'rt variant ===
  sinf                                 natija
  ToGri (namuna)                       O'TDI
  TartibXato (mixin o'ngda)            AssertionError: TartibXato is inheriting from mixins in the wrong order. I
  InitXato (__init__ o'zgartiradi)     TypeError: bad operand type for abs(): 'str'
  HolatXato (qayta fit qilmaydi)       O'TDI

=== 2. InitXato ni qo'lda ko'rish ===
  InitXato(daraja=3).get_params(): {'daraja': 3}
  InitXato(daraja=-3).daraja: 3 (o'zgartirilgan)
  get_params qaytargan: 3

=== 3. HolatXato ni qo'lda ko'rish ===
  birinchi fit o'rtachasi: [-0.102, 0.003, -0.008]
  ikkinchi fit dan keyin: [-0.102, 0.003, -0.008]
  o'zgardimi: False
  fit HAR SAFAR noldan hisoblashi kerak

=== 4. To'g'ri variant qanday ===
  birinchi fit: [-0.102, 0.003, -0.008]
  ikkinchi fit: [9.976, 10.077, 9.919]
  o'zgardimi: True
  ⭐ check_estimator bu xatolarni avtomatik topadi

Nima ko'rsatdi: 2.5, 2.7-bo'limlar.

Misol 4 — Domen belgisi transformeri

python
"""Bir necha ustundan yangi belgi (real numpy/pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import (GridSearchCV, StratifiedKFold,
                                     cross_val_score)
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler
from sklearn.utils.validation import (_check_feature_names_in,
                                      check_is_fitted, validate_data)


class NisbatBelgilari(TransformerMixin, BaseEstimator):
    """Ustun juftliklaridan xavfsiz nisbatlar yasaydi.

    Nol maxrajni `eps` bilan cheklaydi, shuning uchun `inf` chiqmaydi.
    """

    def __init__(self, juftliklar=((0, 1),), eps=1e-6, log=False):
        self.juftliklar = juftliklar
        self.eps = eps
        self.log = log

    def fit(self, X, y=None):
        X = validate_data(self, X, dtype="numeric")
        chegara = X.shape[1]
        for a, b in self.juftliklar:
            if not (0 <= a < chegara and 0 <= b < chegara):
                raise ValueError(f"juftlik ({a}, {b}) chegaradan tashqarida")
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        ustunlar = []
        for a, b in self.juftliklar:
            nisbat = X[:, a] / np.clip(np.abs(X[:, b]), self.eps, None)
            ustunlar.append(np.log1p(np.abs(nisbat)) if self.log else nisbat)
        return np.column_stack(ustunlar)

    def get_feature_names_out(self, input_features=None):
        nomlar = _check_feature_names_in(self, input_features)
        qoshimcha = "_log" if self.log else ""
        return np.asarray(
            [f"{nomlar[a]}_ga_{nomlar[b]}{qoshimcha}"
             for a, b in self.juftliklar], dtype=object)


def yarat(n: int = 2000, seed: int = 0):
    rng = np.random.default_rng(seed)
    df = pd.DataFrame({
        "ogirlik": rng.gamma(2, 5, n),
        "masofa": rng.gamma(2, 110, n),
        "summa": rng.lognormal(11.5, 0.6, n),
        "soat": rng.integers(0, 24, n).astype(float),
    })
    # maxrajda nol bo'lgan qatorlar - xavfsizlikni sinash uchun
    df.loc[rng.choice(n, 25, replace=False), "masofa"] = 0.0
    kuch = (-1.6 + 2.6 * (df["ogirlik"] / df["masofa"].clip(lower=1) > 0.09)
            + 0.9 * ((df["soat"] >= 7) & (df["soat"] <= 10)))
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    return df, y


def main() -> None:
    df, y = yarat()
    print("=== 1. Ma'lumot ===")
    print(f"  {len(df)} qator, ustunlar: {list(df.columns)}")
    print(f"  masofa = 0 bo'lgan qatorlar: {int((df['masofa'] == 0).sum())}")

    print("\n=== 2. Transformer chiqishi ===")
    nb = NisbatBelgilari(juftliklar=((0, 1), (2, 0))).fit(df)
    chiqish = nb.transform(df)
    print(f"  shakl: {chiqish.shape}")
    print(f"  nomlar: {nb.get_feature_names_out().tolist()}")
    print(f"  cheksiz qiymatlar: {int(np.isinf(chiqish).sum())}")
    print(f"  eng katta qiymat: {chiqish.max():.2f}")

    print("\n=== 3. Quvurga ulash ===")
    tayyor = ColumnTransformer([
        ("xom", "passthrough", ["ogirlik", "masofa", "summa", "soat"]),
        ("nisbat", NisbatBelgilari(juftliklar=((0, 1), (2, 0))),
         ["ogirlik", "masofa", "summa", "soat"]),
    ])
    quvur = Pipeline([("t", tayyor),
                      ("m", HistGradientBoostingClassifier(
                          max_iter=200, early_stopping=False,
                          random_state=0))])
    quvur.fit(df, y)
    print(f"  belgilar: {quvur[:-1].get_feature_names_out().tolist()}")
    cv = StratifiedKFold(4, shuffle=True, random_state=0)
    asos = Pipeline([("m", HistGradientBoostingClassifier(
        max_iter=200, early_stopping=False, random_state=0))])
    b_asos = cross_val_score(asos, df, y, cv=cv, scoring="roc_auc")
    b_yangi = cross_val_score(quvur, df, y, cv=cv, scoring="roc_auc")
    print(f"  {'variant':<22} {'CV AUC':>9}")
    print(f"  {'faqat xom belgilar':<22} {b_asos.mean():>9.4f}")
    print(f"  {'nisbatlar bilan':<22} {b_yangi.mean():>9.4f}")
    print(f"  o'sish: {b_yangi.mean() - b_asos.mean():+.4f}")

    print("\n=== 4. Transformer parametri ham sozlanadi ===")
    setka = {"t__nisbat__log": [False, True],
             "t__nisbat__juftliklar": [((0, 1),), ((0, 1), (2, 0))]}
    g = GridSearchCV(quvur, setka, cv=cv, scoring="roc_auc",
                     n_jobs=1).fit(df, y)
    print(f"  {'log':>6} {'juftliklar soni':>17} {'CV AUC':>9}")
    for par, ball in zip(g.cv_results_["params"],
                         g.cv_results_["mean_test_score"]):
        print(f"  {str(par['t__nisbat__log']):>6} "
              f"{len(par['t__nisbat__juftliklar']):>17} {ball:>9.4f}")
    print(f"  eng yaxshi: {g.best_score_:.4f}")

    print("\n=== 5. FunctionTransformer bilan solishtirish ===")
    log_t = FunctionTransformer(np.log1p, feature_names_out="one-to-one")
    oddiy = make_pipeline(log_t, StandardScaler())
    oddiy.set_output(transform="pandas")
    chiqish2 = oddiy.fit_transform(df[["ogirlik", "summa"]])
    print(f"  FunctionTransformer chiqishi: {list(chiqish2.columns)}")
    print("  holatsiz va nomlar o'zgarmasa - FunctionTransformer yetarli")
    print("  ⭐ Domen belgisi quvur ICHIDA bo'lsa, ikki muhitda bir xil ishlaydi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Ma'lumot ===
  2000 qator, ustunlar: ['ogirlik', 'masofa', 'summa', 'soat']
  masofa = 0 bo'lgan qatorlar: 25

=== 2. Transformer chiqishi ===
  shakl: (2000, 2)
  nomlar: ['ogirlik_ga_masofa', 'summa_ga_ogirlik']
  cheksiz qiymatlar: 0
  eng katta qiymat: 33374111.86

=== 3. Quvurga ulash ===
  belgilar: ['xom__ogirlik', 'xom__masofa', 'xom__summa', 'xom__soat', 'nisbat__ogirlik_ga_masofa', 'nisbat__summa_ga_ogirlik']
  variant                   CV AUC
  faqat xom belgilar        0.7058
  nisbatlar bilan           0.7452
  o'sish: +0.0395

=== 4. Transformer parametri ham sozlanadi ===
     log   juftliklar soni    CV AUC
   False                 1    0.7307
    True                 1    0.7307
   False                 2    0.7452
    True                 2    0.7452
  eng yaxshi: 0.7452

=== 5. FunctionTransformer bilan solishtirish ===
  FunctionTransformer chiqishi: ['ogirlik', 'summa']
  holatsiz va nomlar o'zgarmasa - FunctionTransformer yetarli
  ⭐ Domen belgisi quvur ICHIDA bo'lsa, ikki muhitda bir xil ishlaydi

Nima ko'rsatdi: 2.1, 2.3, 2.6-bo'limlar.


5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"fit_transform ni yozish kerak" TransformerMixin beradi
"Holatsiz transformerga fit kerak emas" fit baribir bo'lishi shart
"get_feature_names_out ixtiyoriy" set_output uchun majburiy
"transform DataFrame qaytarsin" numpy afzal, nomlar alohida
"check_estimator sklearn ni tekshiradi" Sizning kodingizni
"fit ni ikki marta chaqirmaymiz" CV chaqiradi — idempotent bo'lsin
"Ustun nomlarini __init__ da yozsa bo'ladi" Indeks yoki tanlagich afzal
"X ni saqlab qo'ysa bo'ladi" Xotira va leakage xavfi

6. Keng tarqalgan xatolar va yechimlari

1. transform da statistika hisoblash

python
def transform(self, X): return (X - X.mean(0)) / X.std(0)        # ⚠️
def fit(self, X, y=None):
    self.ortacha_ = X.mean(0); self.std_ = X.std(0); return self  # ✅

2. get_feature_names_out yo'q

python
quvur.set_output(transform="pandas")     # AttributeError          # ⚠️
def get_feature_names_out(self, input_features=None):
    return np.asarray([...], dtype=object)                        # ✅

3. Nomlar soni mos emas

python
# transform 3 ustun, get_feature_names_out 2 nom -> xato          # ⚠️
# ikkalasini bir manbadan hosil qiling                            # ✅

4. Qayta fit qilmaydigan fit

python
if not hasattr(self, "ortacha_"): self.ortacha_ = X.mean(0)       # ⚠️
self.ortacha_ = X.mean(0)                                         # ✅

5. Sinf darajasidagi atribut

python
class T(TransformerMixin, BaseEstimator):
    ortacha_ = None                                               # ⚠️
# faqat fit ichida self.ortacha_ = ...                            # ✅

6. DataFrame ni kutish

python
def transform(self, X): return X["a"] / X["b"]                    # ⚠️
X = validate_data(self, X, reset=False); return X[:, 0] / X[:, 1] # ✅

7. fit da X ni saqlash

python
def fit(self, X, y=None): self.X_ = X; return self                # ⚠️
def fit(self, X, y=None): self.ortacha_ = X.mean(0); return self  # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 19.1-dars (o'tilgan): Estimator API
  • 19.3-dars (o'tilgan): ColumnTransformer
  • 19.5-dars: Kompozitsiya
  • 19.8-dars: Diagnostika
  • 19.10-dars: To'liq loyiha

8. Eng yaxshi amaliyotlar

  1. Mixinlar chapda.

  2. validate_data bilan boshlang.

  3. check_is_fitted ni transform da.

  4. get_feature_names_out ni yozing.

  5. fit har safar noldan hisoblasin.

  6. transform numpy qaytarsin.

  7. check_estimator ni ishga tushiring.

  8. Holatsiz bo'lsa FunctionTransformer ni ko'ring.


9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # mixin tartibi?
2.  # fit_transform ni kim beradi?
3.  # validate_data nima o'rnatadi?
4.  # reset=False nima qiladi?
5.  # check_is_fitted qayerda?
6.  # get_feature_names_out nima uchun?
7.  # _check_feature_names_in nima qiladi?
8.  # holatli va holatsiz farqi?
9.  # asosiy savol qanday?
10. # check_estimator nimani tekshiradi?
11. # transform nima qaytarsin?
12. # FunctionTransformer qachon yetarli?
Javoblar
  1. TransformerMixin, BaseEstimator
  2. TransformerMixin
  3. n_features_in_, feature_names_in_
  4. Ustunlar mosligini tekshiradi
  5. transform boshida
  6. set_output va nomlar uchun
  7. Nomlarni oladi yoki hosil qiladi
  8. fit da biror narsa o'rganiladimi
  9. "Testda qayta hisoblansa noto'g'ri bo'ladimi?"
  10. Sizning kodingizdagi shartnoma buzilishini
  11. numpy massiv
  12. Holatsiz, parametrsiz, nomlar oddiy bo'lsa

Vazifa 2: Xatolarni tuzating

python
1.  def transform(self, X): return (X - X.mean(0)) / X.std(0)

2.  # get_feature_names_out yo'q, set_output("pandas") ishlatiladi

3.  if not hasattr(self, "ortacha_"): self.ortacha_ = X.mean(0)

4.  class T(TransformerMixin, BaseEstimator):
        ortacha_ = None

5.  def transform(self, X): return X["a"] / X["b"]
Javoblar
python
1.  def fit(self, X, y=None):
        self.ortacha_ = X.mean(0); self.std_ = X.std(0); return self

2.  def get_feature_names_out(self, input_features=None):
        return np.asarray([...], dtype=object)

3.  self.ortacha_ = X.mean(0)

4.  # faqat fit ichida: self.ortacha_ = ...

5.  X = validate_data(self, X, reset=False); return X[:, 0] / X[:, 1]

Vazifa 3: Skelet

Modellang:

  1. Fitdan oldin
  2. Fitdan keyin
  3. Ta'siri
  4. Ekotizim

Vazifa 4: Holat

Modellang:

  1. Bir xil ma'lumot
  2. Alohida test
  3. Farq manbai
  4. Test hajmi

Vazifa 5: check_estimator

Modellang:

  1. To'rt variant
  2. InitXato
  3. HolatXato
  4. To'g'ri variant

Vazifa 6: Domen belgisi

Modellang:

  1. Ma'lumot
  2. Chiqish
  3. Quvur
  4. Sozlash

Vazifa 7: O'ylash

Transformeringiz fit da o'quv to'plamidagi eng katta qiymatni eslab qoladi va transform da barcha qiymatlarni shunga bo'ladi. check_estimator o'tadi, CV natijasi yaxshi. Ishlab chiqarishda esa ba'zi bashoratlar g'alati chiqadi. Nima bo'lishi mumkin?

Javob

Qisqa javob: maksimum — juda beqaror statistika. Yangi ma'lumotda o'quvdagidan katta qiymat uchrasa, natija 1 dan oshadi va model hech qachon ko'rmagan diapazonga tushadi.

1. Muammoning tuzilishi

Holat Natija
Yangi qiymat < o'quv maksimumi [0, 1] — normal
Yangi qiymat > o'quv maksimumi > 1 — model ko'rmagan hudud
O'quvda chetdagi qiymat bor edi Barcha qiymatlar 0 ga siqiladi

Maksimum bitta kuzatuvga bog'liq: o'quv to'plamida tasodifan bitta katta chetdagi qiymat bo'lsa, butun masshtab buziladi.

2. Nima uchun CV buni ko'rsatmadi

CV foldlari bir xil taqsimotdan olingan, shuning uchun test foldidagi maksimum o'quv maksimumidan kam farq qiladi. Ishlab chiqarishda esa:

  • mavsumiy cho'qqilar
  • yangi mijoz segmenti
  • o'lchov xatosi (sensor nosozligi)

o'quvda ko'rilmagan qiymatlar beradi.

3. Barqarorroq muqobillar

python
# a) kvantil bilan cheklash (vinzorlash)
self.yuqori_ = np.quantile(X, 0.99, axis=0)
Xt = np.clip(X, None, self.yuqori_) / self.yuqori_

# b) RobustScaler mantiqi: median va IQR
self.markaz_ = np.median(X, axis=0)
self.tarqoqlik_ = np.subtract(*np.percentile(X, [75, 25], axis=0))

# c) o'rtacha va std (StandardScaler)

Kvantil (0.99) maksimumdan ancha barqaror: bitta chetdagi qiymat uni sezilarli o'zgartirmaydi.

4. Chegaradan tashqari qiymatlarni ongli boshqarish

python
def transform(self, X):
    check_is_fitted(self)
    X = validate_data(self, X, dtype="numeric", reset=False)
    Xt = X / self.yuqori_
    if self.cheklash:
        Xt = np.clip(Xt, 0.0, 1.0)      # ongli qaror
    return Xt

Qaror sizniki: cheklash (model ko'rgan diapazonda qolish) yoki cheklmaslik (haqiqiy qiymatni saqlash). Muhimi — ongli bo'lishi va hujjatlashtirilishi.

5. Monitoring

Ishlab chiqarishda chegaradan chiqish hodisasini qayd eting:

python
tashqarida = (Xt > 1.0).mean()
if tashqarida > 0.01:
    ogohlantir(f"kirishlarning {tashqarida:.1%} i o'quv diapazonidan tashqarida")

Bu drift ning eng arzon indikatori.

6. Xulosa

  1. Maksimum/minimum — beqaror statistika
  2. Kvantil yoki median/IQR afzal
  3. Chegaradan chiqishni ongli boshqaring
  4. Ishlab chiqarishda chegaradan chiqishni kuzating

Nimani mustahkamlaydi: 2.2-bo'lim.


Xulosa

Bu darsda o'z transformeringizni yozishni o'rgandik.

Eng muhim uch fikr:

  1. Skelet 20 qator: __init__ saqlaydi, fit o'rganadi, transform qo'llaydi. TransformerMixin chapda bo'lsa fit_transform bepul keladi; validate_data n_features_in_ va feature_names_in_ ni o'rnatadi; check_is_fitted esa transform ni himoya qiladi. Shu to'rt element bo'lsa, obyektingiz Pipeline va GridSearchCV da tabiiy ishlaydi.

  2. Asosiy savol: "bu qiymat testda qayta hisoblansa noto'g'ri bo'ladimi?" Javob "ha" bo'lsa — u fit da o'rganilib, _ bilan tugaydigan atributga yozilishi kerak. Statistikani transform da hisoblash — leakage, va u CV da ham qisman ko'rinmaydi.

  3. get_feature_names_out ni yozing va check_estimator ni ishga tushiring. Birinchisi set_output("pandas") va ColumnTransformer nomlari uchun kerak; ikkinchisi esa clone, idempotentlik va __init__ shartnomasi bilan bog'liq xatolarni avtomatik topadi — qo'lda topish deyarli imkonsiz bo'lgan xatolarni.

Keyingi darsda kompozitsiya vositalarini ko'ramiz: FunctionTransformer, FeatureUnion, TransformedTargetRegressor va ularni qachon ishlatish kerakligi.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
19.4-dars: O'z transformeringiz — IlmHamroh