Mundarija (22)
- 1. Kirish va motivatsiya
- 2. Nazariya — chuqur tushuntirish
- 2.1. Minimal skelet
- 2.2. fit nimani o'rganadi
- 2.3. get_feature_names_out
- 2.4. set_output qo'llab-quvvatlashi
- 2.5. check_estimator
- 2.6. Holatsiz transformerlar
- 2.7. Tuzoqlar
- 2.8. Yigirma qator
- 3. Tez ma'lumotnoma
- 4. Batafsil misollar
- Misol 1 — Minimal transformer va uning xatti-harakati
- Misol 2 — Holatli va holatsiz farqi
- Misol 3 — check_estimator xatolarni topadi
- Misol 4 — Domen belgisi transformeri
- 5. To'g'ri va noto'g'ri tushunishlar
- 6. Keng tarqalgan xatolar va yechimlari
- 7. Integratsiya — bu bilim qayerda kerak bo'ladi
- 8. Eng yaxshi amaliyotlar
- 9. Amaliy topshiriq
- Xulosa
19.4-dars: O'z transformeringiz
19-QISM — SCIKIT-LEARN TO'LIQ · 4-dars
1. Kirish va motivatsiya
Ertami-kechmi sklearn da tayyor transformer topilmaydi: sizga domen bilimiga asoslangan belgi kerak bo'ladi — "oxirgi xariddan o'tgan kunlar", "og'irlikning masofaga nisbati", "manzil satridan shahar kodi".
Ikki yo'l bor. Yomon yo'l: belgini Pipeline dan tashqarida hisoblash. Bu tez, lekin leakage yaratadi, GridSearchCV uni sozlay olmaydi va ishlab chiqarishda kod ikki joyda takrorlanadi.
Yaxshi yo'l: o'z transformeringizni yozish. Bu atigi 15-20 qator kod, lekin natijada obyektingiz Pipeline, ColumnTransformer, GridSearchCV va cross_val_score da tabiiy ishlaydi.
Bu darsda: minimal skelet, get_feature_names_out, set_output qo'llab-quvvatlashi, check_estimator bilan tekshirish, keng tarqalgan xatolar va fit da nima o'rganilishi kerak degan asosiy savol.
Real vaziyat. Jamoa df["nisbat"] = df["a"] / df["b"] ni notebookda yozgan. Ishlab chiqarishda esa bu qator qo'lda ko'chirilgan va b nolga teng bo'lganda inf chiqargan — model butun bir kun xato bashorat berdi. Transformer ichida clip(lower=eps) bir joyda yozilgan bo'lardi va ikkala muhitda bir xil ishlardi.
Bu darsda o'z transformeringizni yozamiz.
Bu darsda:
- Minimal skelet
- fit nimani o'rganadi
- get_feature_names_out
- set_output qo'llab-quvvatlashi
- check_estimator
- Holatsiz transformerlar
- Tuzoqlar
- Amaliy: domen belgisi
ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14, sklearn 1.9).
2. Nazariya — chuqur tushuntirish
2.1. Minimal skelet
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.utils.validation import (_check_feature_names_in,
check_is_fitted, validate_data)
class MeningT(TransformerMixin, BaseEstimator): # mixin CHAPDA
def __init__(self, parametr=1.0):
self.parametr = parametr # faqat saqlash
def fit(self, X, y=None):
X = validate_data(self, X, dtype="numeric") # tekshirish + metama'lumot
self.narsa_ = X.mean(axis=0) # o'rganish, _ bilan
return self # SHART
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
return X - self.narsa_
def get_feature_names_out(self, input_features=None):
nomlar = _check_feature_names_in(self, input_features)
return np.asarray([f"{n}_markazlangan" for n in nomlar],
dtype=object) TransformerMixin fit_transform ni bepul beradi — uni o'zingiz yozishingiz shart emas.
2.2. fit nimani o'rganadi
ASOSIY SAVOL: transformer O'QUV ma'lumotidan nimani eslab qolishi kerak?
O'RGANADIGAN (holatli):
masshtablash -> o'rtacha va std
imputatsiya -> median yoki mod
kodlash -> ko'rilgan kategoriyalar
tanlash -> tanlangan ustunlar
-> BU QIYMATLAR TESTDA QAYTA HISOBLANMASLIGI kerak (leakage!)
O'RGANMAYDIGAN (holatsiz):
log(x), x^2, a/b, sana -> hafta kuni
-> fit da faqat tekshirish, transform da hisob
HOLATSIZ BO'LSA: FunctionTransformer yetarli bo'lishi mumkin
lekin get_feature_names_out va parametr kerak bo'lsa - o'z sinfingiz "Test ma'lumotidan hisoblansa noto'g'ri bo'ladimi?" — javob "ha" bo'lsa, u fit da o'rganilishi kerak.
2.3. get_feature_names_out
def get_feature_names_out(self, input_features=None):
nomlar = _check_feature_names_in(self, input_features)
return np.asarray([...], dtype=object)
_check_feature_names_in NIMA QILADI:
input_features berilgan bo'lsa - uni tekshiradi
berilmagan bo'lsa - self.feature_names_in_ dan oladi
u ham yo'q bo'lsa - x0, x1, ... hosil qiladi
NIMA UCHUN KERAK:
set_output("pandas") ustun nomlari uchun shuni chaqiradi
ColumnTransformer chiqish nomlarini shundan quradi
quvur[:-1].get_feature_names_out() ishlashi uchun
NOMLAR SONI transform CHIQISHIGA MOS BO'LISHI SHART get_feature_names_out bo'lmasa, set_output("pandas") xato beradi — bu metodni yozish bir daqiqalik ish.
2.4. set_output qo'llab-quvvatlashi
TransformerMixin + get_feature_names_out = set_output ISHLAYDI
transform numpy massiv qaytarsa ham, sklearn uni
DataFrame ga o'raydi va nomlarni get_feature_names_out dan oladi
AGAR transform DataFrame qaytarsa:
o'ralmaydi, sizning ustun nomlaringiz saqlanadi
lekin get_feature_names_out baribir mos bo'lishi kerak
TAVSIYA: transform dan numpy massiv qaytaring,
nomlarni get_feature_names_out ga qoldiring transform dan numpy qaytaring — shunda set_output ikkala rejimda ham to'g'ri ishlaydi.
2.5. check_estimator
import warnings
from sklearn.exceptions import SkipTestWarning
from sklearn.utils.estimator_checks import check_estimator
with warnings.catch_warnings():
warnings.simplefilter("ignore", SkipTestWarning)
check_estimator(MeningT()) # xato bo'lsa - AssertionError
NIMANI TEKSHIRADI:
clone dan keyin parametrlar saqlanadimi
fit ikki marta chaqirilsa bir xil natija beradimi (idempotent)
fit __init__ parametrlarini o'zgartirmaydimi
NotFittedError to'g'ri chiqadimi
turli dtype va shakllarda ishlaydimi
mixin tartibi to'g'rimi
BU TEST SIZNING KODINGIZDAGI XATONI TOPADI, sklearn nikini emas check_estimator ni bir marta ishga tushiring — u qo'lda topib bo'lmaydigan xatolarni ochadi.
2.6. Holatsiz transformerlar
from sklearn.preprocessing import FunctionTransformer
log_t = FunctionTransformer(np.log1p, inverse_func=np.expm1,
feature_names_out="one-to-one")
QACHON FunctionTransformer YETARLI:
fit da hech narsa o'rganilmasa
parametr kerak bo'lmasa (yoki kwargs bilan yetsa)
nomlar o'zgarmasa yoki oddiy bo'lsa
QACHON O'Z SINFINGIZ KERAK:
giperparametr sozlanishi kerak bo'lsa
murakkab nom mantiqi bo'lsa
fit da biror narsa o'rganilsa
bir necha ustundan yangi ustun yasalsa Holatsiz o'zgartirish uchun FunctionTransformer — o'z sinfingizni yozishdan oldin uni ko'rib chiqing.
2.7. Tuzoqlar
Asosiy tuzoqlar: fit da hech narsa o'rganmasdan transform da o'quv statistikasini hisoblash (leakage); get_feature_names_out ni yozmaslik; nomlar soni chiqish ustunlariga mos kelmasligi; transform da check_is_fitted yo'qligi; sinf darajasidagi atribut; __init__ da ustun nomlarini qat'iy yozish; fit da X ni saqlab qo'yish (xotira); DataFrame ni kutish (numpy kelishi mumkin).
2.8. Yigirma qator
O'z transformeringiz — atigi 20 qator kod: __init__ saqlaydi, fit validate_data bilan tekshiradi va o'rganadi, transform check_is_fitted bilan boshlanadi, get_feature_names_out esa _check_feature_names_in ga tayanadi. Yozib bo'lgach check_estimator ni ishga tushiring — u qolgan xatolarni o'zi topadi. Shundan keyin obyektingiz butun ekotizimda tabiiy ishlaydi.
3. Tez ma'lumotnoma
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import FunctionTransformer
from sklearn.utils.validation import (_check_feature_names_in,
check_is_fitted, validate_data)
class Nisbat(TransformerMixin, BaseEstimator):
def __init__(self, eps=1e-6):
self.eps = eps
def fit(self, X, y=None):
validate_data(self, X, dtype="numeric")
return self
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
return (X[:, [0]] / np.clip(X[:, [1]], self.eps, None))
def get_feature_names_out(self, input_features=None):
n = _check_feature_names_in(self, input_features)
return np.asarray([f"{n[0]}_ga_{n[1]}"], dtype=object)
FunctionTransformer(np.log1p, inverse_func=np.expm1,
feature_names_out="one-to-one")
QOIDA: mixin chapda · fit self qaytaradi · _ bilan ·
get_feature_names_out yoz · check_estimator ishga tushirO'z transformeringiz xulosasi
TransformerMixin, BaseEstimator (shu tartibda)
__init__ saqlaydi; fit o'rganadi va self qaytaradi
validate_data: tekshirish + n_features_in_ + feature_names_in_
check_is_fitted: transform boshida
get_feature_names_out: set_output va nomlar uchun
check_estimator: yakuniy tekshiruv4. Batafsil misollar
Misollar real numpy/pandas/sklearn bilan (Python 3.14).
Misol 1 — Minimal transformer va uning xatti-harakati
"""Skeletdan to'liq ishlaydigan transformergacha (real numpy/pandas/sklearn)."""
import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin, clone
from sklearn.exceptions import NotFittedError
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.utils.validation import (_check_feature_names_in,
check_is_fitted, validate_data)
class Vinzorlash(TransformerMixin, BaseEstimator):
"""Chetdagi qiymatlarni o'quv kvantillari bilan cheklaydi."""
def __init__(self, past=0.05, yuqori=0.95):
self.past = past
self.yuqori = yuqori
def fit(self, X, y=None):
X = validate_data(self, X, dtype="numeric")
if not 0.0 <= self.past < self.yuqori <= 1.0:
raise ValueError("0 <= past < yuqori <= 1 bo'lishi kerak")
self.past_ = np.quantile(X, self.past, axis=0)
self.yuqori_ = np.quantile(X, self.yuqori, axis=0)
return self
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
return np.clip(X, self.past_, self.yuqori_)
def get_feature_names_out(self, input_features=None):
nomlar = _check_feature_names_in(self, input_features)
return np.asarray([f"{n}_vinzor" for n in nomlar], dtype=object)
def main() -> None:
rng = np.random.default_rng(0)
df = pd.DataFrame({
"a": np.concatenate([rng.normal(0, 1, 195), [50, -40, 60, -55, 70]]),
"b": np.concatenate([rng.normal(5, 2, 195), [99, -80, 120, -90, 150]]),
})
print("=== 1. Fitdan oldin ===")
v = Vinzorlash()
try:
v.transform(df)
except NotFittedError as xato:
print(f" NotFittedError: {str(xato)[:64]}...")
print("\n=== 2. Fitdan keyin ===")
v.fit(df)
print(f" o'rganilgan atributlar: "
f"{[a for a in vars(v) if a.endswith('_')]}")
print(f" past_: {np.round(v.past_, 3).tolist()}")
print(f" yuqori_: {np.round(v.yuqori_, 3).tolist()}")
print(f" n_features_in_: {v.n_features_in_}")
print(f" feature_names_in_: {v.feature_names_in_.tolist()}")
print(f" get_feature_names_out: "
f"{v.get_feature_names_out().tolist()}")
print("\n=== 3. Ta'siri ===")
natija = v.transform(df)
print(f" {'ustun':<6} {'asl min':>10} {'asl max':>10} "
f"{'yangi min':>11} {'yangi max':>11}")
for i, ustun in enumerate(df.columns):
print(f" {ustun:<6} {df[ustun].min():>10.2f} "
f"{df[ustun].max():>10.2f} {natija[:, i].min():>11.2f} "
f"{natija[:, i].max():>11.2f}")
print("\n=== 4. Ekotizimda ishlaydi ===")
nusxa = clone(v)
print(f" clone parametrlari saqlandi: "
f"{nusxa.get_params() == v.get_params()}")
print(f" clone fit qilinmagan: {not hasattr(nusxa, 'past_')}")
quvur = Pipeline([("v", Vinzorlash(past=0.1, yuqori=0.9)),
("s", StandardScaler())])
quvur.set_output(transform="pandas")
chiqish = quvur.fit_transform(df)
print(f" Pipeline + set_output: {list(chiqish.columns)}")
print(f" quvur parametrlari: "
f"{[k for k in quvur.get_params() if k.startswith('v__')]}")
print(f" set_params: ", end="")
quvur.set_params(v__past=0.01)
print(f"v__past = {quvur.get_params()['v__past']}")
print(" ⭐ 20 qator kod - butun ekotizimda ishlaydi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Fitdan oldin ===
NotFittedError: This Vinzorlash instance is not fitted yet. Call 'fit' with appr...
=== 2. Fitdan keyin ===
o'rganilgan atributlar: ['feature_names_in_', 'n_features_in_', 'past_', 'yuqori_']
past_: [-1.508, 1.979]
yuqori_: [1.649, 8.545]
n_features_in_: 2
feature_names_in_: ['a', 'b']
get_feature_names_out: ['a_vinzor', 'b_vinzor']
=== 3. Ta'siri ===
ustun asl min asl max yangi min yangi max
a -55.00 70.00 -1.51 1.65
b -90.00 150.00 1.98 8.55
=== 4. Ekotizimda ishlaydi ===
clone parametrlari saqlandi: True
clone fit qilinmagan: True
Pipeline + set_output: ['a_vinzor', 'b_vinzor']
quvur parametrlari: ['v__past', 'v__yuqori']
set_params: v__past = 0.01
⭐ 20 qator kod - butun ekotizimda ishlaydiNima ko'rsatdi: 2.1, 2.3, 2.4-bo'limlar.
Misol 2 — Holatli va holatsiz farqi
"""fit da nima o'rganilishi kerak (real numpy/sklearn)."""
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.utils.validation import check_is_fitted, validate_data
class ToGriMasshtab(TransformerMixin, BaseEstimator):
"""O'rtacha va std ni FIT da o'rganadi."""
def fit(self, X, y=None):
X = validate_data(self, X, dtype="numeric")
self.ortacha_ = X.mean(axis=0)
self.std_ = X.std(axis=0)
self.std_[self.std_ == 0] = 1.0
return self
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
return (X - self.ortacha_) / self.std_
class NotoGriMasshtab(TransformerMixin, BaseEstimator):
"""Statistikani TRANSFORM da hisoblaydi - LEAKAGE."""
def fit(self, X, y=None):
validate_data(self, X, dtype="numeric")
self.fit_qilindi_ = True
return self
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
std = X.std(axis=0)
std[std == 0] = 1.0
return (X - X.mean(axis=0)) / std # ⚠️ har safar qaytadan
def main() -> None:
# signal FAQAT o'rtachaning siljishida bo'lgan ma'lumot
rng = np.random.default_rng(0)
n = 1200
y = rng.integers(0, 2, n)
X = rng.normal(0, 1, (n, 6))
X[:, 0] += y * 0.8 # sinf o'rtachasi farq qiladi
X[:, 1] += y * 0.5
cv = StratifiedKFold(5, shuffle=True, random_state=0)
print("=== 1. Ikki transformer bir xil ma'lumotda ===")
a = ToGriMasshtab().fit_transform(X)
b = NotoGriMasshtab().fit_transform(X)
print(f" to'liq to'plamda natijalar bir xil: "
f"{np.allclose(a, b)}")
print("\n=== 2. O'quv va test alohida bo'lganda ===")
Xtr, Xte = X[:800], X[800:]
tg = ToGriMasshtab().fit(Xtr)
ntg = NotoGriMasshtab().fit(Xtr)
a_te = tg.transform(Xte)
b_te = ntg.transform(Xte)
print(f" to'g'ri: test o'rtachasi = "
f"{np.round(a_te.mean(axis=0)[:3], 4).tolist()}")
print(f" noto'g'ri: test o'rtachasi = "
f"{np.round(b_te.mean(axis=0)[:3], 4).tolist()}")
print(" noto'g'ri variant testni O'Z statistikasi bilan markazlaydi")
print("\n=== 3. Farq qayerdan ko'rinadi ===")
print(f" o'quv o'rtachasi (to'g'ri ishlatiladi): "
f"{np.round(tg.ortacha_[:3], 4).tolist()}")
print(f" test o'rtachasi (noto'g'ri ishlatiladi): "
f"{np.round(Xte.mean(axis=0)[:3], 4).tolist()}")
print(f" farq: "
f"{np.round(np.abs(tg.ortacha_ - Xte.mean(axis=0))[:3], 4).tolist()}")
print("\n=== 4. Kichik test to'plamida ta'sir kuchayadi ===")
print(f" {'test hajmi':>12} {'to_gri AUC':>12} {'noto_gri AUC':>14}")
for hajm in [30, 100, 400]:
ballar = {"tg": [], "ntg": []}
for boshi in range(0, 1200 - hajm, hajm):
Xte2 = X[boshi:boshi + hajm]
yte2 = y[boshi:boshi + hajm]
if len(set(yte2.tolist())) < 2:
continue
Xtr2 = np.delete(X, slice(boshi, boshi + hajm), axis=0)
ytr2 = np.delete(y, slice(boshi, boshi + hajm))
for nom, sinf in [("tg", ToGriMasshtab), ("ntg", NotoGriMasshtab)]:
q = Pipeline([("t", sinf()),
("m", LogisticRegression(max_iter=2000))])
q.fit(Xtr2, ytr2)
from sklearn.metrics import roc_auc_score
ballar[nom].append(
roc_auc_score(yte2, q.predict_proba(Xte2)[:, 1]))
print(f" {hajm:>12} {np.mean(ballar['tg']):>12.4f} "
f"{np.mean(ballar['ntg']):>14.4f}")
print(" ⭐ 'Testda qayta hisoblansa noto'g'ri bo'ladimi?' - asosiy savol")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Ikki transformer bir xil ma'lumotda ===
to'liq to'plamda natijalar bir xil: True
=== 2. O'quv va test alohida bo'lganda ===
to'g'ri: test o'rtachasi = [0.0114, -0.0296, -0.0121]
noto'g'ri: test o'rtachasi = [0.0, -0.0, -0.0]
noto'g'ri variant testni O'Z statistikasi bilan markazlaydi
=== 3. Farq qayerdan ko'rinadi ===
o'quv o'rtachasi (to'g'ri ishlatiladi): [0.4505, 0.2487, -0.0027]
test o'rtachasi (noto'g'ri ishlatiladi): [0.4628, 0.2182, -0.0146]
farq: [0.0124, 0.0304, 0.0119]
=== 4. Kichik test to'plamida ta'sir kuchayadi ===
test hajmi to_gri AUC noto_gri AUC
30 0.7265 0.7261
100 0.7302 0.7312
400 0.7316 0.7320
⭐ 'Testda qayta hisoblansa noto'g'ri bo'ladimi?' - asosiy savolNima ko'rsatdi: 2.2-bo'lim.
Misol 3 — check_estimator xatolarni topadi
"""To'rt xil xato va ularning tashxisi (real sklearn)."""
import warnings
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.exceptions import SkipTestWarning
from sklearn.utils.estimator_checks import check_estimator
from sklearn.utils.validation import check_is_fitted, validate_data
class ToGri(TransformerMixin, BaseEstimator):
def __init__(self, daraja=2):
self.daraja = daraja
def fit(self, X, y=None):
X = validate_data(self, X, dtype="numeric")
self.ortacha_ = X.mean(axis=0)
return self
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
return (X - self.ortacha_) ** self.daraja
class TartibXato(BaseEstimator, TransformerMixin): # mixin O'NGDA
def __init__(self, daraja=2):
self.daraja = daraja
def fit(self, X, y=None):
X = validate_data(self, X, dtype="numeric")
self.ortacha_ = X.mean(axis=0)
return self
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
return (X - self.ortacha_) ** self.daraja
class InitXato(TransformerMixin, BaseEstimator):
def __init__(self, daraja=2):
self.daraja = int(abs(daraja)) # O'ZGARTIRISH
def fit(self, X, y=None):
X = validate_data(self, X, dtype="numeric")
self.ortacha_ = X.mean(axis=0)
return self
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
return (X - self.ortacha_) ** self.daraja
class HolatXato(TransformerMixin, BaseEstimator):
def __init__(self, daraja=2):
self.daraja = daraja
def fit(self, X, y=None):
X = validate_data(self, X, dtype="numeric")
if not hasattr(self, "ortacha_"): # QAYTA FIT QILMAYDI
self.ortacha_ = X.mean(axis=0)
return self
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
return (X - self.ortacha_) ** self.daraja
def tekshir(sinf) -> str:
with warnings.catch_warnings():
warnings.simplefilter("ignore", SkipTestWarning)
try:
check_estimator(sinf())
return "O'TDI"
except Exception as xato:
birinchi = str(xato).splitlines()[0]
return f"{type(xato).__name__}: {birinchi[:58]}"
def main() -> None:
print("=== 1. To'rt variant ===")
variantlar = {
"ToGri (namuna)": ToGri,
"TartibXato (mixin o'ngda)": TartibXato,
"InitXato (__init__ o'zgartiradi)": InitXato,
"HolatXato (qayta fit qilmaydi)": HolatXato,
}
print(f" {'sinf':<36} natija")
for nom, sinf in variantlar.items():
print(f" {nom:<36} {tekshir(sinf)}")
print("\n=== 2. InitXato ni qo'lda ko'rish ===")
ob = InitXato(daraja=3)
print(f" InitXato(daraja=3).get_params(): {ob.get_params()}")
ob2 = InitXato(daraja=-3)
print(f" InitXato(daraja=-3).daraja: {ob2.daraja} (o'zgartirilgan)")
print(f" get_params qaytargan: {ob2.get_params()['daraja']}")
print("\n=== 3. HolatXato ni qo'lda ko'rish ===")
rng = np.random.default_rng(0)
X1 = rng.normal(0, 1, (100, 3))
X2 = rng.normal(10, 1, (100, 3))
h = HolatXato().fit(X1)
birinchi = h.ortacha_.copy()
h.fit(X2)
print(f" birinchi fit o'rtachasi: {np.round(birinchi, 3).tolist()}")
print(f" ikkinchi fit dan keyin: {np.round(h.ortacha_, 3).tolist()}")
print(f" o'zgardimi: {not np.allclose(birinchi, h.ortacha_)}")
print(" fit HAR SAFAR noldan hisoblashi kerak")
print("\n=== 4. To'g'ri variant qanday ===")
t = ToGri().fit(X1)
birinchi = t.ortacha_.copy()
t.fit(X2)
print(f" birinchi fit: {np.round(birinchi, 3).tolist()}")
print(f" ikkinchi fit: {np.round(t.ortacha_, 3).tolist()}")
print(f" o'zgardimi: {not np.allclose(birinchi, t.ortacha_)}")
print(" ⭐ check_estimator bu xatolarni avtomatik topadi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. To'rt variant ===
sinf natija
ToGri (namuna) O'TDI
TartibXato (mixin o'ngda) AssertionError: TartibXato is inheriting from mixins in the wrong order. I
InitXato (__init__ o'zgartiradi) TypeError: bad operand type for abs(): 'str'
HolatXato (qayta fit qilmaydi) O'TDI
=== 2. InitXato ni qo'lda ko'rish ===
InitXato(daraja=3).get_params(): {'daraja': 3}
InitXato(daraja=-3).daraja: 3 (o'zgartirilgan)
get_params qaytargan: 3
=== 3. HolatXato ni qo'lda ko'rish ===
birinchi fit o'rtachasi: [-0.102, 0.003, -0.008]
ikkinchi fit dan keyin: [-0.102, 0.003, -0.008]
o'zgardimi: False
fit HAR SAFAR noldan hisoblashi kerak
=== 4. To'g'ri variant qanday ===
birinchi fit: [-0.102, 0.003, -0.008]
ikkinchi fit: [9.976, 10.077, 9.919]
o'zgardimi: True
⭐ check_estimator bu xatolarni avtomatik topadiNima ko'rsatdi: 2.5, 2.7-bo'limlar.
Misol 4 — Domen belgisi transformeri
"""Bir necha ustundan yangi belgi (real numpy/pandas/sklearn)."""
import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import (GridSearchCV, StratifiedKFold,
cross_val_score)
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler
from sklearn.utils.validation import (_check_feature_names_in,
check_is_fitted, validate_data)
class NisbatBelgilari(TransformerMixin, BaseEstimator):
"""Ustun juftliklaridan xavfsiz nisbatlar yasaydi.
Nol maxrajni `eps` bilan cheklaydi, shuning uchun `inf` chiqmaydi.
"""
def __init__(self, juftliklar=((0, 1),), eps=1e-6, log=False):
self.juftliklar = juftliklar
self.eps = eps
self.log = log
def fit(self, X, y=None):
X = validate_data(self, X, dtype="numeric")
chegara = X.shape[1]
for a, b in self.juftliklar:
if not (0 <= a < chegara and 0 <= b < chegara):
raise ValueError(f"juftlik ({a}, {b}) chegaradan tashqarida")
return self
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
ustunlar = []
for a, b in self.juftliklar:
nisbat = X[:, a] / np.clip(np.abs(X[:, b]), self.eps, None)
ustunlar.append(np.log1p(np.abs(nisbat)) if self.log else nisbat)
return np.column_stack(ustunlar)
def get_feature_names_out(self, input_features=None):
nomlar = _check_feature_names_in(self, input_features)
qoshimcha = "_log" if self.log else ""
return np.asarray(
[f"{nomlar[a]}_ga_{nomlar[b]}{qoshimcha}"
for a, b in self.juftliklar], dtype=object)
def yarat(n: int = 2000, seed: int = 0):
rng = np.random.default_rng(seed)
df = pd.DataFrame({
"ogirlik": rng.gamma(2, 5, n),
"masofa": rng.gamma(2, 110, n),
"summa": rng.lognormal(11.5, 0.6, n),
"soat": rng.integers(0, 24, n).astype(float),
})
# maxrajda nol bo'lgan qatorlar - xavfsizlikni sinash uchun
df.loc[rng.choice(n, 25, replace=False), "masofa"] = 0.0
kuch = (-1.6 + 2.6 * (df["ogirlik"] / df["masofa"].clip(lower=1) > 0.09)
+ 0.9 * ((df["soat"] >= 7) & (df["soat"] <= 10)))
y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
return df, y
def main() -> None:
df, y = yarat()
print("=== 1. Ma'lumot ===")
print(f" {len(df)} qator, ustunlar: {list(df.columns)}")
print(f" masofa = 0 bo'lgan qatorlar: {int((df['masofa'] == 0).sum())}")
print("\n=== 2. Transformer chiqishi ===")
nb = NisbatBelgilari(juftliklar=((0, 1), (2, 0))).fit(df)
chiqish = nb.transform(df)
print(f" shakl: {chiqish.shape}")
print(f" nomlar: {nb.get_feature_names_out().tolist()}")
print(f" cheksiz qiymatlar: {int(np.isinf(chiqish).sum())}")
print(f" eng katta qiymat: {chiqish.max():.2f}")
print("\n=== 3. Quvurga ulash ===")
tayyor = ColumnTransformer([
("xom", "passthrough", ["ogirlik", "masofa", "summa", "soat"]),
("nisbat", NisbatBelgilari(juftliklar=((0, 1), (2, 0))),
["ogirlik", "masofa", "summa", "soat"]),
])
quvur = Pipeline([("t", tayyor),
("m", HistGradientBoostingClassifier(
max_iter=200, early_stopping=False,
random_state=0))])
quvur.fit(df, y)
print(f" belgilar: {quvur[:-1].get_feature_names_out().tolist()}")
cv = StratifiedKFold(4, shuffle=True, random_state=0)
asos = Pipeline([("m", HistGradientBoostingClassifier(
max_iter=200, early_stopping=False, random_state=0))])
b_asos = cross_val_score(asos, df, y, cv=cv, scoring="roc_auc")
b_yangi = cross_val_score(quvur, df, y, cv=cv, scoring="roc_auc")
print(f" {'variant':<22} {'CV AUC':>9}")
print(f" {'faqat xom belgilar':<22} {b_asos.mean():>9.4f}")
print(f" {'nisbatlar bilan':<22} {b_yangi.mean():>9.4f}")
print(f" o'sish: {b_yangi.mean() - b_asos.mean():+.4f}")
print("\n=== 4. Transformer parametri ham sozlanadi ===")
setka = {"t__nisbat__log": [False, True],
"t__nisbat__juftliklar": [((0, 1),), ((0, 1), (2, 0))]}
g = GridSearchCV(quvur, setka, cv=cv, scoring="roc_auc",
n_jobs=1).fit(df, y)
print(f" {'log':>6} {'juftliklar soni':>17} {'CV AUC':>9}")
for par, ball in zip(g.cv_results_["params"],
g.cv_results_["mean_test_score"]):
print(f" {str(par['t__nisbat__log']):>6} "
f"{len(par['t__nisbat__juftliklar']):>17} {ball:>9.4f}")
print(f" eng yaxshi: {g.best_score_:.4f}")
print("\n=== 5. FunctionTransformer bilan solishtirish ===")
log_t = FunctionTransformer(np.log1p, feature_names_out="one-to-one")
oddiy = make_pipeline(log_t, StandardScaler())
oddiy.set_output(transform="pandas")
chiqish2 = oddiy.fit_transform(df[["ogirlik", "summa"]])
print(f" FunctionTransformer chiqishi: {list(chiqish2.columns)}")
print(" holatsiz va nomlar o'zgarmasa - FunctionTransformer yetarli")
print(" ⭐ Domen belgisi quvur ICHIDA bo'lsa, ikki muhitda bir xil ishlaydi")
if __name__ == "__main__":
main()Natijaning muhim qismi:
=== 1. Ma'lumot ===
2000 qator, ustunlar: ['ogirlik', 'masofa', 'summa', 'soat']
masofa = 0 bo'lgan qatorlar: 25
=== 2. Transformer chiqishi ===
shakl: (2000, 2)
nomlar: ['ogirlik_ga_masofa', 'summa_ga_ogirlik']
cheksiz qiymatlar: 0
eng katta qiymat: 33374111.86
=== 3. Quvurga ulash ===
belgilar: ['xom__ogirlik', 'xom__masofa', 'xom__summa', 'xom__soat', 'nisbat__ogirlik_ga_masofa', 'nisbat__summa_ga_ogirlik']
variant CV AUC
faqat xom belgilar 0.7058
nisbatlar bilan 0.7452
o'sish: +0.0395
=== 4. Transformer parametri ham sozlanadi ===
log juftliklar soni CV AUC
False 1 0.7307
True 1 0.7307
False 2 0.7452
True 2 0.7452
eng yaxshi: 0.7452
=== 5. FunctionTransformer bilan solishtirish ===
FunctionTransformer chiqishi: ['ogirlik', 'summa']
holatsiz va nomlar o'zgarmasa - FunctionTransformer yetarli
⭐ Domen belgisi quvur ICHIDA bo'lsa, ikki muhitda bir xil ishlaydiNima ko'rsatdi: 2.1, 2.3, 2.6-bo'limlar.
5. To'g'ri va noto'g'ri tushunishlar
| Noto'g'ri fikr | To'g'risi |
|---|---|
"fit_transform ni yozish kerak" |
TransformerMixin beradi |
"Holatsiz transformerga fit kerak emas" |
fit baribir bo'lishi shart |
"get_feature_names_out ixtiyoriy" |
set_output uchun majburiy |
"transform DataFrame qaytarsin" |
numpy afzal, nomlar alohida |
"check_estimator sklearn ni tekshiradi" |
Sizning kodingizni |
"fit ni ikki marta chaqirmaymiz" |
CV chaqiradi — idempotent bo'lsin |
"Ustun nomlarini __init__ da yozsa bo'ladi" |
Indeks yoki tanlagich afzal |
"X ni saqlab qo'ysa bo'ladi" |
Xotira va leakage xavfi |
6. Keng tarqalgan xatolar va yechimlari
1. transform da statistika hisoblash
def transform(self, X): return (X - X.mean(0)) / X.std(0) # ⚠️
def fit(self, X, y=None):
self.ortacha_ = X.mean(0); self.std_ = X.std(0); return self # ✅2. get_feature_names_out yo'q
quvur.set_output(transform="pandas") # AttributeError # ⚠️
def get_feature_names_out(self, input_features=None):
return np.asarray([...], dtype=object) # ✅3. Nomlar soni mos emas
# transform 3 ustun, get_feature_names_out 2 nom -> xato # ⚠️
# ikkalasini bir manbadan hosil qiling # ✅4. Qayta fit qilmaydigan fit
if not hasattr(self, "ortacha_"): self.ortacha_ = X.mean(0) # ⚠️
self.ortacha_ = X.mean(0) # ✅5. Sinf darajasidagi atribut
class T(TransformerMixin, BaseEstimator):
ortacha_ = None # ⚠️
# faqat fit ichida self.ortacha_ = ... # ✅6. DataFrame ni kutish
def transform(self, X): return X["a"] / X["b"] # ⚠️
X = validate_data(self, X, reset=False); return X[:, 0] / X[:, 1] # ✅7. fit da X ni saqlash
def fit(self, X, y=None): self.X_ = X; return self # ⚠️
def fit(self, X, y=None): self.ortacha_ = X.mean(0); return self # ✅7. Integratsiya — bu bilim qayerda kerak bo'ladi
- 19.1-dars (o'tilgan): Estimator API
- 19.3-dars (o'tilgan): ColumnTransformer
- 19.5-dars: Kompozitsiya
- 19.8-dars: Diagnostika
- 19.10-dars: To'liq loyiha
8. Eng yaxshi amaliyotlar
Mixinlar chapda.
validate_databilan boshlang.check_is_fittednitransformda.get_feature_names_outni yozing.fithar safar noldan hisoblasin.transformnumpy qaytarsin.check_estimatorni ishga tushiring.Holatsiz bo'lsa
FunctionTransformerni ko'ring.
9. Amaliy topshiriq
Vazifa 1: Bashorat qiling
1. # mixin tartibi?
2. # fit_transform ni kim beradi?
3. # validate_data nima o'rnatadi?
4. # reset=False nima qiladi?
5. # check_is_fitted qayerda?
6. # get_feature_names_out nima uchun?
7. # _check_feature_names_in nima qiladi?
8. # holatli va holatsiz farqi?
9. # asosiy savol qanday?
10. # check_estimator nimani tekshiradi?
11. # transform nima qaytarsin?
12. # FunctionTransformer qachon yetarli?Javoblar
TransformerMixin, BaseEstimatorTransformerMixinn_features_in_,feature_names_in_- Ustunlar mosligini tekshiradi
transformboshidaset_outputva nomlar uchun- Nomlarni oladi yoki hosil qiladi
fitda biror narsa o'rganiladimi- "Testda qayta hisoblansa noto'g'ri bo'ladimi?"
- Sizning kodingizdagi shartnoma buzilishini
- numpy massiv
- Holatsiz, parametrsiz, nomlar oddiy bo'lsa
Vazifa 2: Xatolarni tuzating
1. def transform(self, X): return (X - X.mean(0)) / X.std(0)
2. # get_feature_names_out yo'q, set_output("pandas") ishlatiladi
3. if not hasattr(self, "ortacha_"): self.ortacha_ = X.mean(0)
4. class T(TransformerMixin, BaseEstimator):
ortacha_ = None
5. def transform(self, X): return X["a"] / X["b"]Javoblar
1. def fit(self, X, y=None):
self.ortacha_ = X.mean(0); self.std_ = X.std(0); return self
2. def get_feature_names_out(self, input_features=None):
return np.asarray([...], dtype=object)
3. self.ortacha_ = X.mean(0)
4. # faqat fit ichida: self.ortacha_ = ...
5. X = validate_data(self, X, reset=False); return X[:, 0] / X[:, 1]Vazifa 3: Skelet
Modellang:
- Fitdan oldin
- Fitdan keyin
- Ta'siri
- Ekotizim
Vazifa 4: Holat
Modellang:
- Bir xil ma'lumot
- Alohida test
- Farq manbai
- Test hajmi
Vazifa 5: check_estimator
Modellang:
- To'rt variant
InitXatoHolatXato- To'g'ri variant
Vazifa 6: Domen belgisi
Modellang:
- Ma'lumot
- Chiqish
- Quvur
- Sozlash
Vazifa 7: O'ylash
Transformeringiz fit da o'quv to'plamidagi eng katta qiymatni eslab qoladi va transform da barcha qiymatlarni shunga bo'ladi. check_estimator o'tadi, CV natijasi yaxshi. Ishlab chiqarishda esa ba'zi bashoratlar g'alati chiqadi. Nima bo'lishi mumkin?
Javob
Qisqa javob: maksimum — juda beqaror statistika. Yangi ma'lumotda o'quvdagidan katta qiymat uchrasa, natija 1 dan oshadi va model hech qachon ko'rmagan diapazonga tushadi.
1. Muammoning tuzilishi
| Holat | Natija |
|---|---|
| Yangi qiymat < o'quv maksimumi | [0, 1] — normal |
| Yangi qiymat > o'quv maksimumi | > 1 — model ko'rmagan hudud |
| O'quvda chetdagi qiymat bor edi | Barcha qiymatlar 0 ga siqiladi |
Maksimum bitta kuzatuvga bog'liq: o'quv to'plamida tasodifan bitta katta chetdagi qiymat bo'lsa, butun masshtab buziladi.
2. Nima uchun CV buni ko'rsatmadi
CV foldlari bir xil taqsimotdan olingan, shuning uchun test foldidagi maksimum o'quv maksimumidan kam farq qiladi. Ishlab chiqarishda esa:
- mavsumiy cho'qqilar
- yangi mijoz segmenti
- o'lchov xatosi (sensor nosozligi)
o'quvda ko'rilmagan qiymatlar beradi.
3. Barqarorroq muqobillar
# a) kvantil bilan cheklash (vinzorlash)
self.yuqori_ = np.quantile(X, 0.99, axis=0)
Xt = np.clip(X, None, self.yuqori_) / self.yuqori_
# b) RobustScaler mantiqi: median va IQR
self.markaz_ = np.median(X, axis=0)
self.tarqoqlik_ = np.subtract(*np.percentile(X, [75, 25], axis=0))
# c) o'rtacha va std (StandardScaler)Kvantil (0.99) maksimumdan ancha barqaror: bitta chetdagi qiymat uni sezilarli o'zgartirmaydi.
4. Chegaradan tashqari qiymatlarni ongli boshqarish
def transform(self, X):
check_is_fitted(self)
X = validate_data(self, X, dtype="numeric", reset=False)
Xt = X / self.yuqori_
if self.cheklash:
Xt = np.clip(Xt, 0.0, 1.0) # ongli qaror
return XtQaror sizniki: cheklash (model ko'rgan diapazonda qolish) yoki cheklmaslik (haqiqiy qiymatni saqlash). Muhimi — ongli bo'lishi va hujjatlashtirilishi.
5. Monitoring
Ishlab chiqarishda chegaradan chiqish hodisasini qayd eting:
tashqarida = (Xt > 1.0).mean()
if tashqarida > 0.01:
ogohlantir(f"kirishlarning {tashqarida:.1%} i o'quv diapazonidan tashqarida")Bu drift ning eng arzon indikatori.
6. Xulosa
- Maksimum/minimum — beqaror statistika
- Kvantil yoki median/IQR afzal
- Chegaradan chiqishni ongli boshqaring
- Ishlab chiqarishda chegaradan chiqishni kuzating
Nimani mustahkamlaydi: 2.2-bo'lim.
Xulosa
Bu darsda o'z transformeringizni yozishni o'rgandik.
Eng muhim uch fikr:
Skelet 20 qator:
__init__saqlaydi,fito'rganadi,transformqo'llaydi.TransformerMixinchapda bo'lsafit_transformbepul keladi;validate_datan_features_in_vafeature_names_in_ni o'rnatadi;check_is_fittedesatransformni himoya qiladi. Shu to'rt element bo'lsa, obyektingizPipelinevaGridSearchCVda tabiiy ishlaydi.Asosiy savol: "bu qiymat testda qayta hisoblansa noto'g'ri bo'ladimi?" Javob "ha" bo'lsa — u
fitda o'rganilib,_bilan tugaydigan atributga yozilishi kerak. Statistikanitransformda hisoblash — leakage, va u CV da ham qisman ko'rinmaydi.get_feature_names_outni yozing vacheck_estimatorni ishga tushiring. Birinchisiset_output("pandas")vaColumnTransformernomlari uchun kerak; ikkinchisi esaclone, idempotentlik va__init__shartnomasi bilan bog'liq xatolarni avtomatik topadi — qo'lda topish deyarli imkonsiz bo'lgan xatolarni.
Keyingi darsda kompozitsiya vositalarini ko'ramiz: FunctionTransformer, FeatureUnion, TransformedTargetRegressor va ularni qachon ishlatish kerakligi.
Izohlar (0)
Izoh yozish uchun kiring.
- Hozircha izoh yo'q. Birinchi bo'ling!