IlmHamroh
Data Science va sun'iy intellekt/scikit learn toliq2/10-dars18 daqiqa
Mundarija (22)

19.2-dars: Pipeline chuqur

19-QISM — SCIKIT-LEARN TO'LIQ · 2-dars


1. Kirish va motivatsiya

Pipeline ni 17.9-darsda leakage dan himoya vositasi sifatida ko'rgandik. Lekin u ancha ko'proq narsa: Pipeline ning o'zi estimator. Uning fit, predict, get_params metodlari bor, uni clone qilish mumkin, uni boshqa Pipeline ichiga joylash mumkin.

Bu shuni anglatadiki, Pipeline bilan ishlashning butun bir qatlami bor: qadamlarga nom bilan murojaat, kesish (quvur[:2]), qimmat qadamlarni keshlash (memory), qadamni o'chirish ("passthrough"), oraliq natijalarni DataFrame sifatida olish (set_output) va qadamlar orasida nima sodir bo'layotganini ko'rish.

Ko'pchilik bu imkoniyatlarni bilmaydi va natijada Pipeline ni "qora quti" deb hisoblaydi: nimadir noto'g'ri ketsa, uni buzib, qadamlarni qo'lda bajaradi — va leakage ni qaytarib keltiradi.

Bu darsda: qurilishi, named_steps va kesish, memory keshi, passthrough, set_output, oraliq natijalarni tekshirish va ichma-ich Pipeline.

Real vaziyat. Matn loyihasida TfidfVectorizer har GridSearchCV nomzodida qaytadan hisoblanardi — 40 nomzod × 5 fold = 200 marta, har biri 25 soniya. memory="kesh" qo'shilgach vektorizatsiya fold boshiga bir marta bajarildi va qidiruv 80 daqiqadan 9 daqiqaga tushdi.

Bu darsda Pipeline ni chuqur o'rganamiz.

Bu darsda:

  • Pipeline ning o'zi estimator
  • named_steps va kesish
  • memory keshi
  • passthrough
  • set_output
  • Oraliq natijalarni tekshirish
  • Tuzoqlar
  • Amaliy: quvurni tahlil qilish

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14, sklearn 1.9).


2. Nazariya — chuqur tushuntirish

2.1. Pipeline ning o'zi estimator

text
Pipeline([("a", A()), ("b", B()), ("m", M())])

fit:       A.fit_transform -> B.fit_transform -> M.fit
predict:   A.transform -> B.transform -> M.predict

QOIDA: OXIRGI qadamdan boshqa hammasi TRANSFORMER bo'lishi kerak
       oxirgisi transformer HAM, predictor HAM bo'lishi mumkin

Pipeline ning o'zi:
  fit, predict, score, get_params, set_params bor
  -> GridSearchCV, cross_val_score, boshqa Pipeline ichiga kiradi

make_pipeline(A(), B(), M()) -> nomlarni AVTOMATIK beradi
  ("a", "b", "m" o'rniga "a", "b", "m" kichik harf sinf nomi)

Nomlarni o'zingiz bering: make_pipeline avtomatik nomlari sinf nomiga bog'liq va sinfni almashtirsangiz param_grid buziladi.

2.2. named_steps va kesish

python
quvur.named_steps["m"]          # qadam obyekti
quvur["m"]                      # qisqa shakl
quvur.steps                     # [(nom, obyekt), ...]
quvur[-1]                       # oxirgi qadam

quvur[:2]                       # YANGI Pipeline: faqat birinchi 2 qadam
quvur[:-1].transform(X)         # modelgacha bo'lgan natija
quvur[1:]                       # ikkinchi qadamdan oxirigacha

DIQQAT: kesish YANGI Pipeline qaytaradi, lekin qadamlar
        NUSXALANMAYDI - o'sha obyektlarga havola

quvur[:-1].transform(X) — modelga nima kirayotganini ko'rishning eng oson yo'li.

2.3. memory keshi

python
from joblib import Memory
quvur = Pipeline(qadamlar, memory=Memory("kesh_papkasi", verbose=0))
# yoki
quvur = Pipeline(qadamlar, memory="kesh_papkasi")

QANDAY ISHLAYDI:
  har transformerning fit_transform natijasi DISKKA yoziladi
  kalit: transformer parametrlari + kirish massivi
  keyingi safar bir xil kalit -> diskdan o'qiladi

QACHON FOYDALI:
  GridSearchCV da faqat OXIRGI qadam parametri o'zgarsa
  qimmat tayyorlash (TF-IDF, tasvir belgilari, katta PCA)

DIQQAT:
  disk joyi va eskirgan keshni tozalash sizning ishingiz
  kichik ma'lumotda kesh QIMMATROQ bo'lishi mumkin

memory faqat oldingi qadamlar o'zgarmaganda yordam beradi: birinchi qadam parametri qidiruvda bo'lsa, kesh deyarli foydasiz.

2.4. passthrough

python
Pipeline([("s", StandardScaler()), ("p", PCA()), ("m", Ridge())])

# qadamni O'CHIRISH:
quvur.set_params(p="passthrough")

# QIDIRUVDA: qadam kerakmi yoki yo'qmi - shuni sinash
setka = [{"p": ["passthrough"]},
         {"p": [PCA()], "p__n_components": [2, 5, 10]}]

QOIDA: "passthrough" satri ma'lumotni O'ZGARTIRMASDAN o'tkazadi
       None ham ishlaydi, lekin "passthrough" aniqroq

"passthrough" bilan qadamning o'zi ham giperparametr bo'ladi: "PCA kerakmi?" degan savolga qidiruv javob beradi.

2.5. set_output

python
quvur.set_output(transform="pandas")     # yoki "polars", "default"

NATIJA: transform DataFrame qaytaradi, ustun nomlari saqlanadi
  -> oraliq natijani o'qish osonlashadi
  -> get_feature_names_out avtomatik ishlatiladi

GLOBAL:
  from sklearn import set_config
  set_config(transform_output="pandas")

DIQQAT:
  ba'zi transformerlar siyrak (sparse) chiqish beradi -
  ularni pandas ga aylantirib bo'lmaydi (sparse_output=False qo'ying)

set_output(transform="pandas") ishlab chiqish paytida ustun nomlarini saqlaydi va xatolarni topishni ancha osonlashtiradi.

2.6. Oraliq natijalarni tekshirish

text
1. quvur[:-1].transform(X)        modelga kiruvchi matritsa
2. quvur[:k].transform(X)         k-qadamgacha
3. quvur[-1].coef_                yakuniy model koeffitsiyentlari
4. quvur[:-1].get_feature_names_out()   belgilar nomlari
5. quvur.named_steps["s"].mean_   qadamning o'rganganlari

SHAKL O'ZGARISHINI KUZATISH:
  for i in range(len(quvur)):
      print(quvur[:i + 1].transform(X).shape)

Shakllar ketma-ketligini chop eting — noto'g'ri qadamni topishning eng tez yo'li.

2.7. Tuzoqlar

Asosiy tuzoqlar: make_pipeline avtomatik nomlariga tayanib param_grid yozish; oxirgi qadamdan oldin predictor qo'yish; memory ni birinchi qadam sozlanayotganda ishlatish; keshni tozalamaslik; siyrak chiqishda set_output("pandas"); kesishdan keyin fit qilib asl quvurni buzish; quvur.steps ni to'g'ridan-to'g'ri o'zgartirish.

2.8. Quvur — ochiq quti

Pipeline — qora quti emas. Unga nom bilan murojaat qiling (quvur["m"]), kesib ko'ring (quvur[:-1].transform(X)), oraliq shakllarni chop eting, set_output bilan ustun nomlarini saqlang. Qimmat tayyorlash bo'lsa memory qo'shing, qadam kerakligini tekshirmoqchi bo'lsangiz "passthrough" ni qidiruvga kiriting. Bularning hammasi leakage dan himoyani buzmasdan ishlaydi.


3. Tez ma'lumotnoma

python
from joblib import Memory
from sklearn.pipeline import Pipeline, make_pipeline

quvur = Pipeline([("s", StandardScaler()), ("p", PCA(n_components=5)),
                  ("m", Ridge())], memory=Memory("kesh", verbose=0))
quvur.set_output(transform="pandas")

quvur["m"] · quvur.named_steps["s"] · quvur.steps · quvur[-1]
quvur[:-1].transform(X)                # modelga kiruvchi
quvur[:-1].get_feature_names_out()     # belgilar nomlari
quvur.set_params(p="passthrough")      # qadamni o'chirish
quvur.set_params(m__alpha=10.0)        # ichki parametr

setka = [{"p": ["passthrough"]},
         {"p": [PCA()], "p__n_components": [2, 5]}]
QOIDA: nomlarni o'zing ber · kesib tekshir · qimmat bo'lsa memory ·
       qadam kerakmi -> passthrough

Pipeline xulosasi

Pipeline ning o'zi estimator
Oxirgidan boshqa hammasi transformer
named_steps / [] / kesish -> yangi Pipeline
memory: fit_transform natijasini keshlaydi
passthrough: qadamni o'chiradi
set_output("pandas"): ustun nomlari saqlanadi

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14).

Misol 1 — Qurilish, nomlar va kesish

python
"""Pipeline ichiga qarash (real numpy/pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import StandardScaler


def main() -> None:
    X, y = make_classification(n_samples=600, n_features=12, n_informative=6,
                               flip_y=0.15, random_state=0)
    ustunlar = [f"x{i}" for i in range(X.shape[1])]
    df = pd.DataFrame(X, columns=ustunlar)

    quvur = Pipeline([
        ("s", StandardScaler()),
        ("t", SelectKBest(f_classif, k=8)),
        ("p", PCA(n_components=4, random_state=0)),
        ("m", LogisticRegression(max_iter=2000)),
    ]).fit(df, y)

    print("=== 1. Qadamlarga murojaat ===")
    print(f"  qadam nomlari: {list(quvur.named_steps)}")
    print(f"  quvur['s'] turi: {type(quvur['s']).__name__}")
    print(f"  quvur[-1] turi: {type(quvur[-1]).__name__}")
    print(f"  quvur.steps[1][0]: {quvur.steps[1][0]}")
    print(f"  uzunlik: {len(quvur)}")

    print("\n=== 2. Kesish yangi Pipeline beradi ===")
    kesim = quvur[:2]
    print(f"  quvur[:2] turi: {type(kesim).__name__}")
    print(f"  qadamlari: {list(kesim.named_steps)}")
    print(f"  asl quvur o'zgarmadi: {list(quvur.named_steps)}")
    print(f"  bir xil obyektmi: {kesim['s'] is quvur['s']}")

    print("\n=== 3. Shakl har qadamda ===")
    print(f"  {'qadamgacha':<22} {'shakl':>12}")
    print(f"  {'kirish':<22} {str(df.shape):>12}")
    for i in range(len(quvur) - 1):
        nomi = quvur.steps[i][0]
        shakl = quvur[:i + 1].transform(df).shape
        print(f"  {nomi + ' gacha':<22} {str(shakl):>12}")
    print(f"  {'modelga kiruvchi':<22} "
          f"{str(quvur[:-1].transform(df).shape):>12}")

    print("\n=== 4. Har qadam nimani o'rgandi ===")
    print(f"  s.mean_ (birinchi 3): "
          f"{np.round(quvur['s'].mean_[:3], 4).tolist()}")
    tanlangan = quvur["t"].get_support(indices=True)
    print(f"  t tanlagan ustunlar: {tanlangan.tolist()}")
    print(f"  p tushuntirgan dispersiya: "
          f"{np.round(quvur['p'].explained_variance_ratio_, 4).tolist()}")
    print(f"  m koeffitsiyentlari: "
          f"{np.round(quvur['m'].coef_[0], 4).tolist()}")
    print(f"  belgilar nomlari (model kirishi): "
          f"{quvur[:-1].get_feature_names_out().tolist()}")

    print("\n=== 5. make_pipeline avtomatik nomlari ===")
    avto = make_pipeline(StandardScaler(), PCA(n_components=3),
                         LogisticRegression(max_iter=2000))
    print(f"  {list(avto.named_steps)}")
    print("  param_grid: 'pca__n_components' (sinf nomiga bog'liq!)")
    print("  ⭐ quvur[:-1].transform(X) - modelga nima kirayotgani")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Qadamlarga murojaat ===
  qadam nomlari: ['s', 't', 'p', 'm']
  quvur['s'] turi: StandardScaler
  quvur[-1] turi: LogisticRegression
  quvur.steps[1][0]: t
  uzunlik: 4

=== 2. Kesish yangi Pipeline beradi ===
  quvur[:2] turi: Pipeline
  qadamlari: ['s', 't']
  asl quvur o'zgarmadi: ['s', 't', 'p', 'm']
  bir xil obyektmi: True

=== 3. Shakl har qadamda ===
  qadamgacha                    shakl
  kirish                    (600, 12)
  s gacha                   (600, 12)
  t gacha                    (600, 8)
  p gacha                    (600, 4)
  modelga kiruvchi           (600, 4)

=== 4. Har qadam nimani o'rgandi ===
  s.mean_ (birinchi 3): [-0.483, 0.0508, 0.535]
  t tanlagan ustunlar: [0, 2, 3, 5, 7, 8, 9, 11]
  p tushuntirgan dispersiya: [0.3938, 0.1906, 0.1563, 0.1247]
  m koeffitsiyentlari: [-0.543, -0.1008, 0.2281, -0.2287]
  belgilar nomlari (model kirishi): ['pca0', 'pca1', 'pca2', 'pca3']

=== 5. make_pipeline avtomatik nomlari ===
  ['standardscaler', 'pca', 'logisticregression']
  param_grid: 'pca__n_components' (sinf nomiga bog'liq!)
  ⭐ quvur[:-1].transform(X) - modelga nima kirayotgani

Nima ko'rsatdi: 2.1, 2.2, 2.6-bo'limlar.

Misol 2 — passthrough va qadamni sozlash

python
"""Qadamning o'zi giperparametr sifatida (real numpy/sklearn)."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler


def main() -> None:
    X, y = make_classification(n_samples=800, n_features=10, n_informative=6,
                               n_redundant=2, flip_y=0.15, class_sep=0.9,
                               random_state=0)
    cv = StratifiedKFold(4, shuffle=True, random_state=0)

    quvur = Pipeline([
        ("s", StandardScaler()),
        ("poly", PolynomialFeatures(degree=2, include_bias=False)),
        ("p", PCA(n_components=5, random_state=0)),
        ("m", LogisticRegression(max_iter=3000)),
    ])

    print("=== 1. Qadamni qo'lda o'chirish ===")
    print(f"  {'sozlama':<34} {'model kirishi':>15}")
    quvur.fit(X, y)
    print(f"  {'hamma qadam':<34} "
          f"{str(quvur[:-1].transform(X).shape):>15}")
    quvur.set_params(poly="passthrough").fit(X, y)
    print(f"  {'poly = passthrough':<34} "
          f"{str(quvur[:-1].transform(X).shape):>15}")
    quvur.set_params(p="passthrough").fit(X, y)
    print(f"  {'poly va p = passthrough':<34} "
          f"{str(quvur[:-1].transform(X).shape):>15}")
    quvur.set_params(poly=PolynomialFeatures(degree=2, include_bias=False),
                     p=PCA(n_components=5, random_state=0))

    print("\n=== 2. Qidiruvda qadam kerakmi ===")
    setka = [
        {"poly": ["passthrough"], "p": ["passthrough"],
         "m__C": [0.05, 1.0]},
        {"poly": ["passthrough"], "p": [PCA(random_state=0)],
         "p__n_components": [3, 6], "m__C": [0.05, 1.0]},
        {"poly": [PolynomialFeatures(degree=2, include_bias=False)],
         "p": ["passthrough"], "m__C": [0.05, 1.0]},
        {"poly": [PolynomialFeatures(degree=2, include_bias=False)],
         "p": [PCA(random_state=0)], "p__n_components": [6, 12],
         "m__C": [0.05, 1.0]},
    ]
    g = GridSearchCV(quvur, setka, cv=cv, scoring="roc_auc",
                     n_jobs=1).fit(X, y)
    print(f"  nomzodlar: {len(g.cv_results_['params'])}")
    print(f"  eng yaxshi ball: {g.best_score_:.4f}")

    print("\n=== 3. Konfiguratsiyalar bo'yicha eng yaxshi ===")
    print(f"  {'konfiguratsiya':<28} {'eng yaxshi ball':>16}")
    for par, ball in zip(g.cv_results_["params"],
                         g.cv_results_["mean_test_score"]):
        pass
    guruhlar = {}
    for par, ball in zip(g.cv_results_["params"],
                         g.cv_results_["mean_test_score"]):
        poly_bor = par["poly"] != "passthrough"
        pca_bor = par["p"] != "passthrough"
        kalit = (f"{'poly+' if poly_bor else ''}"
                 f"{'pca' if pca_bor else 'xom'}")
        guruhlar[kalit] = max(guruhlar.get(kalit, -1), float(ball))
    for kalit, ball in sorted(guruhlar.items(), key=lambda kv: -kv[1]):
        print(f"  {kalit:<28} {ball:>16.4f}")

    print("\n=== 4. Tanlangan konfiguratsiya ===")
    eng = g.best_params_
    print(f"  poly: {'bor' if eng['poly'] != 'passthrough' else 'yo_q'}")
    print(f"  pca: {'bor' if eng['p'] != 'passthrough' else 'yo_q'}")
    if eng["p"] != "passthrough":
        print(f"  n_components: {eng.get('p__n_components')}")
    print(f"  C: {eng['m__C']}")
    print(f"  model kirishi: "
          f"{g.best_estimator_[:-1].transform(X).shape}")
    print("  ⭐ 'passthrough' qadamni giperparametrga aylantiradi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Qadamni qo'lda o'chirish ===
  sozlama                              model kirishi
  hamma qadam                               (800, 5)
  poly = passthrough                        (800, 5)
  poly va p = passthrough                  (800, 10)

=== 2. Qidiruvda qadam kerakmi ===
  nomzodlar: 12
  eng yaxshi ball: 0.8784

=== 3. Konfiguratsiyalar bo'yicha eng yaxshi ===
  konfiguratsiya                eng yaxshi ball
  poly+xom                               0.8784
  xom                                    0.8569
  pca                                    0.7920
  poly+pca                               0.7446

=== 4. Tanlangan konfiguratsiya ===
  poly: bor
  pca: yo_q
  C: 0.05
  model kirishi: (800, 65)
  ⭐ 'passthrough' qadamni giperparametrga aylantiradi

Nima ko'rsatdi: 2.4-bo'lim.

Misol 3 — set_output va ustun nomlari

python
"""DataFrame chiqish bilan quvurni o'qish (real pandas/sklearn)."""

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def yarat(n: int = 400, seed: int = 0) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    return pd.DataFrame({
        "yosh": rng.integers(18, 70, n).astype(float),
        "daromad": rng.lognormal(10, 0.5, n),
        "bolalar": rng.integers(0, 4, n).astype(float),
        "hudud": rng.choice(["shimol", "janub", "markaz"], n),
        "tarif": rng.choice(["oddiy", "premium"], n),
    })


def main() -> None:
    df = yarat()
    y = ((df["daromad"] > df["daromad"].median())
         & (df["yosh"] < 50)).astype(int).to_numpy()

    tayyor = ColumnTransformer([
        ("son", StandardScaler(), make_column_selector(dtype_include=np.number)),
        ("kat", OneHotEncoder(sparse_output=False, handle_unknown="ignore"),
         make_column_selector(dtype_include="str")),
    ])
    quvur = Pipeline([("t", tayyor),
                      ("p", PCA(n_components=4, random_state=0)),
                      ("m", LogisticRegression(max_iter=2000))])

    print("=== 1. Sukut chiqish: numpy ===")
    quvur.fit(df, y)
    xom = quvur[:1].transform(df)
    print(f"  turi: {type(xom).__name__}, shakli: {xom.shape}")
    print(f"  birinchi qator: {np.round(xom[0], 3).tolist()}")

    print("\n=== 2. set_output('pandas') ===")
    quvur.set_output(transform="pandas")
    quvur.fit(df, y)
    ramka = quvur[:1].transform(df)
    print(f"  turi: {type(ramka).__name__}")
    print(f"  ustunlar: {list(ramka.columns)}")
    print(f"  birinchi qator:")
    print("   ", ramka.iloc[0].round(3).to_dict())

    print("\n=== 3. Har qadamdan keyingi ustunlar ===")
    print(f"  {'qadam':<10} {'ustunlar soni':>14}  nomlari (birinchi 4)")
    print(f"  {'kirish':<10} {df.shape[1]:>14}  {list(df.columns)[:4]}")
    for i in range(len(quvur) - 1):
        nomi = quvur.steps[i][0]
        chiqish = quvur[:i + 1].transform(df)
        nomlar = list(chiqish.columns)
        print(f"  {nomi:<10} {len(nomlar):>14}  {nomlar[:4]}")

    print("\n=== 4. Koeffitsiyentlarni nomlar bilan o'qish ===")
    nomlar = quvur[:-1].get_feature_names_out()
    koef = quvur["m"].coef_[0]
    tartib = np.argsort(-np.abs(koef))
    print(f"  {'belgi':<12} {'koeffitsiyent':>14}")
    for i in tartib:
        print(f"  {nomlar[i]:<12} {koef[i]:>+14.4f}")
    print("  ⭐ set_output('pandas') quvurni o'qishga qulay qiladi")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Sukut chiqish: numpy ===
  turi: ndarray, shakli: (400, 8)
  birinchi qator: [1.119, -0.617, -1.356, 0.0, 0.0, 1.0, 1.0, 0.0]

=== 2. set_output('pandas') ===
  turi: DataFrame
  ustunlar: ['son__yosh', 'son__daromad', 'son__bolalar', 'kat__hudud_janub', 'kat__hudud_markaz', 'kat__hudud_shimol', 'kat__tarif_oddiy', 'kat__tarif_premium']
  birinchi qator:
    {'son__yosh': 1.119, 'son__daromad': -0.617, 'son__bolalar': -1.356, 'kat__hudud_janub': 0.0, 'kat__hudud_markaz': 0.0, 'kat__hudud_shimol': 1.0, 'kat__tarif_oddiy': 1.0, 'kat__tarif_premium': 0.0}

=== 3. Har qadamdan keyingi ustunlar ===
  qadam       ustunlar soni  nomlari (birinchi 4)
  kirish                  5  ['yosh', 'daromad', 'bolalar', 'hudud']
  t                       8  ['son__yosh', 'son__daromad', 'son__bolalar', 'kat__hudud_janub']
  p                       4  ['pca0', 'pca1', 'pca2', 'pca3']

=== 4. Koeffitsiyentlarni nomlar bilan o'qish ===
  belgi         koeffitsiyent
  pca2                +2.1334
  pca1                -0.7789
  pca3                -0.1307
  pca0                +0.0193
  ⭐ set_output('pandas') quvurni o'qishga qulay qiladi

Nima ko'rsatdi: 2.5, 2.6-bo'limlar.

Misol 4 — memory keshi

python
"""Qimmat qadamni keshlash (real numpy/sklearn/joblib)."""

import shutil
import tempfile
from pathlib import Path

import numpy as np
from joblib import Memory
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.utils.validation import check_is_fitted, validate_data


class QimmatTayyorlash(TransformerMixin, BaseEstimator):
    """Har fit_transform da hisoblashni SANAYDIGAN transformer."""

    hisoblagich = {"soni": 0}          # namoyish uchun umumiy hisoblagich

    def __init__(self, daraja=2):
        self.daraja = daraja

    def fit(self, X, y=None):
        X = validate_data(self, X, dtype="numeric")
        QimmatTayyorlash.hisoblagich["soni"] += 1
        # "qimmat" hisob: bir necha matritsa ko'paytmasi
        self.proyeksiya_ = np.linalg.svd(X, full_matrices=False)[2][:6].T
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, dtype="numeric", reset=False)
        return np.hstack([X @ self.proyeksiya_,
                          (X[:, :4] ** self.daraja)])


def main() -> None:
    X, y = make_classification(n_samples=1200, n_features=20,
                               n_informative=8, flip_y=0.15, random_state=0)
    cv = StratifiedKFold(4, shuffle=True, random_state=0)
    setka = {"m__C": [0.01, 0.1, 1.0, 10.0, 100.0]}

    print("=== 1. Keshsiz: nechta fit_transform ===")
    QimmatTayyorlash.hisoblagich["soni"] = 0
    quvur = Pipeline([("s", StandardScaler()),
                      ("q", QimmatTayyorlash()),
                      ("m", LogisticRegression(max_iter=3000))])
    g1 = GridSearchCV(quvur, setka, cv=cv, scoring="roc_auc",
                      n_jobs=1).fit(X, y)
    keshsiz = QimmatTayyorlash.hisoblagich["soni"]
    print(f"  nomzodlar: {len(setka['m__C'])}, foldlar: 4")
    print(f"  QimmatTayyorlash.fit chaqirilishi: {keshsiz}")
    print(f"  kutilgan: 5 * 4 + 1 (yakuniy refit) = 21")
    print(f"  eng yaxshi ball: {g1.best_score_:.4f}")

    print("\n=== 2. Kesh bilan ===")
    papka = Path(tempfile.mkdtemp(prefix="sk_kesh_"))
    try:
        QimmatTayyorlash.hisoblagich["soni"] = 0
        quvur2 = Pipeline([("s", StandardScaler()),
                           ("q", QimmatTayyorlash()),
                           ("m", LogisticRegression(max_iter=3000))],
                          memory=Memory(papka, verbose=0))
        g2 = GridSearchCV(quvur2, setka, cv=cv, scoring="roc_auc",
                          n_jobs=1).fit(X, y)
        keshli = QimmatTayyorlash.hisoblagich["soni"]
        print(f"  QimmatTayyorlash.fit chaqirilishi: {keshli}")
        print(f"  kutilgan: har fold uchun 1 marta = 4 + 1")
        print(f"  eng yaxshi ball: {g2.best_score_:.4f}")
        print(f"  natijalar bir xil: "
              f"{abs(g1.best_score_ - g2.best_score_) < 1e-12}")

        print("\n=== 3. Tejash ===")
        print(f"  {'variant':<16} {'fit soni':>10} {'nisbat':>9}")
        print(f"  {'keshsiz':<16} {keshsiz:>10} {1.0:>8.1f}x")
        print(f"  {'kesh bilan':<16} {keshli:>10} "
              f"{keshsiz / max(keshli, 1):>8.1f}x")
        fayllar = sum(1 for _ in papka.rglob("*") if _.is_file())
        print(f"  keshdagi fayllar: {fayllar}")

        print("\n=== 4. Kesh qachon YORDAM BERMAYDI ===")
        QimmatTayyorlash.hisoblagich["soni"] = 0
        setka2 = {"q__daraja": [2, 3], "m__C": [0.1, 1.0]}
        quvur3 = Pipeline([("s", StandardScaler()),
                           ("q", QimmatTayyorlash()),
                           ("m", LogisticRegression(max_iter=3000))],
                          memory=Memory(papka, verbose=0))
        GridSearchCV(quvur3, setka2, cv=cv, scoring="roc_auc",
                     n_jobs=1).fit(X, y)
        print(f"  q__daraja ham sozlanganda fit soni: "
              f"{QimmatTayyorlash.hisoblagich['soni']}")
        print("  qadam parametri o'zgarsa kesh kaliti ham o'zgaradi")
        print("  ⭐ memory faqat OLDINGI qadamlar qat'iy bo'lsa foydali")
    finally:
        shutil.rmtree(papka, ignore_errors=True)


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Keshsiz: nechta fit_transform ===
  nomzodlar: 5, foldlar: 4
  QimmatTayyorlash.fit chaqirilishi: 21
  kutilgan: 5 * 4 + 1 (yakuniy refit) = 21
  eng yaxshi ball: 0.8919

=== 2. Kesh bilan ===
  QimmatTayyorlash.fit chaqirilishi: 5
  kutilgan: har fold uchun 1 marta = 4 + 1
  eng yaxshi ball: 0.8919
  natijalar bir xil: True

=== 3. Tejash ===
  variant            fit soni    nisbat
  keshsiz                  21      1.0x
  kesh bilan                5      4.2x
  keshdagi fayllar: 22

=== 4. Kesh qachon YORDAM BERMAYDI ===
  q__daraja ham sozlanganda fit soni: 4
  qadam parametri o'zgarsa kesh kaliti ham o'zgaradi
  ⭐ memory faqat OLDINGI qadamlar qat'iy bo'lsa foydali

Nima ko'rsatdi: 2.3-bo'lim.


5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"Pipeline — qora quti" Kesib, nom bilan ochish mumkin
"Kesish qadamlarni nusxalaydi" O'sha obyektlarga havola
"memory har doim tezlashtiradi" Faqat oldingi qadamlar qat'iy bo'lsa
"make_pipeline qulayroq" Nomlar sinfga bog'liq — mo'rt
"Qadamni o'chirish uchun qayta qurish kerak" "passthrough"
"set_output faqat chiroylik uchun" Xatolarni topishni osonlashtiradi
"Oxirgi qadam predictor bo'lishi shart" Transformer ham bo'lishi mumkin
"Pipeline estimator emas" To'liq estimator

6. Keng tarqalgan xatolar va yechimlari

1. Avtomatik nomlarga tayanish

python
make_pipeline(StandardScaler(), PCA())
setka = {"pca__n_components": [2, 5]}     # sinf almashsa buziladi  # ⚠️
Pipeline([("s", StandardScaler()), ("p", PCA())])
setka = {"p__n_components": [2, 5]}                                # ✅

2. Qadamni qo'lda chiqarib tashlash

python
X2 = quvur["s"].transform(X); model.fit(X2, y)   # leakage xavfi   # ⚠️
quvur.set_params(p="passthrough")                                  # ✅

3. Kesimni fit qilish

python
quvur[:2].fit(X, y)          # asl quvur qadamlarini o'zgartiradi  # ⚠️
from sklearn.base import clone
clone(quvur[:2]).fit(X, y)                                         # ✅

4. Siyrak chiqishda pandas

python
OneHotEncoder()               # sparse_output=True (sukut)
quvur.set_output(transform="pandas")     # xato                    # ⚠️
OneHotEncoder(sparse_output=False)                                 # ✅

5. Keshni tozalamaslik

python
Pipeline(qadamlar, memory="kesh")   # papka cheksiz o'sadi         # ⚠️
memory = Memory(papka, verbose=0); ...; memory.clear()             # ✅

6. Oxirgidan oldin predictor

python
Pipeline([("m", Ridge()), ("s", StandardScaler())])                # ⚠️
Pipeline([("s", StandardScaler()), ("m", Ridge())])                # ✅

7. steps ni qo'lda o'zgartirish

python
quvur.steps[1] = ("p", PCA(n_components=3))                        # ⚠️
quvur.set_params(p=PCA(n_components=3))                            # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 17.9-dars (o'tilgan): Pipeline va leakage
  • 19.1-dars (o'tilgan): Estimator API
  • 19.3-dars: ColumnTransformer
  • 19.5-dars: Kompozitsiya
  • 19.10-dars: To'liq loyiha

8. Eng yaxshi amaliyotlar

  1. Nomlarni o'zingiz bering.

  2. quvur[:-1].transform(X) bilan tekshiring.

  3. Shakllar ketma-ketligini chop eting.

  4. Ishlab chiqishda set_output("pandas").

  5. Qimmat tayyorlash bo'lsa memory.

  6. Qadam kerakligini "passthrough" bilan sinang.

  7. Kesimni clone qilib fit qiling.

  8. Keshni davriy tozalang.


9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # Pipeline ning o'zi nima?
2.  # oxirgidan boshqa qadamlar qanday bo'lishi kerak?
3.  # quvur[:2] nima qaytaradi?
4.  # quvur[:-1].transform(X) nima beradi?
5.  # memory nimani keshlaydi?
6.  # memory qachon foydasiz?
7.  # passthrough nima qiladi?
8.  # set_output nima beradi?
9.  # make_pipeline kamchiligi?
10. # named_steps nima?
11. # kesim nusxa oladimi?
12. # oxirgi qadam transformer bo'la oladimi?
Javoblar
  1. Estimator
  2. Transformer
  3. Yangi Pipeline
  4. Modelga kiruvchi matritsa
  5. Transformerlarning fit_transform natijasini
  6. Oldingi qadam parametri sozlanayotganda
  7. Qadamni o'chiradi
  8. DataFrame chiqish, ustun nomlari
  9. Nomlar sinfga bog'liq
  10. Nom → qadam lug'ati
  11. Yo'q, havola
  12. Ha

Vazifa 2: Xatolarni tuzating

python
1.  make_pipeline(StandardScaler(), PCA())
    setka = {"pca__n_components": [2, 5]}

2.  X2 = quvur["s"].transform(X); model.fit(X2, y)

3.  quvur[:2].fit(X, y)

4.  OneHotEncoder(); quvur.set_output(transform="pandas")

5.  quvur.steps[1] = ("p", PCA(n_components=3))
Javoblar
python
1.  Pipeline([("s", StandardScaler()), ("p", PCA())])
    setka = {"p__n_components": [2, 5]}

2.  quvur.set_params(p="passthrough")

3.  clone(quvur[:2]).fit(X, y)

4.  OneHotEncoder(sparse_output=False)

5.  quvur.set_params(p=PCA(n_components=3))

Vazifa 3: Kesish

Modellang:

  1. Murojaat
  2. Kesish
  3. Shakllar
  4. O'rganilganlar

Vazifa 4: passthrough

Modellang:

  1. Qo'lda o'chirish
  2. Qidiruvda
  3. Konfiguratsiyalar
  4. Tanlov

Vazifa 5: set_output

Modellang:

  1. numpy
  2. pandas
  3. Ustunlar
  4. Koeffitsiyentlar

Vazifa 6: memory

Modellang:

  1. Keshsiz
  2. Kesh bilan
  3. Tejash
  4. Qachon foydasiz

Vazifa 7: O'ylash

GridSearchCV da memory qo'shdingiz, lekin tezlik umuman o'zgarmadi. Qidiruv maydoni: {"tfidf__max_features": [5000, 20000], "svc__C": [0.1, 1, 10]}. Nima bo'lgan va qanday tuzatasiz?

Javob

Qisqa javob: tfidf__max_features ham qidirilayotgani uchun kesh kaliti har nomzodda o'zgaradi — kesh hech qachon urilmaydi.

1. Nima uchun

memory transformerning fit_transform natijasini (parametrlar, kirish ma'lumoti) kaliti bo'yicha saqlaydi. max_features o'zgarsa — kalit o'zgaradi — kesh yangi yozuv yaratadi, eskisidan foydalanmaydi.

nomzodlar: 2 (max_features) x 3 (C) = 6
kesh kalitlari: 2 x 4 fold = 8 ta noyob
kesh urilishi: har kalit uchun 3 marta C o'zgarganda... ?

Aslida kesh qisman ishlaydi: bir xil max_features bilan uchta C bir xil kalitni ishlatadi. Ya'ni 6 nomzod × 4 fold = 24 o'rniga 2 × 4 = 8 marta TF-IDF hisoblanadi — uch barobar tejash. Agar tezlik umuman o'zgarmagan bo'lsa, boshqa sabab bor.

2. Tezlik o'zgarmasligining boshqa sabablari

Sabab Tekshirish
TF-IDF qimmat emas, SVC qimmat Har qadamning vaqtini alohida o'lchang
n_jobs > 1 — har protsess o'z keshini qayta o'qiydi n_jobs=1 bilan solishtiring
Kesh diski sekin (tarmoq diski) Mahalliy tempfile ga o'tkazing
Ma'lumot katta — diskka yozish/o'qish hisobdan qimmat Kesh hajmini ko'ring

3. Tuzatish: maydonni ikki bosqichga bo'ling

python
# 1-bosqich: max_features ni qat'iy qilib, C ni qidiring
quvur.set_params(tfidf__max_features=20000)
g1 = GridSearchCV(quvur, {"svc__C": [0.1, 1, 10]}, cv=cv)   # kesh TO'LIQ ishlaydi

# 2-bosqich: eng yaxshi C bilan max_features ni sinang
quvur.set_params(svc__C=g1.best_params_["svc__C"])
g2 = GridSearchCV(quvur, {"tfidf__max_features": [5000, 20000]}, cv=cv)

4. Muqobil: tayyorlashni qidiruvdan chiqarish

Agar max_features natijaga kam ta'sir qilsa, uni umuman sozlamang: bitta oqilona qiymat tanlang va butun byudjetni modelga bering.

5. Qachon memory haqiqatan foydali

Pipeline([("qimmat_tayyorlash", ...),   # parametrlari QAT'IY
          ("model", ...)])              # faqat SHU sozlanadi

Ya'ni qidirilayotgan parametrlar quvurning oxiriga yaqin bo'lishi kerak.

6. Xulosa

  1. Kesh kaliti parametrlarga bog'liq
  2. Tayyorlash parametrini qidirsangiz kesh foyda bermaydi
  3. Maydonni ikki bosqichga bo'ling
  4. Har qadamning haqiqiy narxini o'lchang

Nimani mustahkamlaydi: 2.3-bo'lim.


Xulosa

Bu darsda Pipeline ni chuqur ko'rib chiqdik.

Eng muhim uch fikr:

  1. Pipeline — estimator, va u ochiq quti. Unga nom bilan murojaat qiling (quvur["m"]), kesing (quvur[:2] yangi Pipeline qaytaradi), va eng muhimi — quvur[:-1].transform(X) bilan modelga nima kirayotganini ko'ring. Shakllar ketma-ketligini chop etish noto'g'ri qadamni topishning eng tez yo'li.

  2. Nomlarni o'zingiz bering va "passthrough" ni eslang. make_pipeline nomlari sinf nomiga bog'liq, shuning uchun modelni almashtirganda param_grid jim buziladi. "passthrough" esa qadamni o'chiradi va uni giperparametrga aylantiradi: "PCA kerakmi?" degan savolga qidiruvning o'zi javob beradi.

  3. memory faqat oldingi qadamlar qat'iy bo'lganda tezlashtiradi. Kesh kaliti transformer parametrlari va kirish ma'lumotidan tuziladi; tayyorlash parametrini qidirsangiz kalit har safar o'zgaradi va kesh foydasiz bo'ladi. Shuning uchun maydonni ikki bosqichga bo'ling: avval qat'iy tayyorlash bilan modelni sozlang.

Keyingi darsda ColumnTransformer ni batafsil ko'ramiz: ustun tanlagichlari, remainder, get_feature_names_out, siyrak chiqish va ichma-ich quvurlar.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!