IlmHamroh
Data Science va sun'iy intellekt/scikit learn toliq6/10-dars19 daqiqa
Mundarija (22)

19.6-dars: Model saqlash

19-QISM — SCIKIT-LEARN TO'LIQ · 6-dars


1. Kirish va motivatsiya

O'rgatilgan model diskka saqlanmasa, u faqat ishlab turgan Python jarayonida mavjud. Saqlash — ishlab chiqarishga chiqishning birinchi shartidir.

joblib.dump(model, "model.joblib") — bir qator. Lekin bu bir qatorning ortida bir nechta jiddiy masala turadi: versiya mosligi (boshqa sklearn versiyasida yuklash), xavfsizlik (pickle ixtiyoriy kod bajaradi), nima saqlanishi (faqat modelmi yoki metama'lumot ham), hajm va nima saqlab bo'lmasligi (lambda, mahalliy funksiya, ochiq fayl).

Ko'pchilik jamoalar bu masalalarga faqat birinchi nosozlikdan keyin duch keladi: model olti oydan keyin yuklanmay qoladi yoki yuklanadi-yu, jim noto'g'ri bashorat beradi.

Bu darsda: joblib va pickle, nima saqlab bo'lmaydi, versiya muammolari, paket g'oyasi (model + metama'lumot), hajmni kamaytirish va xavfsizlik.

Real vaziyat. Model joblib bilan saqlandi va bir yildan keyin yangi serverda yuklandi — xatosiz. Lekin sklearn versiyasi 1.2 dan 1.5 ga o'zgargandi va bitta transformerning ichki ifodasi o'zgargan edi: model yuklandi, predict ishladi, natijalar esa jim siljidi. Xato uch hafta davom etdi. Endi jamoada har paketda sklearn.__version__ saqlanadi va yuklashda tekshiriladi.

Bu darsda model saqlashni o'rganamiz.

Bu darsda:

  • joblib va pickle
  • Nima saqlab bo'lmaydi
  • Versiya mosligi
  • Paket g'oyasi
  • Hajm va siqish
  • Xavfsizlik
  • Tuzoqlar
  • Amaliy: topshirish paketi

ℹ Misollar real numpy/pandas/sklearn bilan (Python 3.14, sklearn 1.9).


2. Nazariya — chuqur tushuntirish

2.1. joblib va pickle

python
import joblib
joblib.dump(model, "model.joblib", compress=3)
model = joblib.load("model.joblib")

import pickle
with open("model.pkl", "wb") as f:
    pickle.dump(model, f, protocol=pickle.HIGHEST_PROTOCOL)

FARQI:
  joblib katta numpy massivlarini SAMARALIROQ saqlaydi
  joblib compress= bilan siqadi (0-9 yoki ("lz4", 3))
  joblib memmap bilan yuklashi mumkin (mmap_mode="r")
  ichkarida joblib ham pickle protokolini ishlatadi

TAVSIYA: sklearn modellari uchun joblib

sklearn modellari uchun joblib — u numpy massivlarini pickle dan tezroq va ixchamroq saqlaydi.

2.2. Nima saqlab bo'lmaydi

text
PICKLE QILINMAYDIGANLAR:
  lambda                      -> modul darajasidagi funksiya yozing
  mahalliy (ichki) funksiya   -> modul darajasiga chiqaring
  ochiq fayl, soket, qulf     -> obyektda saqlamang
  generator, korutina         -> ro'yxatga aylantiring
  __main__ dagi sinf          -> yuklashda modul kerak bo'ladi

ENG KO'P UCHRAYDIGANI:
  FunctionTransformer(lambda X: ...)   -> PicklingError

MUHIM NUANS:
  pickle SINF KODINI saqlamaydi, faqat MODUL YO'LINI
  -> yuklashda o'sha modul import qilinishi SHART
  -> o'z transformeringiz alohida modulda bo'lsin

pickle sinf kodini saqlamaydi, faqat "qaysi moduldan import qilish" ni — shuning uchun o'z sinflaringiz o'rnatiladigan modulda bo'lishi kerak.

2.3. Versiya mosligi

text
sklearn KAFOLAT BERMAYDI: turli versiyalar orasida pickle mosligi yo'q

XAVFLAR:
  1. Yuklanmaydi           -> AttributeError, ochiq xato (YAXSHI)
  2. InconsistentVersionWarning -> ogohlantirish (E'TIBOR BERING)
  3. Yuklanadi, lekin boshqacha ishlaydi -> JIM XATO (ENG YOMON)

HIMOYA:
  paketga versiyalarni YOZING va yuklashda TEKSHIRING
  requirements ni qulflang (pin): scikit-learn==1.9.1
  yuklashdan keyin NAZORAT NAMUNASIDA bashoratni solishtiring

NAZORAT NAMUNASI (eng ishonchli usul):
  saqlashda 20-50 qator va ularning bashoratlarini ham saqlang
  yuklashda qayta hisoblab, farqni tekshiring

Nazorat namunasi — versiya nosozligini aniqlashning eng ishonchli usuli: model yuklanib, jim boshqacha ishlay boshlasa ham darhol bilinadi.

2.4. Paket g'oyasi

python
paket = {
    "quvur": quvur,                       # butun tayyorlash + model
    "belgilar": list(X.columns),          # kirish ustunlari va TARTIBI
    "metrika": {"cv_auc": 0.84, "test_auc": 0.83},
    "versiyalar": {"sklearn": sklearn.__version__,
                   "numpy": np.__version__,
                   "pandas": pd.__version__,
                   "python": sys.version.split()[0]},
    "sana": "2026-09-21",
    "nazorat": {"X": X_nazorat, "y_pred": p_nazorat},
}
joblib.dump(paket, "model.joblib", compress=3)

NIMA UCHUN SHUNCHAKI MODEL EMAS:
  model yolg'iz o'zi kontekstsiz
  olti oydan keyin "bu qaysi ma'lumotda o'rgatilgan?" savoli chiqadi

Modelni yolg'iz saqlamang — paketga versiyalar, kirish ustunlari va nazorat namunasini ham qo'shing.

2.5. Hajm va siqish

text
compress=0 (sukut)   tez, katta
compress=3           muvozanat (tavsiya)
compress=9           sekin, eng kichik
compress=("zlib", 3) siqish algoritmini aniq tanlash
compress=("lz4", 3)  juda tez (python-lz4 o'rnatilgan bo'lsa)

HAJMNI KAMAYTIRISH:
  RandomForest -> n_estimators ni kamaytiring yoki
                  HistGradientBoosting ga o'ting
  KNN -> butun o'quv to'plamini saqlaydi (juda katta)
  SVC -> support vektorlar soni
  TfidfVectorizer -> lug'at hajmi (max_features)

mmap_mode="r": katta massivlarni xotiraga to'liq o'qimasdan

KNN va SVC o'quv ma'lumotining bir qismini saqlaydi — ularning fayl hajmi ma'lumot hajmiga proporsional.

2.6. Xavfsizlik

text
PICKLE IXTIYORIY KOD BAJARADI:
  ishonchsiz manbadan model yuklash = kod bajarishga ruxsat berish

QOIDALAR:
  1. Faqat O'ZINGIZ yaratgan yoki ishonchli manbadagi fayllarni yuklang
  2. Fayl yaxlitligini tekshiring (SHA-256 xesh)
  3. Ichki tarmoqda saqlang, ommaviy URL dan yuklamang

MUQOBILLAR (ishonchsiz muhit uchun):
  skops    - sklearn uchun xavfsizroq format (ixtiyoriy kod bajarmaydi)
  ONNX     - tildan mustaqil, faqat hisoblash grafi
  o'z formatingiz - koeffitsiyentlarni JSON da saqlash (sodda modellar)

Ishonchsiz pickle faylini yuklash — kod bajarishga ruxsat berish bilan barobar.

2.7. Tuzoqlar

Asosiy tuzoqlar: faqat modelni saqlash (metama'lumotsiz); versiyalarni yozmaslik; lambda ishlatish; o'z sinfini __main__ da e'lon qilish; yuklashdan keyin tekshirmaslik; ustunlar tartibini saqlamaslik; ishonchsiz faylni yuklash; siqishni umuman ishlatmaslik yoki compress=9 ni har doim qo'yish.

2.8. Paket, yolg'iz model emas

Saqlashning to'g'ri birligi — model emas, paket: butun Pipeline, kirish ustunlari va ularning tartibi, versiyalar, metrikalar va nazorat namunasi. Yuklashdan keyin esa doim tekshiring: ustunlar mosmi, versiyalar mosmi, nazorat bashoratlari bir xilmi. Bu uch tekshiruv jim xatolarning katta qismini oldini oladi.


3. Tez ma'lumotnoma

python
import sys
import joblib
import numpy as np
import sklearn

paket = {"quvur": quvur, "belgilar": list(X.columns),
         "metrika": {...},
         "versiyalar": {"sklearn": sklearn.__version__,
                        "numpy": np.__version__,
                        "python": sys.version.split()[0]},
         "nazorat": {"X": X_naz, "p": quvur.predict_proba(X_naz)[:, 1]}}
joblib.dump(paket, "model.joblib", compress=3)

# yuklash va TEKSHIRISH
p = joblib.load("model.joblib")
assert list(yangi.columns) == p["belgilar"]
assert sklearn.__version__ == p["versiyalar"]["sklearn"]
assert np.allclose(p["quvur"].predict_proba(p["nazorat"]["X"])[:, 1],
                   p["nazorat"]["p"])
QOIDA: paket saqla · versiyalarni yoz · nazorat namunasi ·
       lambda yozma · ishonchsiz fayl yuklama

Model saqlash xulosasi

joblib: sklearn uchun afzal, compress=3
Saqlab bo'lmaydi: lambda, mahalliy funksiya, ochiq fayl
pickle sinf KODINI saqlamaydi - modul kerak
Paket: quvur + belgilar + versiyalar + metrika + nazorat
Xavfsizlik: pickle ixtiyoriy kod bajaradi

4. Batafsil misollar

Misollar real numpy/pandas/sklearn bilan (Python 3.14).

Misol 1 — joblib va pickle, hajm va siqish

python
"""Saqlash usullari va hajm (real numpy/sklearn/joblib)."""

import io
import pickle
import shutil
import tempfile
from pathlib import Path

import joblib
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import (HistGradientBoostingClassifier,
                              RandomForestClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC


def hajm_kb(yol: Path) -> float:
    return yol.stat().st_size / 1024


def main() -> None:
    X, y = make_classification(n_samples=4000, n_features=20,
                               n_informative=8, flip_y=0.15, random_state=0)
    papka = Path(tempfile.mkdtemp(prefix="sk_saqlash_"))
    try:
        modellar = {
            "LogisticRegression": make_pipeline(
                StandardScaler(), LogisticRegression(max_iter=2000)),
            "RandomForest(200)": RandomForestClassifier(
                n_estimators=200, random_state=0, n_jobs=1),
            "HistGradientBoosting": HistGradientBoostingClassifier(
                max_iter=200, early_stopping=False, random_state=0),
            "KNN": make_pipeline(StandardScaler(),
                                 KNeighborsClassifier(n_neighbors=15)),
            "SVC": make_pipeline(StandardScaler(), SVC(random_state=0)),
        }

        print("=== 1. Model hajmlari (compress=3) ===")
        print(f"  {'model':<24} {'hajm (KB)':>11}")
        for nom, m in modellar.items():
            m.fit(X, y)
            yol = papka / f"{nom}.joblib"
            joblib.dump(m, yol, compress=3)
            print(f"  {nom:<24} {hajm_kb(yol):>11.1f}")

        print("\n=== 2. Siqish darajasi ===")
        rf = modellar["RandomForest(200)"]
        print(f"  {'compress':>18} {'hajm (KB)':>11} {'nisbat':>8}")
        asos = None
        for siqish in [0, 3, 6, 9, ("zlib", 3)]:
            yol = papka / f"rf_{siqish}.joblib"
            joblib.dump(rf, yol, compress=siqish)
            kb = hajm_kb(yol)
            asos = asos or kb
            print(f"  {str(siqish):>18} {kb:>11.1f} {asos / kb:>7.1f}x")

        print("\n=== 3. joblib va pickle taqqoslash ===")
        yol_j = papka / "rf.joblib"
        joblib.dump(rf, yol_j, compress=0)
        buf = io.BytesIO()
        pickle.dump(rf, buf, protocol=pickle.HIGHEST_PROTOCOL)
        print(f"  {'usul':<20} {'hajm (KB)':>11}")
        print(f"  {'joblib (compress=0)':<20} {hajm_kb(yol_j):>11.1f}")
        print(f"  {'pickle':<20} {buf.tell() / 1024:>11.1f}")

        print("\n=== 4. Yuklash va tenglik ===")
        yuklangan = joblib.load(papka / "LogisticRegression.joblib")
        asl = modellar["LogisticRegression"]
        p1 = asl.predict_proba(X[:200])[:, 1]
        p2 = yuklangan.predict_proba(X[:200])[:, 1]
        print(f"  bashoratlar aynan bir xil: {np.array_equal(p1, p2)}")
        print(f"  maksimal farq: {np.abs(p1 - p2).max():.2e}")
        print(f"  parametrlar bir xil: "
              f"{asl.get_params().keys() == yuklangan.get_params().keys()}")
        print("  ⭐ KNN va SVC o'quv ma'lumotining bir qismini saqlaydi")
    finally:
        shutil.rmtree(papka, ignore_errors=True)


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Model hajmlari (compress=3) ===
  model                      hajm (KB)
  LogisticRegression               1.4
  RandomForest(200)             3028.9
  HistGradientBoosting           345.6
  KNN                            603.6
  SVC                            279.8

=== 2. Siqish darajasi ===
            compress   hajm (KB)   nisbat
                   0     13163.4     1.0x
                   3      3028.9     4.3x
                   6      2679.9     4.9x
                   9      2612.4     5.0x
         ('zlib', 3)      3028.9     4.3x

=== 3. joblib va pickle taqqoslash ===
  usul                   hajm (KB)
  joblib (compress=0)      13163.4
  pickle                   13144.8

=== 4. Yuklash va tenglik ===
  bashoratlar aynan bir xil: True
  maksimal farq: 0.00e+00
  parametrlar bir xil: True
  ⭐ KNN va SVC o'quv ma'lumotining bir qismini saqlaydi

Nima ko'rsatdi: 2.1, 2.5-bo'limlar.

Misol 2 — Nima saqlab bo'lmaydi

python
"""Pickle qilinmaydigan obyektlar (real sklearn/pickle)."""

import io
import pickle

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler


def log_modul(X):
    """Modul darajasidagi funksiya - saqlanadi."""
    return np.log1p(np.abs(np.asarray(X, dtype=float)))


def quvur_yasa_ichki():
    """Ichki funksiya bilan quvur - SAQLANMAYDI."""

    def log_ichki(X):
        return np.log1p(np.abs(np.asarray(X, dtype=float)))

    return make_pipeline(FunctionTransformer(log_ichki), StandardScaler())


def saqlanadimi(obyekt) -> str:
    try:
        buf = io.BytesIO()
        pickle.dump(obyekt, buf, protocol=pickle.HIGHEST_PROTOCOL)
        return f"saqlandi ({buf.tell()} bayt)"
    except Exception as xato:
        return f"{type(xato).__name__}: {str(xato).splitlines()[0][:52]}"


def main() -> None:
    rng = np.random.default_rng(0)
    X = rng.gamma(2, 5, (200, 3))

    print("=== 1. Uch xil funksiya ===")
    variantlar = {
        "modul funksiyasi": make_pipeline(
            FunctionTransformer(log_modul), StandardScaler()),
        "lambda": make_pipeline(
            FunctionTransformer(lambda a: np.log1p(np.abs(a))),
            StandardScaler()),
        "ichki funksiya": quvur_yasa_ichki(),
    }
    print(f"  {'variant':<20} natija")
    for nom, q in variantlar.items():
        q.fit(X)
        print(f"  {nom:<20} {saqlanadimi(q)}")

    print("\n=== 2. Boshqa saqlanmaydigan obyektlar ===")
    ochiq_fayl = io.StringIO("matn")
    generator = (i for i in range(5))
    sinashlar = {
        "ochiq fayl obyekti": ochiq_fayl,
        "generator": generator,
        "lambda": (lambda x: x),
        "oddiy ro'yxat": [1, 2, 3],
        "numpy massiv": np.arange(5),
        "lug'at": {"a": 1, "b": [2, 3]},
    }
    print(f"  {'obyekt':<22} natija")
    for nom, ob in sinashlar.items():
        print(f"  {nom:<22} {saqlanadimi(ob)}")

    print("\n=== 3. Nima uchun lambda saqlanmaydi ===")
    print("  pickle funksiyani MODUL YO'LI bilan saqlaydi:")
    print(f"    log_modul.__module__ = {log_modul.__module__}")
    print(f"    log_modul.__qualname__ = {log_modul.__qualname__}")
    lam = lambda x: x
    print(f"    lambda.__qualname__ = {lam.__qualname__}")
    print("  <lambda> nomi bilan modulda qidiriladi va topilmaydi")

    print("\n=== 4. To'g'ri yechim ===")
    toGri = make_pipeline(FunctionTransformer(log_modul),
                          StandardScaler()).fit(X)
    buf = io.BytesIO()
    pickle.dump(toGri, buf, protocol=pickle.HIGHEST_PROTOCOL)
    buf.seek(0)
    yuklangan = pickle.load(buf)
    print(f"  saqlandi va yuklandi: {type(yuklangan).__name__}")
    print(f"  bashoratlar bir xil: "
          f"{np.allclose(toGri.transform(X), yuklangan.transform(X))}")
    print("  ⭐ Modul darajasidagi funksiya - yagona ishonchli yo'l")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Uch xil funksiya ===
  variant              natija
  modul funksiyasi     saqlandi (846 bayt)
  lambda               PicklingError: Can't pickle local object <function main.<locals>.<l
  ichki funksiya       PicklingError: Can't pickle local object <function quvur_yasa_ichki

=== 2. Boshqa saqlanmaydigan obyektlar ===
  obyekt                 natija
  ochiq fayl obyekti     saqlandi (52 bayt)
  generator              TypeError: cannot pickle 'generator' object
  lambda                 PicklingError: Can't pickle local object <function main.<locals>.<l
  oddiy ro'yxat          saqlandi (22 bayt)
  numpy massiv           saqlandi (167 bayt)
  lug'at                 saqlandi (34 bayt)

=== 3. Nima uchun lambda saqlanmaydi ===
  pickle funksiyani MODUL YO'LI bilan saqlaydi:
    log_modul.__module__ = __main__
    log_modul.__qualname__ = log_modul
    lambda.__qualname__ = main.<locals>.<lambda>
  <lambda> nomi bilan modulda qidiriladi va topilmaydi

=== 4. To'g'ri yechim ===
  saqlandi va yuklandi: Pipeline
  bashoratlar bir xil: True
  ⭐ Modul darajasidagi funksiya - yagona ishonchli yo'l

Nima ko'rsatdi: 2.2-bo'lim.

Misol 3 — Paket va yuklashdagi tekshiruvlar

python
"""Model + metama'lumot + nazorat namunasi (real pandas/sklearn/joblib)."""

import shutil
import sys
import tempfile
from pathlib import Path

import joblib
import numpy as np
import pandas as pd
import sklearn
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def yarat(n: int = 1500, seed: int = 0):
    rng = np.random.default_rng(seed)
    df = pd.DataFrame({
        "yosh": rng.integers(18, 70, n).astype(float),
        "daromad": rng.lognormal(10, 0.5, n),
        "ball": rng.normal(600, 80, n),
        "hudud": rng.choice(["shimol", "janub", "markaz"], n),
    })
    kuch = (-3.0 + 0.02 * df["yosh"] + 0.004 * df["ball"]
            + 0.8 * (df["hudud"] == "markaz"))
    y = (rng.random(n) < 1 / (1 + np.exp(-kuch))).astype(int)
    return df, y


def paket_yasa(quvur, df, y, cv_ball) -> dict:
    nazorat = df.head(40)
    return {
        "quvur": quvur,
        "belgilar": list(df.columns),
        "metrika": {"cv_auc": round(float(cv_ball), 4)},
        "versiyalar": {"sklearn": sklearn.__version__,
                       "numpy": np.__version__,
                       "pandas": pd.__version__,
                       "python": sys.version.split()[0]},
        "sana": "2026-09-21",
        "nazorat": {"X": nazorat,
                    "p": quvur.predict_proba(nazorat)[:, 1]},
    }


def paketni_tekshir(paket: dict, yangi: pd.DataFrame) -> list:
    natijalar = []
    ustunlar_mos = list(yangi.columns) == paket["belgilar"]
    natijalar.append(("ustunlar va tartibi", ustunlar_mos))
    sk_mos = sklearn.__version__ == paket["versiyalar"]["sklearn"]
    natijalar.append(("sklearn versiyasi", sk_mos))
    p = paket["quvur"].predict_proba(paket["nazorat"]["X"])[:, 1]
    nazorat_mos = bool(np.allclose(p, paket["nazorat"]["p"]))
    natijalar.append(("nazorat namunasi", nazorat_mos))
    return natijalar


def main() -> None:
    df, y = yarat()
    tayyor = ColumnTransformer([
        ("son", StandardScaler(),
         make_column_selector(dtype_include=np.number)),
        ("kat", OneHotEncoder(handle_unknown="ignore", sparse_output=False),
         make_column_selector(dtype_include="str"))])
    quvur = Pipeline([("t", tayyor),
                      ("m", HistGradientBoostingClassifier(
                          max_iter=200, early_stopping=False,
                          random_state=0))])
    cv = StratifiedKFold(4, shuffle=True, random_state=0)
    cv_ball = cross_val_score(quvur, df, y, cv=cv, scoring="roc_auc").mean()
    quvur.fit(df, y)

    papka = Path(tempfile.mkdtemp(prefix="sk_paket_"))
    try:
        print("=== 1. Paket tuzilishi ===")
        paket = paket_yasa(quvur, df, y, cv_ball)
        print(f"  kalitlar: {sorted(paket)}")
        print(f"  belgilar: {paket['belgilar']}")
        print(f"  metrika: {paket['metrika']}")
        print(f"  versiyalar: {paket['versiyalar']}")
        print(f"  nazorat namunasi: {paket['nazorat']['X'].shape}")

        yol = papka / "model.joblib"
        joblib.dump(paket, yol, compress=3)
        print(f"  fayl hajmi: {yol.stat().st_size / 1024:.1f} KB")

        print("\n=== 2. Yuklash va tekshiruvlar ===")
        yuklangan = joblib.load(yol)
        print(f"  {'tekshiruv':<24} {'natija':>8}")
        for nom, holat in paketni_tekshir(yuklangan, df):
            print(f"  {nom:<24} {'OK' if holat else 'XATO':>8}")

        print("\n=== 3. Ustun tartibi buzilganda ===")
        buzilgan = df[["daromad", "yosh", "ball", "hudud"]]
        print(f"  kutilgan tartib: {yuklangan['belgilar']}")
        print(f"  kelgan tartib: {list(buzilgan.columns)}")
        for nom, holat in paketni_tekshir(yuklangan, buzilgan):
            print(f"  {nom:<24} {'OK' if holat else 'XATO':>8}")
        p_asl = yuklangan["quvur"].predict_proba(df.head(5))[:, 1]
        p_buzilgan = yuklangan["quvur"].predict_proba(buzilgan.head(5))[:, 1]
        print(f"  asl bashorat: {np.round(p_asl, 4).tolist()}")
        print(f"  buzilgan tartibda: {np.round(p_buzilgan, 4).tolist()}")
        print(f"  farq bormi: {not np.allclose(p_asl, p_buzilgan)}")
        print("  (ColumnTransformer nom bilan ishlagani uchun bu holda")
        print("   natija bir xil; indeks bilan ishlaganda BUZILARDI)")

        print("\n=== 4. Soxta versiya nomuvofiqligi ===")
        soxta = dict(yuklangan)
        soxta["versiyalar"] = dict(yuklangan["versiyalar"])
        soxta["versiyalar"]["sklearn"] = "1.2.0"
        for nom, holat in paketni_tekshir(soxta, df):
            print(f"  {nom:<24} {'OK' if holat else 'XATO':>8}")
        print("  versiya nomuvofiqligi DARHOL aniqlandi")
        print("  ⭐ Uch tekshiruv jim xatolarning katta qismini to'sadi")
    finally:
        shutil.rmtree(papka, ignore_errors=True)


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Paket tuzilishi ===
  kalitlar: ['belgilar', 'metrika', 'nazorat', 'quvur', 'sana', 'versiyalar']
  belgilar: ['yosh', 'daromad', 'ball', 'hudud']
  metrika: {'cv_auc': 0.5551}
  versiyalar: {'sklearn': '1.9.1', 'numpy': '2.5.3', 'pandas': '3.0.6', 'python': '3.14.5'}
  nazorat namunasi: (40, 4)
  fayl hajmi: 275.5 KB

=== 2. Yuklash va tekshiruvlar ===
  tekshiruv                  natija
  ustunlar va tartibi            OK
  sklearn versiyasi              OK
  nazorat namunasi               OK

=== 3. Ustun tartibi buzilganda ===
  kutilgan tartib: ['yosh', 'daromad', 'ball', 'hudud']
  kelgan tartib: ['daromad', 'yosh', 'ball', 'hudud']
  ustunlar va tartibi          XATO
  sklearn versiyasi              OK
  nazorat namunasi               OK
  asl bashorat: [0.1453, 0.726, 0.0495, 0.044, 0.079]
  buzilgan tartibda: [0.1453, 0.726, 0.0495, 0.044, 0.079]
  farq bormi: False
  (ColumnTransformer nom bilan ishlagani uchun bu holda
   natija bir xil; indeks bilan ishlaganda BUZILARDI)

=== 4. Soxta versiya nomuvofiqligi ===
  ustunlar va tartibi            OK
  sklearn versiyasi            XATO
  nazorat namunasi               OK
  versiya nomuvofiqligi DARHOL aniqlandi
  ⭐ Uch tekshiruv jim xatolarning katta qismini to'sadi

Nima ko'rsatdi: 2.3, 2.4-bo'limlar.

Misol 4 — Yaxlitlik va xavfsizlik

python
"""Xesh bilan tekshirish va xavfsizlik (real hashlib/joblib/sklearn)."""

import hashlib
import shutil
import tempfile
from pathlib import Path

import joblib
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def fayl_xeshi(yol: Path) -> str:
    h = hashlib.sha256()
    with open(yol, "rb") as f:
        for bolak in iter(lambda: f.read(65536), b""):
            h.update(bolak)
    return h.hexdigest()


def main() -> None:
    X, y = make_classification(n_samples=1000, n_features=10,
                               n_informative=5, random_state=0)
    model = make_pipeline(StandardScaler(),
                          LogisticRegression(max_iter=2000)).fit(X, y)
    papka = Path(tempfile.mkdtemp(prefix="sk_xavfsiz_"))
    try:
        yol = papka / "model.joblib"
        joblib.dump(model, yol, compress=3)

        print("=== 1. Fayl xeshi ===")
        xesh = fayl_xeshi(yol)
        print(f"  SHA-256: {xesh[:32]}...")
        print(f"  hajm: {yol.stat().st_size} bayt")

        print("\n=== 2. Xesh takrorlanadimi ===")
        yol2 = papka / "model2.joblib"
        joblib.dump(model, yol2, compress=3)
        xesh2 = fayl_xeshi(yol2)
        print(f"  ikkinchi saqlash xeshi: {xesh2[:32]}...")
        print(f"  bir xil: {xesh == xesh2}")
        print("  (bir xil bo'lsa - faylni yaxlitlik uchun tekshirsa bo'ladi)")

        print("\n=== 3. Fayl o'zgartirilganda ===")
        buzilgan = papka / "buzilgan.joblib"
        baytlar = bytearray(yol.read_bytes())
        baytlar[len(baytlar) // 2] ^= 0xFF          # bitta baytni o'zgartirish
        buzilgan.write_bytes(bytes(baytlar))
        print(f"  buzilgan fayl xeshi: {fayl_xeshi(buzilgan)[:32]}...")
        print(f"  asl xesh bilan mos: {fayl_xeshi(buzilgan) == xesh}")
        try:
            joblib.load(buzilgan)
            print("  yuklandi (xavfli - xato sezilmadi)")
        except Exception as xato:
            print(f"  yuklashda xato: {type(xato).__name__}")

        print("\n=== 4. Xavfsiz yuklash tartibi ===")
        kutilgan_xesh = xesh
        qadamlar = []
        qadamlar.append(("manba ishonchli", True))
        qadamlar.append(("xesh mos", fayl_xeshi(yol) == kutilgan_xesh))
        yuklangan = joblib.load(yol)
        qadamlar.append(("yuklandi", yuklangan is not None))
        p1 = model.predict_proba(X[:50])[:, 1]
        p2 = yuklangan.predict_proba(X[:50])[:, 1]
        qadamlar.append(("nazorat bashorati mos", bool(np.allclose(p1, p2))))
        print(f"  {'qadam':<26} {'natija':>8}")
        for nom, holat in qadamlar:
            print(f"  {nom:<26} {'OK' if holat else 'XATO':>8}")
        print("  ⭐ pickle ixtiyoriy kod bajaradi - manbaga ishoning")
    finally:
        shutil.rmtree(papka, ignore_errors=True)


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Fayl xeshi ===
  SHA-256: 40ffb9371c663ca8d9df28a49df1cfdb...
  hajm: 1146 bayt

=== 2. Xesh takrorlanadimi ===
  ikkinchi saqlash xeshi: 40ffb9371c663ca8d9df28a49df1cfdb...
  bir xil: True
  (bir xil bo'lsa - faylni yaxlitlik uchun tekshirsa bo'ladi)

=== 3. Fayl o'zgartirilganda ===
  buzilgan fayl xeshi: 55af8312712ac2be8c3ef41adde9356d...
  asl xesh bilan mos: False
  yuklashda xato: error

=== 4. Xavfsiz yuklash tartibi ===
  qadam                        natija
  manba ishonchli                  OK
  xesh mos                         OK
  yuklandi                         OK
  nazorat bashorati mos            OK
  ⭐ pickle ixtiyoriy kod bajaradi - manbaga ishoning

Nima ko'rsatdi: 2.6-bo'lim.


5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"joblib.dump yetarli" Paket kerak
"pickle sinf kodini saqlaydi" Faqat modul yo'lini
"Versiyalar mos kelishi kafolatlangan" Kafolat yo'q
"Yuklandi — demak to'g'ri" Jim siljish bo'lishi mumkin
"lambda qulay" Saqlanmaydi
"compress=9 har doim yaxshi" Sekin, foyda kichik
"Model fayli kichik bo'ladi" KNN/SVC katta bo'ladi
"pickle xavfsiz" Ixtiyoriy kod bajaradi

6. Keng tarqalgan xatolar va yechimlari

1. Yolg'iz modelni saqlash

python
joblib.dump(model, "m.joblib")                                   # ⚠️
joblib.dump({"quvur": model, "belgilar": ..., "versiyalar": ...}) # ✅

2. lambda ishlatish

python
FunctionTransformer(lambda X: np.log1p(X))                       # ⚠️
def log_t(X): return np.log1p(X)                                 # ✅

3. Sinfni __main__ da e'lon qilish

python
# skript ichida class MeningT(...)  -> yuklashda topilmaydi      # ⚠️
# alohida modul: from loyiha.transformerlar import MeningT       # ✅

4. Yuklashdan keyin tekshirmaslik

python
m = joblib.load("m.joblib"); m.predict(yangi)                    # ⚠️
assert list(yangi.columns) == paket["belgilar"]                  # ✅

5. Versiyalarni yozmaslik

python
paket = {"quvur": q}                                             # ⚠️
paket = {"quvur": q, "versiyalar": {"sklearn": sklearn.__version__}}  # ✅

6. Ishonchsiz fayl

python
joblib.load(urlopen("http://noma_lum/model.joblib"))             # ⚠️
# faqat ichki, xeshi tekshirilgan fayllar                        # ✅

7. Ustunlar tartibini saqlamaslik

python
paket = {"quvur": q}                                             # ⚠️
paket = {"quvur": q, "belgilar": list(X.columns)}                # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 15.13-dars (o'tilgan): Ishlab chiqarish
  • 17.10-dars (o'tilgan): Paket
  • 19.5-dars (o'tilgan): lambda muammosi
  • 19.7-dars: Takrorlanuvchanlik
  • 29-qism: MLOps va deploy

8. Eng yaxshi amaliyotlar

  1. Paket saqlang, yolg'iz model emas.

  2. Versiyalarni yozing.

  3. Nazorat namunasini qo'shing.

  4. Yuklashdan keyin tekshiring.

  5. Modul darajasidagi funksiya.

  6. O'z sinflaringiz alohida modulda.

  7. compress=3 dan boshlang.

  8. Faqat ishonchli faylni yuklang.


9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # sklearn uchun qaysi vosita?
2.  # joblib va pickle farqi?
3.  # lambda nima uchun saqlanmaydi?
4.  # pickle funksiyani qanday saqlaydi?
5.  # o'z sinfingiz qayerda bo'lishi kerak?
6.  # versiya mosligi kafolatlanganmi?
7.  # eng yomon versiya muammosi?
8.  # nazorat namunasi nima uchun?
9.  # paketda nima bo'lishi kerak?
10. # compress qanday tanlanadi?
11. # qaysi modellar katta bo'ladi?
12. # pickle xavfsizmi?
Javoblar
  1. joblib
  2. joblib numpy massivlarini samaraliroq
  3. Modul yo'li bilan topilmaydi
  4. Modul yo'li va nomi bilan
  5. Alohida, o'rnatiladigan modulda
  6. Yo'q
  7. Yuklanadi, lekin boshqacha ishlaydi
  8. Jim siljishni aniqlash uchun
  9. Quvur, belgilar, versiyalar, metrika, nazorat
  10. 3 — muvozanat
  11. KNN, SVC, katta RF
  12. Yo'q, ixtiyoriy kod bajaradi

Vazifa 2: Xatolarni tuzating

python
1.  joblib.dump(model, "m.joblib")

2.  FunctionTransformer(lambda X: np.log1p(X))

3.  m = joblib.load("m.joblib"); m.predict(yangi)

4.  paket = {"quvur": q}

5.  joblib.load(urlopen("http://noma_lum/model.joblib"))
Javoblar
python
1.  joblib.dump({"quvur": model, "belgilar": ..., "versiyalar": ...}, ...)

2.  def log_t(X): return np.log1p(X)

3.  assert list(yangi.columns) == paket["belgilar"]

4.  paket = {"quvur": q, "belgilar": ..., "versiyalar": ..., "nazorat": ...}

5.  # faqat ichki, xeshi tekshirilgan fayllar

Vazifa 3: Hajm

Modellang:

  1. Model hajmlari
  2. Siqish
  3. joblib/pickle
  4. Tenglik

Vazifa 4: Saqlanmaydiganlar

Modellang:

  1. Uch funksiya
  2. Boshqa obyektlar
  3. Sabab
  4. Yechim

Vazifa 5: Paket

Modellang:

  1. Tuzilish
  2. Tekshiruvlar
  3. Tartib buzilishi
  4. Versiya

Vazifa 6: Xavfsizlik

Modellang:

  1. Xesh
  2. Takrorlanuvchanlik
  3. Buzilgan fayl
  4. Yuklash tartibi

Vazifa 7: O'ylash

Model olti oy oldin saqlangan. Uni yuklaganingizda InconsistentVersionWarning chiqdi, lekin model ishladi va bashoratlar oqilona ko'rinadi. Nima qilasiz?

Javob

Qisqa javob: "Oqilona ko'rinadi" — yetarli emas. Nazorat namunasi bilan tekshiring; nazorat namunasi bo'lmasa, modelni qayta o'rgating.

1. Nima uchun "oqilona ko'rinish" yetarli emas

Versiya siljishi odatda kichik bo'ladi: bashoratlar 0.01-0.05 ga o'zgaradi. Bunday siljish ko'zga tashlanmaydi, lekin:

  • qaror chegarasi atrofidagi holatlar boshqa tomonga o'tadi
  • reyting tartibi o'zgaradi (top-N ro'yxati boshqacha bo'ladi)
  • kalibrlash buziladi

2. Birinchi qadam: nazorat namunasi

python
paket = joblib.load("model.joblib")
p_yangi = paket["quvur"].predict_proba(paket["nazorat"]["X"])[:, 1]
p_eski = paket["nazorat"]["p"]
farq = np.abs(p_yangi - p_eski)
print(f"maksimal farq: {farq.max():.6f}, o'rtacha: {farq.mean():.6f}")
  • farq.max() < 1e-10 → model aynan bir xil, ogohlantirishni e'tiborsiz qoldirsa bo'ladi
  • farq.max() > 1e-6 → xatti-harakat o'zgargan, ishlatmang

3. Nazorat namunasi yo'q bo'lsa

Uch variant, tartib bo'yicha:

Variant Ishonchlilik Narx
Eski sklearn versiyasini o'rnatib qayta baholash Yuqori Past
Modelni qayta o'rgatish Eng yuqori O'rta
Ogohlantirishni e'tiborsiz qoldirish Past Nol
bash
# eski versiyada tekshirish
pip install scikit-learn==1.2.0
python -c "import joblib; m = joblib.load('model.joblib'); print(m.predict_proba(X)[:5])"

Ikki versiyadagi natijalarni solishtiring.

4. Uzoq muddatli yechim

  1. Har paketda nazorat namunasi (30-50 qator + bashoratlar)
  2. requirements.txt da versiyani qulflash: scikit-learn==1.9.1
  3. Ishlab chiqarish muhitini konteyner bilan qotirish
  4. Modelni davriy qayta o'rgatish (drift uchun ham foydali)

5. Qachon qayta o'rgatish shart

  • Nazorat namunasi yo'q
  • Farq 1e-6 dan katta
  • Model qaror chegarasi bilan ishlaydi (klassifikatsiya)
  • Model kalibrlangan ehtimollik beradi

6. Xulosa

  1. Nazorat namunasi bilan tekshiring
  2. Farq bo'lsa ishlatmang
  3. Namuna yo'q bo'lsa — eski versiyada solishtiring yoki qayta o'rgating
  4. Kelajak uchun: nazorat namunasi + qulflangan versiyalar

Nimani mustahkamlaydi: 2.3, 2.4-bo'limlar.


Xulosa

Bu darsda model saqlashni o'rgandik.

Eng muhim uch fikr:

  1. Saqlashning birligi — paket, yolg'iz model emas. Paketga butun Pipeline, kirish ustunlari va tartibi, versiyalar (sklearn, numpy, pandas, Python), metrikalar va nazorat namunasi kiradi. Model yolg'iz saqlansa, olti oydan keyin "bu nima, qanday ishlatiladi, ishonsa bo'ladimi?" degan savollarga javob bo'lmaydi.

  2. pickle sinf kodini saqlamaydi — faqat modul yo'lini. Shuning uchun lambda va ichki funksiyalar saqlanmaydi, o'z transformerlaringiz esa alohida, o'rnatiladigan modulda bo'lishi kerak. FunctionTransformer ga har doim modul darajasidagi funksiya bering.

  3. Yuklashdan keyin uch tekshiruv. Ustunlar mosmi, versiyalar mosmi, nazorat namunasidagi bashoratlar bir xilmi. Versiya nomuvofiqligining eng xavfli ko'rinishi — model yuklanib, predict ishlab, natijalar jim siljishi; uni faqat nazorat namunasi aniqlaydi. Va pickle ixtiyoriy kod bajaradi, shuning uchun faqat ishonchli fayllarni yuklang.

Keyingi darsda takrorlanuvchanlik ni ko'ramiz: random_state, global seedlar, oqim soni ta'siri, versiyalarni qulflash va bir xil natijani ikki marta olish.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
19.6-dars: Model saqlash — IlmHamroh