27.6-dars: Modelni paketlash

mijoz_ketishi-1.2.0/
  model.joblib       Pipeline: oldindan ishlov berish + model (BITTA obyekt)
  konfig.json        chegara 0.35-bob, sinf nomlari, biznes sozlamalari
  sxema.json         kirish ustunlari: nom, tur, oraliq, kategoriyalar, majburiylik
  metadata.json      versiya, python/sklearn/numpy versiyalari, ma'lumot xeshi,
                     metrikalar, run_id 27.4-bob, model kartasi havolasi 26.10-bob
  nazorat.json       nazorat namunalari: kirish + KUTILGAN chiqish
  MANIFEST.json      har faylning sha256 xeshi

YUKLASH TARTIBI:
  1. MANIFEST xeshi registrdagiga mosmi?        27.5-bob
  2. har fayl xeshi MANIFEST ga mosmi?          (buzilish / almashtirish)
  3. muhit versiyalari metadata ga mosmi?        2.4-bob
  4. model yuklanadi                             (faqat 1-3 dan KEYIN)
  5. nazorat namunalari: bashorat aynan mosmi?   (smoke test)
  6. tayyor -> Predictor

FORMAT            NIMA SAQLANADI           XAVFSIZMI       KO'CHUVCHANLIK
pickle            ixtiyoriy Python obyekt  YO'Q (kod!)     faqat Python, versiyaga sezgir
joblib            pickle + numpy uchun     YO'Q (pickle)   xuddi pickle
                  samarali, siqish
JSON parametrlar  sonlar (koef, o'rtacha)  HA (ma'lumot)   istalgan til; faqat oddiy modellar
torch state_dict  tenzorlar lug'ati        HA, agar         arxitektura kodi alohida kerak
                                           weights_only=True
safetensors       faqat tenzorlar          HA              tez, xotiraga to'g'ridan-to'g'ri
ONNX              hisob grafi + vaznlar    HA (graf)        tillar va runtime lar arasi
skops (sklearn)   sklearn obyektlari,      qisman           ruxsat etilgan turlar ro'yxati
                  cheklangan yuklash

class YashirinYuk:
    def __reduce__(self):
        return (funksiya, (argumentlar,))

pickle.dumps(YashirinYuk())  ->  baytlar: "import funksiya; funksiya(argumentlar)"
pickle.loads(baytlar)        ->  funksiya(argumentlar) CHAQIRILADI

MUHIM:
  - obyektning hech bir metodini chaqirish shart emas - YUKLASHNING O'ZI yetarli
  - funksiya - istalgan import qilinadigan narsa (builtins, os, subprocess ...)
  - joblib.load, torch.load(weights_only=False), pandas.read_pickle - hammasi pickle
  - fayl kengaytmasi (.pkl, .joblib, .pt, .bin) hech narsani kafolatlamaydi

1. MANBA:      faqat o'z registringizdan 27.5-bob, xesh tekshiruvi bilan
2. SKANER:     pickletools bilan ichidagi import nomlarini BAJARMASDAN o'qish
               (sklearn.*, numpy.* - kutilgan; builtins.exec, os.system - XAVF)
3. CHEKLANGAN YUKLOVCHI:  pickle.Unpickler.find_class ni qayta yozish -
               faqat ruxsat etilgan modullar import qilinadi
4. FORMAT:     imkon bo'lsa - JSON / safetensors / state_dict + weights_only=True
5. IZOLYATSIYA: noma'lum faylni faqat alohida, huquqsiz muhitda (konteyner) ochish

MUAMMO:
  pickle klass NOMINI va uning ichki ATRIBUTLARINI saqlaydi
  sklearn 1.3 da saqlangan -> sklearn 1.9 da yuklangan:
    atribut nomi o'zgargan  -> AttributeError (yaxshi - darhol ko'rinadi)
    ichki mantiq o'zgargan  -> BOSHQA bashorat (yomon - jim)
  sklearn buni sezadi: InconsistentVersionWarning - lekin faqat OGOHLANTIRISH

YECHIM:
  saqlashda:  metadata.json ga python, numpy, sklearn versiyalari
  yuklashda:  siyosat bo'yicha tekshirish, mos kelmasa - RAD
    sklearn:  major.minor aynan mos      (1.9.x == 1.9.x)
    numpy:    major mos                  (2.x == 2.x)
    python:   major.minor mos            (3.14.x == 3.14.x)
  qo'shimcha: nazorat namunalari - bashorat aynan mosmi

ENG YAXSHISI: ishlab chiqarish muhiti o'rgatish muhitining AYNAN nusxasi
              (requirements.lock - 27.2, Docker - 27.9)

SAQLASH:  torch.save({"state_dict": model.state_dict(),
                      "meta": {"kirish": 4, "yashirin": 16, "torch": "2.14"}}, yol)
YUKLASH:  paket = torch.load(yol, weights_only=True)
          model = Tarmoq(**arxitektura)        # KOD - sizning repozitoriyangizda
          model.load_state_dict(paket["state_dict"])

weights_only=True:  faqat tenzorlar, oddiy turlar (dict, list, int, str ...)
                    boshqa har qanday global -> UnpicklingError
torch 2.6 dan boshlab bu sukut qiymati

torch model --torch.onnx.export--> model.onnx --onnxruntime (istalgan til)
sklearn Pipeline --skl2onnx------> model.onnx

AFZALLIKLAR:  Python siz ishlash, ko'pincha tezroq, pickle xavfi yo'q
KAMCHILIKLAR: har operator ham qo'llab-quvvatlanmaydi (ayniqsa maxsus
              oldindan ishlov berish), konvertatsiyadan keyin natijani
              ASL model bilan solishtirish SHART (nazorat namunalari!)

sxema.json:
  {"ustunlar": [
     {"nom": "oylar",    "tur": "son", "min": 0, "max": 600, "majburiy": true},
     {"nom": "tolov",    "tur": "son", "min": 0, "max": 10000, "majburiy": true},
     {"nom": "shikoyat", "tur": "son", "min": 0, "max": 100, "majburiy": false},
     {"nom": "tarif",    "tur": "kategoriya", "qiymatlar": [...],
      "nomalum": "rad"},
     {"nom": "hudud",    "tur": "kategoriya", "qiymatlar": [...],
      "nomalum": "ruxsat"}],
   "chiqish": {"nom": "ketish_ehtimoli", "oraliq": [0, 1]}}

TEKSHIRUVLAR:
  ustun bormi?  turi to'g'rimi?  oraliqdami?  bo'sh bo'lsa - ruxsatmi?
  noma'lum kategoriya: "rad" (tarif - mahsulotlar ro'yxati yopiq)
                       "ruxsat" + ogohlantirish (hudud - yangilari paydo bo'ladi)
  ortiqcha ustunlar: e'tiborsiz, lekin ro'yxati qaytariladi
  BARCHA xatolar birdaniga qaytariladi (birinchisida to'xtamaslik)

NAZORAT NAMUNALARI (smoke test):
  paketlashda: 5-20 ta xilma-xil kirish + model bergan AYNAN chiqish
  yuklashda:   shu kirishlar bilan bashorat -> kutilgan bilan solishtirish
  USHLAYDI:    boshqa model fayli, boshqa kutubxona versiyasi,
               boshqa oldindan ishlov berish, buzilgan konvertatsiya (ONNX)

XESHLAR (MANIFEST):
  USHLAYDI:    yo'lda buzilish, qo'lda tahrir (konfig chegarasi!), almashtirish
  MANIFEST ning O'ZI xeshi registrda 27.5-bob - aks holda almashtiruvchi
  MANIFEST ni ham yangilab qo'yishi mumkin

IKKALASI KERAK: xesh "fayl o'shami" deydi, nazorat "natija o'shami" deydi

joblib.dump(model, yol, compress=N)      N = 0 (siqishsiz) .. 9 (eng kuchli)
  compress=0:  eng katta fayl, eng tez yuklash
  compress=3:  odatda yaxshi murosa (sukut tavsiya)
  compress=9:  biroz kichikroq, lekin ancha sekin saqlash
  ("lzma", 3): ko'pincha eng kichik, sekinroq

QACHON MUHIM:
  konteyner obrazi hajmi 27.9-bob, sovuq start (serverless), tarmoqdan yuklash
  daraxt ansambllari (RandomForest) - katta, yaxshi siqiladi
  chiziqli modellar - kichik, siqish deyarli ahamiyatsiz

class Predictor:
    __init__(papka)          xesh + versiya + nazorat tekshiruvlari
    versiya                  "1.2.0"
    sxema                    kirish shartnomasi
    validatsiya(kirish)      -> jadval yoki SxemaXatosi (barcha xatolar)
    predict_proba(kirish)    -> ehtimollar
    predict(kirish)          -> sinflar (konfig dagi chegara bilan)

KIRISH: bitta dict | dict lar ro'yxati | DataFrame - natija bir xil
FOYDASI: API 27.7-bob, batch skript, noutbuk - HAMMASI bir xil kod orqali

import hashlib
import io
import json
import pickle

import joblib
import torch


class CheklanganYuklovchi(pickle.Unpickler):
    def find_class(self, modul, nom):
        if modul.split(".")[0] in {"sklearn", "numpy"}:
            return super().find_class(modul, nom)
        raise pickle.UnpicklingError(f"taqiqlangan: {modul}.{nom}")


def xavfsiz_yukla(baytlar):
    return CheklanganYuklovchi(io.BytesIO(baytlar)).load()


def fayl_xeshi(yol):
    return hashlib.sha256(yol.read_bytes()).hexdigest()


# saqlash
# joblib.dump(pipeline, papka / "model.joblib", compress=3)
# torch.save({"state_dict": net.state_dict(), "meta": meta}, papka / "model.pt")

# yuklash: avval xesh, keyin yuklash, keyin nazorat
# assert fayl_xeshi(papka / "model.joblib") == manifest["model.joblib"]
# model = joblib.load(papka / "model.joblib")
# paket = torch.load(papka / "model.pt", weights_only=True)

paket = Pipeline + konfig + sxema + metadata + nazorat + MANIFEST
pickle/joblib = kod bajarilishi -> faqat ishonchli manba + xesh
torch: state_dict + weights_only=True; ONNX: graf, runtime lar arasi
yuklash: registr xeshi -> fayl xeshlari -> versiyalar -> model -> nazorat
Predictor: versiya, sxema, validatsiya, predict, predict_proba

"""pickle yuklanganda kod bajaradi: namoyish, skaner, cheklangan yuklovchi, JSON."""

import io
import json
import pickle
import pickletools

import joblib
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

BELGILAR = []


def belgi_qoy(xabar):
    """Zararsiz 'yuk': faqat ro'yxatga yozadi va soxta model qaytaradi."""
    BELGILAR.append(xabar)
    return {"model": "soxta"}


class YashirinYuk:
    def __reduce__(self):
        # retsept: "yuklashda belgi_qoy(...) ni chaqir, natijasini qaytar"
        return (belgi_qoy, ("pickle.loads paytida chaqirildi",))


class ChopEtuvchi:
    def __reduce__(self):
        return (print, ("  [!] bu qator pickle.loads ichidan chop etildi",))


def globallar(baytlar):
    """pickle ichidagi import qilinadigan nomlar - kodni BAJARMASDAN."""
    nomlar, satrlar = [], []
    for op, arg, _ in pickletools.genops(baytlar):
        if op.name in ("SHORT_BINUNICODE", "BINUNICODE", "UNICODE"):
            satrlar.append(arg)
        elif op.name == "GLOBAL":
            nomlar.append(arg.replace(" ", "."))
        elif op.name == "STACK_GLOBAL":
            nomlar.append(f"{satrlar[-2]}.{satrlar[-1]}")
    return sorted(set(nomlar))


RUXSAT = {"sklearn", "numpy"}


class CheklanganYuklovchi(pickle.Unpickler):
    def find_class(self, modul, nom):
        if modul.split(".")[0] in RUXSAT:
            return super().find_class(modul, nom)
        raise pickle.UnpicklingError(f"taqiqlangan: {modul}.{nom}")


def xavfsiz_yukla(baytlar):
    return CheklanganYuklovchi(io.BytesIO(baytlar)).load()


def main() -> None:
    rng = np.random.default_rng(0)
    X = rng.normal(size=(500, 4))
    y = (X[:, 0] - 0.5 * X[:, 1] + rng.normal(0, 0.5, 500) > 0).astype(int)
    model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)

    print("=== 1. Oddiy model: saqlash va yuklash ===")
    baytlar = pickle.dumps(model)
    qayta = pickle.loads(baytlar)
    farq = np.abs(qayta.predict_proba(X) - model.predict_proba(X)).max()
    print(f"  pickle hajmi: {len(baytlar)} bayt, bashorat farqi: {farq}")
    print(f"  belgilar: {BELGILAR}")

    print("\n=== 2. Yashirin yuk: faqat YUKLASH yetarli ===")
    yuk = pickle.dumps(YashirinYuk())
    natija = pickle.loads(yuk)
    print(f"  pickle.loads qaytardi: {natija}")
    print(f"  belgilar: {BELGILAR}")
    pickle.loads(pickle.dumps(ChopEtuvchi()))
    bufer = io.BytesIO()
    joblib.dump(YashirinYuk(), bufer)
    bufer.seek(0)
    joblib.load(bufer)
    print(f"  joblib.load dan keyin belgilar soni: {len(BELGILAR)}")
    print("  hech bir metod chaqirilmadi - kod yuklash paytida bajarildi")

    print("\n=== 3. Skaner: ichidagi importlar (bajarmasdan) ===")
    for nom, b in [("model", baytlar), ("YashirinYuk", yuk),
                   ("ChopEtuvchi", pickle.dumps(ChopEtuvchi()))]:
        importlar = globallar(b)
        begona = [i for i in importlar if i.split(".")[0] not in RUXSAT]
        holat = "XAVFLI" if begona else "toza"
        print(f"  {nom:<12} {holat:<7} {len(importlar)} import, begona: {begona}")
    for i in globallar(baytlar):
        print(f"    {i}")

    print("\n=== 4. Cheklangan yuklovchi ===")
    oldin = len(BELGILAR)
    qayta2 = xavfsiz_yukla(baytlar)
    farq2 = np.abs(qayta2.predict_proba(X) - model.predict_proba(X)).max()
    print(f"  model: yuklandi, bashorat farqi {farq2}")
    for nom, b in [("YashirinYuk", yuk),
                   ("ChopEtuvchi", pickle.dumps(ChopEtuvchi()))]:
        try:
            xavfsiz_yukla(b)
            print(f"  {nom}: yuklandi (kutilmagan)")
        except pickle.UnpicklingError as xato:
            print(f"  {nom}: UnpicklingError - {xato}")
    print(f"  yangi belgilar: {len(BELGILAR) - oldin}")

    print("\n=== 5. Eng xavfsiz: parametrlar JSON da ===")
    olchov, lr = model[0], model[1]
    parametrlar = {"ortacha": olchov.mean_.tolist(),
                   "masshtab": olchov.scale_.tolist(),
                   "koef": lr.coef_[0].tolist(),
                   "ozod": float(lr.intercept_[0])}
    matn = json.dumps(parametrlar)
    p = json.loads(matn)
    z = ((X - p["ortacha"]) / p["masshtab"]) @ np.array(p["koef"]) + p["ozod"]
    qolda = 1 / (1 + np.exp(-z))
    farq3 = np.abs(qolda - model.predict_proba(X)[:, 1]).max()
    print(f"  JSON hajmi: {len(matn)} bayt (pickle: {len(baytlar)} bayt)")
    print(f"  qo'lda hisoblangan ehtimol farqi: {farq3:.1e}")
    print("  ⭐ JSON - ma'lumot, pickle - dastur; begona dasturni ishga tushirmang")


if __name__ == "__main__":
    main()

=== 1. Oddiy model: saqlash va yuklash ===
  pickle hajmi: 1192 bayt, bashorat farqi: 0.0
  belgilar: []

=== 2. Yashirin yuk: faqat YUKLASH yetarli ===
  pickle.loads qaytardi: {'model': 'soxta'}
  belgilar: ['pickle.loads paytida chaqirildi']
  [!] bu qator pickle.loads ichidan chop etildi
  joblib.load dan keyin belgilar soni: 2
  hech bir metod chaqirilmadi - kod yuklash paytida bajarildi

=== 3. Skaner: ichidagi importlar (bajarmasdan) ===
  model        toza    6 import, begona: []
  YashirinYuk  XAVFLI  1 import, begona: ['__main__.belgi_qoy']
  ChopEtuvchi  XAVFLI  1 import, begona: ['builtins.print']
    numpy._core.multiarray.scalar
    numpy._core.numeric._frombuffer
    numpy.dtype
    sklearn.linear_model._logistic.LogisticRegression
    sklearn.pipeline.Pipeline
    sklearn.preprocessing._data.StandardScaler

=== 4. Cheklangan yuklovchi ===
  model: yuklandi, bashorat farqi 0.0
  YashirinYuk: UnpicklingError - taqiqlangan: __main__.belgi_qoy
  ChopEtuvchi: UnpicklingError - taqiqlangan: builtins.print
  yangi belgilar: 0

=== 5. Eng xavfsiz: parametrlar JSON da ===
  JSON hajmi: 314 bayt (pickle: 1192 bayt)
  qo'lda hisoblangan ehtimol farqi: 0.0e+00
  ⭐ JSON - ma'lumot, pickle - dastur; begona dasturni ishga tushirmang

"""sklearn versiya metadata si, InconsistentVersionWarning, torch weights_only."""

import io
import platform
import re
import warnings

import numpy as np
import sklearn
import torch
import torch.nn as nn
from sklearn.exceptions import InconsistentVersionWarning
from sklearn.linear_model import LogisticRegression

BELGILAR = []


def belgi_qoy(xabar):
    BELGILAR.append(xabar)
    return 0


class YashirinYuk:
    def __reduce__(self):
        return (belgi_qoy, ("torch.load paytida chaqirildi",))


SIYOSAT = {"python": 2, "numpy": 1, "sklearn": 2}     # nechta raqam mos bo'lsin


def muhit():
    return {"python": platform.python_version(),
            "numpy": np.__version__, "sklearn": sklearn.__version__}


def mosligini_tekshir(paket_meta, joriy):
    xatolar = []
    for kutubxona, n in SIYOSAT.items():
        a = paket_meta[kutubxona].split(".")[:n]
        b = joriy[kutubxona].split(".")[:n]
        if a != b:
            xatolar.append(f"{kutubxona}: paket {paket_meta[kutubxona]}, "
                           f"muhit {joriy[kutubxona]}")
    return xatolar


def tarmoq():
    return nn.Sequential(nn.Linear(4, 16), nn.ReLU(), nn.Linear(16, 2))


def main() -> None:
    rng = np.random.default_rng(0)
    X = rng.normal(size=(300, 4))
    y = (X[:, 0] > 0).astype(int)
    model = LogisticRegression().fit(X, y)

    print("=== 1. sklearn versiyani o'zi yozadi ===")
    holat = model.__getstate__()
    print(f"  __getstate__ ichida _sklearn_version: "
          f"{holat['_sklearn_version'] == sklearn.__version__}")

    print("\n=== 2. Boshqa versiyada saqlangan model ===")
    holat["_sklearn_version"] = "1.3.0"            # eski muhitni simulyatsiya
    eski = LogisticRegression.__new__(LogisticRegression)
    with warnings.catch_warnings(record=True) as ushlangan:
        warnings.simplefilter("always")
        eski.__setstate__(holat)
    for w in ushlangan:
        print(f"  {w.category.__name__}:")
        print(f"    {str(w.message).split('. ')[0]}")
    print(f"  model baribir ishlaydimi: {eski.predict(X[:3]).tolist()}")
    with warnings.catch_warnings():
        warnings.simplefilter("error", InconsistentVersionWarning)
        try:
            LogisticRegression.__new__(LogisticRegression).__setstate__(holat)
            print("  qabul qilindi")
        except InconsistentVersionWarning:
            print("  simplefilter('error') bilan: yuklash TO'XTATILDI")

    print("\n=== 3. Metadata va yuklash siyosati ===")
    joriy = muhit()
    stsenariylar = {
        "aynan shu muhit": dict(joriy),
        "python patch farqi": {**joriy, "python": joriy["python"].rsplit(".", 1)[0] + ".99"},
        "eski sklearn": {**joriy, "sklearn": "1.5.2"},
        "numpy 1.x": {**joriy, "numpy": "1.26.4"},
    }
    for nom, meta in stsenariylar.items():
        xatolar = mosligini_tekshir(meta, joriy)
        print(f"  {nom:<20} {'RAD' if xatolar else 'OK '}  "
              f"{xatolar[0] if xatolar else ''}")

    print("\n=== 4. torch: state_dict + weights_only=True ===")
    torch.manual_seed(0)
    net = tarmoq()
    bufer = io.BytesIO()
    torch.save({"state_dict": net.state_dict(),
                "meta": {"kirish": 4, "yashirin": 16, "chiqish": 2}}, bufer)
    bufer.seek(0)
    paket = torch.load(bufer, weights_only=True)
    yangi = tarmoq()
    yangi.load_state_dict(paket["state_dict"])
    x = torch.randn(5, 4)
    with torch.no_grad():
        teng = torch.equal(net(x), yangi(x))
    print(f"  kalitlar: {list(paket['state_dict'])}")
    print(f"  meta: {paket['meta']}, chiqishlar aynan teng: {teng}")

    print("\n=== 5. weights_only=True nimani rad etadi ===")
    holatlar = {
        "butun nn.Module": net,
        "yashirin yuk": {"state_dict": net.state_dict(), "x": YashirinYuk()},
    }
    for nom, obyekt in holatlar.items():
        bufer = io.BytesIO()
        torch.save(obyekt, bufer)
        bufer.seek(0)
        try:
            torch.load(bufer, weights_only=True)
            print(f"  {nom}: yuklandi")
        except Exception as xato:
            sabab = re.findall(r"Unsupported global: GLOBAL [\w.]+", str(xato))
            print(f"  {nom}: {type(xato).__name__}")
            print(f"    {sabab[0] if sabab else str(xato)[:60]}")
    print(f"  belgilar (yuk bajarildimi): {BELGILAR}")
    print("  ⭐ Versiya - metadata da, tekshiruv - yuklashda; torch - weights_only")


if __name__ == "__main__":
    main()

=== 1. sklearn versiyani o'zi yozadi ===
  __getstate__ ichida _sklearn_version: True

=== 2. Boshqa versiyada saqlangan model ===
  InconsistentVersionWarning:
    Trying to unpickle estimator LogisticRegression from version 1.3.0 when using version 1.9.1
  model baribir ishlaydimi: [1, 0, 0]
  simplefilter('error') bilan: yuklash TO'XTATILDI

=== 3. Metadata va yuklash siyosati ===
  aynan shu muhit      OK
  python patch farqi   OK
  eski sklearn         RAD  sklearn: paket 1.5.2, muhit 1.9.1
  numpy 1.x            RAD  numpy: paket 1.26.4, muhit 2.5.3

=== 4. torch: state_dict + weights_only=True ===
  kalitlar: ['0.weight', '0.bias', '2.weight', '2.bias']
  meta: {'kirish': 4, 'yashirin': 16, 'chiqish': 2}, chiqishlar aynan teng: True

=== 5. weights_only=True nimani rad etadi ===
  butun nn.Module: UnpicklingError
    Unsupported global: GLOBAL torch.nn.modules.container.Sequential
  yashirin yuk: UnpicklingError
    Unsupported global: GLOBAL __main__.belgi_qoy
  belgilar (yuk bajarildimi): []
  ⭐ Versiya - metadata da, tekshiruv - yuklashda; torch - weights_only

"""Model paketi: Pipeline + konfig + sxema + metadata + nazorat + MANIFEST."""

import hashlib
import io
import json
import platform
import tempfile
from pathlib import Path

import joblib
import numpy as np
import pandas as pd
import sklearn
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder

SONLI = ["oylar", "tolov", "shikoyat"]
KATEGORIAL = ["tarif", "hudud"]
FAYLLAR = ["konfig.json", "metadata.json", "model.joblib", "nazorat.json",
           "sxema.json"]


def malumot(n, seed):
    rng = np.random.default_rng(seed)
    df = pd.DataFrame({
        "oylar": rng.integers(1, 72, n),
        "tolov": rng.normal(60, 20, n).clip(10, 150).round(2),
        "shikoyat": rng.poisson(0.8, n).astype(float),
        "tarif": rng.choice(["oylik", "yillik", "ikki_yillik"], n,
                            p=[0.55, 0.3, 0.15]),
        "hudud": rng.choice(["Toshkent", "Samarqand", "Buxoro", "Andijon"], n),
    })
    logit = (-1.0 - 0.04 * df["oylar"] + 0.02 * (df["tolov"] - 60)
             + 0.5 * df["shikoyat"]
             + np.where(df["tarif"] == "oylik", 1.0, -0.5))
    df["ketdi"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    df.loc[rng.random(n) < 0.05, "shikoyat"] = np.nan    # ba'zan bo'sh
    return df


def orgat(df, seed):
    pre = ColumnTransformer([
        ("son", SimpleImputer(strategy="median"), SONLI),
        ("kat", OneHotEncoder(handle_unknown="ignore"), KATEGORIAL),
    ])
    model = Pipeline([("pre", pre),
                      ("model", RandomForestClassifier(
                          n_estimators=60, max_depth=8, random_state=seed,
                          n_jobs=1))])
    return model.fit(df[SONLI + KATEGORIAL], df["ketdi"])


def fayl_xeshi(yol):
    return hashlib.sha256(yol.read_bytes()).hexdigest()


def json_yoz(yol, obyekt):
    yol.write_text(json.dumps(obyekt, indent=2, ensure_ascii=False,
                              sort_keys=True), encoding="utf-8")


def paketla(papka, model, df_orgatish, df_sinov):
    papka.mkdir(parents=True)
    joblib.dump(model, papka / "model.joblib", compress=3)
    json_yoz(papka / "konfig.json", {"chegara": 0.35,
                                     "sinflar": ["qoladi", "ketadi"]})
    json_yoz(papka / "sxema.json", {"ustunlar": [
        {"nom": "oylar", "tur": "son", "min": 0, "max": 600, "majburiy": True},
        {"nom": "tolov", "tur": "son", "min": 0, "max": 10000, "majburiy": True},
        {"nom": "shikoyat", "tur": "son", "min": 0, "max": 100,
         "majburiy": False},
        {"nom": "tarif", "tur": "kategoriya", "nomalum": "rad",
         "qiymatlar": sorted(df_orgatish["tarif"].unique().tolist())},
        {"nom": "hudud", "tur": "kategoriya", "nomalum": "ruxsat",
         "qiymatlar": sorted(df_orgatish["hudud"].unique().tolist())},
    ]})
    p = model.predict_proba(df_sinov[SONLI + KATEGORIAL])[:, 1]
    json_yoz(papka / "metadata.json", {
        "nom": "mijoz_ketishi", "versiya": "1.2.0", "run_id": "run-0042",
        "python": platform.python_version(), "numpy": np.__version__,
        "sklearn": sklearn.__version__,
        "metrikalar": {"auc": round(float(roc_auc_score(df_sinov["ketdi"], p)), 4)},
    })
    namunalar = df_sinov[SONLI + KATEGORIAL].head(8)
    kutilgan = model.predict_proba(namunalar)[:, 1]
    json_yoz(papka / "nazorat.json", {
        "kirish": namunalar.to_dict(orient="records"),
        "kutilgan": [float(x) for x in kutilgan]})
    json_yoz(papka / "MANIFEST.json",
             {nom: fayl_xeshi(papka / nom) for nom in FAYLLAR})
    return fayl_xeshi(papka / "MANIFEST.json")     # registrga yoziladi (27.5)


def yukla(papka, registr_xeshi):
    if fayl_xeshi(papka / "MANIFEST.json") != registr_xeshi:
        raise ValueError("MANIFEST registrdagi xeshga mos emas")
    manifest = json.loads((papka / "MANIFEST.json").read_text(encoding="utf-8"))
    buzilgan = [nom for nom in FAYLLAR
                if fayl_xeshi(papka / nom) != manifest[nom]]
    if buzilgan:
        raise ValueError(f"xesh mos emas: {buzilgan}")
    model = joblib.load(papka / "model.joblib")     # faqat tekshiruvdan KEYIN
    nazorat = json.loads((papka / "nazorat.json").read_text(encoding="utf-8"))
    haqiqiy = model.predict_proba(pd.DataFrame(nazorat["kirish"]))[:, 1]
    farq = float(np.abs(haqiqiy - np.array(nazorat["kutilgan"])).max())
    if farq > 1e-12:
        raise ValueError(f"nazorat namunalari mos emas: max farq {farq:.4f}")
    return model, farq


def main() -> None:
    df = malumot(5000, seed=0)
    orgatish, sinov = df.iloc[:4000], df.iloc[4000:]
    model = orgat(orgatish, seed=0)

    with tempfile.TemporaryDirectory() as t:
        papka = Path(t) / "mijoz_ketishi-1.2.0"
        registr_xeshi = paketla(papka, model, orgatish, sinov)

        print("=== 1. Paket tuzilishi ===")
        print(f"  {papka.name}/")
        for yol in sorted(papka.iterdir(), key=lambda p: p.name.lower()):
            print(f"    {yol.name:<15} {yol.stat().st_size:>8} bayt")
        print(f"  registrga yozilgan MANIFEST xeshi: {registr_xeshi[:12]}")

        print("\n=== 2. Siqish darajalari (model.joblib) ===")
        asl = None
        for daraja in [0, 1, 3, 9, ("lzma", 3)]:
            bufer = io.BytesIO()
            joblib.dump(model, bufer, compress=daraja)
            hajm = len(bufer.getvalue())
            asl = asl or hajm
            bufer.seek(0)
            teng = np.array_equal(joblib.load(bufer).predict_proba(sinov),
                                  model.predict_proba(sinov))
            print(f"  compress={str(daraja):<12} {hajm:>9} bayt  "
                  f"{hajm / asl:6.1%}  bashorat teng: {teng}")

        print("\n=== 3. To'g'ri yuklash ===")
        _, farq = yukla(papka, registr_xeshi)
        print(f"  xeshlar: mos; nazorat namunalari max farqi: {farq}")

        print("\n=== 4. Buzilish A: konfig qo'lda tahrirlandi ===")
        konfig = json.loads((papka / "konfig.json").read_text(encoding="utf-8"))
        asl_matn = (papka / "konfig.json").read_text(encoding="utf-8")
        json_yoz(papka / "konfig.json", {**konfig, "chegara": 0.9})
        try:
            yukla(papka, registr_xeshi)
        except ValueError as xato:
            print(f"  ValueError - {xato}")
        (papka / "konfig.json").write_text(asl_matn, encoding="utf-8")

        print("\n=== 5. Buzilish B: model almashtirildi, MANIFEST ham ===")
        joblib.dump(orgat(orgatish, seed=1), papka / "model.joblib", compress=3)
        json_yoz(papka / "MANIFEST.json",
                 {nom: fayl_xeshi(papka / nom) for nom in FAYLLAR})
        try:
            yukla(papka, registr_xeshi)
        except ValueError as xato:
            print(f"  registr bilan:   ValueError - {xato}")
        yangi_xesh = fayl_xeshi(papka / "MANIFEST.json")
        try:
            yukla(papka, yangi_xesh)      # registr ham "yangilangan" bo'lsa
        except ValueError as xato:
            print(f"  registrsiz ham:  ValueError - {xato}")
        print("  ⭐ Xesh 'fayl o'shami', nazorat 'natija o'shami' deydi")


if __name__ == "__main__":
    main()

=== 1. Paket tuzilishi ===
  mijoz_ketishi-1.2.0/
    konfig.json           74 bayt
    MANIFEST.json        436 bayt
    metadata.json        189 bayt
    model.joblib      387198 bayt
    nazorat.json        1331 bayt
    sxema.json           776 bayt
  registrga yozilgan MANIFEST xeshi: 8ea4dad1af1f

=== 2. Siqish darajalari (model.joblib) ===
  compress=0              1478178 bayt  100.0%  bashorat teng: True
  compress=1               476771 bayt   32.3%  bashorat teng: True
  compress=3               387198 bayt   26.2%  bashorat teng: True
  compress=9               334337 bayt   22.6%  bashorat teng: True
  compress=('lzma', 3)     223139 bayt   15.1%  bashorat teng: True

=== 3. To'g'ri yuklash ===
  xeshlar: mos; nazorat namunalari max farqi: 0.0

=== 4. Buzilish A: konfig qo'lda tahrirlandi ===
  ValueError - xesh mos emas: ['konfig.json']

=== 5. Buzilish B: model almashtirildi, MANIFEST ham ===
  registr bilan:   ValueError - MANIFEST registrdagi xeshga mos emas
  registrsiz ham:  ValueError - nazorat namunalari mos emas: max farq 0.0438
  ⭐ Xesh 'fayl o'shami', nazorat 'natija o'shami' deydi

"""Predictor: paketni yuklaydi, kirishni tekshiradi, bashorat qiladi."""

import hashlib
import json
import tempfile
from pathlib import Path

import joblib
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

SONLI = ["oylar", "tolov", "shikoyat"]
KATEGORIAL = ["tarif", "hudud"]
FAYLLAR = ["konfig.json", "metadata.json", "model.joblib", "nazorat.json",
           "sxema.json"]


def malumot(n, seed):
    rng = np.random.default_rng(seed)
    df = pd.DataFrame({
        "oylar": rng.integers(1, 72, n),
        "tolov": rng.normal(60, 20, n).clip(10, 150).round(2),
        "shikoyat": rng.poisson(0.8, n).astype(float),
        "tarif": rng.choice(["oylik", "yillik", "ikki_yillik"], n,
                            p=[0.55, 0.3, 0.15]),
        "hudud": rng.choice(["Toshkent", "Samarqand", "Buxoro", "Andijon"], n),
    })
    logit = (-1.0 - 0.04 * df["oylar"] + 0.02 * (df["tolov"] - 60)
             + 0.5 * df["shikoyat"]
             + np.where(df["tarif"] == "oylik", 1.0, -0.5))
    df["ketdi"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return df


def fayl_xeshi(yol):
    return hashlib.sha256(yol.read_bytes()).hexdigest()


def json_yoz(yol, obyekt):
    yol.write_text(json.dumps(obyekt, indent=2, ensure_ascii=False,
                              sort_keys=True), encoding="utf-8")


def paket_yasa(papka):
    """3-misoldagi paketning qisqa nusxasi (logistik regressiya bilan)."""
    df = malumot(3000, seed=0)
    pre = ColumnTransformer([
        ("son", Pipeline([("bosh", SimpleImputer(strategy="median")),
                          ("olchov", StandardScaler())]), SONLI),
        ("kat", OneHotEncoder(handle_unknown="ignore"), KATEGORIAL),
    ])
    model = Pipeline([("pre", pre), ("model", LogisticRegression())])
    model.fit(df[SONLI + KATEGORIAL], df["ketdi"])
    papka.mkdir(parents=True)
    joblib.dump(model, papka / "model.joblib", compress=3)
    json_yoz(papka / "konfig.json", {"chegara": 0.35})
    json_yoz(papka / "metadata.json", {"nom": "mijoz_ketishi",
                                       "versiya": "1.2.0"})
    json_yoz(papka / "sxema.json", {"ustunlar": [
        {"nom": "oylar", "tur": "son", "min": 0, "max": 600, "majburiy": True},
        {"nom": "tolov", "tur": "son", "min": 0, "max": 10000, "majburiy": True},
        {"nom": "shikoyat", "tur": "son", "min": 0, "max": 100,
         "majburiy": False},
        {"nom": "tarif", "tur": "kategoriya", "nomalum": "rad",
         "qiymatlar": ["ikki_yillik", "oylik", "yillik"]},
        {"nom": "hudud", "tur": "kategoriya", "nomalum": "ruxsat",
         "qiymatlar": ["Andijon", "Buxoro", "Samarqand", "Toshkent"]},
    ]})
    namunalar = df[SONLI + KATEGORIAL].tail(5)
    json_yoz(papka / "nazorat.json", {
        "kirish": namunalar.to_dict(orient="records"),
        "kutilgan": [float(x) for x in model.predict_proba(namunalar)[:, 1]]})
    json_yoz(papka / "MANIFEST.json",
             {nom: fayl_xeshi(papka / nom) for nom in FAYLLAR})
    return df


class SxemaXatosi(ValueError):
    def __init__(self, xatolar):
        super().__init__(f"{len(xatolar)} ta xato")
        self.xatolar = xatolar


class Predictor:
    def __init__(self, papka):
        papka = Path(papka)
        manifest = json.loads((papka / "MANIFEST.json").read_text(encoding="utf-8"))
        for nom in FAYLLAR:
            if fayl_xeshi(papka / nom) != manifest[nom]:
                raise ValueError(f"xesh mos emas: {nom}")
        o = lambda nom: json.loads((papka / nom).read_text(encoding="utf-8"))
        self.konfig, self.sxema = o("konfig.json"), o("sxema.json")
        self.metadata = o("metadata.json")
        self._model = joblib.load(papka / "model.joblib")
        nazorat = o("nazorat.json")
        farq = np.abs(self._model.predict_proba(pd.DataFrame(nazorat["kirish"]))
                      [:, 1] - np.array(nazorat["kutilgan"])).max()
        if farq > 1e-12:
            raise ValueError(f"nazorat namunalari mos emas: {farq:.4f}")
        self.ustunlar = [u["nom"] for u in self.sxema["ustunlar"]]
        self.ogohlantirishlar = []

    @property
    def versiya(self):
        return self.metadata["versiya"]

    def validatsiya(self, kirish):
        if isinstance(kirish, dict):
            kirish = [kirish]
        df = (kirish.copy() if isinstance(kirish, pd.DataFrame)
              else pd.DataFrame(list(kirish)))
        if len(df) == 0:
            raise SxemaXatosi(["bo'sh kirish"])
        xatolar, self.ogohlantirishlar = [], []
        for u in self.sxema["ustunlar"]:
            nom = u["nom"]
            if nom not in df.columns:
                xatolar.append(f"{nom}: ustun yo'q")
                continue
            if u["tur"] == "son":
                son = pd.to_numeric(df[nom], errors="coerce")
                for i in df.index[son.isna() & df[nom].notna()]:
                    xatolar.append(f"qator {i}, {nom}: son emas ({df.at[i, nom]!r})")
                if u["majburiy"]:
                    for i in df.index[df[nom].isna()]:
                        xatolar.append(f"qator {i}, {nom}: majburiy, lekin bo'sh")
                for i in df.index[(son < u["min"]) | (son > u["max"])]:
                    xatolar.append(f"qator {i}, {nom}: {son[i]:g} "
                                   f"[{u['min']}, {u['max']}] dan tashqarida")
                df[nom] = son.astype(float)
            else:
                for i in df.index[~df[nom].isin(u["qiymatlar"])]:
                    matn = f"qator {i}, {nom}: noma'lum {df.at[i, nom]!r}"
                    if u["nomalum"] == "rad":
                        xatolar.append(matn)
                    else:
                        self.ogohlantirishlar.append(matn)
        if xatolar:
            raise SxemaXatosi(xatolar)
        ortiqcha = sorted(set(df.columns) - set(self.ustunlar))
        if ortiqcha:
            self.ogohlantirishlar.append(f"e'tiborsiz ustunlar: {ortiqcha}")
        return df[self.ustunlar]

    def predict_proba(self, kirish):
        return self._model.predict_proba(self.validatsiya(kirish))[:, 1]

    def predict(self, kirish):
        return (self.predict_proba(kirish) >= self.konfig["chegara"]).astype(int)


def main() -> None:
    with tempfile.TemporaryDirectory() as t:
        df = paket_yasa(Path(t) / "paket")
        pr = Predictor(Path(t) / "paket")

        print("=== 1. Predictor yuklandi ===")
        print(f"  versiya: {pr.versiya}, chegara: {pr.konfig['chegara']}")
        for u in pr.sxema["ustunlar"]:
            qoshimcha = (f"[{u['min']}, {u['max']}]" if u["tur"] == "son"
                         else f"{len(u['qiymatlar'])} qiymat, noma'lum: {u['nomalum']}")
            print(f"  {u['nom']:<9} {u['tur']:<11} {qoshimcha}")

        print("\n=== 2. Uch xil kirish - bir xil natija ===")
        mijoz = {"hudud": "Buxoro", "tarif": "oylik", "tolov": 95.5,
                 "oylar": 3, "shikoyat": 2}
        a = pr.predict_proba(mijoz)
        b = pr.predict_proba([mijoz])
        c = pr.predict_proba(pd.DataFrame([mijoz])[list(reversed(list(mijoz)))])
        print(f"  dict: {a[0]:.4f}  list: {b[0]:.4f}  DataFrame: {c[0]:.4f}")
        print(f"  sinf (chegara {pr.konfig['chegara']}): {pr.predict(mijoz)[0]}")

        print("\n=== 3. Yomon kirish: BARCHA xatolar birdaniga ===")
        yomon = [
            {"oylar": 12, "tolov": 50, "shikoyat": 0, "tarif": "oylik",
             "hudud": "Toshkent"},
            {"oylar": -5, "tolov": "ellik", "shikoyat": 1, "tarif": "oylik",
             "hudud": "Buxoro"},
            {"oylar": None, "tolov": 1e9, "tarif": "haftalik",
             "hudud": "Andijon"},
            {"tolov": 40, "shikoyat": 0, "tarif": "yillik", "hudud": "Buxoro"},
        ]
        try:
            pr.predict_proba(yomon)
        except SxemaXatosi as xato:
            print(f"  SxemaXatosi: {xato}")
            for x in xato.xatolar:
                print(f"    {x}")

        print("\n=== 4. Ixtiyoriy maydon, noma'lum hudud, ortiqcha ustun ===")
        yaxshi = [
            {"oylar": 24, "tolov": 60, "tarif": "yillik", "hudud": "Xiva"},
            {"oylar": 24, "tolov": 60, "shikoyat": None, "tarif": "yillik",
             "hudud": "Toshkent", "ism": "Ali"},
        ]
        p = pr.predict_proba(yaxshi)
        print(f"  ehtimollar: {np.round(p, 4).tolist()}")
        for o in pr.ogohlantirishlar:
            print(f"  ogohlantirish: {o}")

        print("\n=== 5. Batch va yakka bashorat mosligi ===")
        partiya = df[SONLI + KATEGORIAL].head(300)
        pb = pr.predict_proba(partiya)
        py = np.array([pr.predict_proba(q)[0]
                       for q in partiya.head(60).to_dict(orient="records")])
        print(f"  300 qatorlik batch, 60 ta yakka: max farq "
              f"{np.abs(pb[:60] - py).max():.1e}")
        print(f"  ketadi deb belgilanganlar: {int(pr.predict(partiya).sum())} / 300")
        print("  ⭐ Bitta interfeys: dict, ro'yxat, DataFrame - bir xil natija")


if __name__ == "__main__":
    main()

=== 1. Predictor yuklandi ===
  versiya: 1.2.0, chegara: 0.35
  oylar     son         [0, 600]
  tolov     son         [0, 10000]
  shikoyat  son         [0, 100]
  tarif     kategoriya  3 qiymat, noma'lum: rad
  hudud     kategoriya  4 qiymat, noma'lum: ruxsat

=== 2. Uch xil kirish - bir xil natija ===
  dict: 0.8268  list: 0.8268  DataFrame: 0.8268
  sinf (chegara 0.35): 1

=== 3. Yomon kirish: BARCHA xatolar birdaniga ===
  SxemaXatosi: 6 ta xato
    qator 2, oylar: majburiy, lekin bo'sh
    qator 3, oylar: majburiy, lekin bo'sh
    qator 1, oylar: -5 [0, 600] dan tashqarida
    qator 1, tolov: son emas ('ellik')
    qator 2, tolov: 1e+09 [0, 10000] dan tashqarida
    qator 2, tarif: noma'lum 'haftalik'

=== 4. Ixtiyoriy maydon, noma'lum hudud, ortiqcha ustun ===
  ehtimollar: [0.1261, 0.1352]
  ogohlantirish: qator 0, hudud: noma'lum 'Xiva'
  ogohlantirish: e'tiborsiz ustunlar: ['ism']

=== 5. Batch va yakka bashorat mosligi ===
  300 qatorlik batch, 60 ta yakka: max farq 2.2e-16
  ketadi deb belgilanganlar: 55 / 300
  ⭐ Bitta interfeys: dict, ro'yxat, DataFrame - bir xil natija

model = joblib.load("internetdan_model.joblib")                    # ⚠️
# faqat o'z registringizdan, xesh tekshiruvidan keyin                 # ✅

joblib.dump(scaler, "scaler.pkl"); joblib.dump(clf, "clf.pkl")     # ⚠️
joblib.dump(make_pipeline(scaler, clf), "model.joblib")            # ✅

sinf = (p >= 0.35).astype(int)                                     # ⚠️
sinf = (p >= konfig["chegara"]).astype(int)                        # ✅ paketdan

joblib.dump(model, "model.joblib")                                 # ⚠️
json_yoz(papka / "metadata.json", {"sklearn": sklearn.__version__, ...})  # ✅

torch.save(model, "model.pt"); torch.load("model.pt", weights_only=False)  # ⚠️
torch.save(model.state_dict(), "model.pt"); torch.load("model.pt", weights_only=True)  # ✅

model = joblib.load(yol)                                           # ⚠️
model, farq = yukla(papka, registr_xeshi)   # xesh + nazorat         # ✅

if "oylar" not in df: raise ValueError("oylar yo'q")               # ⚠️
xatolar.append(...); ...; raise SxemaXatosi(xatolar)               # ✅ hammasi

1.  # paketning oltita tarkibiy qismi?
2.  # pickle.loads qachon kod bajaradi?
3.  # joblib.load xavfsizroqmi?
4.  # pickle skaneri nimani o'qiydi?
5.  # cheklangan yuklovchi qaysi metodni qayta yozadi?
6.  # InconsistentVersionWarning yuklashni to'xtatadimi?
7.  # torch.save(model) ni weights_only=True bilan yuklash mumkinmi?
8.  # nazorat namunalari nimani ushlaydi, xesh esa nimani?
9.  # MANIFEST xeshi qayerda saqlanadi?
10. # sxema oraliqlari qayerdan olinadi?
11. # noma'lum kategoriya uchun ikki siyosat?
12. # Predictor qanday kirishlarni qabul qiladi?

1.  model = pickle.load(open("yuklab_olingan.pkl", "rb"))

2.  joblib.dump(scaler, "s.pkl"); joblib.dump(clf, "c.pkl")

3.  torch.save(net, "net.pt")

4.  warnings.simplefilter("ignore")  # yuklashdagi ogohlantirishlar jonga tegdi

5.  sxema = {"oylar": {"min": df.oylar.min(), "max": df.oylar.max()}}