27.2-dars: Reproduksiya va muhit

1. TAKRORLANUVCHANLIK (bit darajasida)
   bir xil kod + bir xil ma'lumot + bir xil muhit + bir xil urug'
   -> AYNAN bir xil bashoratlar (xeshlari teng)
   kim uchun: debug, audit, regressiya testlari, CI

2. REPRODUKSIYA (statistik)
   bir xil kod + bir xil ma'lumot, lekin boshqa mashina/versiya/urug'
   -> metrika ishonch oralig'i ichida, XULOSA o'zgarmaydi
   kim uchun: hisobot, maqola, qaror

3. REPLIKATSIYA (umumlashish)
   yangi ma'lumot (keyingi oy, boshqa hudud)
   -> xulosa hali ham to'g'rimi?
   kim uchun: ishlab chiqarish, monitoring 27.11-bob

MUHIM:
  1-daraja 2-darajaning o'rnini BOSMAYDI:
    aynan takrorlanadigan natija ham bitta tasodifiy namuna bo'lishi mumkin
  2-daraja 1-darajasiz QIYIN:
    natija o'zgarganda sababni topish uchun avval uni takrorlash kerak

1. URUG'LAR
   o'rnatilmagan yoki faqat bittasi o'rnatilgan (python, lekin numpy emas)
   random_state=None -> global holatga bog'liq 19.7-bob

2. TARTIB
   set / dict tartibi (satrlar uchun PYTHONHASHSEED ga bog'liq)
   fayl ro'yxati tartibi (os.listdir, glob - fayl tizimiga bog'liq)
   SQL so'rov natijasi (ORDER BY siz tartib kafolatlanmagan)
   qatorlar tartibi -> SGD, bootstrap, bo'lish natijasi o'zgaradi

3. MA'LUMOT
   "o'sha jadval" - lekin kecha yangi qatorlar qo'shilgan
   kechikib kelgan yozuvlar, tuzatilgan xatolar
   -> ma'lumot XESHI yoki snapshot/versiya 27.3-bob

4. KUTUBXONA VERSIYALARI
   sukut parametr o'zgaradi (masalan, solver, max_iter)
   algoritm ichki tartibi o'zgaradi
   pickle/joblib boshqa versiyada ochilmasligi mumkin 19.6-bob

5. FLOAT VA PARALLEL HISOB
   (a + b) + c != a + (b + c)
   oqimlar soni -> yig'indi tartibi (BLAS, OpenMP, n_jobs)
   GPU: atom amallari tartibi aniqlanmagan (cuDNN algoritmlari)

6. MUHIT
   muhit o'zgaruvchilari (PYTHONHASHSEED, OMP_NUM_THREADS,
     CUBLAS_WORKSPACE_CONFIG, TZ, LANG)
   apparat (CPU ko'rsatmalar to'plami, GPU modeli)
   operatsion tizim (fayl tizimi, satr oxiri CRLF/LF)

INTERPRETATOR DARAJASI (jarayon boshlanishidan OLDIN):
  PYTHONHASHSEED           -> str/bytes hash i -> set/dict tartibi

JARAYON DARAJASI (global generatorlar):
  random.seed(s)           -> Python random moduli
  np.random.seed(s)        -> numpy GLOBAL holati (eski API)
  torch.manual_seed(s)     -> torch CPU (+ CUDA) global generatori
  torch.cuda.manual_seed_all(s)  -> barcha GPU lar

OBYEKT DARAJASI (tavsiya etiladi - mustaqil va aniq):
  rng = np.random.default_rng(s)         -> o'z Generator i
  rng.spawn(k)                           -> k ta MUSTAQIL oqim
  RandomForestClassifier(random_state=s) -> sklearn (int!)
  DataLoader(..., generator=torch.Generator().manual_seed(s))

ISHCHI JARAYONLAR (DataLoader num_workers > 0):
  har ishchi: torch urug'i = base_seed + worker_id
  torch python random va numpy global holatini ham o'zi o'rnatadi
  O'Z generatoringiz bo'lsa (Dataset ichida default_rng) -
    worker_init_fn ichida get_worker_info().seed dan hosil qiling

DETERMINISTIK ALGORITMLAR (GPU):
  torch.use_deterministic_algorithms(True)
  torch.backends.cudnn.benchmark = False
  CUBLAS_WORKSPACE_CONFIG=:4096:8

Python 3.3+ da xavfsizlik uchun (hash-flooding hujumiga qarshi)
str va bytes hash i HAR JARAYONDA tasodifiy "tuz" bilan hisoblanadi.

  hash("salom")  -> jarayon A:  7213...,  jarayon B: -4410...
  hash(12345)    -> har doim 12345 (int tuzlanmaydi)

OQIBAT:
  set(satrlar) tartibi - jarayonga bog'liq
  dict tartibi - QO'SHILISH tartibida (3.7+), xeshga emas -> barqaror
  lekin dict(set(...)) yoki {k: i for i, k in enumerate(set(...))} -> YO'Q

BOSHQARISH:
  PYTHONHASHSEED=0 python orgat.py     # interpretator ISHGA TUSHISHIDAN oldin
  os.environ["PYTHONHASHSEED"] = "0"   # skript ichida - KECH, ta'sir qilmaydi
                                       # (faqat bola jarayonlarga o'tadi)

TO'G'RI YECHIM:
  hash() ga tayanmang: sorted(set(...)), barqaror xesh (hashlib)
  PYTHONHASHSEED ni qat'iylash - qo'shimcha himoya, asosiy yechim emas

NIMA QAYD QILINADI:
  python versiyasi         platform.python_version()
  platforma                platform.system(), platform.machine()
  paketlar (lock)          importlib.metadata.version(nom) -> "numpy==2.5.3"
  kod versiyasi            git rev-parse HEAD + git status --porcelain (toza?)
  ma'lumot                 sha256(fayl baytlari) yoki snapshot ID
  konfig                   konfig + sha256(kanonik JSON)
  urug'lar                 konfig ichida
  muhit o'zgaruvchilari    PYTHONHASHSEED, OMP_NUM_THREADS, ... (sirlar EMAS!)
  natija                   metrikalar + bashorat xeshi

KANONIK JSON (konfig xeshi uchun):
  json.dumps(k, sort_keys=True, separators=(",", ":"))
  -> kalit tartibi ahamiyatsiz, bo'sh joylar ahamiyatsiz

pip freeze VA importlib.metadata:
  pip freeze      - butun muhit, tashqi buyruq
  importlib.metadata - kod ichidan, faqat kerakli paketlar

manifest.json:
{
  "python": "3.14.5",
  "platforma": {"tizim": "Windows", "arxitektura": "AMD64"},
  "paketlar": {"numpy": "2.5.3", "scikit-learn": "1.9.1", ...},
  "git": {"commit": "a1b2c3...", "toza": true},
  "muhit": {"PYTHONHASHSEED": "0", "OMP_NUM_THREADS": "1"},
  "malumot": {"fayl": "malumot.csv", "sha256": "952c6655a350"},
  "konfig": {"urug": 0, "lr": 0.1, "max_iter": 100},
  "konfig_xesh": "507dfdb81878",
  "natija": {"auc": 0.716766, "bashorat_xesh": "818897a04902"}
}

QAYTA ISHGA TUSHIRISHDAN OLDIN tekshir(manifest):
  python/paket versiyalari mosmi?      -> farq: OGOHLANTIRISH
  ma'lumot xeshi mosmi?                -> farq: TO'XTATISH
  konfig xeshi mosmi?                  -> farq: TO'XTATISH
KEYIN:
  natija == manifest["natija"]?        -> reproduksiya testi

BITTA URUG':
  A: 0.924, B: 0.926 -> "B yaxshiroq"
  aslida: A va B ning urug'lar bo'yicha std ~ 0.01
          farq 0.002 - shovqin ichida

TO'G'RI HISOBOT (18-qism):
  bir necha urug' (bo'lish + model), o'rtacha +- std
  JUFTLASHGAN farq: d_s = A_s - B_s (bir xil urug' - bir xil bo'lish)
  SE(d) = std(d) / sqrt(n); |mean(d)| > 2 * SE -> sezilarli
  qaror: eng yaxshisidan sezilarli yomon bo'lmagan ENG ARZON variant

TARQOQLIK MANBALARI:
  bo'lish urug'i  - qaysi qatorlar testga tushdi (ko'pincha KATTA)
  model urug'i    - bootstrap, belgi tanlash, init
  ikkalasini alohida o'zgartirib ko'rish -> qaysi biri ustun

# venv + pip (standart kutubxona)
python -m venv .venv
.venv/Scripts/activate            # Windows (Linux/macOS: source .venv/bin/activate)
pip install -r requirements.txt   # requirements.txt: numpy==2.5.3 ...
pip freeze > requirements.lock    # butun muhitning aniq versiyalari

# pip-tools: to'g'ridan-to'g'ri bog'liqliklar -> to'liq lock
pip-compile requirements.in --generate-hashes

# uv (tez, lock fayl bilan)
uv venv
uv pip install -r requirements.txt
uv lock                           # uv.lock - bog'liqliklar daraxti + xeshlar

# poetry
poetry add scikit-learn==1.9.1    # pyproject.toml + poetry.lock

# conda (python dan tashqari kutubxonalar: CUDA, MKL, GDAL)
conda env create -f environment.yml
conda env export --from-history > environment.yml

import hashlib
import importlib.metadata as md
import json
import platform
import random

import numpy as np
import torch


def seed_everything(urug):
    random.seed(urug)
    np.random.seed(urug)
    torch.manual_seed(urug)                      # CUDA ham (bor bo'lsa)


def worker_init_fn(worker_id):
    urug = torch.utils.data.get_worker_info().seed % 2**32
    np.random.seed(urug)                         # yoki o'z Generator ingiz
    random.seed(urug)


def lock(paketlar):
    return {p: md.version(p) for p in sorted(paketlar)}


def konfig_xesh(konfig):
    kanonik = json.dumps(konfig, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(kanonik.encode()).hexdigest()[:12]


def fayl_xesh(yol, blok=1 << 20):
    h = hashlib.sha256()
    with open(yol, "rb") as f:
        for bolak in iter(lambda: f.read(blok), b""):
            h.update(bolak)                      # katta fayl - bo'lakma-bo'lak
    return h.hexdigest()[:12]


def barqaror_savat(satr, n=1000):
    return int(hashlib.md5(satr.encode()).hexdigest(), 16) % n   # hash() EMAS


muhit = {"python": platform.python_version(), "tizim": platform.system(),
         "paketlar": lock(["numpy", "scikit-learn"])}
kategoriyalar = sorted(set(["Toshkent", "Andijon"]))  # list(set()) EMAS

darajalar: takrorlanuvchanlik (bit) / reproduksiya (statistik) / replikatsiya
manbalar: urug', tartib, ma'lumot, versiya, float/parallel, muhit
urug'lar: PYTHONHASHSEED (jarayondan oldin) > global > obyekt (tavsiya)
hash(): jarayonga bog'liq -> sorted() va hashlib
manifest: python, paketlar, git (+toza), ma'lumot xeshi, konfig xeshi, natija
tekshiruv: yangi jarayonda, manifestga qarshi, natija xeshi teng
sezgirlik: bir necha urug', juftlashgan farq, 2*SE

"""Reproduksiya testi: bir xil skript, ikki YANGI jarayon, natija tengmi?"""

import os
import subprocess
import sys
import tempfile
from pathlib import Path

import numpy as np
import pandas as pd

# o'qitish skripti - tipik "noutbukdan ko'chirilgan" kod
SKRIPT = r'''
import csv, hashlib, sys
from pathlib import Path
import numpy as np

urug = "--urug" in sys.argv
tartibla = "--sorted" in sys.argv
papka = Path(sys.argv[1])

nomlar = {p.name for p in papka.glob("qism_*.csv")}          # dublikatsiz
nomlar = sorted(nomlar) if tartibla else list(nomlar)
qatorlar = []
for n in nomlar:
    with open(papka / n, encoding="utf-8") as f:
        qatorlar += list(csv.DictReader(f))

hududlar = {q["hudud"] for q in qatorlar}
hududlar = sorted(hududlar) if tartibla else list(hududlar)
kod = {h: i for i, h in enumerate(hududlar)}                # label encoding
X = np.array([[float(q["x1"]), float(q["x2"]), kod[q["hudud"]]]
              for q in qatorlar])
X = (X - X.mean(0)) / X.std(0)
y = np.array([int(q["y"]) for q in qatorlar])
Xtr, ytr, Xte, yte = X[:3000], y[:3000], X[3000:], y[3000:]

if urug:
    np.random.seed(0)
w = np.random.normal(0, 0.1, 3)
b = 0.0
for davr in range(5):                                       # mini-batch SGD
    tartib = np.random.permutation(len(ytr))
    for i in range(0, len(ytr), 50):
        j = tartib[i:i + 50]
        g = 1 / (1 + np.exp(-(Xtr[j] @ w + b))) - ytr[j]
        w -= 0.1 * Xtr[j].T @ g / len(j)
        b -= 0.1 * g.mean()
p = 1 / (1 + np.exp(-(Xte @ w + b)))
r = p.argsort().argsort() + 1                               # AUC ranglardan
auc = (r[yte == 1].sum() - (yte == 1).sum() * ((yte == 1).sum() + 1) / 2) / (
    (yte == 1).sum() * (yte == 0).sum())
print(hashlib.sha256(np.round(p, 10).tobytes()).hexdigest()[:12], round(auc, 4))
'''


def ishga(skript, papka, bayroqlar, hashseed):
    """Skriptni YANGI jarayonda, berilgan PYTHONHASHSEED bilan ishga tushiradi."""
    env = dict(os.environ, PYTHONHASHSEED=str(hashseed))
    r = subprocess.run([sys.executable, str(skript), str(papka), *bayroqlar],
                       capture_output=True, text=True, env=env, timeout=120)
    if r.returncode != 0:
        raise RuntimeError(r.stderr.strip().splitlines()[-1])
    xesh, auc = r.stdout.split()
    return xesh, float(auc)


def main() -> None:
    with tempfile.TemporaryDirectory() as t:
        papka = Path(t)
        rng = np.random.default_rng(0)
        hudud = ["Andijon", "Buxoro", "Jizzax", "Namangan", "Navoiy",
                 "Samarqand", "Toshkent", "Xorazm"]
        eff = dict(zip(hudud, [0.4, -0.3, 0.8, 0.1, -0.6, 0.0, -0.9, 0.5]))
        for i in range(4):                         # ma'lumot 4 bo'lak faylda
            n = 1000
            h = rng.choice(hudud, n)
            x1, x2 = rng.normal(size=n), rng.normal(size=n)
            logit = x1 - 0.5 * x2 + pd.Series(h).map(eff).to_numpy()
            y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
            pd.DataFrame({"x1": x1.round(5), "x2": x2.round(5), "hudud": h,
                          "y": y}).to_csv(papka / f"qism_{i}.csv", index=False)
        skript = papka / "orgat.py"
        skript.write_text(SKRIPT, encoding="utf-8")

        print("=== 1. Bir xil skript, ikki yangi jarayon (PYTHONHASHSEED 1 va 2) ===")
        print(f"  {'variant':<26} {'xesh teng':>9}  izoh")
        variantlar = [("urug' yo'q, set tartibi", []),
                      ("urug' bor, set tartibi", ["--urug"]),
                      ("urug' yo'q, sorted", ["--sorted"]),
                      ("urug' bor, sorted", ["--urug", "--sorted"])]
        for nom, bayroq in variantlar:
            (x1, a1), (x2, a2) = [ishga(skript, papka, bayroq, hs) for hs in (1, 2)]
            izoh = f"AUC {a1:.4f} va {a2:.4f}" if "--urug" in bayroq else ""
            print(f"  {nom:<26} {str(x1 == x2):>9}  {izoh}")

        print("\n=== 2. PYTHONHASHSEED qat'iy bo'lsa ===")
        (xa, _), (xb, _) = [ishga(skript, papka, ["--urug"], hs) for hs in (5, 5)]
        print(f"  urug' bor, set tartibi, ikkalasida PYTHONHASHSEED=5: teng {xa == xb}")
        print("  -> set tartibi faqat PYTHONHASHSEED ga bog'liq; sukut bo'yicha u")
        print("     tasodifiy, ya'ni 'urug' qo'yilgan' skript ham har safar boshqacha")

        print("\n=== 3. Olti yangi jarayon (PYTHONHASHSEED 10..15) ===")
        for nom, bayroq in [("urug' bor, set tartibi", ["--urug"]),
                            ("urug' bor, sorted", ["--urug", "--sorted"])]:
            natijalar = {ishga(skript, papka, bayroq, hs) for hs in range(10, 16)}
            auclar = sorted(a for _, a in natijalar)
            print(f"  {nom:<24} noyob natija: {len(natijalar)}; AUC "
                  f"{auclar[0]:.4f} - {auclar[-1]:.4f}")


if __name__ == "__main__":
    main()

=== 1. Bir xil skript, ikki yangi jarayon (PYTHONHASHSEED 1 va 2) ===
  variant                    xesh teng  izoh
  urug' yo'q, set tartibi        False
  urug' bor, set tartibi         False  AUC 0.7376 va 0.7456
  urug' yo'q, sorted             False
  urug' bor, sorted               True  AUC 0.7584 va 0.7584

=== 2. PYTHONHASHSEED qat'iy bo'lsa ===
  urug' bor, set tartibi, ikkalasida PYTHONHASHSEED=5: teng True
  -> set tartibi faqat PYTHONHASHSEED ga bog'liq; sukut bo'yicha u
     tasodifiy, ya'ni 'urug' qo'yilgan' skript ham har safar boshqacha

=== 3. Olti yangi jarayon (PYTHONHASHSEED 10..15) ===
  urug' bor, set tartibi   noyob natija: 6; AUC 0.7384 - 0.7590
  urug' bor, sorted        noyob natija: 1; AUC 0.7584 - 0.7584

"""Urug'lar ierarxiyasi va PYTHONHASHSEED tuzog'i."""

import os
import random
import subprocess
import sys

import numpy as np
import torch
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from torch.utils.data import DataLoader, TensorDataset


def seed_everything(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)


def jarayon(kod, hashseed):
    """Kodni yangi Python jarayonida ishga tushiradi."""
    env = dict(os.environ)
    if hashseed is None:
        env.pop("PYTHONHASHSEED", None)
    else:
        env["PYTHONHASHSEED"] = str(hashseed)
    r = subprocess.run([sys.executable, "-c", kod], capture_output=True,
                       text=True, env=env, timeout=60)
    return r.stdout.strip()


def main() -> None:
    print("=== 1. Uch generator - uch alohida urug' ===")
    seed_everything(0)
    a = (random.random(), np.random.rand(), torch.rand(1).item())
    random.seed(0)                           # FAQAT python random
    b = (random.random(), np.random.rand(), torch.rand(1).item())
    print(f"  seed_everything(0) dan keyin:   {[round(v, 4) for v in a]}")
    print(f"  faqat random.seed(0) dan keyin: {[round(v, 4) for v in b]}")
    print(f"  mos keladi: python {a[0] == b[0]}, numpy {a[1] == b[1]}, "
          f"torch {a[2] == b[2]}")

    print("\n=== 2. sklearn random_state: None va int ===")
    X, y = make_classification(n_samples=600, n_features=10, random_state=0)

    def rf(rs, oraliq=False):
        np.random.seed(0)
        if oraliq:
            np.random.rand(3)                # boshqa kod global holatni ishlatdi
        return RandomForestClassifier(n_estimators=20, random_state=rs).fit(
            X, y).predict_proba(X[:50])

    print(f"  None, global seed(0):              teng {np.array_equal(rf(None), rf(None))}")
    print(f"  None, oraliqda boshqa np.random:   teng "
          f"{np.array_equal(rf(None), rf(None, True))}")
    print(f"  int 42, oraliqda boshqa np.random: teng "
          f"{np.array_equal(rf(42), rf(42, True))}")

    print("\n=== 3. numpy Generator: mustaqil oqimlar ===")
    g = np.random.default_rng(123)
    print(f"  default_rng(123) birinchi 3: {np.round(g.random(3), 4).tolist()}")
    oqim = np.random.default_rng(123).spawn(2)
    print(f"  spawn qilingan 2 oqim bir xilmi: "
          f"{np.array_equal(oqim[0].random(3), oqim[1].random(3))}")
    oqim2 = np.random.default_rng(123).spawn(2)
    print(f"  qayta spawn - 1-oqim takrorlandimi: "
          f"{np.array_equal(np.random.default_rng(123).spawn(2)[0].random(3), oqim2[0].random(3))}")

    print("\n=== 4. DataLoader: generator global holatdan mustaqil ===")
    ds = TensorDataset(torch.arange(8))

    def tartib(oraliq):
        torch.manual_seed(0)
        if oraliq:
            torch.rand(5)                     # masalan, model init o'zgardi
        g = torch.Generator().manual_seed(7)
        return next(iter(DataLoader(ds, batch_size=8, shuffle=True,
                                    generator=g)))[0].tolist()

    print(f"  generator(7):                 {tartib(False)}")
    print(f"  generator(7), oraliqda rand:  {tartib(True)}")

    print("\n=== 5. PYTHONHASHSEED: str hash i jarayonga bog'liq ===")
    kod = "print(hash('salom'))"
    print(f"  =0 va =0 (ikki jarayon) teng: {jarayon(kod, 0) == jarayon(kod, 0)}")
    print(f"  =0 va =1 teng:                {jarayon(kod, 0) == jarayon(kod, 1)}")
    tasodifiy = {jarayon(kod, None) for _ in range(5)}
    print(f"  o'rnatilmagan: 5 jarayonda {len(tasodifiy)} xil qiymat")
    ichida = ("import os; os.environ['PYTHONHASHSEED'] = '0'; "
              "print(hash('salom'))")
    print(f"  skript ICHIDA '0' qilingan (tashqarida 1 va 2) teng: "
          f"{jarayon(ichida, 1) == jarayon(ichida, 2)}")
    kod_int = "print(hash(12345), hash((1, 2)))"
    print(f"  int va tuple hash i (=0 va =1) teng: "
          f"{jarayon(kod_int, 0) == jarayon(kod_int, 1)}")

    print("\n=== 6. Oqibat: list(set(...)) bilan kategoriya kodlash ===")
    kod = ("s = {'Andijon','Buxoro','Namangan','Samarqand','Toshkent',"
           "'Xorazm','Qarshi','Nukus'}; print(list(s).index('Toshkent'))")
    print(f"  PYTHONHASHSEED 0..5 da 'Toshkent' kodi: "
          f"{[int(jarayon(kod, hs)) for hs in range(6)]}")
    kod_s = kod.replace("list(s)", "sorted(s)")
    print(f"  sorted() bilan:                        "
          f"{[int(jarayon(kod_s, hs)) for hs in range(6)]}")


if __name__ == "__main__":
    main()

=== 1. Uch generator - uch alohida urug' ===
  seed_everything(0) dan keyin:   [0.8444, 0.5488, 0.4963]
  faqat random.seed(0) dan keyin: [0.8444, 0.7152, 0.7682]
  mos keladi: python True, numpy False, torch False

=== 2. sklearn random_state: None va int ===
  None, global seed(0):              teng True
  None, oraliqda boshqa np.random:   teng False
  int 42, oraliqda boshqa np.random: teng True

=== 3. numpy Generator: mustaqil oqimlar ===
  default_rng(123) birinchi 3: [0.6824, 0.0538, 0.2204]
  spawn qilingan 2 oqim bir xilmi: False
  qayta spawn - 1-oqim takrorlandimi: True

=== 4. DataLoader: generator global holatdan mustaqil ===
  generator(7):                 [1, 2, 3, 5, 7, 4, 6, 0]
  generator(7), oraliqda rand:  [1, 2, 3, 5, 7, 4, 6, 0]

=== 5. PYTHONHASHSEED: str hash i jarayonga bog'liq ===
  =0 va =0 (ikki jarayon) teng: True
  =0 va =1 teng:                False
  o'rnatilmagan: 5 jarayonda 5 xil qiymat
  skript ICHIDA '0' qilingan (tashqarida 1 va 2) teng: False
  int va tuple hash i (=0 va =1) teng: True

=== 6. Oqibat: list(set(...)) bilan kategoriya kodlash ===
  PYTHONHASHSEED 0..5 da 'Toshkent' kodi: [0, 0, 2, 6, 7, 0]
  sorted() bilan:                        [6, 6, 6, 6, 6, 6]

"""Reproduksiya manifesti: muhit, ma'lumot va konfig xeshi, tekshirish."""

import copy
import hashlib
import importlib.metadata as md
import json
import os
import platform
import subprocess
import tempfile
from pathlib import Path

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score

PAKETLAR = ["numpy", "pandas", "scikit-learn", "scipy", "joblib"]
MUHIT_ROYXAT = ["PYTHONHASHSEED", "OMP_NUM_THREADS", "MKL_NUM_THREADS"]  # sirlar EMAS


def xesh(b):
    return hashlib.sha256(b).hexdigest()[:12]


def konfig_xesh(konfig):
    return xesh(json.dumps(konfig, sort_keys=True, separators=(",", ":")).encode())


def lock(paketlar):
    return {p: md.version(p) for p in sorted(paketlar)}


def git_holati(papka):
    try:
        r = subprocess.run(["git", "rev-parse", "HEAD"], cwd=papka,
                           capture_output=True, text=True, timeout=10)
    except FileNotFoundError:
        return {"commit": None, "sabab": "git topilmadi"}
    if r.returncode != 0:
        return {"commit": None, "sabab": "git repozitoriy emas"}
    s = subprocess.run(["git", "status", "--porcelain"], cwd=papka,
                       capture_output=True, text=True, timeout=10)
    return {"commit": r.stdout.strip(), "toza": s.stdout.strip() == ""}


def malumot_yarat(yol, seed=0, n=3000):
    rng = np.random.default_rng(seed)
    df = pd.DataFrame({"x1": rng.normal(size=n).round(4),
                       "x2": rng.normal(size=n).round(4),
                       "x3": rng.integers(0, 5, n)})
    logit = 1.2 * df["x1"] - 0.8 * df["x2"] * (df["x3"] > 2) + 0.3
    df["y"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    df.to_csv(yol, index=False, lineterminator="\n")   # LF - OS dan mustaqil


def orgat(df, konfig):
    tr, te = df.iloc[:2000], df.iloc[2000:]
    X = ["x1", "x2", "x3"]
    m = HistGradientBoostingClassifier(
        max_iter=konfig["max_iter"], learning_rate=konfig["lr"],
        random_state=konfig["urug"]).fit(tr[X], tr["y"])
    p = m.predict_proba(te[X])[:, 1]
    return {"auc": round(float(roc_auc_score(te["y"], p)), 6),
            "bashorat_xesh": xesh(np.round(p, 8).tobytes())}


def manifest_yarat(papka, malumot_yol, konfig, natija):
    return {
        "python": platform.python_version(),
        "platforma": {"tizim": platform.system(), "arxitektura": platform.machine()},
        "paketlar": lock(PAKETLAR),
        "git": git_holati(papka),
        "muhit": {k: os.environ.get(k) for k in MUHIT_ROYXAT},
        "malumot": {"fayl": malumot_yol.name,
                    "sha256": xesh(malumot_yol.read_bytes())},
        "konfig": konfig, "konfig_xesh": konfig_xesh(konfig),
        "natija": natija,
    }


def tekshir(manifest, papka):
    """Hozirgi muhit va fayllarni manifest bilan solishtiradi."""
    farqlar = []
    if manifest["python"] != platform.python_version():
        farqlar.append(f"python: {manifest['python']} != {platform.python_version()}")
    joriy = lock(manifest["paketlar"])
    for p, v in manifest["paketlar"].items():
        if joriy[p] != v:
            farqlar.append(f"OGOHLANTIRISH {p}: manifest {v}, hozir {joriy[p]}")
    yol = Path(papka) / manifest["malumot"]["fayl"]
    if xesh(yol.read_bytes()) != manifest["malumot"]["sha256"]:
        farqlar.append("TO'XTATISH: ma'lumot xeshi mos emas")
    if konfig_xesh(manifest["konfig"]) != manifest["konfig_xesh"]:
        farqlar.append("TO'XTATISH: konfig o'zgartirilgan")
    return farqlar


def main() -> None:
    with tempfile.TemporaryDirectory() as t:
        papka = Path(t)
        yol = papka / "malumot.csv"
        malumot_yarat(yol)
        konfig = {"urug": 0, "lr": 0.1, "max_iter": 100}

        print("=== 1. Konfig xeshi: kalit tartibi ahamiyatsiz, qiymat - muhim ===")
        k2 = {"max_iter": 100, "lr": 0.1, "urug": 0}
        k3 = dict(konfig, lr=0.05)
        print(f"  konfig           {konfig_xesh(konfig)}")
        print(f"  tartibi boshqa   {konfig_xesh(k2)}  teng: "
              f"{konfig_xesh(k2) == konfig_xesh(konfig)}")
        print(f"  lr=0.05          {konfig_xesh(k3)}  teng: "
              f"{konfig_xesh(k3) == konfig_xesh(konfig)}")

        print("\n=== 2. O'qitish va manifest ===")
        natija = orgat(pd.read_csv(yol), konfig)
        m = manifest_yarat(papka, yol, konfig, natija)
        (papka / "manifest.json").write_text(json.dumps(m, indent=2),
                                             encoding="utf-8")
        print(f"  kalitlar: {list(m)}")
        print(f"  paketlar (lock): {len(m['paketlar'])} ta, masalan "
              f"'scikit-learn=={m['paketlar']['scikit-learn']}'")
        print(f"  muhit o'zgaruvchilari (faqat ro'yxatdagilar): {sorted(m['muhit'])}")
        print(f"  git: {m['git']}")
        print(f"  ma'lumot sha256: {m['malumot']['sha256']}")
        print(f"  natija: {natija}")

        print("\n=== 3. requirements.txt (pin) ===")
        req = "\n".join(f"{p}=={v}" for p, v in m["paketlar"].items())
        (papka / "requirements.txt").write_text(req + "\n", encoding="utf-8")
        for qator in req.splitlines():
            print(f"  {qator}")

        print("\n=== 4. Manifestdan qayta ishga tushirish ===")
        m2 = json.loads((papka / "manifest.json").read_text(encoding="utf-8"))
        print(f"  tekshiruv farqlari: {tekshir(m2, papka) or 'yoq'}")
        qayta = orgat(pd.read_csv(yol), m2["konfig"])
        print(f"  qayta natija: {qayta}")
        print(f"  aynan teng: {qayta == m2['natija']}")

        print("\n=== 5. Buzilishlarni ushlash ===")
        df = pd.read_csv(yol)
        df.loc[17, "x1"] = df.loc[17, "x1"] + 0.0001        # bitta qiymat
        df.to_csv(yol, index=False, lineterminator="\n")
        print(f"  1 qiymat o'zgardi: {tekshir(m2, papka)}")
        print(f"  shu o'zgarishdan natija o'zgardimi: "
              f"{orgat(df, m2['konfig']) != m2['natija']}")
        malumot_yarat(yol)                                   # tiklash
        crlf = yol.read_bytes().replace(b"\n", b"\r\n")      # Windows satr oxiri
        (papka / "crlf.csv").write_bytes(crlf)
        print(f"  CRLF nusxa: xesh teng {xesh(crlf) == m2['malumot']['sha256']}, "
              f"pandas jadvali teng {pd.read_csv(papka / 'crlf.csv').equals(pd.read_csv(yol))}")
        m3 = copy.deepcopy(m2)
        m3["paketlar"]["scikit-learn"] = "1.4.2"            # boshqa muhitda yozilgan
        m3["konfig"]["lr"] = 0.2                             # qo'lda o'zgartirilgan
        for f in tekshir(m3, papka):
            print(f"  {f}")


if __name__ == "__main__":
    main()

=== 1. Konfig xeshi: kalit tartibi ahamiyatsiz, qiymat - muhim ===
  konfig           507dfdb81878
  tartibi boshqa   507dfdb81878  teng: True
  lr=0.05          c3db07c8be99  teng: False

=== 2. O'qitish va manifest ===
  kalitlar: ['python', 'platforma', 'paketlar', 'git', 'muhit', 'malumot', 'konfig', 'konfig_xesh', 'natija']
  paketlar (lock): 5 ta, masalan 'scikit-learn==1.9.1'
  muhit o'zgaruvchilari (faqat ro'yxatdagilar): ['MKL_NUM_THREADS', 'OMP_NUM_THREADS', 'PYTHONHASHSEED']
  git: {'commit': None, 'sabab': 'git repozitoriy emas'}
  ma'lumot sha256: 952c6655a350
  natija: {'auc': 0.716766, 'bashorat_xesh': '818897a04902'}

=== 3. requirements.txt (pin) ===
  joblib==1.6.0
  numpy==2.5.3
  pandas==3.0.6
  scikit-learn==1.9.1
  scipy==1.18.1

=== 4. Manifestdan qayta ishga tushirish ===
  tekshiruv farqlari: yoq
  qayta natija: {'auc': 0.716766, 'bashorat_xesh': '818897a04902'}
  aynan teng: True

=== 5. Buzilishlarni ushlash ===
  1 qiymat o'zgardi: ["TO'XTATISH: ma'lumot xeshi mos emas"]
  shu o'zgarishdan natija o'zgardimi: False
  CRLF nusxa: xesh teng False, pandas jadvali teng True
  OGOHLANTIRISH scikit-learn: manifest 1.4.2, hozir 1.9.1
  TO'XTATISH: konfig o'zgartirilgan

"""Urug'ga sezgirlik: bitta urug'dagi natija - bitta namuna."""

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, Y = make_classification(n_samples=2000, n_features=20, n_informative=5,
                           n_redundant=5, flip_y=0.08, class_sep=0.8,
                           random_state=3)
MODELLAR = {"A": {"max_features": "sqrt"},      # 4 belgi har bo'linishda
            "B": {"max_features": 0.3}}         # 6 belgi - sekinroq


def yurish(nom, bolish_urug, model_urug):
    Xtr, Xte, ytr, yte = train_test_split(X, Y, test_size=0.3,
                                          random_state=bolish_urug, stratify=Y)
    m = RandomForestClassifier(n_estimators=40, random_state=model_urug,
                               n_jobs=1, **MODELLAR[nom]).fit(Xtr, ytr)
    return roc_auc_score(yte, m.predict_proba(Xte)[:, 1])


def main() -> None:
    urug = range(15)
    a = np.array([yurish("A", s, s) for s in urug])
    b = np.array([yurish("B", s, s) for s in urug])
    d = a - b

    print("=== 1. Bitta urug' - 'aniq' javob ===")
    for s in range(4):
        print(f"  urug' {s}: A {a[s]:.4f}, B {b[s]:.4f} -> g'olib "
              f"{'A' if d[s] > 0 else 'B'} ({d[s]:+.4f})")
    print(f"  takror: urug' 0 dagi A qayta hisoblandi, teng: "
          f"{yurish('A', 0, 0) == a[0]}")

    print("\n=== 2. 15 urug' (bo'lish va model urug'i birga) ===")
    for nom, v in [("A", a), ("B", b)]:
        print(f"  {nom}: {v.mean():.4f} +- {v.std(ddof=1):.4f}  "
              f"[{v.min():.4f}, {v.max():.4f}]")
    print(f"  A g'olib bo'lgan urug'lar: {int((d > 0).sum())}/15")
    se = d.std(ddof=1) / np.sqrt(len(d))
    se_juftsiz = np.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))
    print(f"  farq A-B: {d.mean():+.4f}; juftlashgan SE {se:.4f}, "
          f"juftlashmagan SE {se_juftsiz:.4f}")
    if abs(d.mean()) > 2 * se:
        print(f"  farq SEZILARLI -> {'A' if d.mean() > 0 else 'B'}")
    else:
        print("  farq sezilarli emas -> qoida: arzonroq A")

    print("\n=== 3. Tarqoqlik manbalari: faqat bittasini o'zgartirib ===")
    faqat_bolish = np.array([yurish("A", s, 0) for s in urug])
    faqat_model = np.array([yurish("A", 0, s) for s in urug])
    print(f"  faqat bo'lish urug'i: std {faqat_bolish.std(ddof=1):.4f}")
    print(f"  faqat model urug'i:   std {faqat_model.std(ddof=1):.4f}")
    katta = "bo'lish" if faqat_bolish.std() > faqat_model.std() else "model"
    print(f"  asosiy tarqoqlik manbai: {katta} urug'i")

    print("\n=== 4. 'Eng yaxshi urug'' ni hisobot qilish ===")
    eng = int(b.argmax())
    print(f"  B ning eng yaxshi urug'i {eng}: {b[eng]:.4f}; o'rtacha {b.mean():.4f}; "
          f"optimizm {b[eng] - b.mean():+.4f} "
          f"({(b[eng] - b.mean()) / b.std(ddof=1):.1f} std)")
    print(f"  shu urug'da B ning A dan ustunligi: {-d[eng]:+.4f}; "
          f"o'rtacha ustunlik {-d.mean():+.4f}")
    print("  hisobotga: o'rtacha +- std (15 urug'), juftlashgan farq va SE")


if __name__ == "__main__":
    main()

=== 1. Bitta urug' - 'aniq' javob ===
  urug' 0: A 0.9238, B 0.9264 -> g'olib B (-0.0025)
  urug' 1: A 0.9111, B 0.9042 -> g'olib A (+0.0069)
  urug' 2: A 0.9355, B 0.9293 -> g'olib A (+0.0062)
  urug' 3: A 0.9124, B 0.9175 -> g'olib B (-0.0051)
  takror: urug' 0 dagi A qayta hisoblandi, teng: True

=== 2. 15 urug' (bo'lish va model urug'i birga) ===
  A: 0.9226 +- 0.0106  [0.9082, 0.9421]
  B: 0.9246 +- 0.0100  [0.9042, 0.9411]
  A g'olib bo'lgan urug'lar: 6/15
  farq A-B: -0.0020; juftlashgan SE 0.0013, juftlashmagan SE 0.0038
  farq sezilarli emas -> qoida: arzonroq A

=== 3. Tarqoqlik manbalari: faqat bittasini o'zgartirib ===
  faqat bo'lish urug'i: std 0.0097
  faqat model urug'i:   std 0.0033
  asosiy tarqoqlik manbai: bo'lish urug'i

=== 4. 'Eng yaxshi urug'' ni hisobot qilish ===
  B ning eng yaxshi urug'i 10: 0.9411; o'rtacha 0.9246; optimizm +0.0165 (1.6 std)
  shu urug'da B ning A dan ustunligi: -0.0011; o'rtacha ustunlik +0.0020
  hisobotga: o'rtacha +- std (15 urug'), juftlashgan farq va SE

kod = {h: i for i, h in enumerate(set(df["hudud"]))}          # ⚠️ jarayonga bog'liq
kod = {h: i for i, h in enumerate(sorted(set(df["hudud"])))}  # ✅

fayllar = os.listdir(papka)                                   # ⚠️ OS ga bog'liq
fayllar = sorted(os.listdir(papka))                           # ✅

random.seed(0)                                                # ⚠️ numpy/torch erkin
seed_everything(0); model = RandomForestClassifier(random_state=0)   # ✅

savat = hash(foydalanuvchi_id) % 100                          # ⚠️ har jarayonda boshqa
savat = int(hashlib.md5(foydalanuvchi_id.encode()).hexdigest(), 16) % 100   # ✅

os.environ["PYTHONHASHSEED"] = "0"                            # ⚠️ kech
# PYTHONHASHSEED=0 python orgat.py  (yoki Dockerfile da ENV)  # ✅ jarayondan oldin

# requirements.txt: scikit-learn                              # ⚠️
# requirements.txt: scikit-learn==1.9.1  (+ lock fayl)        # ✅

manifest["muhit"] = dict(os.environ)                          # ⚠️ sirlar tushadi
manifest["muhit"] = {k: os.environ.get(k) for k in RUXSAT_ROYXATI}   # ✅

1.  # takrorlanuvchanlik va reproduksiya farqi?
2.  # natija qaytarilmasligining olti manbasi?
3.  # random.seed(0) numpy ga ta'sir qiladimi?
4.  # random_state=None nima uchun mo'rt?
5.  # hash("salom") ikki yangi jarayonda tengmi (PYTHONHASHSEED o'rnatilmagan)?
6.  # hash(12345) chi?
7.  # os.environ["PYTHONHASHSEED"] = "0" skript boshida ishlaydimi?
8.  # dict tartibi xeshga bog'liqmi?
9.  # manifestda nima bo'ladi?
10. # konfig xeshi uchun JSON qanday yoziladi?
11. # bir xil jadval CRLF va LF da saqlansa, fayl xeshi tengmi?
12. # tarqoqlikning asosiy manbai odatda qaysi urug'?

1.  kod = {h: i for i, h in enumerate(set(df["hudud"]))}

2.  for f in os.listdir(papka): qismlar.append(pd.read_csv(f))

3.  model = RandomForestClassifier()

4.  savat = hash(user_id) % 10

5.  manifest["muhit"] = dict(os.environ)