IlmHamroh
Python kursi/Avtomatlashtirish3/10-dars16 daqiqa
Mundarija (22)

26.3-dars: Web scraping — requests

26-QISM — AVTOMATLASHTIRISH · 3-dars

1. Kirish va motivatsiya

Internet — ulkan ma'lumot manbai: narxlar, yangiliklar, ob-havo, valyuta kurslari, mahsulot ma'lumoti. Ko'p ma'lumot faqat veb sahifada (API yo'q). Uni qo'lda ko'chirish — sekin, zerikarli. Web scraping — veb sahifalardan ma'lumotni avtomatik yig'ish. Birinchi qadam — veb sahifani yuklab olish (HTTP so'rov). Bu — requests kutubxonasi.

requests — Python'ning HTTP kutubxonasi (veb bilan ishlash): GET (ma'lumot olish — sahifa, API), POST (ma'lumot yuborish), status kod (200 OK, 404 topilmadi), headers (metama'lumot — User-Agent), params (so'rov parametrlari). Bu 20-qism (FastAPI — server tomoni) ning mijoz tomoni: siz so'rov yuborasiz, javob olasiz. Muhim tushunchalar: HTTP metodlari, status kodlar, JSON javob, so'rov sozlash. requests — internetdan ma'lumot olishning asosi. Web scraping va API integratsiyasining boshi.

Real vaziyat. Bir tahlilchi har kuni 10 ta veb-saytdan valyuta kurslarini qo'lda ko'chirardi — 20 daqiqa. Python requests skript yozildi: har saytga GET so'rov, JSON javobdan kursni oldi, jadvalga yig'di. Bir soniyada. Avtomatik, xatosiz. requests — internetdagi ma'lumotni avtomatik yig'di. Qo'lda ko'chirish o'rniga skript.

Bu darsda requests bilan veb so'rovlarni o'rganamiz.

Bu darsda:

  • HTTP va requests nima
  • GET so'rov va javob
  • Status kodlar
  • JSON javob (API)
  • So'rov parametrlari (params)
  • Headers (User-Agent)
  • Xatolarni boshqarish
  • Amaliy: API'dan ma'lumot

ℹ Misollarda urllib.parse (URL) va mock javob (soxta — deterministik) bilan sinaladi. Real requests tarmoq talab qiladi (bu yerda chaqirilmaydi).

2. Nazariya — chuqur tushuntirish

2.1. HTTP va requests nima

Veb — HTTP so'rov/javob:

python
import requests
response = requests.get("https://example.com")   # so'rov
response.status_code    # 200 (OK)
response.text           # HTML matni
response.json()         # JSON (API bo'lsa)

HTTP (HyperText Transfer Protocol) — veb aloqa protokoli: so'rov (mijoz → server) va javob (server → mijoz). requests — Python HTTP kutubxonasi (so'rov yuborib javob olish): requests.get(url) (ma'lumot olish), response.status_code (holat), .text (HTML/matn), .json() (JSON — API). Bu 20-qism (FastAPI — server) ning mijoz tomoni. Har veb amal — HTTP so'rov (brauzer ham). requests — veb bilan ishlashning asosi.

2.2. GET so'rov va javob

Ma'lumot olish:

python
response = requests.get("https://api.example.com/data")
response.status_code    # 200
response.text           # javob matni
response.headers        # javob metama'lumoti
response.url            # so'ralgan URL

GET so'rov (requests.get(url)) — serverdan ma'lumot olish (eng ko'p ishlatiladigan): veb sahifa, API ma'lumoti, rasm. Javob (response): .status_code (holat — 200 OK), .text (matn — HTML yoki JSON), .headers (metama'lumot — tur, hajm), .content (binar — rasm). GET — ma'lumot o'qish (o'zgartmaydi — xavfsiz). POST — yuborish 26.7-bob. Web scraping asosan GET (sahifa yuklash).

2.3. Status kodlar

Javob holati:

Kod Ma'no
200 OK (muvaffaqiyat)
404 Not Found (topilmadi)
403 Forbidden (ruxsat yo'q)
500 Server xatosi
429 Too Many Requests (ko'p so'rov)

Status kodlar — javob holati (server javobi): 2xx (muvaffaqiyat — 200 OK), 4xx (mijoz xatosi — 404 topilmadi, 403 ruxsat yo'q, 429 ko'p so'rov), 5xx (server xatosi — 500). response.status_code (tekshir — 200mi?), response.raise_for_status() (xato bo'lsa istisno). Muhim: har so'rovdan keyin status tekshir (200 emasmi?). Bu 20-qism (FastAPI status kodlari) ning mijoz tomoni. Status — so'rov muvaffaqiyati.

2.4. JSON javob (API)

API ma'lumoti — JSON:

python
response = requests.get("https://api.example.com/users")
data = response.json()          # JSON → Python (dict/list)
data["users"][0]["name"]        # ma'lumotga kirish
# 19-qism: JSON — API standart formati

JSON javob (response.json()) — API ma'lumotni JSON'da qaytaradi (19-qism): .json() (JSON matnni Python dict/list'ga). Bu API (Application Programming Interface — dastur uchun ma'lumot) — HTML'dan farq (inson uchun). API bilan ishlash oson (data["kalit"] — 19-qism JSON). Ko'p sayt API beradi (rasmiy ma'lumot — narx, ob-havo). Web scraping (HTML) va API (JSON) — ikki usul: API afzal (toza, barqaror). .json() — API bilan ishlashning asosi.

2.5. So'rov parametrlari (params)

So'rovni sozlash:

python
params = {"q": "python", "page": 2, "limit": 10}
response = requests.get("https://api.example.com/search", params=params)
# URL: .../search?q=python&page=2&limit=10

So'rov parametrlari (params) — so'rovni sozlash (qidirish, sahifa, filtr): requests.get(url, params={"q": "python"}) → URL'ga ?q=python qo'shiladi. requests avtomatik URL quradi (kodlash, birlashtirish). Bu API'dan aniq ma'lumot so'rash (q qidirish, page sahifa, limit cheklov). Qo'lda URL qurish o'rniga params (xavfsiz, toza). Parametrlar — so'rovni moslash.

2.6. Headers (User-Agent)

So'rov metama'lumoti:

python
headers = {"User-Agent": "MyBot/1.0", "Accept": "application/json"}
response = requests.get(url, headers=headers)
# User-Agent: kim so'rayapti (brauzer/bot)
# Authorization: kalit (API auth)

Headers — so'rov metama'lumoti: User-Agent (kim so'rayapti — brauzer yoki bot; ba'zi sayt botni bloklaydi — real User-Agent kerak), Authorization (API kaliti — 25.10, 20.7), Accept (kutilgan format — JSON). requests.get(url, headers={...}). Muhim: ba'zi sayt User-Agent'siz so'rovni rad qiladi (403). Odob: to'g'ri User-Agent (bot ekanini ayt). Headers — so'rov konteksti (kim, nima kutadi).

2.7. Xatolarni boshqarish

Veb so'rov — kutilmagan:

python
try:
    response = requests.get(url, timeout=10)   # timeout muhim
    response.raise_for_status()                 # 4xx/5xx → istisno
    data = response.json()
except requests.RequestException as e:
    logging.error(f"So'rov xatosi: {e}")

Xatolarni boshqarish — veb so'rov kutilmagan (tarmoq, server): timeout=10 (muhim — javob kutmasdan cheksiz osilib qolmaslik), raise_for_status() (4xx/5xx → istisno), try/except requests.RequestException (tarmoq, timeout, xato). Veb — ishonchsiz (server o'chiq, sekin, xato) — har so'rov xato bo'lishi mumkin. timeout va try/except — ishonchli scraping. "Veb kutilmagan — tayyor bo'l" (26.1 skript xato boshqarish kabi).

2.8. Web scraping — mas'uliyat

Web scraping kuchli, lekin mas'uliyatli: serverga yuk (ko'p so'rov — serverni sekinlashtiradi; time.sleep bilan sekinlash), robots.txt (sayt qoidasi — nima scraping mumkin, 26.6), qonuniy (ba'zi ma'lumot himoyalangan — mualliflik, shaxsiy), API afzal (bor bo'lsa — API, HTML scraping emas). Odob: sekin so'rov (yukni kamaytir), User-Agent (bot ekanini ayt), robots.txt hurmat. Bu 26.6 (scraping etikasi) da chuqurroq. "Kuch bilan mas'uliyat" — scraping boshqalar serveriga ta'sir qiladi. Ehtiyotli va odobli scraping.

3. Tez ma'lumotnoma

python
import requests

# GET so'rov:
response = requests.get("https://api.example.com/data", timeout=10)
response.status_code       # 200
response.text              # matn (HTML/JSON)
response.json()            # JSON → dict/list
response.headers           # javob metama'lumoti

# params (so'rovni sozlash):
requests.get(url, params={"q": "python", "page": 2})
# URL: ...?q=python&page=2

# headers:
requests.get(url, headers={"User-Agent": "MyBot/1.0"})

# POST (yuborish, 26.7):
requests.post(url, json={"key": "value"})

# xato boshqarish:
try:
    r = requests.get(url, timeout=10)
    r.raise_for_status()
except requests.RequestException as e:
    ...

# URL qurish (urllib):
from urllib.parse import urlencode
url + "?" + urlencode(params)

requests xulosasi

HTTP: so'rov/javob · GET (ma'lumot ol) · status (200 OK, 404) · .json() (API)
params (sozlash) · headers (User-Agent) · timeout + raise_for_status (xato)
Web scraping — mas'uliyatli (yuk, robots.txt, API afzal)

4. Batafsil misollar

Misollarda urllib.parse (URL) va mock javob (soxta Response — deterministik) bilan sinaladi. Real requests.get tarmoq talab qiladi.

Misol 1 — URL qurish va parametrlar

python
"""URL qurish (urlencode); parametrlar; URL tahlil (urlparse) — so'rov sozlash."""

from urllib.parse import parse_qs, urlencode, urlparse


def main() -> None:
    print("=== 1. So'rov parametrlari (urlencode) ===")
    base = "https://api.example.com/search"
    params = {"q": "python", "page": 2, "limit": 10}
    url = base + "?" + urlencode(params)
    print(f"  {url}")

    print("\n=== 2. URL tahlil (urlparse) ===")
    p = urlparse(url)
    print(f"  domen: {p.netloc}")
    print(f"  yo'l: {p.path}")
    print(f"  parametrlar: {parse_qs(p.query)}")

    print("\n=== 3. Maxsus belgilar (avtomatik kodlash) ===")
    params2 = {"q": "web scraping", "til": "o'zbek"}
    print(f"  {urlencode(params2)}")

    print("\n=== 4. requests bilan (real) ===")
    print("  requests.get(url, params={'q': 'python'})")
    print("  requests avtomatik URL quradi (params)")
    print("  ⭐ params — so'rovni sozlash (qidirish, sahifa)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. So'rov parametrlari (urlencode) ===
  https://api.example.com/search?q=python&page=2&limit=10

=== 2. URL tahlil (urlparse) ===
  domen: api.example.com
  yo'l: /search
  parametrlar: {'q': ['python'], 'page': ['2'], 'limit': ['10']}

=== 3. Maxsus belgilar (avtomatik kodlash) ===
  q=web+scraping&til=o%27zbek

=== 4. requests bilan (real) ===
  requests.get(url, params={'q': 'python'})
  requests avtomatik URL quradi (params)
  ⭐ params — so'rovni sozlash (qidirish, sahifa)

Nima ko'rsatdi: 2.5-bo'lim.

Misol 2 — Javob va status kodlar (mock)

python
"""mock Response: status_code, json(), raise_for_status; status kodlar ma'nosi."""

import json


class MockResponse:
    """Soxta requests.Response (deterministik sinash)."""

    def __init__(self, status: int, data: dict) -> None:
        self.status_code = status
        self._data = data

    def json(self) -> dict:
        return self._data

    @property
    def text(self) -> str:
        return json.dumps(self._data)

    def raise_for_status(self) -> None:
        if self.status_code >= 400:
            raise Exception(f"HTTP {self.status_code}")


def main() -> None:
    print("=== 1. Muvaffaqiyatli javob (200) ===")
    resp = MockResponse(200, {"results": ["a", "b", "c"], "total": 3})
    print(f"  status: {resp.status_code}, OK: {resp.status_code == 200}")

    print("\n=== 2. JSON javob (API) ===")
    data = resp.json()
    print(f"  total: {data['total']}, birinchi: {data['results'][0]}")

    print("\n=== 3. Status kodlar ma'nosi ===")
    kodlar = {200: "OK", 404: "Not Found", 403: "Forbidden", 500: "Server xato"}
    for kod, ma_no in kodlar.items():
        print(f"  {kod}: {ma_no}")

    print("\n=== 4. Xato javob (raise_for_status) ===")
    xato_resp = MockResponse(404, {})
    try:
        xato_resp.raise_for_status()
    except Exception as e:
        print(f"  404 → istisno: {e}")
    print("  ⭐ status_code (holat), json() (ma'lumot), raise_for_status")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Muvaffaqiyatli javob (200) ===
  status: 200, OK: True

=== 2. JSON javob (API) ===
  total: 3, birinchi: a

=== 3. Status kodlar ma'nosi ===
  200: OK
  404: Not Found
  403: Forbidden
  500: Server xato

=== 4. Xato javob (raise_for_status) ===
  404 → istisno: HTTP 404
  ⭐ status_code (holat), json() (ma'lumot), raise_for_status

Nima ko'rsatdi: 2.2, 2.3, 2.4-bo'limlar.

Misol 3 — Headers va xato boshqarish (mock)

python
"""headers (User-Agent); xato boshqarish (timeout, try/except) — ishonchli so'rov namunasi."""


def build_headers(bot_nomi: str) -> dict:
    return {
        "User-Agent": f"{bot_nomi}/1.0",
        "Accept": "application/json",
    }


def safe_request(url: str, mock_status: int) -> dict:
    """Xato boshqarish bilan so'rov (mock status)."""
    try:
        if mock_status >= 400:
            raise Exception(f"HTTP {mock_status}")
        return {"holat": "muvaffaqiyat", "status": mock_status}
    except Exception as e:
        return {"holat": "xato", "sabab": str(e)}


def main() -> None:
    print("=== 1. Headers qurish ===")
    headers = build_headers("MyBot")
    print(f"  User-Agent: {headers['User-Agent']}")
    print(f"  Accept: {headers['Accept']}")

    print("\n=== 2. Nega User-Agent ===")
    print("  ba'zi sayt botni bloklaydi (User-Agent tekshiradi)")
    print("  odob: bot ekanini ayt")

    print("\n=== 3. Muvaffaqiyatli so'rov (mock 200) ===")
    print(f"  {safe_request('https://api.example.com', 200)}")

    print("\n=== 4. Xato so'rov (mock 403) ===")
    print(f"  {safe_request('https://api.example.com', 403)}")
    print("  timeout + raise_for_status + try/except — ishonchli")
    print("  ⭐ headers (User-Agent) + xato boshqarish")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Headers qurish ===
  User-Agent: MyBot/1.0
  Accept: application/json

=== 2. Nega User-Agent ===
  ba'zi sayt botni bloklaydi (User-Agent tekshiradi)
  odob: bot ekanini ayt

=== 3. Muvaffaqiyatli so'rov (mock 200) ===
  {'holat': 'muvaffaqiyat', 'status': 200}

=== 4. Xato so'rov (mock 403) ===
  {'holat': 'xato', 'sabab': 'HTTP 403'}
  timeout + raise_for_status + try/except — ishonchli
  ⭐ headers (User-Agent) + xato boshqarish

Nima ko'rsatdi: 2.6, 2.7-bo'limlar.

Misol 4 — Amaliy: API'dan ma'lumot yig'ish (mock)

Real scraping: API'dan ma'lumot olish, JSON'ni qayta ishlash, natijani yig'ish. Bu — API integratsiyasining namunasi (deterministik mock bilan).

python
"""to'liq API ishi (mock): so'rov, JSON javob, qayta ishlash, statistika."""

import json


class MockAPI:
    """Soxta API (deterministik javob)."""

    def get(self, endpoint: str) -> dict:
        data = {
            "/kurslar": {"kurslar": [
                {"nom": "Python", "narx": 100, "til": "uz"},
                {"nom": "Go", "narx": 120, "til": "en"},
                {"nom": "Rust", "narx": 150, "til": "en"},
            ]},
            "/statistika": {"jami": 3, "o'rtacha_narx": 123.3},
        }
        return {"status": 200, "data": data.get(endpoint, {})}


def main() -> None:
    api = MockAPI()

    print("=== 1. API'dan kurslar olish (GET) ===")
    resp = api.get("/kurslar")
    print(f"  status: {resp['status']}")
    kurslar = resp["data"]["kurslar"]
    print(f"  kurslar soni: {len(kurslar)}")

    print("\n=== 2. JSON'ni qayta ishlash ===")
    nomlar = [k["nom"] for k in kurslar]
    print(f"  nomlar: {nomlar}")

    print("\n=== 3. Filtrlash (en til) ===")
    en = [k["nom"] for k in kurslar if k["til"] == "en"]
    print(f"  en kurslar: {en}")

    print("\n=== 4. Statistika (hisoblash) ===")
    narxlar = [k["narx"] for k in kurslar]
    print(f"  jami narx: {sum(narxlar)}, o'rtacha: {round(sum(narxlar) / len(narxlar), 1)}")
    print("  ⭐ API — GET + JSON + qayta ishlash (avtomatik yig'ish)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. API'dan kurslar olish (GET) ===
  status: 200
  kurslar soni: 3

=== 2. JSON'ni qayta ishlash ===
  nomlar: ['Python', 'Go', 'Rust']

=== 3. Filtrlash (en til) ===
  en kurslar: ['Go', 'Rust']

=== 4. Statistika (hisoblash) ===
  jami narx: 370, o'rtacha: 123.3
  ⭐ API — GET + JSON + qayta ishlash (avtomatik yig'ish)

Nima ko'rsatdi: 2.1–2.8-bo'limlar.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"GET — ma'lumot yuborish" Olish (POST — yuborish)
"Status tekshirmaslik" 200mi? (raise_for_status)
"URL qo'lda qurish" params (avtomatik)
"User-Agent keraksiz" Ba'zi sayt bloklaydi
"timeout keraksiz" Cheksiz osilish xavfi
"Scraping erkin" Mas'uliyat (yuk, robots.txt)
"HTML = API" JSON (API) — toza, barqaror
"Veb so'rov ishonchli" Kutilmagan (xato boshqar)

6. Keng tarqalgan xatolar va yechimlari

1. timeout yo'q (osilish)

python
requests.get(url)   # cheksiz kutishi mumkin           # ⚠️
requests.get(url, timeout=10)                            # ✅

2. Status tekshirmaslik

python
data = requests.get(url).json()   # 404 bo'lsa?         # ⚠️
r = requests.get(url); r.raise_for_status()              # ✅

3. URL qo'lda qurish (kodlash)

python
url + "?q=" + qidiruv   # maxsus belgi buziladi         # ⚠️
requests.get(url, params={"q": qidiruv})                 # ✅

4. User-Agent'siz (bloklanish)

python
requests.get(url)   # ba'zan 403                         # ⚠️
requests.get(url, headers={"User-Agent": "..."})         # ✅

5. Ko'p so'rov (serverga yuk)

python
for u in urls: requests.get(u)   # tez, ko'p            # ⚠️
for u in urls: requests.get(u); time.sleep(1)            # ✅ sekinla

6. Xato boshqarilmagan

python
data = requests.get(url).json()   # tarmoq xato — crash  # ⚠️
try: ... except requests.RequestException: ...            # ✅

7. API o'rniga HTML scraping

python
# HTML parse (murakkab, buziluvchan)                     # ⚠️ (API bor bo'lsa)
requests.get(api_url).json()   # API afzal               # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 26.4-dars: BeautifulSoup — HTML tahlil
  • 26.7-dars: API integratsiya — chuqurroq
  • 19-qism (o'tilgan): JSON — API format
  • 20-qism (o'tilgan): FastAPI — server tomoni
  • 25.10-dars (o'tilgan): LLM API — requests bilan

8. Eng yaxshi amaliyotlar

  1. timeout (osilishni oldini oladi).

  2. Status tekshir (raise_for_status).

  3. params (URL qo'lda qurma).

  4. User-Agent (bot ekanini ayt).

  5. Xatolarni boshqar (try/except).

  6. Sekin so'rov (sleep — serverga yuk).

  7. API afzal (HTML scraping emas).

  8. robots.txt hurmat 26.6-bob.

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # HTTP nima?
2.  # requests nima?
3.  # GET nima?
4.  # status_code nima?
5.  # 200 va 404?
6.  # json() nima?
7.  # params nima?
8.  # headers nima?
9.  # User-Agent nima?
10. # timeout nega?
11. # raise_for_status nima?
12. # scraping mas'uliyatmi?
Javoblar
  1. Veb aloqa protokoli (so'rov/javob)
  2. Python HTTP kutubxonasi
  3. Ma'lumot olish so'rovi
  4. Javob holati
  5. 200 OK, 404 topilmadi
  6. JSON → Python (dict/list)
  7. So'rov parametrlari (sozlash)
  8. So'rov metama'lumoti
  9. Kim so'rayapti (bot/brauzer)
  10. Cheksiz osilishni oldini oladi
  11. 4xx/5xx → istisno
  12. Ha (yuk, robots.txt, API afzal)

Vazifa 2: Xatolarni tuzating

python
1.  requests.get(url)   # timeout yo'q       # timeout=10

2.  requests.get(url).json()   # status?     # raise_for_status

3.  url + "?q=" + qidiruv                     # params

4.  requests.get(url)   # 403                 # User-Agent

5.  for u in urls: requests.get(u)            # sleep
Javoblar
python
1.  requests.get(url, timeout=10)

2.  r = requests.get(url); r.raise_for_status()

3.  requests.get(url, params={"q": qidiruv})

4.  requests.get(url, headers={"User-Agent": "..."})

5.  ...; time.sleep(1)

Vazifa 3: URL/params

So'rov:

  1. urlencode
  2. urlparse
  3. Kodlash
  4. params

Vazifa 4: Javob/status

Mock:

  1. status_code
  2. json()
  3. Status kodlar
  4. raise_for_status

Vazifa 5: Headers/xato

Ishonchli:

  1. User-Agent
  2. timeout
  3. try/except
  4. Boshqarish

Vazifa 6: API

Yig'ish:

  1. GET
  2. JSON
  3. Qayta ishlash
  4. Statistika

Vazifa 7: O'ylash

Web scraping internetdagi ma'lumotni avtomatik yig'adi — kuchli. Lekin mas'uliyatli: serverga yuk (ko'p so'rov), robots.txt (sayt qoidasi), qonuniy (himoyalangan ma'lumot). Nima uchun "API afzal (HTML scraping emas)", va nega scraping "kuch bilan mas'uliyat" tamoyilini ko'rsatadi — boshqalar serveriga ta'sir qiladigan avtomatlashtirish nega ehtiyot talab qiladi?

Javob

Qisqa javob: API afzal (HTML scraping emas), chunki: API dastur uchun (toza JSON, barqaror — 19-qism), HTML inson uchun (murakkab, dizayn o'zgarsa buziladi). API bor bo'lsa — uni ishlat (oson, ishonchli); HTML scraping — oxirgi chora (API yo'q bo'lsa). Scraping "kuch bilan mas'uliyat": sizning skriptingiz boshqalar serveriga so'rov yuboradi — ko'p so'rov serverni sekinlashtiradi (yoki yiqitadi — DDoS kabi), qonuniy muammo (himoyalangan ma'lumot — mualliflik, shaxsiy), sayt qoidasi (robots.txt). Ehtiyot talab qiladi, chunki: bu sizning emas, boshqaning resursi (server, ma'lumot) — hurmat kerak (sekin so'rov, robots.txt, User-Agent). "Boshqaning uyiga mehmon" kabi — odob bilan. Avtomatlashtirish boshqalarga ta'sir qilsa (server yuki), mas'uliyat oshadi.

1. API vs HTML scraping

API HTML scraping
Dastur uchun (JSON) Inson uchun (HTML)
Toza, barqaror Murakkab, buziluvchan
Rasmiy Norasmiy
Afzal Oxirgi chora

2. Scraping — boshqaning resursi

Skript boshqalar serveriga so'rov — ularning resursi (server, tarmoq). Ko'p so'rov — yuk (sekinlashadi, yiqiladi).

3. Mas'uliyat sabablari

  • Server yuki: ko'p so'rov — DDoS kabi
  • Qonuniy: himoyalangan ma'lumot
  • Sayt qoidasi: robots.txt (26.6)
  • Odob: User-Agent, sekin so'rov

4. Kuch bilan mas'uliyat

Avtomatlashtirish boshqalarga ta'sir (server) → mas'uliyat. "Boshqaning resursi" — hurmat (sekin, robots.txt).

5. Muhandislik saboqlari

  1. API afzal (toza, barqaror)
  2. Scraping — boshqaning resursi
  3. Mas'uliyat (yuk, qonun, robots.txt)
  4. Kuch bilan mas'uliyat (ehtiyot)

6. Xulosa

  1. API afzal (HTML — oxirgi chora)
  2. Scraping boshqalar serveriga ta'sir
  3. Mas'uliyatli (yuk, robots.txt)
  4. Boshqaning resursi — hurmat

Nimani mustahkamlaydi: 2.1–2.8-bo'limlar.

Xulosa

Bu darsda requests bilan veb so'rovlarni o'rgandik.

Eng muhim uch fikr:

  1. HTTP, GET va status. HTTP — veb aloqa protokoli (so'rov/javob); requests — Python HTTP kutubxonasi. GET so'rov (requests.get(url)) — serverdan ma'lumot olish (sahifa, API): javob (response) — .status_code (holat), .text (HTML/matn), .json() (JSON), .headers. Status kodlar: 2xx (muvaffaqiyat — 200), 4xx (mijoz xatosi — 404 topilmadi, 403 ruxsat yo'q, 429 ko'p so'rov), 5xx (server xatosi). raise_for_status() (4xx/5xx → istisno). Har so'rovdan keyin status tekshir. Bu 20-qism (FastAPI server) ning mijoz tomoni.

  2. JSON, params, headers, xato. JSON javob (response.json()) — API ma'lumotni Python dict/list'ga (19-qism); API (dastur uchun — toza) vs HTML (inson uchun). So'rov parametrlari (params={"q": "python"}) — so'rovni sozlash (qidirish, sahifa, filtr — requests avtomatik URL quradi). Headers ({"User-Agent": "MyBot/1.0"}) — so'rov metama'lumoti (kim so'rayapti; ba'zi sayt botni bloklaydi — real User-Agent; Authorization — API kaliti). Xato boshqarish: timeout=10 (muhim — osilishni oldini oladi), raise_for_status(), try/except requests.RequestException (veb ishonchsiz).

  3. Web scraping — mas'uliyat. Web scraping kuchli (internetdan avtomatik ma'lumot), lekin mas'uliyatli: serverga yuk (ko'p so'rov — sekinlashadi/yiqiladi; time.sleep bilan sekinla), robots.txt (sayt qoidasi — 26.6), qonuniy (himoyalangan ma'lumot), API afzal (bor bo'lsa — toza, barqaror; HTML scraping — oxirgi chora). Scraping "kuch bilan mas'uliyat": skript boshqaning serveriga so'rov yuboradi (ularning resursi — hurmat kerak: sekin so'rov, User-Agent, robots.txt). Avtomatlashtirish boshqalarga ta'sir qilsa (server yuki), ehtiyot oshadi. "Boshqaning resursiga mehmon" — odob bilan. requests — internetdan ma'lumot olishning asosi (26.4 HTML tahlil, 26.7 API).

Keyingi darsda BeautifulSoup ni o'rganamiz: requests bilan yuklab olingan HTML sahifadan ma'lumotni ajratib olish — teglar, klasslar bo'yicha qidirish — HTML scraping'ning ikkinchi qismi.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!