IlmHamroh
Data Science va sun'iy intellekt/Malumot yigish9/12-dars17 daqiqa
Mundarija (22)

7.9-dars: Web scraping (BeautifulSoup)

7-QISM — MA'LUMOT YIG'ISH · 9-dars

1. Kirish va motivatsiya

Ba'zan ma'lumot faqat veb-sahifada ko'rinadi — API yo'q, fayl yo'q, baza yo'q. Masalan e'lon sayti, yangiliklar, narxlar. Web scraping — veb-sahifa HTML'idan ma'lumotni avtomatik olish. Python'da eng ko'p ishlatiladigan vosita — BeautifulSoup (HTML'ni parse qilish). Bu darsda: HTML struktura (teg, atribut, klass), BeautifulSoup (find, find_all, select), ma'lumot olish (matn, atribut), va scraping etikasi (robots.txt, hurmat). Muhim: scraping oxirgi chora (7.1 — API afzal), va etika/qonun bilan 7.11-bob. Bu darsda statik HTML bilan ishlaymiz (real sayt emas — deterministik, etik). Nega muhim? (1) API yo'q — faqat veb-sahifa; (2) HTML parse — BeautifulSoup (teg, klass); (3) Etika — robots.txt, hurmat (oxirgi chora). Bu dars web scraping'ni o'rgatadi — veb-sahifadan ma'lumot.

Web scraping (BeautifulSoup) — veb-sahifadan: HTML struktura (teg <div>, atribut class/id, ichma-ich), BeautifulSoup(html, "html.parser") (parse), find/find_all (teg/klass topish), select (CSS selektor), olish (.text matn, ["href"] atribut), etika (robots.txt, hurmat — oxirgi chora; 7.11). Foydalanish: API yo'q veb-sahifa, HTML parse. Bu 7.1 (manba — scraping oxirgi), 7.7 (API afzal), 7.11 (etika) bilan bog'liq. Web scraping — veb-sahifadan (BeautifulSoup; etika). Scraping. HTML.

Real vaziyat. Data Scientist e'lon sayti narxlarini olmoqchi (API yo'q — faqat veb-sahifa). Jarayon: (1) etika (robots.txt tekshir — ruxsat?; foydalanish sharti — 7.11), (2) HTML olish (requests.get(url) — 7.7; sahifa HTML), (3) parse (BeautifulSoup(html) — HTML → obyekt), (4) topish (soup.find_all("div", class_="narx") — narx teglari), (5) olish (.text — narx matni). Muhim: hurmat (kam so'rov, sleep — server; robots.txt). Data Scientist etika tekshirdi (robots.txt), HTML parse qildi (BeautifulSoup), narx oldi (find_all, .text). Web scraping — veb-sahifadan (etika bilan).

Bu darsda web scraping'ni o'rganamiz.

Bu darsda:

  • HTML struktura (teg, atribut, klass)
  • BeautifulSoup (find, find_all)
  • select (CSS selektor)
  • Ma'lumot olish (matn, atribut)
  • Scraping etikasi (robots.txt)
  • Scraping amaliyoti
  • Scraping tuzoqlari
  • Amaliy: web scraping

ℹ Misollar statik HTML bilan deterministik (real sayt scrape qilinmaydi — etik/deterministik).

2. Nazariya — chuqur tushuntirish

2.1. HTML struktura (teg, atribut, klass)

HTML asoslari:

html
<div class="mahsulot" id="m1">
    <h2 class="nom">Telefon</h2>
    <span class="narx">1200</span>
    <a href="/mahsulot/1">batafsil</a>
</div>

HTML struktura (teg, atribut, klass) — veb-sahifa tuzilishi: teg (<div>, <h2>, <span>, <a> — element; ochilish/yopilish <div>...</div>), atribut (class="narx", id="m1", href="..." — teg xususiyati), ichma-ich (teg ichida teg — ierarxik; DOM daraxt), klass/id (class — ko'p element; id — noyob). Sabab: HTML — veb-sahifa tili (teg — struktura; brauzer ko'rsatadi); scraping — teg/klass bo'yicha topish (class="narx" — narx elementi); "HTML tuzilishini tushun" (teg, atribut). Teg (element), atribut (class/id/href), ichma-ich (daraxt), klass (topish uchun). HTML struktura — teg/atribut/klass. HTML. Teg.

2.2. BeautifulSoup (find, find_all)

Parse va topish:

python
from bs4 import BeautifulSoup

# HTML → obyekt (parse)
soup = BeautifulSoup(html, "html.parser")

# find — birinchi mos teg
soup.find("span", class_="narx")        # 1-narx

# find_all — barcha mos teg (ro'yxat)
soup.find_all("div", class_="mahsulot") # barcha mahsulot

# teg + atribut
soup.find("a")["href"]                  # href atribut

BeautifulSoup (find, find_all) — parse va topish: BeautifulSoup(html, "html.parser") (HTML matn → obyekt — parse; "html.parser" yoki "lxml"), find(teg, class_=) (birinchi mos teg — bitta; yo'q — None), find_all(teg, class_=) (barcha mos teg — ro'yxat; yo'q — bo'sh); class_ (class Python kalit so'z — class_). Sabab: HTML parse (matn → obyekt — teg bo'yicha kirish); find (bitta — sarlavha), find_all (ko'p — barcha mahsulot; sikl bilan). BeautifulSoup (parse), find (birinchi), find_all (barcha — ro'yxat), class_ (klass). BeautifulSoup — find/find_all (parse, topish). BeautifulSoup. find.

2.3. select (CSS selektor)

select (CSS selektor) — CSS bilan topish: soup.select("div.mahsulot") (CSS selektor — .klass, #id, teg; barcha — ro'yxat), soup.select_one(...) (birinchi); selektorlar: .narx (klass), #m1 (id), div span (ichma-ich), div.mahsulot .narx (ichma-ich klass). Sabab: CSS selektor (veb-dasturchi tanish — qisqa, kuchli; div.mahsulot .narx — mahsulot ichidagi narx); find/find_all (alternativa — bir xil natija); tanlov (odat). select (CSS — barcha), select_one (birinchi), selektor (.klass, #id, ichma-ich). select — CSS selektor (topish). select. CSS.

2.4. Ma'lumot olish (matn, atribut)

Ma'lumot olish (matn, atribut) — tegdan: .text/.get_text() (teg ichidagi matn — <span>1200</span> → "1200"; strip=True — bo'shliq tozala), teg["atribut"] (atribut — <a>["href"] havola; .get("href") xavfsiz), .attrs (barcha atribut). Sabab: ma'lumot — teg ichida (matn) yoki atributda (href, src); .text (ko'rinadigan matn), ["href"] (havola, rasm URL). .text (matn), ["atribut"] (atribut), .get (xavfsiz). Ma'lumot olish — .text/["atribut"] (tegdan). Matn. Atribut.

2.5. Scraping etikasi (robots.txt)

Scraping etikasi (robots.txt) — mas'uliyat: robots.txt (sayt.com/robots.txt — qaysi sahifa scrape mumkin/mumkin emas; hurmat), foydalanish sharti (Terms — scraping taqiqlangan bo'lishi mumkin), hurmat (kam so'rov, sleep — server yuklama; rate limit — 7.7), shaxsiy ma'lumot (GDPR/maxfiylik — 7.11), User-Agent (kim — yashirma), API afzal (7.1 — scraping oxirgi chora). Sabab: etika/qonun (scraping — server resursi, ruxsat, maxfiylik; ruxsatsiz — jinoyiy/axloqsiz); robots.txt (sayt ruxsati — hurmat); "scraping — mas'uliyat" (etika, qonun; 7.11). robots.txt (ruxsat), hurmat (kam so'rov, sleep), shaxsiy (maxfiylik), API afzal (oxirgi chora). Scraping etikasi — robots.txt/hurmat (mas'uliyat). Etika. robots.txt.

2.6. Scraping amaliyoti

Scraping amaliyoti: etika (robots.txt, Terms — ruxsat?; API afzal — 7.1; 7.11); HTML olish (requests.get — 7.7; timeout); parse (BeautifulSoup(html, "html.parser")); struktura (sahifa HTML ko'r — teg/klass; brauzer "inspect"); topish (find/find_all/select — teg/klass); olish (.text/["atribut"]; .get xavfsiz); hurmat (kam so'rov, sleep; server); DataFrame (natija → jadval). Tuzoqlar: etika yo'q (ruxsat/robots.txt), ko'p so'rov (server — sleep), None (teg yo'q — tekshir), HTML o'zgaradi (mo'rt — 7.1), API afzal (oxirgi). Amaliyot — etika, parse, topish, hurmat. Scraping. HTML.

2.7. Scraping tuzoqlari

Web scraping asosiy tuzoqlari: etika/qonun (robots.txt/Terms tekshirmas — scraping taqiqlangan bo'lishi mumkin (huquqiy/axloqiy; 7.11); shaxsiy ma'lumot — GDPR; ruxsat); ko'p so'rov (tez ko'p so'rov — server yuklama (sayt sekinlashadi; bloklash; DoS kabi); sleep, kam so'rov — hurmat); None (find teg yo'q → None; .text → AttributeError (None.text); tekshir — if teg:); HTML o'zgaradi (sayt HTML o'zgarsa — scraper buziladi (mo'rt; ertaga ishlamasligi mumkin; API barqaror — 7.1)); API afzal (API bor bo'lsa — scraping o'rniga (rasmiy, barqaror; scraping oxirgi chora)); dinamik kontent (JavaScript bilan yuklanadigan (React/Vue) — requests/BeautifulSoup ko'rmaydi (faqat boshlang'ich HTML); Selenium/Playwright kerak); klass o'zgaruvchan (avtomatik klass (css-1a2b3c) — o'zgaradi; barqaror atribut (id, data-*)); kodlash (HTML kodlash — response.encoding; o'zbekcha buziq bo'lsa moslab); User-Agent (ba'zi sayt — User-Agent tekshiradi (bloklaydi); to'g'ri header); ko'chirish huquqi (kontent — mualliflik huquqi; foydalanish cheklovi). Sabab: scraping etika/mo'rtlik nozik (ruxsat, server, HTML o'zgarish — qonun yoki buziladi). Yechim: robots.txt/API afzal, sleep/kam so'rov, None tekshir, barqaror atribut. Tuzoqlar — etika, ko'p so'rov, None, HTML o'zgarish.

2.8. Web scraping — veb-sahifadan ma'lumot (etika bilan)

Web scraping asosiy g'oyasi — veb-sahifadan ma'lumot (etika bilan): ba'zan ma'lumot faqat veb-sahifada (API/fayl/baza yo'q — e'lon, yangilik, narx); web scraping — HTML'dan ma'lumot avtomatik olish (BeautifulSoup); lekin oxirgi chora (7.1 — API afzal) va etika bilan 7.11-bob. HTML struktura (teg <div>, atribut class/id, ichma-ich DOM), BeautifulSoup(html, "html.parser") (parse), find/find_all/select (teg/klass/CSS topish), olish (.text matn, ["href"] atribut; .get xavfsiz), etika (robots.txt ruxsat, hurmat — kam so'rov/sleep, shaxsiy ma'lumot maxfiylik; API afzal). Foydalanish: API yo'q veb-sahifa (oxirgi chora), HTML parse (teg/klass). Tuzoqlar: etika/qonun (robots.txt/Terms; 7.11), ko'p so'rov (server — sleep/hurmat), None (teg yo'q — AttributeError; tekshir), HTML o'zgaradi (mo'rt), dinamik kontent (JS — Selenium), API afzal (oxirgi chora). Bu 7.1 (manba — scraping oxirgi), 7.7 (API afzal, requests), 7.8 (parse), 7.11 (etika/qonun) bilan. Web scraping — veb-sahifadan ma'lumot (BeautifulSoup; etika bilan; oxirgi chora). Scraping. HTML. Etika.

3. Tez ma'lumotnoma

python
from bs4 import BeautifulSoup

# PARSE (HTML → obyekt):
soup = BeautifulSoup(html, "html.parser")

# TOPISH:
soup.find("span", class_="narx")           # birinchi (yo'q → None)
soup.find_all("div", class_="mahsulot")    # barcha (ro'yxat)
soup.select("div.mahsulot .narx")          # CSS selektor (barcha)
soup.select_one("#m1")                     # CSS (birinchi)

# OLISH:
teg.text                                    # matn (strip=True — tozala)
teg.get_text(strip=True)
teg["href"]                                 # atribut (yo'q → xato)
teg.get("href")                             # xavfsiz atribut

# XAVFSIZ (None tekshir):
teg = soup.find("span", class_="narx")
if teg:
    narx = teg.text

# ETIKA (MUHIM — 7.11):
#   robots.txt tekshir · foydalanish sharti · sleep (hurmat) · API afzal
QOIDA: robots.txt/API afzal · sleep (server) · None tekshir · HTML mo'rt

Web scraping xulosasi

Web scraping — veb-sahifadan ma'lumot (BeautifulSoup; oxirgi chora)
HTML struktura — teg (<div>), atribut (class/id), ichma-ich
BeautifulSoup — find (birinchi), find_all (barcha), select (CSS)
Olish — .text (matn), ["href"] (atribut); .get (xavfsiz)
Etika — robots.txt, hurmat (sleep), API afzal (7.11)

4. Batafsil misollar

Misollar statik HTML bilan deterministik (real sayt scrape qilinmaydi).

Misol 1 — Parse va find

python
"""Parse va find (real bs4)."""

from bs4 import BeautifulSoup


def main() -> None:
    html = """
    <div class="mahsulot">
        <h2 class="nom">Telefon</h2>
        <span class="narx">1200</span>
    </div>
    """
    soup = BeautifulSoup(html, "html.parser")

    print("=== 1. find (birinchi) ===")
    nom = soup.find("h2", class_="nom")
    print(f"  nom: {nom.text}")

    print("\n=== 2. Narx ===")
    narx = soup.find("span", class_="narx")
    print(f"  narx: {narx.text}")

    print("\n=== 3. Teg topilmadi (None) ===")
    yoq = soup.find("span", class_="chegirma")
    print(f"  chegirma: {yoq} (None — yo'q)")

    print("\n=== 4. Xavfsiz (None tekshir) ===")
    if yoq:
        print(yoq.text)
    else:
        print("  chegirma yo'q (AttributeError oldini oldi)")
    print("  ⭐ find — birinchi (None tekshir)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. find (birinchi) ===
  nom: Telefon

=== 2. Narx ===
  narx: 1200

=== 3. Teg topilmadi (None) ===
  chegirma: None (None — yo'q)

=== 4. Xavfsiz (None tekshir) ===
  chegirma yo'q (AttributeError oldini oldi)
  ⭐ find — birinchi (None tekshir)

Nima ko'rsatdi: 2.1, 2.2-bo'limlar.

Misol 2 — find_all (barcha)

python
"""find_all: barcha (real bs4)."""

from bs4 import BeautifulSoup


def main() -> None:
    html = """
    <div class="mahsulot"><span class="narx">1200</span></div>
    <div class="mahsulot"><span class="narx">800</span></div>
    <div class="mahsulot"><span class="narx">1500</span></div>
    """
    soup = BeautifulSoup(html, "html.parser")

    print("=== 1. find_all (barcha mahsulot) ===")
    mahsulotlar = soup.find_all("div", class_="mahsulot")
    print(f"  soni: {len(mahsulotlar)}")

    print("\n=== 2. Har narx (sikl) ===")
    narxlar = []
    for m in mahsulotlar:
        narx = int(m.find("span", class_="narx").text)
        narxlar.append(narx)
    print(f"  narxlar: {narxlar}")

    print("\n=== 3. Tahlil ===")
    print(f"  o'rtacha: {sum(narxlar) / len(narxlar):.0f}")
    print(f"  eng qimmat: {max(narxlar)}")

    print("\n=== 4. Comprehension ===")
    n2 = [int(s.text) for s in soup.find_all("span", class_="narx")]
    print(f"  narxlar: {n2}")
    print("  ⭐ find_all — barcha (ro'yxat)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. find_all (barcha mahsulot) ===
  soni: 3

=== 2. Har narx (sikl) ===
  narxlar: [1200, 800, 1500]

=== 3. Tahlil ===
  o'rtacha: 1167
  eng qimmat: 1500

=== 4. Comprehension ===
  narxlar: [1200, 800, 1500]
  ⭐ find_all — barcha (ro'yxat)

Nima ko'rsatdi: 2.2-bo'lim.

Misol 3 — select va atribut

python
"""select va atribut (real bs4)."""

from bs4 import BeautifulSoup


def main() -> None:
    html = """
    <div class="mahsulot">
        <a href="/m/1" class="link">Telefon</a>
        <span class="narx">1200</span>
    </div>
    """
    soup = BeautifulSoup(html, "html.parser")

    print("=== 1. select (CSS selektor) ===")
    narx = soup.select_one("div.mahsulot .narx")
    print(f"  narx: {narx.text}")

    print("\n=== 2. Atribut (href) ===")
    link = soup.find("a", class_="link")
    print(f"  href: {link['href']}")

    print("\n=== 3. .get (xavfsiz atribut) ===")
    print(f"  target (yo'q): {link.get('target', 'yo`q')}")

    print("\n=== 4. Matn + atribut ===")
    print(f"  matn: {link.text}, havola: {link['href']}")
    print("  ⭐ select (CSS) + atribut ([href])")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. select (CSS selektor) ===
  narx: 1200

=== 2. Atribut (href) ===
  href: /m/1

=== 3. .get (xavfsiz atribut) ===
  target (yo'q): yo`q

=== 4. Matn + atribut ===
  matn: Telefon, havola: /m/1
  ⭐ select (CSS) + atribut ([href])

Nima ko'rsatdi: 2.3, 2.4-bo'limlar.

Misol 4 — Jadvalga va etika

python
"""Scraping → DataFrame va etika (real bs4/pandas)."""

from bs4 import BeautifulSoup
import pandas as pd


def main() -> None:
    html = """
    <div class="mahsulot"><h2>Telefon</h2><span class="narx">1200</span></div>
    <div class="mahsulot"><h2>Noutbuk</h2><span class="narx">5000</span></div>
    """
    soup = BeautifulSoup(html, "html.parser")

    print("=== 1. Scrape → ro'yxat ===")
    natija = []
    for m in soup.find_all("div", class_="mahsulot"):
        natija.append({
            "nom": m.find("h2").text,
            "narx": int(m.find("span", class_="narx").text),
        })
    print(f"  yozuvlar: {len(natija)}")

    print("\n=== 2. DataFrame ===")
    df = pd.DataFrame(natija)
    print(df.to_string(index=False))

    print("\n=== 3. Etika (real scrapingda) ===")
    print("  robots.txt tekshir; sleep (server); API afzal")

    print("\n=== 4. Oxirgi chora ===")
    print("  API bor bo'lsa — scraping emas 7.1-bob")
    print("  ⭐ Scraping → DataFrame (etika bilan)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Scrape → ro'yxat ===
  yozuvlar: 2

=== 2. DataFrame ===
    nom  narx
Telefon  1200
Noutbuk  5000

=== 3. Etika (real scrapingda) ===
  robots.txt tekshir; sleep (server); API afzal

=== 4. Oxirgi chora ===
  API bor bo'lsa — scraping emas 7.1-bob
  ⭐ Scraping → DataFrame (etika bilan)

Nima ko'rsatdi: 2.5, 2.6-bo'limlar.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"har sayt scrape mumkin" robots.txt/Terms (7.11)
"scraping birinchi" Oxirgi chora (API afzal)
"find doim teg" None (yo'q — tekshir)
"ko'p so'rov OK" Server (sleep, hurmat)
"HTML barqaror" O'zgaradi (mo'rt)
"JS kontent ko'rinadi" Selenium (dinamik)
"klass barqaror" O'zgaruvchan (id, data-*)
"scraping = API" API afzal (rasmiy)

6. Keng tarqalgan xatolar va yechimlari

1. Etika/robots.txt

python
# har sayt scrape (robots.txt e'tiborsiz)                         # ⚠️
# robots.txt tekshir; Terms; API afzal 7.11-bob                     # ✅

2. Ko'p so'rov (server)

python
for url in urllar: requests.get(url)   # tez ko'p (server yuk)    # ⚠️
import time; time.sleep(1)   # kutib (hurmat)                    # ✅

3. None (teg yo'q)

python
soup.find("span", class_="narx").text   # yo'q → AttributeError   # ⚠️
teg = soup.find(...); teg.text if teg else None   # tekshir      # ✅

4. HTML o'zgaradi

python
soup.find("div", class_="css-1a2b3c")   # avtomatik klass o'zgaradi # ⚠️
soup.find("div", attrs={"data-id": "narx"})   # barqaror atribut  # ✅

5. Dinamik kontent (JS)

python
requests.get(url)   # JS bilan yuklangan → bo'sh                  # ⚠️
# Selenium/Playwright (JS render) yoki API                        # ✅

6. API o'rniga scraping

python
# API bor, lekin scraping (mo'rt, huquqiy)                        # ⚠️
# avval API tekshir (rasmiy, barqaror)                            # ✅

7. Kodlash

python
BeautifulSoup(response.content)   # o'zbekcha buzuq                # ⚠️
response.encoding = "utf-8"; BeautifulSoup(response.text)         # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 7.1-dars (o'tilgan): Manbalar (scraping oxirgi)
  • 7.7-dars (o'tilgan): API (requests, afzal)
  • 7.8-dars (o'tilgan): JSON parse (o'xshash)
  • 7.11-dars: Etika va qonun (robots.txt)
  • 7.12-dars: Amaliyot (loyiha)

8. Eng yaxshi amaliyotlar

  1. robots.txt/Terms — tekshir (etika).

  2. API afzal — scraping oxirgi.

  3. sleep — hurmat (server).

  4. None tekshir — teg yo'q.

  5. Barqaror atribut — id, data-*.

  6. .get — xavfsiz atribut.

  7. Kodlash — utf-8 (o'zbekcha).

  8. Dinamik — Selenium (JS).

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # web scraping nima?
2.  # BeautifulSoup?
3.  # find vs find_all?
4.  # select nima?
5.  # .text nima?
6.  # atribut olish?
7.  # robots.txt?
8.  # None sababi?
9.  # HTML mo'rtligi?
10. # dinamik kontent?
11. # API vs scraping?
12. # nega etika?
Javoblar
  1. Veb-sahifadan ma'lumot olish
  2. HTML parse
  3. find birinchi, find_all barcha
  4. CSS selektor
  5. Teg matni
  6. teg["href"] / .get
  7. Scraping ruxsati
  8. Teg yo'q (AttributeError)
  9. HTML o'zgaradi (buziladi)
  10. JS (Selenium kerak)
  11. API afzal (rasmiy)
  12. Server/ruxsat/maxfiylik

Vazifa 2: Xatolarni tuzating

python
1.  # har sayt scrape

2.  for url in urllar: requests.get(url)

3.  soup.find("span").text   # yo'q

4.  soup.find(class_="css-1a2b3c")

5.  requests.get(url)   # JS kontent
Javoblar
python
1.  robots.txt / API (7.11)

2.  time.sleep(1) (hurmat)

3.  teg.text if teg else None

4.  data-id (barqaror)

5.  Selenium / API

Vazifa 3: HTML va parse

Modellang:

  1. Teg
  2. Atribut/klass
  3. BeautifulSoup
  4. find/find_all

Vazifa 4: Topish va olish

Modellang:

  1. find_all
  2. select (CSS)
  3. .text
  4. ["href"]

Vazifa 5: Etika

Modellang:

  1. robots.txt
  2. Hurmat (sleep)
  3. API afzal
  4. Maxfiylik

Vazifa 6: Integratsiya

Modellang:

  1. Manbalar (7.1)
  2. API (7.7)
  3. Etika (7.11)
  4. Loyiha (7.12)

Vazifa 7: O'ylash

Web scraping texnik jihatdan oson (BeautifulSoup), lekin "oxirgi chora" deb ataladi va etika/qonun bilan cheklangan. Ko'p so'rov serverni yuklaydi, robots.txt scrapingni taqiqlashi mumkin, shaxsiy ma'lumot maxfiylik masalasi. Nima uchun "shunchaki texnik imkoniyat bor deb, uni qilish kerak degani emas" tamoyili scraping'da (va umuman Data Science'da) muhim?

Javob

Qisqa javob: Web scraping texnik oson (BeautifulSoup), lekin "oxirgi chora" va etika/qonun bilan cheklangan (server yuklama, robots.txt, maxfiylik); "texnik imkoniyat bor deb, qilish kerak degani emas" tamoyili scraping'da (va Data Science'da) muhim, chunki: (1) mumkin ≠ to'g'ri — texnik imkoniyat (scrape qilish mumkin) ≠ ruxsat/to'g'rilik (qilish kerak/mumkin — etika, qonun); "qila olaman" boshqa, "qilishim kerak/mumkin" boshqa; (2) boshqalarga ta'sir — scraping serverni yuklaydi (boshqa foydalanuvchi; sayt egasi; ko'p so'rov — zarar); harakat oqibati (o'zi uchun emas — boshqalar); (3) ruxsat — sayt egasi bor (robots.txt/Terms — ruxsat/taqiq; ruxsatsiz — huquqiy/axloqiy buzish); (4) maxfiylik — shaxsiy ma'lumot (inson — GDPR; scrape — maxfiylik buzish; 7.11). "Nega Data Science'da umuman": (a) kuch — mas'uliyat (Data Scientist — ma'lumot kuchi (yig'ish, tahlil, model); kuch — mas'uliyat; "buyuk kuch — buyuk mas'uliyat"); (b) ta'sir (ma'lumot/model — inson hayotiga (kredit, ish, sud); to'g'ri/adolatli; 5.13 halollik); (c) etika (texnik qaror — axloqiy oqibat; ma'lumot — maxfiylik, adolat, zarar); (d) kasbiy (Data Scientist — etika standart; texnik + axloqiy). "Scraping'da qanday": (1) robots.txt/Terms (ruxsat — tekshir; taqiq — qilma); (2) API afzal (rasmiy — scraping oxirgi); (3) hurmat (kam so'rov, sleep — server); (4) maxfiylik (shaxsiy ma'lumot — olma/anonimlash); (5) savol ("qilishim kerakmi? ruxsatmi? zararmi?" — texnik emas). "Umuman Data Science'da": (1) ma'lumot yig'ish (ruxsat, maxfiylik); (2) model (adolat — noxolislik; zarar; 4.10/5.13); (3) xulosa (halol — aldamas; 5.13); (4) ta'sir (inson — mas'uliyat). "Nega muhim": mumkin ≠ to'g'ri (texnik ≠ etika); boshqalarga ta'sir (server, inson); ruxsat/maxfiylik (qonun/etika); kuch — mas'uliyat. Saboqlar: texnik imkoniyat ≠ qilish kerak (etika, ruxsat); boshqalarga ta'sir (server, maxfiylik); kuch — mas'uliyat (Data Science); savol (kerakmi, ruxsatmi, zararmi). To'g'ri: mumkin ≠ to'g'ri (etika, ruxsat); scraping — robots.txt/API/hurmat/maxfiylik; Data Science — kuch mas'uliyat; savol. Muvozanat: texnik imkoniyat (qila olaman) + etika (qilishim kerakmi — ruxsat, ta'sir, maxfiylik). Bu Data Science etika asosiy (mumkin ≠ to'g'ri; ruxsat, maxfiylik, ta'sir; kuch mas'uliyat; 7.11). Web scraping — mumkin ≠ to'g'ri (etika, ruxsat; texnik imkoniyat — mas'uliyat bilan).

1. Nega mumkin ≠ to'g'ri

  • Texnik imkoniyat ≠ ruxsat/to'g'rilik
  • Boshqalarga ta'sir (server, inson)
  • Ruxsat (sayt egasi — robots.txt/Terms)
  • Maxfiylik (shaxsiy — GDPR)

2. Nega Data Science'da umuman

  • Kuch — mas'uliyat (ma'lumot kuchi)
  • Ta'sir (model — inson hayoti)
  • Etika (texnik qaror — axloqiy oqibat)
  • Kasbiy standart (texnik + axloqiy)

3. Mumkin vs to'g'ri

Mumkin (texnik) To'g'ri (etika)
Scrape qila olaman Ruxsatmi? (robots.txt)
Ko'p so'rov Server (hurmat)
Ma'lumot olish Maxfiylik (shaxsiy)

4. Scraping'da qanday

  1. robots.txt/Terms (ruxsat)
  2. API afzal (oxirgi chora)
  3. Hurmat (sleep — server)
  4. Maxfiylik (shaxsiy — olma)

5. Xulosa

  1. Texnik imkoniyat ≠ qilish kerak (etika, ruxsat)
  2. Boshqalarga ta'sir (server, inson — maxfiylik)
  3. Kuch — mas'uliyat (Data Science — texnik + axloqiy)
  4. Savol (kerakmi, ruxsatmi, zararmi — texnik emas)

Nimani mustahkamlaydi: 2.5, 2.7-bo'limlar.

Xulosa

Bu darsda web scraping'ni o'rgandik.

Eng muhim uch fikr:

  1. HTML va BeautifulSoup. HTML struktura — teg (<div>, <span>), atribut (class/id/href), ichma-ich (DOM daraxt); klass — topish uchun. BeautifulSoup(html, "html.parser") (parse) — find(teg, class_=) (birinchi — yo'q → None), find_all (barcha — ro'yxat), select (CSS selektor — .klass, #id, ichma-ich).

  2. Olish va etika. Ma'lumot olish — .text/.get_text(strip=True) (matn), teg["href"] (atribut; .get xavfsiz). Scraping etikasi — robots.txt (ruxsat), foydalanish sharti (Terms), hurmat (kam so'rov, sleep — server), shaxsiy ma'lumot (maxfiylik); API afzal (7.1 — scraping oxirgi chora).

  3. Veb-sahifadan (etika bilan). Web scraping — veb-sahifadan ma'lumot (API yo'q; BeautifulSoup) lekin oxirgi chora va etika bilan 7.11-bob. "Texnik imkoniyat bor deb, qilish kerak degani emas" (mumkin ≠ to'g'ri; ruxsat, server, maxfiylik — kuch mas'uliyat). Tuzoqlar: etika/qonun (robots.txt/Terms), ko'p so'rov (server — sleep), None (teg yo'q — AttributeError; tekshir), HTML o'zgaradi (mo'rt — barqaror atribut), dinamik kontent (JS — Selenium), API afzal (oxirgi), kodlash (utf-8).

Keyingi darsda katta fayllar (chunks)ni o'rganamiz: xotiraga sig'maydigan katta fayllar (GB) — qism-qism o'qish (chunksize), tur optimizatsiya (dtype, kategoriya), va katta ma'lumot strategiyasi (Parquet, Dask).

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!
7.9-dars: Web scraping (BeautifulSoup) — IlmHamroh