IlmHamroh
Python kursi/Avtomatlashtirish4/10-dars15 daqiqa
Mundarija (22)

26.4-dars: Scraping — BeautifulSoup

26-QISM — AVTOMATLASHTIRISH · 4-dars

1. Kirish va motivatsiya

26.3 da requests bilan veb sahifani yuklab oldik — lekin natija HTML matn (teglar, atributlar aralash). Undan kerakli ma'lumot (narx, nom, havola) ni ajratib olish kerak. HTML'ni qo'lda qidirish (find, split) — juda qiyin va buziluvchan. HTML tahlilchi (parser) kerak — HTML tuzilishini tushunib, teglar/klasslar bo'yicha qidiradi. Bu — BeautifulSoup.

BeautifulSoup — Python'ning HTML tahlil kutubxonasi (bs4): HTML matnini daraxt (tuzilma) qiladi va oson qidirish beradi. find (birinchi mos teg), find_all (barcha mos), teg/klass bo'yicha (find("div", class_="kurs")), atribut (tag["href"]), matn (tag.text), CSS selektorlar (select(".kurs .nom")). requests (yuklash) + BeautifulSoup (tahlil) — web scraping'ning ikki qismi. Veb sahifadan tuzilgan ma'lumot olish. HTML scraping'ning yuragi.

Real vaziyat. Bir do'kon raqobatchi narxlarini kuzatish kerak edi — 100 mahsulot, har kuni. Qo'lda — soatlar. requests bilan sahifa yuklandi, BeautifulSoup bilan find_all("div", class_="mahsulot") — har mahsulot nom va narxi ajratildi, jadvalga yig'di. Bir daqiqada 100 narx. Avtomatik, xatosiz. BeautifulSoup — HTML'dan ma'lumot ajratdi. Qo'lda o'qish o'rniga skript.

Bu darsda BeautifulSoup bilan HTML tahlilni o'rganamiz.

Bu darsda:

  • BeautifulSoup nima
  • find va find_all
  • Teg va klass bo'yicha qidirish
  • Atributlar (href, data-*)
  • Matn ajratish (.text)
  • CSS selektorlar (select)
  • Tuzilgan ma'lumot yig'ish
  • Amaliy: sahifadan ma'lumot

ℹ Misollarda BeautifulSoup (bs4) sobit HTML matni bilan sinaladi — deterministik.

2. Nazariya — chuqur tushuntirish

2.1. BeautifulSoup nima

HTML matnini tuzilma (daraxt) qiladi:

python
from bs4 import BeautifulSoup
html = "<html><body><h1>Salom</h1></body></html>"
soup = BeautifulSoup(html, "html.parser")
soup.find("h1").text     # "Salom"

BeautifulSoup (bs4) — HTML matnini daraxt (tuzilma) qiladi va oson qidirish beradi: BeautifulSoup(html, "html.parser") (matn → soup obyekt). Keyin find/find_all bilan teglarni topish, .text bilan matn, tag["atr"] bilan atribut. HTML'ni qo'lda (split, regex) qidirish — qiyin, buziluvchan; BeautifulSoup — tuzilmani tushunadi. requests (yuklash) + BeautifulSoup (tahlil) — scraping. HTML'dan ma'lumot ajratishning asosi.

2.2. find va find_all

Teglarni topish:

python
soup.find("h1")              # birinchi <h1> (bitta)
soup.find_all("div")         # barcha <div> (ro'yxat)
soup.find("a")["href"]       # birinchi havola
[a["href"] for a in soup.find_all("a")]   # barcha havola

find (birinchi mos teg — bitta) va find_all (barcha mos — ro'yxat): soup.find("h1") (birinchi sarlavha), soup.find_all("div") (barcha div — siklda). find topmasa None (tekshir), find_all bo'sh ro'yxat. Bu HTML'dan teglarni topishning asosi. find_all — ko'p element (mahsulotlar, havolalar — siklda qayta ishlash). Eng ko'p ishlatiladigan metodlar.

2.3. Teg va klass bo'yicha qidirish

Aniqroq qidirish:

python
soup.find("div", class_="kurs")           # klass bo'yicha
soup.find_all("span", class_="narx")      # barcha narx
soup.find("div", id="asosiy")             # id bo'yicha
soup.find("a", href="/home")              # atribut bo'yicha

Klass/atribut bo'yicha qidirish (aniqroq): find("div", class_="kurs") (class_ — class Python kalit so'zi, _ bilan), find(id="asosiy") (id), find("a", href="...") (atribut). Real HTML'da ko'p div bor — klass bilan aniq (class_="mahsulot" — faqat mahsulotlar). Bu aniq ma'lumotni ajratishning kaliti (barcha div emas, faqat kerakli). Klass — HTML'da ma'lumot belgisi (dizayn + tuzilma).

2.4. Atributlar (href, data-*)

Teg atributlari:

python
tag = soup.find("a")
tag["href"]              # havola
tag.get("href")          # xavfsiz (yo'q bo'lsa None)
tag["data-id"]           # maxsus atribut
tag.attrs                # barcha atribut (dict)

Atributlar — teg xususiyatlari: tag["href"] (havola — <a href="...">), tag.get("href") (xavfsiz — yo'q bo'lsa None, xato emas), tag["data-id"] (maxsus data-* atribut — ko'pincha id, narx). tag.attrs (barcha — dict). Atributlar muhim ma'lumot saqlaydi (havola, id, narx). .get() — xavfsiz (atribut bo'lmasa xato bermaydi). Atributlardan ma'lumot olish.

2.5. Matn ajratish (.text)

Teg ichidagi matn:

python
soup.find("h1").text            # teg matni
soup.find("p").get_text()       # bir xil
soup.find("p").text.strip()     # bo'sh joyni olib tashla
[li.text for li in soup.find_all("li")]   # barcha

Matn ajratish: tag.text (teg va ichki teglar matni), tag.get_text() (bir xil), .strip() (chet bo'sh joylarni olib tashla — 24.8 kabi). Real HTML matnda bo'sh joy, yangi qator ko'p — .strip() bilan tozala. find_all + .text — ko'p element matni (siklda). Bu HTML'dan ko'rinadigan ma'lumotni (narx, nom) olish. Matn — foydalanuvchi ko'radigan qism.

2.6. CSS selektorlar (select)

CSS uslubida qidirish:

python
soup.select(".kurs")              # klass (nuqta)
soup.select("#asosiy")            # id (diez)
soup.select("div.kurs span.nom")  # ichma-ich
soup.select_one(".narx")          # birinchi (find kabi)

CSS selektorlar (select) — CSS uslubida qidirish (veb dizaynerlar tanish): .kurs (klass — nuqta), #asosiy (id — diez), div.kurs span.nom (ichma-ich — bo'sh joy), select_one (birinchi — find kabi). find/find_all ga muqobil — CSS selektor qisqaroq (murakkab qidirishda). Ikkalasi ishlaydi (ta'm masalasi). CSS — HTML/veb standart qidirish usuli. Murakkab tuzilmalarda qulay.

2.7. Tuzilgan ma'lumot yig'ish

HTML'dan jadval:

python
kurslar = []
for div in soup.find_all("div", class_="kurs"):
    kurslar.append({
        "nom": div.find("span", class_="nom").text,
        "narx": int(div.find("span", class_="narx").text),
    })
# kurslar → pandas DataFrame 24.5-bob yoki CSV

Tuzilgan ma'lumot yig'ish — HTML'dan jadval (dict/list): find_all (elementlar) → siklda har biridan (nom, narx) ajratib → dict → ro'yxatga. Natija — pandas DataFrame 24.5-bob yoki CSV 24.6-bob uchun tayyor. Bu scraping'ning maqsadi: iflos HTML'dan toza, tuzilgan ma'lumot. Keyin tahlil (24-qism), saqlash. HTML → tuzilgan ma'lumot — scraping'ning yakuniy natijasi.

2.8. HTML scraping — mo'rt

HTML scraping mo'rt (fragile): sayt dizayni o'zgarsa — scraper buziladi (klass nomi, tuzilma o'zgarsa, find topmaydi). Bu API (barqaror — 26.3) dan farq: HTML inson uchun (o'zgaradi), API dastur uchun (barqaror). Yechim: API afzal (bor bo'lsa), mustahkam qidirish (bir necha usul), xato boshqarish (find None qaytaradi — tekshir), kuzatish (scraper buzilganda sezish). "HTML o'zgaradi — scraper ehtiyot" — scraping doim kuzatish, yangilash talab qiladi. Mo'rt, lekin ba'zan yagona usul (API yo'q).

3. Tez ma'lumotnoma

python
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")

# find / find_all:
soup.find("h1")                       # birinchi
soup.find_all("div")                  # barcha

# klass/atribut:
soup.find("div", class_="kurs")       # class_ (kalit so'z)
soup.find(id="asosiy")
soup.find("a", href="/home")

# atribut:
tag["href"]; tag.get("href")          # xavfsiz
tag["data-id"]

# matn:
tag.text; tag.get_text(); tag.text.strip()

# CSS selektor:
soup.select(".kurs")                  # klass
soup.select("div.kurs span.nom")      # ichma-ich
soup.select_one(".narx")

# tuzilgan ma'lumot:
[{"nom": d.find("span", class_="nom").text} for d in soup.find_all("div", class_="kurs")]

BeautifulSoup xulosasi

HTML → daraxt · find (birinchi) / find_all (barcha) · class_ (klass)
tag["href"] (atribut) · tag.text (matn) · select (CSS)
HTML → tuzilgan ma'lumot (dict/list) · HTML scraping mo'rt (API afzal)

4. Batafsil misollar

Misollarda BeautifulSoup (bs4) sobit HTML matni bilan sinaladi — deterministik.

Misol 1 — find va find_all

python
"""find (birinchi teg); find_all (barcha); .text (matn) — HTML'dan teglar."""

import warnings
warnings.filterwarnings("ignore")

from bs4 import BeautifulSoup


HTML = """
<html><body>
<h1>Kurslar ro'yxati</h1>
<p>Eng yaxshi kurslar</p>
<ul><li>Python</li><li>Go</li><li>Rust</li></ul>
</body></html>
"""


def main() -> None:
    soup = BeautifulSoup(HTML, "html.parser")

    print("=== 1. find (birinchi teg) ===")
    print(f"  h1: {soup.find('h1').text}")
    print(f"  p: {soup.find('p').text}")

    print("\n=== 2. find_all (barcha teg) ===")
    items = soup.find_all("li")
    print(f"  li soni: {len(items)}")

    print("\n=== 3. Matn ajratish (siklda) ===")
    tillar = [li.text for li in soup.find_all("li")]
    print(f"  tillar: {tillar}")

    print("\n=== 4. find topmasa (None) ===")
    yoq = soup.find("table")
    print(f"  table topilmadi: {yoq is None}")
    print("  ⭐ find (birinchi), find_all (barcha), .text (matn)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. find (birinchi teg) ===
  h1: Kurslar ro'yxati
  p: Eng yaxshi kurslar

=== 2. find_all (barcha teg) ===
  li soni: 3

=== 3. Matn ajratish (siklda) ===
  tillar: ['Python', 'Go', 'Rust']

=== 4. find topmasa (None) ===
  table topilmadi: True
  ⭐ find (birinchi), find_all (barcha), .text (matn)

Nima ko'rsatdi: 2.2, 2.5-bo'limlar.

Misol 2 — Klass va atributlar

python
"""klass bo'yicha (class_); atributlar (href, data-*); get (xavfsiz)."""

import warnings
warnings.filterwarnings("ignore")

from bs4 import BeautifulSoup


HTML = """
<html><body>
<div class="kurs" data-id="1"><span class="nom">Python</span></div>
<div class="kurs" data-id="2"><span class="nom">Go</span></div>
<a href="/page1">Sahifa 1</a>
<a href="/page2">Sahifa 2</a>
</body></html>
"""


def main() -> None:
    soup = BeautifulSoup(HTML, "html.parser")

    print("=== 1. Klass bo'yicha (class_) ===")
    kurslar = soup.find_all("div", class_="kurs")
    print(f"  kurs div soni: {len(kurslar)}")

    print("\n=== 2. Atribut (data-id) ===")
    for div in kurslar:
        print(f"  id={div['data-id']}: {div.find('span', class_='nom').text}")

    print("\n=== 3. Havola atributi (href) ===")
    havolalar = [a["href"] for a in soup.find_all("a")]
    print(f"  havolalar: {havolalar}")

    print("\n=== 4. get (xavfsiz atribut) ===")
    a = soup.find("a")
    print(f"  href (get): {a.get('href')}")
    print(f"  yo'q atribut (get): {a.get('target')}")
    print("  ⭐ class_ (klass), tag['atr'] (atribut), get (xavfsiz)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Klass bo'yicha (class_) ===
  kurs div soni: 2

=== 2. Atribut (data-id) ===
  id=1: Python
  id=2: Go

=== 3. Havola atributi (href) ===
  havolalar: ['/page1', '/page2']

=== 4. get (xavfsiz atribut) ===
  href (get): /page1
  yo'q atribut (get): None
  ⭐ class_ (klass), tag['atr'] (atribut), get (xavfsiz)

Nima ko'rsatdi: 2.3, 2.4-bo'limlar.

Misol 3 — CSS selektorlar

python
"""CSS selektorlar (select): klass (.), id (#), ichma-ich (bo'sh joy)."""

import warnings
warnings.filterwarnings("ignore")

from bs4 import BeautifulSoup


HTML = """
<html><body>
<div id="asosiy">
  <div class="kurs"><span class="nom">Python</span><span class="narx">100</span></div>
  <div class="kurs"><span class="nom">Go</span><span class="narx">120</span></div>
</div>
</body></html>
"""


def main() -> None:
    soup = BeautifulSoup(HTML, "html.parser")

    print("=== 1. Klass selektori (.kurs) ===")
    kurslar = soup.select(".kurs")
    print(f"  .kurs soni: {len(kurslar)}")

    print("\n=== 2. id selektori (#asosiy) ===")
    asosiy = soup.select_one("#asosiy")
    print(f"  #asosiy ichida kurs: {len(asosiy.select('.kurs'))}")

    print("\n=== 3. Ichma-ich selektor ===")
    nomlar = [s.text for s in soup.select("div.kurs span.nom")]
    print(f"  nomlar: {nomlar}")

    print("\n=== 4. select_one (birinchi) ===")
    birinchi = soup.select_one(".kurs .narx")
    print(f"  birinchi narx: {birinchi.text}")
    print("  ⭐ select (CSS) — klass, id, ichma-ich (qisqa)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. Klass selektori (.kurs) ===
  .kurs soni: 2

=== 2. id selektori (#asosiy) ===
  #asosiy ichida kurs: 2

=== 3. Ichma-ich selektor ===
  nomlar: ['Python', 'Go']

=== 4. select_one (birinchi) ===
  birinchi narx: 100
  ⭐ select (CSS) — klass, id, ichma-ich (qisqa)

Nima ko'rsatdi: 2.6-bo'lim.

Misol 4 — Amaliy: HTML'dan tuzilgan ma'lumot

Real scraping: HTML sahifadan mahsulotlar jadvalini ajratib, tuzilgan ma'lumot (dict/list) yig'ish. Bu — scraping natijasining namunasi.

python
"""to'liq scraping: HTML'dan mahsulotlar (nom, narx, teg), tuzilgan ma'lumot, statistika."""

import warnings
warnings.filterwarnings("ignore")

from bs4 import BeautifulSoup


HTML = """
<html><body>
<div class="mahsulot" data-id="1">
  <span class="nom">Python kursi</span><span class="narx">100</span><span class="kat">kurs</span>
</div>
<div class="mahsulot" data-id="2">
  <span class="nom">SQL kitobi</span><span class="narx">50</span><span class="kat">kitob</span>
</div>
<div class="mahsulot" data-id="3">
  <span class="nom">Go kursi</span><span class="narx">120</span><span class="kat">kurs</span>
</div>
</body></html>
"""


def parse_mahsulotlar(html: str) -> list:
    soup = BeautifulSoup(html, "html.parser")
    natija = []
    for div in soup.find_all("div", class_="mahsulot"):
        natija.append({
            "id": int(div["data-id"]),
            "nom": div.find("span", class_="nom").text,
            "narx": int(div.find("span", class_="narx").text),
            "kat": div.find("span", class_="kat").text,
        })
    return natija


def main() -> None:
    print("=== 1. HTML'dan mahsulotlar ajratish ===")
    mahsulotlar = parse_mahsulotlar(HTML)
    print(f"  mahsulotlar soni: {len(mahsulotlar)}")

    print("\n=== 2. Tuzilgan ma'lumot (dict) ===")
    for m in mahsulotlar:
        print(f"  {m['nom']}: {m['narx']} ({m['kat']})")

    print("\n=== 3. Filtrlash (kurslar) ===")
    kurslar = [m["nom"] for m in mahsulotlar if m["kat"] == "kurs"]
    print(f"  kurslar: {kurslar}")

    print("\n=== 4. Statistika ===")
    narxlar = [m["narx"] for m in mahsulotlar]
    print(f"  jami: {sum(narxlar)}, o'rtacha: {round(sum(narxlar) / len(narxlar), 1)}")
    print("  ⭐ HTML → tuzilgan ma'lumot (pandas/CSV uchun tayyor)")


if __name__ == "__main__":
    main()

Natijaning muhim qismi:

text
=== 1. HTML'dan mahsulotlar ajratish ===
  mahsulotlar soni: 3

=== 2. Tuzilgan ma'lumot (dict) ===
  Python kursi: 100 (kurs)
  SQL kitobi: 50 (kitob)
  Go kursi: 120 (kurs)

=== 3. Filtrlash (kurslar) ===
  kurslar: ['Python kursi', 'Go kursi']

=== 4. Statistika ===
  jami: 270, o'rtacha: 90.0
  ⭐ HTML → tuzilgan ma'lumot (pandas/CSV uchun tayyor)

Nima ko'rsatdi: 2.1–2.8-bo'limlar.

5. To'g'ri va noto'g'ri tushunishlar

Noto'g'ri fikr To'g'risi
"HTML'ni regex bilan" BeautifulSoup (tuzilma)
"class (Python)" class_ (kalit so'z)
"find doim topadi" None bo'lishi mumkin
"tag['x'] xavfsiz" .get() (yo'q — None)
"HTML scraping barqaror" Mo'rt (dizayn o'zgarsa)
"select yoki find yaxshi" Ikkalasi (ta'm)
"Matnda bo'sh joy yo'q" .strip() kerak
"Scraping — API o'rniga" API afzal (bor bo'lsa)

6. Keng tarqalgan xatolar va yechimlari

1. class (class_ o'rniga)

python
soup.find("div", class="kurs")   # SyntaxError       # ⚠️
soup.find("div", class_="kurs")                        # ✅

2. find None tekshirmaslik

python
soup.find("h1").text   # None bo'lsa — AttributeError  # ⚠️
tag = soup.find("h1"); tag.text if tag else ""         # ✅

3. Atribut yo'q ([] o'rniga .get)

python
tag["target"]   # yo'q bo'lsa KeyError                 # ⚠️
tag.get("target")   # None (xavfsiz)                    # ✅

4. Matn tozalamaslik

python
tag.text   # bo'sh joy, yangi qator                    # ⚠️
tag.text.strip()                                         # ✅

5. Regex bilan HTML

python
re.findall(r"<div>(.*?)</div>", html)   # mo'rt        # ⚠️
BeautifulSoup(html).find_all("div")                     # ✅

6. Mo'rt selektor (juda aniq)

python
soup.select("body > div:nth-child(3) > span")   # mo'rt  # ⚠️
soup.find_all("span", class_="narx")   # mustahkam       # ✅

7. Parser tanlamaslik

python
BeautifulSoup(html)   # ogohlantirish                   # ⚠️
BeautifulSoup(html, "html.parser")                       # ✅

7. Integratsiya — bu bilim qayerda kerak bo'ladi

  • 26.3-dars (o'tilgan): requests — HTML yuklash
  • 26.5-dars: Selenium — dinamik sahifa
  • 26.6-dars: Scraping etikasi
  • 24.6-dars (o'tilgan): CSV — ma'lumot saqlash
  • 24.5-dars (o'tilgan): pandas — tuzilgan ma'lumot

8. Eng yaxshi amaliyotlar

  1. BeautifulSoup (regex emas).

  2. class_ (kalit so'z).

  3. find natijasini tekshir (None).

  4. .get() (xavfsiz atribut).

  5. .strip() (matn tozalash).

  6. Mustahkam selektor (klass — nth-child emas).

  7. "html.parser" (parser ko'rsat).

  8. HTML → tuzilgan ma'lumot (dict/list).

9. Amaliy topshiriq

Vazifa 1: Bashorat qiling

python
1.  # BeautifulSoup nima?
2.  # find nima?
3.  # find_all nima?
4.  # class_ nega?
5.  # tag['href'] nima?
6.  # get nima?
7.  # .text nima?
8.  # .strip() nega?
9.  # select nima?
10. # CSS selektor?
11. # tuzilgan ma'lumot?
12. # HTML scraping mo'rtmi?
Javoblar
  1. HTML tahlil kutubxonasi
  2. Birinchi mos teg
  3. Barcha mos teg (ro'yxat)
  4. class Python kalit so'zi
  5. Teg atributi (havola)
  6. Xavfsiz atribut (None)
  7. Teg matni
  8. Bo'sh joyni olib tashlash
  9. CSS selektor bilan qidirish
  10. .klass, #id, ichma-ich
  11. HTML'dan dict/list
  12. Ha (dizayn o'zgarsa buziladi)

Vazifa 2: Xatolarni tuzating

python
1.  soup.find("div", class="kurs")       # class_

2.  soup.find("h1").text   # None?        # tekshir

3.  tag["target"]   # yo'q                 # get

4.  tag.text   # bo'sh joy                 # strip

5.  re.findall(r"<div>...", html)          # BeautifulSoup
Javoblar
python
1.  soup.find("div", class_="kurs")

2.  tag = soup.find("h1"); tag.text if tag else ""

3.  tag.get("target")

4.  tag.text.strip()

5.  BeautifulSoup(html).find_all("div")

Vazifa 3: find

Teglar:

  1. find
  2. find_all
  3. .text
  4. None

Vazifa 4: Klass/atribut

Qidirish:

  1. class_
  2. data-*
  3. href
  4. .get()

Vazifa 5: CSS

Selektor:

  1. .klass
  2. #id
  3. Ichma-ich
  4. select_one

Vazifa 6: Ma'lumot

Yig'ish:

  1. find_all
  2. Dict
  3. Filtr
  4. Statistika

Vazifa 7: O'ylash

HTML scraping mo'rt (fragile) — sayt dizayni o'zgarsa, scraper buziladi (klass nomi o'zgarsa, find topmaydi). Bu API (barqaror — 26.3) dan farq. Nima uchun "HTML inson uchun, API dastur uchun", va nega HTML scraping doim kuzatish va yangilash talab qiladi — "barqaror interfeys" (API) va "o'zgaruvchan ko'rinish" (HTML) orasidagi farq nega muhim?

Javob

Qisqa javob: HTML inson uchun (ko'rinish — dizayn, rang, joylashuv — tez-tez o'zgaradi), API dastur uchun (ma'lumot — barqaror shakl, JSON — 19-qism). HTML scraping mo'rt, chunki HTML ko'rinishni ifodalaydi (klass nomi, tuzilma — dizaynga bog'liq) — sayt dizayni o'zgarsa (yangi ko'rinish), scraper buziladi (find("div", class_="kurs") — klass o'zgardi, topmaydi). API esa shartnoma (barqaror — o'zgarsa ogohlantiriladi, versiyalanadi). "Barqaror interfeys vs o'zgaruvchan ko'rinish" muhim, chunki: HTML'ga tayangan kod o'zgaruvchan narsaga tayanadi (mo'rt), API'ga tayangan kod barqaror narsaga (mustahkam). Shuning uchun API afzal (bor bo'lsa), HTML scraping oxirgi chora (kuzatish, yangilash kerak — dizayn o'zgarsa). "Ko'rinishga emas, shartnomaga tayan" — interfeys barqarorligi tamoyili.

1. HTML vs API

HTML API
Inson uchun (ko'rinish) Dastur uchun (ma'lumot)
Dizayn (o'zgaruvchan) Shartnoma (barqaror)
Mo'rt scraping Mustahkam
Oxirgi chora Afzal

2. Nega HTML mo'rt

HTML — ko'rinish (klass, tuzilma — dizaynga bog'liq). Dizayn o'zgarsa (yangi sayt) → scraper buziladi (find topmaydi). Ko'rinish tez-tez o'zgaradi.

3. API barqaror

API — shartnoma (ma'lumot shakli barqaror). O'zgarsa — versiyalanadi, ogohlantiriladi. Kod ishonch bilan tayanadi.

4. Barqaror interfeys tamoyili

Kod barqaror narsaga tayanishi kerak (API — shartnoma), o'zgaruvchan emas (HTML — ko'rinish). "Ko'rinishga emas, shartnomaga tayan" — mustahkam kod.

5. Muhandislik saboqlari

  1. HTML inson uchun (o'zgaruvchan), API dastur uchun (barqaror)
  2. HTML scraping mo'rt (dizayn o'zgarsa)
  3. Barqaror interfeysga tayan (API)
  4. Scraping kuzatish/yangilash talab qiladi

6. Xulosa

  1. HTML — ko'rinish (mo'rt), API — ma'lumot (barqaror)
  2. Scraping dizaynga bog'liq (buziladi)
  3. API afzal (shartnoma)
  4. Barqaror interfeysga tayan

Nimani mustahkamlaydi: 2.1–2.8-bo'limlar.

Xulosa

Bu darsda BeautifulSoup bilan HTML tahlilni o'rgandik.

Eng muhim uch fikr:

  1. BeautifulSoup va topish. BeautifulSoup (bs4) — HTML matnini daraxt (tuzilma) qiladi va oson qidirish beradi: BeautifulSoup(html, "html.parser"). find (birinchi mos teg — bitta; topmasa None — tekshir), find_all (barcha mos — ro'yxat, siklda). Klass/atribut bo'yicha: find("div", class_="kurs") (class_ — class Python kalit so'zi), find(id="..."), find("a", href="..."). Real HTML'da klass bilan aniq (barcha div emas, faqat kerakli). requests (yuklash) + BeautifulSoup (tahlil) — scraping.

  2. Atributlar, matn, CSS. Atributlar: tag["href"] (havola), tag.get("href") (xavfsiz — yo'q bo'lsa None), tag["data-id"] (maxsus), .attrs (barcha). Matn: tag.text (teg matni), .strip() (bo'sh joyni tozala — real HTML'da ko'p). CSS selektorlar (select): .kurs (klass — nuqta), #asosiy (id — diez), div.kurs span.nom (ichma-ich), select_one (birinchi). find/find_all ga muqobil (CSS qisqaroq). Tuzilgan ma'lumot yig'ish: find_all → siklda har biridan (nom, narx) → dict → ro'yxat (pandas/CSV uchun tayyor). Bu scraping maqsadi (iflos HTML → toza ma'lumot).

  3. HTML scraping mo'rt. HTML scraping mo'rt (fragile) — sayt dizayni o'zgarsa, scraper buziladi (klass nomi, tuzilma o'zgarsa — find topmaydi). Bu API (barqaror) dan farq: HTML inson uchun (ko'rinish — dizayn, tez-tez o'zgaradi), API dastur uchun (ma'lumot — barqaror shartnoma, 19-qism). Yechim: API afzal (bor bo'lsa), mustahkam selektor (klass — nth-child emas), xato boshqarish (find None), kuzatish (buzilganda sezish). "Ko'rinishga emas, shartnomaga tayan" — interfeys barqarorligi tamoyili. HTML scraping doim kuzatish, yangilash talab qiladi (mo'rt, lekin ba'zan yagona usul — API yo'q). BeautifulSoup — HTML'dan ma'lumot ajratishning asosi (26.5 dinamik sahifa, 26.6 etika).

Keyingi darsda Selenium / Playwright ni o'rganamiz: JavaScript bilan yuklanadigan dinamik sahifalarni scraping — requests ko'rmaydigan (brauzer kerak) kontentni olish.

Ulashish:Telegram'da

Izohlar (0)

Izoh yozish uchun kiring.

  • Hozircha izoh yo'q. Birinchi bo'ling!