lesson1Title

lesson2Title

lesson3Title

lesson4Title

lesson5Title

lesson6Title

pythonIntroCrawlingChapter2Title

lesson7Title

lesson8Title

lesson9Title

lesson10Title

lesson11Title

pythonIntroCrawlingChapter1Title

pythonIntroCrawlingChapter3Title

pythonIntroCrawlingChapter4Title

from bs4 import BeautifulSoup

# Example HTML document
html_doc = """
<html>
 <head>
 <title>The Codefriends' story</title>
 </head>
 <body>
 The Codefriends' story
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 ...
 </body>
</html>
"""

# Create a BeautifulSoup object
soup = BeautifulSoup(html_doc, 'html.parser')

# Extract the content of the title tag
title = soup.title.text
print('Title:', title) # Output: The Codefriends' story

print('-' * 10)

# Extract the href attribute values of 'a' tags
for link in soup.find_all('a'):
 print(link.get('href'))

# What is BeautifulSoup?

`BeautifulSoup` is a Python library that facilitates web scraping by extracting and parsing data from HTML files.

Parsing is the process of analyzing HTML documents of a webpage to extract the desired data. BeautifulSoup helps perform this parsing task easily.

 

## Features and Characteristics of BeautifulSoup

1. `Support for Various Parsers`

 - BeautifulSoup supports several types of parsers for parsing HTML/XML documents.

 - The most commonly used parsers are `html.parser` (standard Python library) and `lxml`.

2. `Easy Data Extraction`

 - It allows easy searching for specific tags, IDs, classes, etc.

 - It can effectively extract various elements of a webpage, such as text and attribute values.

3. `Handling Complex HTML Structures`

 - It can easily navigate and extract required data from nested tags or complex HTML structures.

 - It uses the hierarchical relationship of tags to find the exact data location.

4. `Flexible Search Methods`

 - It allows data searching using various methods, such as CSS selectors and regular expressions.

 - It can also find data with specific patterns by combining multiple conditions.

 

## Usage

```python title="Example of Using BeautifulSoup Library"
from bs4 import BeautifulSoup

# Example of an HTML document
html_doc = """
<html>
 <head>
 <title>The Codefriends' story</title>
 </head>
 <body>
 The Codefriends' story
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 ...
 </body>
</html>
"""

# Creating a BeautifulSoup object
soup = BeautifulSoup(html_doc, 'html.parser')

# Extracting contents of the HTML title tag
title = soup.title.text
print('Title:', title) # Output: The Codefriends' story

print('-' * 10)

# Extracting href attribute values of 'a' tags
for link in soup.find_all('a'):
 print(link.get('href'))
```

 

## Practice

Click the _`Run Code`_ button on the right side of the screen to check the crawling results or modify the code!

python_execution