Learn How to Import Data from a Website into Python

In this tutorial, we’ll see how to import data from a website in Python. For this task, you’ll need two Python libraries requests and beautifulsoup4.

Install requests and beautifulsoup4 library in python

Run below command to install requests and beautifulsoup4. You can install them using pip:

pip install requests
pip install beautifulsoup4
# or
pip install requests beautifulsoup4

Example 1

Create a Python Web Scraping Tutorial Fetching Data from Wikipedia using BeautifulSoup and Requests.

import requests
from bs4 import BeautifulSoup

response = requests.get('https://www.wikipedia.org/')
soup = BeautifulSoup(response.text, 'html.parser')
print(soup.prettify)

Example 2

we’ll create a Python web scraping example to demonstrate how to fetch data from Wikipedia using BeautifulSoup and Requests. We’ll focus on extracting headlines from the In the news section.

import requests
from bs4 import BeautifulSoup

# URL of the Wikipedia main page
url = 'https://en.wikipedia.org/wiki/Main_Page'

# Send a GET request to the URL
response = requests.get(url)

# Parse the HTML content of the page
soup = BeautifulSoup(response.text, 'html.parser')

# Find the 'In the news' section by its ID
in_the_news_section = soup.find('div', id='mp-itn')

# Extract headlines within that section
headlines = in_the_news_section.find_all('li')

# Print each headline
for headline in headlines:
  print(headline.text.strip())