Loading URLs

Now that we've confirmed the required libraries and system requirements, we will proceed with loading the URLs. While looking for contents from a URL, it is also necessary to confirm and verify the exact URL that has been chosen for the required content. Contents can be found on single web pages or scattered across multiple pages, and it might not always be the HTML sources we are looking for.

We will load some URLs and explore the content using a couple of tasks.

Task 1: To view data related to the listings of the most popular websites from Wikipedia. We will identify data from the Site, Domain, and Type columns in the page source.

We will follow the steps at the following link to achieve our task (a data extraction-related activity will be done in Chapter 3, Using LXML, XPath and CSS Selectors): https://en.wikipedia.org/wiki/List_of_most_popular_websites. 

Search Wikipedia for the information we are looking for. The preceding link can be easily viewed in a web browser. The content is in tabular format (as shown in the following screenshot), and so the data can be collected by repeatedly using the select, copy, and paste actions, or by collecting all the text inside the table.

However, such actions will not result in the content that we are interested in being in a desirable format, or it will require extra editing and formatting tasks being performed on the text to achieve the desired result. We are also not interested in the page source that's obtained from the browser:

After finalizing the link that contains the content we require, let's load the link using Python. We are making a request to the link and willing to see the response returned by both libraries, that is, urllib and requests:

1. Let's use urllib:

The urlopen() function from urllib.request has been passed with the selected URL or request that has been made to the URL and response is received, that is, HTTPResponse. response that's received for the request made can be read using the read() method.

2. Now, let's use requests:

Here, we are using the requests module to load the page source, just like we did using urllib. requests with the get() method, which accepts a URL as a parameter. The response type for both examples has also been checked.

In the preceding examples, the page content—or the response object—contains the details we were looking for, that is, the Site, Domain, and Type columns.

We can choose any one library to deal with the HTTP request and response. Detailed information on these two Python libraries with examples is provided in the next section, URL handling and operations with urllib and requests.

Let's have a look at the following screenshot:

Further activities like processing and parsing can be applied to content like this in order to extract the required data. More details about further processing tools/techniques and parsing can be found in Chapter 3, Using LXML, XPath, and CSS Selectors, Chapter 4, Scraping Using pyquery – a Python Library, and Chapter 5, Web Scraping Using Scrapy and Beautiful Soup. 

Task 2: Load and save the page content from https://www.samsclub.com/robots.txt and https://www.samsclub.com/sitemap.xml using urllib and requests.

Generally, websites provide files in their root path (for more information on these files, please refer to Chapter 1, Web Scraping Fundamentals, the Data finding techniques for the web section):

• robots.txt: This contains information for the crawler, web agents, and so on

• sitemap.xml: This contains links to recently modified files, published files, and so on

From Task 1, we were able to load the URL and retrieve its content. Saving the content to local files using libraries methods and using file handling concepts will be implemented in this task. Saving content to local files and working on content with tasks like parsing and traversing can be really quick and even reduce network resources:

1. Load and save the content from https://www.samsclub.com/robots.txt using urllib:

The urlretrieve() function, that is, urlretrieve(url, filename=None, reporthook=None, data=None), from urllib.request returns a tuple with the filename and HTTP headers. You can find this file in the C://Users..Temp directory if no path is given; otherwise, the file will be generated in the current working directory with the name provided to the urlretrieve() method as the second argument. This was testrobots.txt in the preceding code:

In the preceding code, we are reading the URL and writing the content found using a file handling concept. 

1. Load and save the content from https://www.samsclub.com/sitemap.xml using requests: