Best Practices for Advanced Python Web Scraping

Scraping is an easy concept in its crux, however, it's also a tricky one! It's like the cat-and-mouse game between a website owner as well as a developer working in the legal area. This blog throws light on a few obstructions that a programmer might face while doing web scraping, as well as different ways of getting around.

What is Web Scraping?

What-is-Web-Scraping

Web scraping services are the work of extracting data from different websites and other online sources. This could either be a manual procedure or an automated process. Although manually scraping data from web pages could be a redundant and tedious procedure that justifies the whole ecosystem of different libraries and tools built to automate data scraping procedure. In auto web scraping services, rather than letting a browser reduce pages, we utilize self-written scripts for parsing raw responses from a server. In this blog post, we will utilize "Web Scraper" for implying "Automated Web Scraping."

How to Do Web Scraping?

How-to-Do-Web-Scraping

Before moving to things, which can make web scraping complicated, let's break the procedure of data scraping into comprehensive steps:

Visual inspection: Finding what to scrape

Making an HTTP request for a webpage

Parsing HTTP responses

Use relevant data

The initial step involves the use of in-built browser tools (including Chrome DevTools as well as Firefox Developer Tools) for finding the information we want on a webpage as well as identifying structures or patterns to scrape it programmatically.

Following are the steps that involve systematically making requests for a webpage as well as implementing the logic of scraping data, using patterns that we have identified. Finally, we utilize the data for whatever objective we planned to.

For instance, let's say that we wish to scrape the total subscribers of PewDiePie as well as compare that with T-Series. An easy Google search results in a YouTube Subscriber Count Page of Socialblade.

Difficulties of Web Scraping

Difficulties-of-Web-Scraping

Analyzing the Request Rate

Asynchronous Loading as well as Client-Side Rendering

Authentication

Captchas and Redirects

Choosing the Right Libraries, Frameworks, and Tools

Header Inspection

Honeypots

Pattern Detection

Resolving Complexities of Python and Web Scraping

For data scraping in Python, many tools are available. We'll use some popular options as well as when to utilize which. For extracting easy websites rapidly, we've found a grouping of Python Requests (for handling sessions as well as making HTTP requests) as well as BeautifulSoup (to parse the response as well as navigate through that to scrape data) to make a perfect pair.

For big-size web scraping projects (where we need to collect as well as process ample data as well as cope with non-JS-related difficulties), Scrapy has been extremely useful.

Scrapy is the framework that extracts many intricacies to scrape efficiently (memory utilization, concurrent requests, etc.), and permits to plug in a group of middleware (for redirects, sessions, cookies, caching, etc.) to cope with various complexities. Scrapy gives a shell also, which can assist in rapid prototyping and authenticating your data extraction approach (responses, selectors, etc.). The framework is quite extensible, mature, and has a very good support community too.

For heavy JavaScript websites (or websites that look too complex), Selenium is generally a way to go. Though data scraping with Selenium isn't as effective as compared to Beautiful Soup or Scrapy, it always offers you the required data (that is a thing, which matters the most).

Authentication Handling

Authentication-Handling

For authentication, we'll need to preserve cookies as well as persist login, the better option is to make a session that deals with all. For unseen fields, we could manually try and log in as well as inspect the payloads getting sent to a server with network tools given by a browser to recognize the hidden data getting sent.

We could also review what headers are getting sent to the server with browser tools so that we could replicate the behavior in code also like if authentication relies on headers like Authentication and Authorization). If a website uses an easy cookie-based authentication (that is highly doubtful these days), we could also copy cookie contents as well as add that to a scraper's code (we could use in-built browser tools to do that).

Handling of Asynchronous Loading

Handling-of-Asynchronous-Loading

Detect Asynchronous Loading

We can find asynchronous loading during a visual inspection by viewing a page source (a "View Source" alternative in a browser by doing a right-click) as well as searching for the content we're searching for. In case, you don't get text in a source, and yet you can still see that in a browser, then probably it's getting rendered using JavaScript. An additional inspection could be done using the browser's network tools to review if all XHR requests are getting made by a website.

Bypassing Asynchronous Loading

Bypassing-Asynchronous-Loading

Use a Web Driver

Using a web driver is like a simulation of the web browser with interfaces to get controlled using scripts. This is capable to do browser stuff including rendering JavaScript, organizing sessions and cookies, and more. Selenium Web Driver is the web automation framework specially designed for testing UI/UX of websites, however, it has become a well-known option for scraping dynamically rendered websites over time.

As web drivers are an imitation of browsers, they're source intensive as well as moderately slower compared to libraries including Scrapy and BeautifulSoup.

Inspect AJAX Calls

This technique works on the idea of "In case, it's getting displayed on a browser, this needs to come from someplace." We can utilize browser developer tools for inspecting AJAX calls as well as try and find requests that are accountable to fetch the data we're searching for. We could require to set the X-Requested-With header for mimicking AJAX requests in the script.

Tackle Infinite Scrolling

We can deal with infinite scrolling by injecting a few JavaScript logics in selenium. Also, generally, the infinite scrolling comprises more AJAX calls to a server that we could inspect using different browser tools as well as replicate it in the scraping program.

Getting the Right Selectors

Getting-the-Right-Selectors

When we locate an element, which we wish to scrape visually, the following step is to discover a selector’s pattern for such elements, which we can utilize to scrape them from HTML. We could filter different elements depending on their attributes and CSS classes with CSS selectors.

CSS selectors are the first choice to do the scraping. Although another method to select elements named XPath (the query language to select nodes in the XML documents) could be helpful in some scenarios. This gives more flexible capabilities.

Handling Captchas and Redirects

Handling-Captchas-and-Redirects

Contemporary libraries like requests take care of the HTTP redirects through following them (maintaining the history) as well as returning a final page. Scrapy comes with the redirect middleware for handling redirects. These redirects aren't given much trouble if we redirect to the pages we seek. However, if we get redirected to any captcha, then problems may come.

Easy text-based captchas could be solved with OCR. Text-based captchas are smooth slopes for implementing these days using the arrival of advanced OCR methods (that depends on Deep Learning), therefore it's getting tougher to make images, which could beat machines as well as humans.

Handling Unstructured Responses and iframe Tags

Handling-Unstructured-Responses-and-iframe-Tags