3 Challenges Extracting Data from Ecommerce Websites

The competition within the e-Commerce market is frenzied. With more than 2 millions sellers on Amazon alone, a massive volume of listings are updated daily. Many businesses thus opt for web scraping to extract data. However, there are three obstacles you need to be aware of that might roadblock your trip of getting quality data, and eventually impact your business in a bad way.

Big Problem 1: large scale extraction

For e-Commerce store owners, it is a daily chore to manage 20+ subcategories under a major category. These add up to a total of more than a hundred items. It doesn’t sound realistic to copy and paste each product’s information including SKU, thumbnail image, description, shipping and customer reviews into a single spreadsheet for records and analysis on a daily basis. The monotonous work not only takes up your time but also leads to lower data quality and precision.

Outsourcing or In-house Team?

In most cases, owners would opt for outsourcing or an in-house team to build a web crawler for them. Notice that all websites are versatile and vary in structures. There is a great possibility that you need to adjust the crawler once a while. The service and maintenance are quite an expense each year. In addition, if the vendor isn’t reliable, you will put the data at risk.

A Web scraping tool is a great alternative

An intuitive web scraping tool like Octoparse would help you achieve a better result at a lower cost. Web scraping is not the privilege of a programmer anymore. And it shouldn’t burden you with an excessive cost.

  • Simplicity: You can build a crawler with simple clicks and drag-and-drops. Better yet, no technical skills are required to use the tool
  • Security： Octoparse allows collaborative work. You have control over the data source and the data’s quality. Extracted data will be only handled within the hands of trusted agents
  • Lower cost: It minimizes the maintenance cost as you can debug the flow by yourself within a few clicks. Compared to a 3rd party service, a web scraping tool reduces the cost per data, and increase the gross margin

Here is how you can leverage Octoparse to solve the problem and upscale your business within a few steps:

  • Download and install Octoparse on your local computer
  • Choose a web scraping template under the Product category
  • Fill in the parameters
  • Run the task on your local device or on a cloud
  • Export the data into a file format of your choice, e.g. spreadsheet or database

By connecting with your database via API, you can update your database automatically. As such, you are able to monitor most major e-Commerce websites like eBay, Flipkart, Target and BestBuy concurrently.

The best part is that Octoparse is having a big sale for the upcoming Black Friday event which is a perfect opportunity to get to know their product.

Big Problem 2: Getting blacklisted/blocked

Another major challenge faced by many is getting blocked by a targeted website. There are many reasons that can trigger such a defensive act, and the most common one is due to the abnormality of the IP address.

For example, when you ask for too many resources in a given time window, the server will think that the user is not a real person. In order to prevent abuse, the server blacklists your IP address. An IP address is your identity to communicate on the internet with an online resource. It’s like a driver license that can get you a pitcher of beer. You can’t get in a bar without showing your identity.

To avoid being blacklisted, a scraper will need to act like a human. What makes a bot different than a human being in front of a computer? As a crawler is scripted, its behavior follows a certain pattern. However, humans’ interactions with the internet are unpredictable. We need to break off the patterns by doing some random acts.

There are three things you can do:

  • Switch User-Agent: A user-agent indicates which browser the website is interacting with. We reveal the robotic identity if consistent requests are sent with the same user-agent. Octoparse provides a list of user-agents that allows the crawler to switch within a certain time interval.
  • Slow down your crawling speed: It’s self-explanatory that humans can’t browse at a crazily fast speeds, but a bot can and will.
  • Rotate IP address: Allocate requests to different IP addresses to make it more difficult for servers to detect an abnormality. IP rotation is the most effective method to keep web scraping smooth without interruption.

There are many IP Proxy providers that are able to change your IP address. However, the quality of the networks vary.

IP Rotation Solution

Luminati leads the market with the largest residential proxy network in the world. They provide 4 types of networks:

  • Rotating residential proxies: you can exchange real-user IPs from city to city across the world. It is extremely useful for information gathering on market analysis and price comparison
  • Mobile Proxy network: it mimics real mobile users, which allows you to work on a marketing campaign on mobile-centric social media platforms
  • Static residential proxies: they simulate real residential IPs without IP rotation which ensures uninterrupted task completion
  • Data Center proxies: this allows you to share proxies which is helpful when a mass crawling is needed

Big Problem 3: Anti-scraping technique 一 CAPTCHAs

However, the above problems are not all. Another problem you may encounter while web scraping is CAPTCHA issues.

What is CAPTCHAs

In order to defend against the malicious scrapers who send too many requests in a given time window and put a strain on their server, some websites might challenge the users with CAPTCHA to single out the automated bots. That's where CAPTCHA solving services are taking the stage.

The idea of solving Captchas is quite simple: a customer sends the Captcha to the server. The server sends Captchas to the agent who solves it and then sends the answer back. It takes around 10 seconds after the initial request was made, the customer can send a request every 5 seconds until it gets solved.

It raises the bar for the data extraction as CAPTCHA appears in many forms and scrapers usually are not intelligent enough to get passed.

CAPTCHA Types

The most common types are:

  • Graphic images, which need to be decoded to text
  • Mathematical CAPTCHA (where you need to do some operations and type answer, like 7 + 5 = ??)
  • Puzzle CAPTCHA
  • Interactive CAPTCHA: reCAPTCHA, FunCaptcha.

Moreover, CAPTCHA evolves and generates other variants like reCAPTCHA v2 and reCAPTCHA v3 that get harder to pass.

How to deal with CAPTCHA?

  1. The whole purpose of CAPTCHA is to prevent abusive traffic imposed on the website. It is important not to overburden the server by sending too many requests in a given time window. With an intuitive web scraper like Octoparse, the problem is easily being taken cared of by imposing artificial speed.
  2. Some simple CAPTCHAs like login form CAPTCHAs can also be resolved by Octoparse.
  3. There are many anti-CAPTCHA providers who are able to solve advanced CAPTHCAs like a mathematical CAPTCHA or an Image-based CAPTCHA.

Take 2Captcha as an example. Their service carries a few notable pros against others on today’s anti-captcha market:

  1. High Solving Speed: 14 seconds for normal Captchas and 38 seconds for reCAPTCHA on average
  2. High accuracy rate with up to 99% (depends on the CAPTCHA type).

There are some other minor challenges that would prevent you from getting quality data from e-Commerce websites like extracting data from consecutive pages, XPath editing and data cleaning. But don’t worry, Octoparse is crafted for non-coders to keep fingers on the pulse of the latest market news.