How to use WebScrapingAPI to gather Yelp data

Robert Sfichi on Aug 12 2021

Have you ever wondered how businesses keep track of the competition? Of course, double agents, private investigators, and binoculars may sound fun, but real life isn’t like the movies. Actual competitor analysis is a tedious process that’s better left to the software. This way, you can focus on delivering the most value to your customers.

Nowadays, your competitors already have all the information you need in the open. With social media gaining a lot of traction in the last couple of decades, businesses are trying to be more present in their customer’s life by creating an online presence. The information is already there. We just have to find a way to automate the data gathering process.

In the following article, we will build a script that will help us gather all the information we need regarding our business competitors by taking advantage of their public Yelp profiles.

Why you should scrape competitor data

Investigating competitors' product pricing, overseeing their product line and categories, or analyzing their social media strategy are just three of the most valuable tactics you can use to improve your business.

Finding and comparing product prices is not a very challenging task, but it is time-consuming. Using a web scraping tool can help you automatically get updated information regarding your competitor’s pricing strategy and flash sales.

Scraping your competitors’ product lines and categories helps you better understand the trends that define the market. For example, if you own a restaurant, you can find out that a specific dish like the vegan burger is trending in your city, so you can make adjustments to your menu to drive more sales.

Social media isn’t useful just for strengthening the bond between business and customer. By analyzing your competitors' social media strategy, you can find out what works and what doesn’t in just a couple of minutes. You can easily find answers to questions like “How many times do they post per week?” or “What visuals keep engagement at a high level?” just by scraping.

Why Yelp is the perfect place to scrape competitor data

Yelp is one of the most popular online platforms where users can share their experiences while helping others make informed decisions about local businesses. If you own a business or are looking to start one, finding out valuable information about the competition can improve your success rate by a wide margin. Pieces of information about the exact location, working hours, delivery services, or business type are just a couple of the ones you can find by browsing Yelp.

What’s even better is that you can decrease the time you spend analyzing competition by using a web scraping tool and focus more on building a better business. This strategy will pay off in the long term as automating the data gathering process can become one of the most tedious and time-consuming tasks.

How to use WebScrapingAPI to extract data from Yelp

In the following section, we will use Node.js and a couple of libraries like got and jsdom to create the script that will get all the data from a Yelp page and format it nicely to be as understandable as possible. Let’s see how all the information presented above can be converted to tangible results:

1. Find the data you need

Let’s assume we are interested in opening up a restaurant in New York City. We would want to create a list of competitors that includes helpful information like:

• Number of reviews
• Restaurant type
• Pricing
• Location

The URL we are going to scrape is the following: https://www.yelp.ie/search?find_desc=restaurant&find_loc=New+York%2C+NY%2C+United+States&ns=1.

You can scrape in tandem with the article if you start your free trial with 5000 requests and access to all functionalities to test our product. You can use whatever scraping tool you feel most comfortable with. Just know that from here on, the tutorial will focus on how to get the results using WebScrapingAPI.

After successfully creating a free account, access the API Playground page by hitting the “Use API Playground” button on the Dashboard page. The page should look like this:

Just as the name suggests, this is the place where we can test the scraping tool before creating our script. Let’s copy the URL presented above in the URL input (left column), scroll down a little bit, and hit the “Send API Request” button. This action should return a result that looks like this:

Now let’s build the script that is going to do most of the work for us.

2. Setting up the project

Create the project’s folder, open up a terminal window, and navigate to it. Just as we have said before, we will use the ‘got’ library to make the requests and ‘jsdom’ for our parsing needs. After you have accessed the project’s folder, run the following commands in the terminal:

npm init -y
npm install got jsdom

The project should now include the ‘package.json’ files. Create a new file called ‘index.js’ and open it up using your preferred IDE. If you do not have any installed, we recommend downloading Visual Studio Code, as it’s considered the most popular IDE at the moment.

3. Making a request

Let’s set the parameters, make the request and parse the HTML. Write the following lines in the newly created ‘index.js’ file:

const {JSDOM} = require("jsdom")

const got = require("got")

(async () => {
const params = {
    api_key: "YOUR_API_KEY,”
    url:"https://www.yelp.ie/search?find_desc=restaurant&find_loc=New+York%2C+NY%2C+United+States&ns=1"
}

const response = await got('https://api.webscrapingapi.com/v1', {searchParams: params})


const {document} = new JSDOM(response.body).window

const competitors = document.querySelectorAll('.container__09f24__sxa9-')

console.log(competitors)

})()

Be sure to replace the “YOUR_API_KEY” string with the API key provided to you by our service. You can find it on the dashboard page.

By running the script, we make a request to the WebScrapingAPI service to get the Yelp page HTML. We then collect all the elements on the page containing information about our competitors and log it to the screen using the ‘console.log’ function.

4. Inspecting the page

Let’s get back to the Yelp page and find out how we can select only the information we need. Right-click on the first restaurant’s name and click ‘Inspect.’

You’ll get a new window containing the HTML source code:

Let’s find all the information we need by looking for the HTML elements that contain it. In the picture presented above, we can easily see that the element containing the restaurant’s name has a CSS class of ‘css-166la90’. To get all our competitors’ names, we have to select all the elements on the page with that class. We will do the same to all the details presented before, like the number of reviews, restaurant type, pricing, and location.

5. Formatting the information

We will now get all the information we need from the resulting API response. We will fetch and format the restaurant’s name, review score, restaurant type, price range, and location. Add the following code lines to the ‘index.js’ file.

competitors.forEach((competitor) => {
    if (competitor) {
        const name = competitor.querySelector('.css-166la90')
        if (name) competitor.name = name.innerHTML

        const reviewScore = competitor.querySelector('.reviewCount__09f24__EUXPN')
        if (reviewScore) competitor.review_score = `${reviewScore.innerHTML}/100`

        const types = competitor.querySelectorAll('.css-1hx6l2b')
        if (types) {
            competitor.types = []
            for (type of types) competitor.types.push(type.innerText)
       }

       const priceRange = competitor.querySelector('.priceRange__09f24__2O6le')
       if (priceRange) competitor.price_range = priceRange.innerHTML

       const locationContainer = competitor.querySelector('.priceCategory__09f24__Ylk7h')
       if (locationContainer) {
	     let location = locationContainer.querySelector('.css-e81eai')
	     competitor.location = location
       }

       results.push(competitor)
    }
})

console.log(results)

In the end, we will have an array of objects, and each of them will contain every competitor on the page and its specific data.

As you can see, scraping Yelp data using WebScrapingAPI is quite easy. We have to use a scraping API to get the HTML content, parse the response, get the relevant information from each element on the page, and add everything to a list.

Getting to know your competition without any extra work

Trying to create a new business nowadays takes a lot more than having a great product. There are endless opportunities depending on how creative you can be. Some of the most valuable strategies business owners should pay attention to are:

• creating an outstanding online presence
• working on providing the most value for the lowest price
• having a complete understanding of the competition’s advantage

All of these strategies can prove vital in one’s business. It feels good to know that web scrapers offer a huge help in tackling these problems. Adding automation to the data gathering process may be the easiest step to improve their business.

We try to offer a helping hand by creating the necessary tools for these kinds of jobs. Thank you for reading all the way through and remember that WebScraping API has a free trial after which users are downgraded to the free plan so you can test the API to your heart’s content without spending a dime.