How To Use Http Headers With Axios To Avoid Detection

Mihnea-Octavian Manolache on Jan 27 2023

HTTP headers are an important aspect of any HTTP request or response, as they allow the client and server to exchange additional information about the request. When building a Node.js web application, it is important to have a reliable and easy-to-use HTTP client for making requests to APIs and other servers.

JavaScript and Node JS in particular offer quite a few options when it comes to HTTP clients. However, one of my personal favorites (and certainly one of the top 5 JavaScript HTTP clients) is Axios. That is why today, we will focus on setting HTTP headers Axios.

Axios is a widely-used JavaScript library that simplifies the process of making HTTP requests and is especially useful when working with APIs in a Node.js environment. In this article, we'll explore the various ways that Axios can be used to work with HTTP headers. By the end of it, you should be able to understand:

• What is an HTTP header and how do they work
• Why is Axios a great HTTP client for Javascript
• How to use HTTP headers Axios in a real web app

How The HTTP Communication Works

Before diving into the definition of an HTTP header, I think it’s important to have at least an overview of how the HTTP protocol works. As I am sure you already know, the Hypertext Transfer Protocol (HTTP) is the foundation upon which the web is built today. On a high level, it allows information to be transferred between a server and its clients. The actual flow of this exchange of information looks something like this:

1. The client opens a new TCP connection
2. It sends a message to the server, which is called the HTTP request
3. The server receives and interprets the request
4. It then sends a message back to the client, which is called the HTTP response
5. The client reads the message and continues or closes the connection

The important part on which we will focus today is the message, particularly the message sent by the client. In order for the communication to be efficient between the server and the client, the messages need to be formatted as described by the HTTP protocol. When it comes to the HTTP request, the elements that compose the message are:

The method describes what we want to perform with our request (if we want to receive, send, update information etc.)

• The path, the URL location we’re trying to reach
• The version HTTP protocol we’re using
• The HTTP headers, which are used to send additional information and metadata alongside our request
• The body, in case we’re using a method that sends information to the server (such as a POST request)

What Are HTTP Headers Axios And How Do They Work

Simply put, HTTP headers are fields that pass additional information and metadata to the message. Again, by message, we understand the request when it is sent by the client and the response when it is sent by the server. So both the server and the client can pass and receive headers. For example, say you want to open a persistent connection to the server. By default, HTTP connections are closed after each request. To avoid this, all you need to do is pass the `Keep-Alive` header.

When it comes to HTTP headers with Axios, there really is nothing special about it. As we’ve discussed, Axios is an HTTP client and we’ve already established that HTTP clients can send and receive headers.

Why Axios Is A Great HTTP Client For JavaScript

Now that we have a highlight of how HTTP works, let’s talk ‘clients’. JavaScript offers a few options to ‘programmatically interact’ with a server over HTTP. Among the most popular choices, there is `axios`, `node-fetch` and `got`.

There are different opinions as to which one to use in the JavaScript community. Of course there are lots of pros and cons to each package, however, I myself chose Axios upon conducting a simple speed test between the three.

Here is the script I’ve used to test the speed:

// index.js

import { get_timing, array_sum } from './helpers.js'

import got from 'got'

import axios from 'axios'

const CALLS = 5

const send = async () => {

   const res = {}

  

   let start = process.hrtime()

   await got('https://httpbin.org/')

   const g = get_timing(start)

   res.got = g

   start = process.hrtime()

   await axios.get('https://httpbin.org/')

   const a = get_timing(start)

   res.axios = a

   start = process.hrtime()

   await fetch('https://httpbin.org/')

   const f = get_timing(start)

   res.fetch = f

   return res

}

let test_results = {

   got: [],

   axios: [],

   fetch: []

}

let avg = {}

console.log(`[i] Process started with ${CALLS} iterations.`)

for (let i = 0; i<=CALLS; i++) {

   let r = await send()

   Object.entries(test_results).map(([key, value]) => test_results[key].push(r[key]))

}

Object.entries(test_results).forEach(([key, value]) => {

       console.log(`\n[+] ${key}`)

       console.log(`    [i] Average: ${array_sum(value)/value.length}`)

       console.log(`    [i] Values: ${value}`)

       avg[key] = array_sum(value)/value.length

   }

)

console.log(`\n🚀🚀🚀 WINNER: ${Object.keys(avg).reduce((key, v) => avg[v] < avg[key] ? v : key)}  [${CALLS} calls sent] 🚀🚀🚀`)

And here are the `helper` functions:

// helpers.js

export const get_timing = (start) => {

   const NS_PER_SEC = 1e9

   const NS_TO_MS = 1e6

   const diff = process.hrtime(start)

   return (diff[0] * NS_PER_SEC + diff[1]) / NS_TO_MS

}

export const array_sum = (array) => {

   return array.reduce((accumulator, value) => {

     return accumulator + value

   }, 0)

}

I’ve tested with 5, 10, 20 and 30 requests sent with each package, 10 iterations each time and here is the overview of the results:

By iteration, I am referring to the number of script executions, using this bash formula, which generates a .txt file with the results from each iteration:

~ » for i in {1..10}

do

node got.js > "${i}.txt"

echo "${i} done"

done

As you will see if you check the detailed results table, there are different timings for each batch and sometimes Axios is not the fastest. Overall, however, Axios scored an average of 387 milliseconds, with a half second faster than its opponents. Got and Fetch had a very similar response time of approximately 435 milliseconds on average. This being said, if speed is important for your project, Axios is maybe the best HTTP client for you.

How To Send HTTP Headers With Axios

I personally find that learning by practice gives almost instant results. So now that we have both the knowledge and the tools to send HTTP headers, let’s start working on a small project. In this section, we will set up a new Node project, install Axios and use it to send HTTP headers to a server.

Set Up The Project

Before moving forward, make sure that your machine is equipped with:

• Node JS - you can download it from here
• An IDE - my day-to-day IDE is Visual Studio Code

Hint: You can check to see if you have Node JS installed by typing the following command in your terminal:

~ » node -v  

v19.3.0

Now let’s create a new folder and open it in our IDE. If you’re using a UNIX like machine (Mac or Linux), you can programmatically create a new directory, from your terminal, by typing the following command:

~ » mkdir axios_project && cd axios_project 

~ » npm init -y

~ » npm i axios

~ » touch index.js

~ » code .

These commands will:

1. Create a new directory (named ‘axios_project’) and navigate to it
2. Initialize a new Node JS project inside the directory
3. Install `axios` inside your project
4. Create a new ‘index.js’ file
5. Open your IDE on the current project

Code To Learn

There are actually a few ways you can send HTTP headers with Axios. For example, you can use a config object as described here, or you can use the instance methods, that will automatically merge the configuration you pass with the instance configuration. You can also use the `axios.defaults.headers.common` object to set default headers for all Axios requests.

Also, note that Axios is a promise based HTTP client. This means that we’ll either have to await for it inside an asynchronous function, or resolve the response.

With these two aspects in mind, let us get started with some actual coding. We will work inside the ‘index.js’ file. For convenience, let’s recap what we need to do beforehand:

1. Import `axios` inside our file
2. Define a config object that will hold our headers
3. Pass the config to `axios` in order to make a request
4. Print the response inside our terminal

#1: Send a GET request using the config object

import axios from "axios"

const config = {

   method: 'GET',

   url: 'https://httpbin.org/headers',

   headers: {

       'HTTP-Axios-Headers': 'This is my custom header.'

   }

}

axios(config)

   .then((response) => {

       console.log(response)

   })

   .catch((err) => {

       console.log(err)

   })

Sending HTTP headers with Axios is as simple as it gets. In order to execute this script, simply run the following command inside your terminal:

~ » node index.js  

{

  status: 200,

  statusText: 'OK',

  headers: ...,

  config: ...,

  request: ...,

  data: {

    headers: {

      'Accept': 'application/json, text/plain, */*',

      'Accept-Encoding': 'gzip, compress, deflate, br',

      'Host': 'httpbin.org',

      'Http-Axios-Headers': 'This is my custom header.',

      'User-Agent': 'axios/1.2.2',

      'X-Amzn-Trace-Id': 'Root=1-63b54d94-7656f02113483dfa036c476c'

    }

  }

}

The entire response is quite large and it follows this schema. However, what we’re mostly interested in is the `data`, which holds the actual response we received from the server. Now check out the response from above. Remember we sent a custom header `Http-Axios-Headers` to the server and, as you can see, the server did receive it.

#2: Send a POST request using the method alias

import axios from "axios"

const data = {

   'foo':'bar'

}

const config = {

   headers: {

       'HTTP-Axios-Headers': 'This is my custom header.'

   }

}

axios.post('http://httpbin.org/post', data, config)

   .then(response => console.log(response.data))

   .catch(err => console.log(err))

Note how in order to send a POST request, I added a new `data` object to our script and I’ve also changed the URL. Now, if you run the script, you will find that this is the data received from the server:

{

  args: {},

  data: '{"foo":"bar"}',

  files: {},

  form: {},

  headers: {

    Accept: 'application/json, text/plain, */*',

    'Accept-Encoding': 'gzip, compress, deflate, br',

    'Content-Length': '13',

    'Content-Type': 'application/json',

    Host: 'httpbin.org',

    'Http-Axios-Headers': 'This is my custom header.',

    'User-Agent': 'axios/1.2.2',

    'X-Amzn-Trace-Id': 'Root=1-63b5508a-3a86493f087662d3169e80ee'

  },

  json: { foo: 'bar' },

  origin: '49.12.221.20',

  url: 'http://httpbin.org/post'

}

How To Use HTTP Headers In Axios For Web Scraping

If you are planning on using Axios for web scraping, please note that most websites have protection rules that block requests coming from automated software (web scrapers included).

Using HTTP headers, particularly the `User-Agent` header, can be a useful technique for avoiding detection when web scraping. The User-Agent header identifies the client browser and operating system to the server, and web servers may serve different content or block requests based on this information. By setting the User-Agent header you can mimic a common web browser, hence increasing your chances to bypass some detection mechanisms.

Here's an example of how to use the User-Agent header with Axios to avoid detection when web scraping:

import axios from "axios"

axios.defaults.headers.common['User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'

axios({

   method: 'GET',

   url: 'https://httpbin.org/get',

}).then(response => {

   console.log(response.data)

});

Note how this time I’ve used the config defaults option from Axios, which applies the header to all upcoming requests. In this example, the User-Agent header is set to mimic the Chrome browser on a Windows 10 operating system. You may want to try using different User-Agent values or a variety of different headers to see what works best for your specific use case.

It's worth noting that while changing the User-Agent header may help you avoid detection in some cases, it is not a foolproof method, and web servers may still be able to identify that you are using a web scraper. As such, it's a good idea to use a combination of techniques to avoid detection and stay within the bounds of the website's terms of service.

Conclusions

Using HTTP headers with Axios (or with any other HTTP client for that matter) can increase the efficiency of communication between a server and a client. Moreover, it can even help you avoid detection when you’re building a web scraper. As a matter of fact, at WebScrapingAPI we’re using various user agents as one of the basic evasion techniques.

Of course, detection is not limited to user agents, but it is a good place to start with. Here is a nice tutorial on how to avoid getting your IP address banned when web scraping that will help you get a better understanding of how evasions work.

Meanwhile, did you know that Web Scraping API allows you to set custom headers to your request? If not, learn more about it here.