TL;DR: If you are using express (3.x, 4.x), be aware that express uses qs for parsing request query strings. By default, the qs package parses a maximum of five levels in nested objects. Also, when parsing arrays qs package parses only 20, any array with members greater than 20 would be converted to an object with indexes as keys.
When making HTTP request, data can be sent via POST message body, or via query string. When sending complex data (large data, complex data structure) is it usually best to send data using HTTP POST method. Query string is less effective when sending large data because URL size is limited by both servers and browser. However, when it comes to sharing and reusing links, GET requests with query strings are more applicable. For example a link can be sent by email. Query strings can contain complex and nested data such as arrays, key value pairs.
Example:
HTTP GET http://example.com?fruits=Banana&fruits=Apple
Is converted to the array below in javascript
{ fruits: [ 'Banana', 'Apples' ] }
Now that we finished with the quick introduction to HTTP requests and Query string, let us dive into the main course. When building a nodejs website, the most popular framework is express. The express package uses package called qs for parsing request query strings. This package (qs) has some default configurations that cause some limitations.
Parsing of objects in requests' query string
Few quotes from qs documentation related to object parsing
By default, when nesting objects qs will only parse up to 5 children deep.
This depth can be overridden by passing a
depthoption toqs.parse(string, [options])
The depth limit helps mitigate abuse when qs is used to parse user input, and it is recommended to keep it a reasonably small number. For similar reasons, by default qs will only parse up to 1000 parameters. This can be overridden by passing a
parameterLimit.
A solution to this problem is to define a custom query parser and define custom options when using qs.parse.
Example: Set depth to 50.
const app = express();
app.set('query parser', function(str) {
return qs.parse(str, { depth: 50 });
});
Parsing of arrays in requests' query string
A quote from qs documentation related to array parsing
qs will also limit specifying indices in an array to a maximum index of
20. Any array members with an index of greater than20will instead be converted to an object with the index as the key.
Let see some examples
Example 1: Array defined using the same key with square brackets and multiple values
HTTP GET http://example.com?arr[]=1&arr[]=3&arr[]=3&arr[]=4&arr[]=5&arr[]=6&arr[]=7&arr[]=8&arr[]=9&arr[]=10&arr[]=11&arr[]=12&arr[]=13&arr[]=14&arr[]=15&arr[]=16&arr[]=17&arr[]=18&arr[]=19&arr[]=20&arr[]=21&arr[]=22
Is parsed to
{ arr: [ 1, 2 , 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21] }
Example 2: Array defined using the same key with multiple values
HTTP GET http://example.com?arr=1&arr=3&arr=3&arr=4&arr=5&arr=6&arr=7&arr=8&arr=9&arr=10&arr=11&arr=12&arr=13&arr=14&arr=15&arr=16&arr=17&arr=18&arr=19&arr=20&arr=21&arr=22
Is parsed to
{ arr: [ 1, 2 , 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21] }
Example 3: Array defined using the same key with square brackets and index and multiple values
HTTP GET http://example.com?arr[0]=1&arr[1]=3&arr[2]=3&arr[3]=4&arr[4]=5&arr[5]=6&arr[6]=7&arr[7]=8&arr[8]=9&arr[9]=10&arr[10]=11&arr[11]=12&arr[12]=13&arr[13]=14&arr[14]=15&arr[15]=16&arr[16]=17&arr[17]=18&arr[18]=19&arr[19]=20&arr[20]=21&arr[21]=22
Is parsed to
{
arr: {
0: "1",
1: "2",
2: "3",
3: "4",
4: "5",
5: "6",
6: "7",
7: "8",
8: "9",
9: "10",
10: "11",
11: "12",
12: "13",
13: "14",
14: "15",
15: "16",
16: "17",
17: "18",
18: "19",
19: "20",
20: "21",
21: "22"
}
}
In the third example we see there is a change in behaviour. We get an object instead of an array.
Solutions
Listed below are 2 solutions to handling large arrays in query string using express.
1. Object.values()
A quick solution to this is using Object.values() to always get an array.
Example: How Object.values work.
const array = [1,2,3];
const object = {a: 1, b: 2, c: 3 };
console.log(Object.values(array));
// prints [1,2,3]
console.log(Object.values(object));
// prints [1,2,3]
Example: Sample request handling
app.get('/path', (req, res) => {
const myBigArray = Object.values(req.query.bigArray);
//...
});
In the snippet above there is a processing of the request query data before it is used. It is not the most efficient since it requires fixing query data in multiple request handlers. But it good for one-off cases. The next solution is a much better one.
2. Custom Query Parser Middleware
A custom query parser can be defined as shown in the code sinppet below.
const app = express();
app.set('query parser', function(str) {
return qs.parse(str, { arrayLimit: 0 });
});
Conclusion
Generally you should not have query string with data that is too complex or with too many nested levels. And if you decide to do so and are using express version 3 or 4, be aware of the default configuration.
