cannot add scalar value without an associeted key

[mrahmatellah]

Hello everyone,

I have deployed a waf on cloud (azure) app gateway waf,

I have a false positive ( CANNOT ADD SCALAR VALUE WITHOUT AN ASSOCIETED KEY)

I don't know why waf detect a request as a bad traffic,
this is the totaly request:
`
POST https://edu-pp.tactileo.fr/profile/internal/permissions HTTP/1.1
cache-control: no-cache
Accept: application/json
Content-Type: application/json
Authorization: Bearer
Host: edu-pp.tactileo.fr
cookie: ARRAffinity=e66763d6606203ca589c508b2ec0185c5b193bb84d15f36169a81d20ed13dc09
accept-encoding: gzip, deflate
content-length: 13

[ "teacher" ]

the rule id generated is 0

@victorhora was telling me that:
"
The error message you mentioned "Cannot add scalar value without an associated key" usually happens when the JSON parser can't associate a key to a given JSON value as the code normally expects a key/value format (e.g. [profession: "teacher"]) data for JSON and your request only contains the value ([ "teacher" ]).

#1576 might be related so maybe you could retry with the current proposed solution for v2 at #1577.

Still, as far as I can remember this error message shouldn't be causing a false positive... Unless maybe if this issue is causing the request body parser to fail and triggering rule 200002..."

I have seen in: https://fossies.org/linux/modsecurity/apache2/msc_json.c
<
"int json_add_argument(modsec_rec *msr, const char *value, unsigned length)
20 {
21 msc_arg arg = (msc_arg ) NULL;
22
23 /
24 * If we do not have a prefix, we cannot create a variable name
25 * to reference this argument; for now we simply ignore these
26 */
27 if (!msr->json->current_key) {
28 msr_log(msr, 3, "Cannot add scalar value without an associated key");
29 return 1;
30 }

thhis information but i'm not sure if the request is really ignored or not (because on logs the waf blocked the request)

But the application works fine, (no error)

[victorhora]

@mrahmatellah,

So you're not getting a false positive after all? :)

I might be missing something here, but what I could understand now is that you don't have any issue really but you're just trying to figure out (and fix?) why you're getting these messages on your logs?

Maybe it would help if you could provide details on the ModSecurity version, environment, logs etc.

Have you tried what what I proposed you at #1073 (comment)

"#1576 might be related so maybe you could retry with the current proposed solution for v2 at #1577."?

[madmox]

Two things to note:

• A naked JSON array is a valid JSON data structure, so the parser should handle it appropriately, to avoid the warning, but more importantly, to be able to analyse the JSON content!
• The warning is just a log pollution. It does not block the request, but makes it very difficult to analyse the logs and identify real blocking issues. Some softwares using ModSecurity provide no way to reliably filter out those minor issues from serious ones.

We are currently in the process of deploying a WAF in front of our production environment, so we are very concerned with every issue and false positive introduced by the new component before we can confidently switch to prevention mode (i.e. active protection instead of just alerts). This makes it a real pain to configure it properly!

Modifying the schema of the request content would work, but:

• It forces us to add a new workaround in our API, and we also have to remember this rule for all other endpoints. Maintainance is made more difficult.
• We could host a third party API component which we can't modify (non-open source 3rd party library or software), and the workaround would be impossible to implement.

[victorhora]

While revisiting an issue related with JSON today (#1879) I was looking into YAJL API and realized there are a few options available which we are not using today. One of them is yajl_allow_multiple_values.

By enabling this option YAJL can parse requests such as (EDIT: to make it clear, case below was tested in libModSecurity/v3):

POST https://edu-pp.tactileo.fr/profile/internal/permissions HTTP/1.1
cache-control: no-cache
Accept: application/json
Content-Type: application/json
Authorization: Bearer
Host: edu-pp.tactileo.fr
cookie: ARRAffinity=e66763d6606203ca589c508b2ec0185c5b193bb84d15f36169a81d20ed13dc09
accept-encoding: gzip, deflate
content-length: 13

[ "teacher" ]

Debug logs:

[4] Starting phase REQUEST_BODY. (SecRules 2)
[4] Adding request argument (JSON): name "json.array_0", value "teacher"
...
[4] (Rule: 1111) Executing operator "Contains" with param "teacher" against ARGS.
[9] Target value: "teacher" (Variable: ARGS:json.array_0)
[9] Matched vars updated.
[4] Running [independent] (non-disruptive) action: log
[9] Saving transaction to logs
[9] Rule contains a `block' action
[4] Rule returned 1.

Also, empty JSONs such as the ones described at #1767, #1392 (maybe #1822) doesn't trigger invalid body errors anymore.

Some manual tests and testcases are passing fine with this but I'm still wondering if there's any scenario I'm missing where yajl_allow_multiple_values would make the parser more fragile or allow for a bypass. Ideas?

[zimmerle]

Hi @madmox, I guess that you are using the old 2.x family where this was an issue till today. Having #1576 and #1577 closed, this was consequently closed. Notice that the solution adopted on 2.x is not fancy as it is on v3. However, should solve this specific problem.

[francoislarouche]

Hi @victorhora and @zimmerle,

I see that the issue has been closed but it would seem I still have the same issue (rule id 0) with a json call.

I'm using WAF with Azure which uses the version 3.0. If I understood well it should have fixed that issue.

In my case I POST an array of values without any key, something like that:

["value1", "value2", value3", "value-n"]

Like madmox I believe that it should be handled without problem, or if there is a real risk then a real Rule Id should be assigned.

Right now, it generates a lot of noise and it's hard to create reliable alerts upon that many false positives

[victorhora]

Hi @francoislarouche

As far as I can recall, Azure WAF uses ModSecurity 2.9.x branch (not libModSecurity/v3). I think they use the Core Rule Set (CRS) 3.0.

That being said, this should not be an issue with both ModSecurity 2.9.x and 3.x, but you will only see the fix on your Azure WAF if Microsoft decides to backport our patches (9b6d4b2 and 25e5543.) to their own ModSecurity 2.9.x branch.

If you want to be certain that the patch works for you, try downloading and setting up your own version of ModSecurity from master and then test it :)

[francoislarouche]

Thanks for your prompt feedback.

I see, I didn't know there were different builds and branches, and you right it's the CRS 3.0. I was assuming it was the same thing.

It seems they didn't port in it unfortunately and the way Azure WAF is built is we have 0 control on how it works, as a matter of fact I wonder how a normal modern website can use the Azure WAF with blocking mode without impeding business. There is no flexibility since it's a managed service and it's kind of a blackbox (no access to the conf file for modification and write exceptions). It's kind of it's on or off, detect or block, and check box to select rules.

As to use the ModSecurity by itself it would require to have an extra machine that we would need to maintain and secure, and of course pay. I think we would go with a 3rd party WAF vendor, it's a shame I think your WAF is great and MS should expend its functionalities further.

Thanks again!

Cheers

[madmox]

After a few weeks of testing, I agree with you: Azure's ModSecurity implementation is just a selling argument for Azure in general ("we have everything you need so join us"). It's really difficult to imagine a production site using it in real life. Too many false positives required disabling half of the CRS rules, thus rendering the WAF useless.

We can still hope that it will become more flexible in the future (I have seen it happen with other Azure products), so it's kind of an open door for them if MS decides the WAF market is a good opportunity at some point. It's also one of the cheapest SaaS ModSecurity implementations out there, so it deserves follow up.

But right now I suggest you go with other professional solutions like Barracuda (PaYG or BYOL), or a custom VM / VM farm (if you have the resources and skills to do it this way), and only create/upvote feature requests on the dedicated Azure page for Azure app gateway's WAF.

[francoislarouche]

Exactly. You end up disabling half the rules so it's useless at the end.

I suspect they will improve it but they should have been smarter on what to deploy first, people like us lost faith and will end up getting something else. Are we ever going back? Perhaps. It's a real shame being managed and fully integrated in the pipeline of a Web Azure and Logs it's sad. I came from AWS and if I recall well they got a WAF version that was way more flexible right off the bat. Shame.

Custom VM holds many challenges by itself and we wish to avoid that at all cost, not only for security perspective but also for time spent managing it. I saw Barracuda has a solution but I think you need a VM as well. And MS feature requests is a great idea. In any cases thank you for your comment!

Cheers