Audit log: Failed to lock global mutex: Permission denied

[rcbarnett-zz]

MODSEC-306: When running both Mod Security and Mod Ruid 2, my Mod Security audit log fills up with the following error 'Audit log: Failed to lock global mutex: Permission denied'.

[rcbarnett-zz]

Original reporter: littleb

[rcbarnett-zz]

littleb: I made an error and listed 2.5.6 as the mod_security version.The correct version is 2.6.3.

[rcbarnett-zz]

bpinto: Hello Chris,

Could you send the necessary configuration options to reproduce the issue ?
I would like to make some tests and see if mod_security can work together with mod_ruid2.

Thanks

[rcbarnett-zz]

tomdchi: I am having the same problem.
Environment:
CENTOS 6.3
Apache 2.2.23
mod_ruid2
DSO handler
WHM 11.34.0 (build 11)
PHP 5.4.9
CSF firewall

This error happens when using ownCloud file sharing software from owncloud.org and using the sync client to sync files. It also triggers the CSF firewall to block the IP address. I have a similar setup that does not use mod_ruid2 and do not have this problem with the ownCloud software. The other difference on the second server is that it uses suphp as the Apache handler.

[rcbarnett-zz]

tomdchi: modsec rules that are being used.

[rcbarnett-zz]

arondeparon: I am having the exact same issue with a similar configuration.

[rcbarnett-zz]

jazz: Debian 2.6.39
Apache 2.2
PHP 5.3
mod_security v2.7.2

The same problem.

[rcbarnett-zz]

eagle1: Same here with MPM-ITK on Debian, with Apache 2.2, PHP5.3 (dotdeb) anche mod_security2 from backports.

[alex-leonhardt]

hi,

same here using nginx 1.4.2 with modsecurity-apache_2.7.5 - if I use Concurrent logging, it seems to work, however, then the nginx workers are being killed (there was a previous bug)

Alex

[paulie51]

I'm getting this on Nginx 1.5 modsec 2.7.5 and owncloud. I don't believe the specific rule is relevant to the problem, but here you go :

--aebe1c6a-A--
[05/Dec/2013:15:06:12 +0000] AIAcAcAcAcAcUcAcALAcAc7c my.ip.ad.dr 49304 127.0.0.1 80
--aebe1c6a-B--
PROPFIND /remote.php/webdav HTTP/1.1
User-Agent: Mozilla/5.0 (Linux) csyncoC/0.90.4 neon/0.30.0
Connection: TE
TE: trailers
Host: my.doma.in
Proxy-Connection: Keep-Alive
Depth: 1
Content-Length: 206
Content-Type: application/xml
Cookie: 5263b0ecd7359=gseioa8k7tkba6oirkchrcdv81;

--aebe1c6a-F--
HTTP/1.1 207 Multi-Status
Strict-Transport-Security: max-age=0
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
Pragma: no-cache
Vary: Brief,Prefer
DAV: 1, 3, extended-mkcol, 2
Content-Type: application/xml; charset=utf-8; charset=utf-8+
Content-Length: 875
Connection: keep-alive

--aebe1c6a-E--

--aebe1c6a-H--
Message: Warning. Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/etc/mod_security/activated_rules/modsecurity_crs_30_http_policy.conf"] [line "31"] [id "960032"] [rev "2"] [msg "Method is not allowed by policy"] [data "PROPFIND"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.7"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"]
Message: Warning. Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/mod_security/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.7"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"]
Message: Warning. Operator LT matched 5 at TX:inbound_anomaly_score. [file "/etc/mod_security/activated_rules/modsecurity_crs_60_correlation.conf"] [line "33"] [id "981203"] [msg "Inbound Anomaly Score (Total Inbound Score: 3, SQLi=, XSS=): Method is not allowed by policy"]
Message: Audit log: Failed to lock global mutex: Permission denied
Apache-Handler: IIS
Stopwatch: 1386255971000859 1022405 (- - -)
Stopwatch2: 1386255971000859 1022405; combined=33261, p1=4314, p2=26648, p3=14, p4=1680, p5=605, sr=82, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for nginx (STABLE)/2.7.5 (http://www.modsecurity.org/); OWASP_CRS/2.2.7.
Server: ModSecurity Standalone
Engine-Mode: "DETECTION_ONLY"

--aebe1c6a-Z--

Obviously owncloud falls foul of the CRS rules, but I'm intrigued by the mutex error as it may help me to identify why I don't have any persistency (IP, USER, presumably SESSION as well) under Nginx.

Off to disable the offending rule (which will probably take me some hours!).

Paul.

[shawnholt]

same here - running mod_itk

[wellumies]

Still getting this error with serial logging. Is it safe to use serial logging at all?

2014/03/31 12:43:43 [notice] 12335#0: ModSecurity for nginx (STABLE)/2.7.7 (http://www.modsecurity.org/) configured.
2014/03/31 12:43:43 [notice] 12335#0: ModSecurity: APR compiled version="1.4.6"; loaded version="1.4.6"
2014/03/31 12:43:43 [notice] 12335#0: ModSecurity: PCRE compiled version="8.30 "; loaded version="8.30 2012-02-04"
2014/03/31 12:43:43 [notice] 12335#0: ModSecurity: LIBXML compiled version="2.8.0"
2014/03/31 12:44:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to lock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AcGPArAlAcAcAcAEtOAcAcAc"]
2014/03/31 12:44:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to unlock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AcGPArAlAcAcAcAEtOAcAcAc"]
2014/03/31 12:45:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to lock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AcAcAcUcAcASAcAcAcAcA@Ac"]
2014/03/31 12:45:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to unlock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AcAcAcUcAcASAcAcAcAcA@Ac"]
2014/03/31 12:46:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to lock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AIcEAcA0Kc3cAcAcAcA6Acdc"]
2014/03/31 12:46:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to unlock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AIcEAcA0Kc3cAcAcAcA6Acdc"]
2014/03/31 12:47:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to lock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AYAcAcAcAcAcAcAcAcAcAiAm"]
2014/03/31 12:47:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to unlock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AYAcAcAcAcAcAcAcAcAcAiAm"]
2014/03/31 12:48:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to lock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AcAcAcAcAck9rcAciVAcAcAQ"]
2014/03/31 12:48:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to unlock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AcAcAcAcAck9rcAciVAcAcAQ"]
2014/03/31 12:49:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to lock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AZAyACAcAuAcAcAcAlrwAcAf"]
2014/03/31 12:49:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to unlock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AZAyACAcAuAcAcAcAlrwAcAf"]
2014/03/31 12:50:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to lock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AcAcAcNcDcQcAcAcAmfcAcAc"]
2014/03/31 12:50:04 [error] 12338#0: [client xxx.xxx.xxx.xxx] ModSecurity: Audit log: Failed to unlock global mutex: Permission denied [hostname ""] [uri "/nginx_stub_status"] [unique_id "AcAcAcNcDcQcAcAcAmfcAcAc"]

[ataliba]

Same here using the mpm itk on CentOS 6 .

mod_security-2.7.3-3.el6.x86_64
httpd-itk-2.2.22-6.el6.x86_64
CentOS release 6.5 (Final)

[NgoHuy]

I viewed code and saw when I ran nginx under www-data or anythin else except root, I got this message, because nginx runs as other users who can not stop master process for mutex "writing to file" even when I changed permission to 777 for /var/log/modsec_audit.log

[Gazoo]

Sorry to bump this issue but its been almost a year. Does anyone know of a working fix for this ?

[couchfault]

I have the same exact issue using nginx 1.7.8

[ton31337]

All,

it's because your mutex (in this case semaphore) is created with another user than nginx is running. You can verify with:

ipcs -s

Most simplest patch this would be:

--- standalone/server.c 2014-10-21 08:27:16.170204999 -0400
+++ standalone/server.c 2014-10-21 08:25:40.554204999 -0400
@@ -940,6 +940,9 @@ AP_DECLARE(void) unixd_pre_config(apr_po
 {
     apr_finfo_t wrapper;

+#define DEFAULT_USER "your_nginx_user"
+#define DEFAULT_GROUP "your_nginx_group"

Of couse this can be compiled with -DDEFAULT_USER and so on, but it would require to have libapr, libapache2-dev, etc.

[ju5t]

I've written a patch that solves it. It basically adds the username to the path where it creates the log files. It works and we use it in production, but to be merged upstream it will probably need a configurable setting. There is a pull request waiting to be merged for mod_ruid2 as that will not work out of the box either.

https://gist.github.com/ju5t/0d61ee80b23f0987d65e

[wellumies]

Hmm I thought you have to have libapr or apache-dev to get this working with nginx. With those libs I can compile with DEFUSER and get no mutex lock errors?

Sent from my iPhone

On 14.12.2014, at 16.16, Donatas [email protected] wrote:

All,

it's because your mutex (in this case semaphore) is created with another user than nginx is running. You can verify with:

ipcs -s
Most simplest patch this would be:

--- standalone/server.c 2014-10-21 08:27:16.170204999 -0400
+++ standalone/server.c 2014-10-21 08:25:40.554204999 -0400
@@ -940,6 +940,9 @@ AP_DECLARE(void) unixd_pre_config(apr_po
{
apr_finfo_t wrapper;

+#define DEFAULT_USER "your_nginx_user"
+#define DEFAULT_GROUP "your_nginx_group"
Of couse this can be compiled with -DDEFAULT_USER and so on, but it would require to have libapr, libapache2-dev, etc.

—
Reply to this email directly or view it on GitHub.

[driehuis]

The patch by ju5t addresses the file permission issues, when the log and persistance directories are set up with mode 1777 (as happens to be the case on Unix for the default location of /tmp).

Unfortunately, getting around the lock issues under mpm-itk requires more jiggery pokery, because evidently apr_global_mutex_create and ap_unixd_set_global_mutex_perms get called when Apache has changed uid to www-data, but references the mutex when running as the site's AssignUserID, thus causing the lock failures that starting this thread.

[64kramsystem]

Affects me as well, on an Nginx 1.6.2 on Ubuntu setup.

[microhuang]

I have the same exact issue using nginx 1.6.3.

nginx 1.6.3
modsecurity 2.9
php-fpm 5.5.23
centos6.5

[IncredibleZeuse]

I think that Modsecurity module doesn't care the user of worker processes.
In master process, modsecurity_init(msc_engine *msce, apr_pool_t *mp) function is called, which creates global mutex but master process is running as a root user.
The global mutex can be run under root.
But worker processes are running as not root user but nobody and so on..
The global mutex can not be run under worker process. Because it has permission of nothing but root.

You can solve this issue if you set user directive to root in nginx.conf. "user root;" but it's dangerous to vulnerability.

[jowrjowr]

You need to change how the audit logging works.

See: https://atomicorp.com/forums/viewtopic.php?f=1&t=7964 as an example solution.

[Gazoo]

@jowrjowr No that isn't a real fix. If you follow all the big hosting panel providers (like Cpanel) they are now all running their own patched versions of mod_security. That's the only way to work around this issue. As this issue has now been opened for years I don't think that the mod_security team really cares about this issue.

[wellumies]

True. Had to stop selling modsecurity to our clients. No way to get it working with nginx. They keep tellung me "just switch the inspection off". That is like wearing a condom over your head. Won't help and makes you sweat

Sent from my iPhone

On 27.4.2015, at 17.28, Gazoo [email protected] wrote:

@jowrjowr No that isn't a real fix. If you follow all the big hosting panel providers (like Cpanel) they are now all running their own patched versions of mod_security. That's the only way to work around this issue. As this issue has now been opened for years I don't think that the mod_security team really cares about this issue.

—
Reply to this email directly or view it on GitHub.

[shawnholt]

@jowrjowrWe have disabled modsec also - it does not seem to be compatable with modruid2 and anel. Both Cpanel and Spiderlabs clearly want this to work (especially a Cpanel) but this issue has not received serious attention. Perhaps its an edge case. Cpanel has an internal ticket but I think they need some expertise from Spiderlabs to actually resolve the issue. Hope SpiderLabs and Cpanel can get on the same page as this is hurting both their reputations.

[jowrjowr]

Hum? I'm not saying switch the inspection off, I'm just saying change how the logging works. That's working for me on one webserver. Though I did hit a 100% CPU bug, of course.

I really want this to work. But its' being made as difficult as possible.

[Gazoo]

@jowrjowr The problem with this approach is that changing it to serial decreases the functionality / performance of mod_security considerably. Also all audit consoles require logging to be set to concurrent. (mlogc requires concurrent logging).

[jowrjowr]

Alright, I can't argue with that.

So, uh, does the Apache version have these problems?

On Mon, Apr 27, 2015 at 12:51 PM, Gazoo [email protected] wrote:

@jowrjowr https://github.com/jowrjowr The problem with this approach is
that changing it to serial decreases the functionality / performance of
mod_security considerably. Also all audit consoles require logging to be
set to concurrent. (mlogc requires concurrent logging).

—
Reply to this email directly or view it on GitHub
#454 (comment)
.

[Gazoo]

@jowrjowr Yes the issue is present in both Apache and Nginx.

[jowrjowr]

What if the log messes were simply dumped to syslog() and we bypass the
issue entirely?

On Mon, Apr 27, 2015 at 12:53 PM, eric gisse [email protected] wrote:

Alright, I can't argue with that.

So, uh, does the Apache version have these problems?

On Mon, Apr 27, 2015 at 12:51 PM, Gazoo [email protected] wrote:

@jowrjowr https://github.com/jowrjowr The problem with this approach
is that changing it to serial decreases the functionality / performance of
mod_security considerably. Also all audit consoles require logging to be
set to concurrent. (mlogc requires concurrent logging).

—
Reply to this email directly or view it on GitHub
#454 (comment)
.

[Gazoo]

The power of concurrent logging is that it stores the full HTTP transaction in dedicated files allowing you to do proper auditing / debugging. That's not something you can just dump to syslog.

[shawnholt]

Note: Please see the following related comments-
ref: https://forums.cpanel.net/threads/modsecurity-rule-processing-failed.453321
#712

[jowrjowr]

Sure it can, if you throw enough log messages at it.

modsec assigns each transaction a unique ID. Throw the HTTP data into
syslog with that transaction, put in audit data with that transaction, and
the rest of the stuff up to and including the kitchen sink as well. Subject
to modsec logging levels.

This solves the log concurrency issue, by putting it elsewhere. Which I
guess people hate because for some reason flat files are preferred?

Is there an internal nginx log output functionality modsec could use
instead? Nginx doesn't have a concurrency issue when writing to its' own
logs...

On Mon, Apr 27, 2015 at 1:00 PM, Gazoo [email protected] wrote:

The power of concurrent logging is that it stores the full HTTP
transaction in dedicated files allowing you to do proper auditing /
debugging. That's not something you can just dump to syslog.

—
Reply to this email directly or view it on GitHub
#454 (comment)
.

[driehuis]

Modsecurity works on Apache, even with mpm-itk (and I would presume, mod_ruid2), provided the vhost it runs in does not use AssignUserID (nor RUidGid, for ruid2). I'm using this to run modsecurity in a vhost that just proxies requests to the vhosts that run the actual websites. Basically, modsecurity runs as a completely separate frontend WAF in this scenario.

This works as long as all sites behind the WAF can live with a common ruleset. I'm happy with this setup, YMMV.

It would be nice if modsecurity would fully support mpm-itk, but it seems to be a hard nut to crack. Maybe something for Google's summer of code.

[ju5t]

Writing the log entries to syslog won't solve the issue. There's another file that is used to track IP addresses and that has the same problem. I'm using my own patched version to get around that for now which is mentioned in #712. As it's a pretty rough and probably dirty patch (I'm not a developer let alone one that can write c very well) I haven't sent in a pull request for it.

Mod_ruid2 has another issue where it doesn't initiate the ap_hook_post_read_request call at the right moment, so even when Mod Security is patched for this issue it will fail because some files are still created with the wrong owner. I've sent a pull request in that particular case as it was an easy fix, but it hasn't been accepted for months now.

[DediData]

To solve the problem Audit log: Failed to lock global mutex: Permission denied
Using mod_ruid2 and mod_security together:

find the configuration file which loads finally in mod security (example: /usr/local/apache/conf/modsec2.cpanel.conf)

Add these line to the end of file:

SecAuditLogDirMode 1733
SecAuditLogFileMode 0550
SecAuditLogType Concurrent
SecAuditLogStorageDir /usr/local/apache/logs/modsec_audit

Check the directory of /usr/local/apache/logs/modsec_audit for proper permissions of : 1733

This solved my problem
Regards
Farhad Sakhaei

[Cacasapo]

parsmizban's solution worked for me, but I added the text to modsec2.user.conf, under the reasoning that modsec2.cpanel.conf will get overwritten by cpanel.

Thanks!

[ghost]

Sorry to stir up an old bug.

Does this error also block the request? or is it just an issue of filling up the logs with errors?

Thank you.

[skurtn]

I am an EasyApache developer for cPanel. If you're using a cPanel system with EasyApache, you do not need to update modsec2.cpanel.conf, nor do you need to update modsec2.user.conf with any directives to enable Concurrent logging.

EasyApache 3 automatically configure Apache within modsec2.conf with the proper directives.
EasyApache 4 will automatically configure Apache within modsec2.conf once we release EA-4430.

Here's an excerpt of what you should see.

EasyApache 3 (/usr/local/apache/conf/modsec2.conf)

<IfModule mod_ruid2.c>
    SecAuditLogStorageDir /usr/local/apache/logs/modsec_audit
    SecAuditLogType Concurrent
</IfModule>
<IfModule itk.c>
    SecAuditLogStorageDir /usr/local/apache/logs/modsec_audit
    SecAuditLogType Concurrent
</IfModule>

EasyApache 4 (/etc/apache2/conf.d/modsec2.conf)

    # Switch to concurrent logging when Apache is running under a multi-uid
    # environment.  This ensures that each user can successfully log to
    # their own log file.
    <IfModule ruid2_module>
        SecAuditLogStorageDir logs/modsec_audit
        SecAuditLogType Concurrent
    </IfModule>
    <IfModule mpm_itk_module>
        SecAuditLogStorageDir logs/modsec_audit
        SecAuditLogType Concurrent
    </IfModule>

IMPORTANT NOTES

• We've been applying this patch since 2014 so that each user logs to their own area.
• We strictly set permissions of the logging directory to 1733 in order to reduce information disclosure.
• We're aware of Issue 712 relating to rules which use a DBM file to store information (e.g. a rule in the OWASP distributed ruleset).

[victorhora]

The global mutex is optional since 112ba45 (included in 2.9.2 release)

Nginx support for ModSecurity 2.9.x branch is unsupported. This shouldn't be a concern with libModSecurity. Please upgrade.

PR #1912 is up for evaluation address the compatibility issue with apache2-itk and mod_ruid2 modules.

[azurit]

Hi guys, i'm still having problems with this in version 2.9.3 (apache with ITK):
Message: Audit log: Failed to lock global mutex: Permission denied

I tried to recompile modsecurity with --disable-collection-global-lock but errors are still logged.

[commeta]

There is also an example of solving the global mutex problem with the Apache2 MPM ITK module. This project (Apache mod security log parser) demonstrates an approach.

[airween]

@commeta thanks.

Anyone who still faces with this issue, could you take a look at this PR?
#3188

[marcstern]

On top of PR #3188, I wonder if it won't be better, at least for the Apache version, to use Apache mutex mechnaism (ap_proc_mutex_create or ap_global_mutex_create?) and use another type of mutex (in the config), like "pthread". Just an idea, I don't know the internals of these but it may worth to dig into this.

[airween]

@marcstern - just my 2 cents: even though that PR is merged but unreleased, so nobody has used yet that, this is why I bring their attention.

[azurit]

Why is this closed?

[airween]

Why is this closed?

I'm afraid nobody knows the reason... It had been closed on 23rd of Sep, 2018

[marcstern]

Does this still occurs with 2.9.8?
We rewrote several things related to locks.

[commeta]

I encountered this issue in the production environment:
Producer: ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache/2.4.6 (CentOS 7) mpm-itk/2.4.7-04; 1 Core, 2.4GHz, 1GB RAM.

In the solution, I used global settings, and some difficulties could have been avoided by using subdirectories in the virtual host for: SecTmpDir, SecDataDir, SecAuditLogStorageDir.

In modsec_audit.log, I regularly encountered messages:
ModSecurity: Audit log: Failed to lock global mutex: Permission denied
ModSecurity: Audit log: Failed to unlock global mutex: Permission denied
Message: collections_remove_stale: Failed to lock proc mutex: Permission denied

My problem is solved, and I revived the issue because I found it in search and used it as a reference in solving my task.

[marcstern]

@commeta: My problem is solved
What do you mean exactly?
Is it solved in new versions, by using file mutex instead of the default, by using different directories for each virtual host for SecTmpDir/SecDataDir/SecAuditLogStorageDir, other?

[commeta]

Apache/2.4.6 mpm-itk/2.4.7-04
ModSecurity for Apache/2.9.2
I was only testing this environment with the mpm-itk module.