How Debian fixed CVE-2016-1247, NGINX log root escalation

Introduction

This article explores CVE-2016-1247 exploit, how it was fixed by Debian and what leasons we can extract from it to even go further to protect/secure your systems.

This article also applies to CVE-2016-6664-5617, MySQL root escalation, though it is not a Debian especific issue, it is the same concept failure at the mysql wrapper that allowed the PoC author (Dawid Golunski) to create a working exploit.

How Dawid Golunski’s PoC works (CVE-2016-1247 background)

Recently Dawid Golunski (https://legalhackers.com/advisories/Nginx-Exploit-Deb-Root-PrivEsc-CVE-2016-1247.html) reported a working PoC that successfully escales from www-data (default nginx user) to full root account power. Fiu!

Taking a closer look into the bug, reducing the steps taken by Dawid to break the system and skipping some details for brevity, these are the following:

1. First, it starts with the assumption (not very hard to achieve) that you already own/control www-data account by controlling a .php file, similar cgi, or just the wordpress administration account to that web, or a similar mechanism, so you can upload files to run arbitrary commands.
2. From there, Dawid discovered that log rotation made by logrotate setups the following permissions every day it rotates:

        /var/log/nginx/*.log
        create 0640 www-data adm

   That is, logrotate will rotate and “create” (that’s the point) an empty file with www-data:adm permissions. It looks harmless. This is the devil in the detail.

3. From there, the rest is history: Dawid’s PoC removes /var/log/nginx/error.log and creates a link to /etc/ld.so.preload:

       rm /var/log/nginx/error.log
       ln -s /etc/ld.so.preload /var/log/nginx/error.log

   NOTE: for those wondering, “what the heck is that file for?”

   Basically that file allows to configure “libraries” to “preload” before any other library before launching the binary.

   In essence, it is a mechanism that allows “intercepting” symbols/functions to replace its code by yours. It can be used as a mechanism “for external fix” without binary modification, but, as many mechanisms, it can be used for evil.

   For more information, see: http://man7.org/linux/man-pages/man8/ld.so.8.html

4. Once Dawid’s PoC do that link, it only has to wait for the next day for log roration to happen and let the system itself “open the door for you” by “setting www-data:adm” to /etc/ld.so.preload.
   NOTE: here is when you can play “Carmina Buruana” in the background
5. From there, Dawid’s PoC creates a small library that is attached to any binary loaded by your system, including code to detect “when it runs as root”….and when it does, “bang!” it creates a shell copy with setuid for you to escale without password.

But, hey, you started saying “how Debian fixed” this problem?

Right, we wanted to go through how PoC works to better understand fix introduced and what we can learn from it.

Going back to the solution, if we take a look at Debian’s changelog for this, what they have done is to “secure” permissions for that log to be owned by “root:adm” rather than “www-data:adm”:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842295

    +  * debian/nginx-common.postinst:
    +    + CVE-2016-1247: Secure log file handling (owner & permissions)
    +      against privilege escalation attacks. /var/log/nginx is now owned
    +      by root:adm. Thanks ro Dawid Golunski for the report.
    +      Changing /var/log/nginx permissions effectively reopens #701112,
    +      since log files can be world-readable. This is a trade-off until
    +      a better log opening solution is implemented upstream (trac:376).
    +      (Closes: #842295)

Certainly, having “/var/log/nginx” be owned by root:adm makes impossible for www-data to remove /var/log/nginx/error.log and then trick “logrotate” to create a /etc/ld.so.preload owned by www-data (which is crux).

However, are there more leassons we can learn to better secure the system against this kind of log rotate escalation? Keep on reading.

How services should handle logs (advices for developers and system administrators)

One of the problems that this nginx service setup has is that it does not separate log handling from log production. By making nginx service to handle log directly, instead of using syslog, it creates a permission problem that cannot be easily solved/secured.

If you separate log handling (so syslog handles all logs produced by nginx), you can secure all logs (owned and accessible by root:adm), rotate them, etc, without worrying about the user/users (hopefully low permission users) can do. They can only attack files owned by that user (hopefully only web files) which are not rotated.

See https://nginx.org/en/docs/syslog.html (to start using syslog for your nginx setup)

As an example, the following is wrong:

   service-user:adm   640        # bad/weak configuration

…and a good definition is:

  root:adm  640   # This is the good/recommended configuration because it prevents user
                  # deleting this log and creating a link to sensitive files.

But wait, what do I do with clamav, roundcube, mysql, dovecot (to name some)?

Some services comes with a default “package setup” that do not allow changing this log user to “root” because these services need to be able to open and write to these logs.

In any case, all these services CAN be configured to use syslog and you should consider configuring your system to do so.

In fact, Dawid Golunski also discovered “exactly” the same type of failure in MySQL packages, where a log escalation to root is possible (https://legalhackers.com/advisories/MySQL-Maria-Percona-RootPrivEsc-CVE-2016-6664-5617-Exploit.html ).

First conclusión: always use log service separation (if security is important for you)

That’s why it is very important to concentrate log reporting/handling to syslog so you can separate service from log handling (which solves all these problems).

Again, if you make your system service to run with a low level permission user (i.e.: mysql, www-data, clamav,..), and log reporting is handled and written by a separate daemon service (rsyslog, for example), you can safely create “lograte” configurations that are “always” safe and do not depend on “wrapper script failures” or “packiging problems” that might end up making very controversial decisions (more about this later).

Second conclusion: why wait for the problem to happen

Because this is life and you cannot control how packaging is done, how supervision wrappers are written, etc, you can block these kind of attacks by doing:

   touch /etc/ld.so.preload
   chown root:root /etc/ld.so.preload # you might be already hacked
   chmod 644 /etc/ld.so.preload       # i hope not, 
   chattr +i /etc/ld.so.preload

This way you make sure file exists and cannot be updated (even for the root! unless the attacker already owns root to remove the +i flag).

With this setting you completely disable Dawid’s PoC and make it more dificult to allow log rotate escalation techniques.

Third conclusion: logrotate must be constantly checked

The “logrotate” service is part of the scheme to attack the system by creating “weak” configurations that “rotates” using low privilege users like following (Debian example):

  /etc/logrotate.d/mysql-server:	create 640 mysql adm

As shown in https://legalhackers.com/advisories/MySQL-Maria-Percona-RootPrivEsc-CVE-2016-6664-5617-Exploit.html, this configuration can be exploited (log escalation from mysql user). The best thing to do is to make MySQL service to use syslog to handle all log reporting and update that file to make log owner system to be:

   /etc/logrotate.d/mysql-server:	create 640 root adm

Knowing this, you can use the following command to review those logrotate configurations that are possible breaches in your system:

   find /etc/logrotate.d /etc/logrotate.conf -type f -exec grep -H create {} \; | grep -v "create 640 root adm"

Fourth conclusion: you cannot escape from this

You might be thinking “well, this is something internal..” or “I can handle it”. Maybe. In any case, let’s take a closer look into what “really” did Debian to fix this issue by looking at the changelog:

  nginx-common (1.6.2-5+deb8u3) jessie-security; urgency=high

  In order to secure nginx against privilege escalation attacks, we are
  changing the way log file owners & permissions are handled so that www-data
  is not allowed to symlink a logfile. /var/log/nginx is now owned by root:adm
  and its permissions are changed to 0755. The package checks for such symlinks
  on existing installations and informs the admin using debconf.

  That unfortunately may come at a cost in terms of privacy. /var/log/nginx is
  now world-readable, and nginx hardcodes permissions of non-existing logs to
  0644. On systems running logrotate log files are private after the first
  logrotate run, since the new log files are created with 0640 permissions.

   -- Christos Trochalakis   Tue, 04 Oct 2016 15:20:33+0300

That is, Debian has taken the path of limiting link creation but, at the cost of allowing all users accessing that directory. In fact, during package upgrade/installation they attempt to detect possible hacking links by reporting to the user:

  Template: nginx/log-symlinks
  Type: note
  _Description: Possible insecure nginx log files
    The following log files under /var/log/nginx directory are symlinks
    owned by www-data:
   .
   ${logfiles}
   .
   Since nginx 1.4.4-4 /var/log/nginx was owned by www-data. As a result
   www-data could symlink log files to sensitive locations, which in turn
   could lead to privilege escalation attacks. Although /var/log/nginx
   permissions are now fixed it is possible that such insecure links
   already exist. So, please make sure to check the above locations.

As you can see, “Yes”, the security update allows all system users to be able to access these logs and “Yes”, the security update do not fixes hackings in place, you have to review them any way.

This might (and it is) interesting but, for the scope of this article, as you can see, if you don’t fix your system setup to “separate log handling from log production” you will have to live with very bad decisions that otherwise can be easily solved (and are not Debian’s responsability, by the way).

How Core-Admin handles and mitigates CVE-2016-1247 and CVE-2016-6664-5617

With all these details explained, Core-Admin does two things to mitigate this kind of “root log escalation” attacks:

1. Renamed process checker ensures, automatically, that /etc/ld.so.preload file is protected. It also reports any change to that file.
2. It has a “log” checker that ensures logrotate configurations are working and has known (or at least accepted) ownership declarations that are secure, reporting any insecure configuration that might lead to problems.