Archive for Debian

How Debian fixed CVE-2016-1247, NGINX log root escalation


This article explores CVE-2016-1247 exploit, how it was fixed by Debian and what leasons we can extract from it to even go further to protect/secure your systems.

This article also applies to CVE-2016-6664-5617, MySQL root escalation, though it is not a Debian especific issue, it is the same concept failure at the mysql wrapper that allowed the PoC author (Dawid Golunski) to create a working exploit.

How Dawid Golunski’s PoC works (CVE-2016-1247 background)

Recently Dawid Golunski ( reported a working PoC that successfully escales from www-data (default nginx user) to full root account power. Fiu!

Taking a closer look into the bug, reducing the steps taken by Dawid to break the system and skipping some details for brevity, these are the following:

  1. First, it starts with the assumption (not very hard to achieve) that you already own/control www-data account by controlling a .php file, similar cgi, or just the wordpress administration account to that web, or a similar mechanism, so you can upload files to run arbitrary commands.
  2. From there, Dawid discovered that log rotation made by logrotate setups the following permissions every day it rotates:
         create 0640 www-data adm

    That is, logrotate will rotate and “create” (that’s the point) an empty file with www-data:adm permissions. It looks harmless. This is the devil in the detail.

  3. From there, the rest is history: Dawid’s PoC removes /var/log/nginx/error.log and creates a link to /etc/
        rm /var/log/nginx/error.log
        ln -s /etc/ /var/log/nginx/error.log

    NOTE: for those wondering, “what the heck is that file for?”

    Basically that file allows to configure “libraries” to “preload” before any other library before launching the binary.

    In essence, it is a mechanism that allows “intercepting” symbols/functions to replace its code by yours. It can be used as a mechanism “for external fix” without binary modification, but, as many mechanisms, it can be used for evil.

    For more information, see:

  4. Once Dawid’s PoC do that link, it only has to wait for the next day for log roration to happen and let the system itself “open the door for you” by “setting www-data:adm” to /etc/
    NOTE: here is when you can play “Carmina Buruana” in the background
  5. From there, Dawid’s PoC creates a small library that is attached to any binary loaded by your system, including code to detect “when it runs as root”….and when it does, “bang!” it creates a shell copy with setuid for you to escale without password.

But, hey, you started saying “how Debian fixed” this problem?

Right, we wanted to go through how PoC works to better understand fix introduced and what we can learn from it.

Going back to the solution, if we take a look at Debian’s changelog for this, what they have done is to “secure” permissions for that log to be owned by “root:adm” rather than “www-data:adm”:

    +  * debian/nginx-common.postinst:
    +    + CVE-2016-1247: Secure log file handling (owner & permissions)
    +      against privilege escalation attacks. /var/log/nginx is now owned
    +      by root:adm. Thanks ro Dawid Golunski for the report.
    +      Changing /var/log/nginx permissions effectively reopens #701112,
    +      since log files can be world-readable. This is a trade-off until
    +      a better log opening solution is implemented upstream (trac:376).
    +      (Closes: #842295)

Certainly, having “/var/log/nginx” be owned by root:adm makes impossible for www-data to remove /var/log/nginx/error.log and then trick “logrotate” to create a /etc/ owned by www-data (which is crux).

However, are there more leassons we can learn to better secure the system against this kind of log rotate escalation? Keep on reading.

How services should handle logs (advices for developers and system administrators)

One of the problems that this nginx service setup has is that it does not separate log handling from log production. By making nginx service to handle log directly, instead of using syslog, it creates a permission problem that cannot be easily solved/secured.

If you separate log handling (so syslog handles all logs produced by nginx), you can secure all logs (owned and accessible by root:adm), rotate them, etc, without worrying about the user/users (hopefully low permission users) can do. They can only attack files owned by that user (hopefully only web files) which are not rotated.

See (to start using syslog for your nginx setup)

Service should never own log
So a general conclusion we can derive from this is: never let a log be owned by the service producing it, otherwise, it can be used to escale using exactly the same mechanism as Dawid’s PoC did.

As an example, the following is wrong:

   service-user:adm   640        # bad/weak configuration

…and a good definition is:

  root:adm  640   # This is the good/recommended configuration because it prevents user
                  # deleting this log and creating a link to sensitive files.

But wait, what do I do with clamav, roundcube, mysql, dovecot (to name some)?

Some services comes with a default “package setup” that do not allow changing this log user to “root” because these services need to be able to open and write to these logs.

In any case, all these services CAN be configured to use syslog and you should consider configuring your system to do so.

In fact, Dawid Golunski also discovered “exactly” the same type of failure in MySQL packages, where a log escalation to root is possible ( ).

Default log configuration might not be good
This article tries to raise awareness about default log configuration. Linux distributions provides you a default configuration but you have to carefully review how log is handled, rotated and owned.

First conclusión: always use log service separation (if security is important for you)

That’s why it is very important to concentrate log reporting/handling to syslog so you can separate service from log handling (which solves all these problems).

Again, if you make your system service to run with a low level permission user (i.e.: mysql, www-data, clamav,..), and log reporting is handled and written by a separate daemon service (rsyslog, for example), you can safely create “lograte” configurations that are “always” safe and do not depend on “wrapper script failures” or “packiging problems” that might end up making very controversial decisions (more about this later).

Second conclusion: why wait for the problem to happen

Because this is life and you cannot control how packaging is done, how supervision wrappers are written, etc, you can block these kind of attacks by doing:

   touch /etc/
   chown root:root /etc/ # you might be already hacked
   chmod 644 /etc/       # i hope not, 
   chattr +i /etc/

This way you make sure file exists and cannot be updated (even for the root! unless the attacker already owns root to remove the +i flag).

With this setting you completely disable Dawid’s PoC and make it more dificult to allow log rotate escalation techniques.

Third conclusion: logrotate must be constantly checked

The “logrotate” service is part of the scheme to attack the system by creating “weak” configurations that “rotates” using low privilege users like following (Debian example):

  /etc/logrotate.d/mysql-server:	create 640 mysql adm

As shown in, this configuration can be exploited (log escalation from mysql user). The best thing to do is to make MySQL service to use syslog to handle all log reporting and update that file to make log owner system to be:

   /etc/logrotate.d/mysql-server:	create 640 root adm

Knowing this, you can use the following command to review those logrotate configurations that are possible breaches in your system:

   find /etc/logrotate.d /etc/logrotate.conf -type f -exec grep -H create {} \; | grep -v "create 640 root adm"

Fourth conclusion: you cannot escape from this

You might be thinking “well, this is something internal..” or “I can handle it”. Maybe. In any case, let’s take a closer look into what “really” did Debian to fix this issue by looking at the changelog:

  nginx-common (1.6.2-5+deb8u3) jessie-security; urgency=high

  In order to secure nginx against privilege escalation attacks, we are
  changing the way log file owners & permissions are handled so that www-data
  is not allowed to symlink a logfile. /var/log/nginx is now owned by root:adm
  and its permissions are changed to 0755. The package checks for such symlinks
  on existing installations and informs the admin using debconf.

  That unfortunately may come at a cost in terms of privacy. /var/log/nginx is
  now world-readable, and nginx hardcodes permissions of non-existing logs to
  0644. On systems running logrotate log files are private after the first
  logrotate run, since the new log files are created with 0640 permissions.

   -- Christos Trochalakis   Tue, 04 Oct 2016 15:20:33+0300

That is, Debian has taken the path of limiting link creation but, at the cost of allowing all users accessing that directory. In fact, during package upgrade/installation they attempt to detect possible hacking links by reporting to the user:

  Template: nginx/log-symlinks
  Type: note
  _Description: Possible insecure nginx log files
    The following log files under /var/log/nginx directory are symlinks
    owned by www-data:
   Since nginx 1.4.4-4 /var/log/nginx was owned by www-data. As a result
   www-data could symlink log files to sensitive locations, which in turn
   could lead to privilege escalation attacks. Although /var/log/nginx
   permissions are now fixed it is possible that such insecure links
   already exist. So, please make sure to check the above locations.

As you can see, “Yes”, the security update allows all system users to be able to access these logs and “Yes”, the security update do not fixes hackings in place, you have to review them any way.

This might (and it is) interesting but, for the scope of this article, as you can see, if you don’t fix your system setup to “separate log handling from log production” you will have to live with very bad decisions that otherwise can be easily solved (and are not Debian’s responsability, by the way).

How Core-Admin handles and mitigates CVE-2016-1247 and CVE-2016-6664-5617

With all these details explained, Core-Admin does two things to mitigate this kind of “root log escalation” attacks:

  1. Renamed process checker ensures, automatically, that /etc/ file is protected. It also reports any change to that file.
  2. It has a “log” checker that ensures logrotate configurations are working and has known (or at least accepted) ownership declarations that are secure, reporting any insecure configuration that might lead to problems.

Posted in: Core-Admin, Debian, LogRotate, Security

Leave a Comment (0) →

KB: 07072014-001: Disabling ptrace() syscall

Keyword index


The following article explains how to disable system call ptrace() in various platforms (see list of supported platforms). By disabling this system call you can remove a large source of security problems and a linux kernel feature that is used by many attacks to implement hard to detect modifications like in-flight memory process modification.

The article proposes disabling the ptrace syscall by installing a kernel module that disables it.

Supported platforms

  • Debian Squeeze amd64
  • Debian Squeeze i686
  • Debian Wheezy amd64
  • Ubuntu Precise LTS 12.04 amd64
  • Linux Mint 13 Maya amd64

Installing the module

To have the module installed, you have to update your /etc/apt/sources.list file to include the right apt sources. See in the following link the right one for your distribution:

After that, you only have to update references and install it by running:

apt-get update
apt-get install noptrace2

After that, the module will be compiled using your current server/system settings and will be loaded if no problem is found.

How do I check if the module is actually blocking ptrace() calls?

Run the following command. You should get a “No child processes”:

strace -p 1
Process 1 attached - interrupt to quit
detach: ptrace(PTRACE_DETACH, ...): No child processes
Process 1 detached

How do I enable/disable it temporally?

You can use the following command to stop/unload the module causing ptrace() blocking to be removed:

service noptrace2 stop

At the same time, you can use the following command to reenable the module that blocks ptrace():

service noptrace2 start

Do this generates any operation log I can inspect?

Sure, take a look at your /var/log/syslog. You should get logs like this:

Jul 7 11:14:40 vulcan kernel: [4721108.617232] [noptrace2] ptrace syscall disabled
Jul 7 11:14:54 vulcan kernel: [4721122.990270] [noptrace2] ptrace() invoked against process 1 by process 20675
Jul 7 11:14:54 vulcan kernel: [4721122.990304] [noptrace2] ptrace() invoked against process 1 by process 20675
Jul 7 11:15:02 vulcan kernel: [4721130.689160] [noptrace2] ptrace() invoked against process 29912 by process 20746
Jul 7 11:15:02 vulcan kernel: [4721130.689188] [noptrace2] ptrace() invoked against process 29912 by process 20746
Jul 7 11:15:22 vulcan kernel: [4721150.219577] [noptrace2] ptrace syscall restored
Jul 7 11:15:44 vulcan kernel: [4721172.921028] [noptrace2] ptrace syscall disabled
Jul 7 18:11:15 vulcan kernel: [4746103.948870] [noptrace2] ptrace() invoked against process 1 by process 9821
Jul 7 18:11:15 vulcan kernel: [4746103.948897] [noptrace2] ptrace() invoked against process 1 by process 9821

Did you like the article, found it useful or something to comment?

That’s good. Please,  contact us at or follow use at or

Posted in: Administration, Debian, Debian Squeeze, Debian Wheezy, Linux Mint, Security, Ubuntu, Ubuntu Precise LTS

Leave a Comment (0) →

KB: 16052014-001: Fixing /usr/sbin/grub-probe: error: no such disk. for device /dev/md0 error

So, you are running a Debian like system, and you are upgrading your kernel and during the process you the the following error:

update-initramfs: Generating /boot/initrd.img-2.6.32-5-amd64
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 2.6.32-5-amd64 /boot/vmlinuz-2.6.32-5-amd64
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 2.6.32-5-amd64 /boot/vmlinuz-2.6.32-5-amd64
Generating grub.cfg ...
/usr/sbin/grub-probe: error: no such disk.
run-parts: /etc/kernel/postinst.d/zz-update-grub exited with return code 1
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-2.6.32-5-amd64.postinst line 799.

Possibly you are googling it out and you’ve found various solutions like the following links but none of them works:

The curious thing is that your system is running perfectly, there is no error at /proc/mdstat (please do a cat over that file just to be sure) and if you run a simple “ls -la ” over /dev/md0 and the disks components that made that disks you find out that everything is right. No error.

At some point, you find that you have to run the following command to “check” what’s grub-probe idea about your hardisks:

/usr/sbin/grub-probe --device-map=/boot/grub/ --target=fs -v /boot/grub

However, it is reporting at the end:

/usr/sbin/grub-probe: info: opening md0.
/usr/sbin/grub-probe: error: no such disk.

If you have all this elements in common, please, just be sure you have mdadm command available. It is possible that you have removed it by mistake. Because grub-probe uses mdadm –examine /dev/md0, it is confusing an error from that command with a command not found error.

Please, try the following to see if it works:

>> apt-get install mdadm
>> apt-get install -f

Note well for Core-Admin users

If you are running Core-Admin’s mdadm checker, it will ensure you have mdadm available apart from checking your hard disks and the details inside /proc/mdstat.

Please, be sure you have mdadm checker to ensure this error do not reach your system.

Posted in: Administration, Debian, Debian Squeeze, KB

Leave a Comment (0) →

KB: 05112013-001: Webhosting management fails to add a new hosting reporting an error about quotas


When trying to create a hosting inside Webhosting management, it fails with an error similar to:

setquota: Cannot open quotafile /home/aquota.user: Permission denied
setquota: Not all specified mountpoints are using quota.

Affected releases



Restart quota system. To do so you can use one the following methods:

  1. Run the following command if you have easy access to a root shell on the server:
    >> /etc/init.d/quota restart
  2. You can restart the service using “Process and services viewer”, by selecting “quota” service under unmanaged services. After selecting, use the restart option available.

Posted in: Core-Admin, Debian, Debian Lenny, Debian Squeeze, Debian Wheezy, KB

Leave a Comment (0) →

KB: 04112013-001: Mail admin installer because libmail-sender-perl is not found


When launched Mail admin installer, it stops and fails reporting that libmail-sender-perl package isn’t found.

Affected releases

Debian Lenny (5.0), Debian Squeeze (6.0), Debian Wheezy (7.0)


  1. Update your /etc/apt/sources.list to include “non-free” declaration to default source of packages. A working example should like like:
    deb squeeze main non-free
  2. After that, restart the installer again.


Posted in: Debian, Debian Lenny, Debian Squeeze, Debian Wheezy, KB

Leave a Comment (0) →