Securing Linux, Part 2

by Michael H. Warfield [Published on 16 Oct. 2002 / Last Updated on 24 Jan. 2013]

Advanced Linux security

These advanced tips are mostly for larger systems or networks which have more resources, or which are a more valuable target, or which are run by more experienced system administrators who are available to implement and maintain the installation. They should be implemented in conjunction with the security precautions discussed in Part 1 of this series, "Elementary Linux Security."

  1. Filesystem defenses
    Turn your filesystem into a staunch ally by mounting major partitions separately, and by making critical partitions read only. Take advantage of native Linux defenses, like the append-only and immutable attributes for the ext2 filesystem.

    Filesystem partitions
    There is a reason for the filesystem standard, and it is well worth your while to invest time in taking full advantage of it. The filesystem can be divided into major partitions, and each partition can be configured and mounted differently. I strongly recommend separate /, /usr, /usr/local, /var, and /home partitions, at the very least.

    /usr can be mounted read only and can be considered inviolate for purposes of validation. If anything ever changes in /usr, that change should ring an alarm bell -- literally. Of course, if you change something in /usr yourself, you will know that the change is coming.

    The same idea applies to /lib, /boot, and /sbin. If you can make them read-only and alarm any attempts to change files, directories, or permissions, then do it.

    It isn't possible to mount all of your major partitions as read only. For example, /var, by its very nature, cannot be read only, and for that reason nothing should be allowed to execute out of it. Things like configuration files for X servers should be symbolically linked to files which are kept in places which can be made read only -- and not through variable storage dumps.

    Extending ext2
    Use of the append-only and immutable attributes on the ext2 file system can provide enhancements to a secure installation. While not perfect in and of themselves, these attributes can be useful in detecting intrusion attempts when an attacker attempts to install rootkits or other backdoors over existing files. To be sure, such measures can be thwarted once they are detected. But by then, you should already have been notified and made aware of the intrusion.

    If you have critical filesystems mounted read only and the files are marked as immutable, an intruder must remount the filesystems and remove the immutable bits -- all without getting caught or triggering an alarm. This is no small feat, and an intruder who recognizes this is more likely to go off in search of more vulnerable prey than risk being caught.

    The immutable and append-only attributes are just two of the extended attribute flags on the ext2 filesystems. A file which is flagged as immutable cannot be changed, not even by root. A file flagged as append only can be changed, but it can only have material appended to it. Even the root user cannot modify it in any other way.

    These attributes are added to a file by the chattr command, and can be listed with the list attribute or lsattr command. For more information on enhanced file protection through ext2 permissions attributes, see man chattr.

    While partitions and ext2 attributes seem simple enough on the surface, they are actually bits of arcana -- and little effort or progress has been made in making them user-friendly. Even sophisticated users and administrators have been known to get tripped up on them, and so you should not treat them trivially.

    Secure log files
    The immutable and append-only attributes are particularly effective when used in combination with log files and log backups. You should set active log files to append only. When the logs are rotated, the backup log file created by the rotation should be set to immutable, while the new active log file becomes append only. This usually requires some manipulation of your log rotation scripts.

  2. Using backups to validate system security
    By making a snapshot of your system upon completion of a successful installation, you can validate system integrity against this backup to see if files have been tampered with. You can also use the backup to return your system to its secure state if it has been compromised.

    CD-ROM snapshots
    Once you have a secure system, cut a snapshot of it to a CD-ROM. Then, periodically, you can validate system integrity against that CD-ROM. In a case where security is a very high priority, such as a firewall system, you can make the CD-ROM bootable and do the validation as part of the bootup. By booting off the CD-ROM, you can be pretty sure that the media has not been tampered with prior to validation.

    If you have created read-only partitions, these can be reloaded periodically from the CD-ROM image (after all, they are not supposed to change). Even if areas like /boot, /lib, and /sbin cannot be mounted read-only (because of /etc), you can still check those areas against the CD-ROM image, or even reload them at boot time from another safe image.

    Other backup
    While many files in /etc are subject to frequent change, some areas in /etc can still be cut to a CD-ROM and verified. Other files, which are subject to infrequent changes, may be backed up to another system -- they may, for instance, be sent to tape, or be tarred and placed in a read-only directory -- for a second level of validation after critical utilities are validated against the CD-ROM image.

    Since most systems now come with a CD-ROM drive, CD-ROM emergency bootdisks or validation disks are practical for any site which is equipped with a CD-ROM burner to create them. The price of CD-ROM blanks is dropping to below the cost of floppies, making this a cheap and effective validation method which is tough for intruders to defeat.

  3. Internal system security enhancements
    Several things can be done to enhance the operating system's internal functions and protect against buffer overflow attacks, access control violations, or poorly-coded applications. Some of these enhancements carry a heavy performance, maintenance, and/or learning-curve tax. But for many systems, it's a necessity.

    Solar Designer's secure Linux patches
    Solar Designer's secure Linux patches for the 2.0 kernels provide a nonexecutable stack to reduce the threat from buffer overflows, as well as patches to deal with some /tmp race conditions.

    It is true that buffer overflow conditions are extremely difficult to exploit, because the intruder must identify the presence of a potential buffer overrun as well as its location in memory. But it is also true that your system can never be too secure. These advanced internal safeguards would not exist if it were impossible to abuse buffer overflow conditions.

    Despite this, some people -- including Linus Torvalds -- consider the secure Linux patches to be "security through obscurity" and not worth the trouble they can cause. Since they eliminate an entire category of attacks and force the attacker to use other techniques, I disagree with this argument.

    But you should be aware that these patches can cause problems with certain programs and libraries which depend on an executable stack. That's why they are an advanced topic, only for those administrators experienced enough to deal with the potential difficulties that come with this kind of patch.

    The nonexecutable stack patches have been nominated for the secure distribution wish list now under discussion on several security mailing lists, like the secure-distros list mentioned later in this article. You'll find links for this list -- and the other resources -- listed in the Resources section at the end of this article.

    StackGuard
    For the really ambitious types, with big powerful boxes, there is StackGuard. You can recompile and relink critical applications with a version of gcc which has been patched with StackGuard.

    Compiling with StackGuard adds stack checking in order to prevent stack smashing buffer overflows, but it comes with a performance penalty that most of us would not want to pay on a normal box. For high-security systems in special purpose applications, however, StackGuard is a valuable asset for those with the time to invest.

    There are some Linux distributions available that are already StackGuarded. This makes it easier to install a system with StackGuard, but you will still have as much as a 10 to 20 percent performance impact. This solution is also not perfect, but it eliminates an entire subclass of buffer overflow attacks.

    Access control lists
    The 2.3 kernels are looking toward the implementation of access control lists in the filesystems. These will provide finer-grained control of access permission beyond the old three-category (owner, group, and other) system currently in place.

    Capabilities
    Still under development in the 2.2 and 2.3 Linux kernels are the introduction of capabilities, which will eventually affect some of the things about the ext2 file attributes which we discussed in Tip 1. Capabilities provide a more precise control over security than the conventional filesystem permissions model with the ext2 extensions and secure level. With this new feature, applications will be granted capabilities to access certain system resources, such as raw sockets, without needing to run as superuser. The secure level functionality of the 2.0 kernels will be implemented as a function of the capabilities set in the 2.2 kernels and beyond.

    Rule set based access control
    The RSBAC (rule-set based access control) project has a stated goal of meeting the requirement of a B1 security rating for the Linux operating system. RSBAC is based on the generalized framework for access control and extends many of the system calls with security enforcement code. It includes support for a number of different access and authentication methods. This is an area of great promise for extending and hardening the internal and local security of Linux systems.

  4. Honey pots and traps
    Traps are any software designed to trigger an alarm when activated, while the term honey pots refers to deceptively tempting traps which are designed to lure snooping intruders into deliberate alarms.

    Your honey pot says 'Yes, yes!' but your system says 'Danger, Will Robinson!'
    With simple traps and alarms, normal system activity is armed to trigger an alarm in the event of an intrusion. In large or highly critical networks, entire systems may be dedicated to being a honey pot. This honey pot is deliberately vulnerable and easy to break into -- but immediately triggers an alarm in the event of any activity.

    Intruders check in, but they don't check out
    One way to set up a honey pot would be to configure a system with an IMAP server designed to falsly claim to be one of several vulnerable versions. Intruders performing bulk port scans for IMAP servers would then attempt to break into the honey pot system and trigger an alarm.

    Another example of a honey pot trap is based on an old, vulnerable Web cgi-bin script called phf. The original script was designed to look up phone numbers. Unfortunately, it had a security hole which would allow an intruder to use it to retrieve your password file or commit other heinous acts on your system. The fake phf, or honey pot version, acts just like the vulnerable script. But instead of mailing the intruder your password file, it sends out some fake information while at the same time sending an alarm to the system's administrator.

    Reactive honey pot
    Some honey pots can be set up to immediately deny access to the intruder by shutting down firewalls that block access from the intruder's address. Such access can be denied for a short period of time, or may be cut off permanently. The firewall code in the Linux kernel is well suited to this. If not implemented correctly, however, this can be turned against the user by spoofing legitimate addresses and blocking valid sites. This turns a honey pot into a self-destructive denial of service attack.

    The Deception Toolkit (see Resources) provides a variety of tools to set up fake servers, fake vulnerable services, and traps for intruders to fall into when snooping through your system.

  5. Reaction systems and dynamic firewalls
    Reaction systems can be as simple as stateful inspection packet filters or as sophisticated as intrusion detection systems and configurable firewalls and alarms.

    Abacus Port Sentry
    Abacus Port Sentry (aka Abacus Sentry) is an open source package that monitors the network interface and interacts with the firewall code to shut off an attack during an attempted port scan. This is a powerful tool, and it needs to be used with caution.

    In the face of an ongoing port scan, Abacus Sentry can rapidly block an intruder before the intruder can get very far. But if it is not configured properly, it can enable a hostile outsider to mount a denial of service attack against your systems.

    Enhancing a firewall with Abacus Sentry
    While it was not specifically designed with this use in mind, Abacus Port Sentry can be combined with the transparent proxy facilities in Linux to provide a very effective reactive firewall against intruders.

    The firewall is set up normally, with or without stateful packet filtering. The last set of rules, or rule chains, redirect unused ports for common services on all IP addresses to the Port Sentry. This gives the Port Sentry a depth of coverage unavailable otherwise, and scans of a single port across multiple addresses can then be detected and cancelled before the intruder progresses deeper into your network.

    Abacus Port Sentry will detect slow scans, but it will not detect structured attacks. Both proccesses seek to obscure probe attacks. An attacker does this with a slow scan by spreading out the test over a long period of time. In a structured attack, the attacker tries to make the probe attempts more obscure by scanning or probing from multiple source addresses.

    Using this software properly will effectively neutralize massive parallel scans for IMAP services and block all access to such intruders. This use of Abacus Sentry is most effective in combination with the Linux 2.2 kernel's IPChains facility -- which allows a separate redirection chain to feed port scans to Port Sentry -- and a preceding blocking chain which allows Port Sentry to block intruders prior to any other rules.

    The Linux 2.0 kernels can be patched to use IPChains, even though the facility is not packaged with this version. Abacus Port Sentry can also work with the earlier ipfwadm facility which is in the 2.0 kernels, and which was replaced by IPChains in version 2.2.

    Abacus Port Sentry can be configured to react to UDP scans and, on Linux systems, to various half scans, such as half-open and FIN scans, which try to avoid detection by sending only initial or final packets instead of establishing an actual connection.

    Avoiding denial of service attacks
    There is a small risk in these configurations, in that they can be spoofed -- and could thereby be exploited for denial of service attacks. While such attacks are extremely difficult to execute, Port Sentry does not have convenient facilities for reenabling blocked addresses. Simple TCP port scanning detection is sufficient in most cases -- particularly in the firewall/transparent proxy configuration.

    Some intrusion detection systems, such as Internet Security System's Real Secure, can reconfigure firewalls based on intrusion alarms and attack signatures. Many of these can provide highly sophisticated reaction profiles, but they also need to be configured with care to prevent the abuse of such systems, by knowledgeable attackers, to perpetrate denial of service attacks.

    Other types of attacks
    You can avoid being the victim of many other types of attacks by following the basic security precautions I laid out in Securing Linux, Part 1 -- specifically, by never enabling services you don't really need or use.

  6. Counterattacks
    While reaction systems counter intrusions by blocking or cancelling the attempts, counterattack systems launch return port scans or other aggressive attacks against the intruding system.

    An equal and opposite overreaction
    Some security systems, such as Abacus Sentry, include counterattack capability. While it is tempting to think about striking back at an intruder, these systems can cause a great deal of trouble. Some of these counterattack measures can be turned against their owners or be used by attackers to attack others in turn. By configuring these systems to counterattack in the face of a UDP or FIN scan (which can be spoofed), you may actually set yourself up to be used to attack other systems.

    One site on a network erected a counterattacker that was triggered by any attempt to telnet to the system. In response to a telnet connection, this system would e-mail a nasty message to the system administrator of the originating system, as well as to the domain and network administrators. Someone then discovered what great fun it would be to include a telnet:// URL in a couple of Web pages and in some HTML-enabled e-mail. The resulting flood of complaints and nasty messages showed just how bad a mistake a poorly thought-out counterattack can be.

    My personal advice on counterattack systems is: don't do it! It's not worth it if someone subverts your system to attack someone else -- and you get the blame!

  7. System logs and logging servers
    Moving your logging server to a separate, isolated machine adds another level of security to your system; replacing the logging utility with one of its secure counterparts makes for an even better solution.

    Secure logging
    Use a separate logging server for syslog services. This could be a spare box with sufficient disk space for all of your logging needs. There should be no other services running on this system. This will severely limit an intruder's ability to penetrate the logging system and tamper with the logs. Periodically writing the logs to a write-once media, for those who have such devices, is also useful to thwart attempts to tamper with logs.

    Secure syslog
    Even with separate log servers, the stock syslog utility is rather insecure. While it is true that prevention is the first line of defense, detection is always the second.

    Bruce Schneier has presented ideas for creating a secure log server in which cryptographic signatures would be incorporated into the logs. This would ensure that intrusions cannot go undetected, even if the logs are tampered with.

    A secure log server replacement for syslog, called ssyslong, is available from the Core SDI site (see Resources). This daemon implements cryptographic security on the logs and enables remote auditing.

    From the Core SDI page on Secure Syslog:

    BLOCKQUOTE: Designed to replace the syslog daemon, ssyslog implements a cryptographic protocol called PEO-1 that allows the remote auditing of system logs. Auditing remains possible even if an intruder gains superuser privileges in the system, [as] the protocol guarantees that the information logged before and during the intrusion process cannot be modified without the auditor (on a remote, trusted host) noticing. :END_BLOCKQUOTE

    syslog-ng
    Another secure alternative to syslog is syslog-ng, the "next generation syslog," available from BalaBit.

    This is a more configurable logging daemon, and it includes cryptographic signatures to detect log tampering. It also allows for directing log traffic based on facility and level, as well as on content and use of regular expressions.

    Remote logging servers, in combination with cryptographically secure logging servers and secure remote audit capability, can make log tampering extremely difficult to perform -- and very easy to detect.

  8. Single sign-on systems
    Single sign-on is a great convenience, and it can actually improve security if implemented correctly. But it can increase your risk if it's not done properly.

    Maintaining numerous user accounts in diverse environments is a continual problem for administrators in large networks. Several single sign-on systems exist to ease this administrative burden, while at the same time attempting to maintain or improve security.

    Network Information Services, or NIS, is a service from Sun originally called Yellow Pages -- until a telephone company in Europe complained about trademark infringement. Its basic security features are less than robust, and some of the bugs and vulnerabilities which have been uncovered have earned it the nickname of Network Intruder Services. A newer version of NIS called NIS+ hardens the security and attempts to address many of the shortcomings of the original. Both are available for Linux.

    Kerberos is another single sign-on system with a mixed history. Kerberos version 4 has some notable security holes, including the ability to brute force crack the Kerberos cookies offline and undetected. Kerberos version 5 is not vulnerable to these problems and modes of exploitation.

    In large networks, single sign-on systems like NIS or Kerberos are both a blessing and a curse. On one hand, having the authentication mechanism on a separate system helps isolate that functionality and reduces the risk of compromise through other services. On the other, once an account is compromised on one system, all systems accessible by that account are equally compromised. Good passwords are a major requirement in a situation where a single cracked password can jeopardize many systems.

    Authenticating against Windows domains
    Windows-based networks can have their own single sign-on system in the Windows NT domain system. Linux systems can authenticate against the Windows system. This allows users to maintain and manage their accounts and passwords under Windows while the changes are reflected in their Unix logins at the same time.

    Using pam_smb, Linux systems can authenticate against Windows SMB Domain Controllers (primary or secondary). This is handy in a network centered around Windows network administration, but it brings with it all of the insecurities of the Windows authentication systems.

  9. A Red Queen's race -- keeping up with advances in security
    The biggest weapon in your arsenal is your ability to stay abreast of developments. When you are dealing with advanced security, it isn't enough to just watch for new vulnerabilities, hacks, and versions. While taking the operation of your systems to the bleeding edge of development may not be appropriate, security development is taking place at a rapid pace and you should remain aware of the state of the art.

    Secure Linux distributions
    There are currently at least three secure Linux projects which have been started, each with slightly different goals. Each of these projects complements the other two, and URLs for all three can be found in our Resources section.

    Secure Linux
    The goal of the Secure Linux project is to create a secure Linux distribution for Internet server systems. The distribution is intended to feature a package system and some file accounting. The project managers are looking to incorporate strong cryptography and some extra Web server functionality into their product.

    Since this is being created outside of the United States, we can expect enhanced cryptographic security without the onerous burden of archaic U.S. export regulations.

    Bastille Linux
    Bastille Linux looks to build upon the example of OpenBSD within the Linux environment. The project's stated goal is to create a secure distribution for the desktop which network administrators can give out to their users without worrying about security.

    Kha0s Linux
    Kha0s Linux is looking to create a minimal and secure Linux distribution with strong encryption and OpenBSD-like policies. The group is currently soliciting contributions and participation on its Web site.

    Common secure Linux mailing lists
    In an effort to unify some of the discussions over these projects in particular, and about Linux security in general, some of the projects have organized a common mailing list, securedistros@nl.linux.org. This is a relatively new list and is intended to cut down on some of the redundancy and fragmentation between the three major secure Linux projects.

    There is another common mailing list -- devoted to security auditing of source code -- at security-audit@ferret.lmh.ox.ac.uk. This list is likely to have significant overlap with the other mailing lists. But it's worth a look if you are looking to get involved in code audits and related security issues.

    Linux Administrators Security Guide
    The Linux Administrators Security Guide, is a large document covering a variety of security issues surrounding Linux. Its author distributes this only as a pdf file in order to maintain more editorial control over its content and modifications. It's an excellent guide with a wealth of information -- but it is still subject to frequent updates. While pdf files can be a pain (some consider them to be contrary to the entire philosophy of the Web), the high value of the content of the document greatly outweighs the inconvenience of its container. I would recommend it to anyone with serious concerns about security.

  10. Defense in depth
    Some would argue that using one of these new secure distributions and the firewall code and the TCP wrappers controls and something like Abacus Sentry on a single system is very redundant. It is, and that is how it should be.

    Layers of defense
    The concept of defense in depth implies layers of security and detection. If an intruder gets past one, there is a chance he or she will trip on the next. This system turns the table on intruders. In the past, to be secure, you had to be perfect -- and the intruder had to detect only one flaw. With security in depth, an intruder must quietly detect and take advantage of a flaw in each and every layer (or bypass the layer) -- while a single failure on his part means detection. With defense in depth, a flaw in one layer should be covered by another layer.

    System security, in turn, becomes a set of layers within network security. Adaptive filtering at the router combines with firewalls, isolated DMZ networks, and intrusion detection systems to lay a minefield for intruders attempting to break into large corporate networks. (Most of the techniques used in these large networks are difficult to apply to smaller, individual networks.)

    To intrude into you system, an attacker should have to circumvent your reactive firewall, avoid your intrusion detection systems, ignore your lures and honey pots, work his way past your system filters, trick your network wrapper code, avoid your logging monitors, bypass the security protections on your filesystems, disarm your filesystem attributes, work past your filesystem protections, and subvert your protected logging servers -- all without knowing what you have prepared for him, and all without triggering the alarms that call you to man the barricades.

    You can turn the tables on intruders by ensuring that, if they get past one or two of your defenses, they still have unpleasant surprises in store for them.

While security is not an exact science with well-defined goals, it rarely needs to be perfect. You don't need to be faster than the bear which is chasing you. You merely need to be faster -- or less tasty -- than the others running with you. About the author
Mike Warfield is a senior security researcher on the Internet Security Systems X-Force. He is also a member of the Samba Development Team, a founding member of the Atlanta Linux Enthusiasts, and a long-standing guru with the Atlanta Unix Users Group. He has been involved in Unix systems for over 15 years and with computer security for over 25 years.

Resources

Further reading and security resources
(Most of the sites listed below feature additional security resources and links.)
    Resources mentioned in this article Invaluable security resources

    See Also

    Featured Links