top-linux-monitoring-tools

eed to monitor Linux server performance? Try these built-in commands and a few add-on tools. Most Linux distributions are equipped with tons of monitoring. These tools provide metrics which can be used to get information about system activities. You can use these tools to find the possible causes of a performance problem. The commands discussed below are some of the most basic commands when it comes to system analysis and debugging server issues such as:

1. Finding out bottlenecks.
2. Disk (storage) bottlenecks.
3. CPU and memory bottlenecks.
4. Network bottlenecks.

#1: top - Process Activity Command

The top program provides a dynamic real-time view of a running system i.e. actual process activity. By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds.

Fig.01: Linux top command

Commonly Used Hot Keys

The top command provides several useful hot keys:

Hot Key	Usage
t	Displays summary information off and on.
m	Displays memory information off and on.
A	Sorts the display by top consumers of various system resources. Useful for quick identification of performance-hungry tasks on a system.
f	Enters an interactive configuration screen for top. Helpful for setting up top for a specific task.
o	Enables you to interactively select the ordering within top.
r	Issues renice command.
k	Issues kill command.
z	Turn on or off color/mono

=> Related: How do I Find Out Linux CPU Utilization?

#2: vmstat - System Activity, Hardware and System Information

The command vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.
# vmstat 3
Sample Outputs:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 2540988 522188 5130400    0    0     2    32    4    2  4  1 96  0  0
 1  0      0 2540988 522188 5130400    0    0     0   720 1199  665  1  0 99  0  0
 0  0      0 2540956 522188 5130400    0    0     0     0 1151 1569  4  1 95  0  0
 0  0      0 2540956 522188 5130500    0    0     0     6 1117  439  1  0 99  0  0
 0  0      0 2540940 522188 5130512    0    0     0   536 1189  932  1  0 98  0  0
 0  0      0 2538444 522188 5130588    0    0     0     0 1187 1417  4  1 96  0  0
 0  0      0 2490060 522188 5130640    0    0     0    18 1253 1123  5  1 94  0  0

Display Memory Utilization Slabinfo

# vmstat -m

Get Information About Active / Inactive Memory Pages

# vmstat -a
=> Related: How do I find out Linux Resource utilization to detect system bottlenecks?

#3: w - Find Out Who Is Logged on And What They Are Doing

w command displays information about the users currently on the machine, and their processes.
# w username
# w vivek
Sample Outputs:

 17:58:47 up 5 days, 20:28,  2 users,  load average: 0.36, 0.26, 0.24
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    10.1.3.145       14:55    5.00s  0.04s  0.02s vim /etc/resolv.conf
root     pts/1    10.1.3.145       17:43    0.00s  0.03s  0.00s w

#4: uptime - Tell How Long The System Has Been Running

The uptime command can be used to see how long the server has been running. The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
# uptime
Output:

 18:02:41 up 41 days, 23:42,  1 user,  load average: 0.00, 0.00, 0.00

1 can be considered as optimal load value. The load can change from system to system. For a single CPU system 1 - 3 and SMP systems 6-10 load value might be acceptable.

#5: ps - Displays The Processes

ps command will report a snapshot of the current processes. To select all processes use the -A or -e option:
# ps -A
Sample Outputs:

  PID TTY          TIME CMD
    1 ?        00:00:02 init
    2 ?        00:00:02 migration/0
    3 ?        00:00:01 ksoftirqd/0
    4 ?        00:00:00 watchdog/0
    5 ?        00:00:00 migration/1
    6 ?        00:00:15 ksoftirqd/1
....
.....
 4881 ?        00:53:28 java
 4885 tty1     00:00:00 mingetty
 4886 tty2     00:00:00 mingetty
 4887 tty3     00:00:00 mingetty
 4888 tty4     00:00:00 mingetty
 4891 tty5     00:00:00 mingetty
 4892 tty6     00:00:00 mingetty
 4893 ttyS1    00:00:00 agetty
12853 ?        00:00:00 cifsoplockd
12854 ?        00:00:00 cifsdnotifyd
14231 ?        00:10:34 lighttpd
14232 ?        00:00:00 php-cgi
54981 pts/0    00:00:00 vim
55465 ?        00:00:00 php-cgi
55546 ?        00:00:00 bind9-snmp-stat
55704 pts/1    00:00:00 ps

ps is just like top but provides more information.

Show Long Format Output

# ps -Al
To turn on extra full mode (it will show command line arguments passed to process):
# ps -AlF

To See Threads ( LWP and NLWP)

# ps -AlFH

To See Threads After Processes

# ps -AlLm

Print All Process On The Server

# ps ax
# ps axu

Print A Process Tree

# ps -ejH
# ps axjf
# pstree

Print Security Information

# ps -eo euser,ruser,suser,fuser,f,comm,label
# ps axZ
# ps -eM

See Every Process Running As User Vivek

# ps -U vivek -u vivek u

Set Output In a User-Defined Format

# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
# ps -eopid,tt,user,fname,tmout,f,wchan

Display Only The Process IDs of Lighttpd

# ps -C lighttpd -o pid=
OR
# pgrep lighttpd
OR
# pgrep -u vivek php-cgi

Display The Name of PID 55977

# ps -p 55977 -o comm=

Find Out The Top 10 Memory Consuming Process

# ps -auxf | sort -nr -k 4 | head -10

Find Out top 10 CPU Consuming Process

# ps -auxf | sort -nr -k 3 | head -10

#6: free - Memory Usage

The command free displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel.
# free
Sample Output:

            total       used       free     shared    buffers     cached
Mem:      12302896    9739664    2563232          0     523124    5154740
-/+ buffers/cache:    4061800    8241096
Swap:      1052248          0    1052248

=> Related: :

1. Linux Find Out Virtual Memory PAGESIZE
2. Linux Limit CPU Usage Per Process
3. How much RAM does my Ubuntu / Fedora Linux desktop PC have?

#7: iostat - Average CPU Load, Disk Activity

The command iostat report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems (NFS).
# iostat
Sample Outputs:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 	06/26/2009
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.50    0.09    0.51    0.03    0.00   95.86
Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              22.04        31.88       512.03   16193351  260102868
sda1              0.00         0.00         0.00       2166        180
sda2             22.04        31.87       512.03   16189010  260102688
sda3              0.00         0.00         0.00       1615          0

=> Related: : Linux Track NFS Directory / Disk I/O Stats

#8: sar - Collect and Report System Activity

The sar command is used to collect, report, and save system activity information. To see network counter, enter:
# sar -n DEV | more
To display the network counters from the 24th:
# sar -n DEV -f /var/log/sa/sa24 | more
You can also display real time usage using sar:
# sar 4 5
Sample Outputs:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 		06/26/2009
06:45:12 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle
06:45:16 PM       all      2.00      0.00      0.22      0.00      0.00     97.78
06:45:20 PM       all      2.07      0.00      0.38      0.03      0.00     97.52
06:45:24 PM       all      0.94      0.00      0.28      0.00      0.00     98.78
06:45:28 PM       all      1.56      0.00      0.22      0.00      0.00     98.22
06:45:32 PM       all      3.53      0.00      0.25      0.03      0.00     96.19
Average:          all      2.02      0.00      0.27      0.01      0.00     97.70

=> Related: : How to collect Linux system utilization data into a file

#9: mpstat - Multiprocessor Usage

The mpstat command displays activities for each available processor, processor 0 being the first one. mpstat -P ALL to display average CPU utilization per processor:
# mpstat -P ALL
Sample Output:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in)	 	06/26/2009
06:48:11 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
06:48:11 PM  all    3.50    0.09    0.34    0.03    0.01    0.17    0.00   95.86   1218.04
06:48:11 PM    0    3.44    0.08    0.31    0.02    0.00    0.12    0.00   96.04   1000.31
06:48:11 PM    1    3.10    0.08    0.32    0.09    0.02    0.11    0.00   96.28     34.93
06:48:11 PM    2    4.16    0.11    0.36    0.02    0.00    0.11    0.00   95.25      0.00
06:48:11 PM    3    3.77    0.11    0.38    0.03    0.01    0.24    0.00   95.46     44.80
06:48:11 PM    4    2.96    0.07    0.29    0.04    0.02    0.10    0.00   96.52     25.91
06:48:11 PM    5    3.26    0.08    0.28    0.03    0.01    0.10    0.00   96.23     14.98
06:48:11 PM    6    4.00    0.10    0.34    0.01    0.00    0.13    0.00   95.42      3.75
06:48:11 PM    7    3.30    0.11    0.39    0.03    0.01    0.46    0.00   95.69     76.89

=> Related: : Linux display each multiple SMP CPU processors utilization individually.

#10: pmap - Process Memory Usage

The command pmap report memory map of a process. Use this command to find out causes of memory bottlenecks.
# pmap -d PID
To display process memory information for pid # 47394, enter:
# pmap -d 47394
Sample Outputs:

47394:   /usr/bin/php-cgi
Address           Kbytes Mode  Offset           Device    Mapping
0000000000400000    2584 r-x-- 0000000000000000 008:00002 php-cgi
0000000000886000     140 rw--- 0000000000286000 008:00002 php-cgi
00000000008a9000      52 rw--- 00000000008a9000 000:00000   [ anon ]
0000000000aa8000      76 rw--- 00000000002a8000 008:00002 php-cgi
000000000f678000    1980 rw--- 000000000f678000 000:00000   [ anon ]
000000314a600000     112 r-x-- 0000000000000000 008:00002 ld-2.5.so
000000314a81b000       4 r---- 000000000001b000 008:00002 ld-2.5.so
000000314a81c000       4 rw--- 000000000001c000 008:00002 ld-2.5.so
000000314aa00000    1328 r-x-- 0000000000000000 008:00002 libc-2.5.so
000000314ab4c000    2048 ----- 000000000014c000 008:00002 libc-2.5.so
.....
......
..
00002af8d48fd000       4 rw--- 0000000000006000 008:00002 xsl.so
00002af8d490c000      40 r-x-- 0000000000000000 008:00002 libnss_files-2.5.so
00002af8d4916000    2044 ----- 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b15000       4 r---- 0000000000009000 008:00002 libnss_files-2.5.so
00002af8d4b16000       4 rw--- 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b17000  768000 rw-s- 0000000000000000 000:00009 zero (deleted)
00007fffc95fe000      84 rw--- 00007ffffffea000 000:00000   [ stack ]
ffffffffff600000    8192 ----- 0000000000000000 000:00000   [ anon ]
mapped: 933712K    writeable/private: 4304K    shared: 768000K

The last line is very important:

• mapped: 933712K total amount of memory mapped to files
• writeable/private: 4304K the amount of private address space
• shared: 768000K the amount of address space this process is sharing with others

=> Related: : Linux find the memory used by a program / process using pmap command

#11 and #12: netstat and ss - Network Statistics

The command netstat displays network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. ss command is used to dump socket statistics. It allows showing information similar to netstat. See the following resources about ss and netstat commands:

• ss: Display Linux TCP / UDP Network and Socket Information
• Get Detailed Information About Particular IP address Connections Using netstat Command

#13: iptraf - Real-time Network Statistics

The iptraf command is interactive colorful IP LAN monitor. It is an ncurses-based IP LAN monitor that generates various network statistics including TCP info, UDP counts, ICMP and OSPF information, Ethernet load info, node stats, IP checksum errors, and others. It can provide the following info in easy to read format:

• Network traffic statistics by TCP connection
• IP traffic statistics by network interface
• Network traffic statistics by protocol
• Network traffic statistics by TCP/UDP port and by packet size
• Network traffic statistics by Layer2 address

Fig.02: General interface statistics: IP traffic statistics by network interface

Fig.03 Network traffic statistics by TCP connection

#14: tcpdump - Detailed Network Traffic Analysis

The tcpdump is simple command that dump traffic on a network. However, you need good understanding of TCP/IP protocol to utilize this tool. For.e.g to display traffic info about DNS, enter:
# tcpdump -i eth1 'udp port 53'
To display all IPv4 HTTP packets to and from port 80, i.e. print only packets that contain data, not, for example, SYN and FIN packets and ACK-only packets, enter:
# tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
To display all FTP session to 202.54.1.5, enter:
# tcpdump -i eth1 'dst 202.54.1.5 and (port 21 or 20'
To display all HTTP session to 192.168.1.5:
# tcpdump -ni eth0 'dst 192.168.1.5 and tcp and port http'
Use wireshark to view detailed information about files, enter:
# tcpdump -n -i eth1 -s 0 -w output.txt src or dst port 80

#15: strace - System Calls

Trace system calls and signals. This is useful for debugging webserver and other server problems. See how to use to trace the process and see What it is doing.

#16: /Proc file system - Various Kernel Statistics

/proc file system provides detailed information about various hardware devices and other Linux kernel information. See Linux kernel /proc documentations for further details. Common /proc examples:
# cat /proc/cpuinfo
# cat /proc/meminfo
# cat /proc/zoneinfo
# cat /proc/mounts

17#: Nagios - Server And Network Monitoring

Nagios is a popular open source computer system and network monitoring application software. You can easily monitor all your hosts, network equipment and services. It can send alert when things go wrong and again when they get better. FAN is "Fully Automated Nagios". FAN goals are to provide a Nagios installation including most tools provided by the Nagios Community. FAN provides a CDRom image in the standard ISO format, making it easy to easilly install a Nagios server. Added to this, a wide bunch of tools are including to the distribution, in order to improve the user experience around Nagios.

18#: Cacti - Web-based Monitoring Tool

Cacti is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices. It can provide data about network, CPU, memory, logged in users, Apache, DNS servers and much more. See how to install and configure Cacti network graphing tool under CentOS / RHEL.

#19: KDE System Guard - Real-time Systems Reporting and Graphing

KSysguard is a network enabled task and system monitor application for KDE desktop. This tool can be run over ssh session. It provides lots of features such as a client/server architecture that enables monitoring of local and remote hosts. The graphical front end uses so-called sensors to retrieve the information it displays. A sensor can return simple values or more complex information like tables. For each type of information, one or more displays are provided. Displays are organized in worksheets that can be saved and loaded independently from each other. So, KSysguard is not only a simple task manager but also a very powerful tool to control large server farms.

Fig.05 KDE System Guard {Image credit: Wikipedia}

See the KSysguard handbook for detailed usage.

#20: Gnome System Monitor - Real-time Systems Reporting and Graphing

The System Monitor application enables you to display basic system information and monitor system processes, usage of system resources, and file systems. You can also use System Monitor to modify the behavior of your system. Although not as powerful as the KDE System Guard, it provides the basic information which may be useful for new users:

• Displays various basic information about the computer's hardware and software.
• Linux Kernel version
• GNOME version
• Hardware
• Installed memory
• Processors and speeds
• System Status
• Currently available disk space
• Processes
• Memory and swap space
• Network usage
• File Systems
• Lists all mounted filesystems along with basic information about each.

Fig.06 The Gnome System Monitor application

Bonus: Additional Tools

A few more tools:

• nmap - scan your server for open ports.
• lsof - list open files, network connections and much more.
• ntop web based tool - ntop is the best tool to see network usage in a way similar to what top command does for processes i.e. it is network traffic monitoring software. You can see network status, protocol wise distribution of traffic for UDP, TCP, DNS, HTTP and other protocols.
• Conky - Another good monitoring tool for the X Window System. It is highly configurable and is able to monitor many system variables including the status of the CPU, memory, swap space, disk storage, temperatures, processes, network interfaces, battery power, system messages, e-mail inboxes etc.
• GKrellM - It can be used to monitor the status of CPUs, main memory, hard disks, network interfaces, local and remote mailboxes, and many other things.
• vnstat - vnStat is a console-based network traffic monitor. It keeps a log of hourly, daily and monthly network traffic for the selected interface(s).
• htop - htop is an enhanced version of top, the interactive process viewer, which can display the list of processes in a tree form.
• mtr - mtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.

php-security-best-practices-tutorial

PHP is an open-source server-side scripting language and it is a widely used. The Apache web server provides access to files and content via the HTTP OR HTTPS protocol. A misconfigured server-side scripting language can create all sorts of problems. So, PHP should be used with caution. Here are twenty-fivephp security best practices for sysadmins for configuring PHP securely.

Our Sample Setup For PHP Security Tips

• DocumentRoot: /var/www/html
• Default Web server: Apache ( you can use Lighttpd or Nginx instead of Apache)
• Default PHP configuration file: /etc/php.ini
• Default PHP extensions config directory: /etc/php.d/
• Our sample php security config file: /etc/php.d/security.ini (you need to create this file using a text editor)
• Operating systems: RHEL / CentOS / Fedora Linux (the instructions should work with any other Linux distributions such as Debian / Ubuntu or other Unix like operating systems such as OpenBSD/FreeBSD/HP-UX).
• Default php server TCP/UDP ports: none

Most of the actions listed in this post are written with the assumption that they will be executed by the root user running the bash or any other modern shell:
$ php -v
Sample outputs:

PHP 5.3.3 (cli) (built: Oct 24 2011 08:35:41)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies

For demonstration purpose I'm going to use the following operating system:
$ cat /etc/redhat-release
Sample outputs:

Red Hat Enterprise Linux Server release 6.1 (Santiago)

#1: Know Your Enemy

PHP based apps can face the different types of attacks. I have noticed the different types of attacks:

1. XSS - Cross-site scripting is a vulnerability in php web applications, which attackers may exploit to steal users' information. You can configure Apache and write more secure PHP scripts (validating all user input) to avoid xss attacks.
2. SQL injection - It is a vulnerability in the database layer of an php application. When user input is incorrectly filtered any SQL statements can be executed by the application. You can configure Apache and write secure code (validating and escaping all user input) to avoid SQL injection attacks. A common practice in PHP is to escape parameters using the function called mysql_real_escape_string() before sending the SQL query.
   Spoofing
3. File uploads - It allows your visitor to place files (upload files) on your server. This can result into various security problems such as delete your files, delete database, get user details and much more. You can disable file uploads using php or write secure code (like validating user input and only allow image file type such as png or gif).
4. Including local and remote files - An attacker can open files from remote server and execute any PHP code. This allows them to upload file, delete file and install backdoors. You can configure php to disable remote file execution.
5. eval() - Evaluate a string as PHP code. This is often used by an attacker to hide their code and tools on the server itself. You can configure php to disable eval().
6. Sea-surf Attack (Cross-site request forgery - CSRF) - This attack forces an end user to execute unwanted actions on a web application in which he/she is currently authenticated. A successful CSRF exploit can compromise end user data and operation in case of normal user. If the targeted end user is the administrator account, this can compromise the entire web application.

#2: Find Built-in PHP Modules

To see the set of compiled-in PHP modules type the following command:
# php -m
Sample outputs:

[PHP Modules]
apc
bcmath
bz2
calendar
Core
ctype
curl
date
dom
ereg
exif
fileinfo
filter
ftp
gd
gettext
gmp
hash
iconv
imap
json
libxml
mbstring
memcache
mysql
mysqli
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
readline
Reflection
session
shmop
SimpleXML
sockets
SPL
sqlite3
standard
suhosin
tokenizer
wddx
xml
xmlreader
xmlrpc
xmlwriter
xsl
zip
zlib
[Zend Modules]
Suhosin

I recommends that you use PHP with a reduced modules for performance and security. For example, you can disable sqlite3 module by deleting (removing) configuration file , ORrenaming (moving) a file called /etc/php.d/sqlite3.ini as follows:
# rm /etc/php.d/sqlite3.ini
OR
# mv /etc/php.d/sqlite3.ini /etc/php.d/sqlite3.disable
Other compiled-in modules can only be removed by reinstallating PHP with a reduced configuration. You can download php source code from php.net and compile it as follows with GD, fastcgi, and MySQL support:

./configure --with-libdir=lib64 --with-gd --with-mysql --prefix=/usr --exec-prefix=/usr \
--bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share \
--includedir=/usr/include --libexecdir=/usr/libexec --localstatedir=/var \
--sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info \
--cache-file=../config.cache --with-config-file-path=/etc \
--with-config-file-scan-dir=/etc/php.d  --enable-fastcgi \
--enable-force-cgi-redirect

See how to compile and reinstall php on Unix like operating system for more information.

#3: Restrict PHP Information Leakage

To restrict PHP information leakage disable expose_php. Edit /etc/php.d/secutity.ini and set the following directive:

expose_php=Off

When enabled, expose_php reports to the world that PHP is installed on the server, which includes the PHP version within the HTTP header (e.g., X-Powered-By: PHP/5.3.3). The PHP logo guids (see example) are also exposed, thus appending them to the URL of a PHP enabled site will display the appropriate logo. When expose_php enabled you can see php version using the following command:
$ curl -I http://www.cyberciti.biz/index.php
Sample outputs:

HTTP/1.1 200 OK
X-Powered-By: PHP/5.3.3
Content-type: text/html; charset=UTF-8
Vary: Accept-Encoding, Cookie
X-Vary-Options: Accept-Encoding;list-contains=gzip,Cookie;string-contains=wikiToken;string-contains=wikiLoggedOut;string-contains=wiki_session
Last-Modified: Thu, 03 Nov 2011 22:32:55 GMT
...

I also recommend that you setup the ServerTokens and ServerSignature directives in httpd.conf to hide Apache version and other information.

#4: Minimize Loadable PHP Modules (Dynamic Extensions)

PHP supports "Dynamic Extensions". By default, RHEL loads all the extension modules found in /etc/php.d/ directory. To enable or disable a particular module, just find the configuration file in /etc/php.d/ directory and comment the module name. You can also rename or delete module configuration file. For best PHP performance and security, you should only enable the extensions your webapps requires. For example, to disable gd extension, type the following commands:
# cd /etc/php.d/
# mv gd.{ini,disable}
# /sbin/service httpd restart
To enable php module called gd, enter:
# mv gd.{disable,ini}
# /sbin/service httpd restart

#5: Log All PHP Errors

Do not expose PHP error messages to all site visitors. Edit /etc/php.d/security.ini and set the following directive:

display_errors=Off

Make sure you log all php errors to a log file:

log_errors=On
error_log=/var/log/httpd/php_scripts_error.log

#6: Disallow Uploading Files

Edit /etc/php.d/security.ini and set the following directive to disable file uploads for security reasons:

file_uploads=Off

If users of your application need to upload files, turn this feature on by settingupload_max_filesize limits the maximum size of files that PHP will accept through uploads:

file_uploads=On
# user can only upload upto 1MB via php
upload_max_filesize=1M
 

#7: Turn Off Remote Code Execution

If enabled, allow_url_fopen allows PHP's file functions -- such as file_get_contents() and the include and require statements -- can retrieve data from remote locations, like an FTP or web site.

The allow_url_fopen option allows PHP's file functions - such as file_get_contents() and the include and require statements - can retrieve data from remote locations using ftp or http protocols. Programmers frequently forget this and don't do proper input filtering when passing user-provided data to these functions, opening them up to code injection vulnerabilities. A large number of code injection vulnerabilities reported in PHP-based web applications are caused by the combination of enabling allow_url_fopen and bad input filtering. Edit /etc/php.d/security.ini and set the following directive:

allow_url_fopen=Off

I also recommend to disable allow_url_include for security reasons:

allow_url_include=Off

#8: Enable SQL Safe Mode

Edit /etc/php.d/security.ini and set the following directive:

sql.safe_mode=On
 

If turned On, mysql_connect() and mysql_pconnect() ignore any arguments passed to them. Please note that you may have to make some changes to your code. Third party and open source application such as WordPress, and others may not work at all when sql.safe_mode enabled. I also recommend that you turn off magic_quotes_gpc for all php 5.3.x installations as the filtering by it is ineffective and not very robust. mysql_escape_string() and custom filtering functions serve a better purpose (hat tip to Eric Hansen):

magic_quotes_gpc=Off

#9: Control POST Size

The HTTP POST request method is used when the client (browser or user) needs to send data to the Apache web server as part of the request, such as when uploading a file or submitting a completed form. Attackers may attempt to send oversized POST requests to eat your system resources. You can limit the maximum size POST request that PHP will process. Edit /etc/php.d/security.ini and set the following directive:

; Set a realistic value here 
post_max_size=1K

The 1K sets max size of post data allowed by php apps. This setting also affects file upload. To upload large files, this value must be larger than upload_max_filesize. I also suggest that you limit available methods using Apache web server. Edit, httpd.conf and set the following directive for DocumentRoot /var/www/html:

 
<Directory /var/www/html>
    <LimitExcept GET POST>
        Order allow,deny
    </LimitExcept>
## Add rest of the config goes here... ##
</Directory>
 

#10: Resource Control (DoS Control)

You can set maximum execution time of each php script, in seconds. Another recommend option is to set maximum amount of time each script may spend parsing request data, and maximum amount of memory a script may consume. Edit /etc/php.d/security.ini and set the following directives:

# set in seconds
max_execution_time =  30
max_input_time = 30
memory_limit = 40M
 

#11: Install Suhosin Advanced Protection System for PHP

From the project page:

Suhosin is an advanced protection system for PHP installations. It was designed to protect servers and users from known and unknown flaws in PHP applications and the PHP core. Suhosin comes in two independent parts, that can be used separately or in combination. The first part is a small patch against the PHP core, that implements a few low-level protections against bufferoverflows or format string vulnerabilities and the second part is a powerful PHP extension that implements all the other protections.

See how to install and configure suhosin under Linux operating systems.

#12 Disabling Dangerous PHP Functions

PHP has a lot of functions which can be used to crack your server if not used properly. You can set list of functions in /etc/php.d/security.ini using disable_functions directive:

 
disable_functions =exec,passthru,shell_exec,system,proc_open,popen,curl_exec,curl_multi_exec,parse_ini_file,show_source
 

#13 PHP Fastcgi / CGI - cgi.force_redirect Directive

PHP work with FastCGI. Fascgi reduces the memory footprint of your web server, but still gives you the speed and power of the entire PHP language. You can configureApache2+PHP+FastCGI or cgi as described here. The configuration directive cgi.force_redirect prevents anyone from calling PHP directly with a URL like http://www.cyberciti.biz/cgi-bin/php/hackerdir/backdoor.php. Turn on cgi.force_redirect for security reasons. Edit /etc/php.d/security.ini and set the following directive:

; Enable cgi.force_redirect for security reasons in a typical *Apache+PHP-CGI/FastCGI* setup
cgi.force_redirect=On
 

#14 PHP User and Group ID

mod_fastcgi is a cgi-module for Apache web server. It can connect to an external FASTCGI server. You need to make sure php run as non-root user. If PHP executes as a root or UID under 100, it may access and/or manipulate system files. You must execute PHP CGIs as a non-privileged user using Apache's suEXEC or mod_suPHP. The suEXEC feature provides Apache users the ability to run CGI programs under user IDs different from the user ID of the calling web server. In this example, my php-cgi is running as phpcgi user and apache is running as apache user:
# ps aux | grep php-cgi
Sample outputs:

phpcgi      6012  0.0  0.4 225036 60140 ?        S    Nov22   0:12 /usr/bin/php-cgi
phpcgi      6054  0.0  0.5 229928 62820 ?        S    Nov22   0:11 /usr/bin/php-cgi
phpcgi      6055  0.1  0.4 224944 53260 ?        S    Nov22   0:18 /usr/bin/php-cgi
phpcgi      6085  0.0  0.4 224680 56948 ?        S    Nov22   0:11 /usr/bin/php-cgi
phpcgi      6103  0.0  0.4 224564 57956 ?        S    Nov22   0:11 /usr/bin/php-cgi
phpcgi      6815  0.4  0.5 228556 61220 ?        S    00:52   0:19 /usr/bin/php-cgi
phpcgi      6821  0.3  0.5 228008 61252 ?        S    00:55   0:12 /usr/bin/php-cgi
phpcgi      6823  0.3  0.4 225536 58536 ?        S    00:57   0:13 /usr/bin/php-cgi

You can use tool such as spawn-fcgi to spawn remote and local FastCGI processes as phpcgi user (first, add phpcgi user to the system):
# spawn-fcgi -a 127.0.0.1 -p 9000 -u phpcgi -g phpcgi -f /usr/bin/php-cgi
Now, you can configure Apache, Lighttpd, and Nginx web server to use external php FastCGI running on port 9000 at 127.0.0.1 IP address.

#15 Limit PHP Access To File System

The open_basedir directive set the directories from which PHP is allowed to access files using functions like fopen(), and others. If a file is outside of the paths defined by open_basdir, PHP will refuse to open it. You cannot use a symbolic link as a workaround. For example only allow access to /var/www/html directory and not to /var/www, or /tmp or /etc directories:

; Limits the PHP process from accessing files outside 
; of specifically designated directories such as /var/www/html/
open_basedir="/var/www/html/"
; ------------------------------------
; Multiple dirs example 
; open_basedir="/home/httpd/vhost/cyberciti.biz/html/:/home/httpd/vhost/nixcraft.com/html/:/home/httpd/vhost/theos.in/html/"
; ------------------------------------
 

#16 Session Path

Session support in PHP consists of a way to preserve certain data across subsequent accesses. This enables you to build more customized applications and increase the appeal of your web site. This path is defined in /etc/php.ini file and all data related to a particular session will be stored in a file in the directory specified by the session.save_path option. The default is as follows under RHEL/CentOS/Fedora Linux:

session.save_path="/var/lib/php/session"
; Set the temporary directory used for storing files when doing file upload
upload_tmp_dir="/var/lib/php/session"
 

Make sure path is outside /var/www/html and not readable or writeable by any other system users:
# ls -Z /var/lib/php/
Sample outputs:

drwxrwx---. root apache system_u:object_r:httpd_var_run_t:s0 session

Note: The -Z option to the ls command display SELinux security context such as file mode, user, group, security context and file name.

#17 Keep PHP, Software, And OS Up to Date

Applying security patches is an important part of maintaining Linux, Apache, PHP, and MySQL server. All php security update should be reviewed and applied as soon as possible using any one of the following tool (if you're installing PHP via a package manager):
# yum update
OR
# apt-get update && apt-get upgrade
You can configure Red hat / CentOS / Fedora Linux to send yum package update notification via email. Another option is to apply all security updates via a cron job. Under Debian / Ubuntu Linux you can use apticron to send security notifications.

Note: Check php.net for the most recent release for source code installations.

#18: Restrict File and Directory Access

Make sure you run Apache as a non-root user such as Apache or www. All files and directory should be owned by non-root user (or apache user) under /var/www/html:
# chown -R apache:apache /var/www/html/
/var/www/html/ is a subdirectory and DocumentRoot which is modifiable by other users since root never executes any files out of there, and shouldn't be creating files in there.

Make sure file permissions are set to 0444 (read-only) under /var/www/html/:
# chmod -R 0444 /var/www/html/
Make sure all directories permissions are set to 0445 under /var/www/html/:
# find /var/www/html/ -type d -print0 | xargs -0 -I {} chmod 0445 {}

A Note About Setting Up Correct File Permissions

The chown and chmod command make sures that under no circumstances DocumentRoot or files contained in DocumentRoot are writable by the Web server user apache. Please note that you need to set permissions that makes the most sense for the development model of your website, so feel free to adjust the chown and chmod command as per your requirements. In this example, the Apache server run as apache user. This is configured with the User and Groupdirectives in your httpd.conf file. The apache user needs to have read access to everything under DocumentRoot but should not have write access to anything.

Make sure httpd.conf has the following directives for restrictive configuration:

 
<Directory / >
    Options None
    AllowOverride None
    Order allow,deny
</Directory>
 

You should only grant write access when required. Some web applications such as wordpress and others may need a caching directory. You can grant a write access to caching directory using the following commands:
# chmod a+w /var/www/html/blog/wp-content/cache
### block access to all ###
# echo 'deny from all' > /var/www/html/blog/wp-content/cache/.htaccess

#19: Write Protect Apache, PHP, and, MySQL Configuration Files

Use the chattr command to write protect configuration files:
# chattr +i /etc/php.ini
# chattr +i /etc/php.d/*
# chattr +i /etc/my.ini
# chattr +i /etc/httpd/conf/httpd.conf
# chattr +i /etc/

The chattr command can write protect your php file or files in /var/www/html directory too:
# chattr +i /var/www/html/file1.php
# chattr +i /var/www/html/

#20: Use Linux Security Extensions (such as SELinux)

Linux comes with various security patches which can be used to guard against misconfigured or compromised server programs. If possible use SELinux and other Linux security extensions to enforce limitations on network and other programs. For example, SELinux provides a variety of security policies for Linux kernel and Apache web server. To list all Apache SELinux protection variables, enter:
# getsebool -a | grep httpd
Sample outputs:

allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> on
httpd_enable_cgi --> on
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_read_user_content --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_tty_comm --> on
httpd_unified --> on
httpd_use_cifs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off

To disable Apache cgi support, enter:
# setsebool -P httpd_enable_cgi off
See Red Hat SELinux guide for more information.

#21 Install Mod_security

ModSecurity is an open source intrusion detection and prevention engine for web applications. You can easily install mod_security under Linux and protect apache and php based apps from xss and various other attacks:

 
## A few Examples ##
# Do not allow to open files in /etc/
SecFilter /etc/
 
# Stop SQL injection
SecFilter "delete[[:space:]]+from"
SecFilter "select.+from"
 

#22 Run Apache / PHP In a Chroot Jail If Possible

Putting PHP and/or Apache in a chroot jail minimizes the damage done by a potential break-in by isolating the web server to a small section of the filesystem. You can use traditional chroot kind of setup with Apache. However, I recommend FreeBSD jails, XEN virtulization, KVM virtulization, or OpenVZ virtualization which uses the concept of containers.

#23 Use Firewall To Restrict Outgoing Connections

The attacker will download file locally on your web-server using tools such as wget. Use iptables to block outgoing connections from apache user. The ipt_owner module attempts to match various characteristics of the packet creator, for locally generated packets. It is only valid in the OUTPUT chain. In this example, allow vivek user to connect outside using port 80 (useful for RHN or centos repo access):

 
/sbin/iptables -A OUTPUT -o eth0 -m owner --uid-owner vivek -p tcp --dport 80 -m state --state NEW,ESTABLISHED  -j ACCEPT
 

Here is another example that blocks all outgoing connections from apache user except to our own smtp server, and spam validation API service:

 
# ....  
/sbin/iptables --new-chain apache_user
/sbin/iptables --append OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables --append OUTPUT -m owner --uid-owner apache -j apache_user
# allow apache user to connec to our smtp server 
/sbin/iptables --append apache_user -p tcp --syn -d 192.168.1.100 --dport 25 -j RETURN
# Allow apache user to connec to api server for spam validation
/sbin/iptables --append apache_user -p tcp --syn -d  66.135.58.62 --dport 80 -j RETURN
/sbin/iptables --append apache_user -p tcp --syn -d  66.135.58.61 --dport 80 -j RETURN
/sbin/iptables --append apache_user -p tcp --syn -d  72.233.69.89 --dport 80 -j RETURN
/sbin/iptables --append apache_user -p tcp --syn -d  72.233.69.88 --dport 80 -j RETURN
#########################
## Add more rules here ##
#########################
# No editing below
# Drop everything for apache outgoing connection
/sbin/iptables --append apache_user -j REJECT
 

#24 Watch Your Logs & Auditing

Check the apache log file:
# tail -f /var/log/httpd/error_log
# grep 'login.php' /var/log/httpd/error_log
# egrep -i "denied|error|warn" /var/log/httpd/error_log
Check the php log file:
# tail -f /var/log/httpd/php_scripts_error.log
# grep "...etc/passwd" /var/log/httpd/php_scripts_error.log
Log files will give you some understanding of what attacks is thrown against the server and allow you to check if the necessary level of security is present or not. The auditd service is provided for system auditing. Turn it on to audit SELinux events, authetication events, file modifications, account modification and so on. I also recommend using standard "Linux System Monitoring Tools" for monitoring your web-server.

#25 Run Service Per System or VM Instance

For large installations it is recommended that you run, database, static, and dynamic content from different servers.

///////////////
/ ISP/Router /
//////////////
  \
   |
   Firewall
     \
      |
     +------------+
     | LB01       |
     +------------+                 +--------------------------+
                  |                 | static.lan.cyberciti.biz |
		  +-----------------+--------------------------+
                                    | phpcgi1.lan.cyberciti.biz|
                                    +--------------------------+
                                    | phpcgi2.lan.cyberciti.biz|
                                    +--------------------------+
                                    | mysql1.lan.cyberciti.biz |
                                    +--------------------------+
                                    | mcache1.lan.cyberciti.biz|
                                    +--------------------------+

(Fig.01: Running Services On Separate Servers)

Run different network services on separate servers or VM instances. This limits the number of other services that can be compromised. For example, if an attacker able to successfully exploit a software such as Apache flow, he / she will get an access to entire server including other services running on the same server (such as MySQL, e-mail server and so on). But, in the above example content are served as follows:

1. static.lan.cyberciti.biz - Use lighttpd or nginx server for static assets such as js/css/images.
2. phpcgi1.lan.cyberciti.biz and phpcgi2.lan.cyberciti.biz - Apache web-server with php used for generating dynamic content.
3. mysql1.lan.cyberciti.biz - MySQL database server.
4. mcache1.lan.cyberciti.biz - Memcached server is very fast caching system for MySQL. It uses libevent or epoll (Linux runtime) to scale to any number of open connections and uses non-blocking network I/O.
5. LB01 - A nginx web and reverse proxy server in front of Apache Web servers. All connections coming from the Internet addressed to one of the Web servers are routed through the nginx proxy server, which may either deal with the request itself or pass the request wholly or partially to the main web servers. LB01 provides simple load-balancing.

#26 Additional Tools

From the project page:

PHPIDS (PHP-Intrusion Detection System) is a simple to use, well structured, fast and state-of-the-art security layer for your PHP based web application. The IDS neither strips, sanitizes nor filters any malicious input, it simply recognizes when an attacker tries to break your site and reacts in exactly the way you want it to.

You can use PHPIDS to detect malicious users, and log any attacks detected for later review. Please note that I've personally not used this tool.

From the project page:

PhpSecInfo provides an equivalent to the phpinfo() function that reports security information about the PHP environment, and offers suggestions for improvement. It is not a replacement for secure development techniques, and does not do any kind of code or app auditing, but can be a useful tool in a multilayered security approach.

Fig.02: Security Information About PHP Application

See Linux security hardening tips which can reduce available vectors of attack on the system.

A Note About PHP Backdoors

You may come across php scripts or so called common backdoors such as c99, c99madshell, r57 and so on. A backdoor php script is nothing but a hidden script for bypassing all authentication and access your server on demand. It is installed by an attackers to access your server while attempting to remain undetected. Typically a PHP (or any other CGI script) script by mistake allows inclusion of code exploiting vulnerabilities in the web browser. An attacker can use such exploiting vulnerabilities to upload backdoor shells which can give him or her a number of capabilities such as:

• Download files
• Upload files
• Install rootkits
• Set a spam mail servers / relay server
• Set a proxy server to hide tracks
• Take control of server
• Take control of database server
• Steal all information
• Delete all information and database
• Open TCP / UDP ports and much more

Tip: How Do I Search PHP Backdoors?

Use Unix / Linux grep command to search c99 or r57 shell:
# grep -iR 'c99' /var/www/html/
# grep -iR 'r57' /var/www/html/
# find /var/www/html/ -name \*.php -type f -print0 | xargs -0 grep c99
# grep -RPn "(passthru|shell_exec|system|base64_decode|fopen|fclose|eval)" /var/www/html/

Conclusion

Your PHP based server is now properly harden and ready to show dynamic webpages. However, vulnerabilities are caused mostly by not following best practice programming rules. You should be consulted further resources for your web applications security needs especially php programming which is beyond the scope of sys admin work.