How to build mod_wsgi,web.py with httpd and python from source code

It is best to compile everything from source code.

Version usedPython: 2.7 (with shared library)httpd 2.2.17 (with shared DSO support)WSGI (integrating with httpd): mod_wsgi 3.3
Web framework: web.py 0.34

Compile Python for shared lib

./configure --prefix= --enable-shared

Set up LD_LIBRARY_PATH so that running "python" would find its shared lib
export LD_LIBRARY_PATH=${python-lib}:$LD_LIBRARY_PATH

Compile (apache) httpd Add DSO support
./configure --prefix= --enable--module=module_so



Compile mod_wsgi
./configure --with-apxs=$APACHE_HOME/bin/apxs --with-python=$PYTHON_BIN/python

Configure httpd.conf for mod_wsgi

LoadModule wsgi_module modules/mod_wsgi.so

WSGIScriptAlias /wsgi/ /u/local/www/wsgi-scripts/

WSGIApplicationGroup admin-scripts

Order allow,deny

Allow from all

WSGIProcessGroup example1.com

WSGIDaemonProcess example1.com processes=2 threads=15 display-name=example1.com

Install a sample python code under /u/local/www/wsgi-scripts/sniff

import sys

def application(environ, start_response):

status = '200 OK'

output = 'sys.version= %s\nsys.prefix=%s\n' % (sys.version, sys.prefix)

output += 'sys.path= %s\n' % (sys.path)

response_headers = [('Content-type', 'text/plain'),

('Content-Length', str(len(output)))]

start_response(status, response_headers)

return [output]

To install Nginx from source

./configure --prefix=/u/pandev/opt/nginx --without-http_rewrite_module

netstat + lsof = my friend

linux

1. "netstat -anpt | grep -i listen | grep 8888" will find out the process id is running on that port.
2. "/usr/sbin/lsof -p " find out what file and sockets it opens.
3. "lsof -P -n -i TCP" will list all the TCP sockets with unresolved (numeric) IP and port

Database connection pooling with firewall troubleshooting tip: Use Netcat

Database connection pooling is useful and common feature for performance. However there are some cavets.
Sometimes your sql query just got timed out, and you wonder why.
If you suspect the TCP connection between your application hosted in DMZ to the database (which is usually sits behind a firewall) got dropped by the firewall. It is very likely firewall has a policy to terminate any idle connection.

Let's go back to the basics to use Netcat.

Server(internal machine):
nc -v -l 9999

Client (DMZ machine):
nc 9999

Assuming port 9999 is allowd on the firewall.

Let's assume the firewall drops the connection after 30 minutes idle time, then the nc will be broken after 30 minutes.

Remote disk space monitor

The following is a poor man's disk space monitor script.
"ssh id@host df -h " is the core of the script. It relies on SSH RSA authentication so that the remote "df" can work like local.


#!/bin/ksh
homedir=`dirname $0`
RUN_DATE_LONG=$(date +'%Y%m%d')
LOGFILE=${homedir}/diskmon_log.$RUN_DATE_LONG

HOSTS="192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 "

alert=90
alert_email="admin@foo.com"
typeset -i email=0

function rdf
{
echo "$1 is sshing into $2 doing a df......." >> $LOGFILE
ssh $1@$2 df -hP | grep -vE '^Filesystem' | awk '{print $4 " " $5 " " $1 " " $2}' | while read line;
do
typeset -i used=$(echo $line | awk '{print $1}' | cut -d "%" -f 1)
if [[ $used -ge $alert ]]; then
email=1
echo $line >> $LOGFILE
fi
done
echo "ssh done......." >> $LOGFILE
}



for h in $HOSTS
do
rdf sshid $h
done


if [[ ${email} -eq 1 ]]; then
echo "alert email will be setnt" >> $LOGFILE
# put whatever mailer you use here
fi

Dtrace could be the secret weapon

I had this vision of setting up a high end consulting firm which only serves some hard problem such as performance. One of secret weapon is Dtrace.

DTrace for linux: http://www.crisp.demon.co.uk/tools.html

DTrace with Python: http://blogs.sun.com/levon/entry/python_and_dtrace_in_build

The power of rsync with hardlink

rsync -aHvz -e ssh /var/src /var/src-hardlinks
remote_id@192.168.1.1:/home/users/foo/

The above would copy the directory "src" and "src-hardlinks" (which holds all the hardlinks) recursively to 1921.68.1.1's folder.

find and cpio to back up hardlinks and archive

It can be used for backup with cpio to maintain the hardlinks, permission and modified timestamp.

find d1 d2 -type f -mtime +${period} | cpio -Bpdmv backup_dir


Archive the data to a single .cpio file.

http://bradthemad.org/tech/notes/cpio_directory.php