python for loop, iterator, iterable, generator

For Loop
 When Python executes the for loop, it first invokes the __iter__() method of the container to get the iterator of the container. It then repeatedly calls the next() method (__next__() method in Python 3.x) of the iterator until the iterator raises a StopIteration exception. Once the exception is raised, the for loop ends.

Generators can be thought of as resumable functions

cProfile pstats

pstats

OS X Python set up

homebrew

ruby <(curl -fsSkL raw.github.com/mxcl/homebrew/go)

install Python from http://www.python.org/getit/mac/

install distribute (easy_install needs it)

wget http://python-distribute.org/distribute_setup.py
$ python distribute_setup.py

install pip,virtualenv 

easy_install pip

pip install virtualenv

pip install virtualenvwrapper

vim .bash_profile

source /virtualenvwrapper.sh

concurrency model with Actor

Actor model everything is a Actor, natively concurrent.

Support in programming language 

Erlang: language level. Erlang is the best platform for concurrency and distributed application so far.

Python: library level.

• eventlet 
• why gevent
• Gevent: good, bad and ugly
• gevent tutorial

Java: library level. Akka

Set up Python for Android development

Python Android
adb cmd example

• Install android-sdk: 
• tools:android
• platform-tools:adb
• Create an AVD ("android avd") test1. 
• "emulator avd test1"
• sudo apt-get install ia32-libs
• While the above is running, "platform-tools/adb install sl4a_r4.apk"
• While the above is running, "platform-tools/adb install PythonForAndroid_r4.apk"
• Use adb for pushing python code for Android testing. 

python eco system setup

python ecosystem


install necessary dev tools
$ sudo apt-get install python-pip

$ sudo pip install --upgrade pip 
$ sudo pip install --upgrade virtualenv 

$ sudo pip install --upgrade virtualenvwrapper  

$ sudo apt-get install git-core mercurial subversion

$ sudo apt-get install python-dev build-essential 

install python 

$ sudo apt-get intsall python

$ sudo apt-get intsall python3

 

management of virtualenv

$ mkvirtualenv my_project_venv
$ workon my_project_venv
$ deactivate
$ rmvirtualenv my_project_venv

 

create python3 virtualenv

mkvirtualenv -p /usr/bin/python py3

workon py3

pip install ipython3

Python study notes

python overview

• abstraction: 
• type: dynamic binding, strong type(for catching bugs)
• object is first class:objects can be freely passed around, inspected, and placed in various data structures (e.g., lists or dictionaries) at run-time.
• organizing code: package, module, class,method, function. 
• flow control: if/else,while, iterator,generator, coroutine
• error handling: try/except/finally

History of Python builtin types   

• Influenced by ABC

python user defined class design

• runtime data structure of instances and class, inheritance
• instance dict, class dict
• why "self" is needed
• class implementation requires no special syntax: just a collection of functions (methods), variables (class variables), computed attributed(properties).  
• new style classes, 
• __new__(cls,*args,**kwargs) is used for change value for subclassing immutable type  
• Attributes,Properties and Descriptors 
• History of descriptor,staticmethod, classmethod, property
• metaclass explained: metaclass 

• type(name, bases, dict) Return a new type object. This is essentially a dynamic form of the class statement. Python will look for __metaclass__ in the class definition. If it finds it, if will use it to create the object class Foo. If it doesn't, it will use type to create the class.

   

object is first class

• how to make method first class? via bound and unbound method

every statement is treated the same

• benefit: share the same grammar rule, and hence the compiler could use the same byte code generation function for all of them.

exception design

• early days implemented as string token (from Modula-3) 
• user defined class as replacement

cpython source code study note

http://www.python.org/dev/peps/pep-0339/

http://eli.thegreenplace.net/articles/

Python compiler

Python 2.7.2 source code

• main():Modules/python.c 
• Py_Main():Modules/main.c
• Py_Initialize(): Python/pythonrun.c
• PyRun_SimpleFileExFlags
• PyRun_FileExFlags
• PyParser_ASTFromFile returns AST module
• Parser build CST: PyParser_ParseFileFlagsEx: Parser/parsetok.c
• Transform CST to AST: PyAST_FromNode(): Python/ast.c
• run_mod(mod) 
• PyAST_Compile(mod) compile AST into byte code
• PyEval_EvalCode runs the byte code
• create first interpreter PyInterperterState_New()
• create first thread PyThreadState_New()
• initialize __builtin__ module: _PyBuiltin_Init()
• initialize sys module: _PySys_init()

Packet crafting and sniffing with Python extensions

• libpcap: sudo apt-get install libpcap-dev

• download and install pypcap, dpkt, dnet
• pypcap 's installation is tricky: download the pyrex and recompile the pcap.pyx file will work. Furthermore, use Makefile instead of setup.py seems fixed issues.

python vim set up

http://wiki.python.org/moin/Vim

cx_Oracle

Follow http://mostperfect.net/blog/2010/07/28/installing-cx_oracle-on-windows/,

A looking at cx_Oracle.pyd with dependency walker (from MS) will show it depends on OCI.dll.
As a result, we need to install Oracle Instant Client from here

http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html

a smtp proxy server

http://code.activestate.com/recipes/577260-multi-threaded-smtp-proxy/

is useful for helping me to test my html email via outlook. This is due to the real MTA that can deliver to outlook is IP based, and my workstation is not allowed but as long as I run the server on the allowed IP, it works.

range(a..z): python generator expression for

In order to generate a..z with generator expression.

import string

for filter in ('uid='+x+'*' for x in string.lowercase):

print(filter)

Generator Expression and List Comprehension

Generator http://en.wikipedia.org/wiki/Generator_(computer_science)

Generator Expression and List Comprehension

http://python-history.blogspot.com/2010/06/from-list-comprehensions-to-generator.html

How to build mod_wsgi,web.py with httpd and python from source code

It is best to compile everything from source code.

Version usedPython: 2.7 (with shared library)httpd 2.2.17 (with shared DSO support)WSGI (integrating with httpd): mod_wsgi 3.3
Web framework: web.py 0.34

Compile Python for shared lib

./configure --prefix= --enable-shared

Set up LD_LIBRARY_PATH so that running "python" would find its shared lib
export LD_LIBRARY_PATH=${python-lib}:$LD_LIBRARY_PATH

Compile (apache) httpd Add DSO support
./configure --prefix= --enable--module=module_so



Compile mod_wsgi
./configure --with-apxs=$APACHE_HOME/bin/apxs --with-python=$PYTHON_BIN/python

Configure httpd.conf for mod_wsgi

LoadModule wsgi_module modules/mod_wsgi.so

WSGIScriptAlias /wsgi/ /u/local/www/wsgi-scripts/

WSGIApplicationGroup admin-scripts

Order allow,deny

Allow from all

WSGIProcessGroup example1.com

WSGIDaemonProcess example1.com processes=2 threads=15 display-name=example1.com

Install a sample python code under /u/local/www/wsgi-scripts/sniff

import sys

def application(environ, start_response):

status = '200 OK'

output = 'sys.version= %s\nsys.prefix=%s\n' % (sys.version, sys.prefix)

output += 'sys.path= %s\n' % (sys.path)

response_headers = [('Content-type', 'text/plain'),

('Content-Length', str(len(output)))]

start_response(status, response_headers)

return [output]