Resolve around File Descriptor limited on Reactor Select (selector)

I've been running into problem file descriptor run out using the default reactor. By default, twisted got hardcode the file descriptor to be 1024 (MAX). If you want to change the default MAX need to modify code and recompile. Or we can use the pollreactor to increase several thousand more.
>>
from twisted.internet import pollreactor
pollreactor.install()
<<
usage as normal:
from twisted.internet import reactor

Even if you change the ulimit on filesystem, it doesn't change with the library. This is something hardcode inside twisted lib.

Creating a self-signed cert with one command

Creating a self-signed cert with one command

The following command will generate a new key and create a certificate all in one line suitable for use by Apache or any other SSL tool.

openssl req -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout www.example.com.pem -out www.example.com.pem

You can then use the file above in apache with the following two lines

SSLEngine On
SSLCertificateFile www.example.com.pem

python twisted interrupt signal handler

from twisted.internet import reactor
import signal
def shutdown(*f):
    sig, ob = f
    if sig == signal.SIGINT: reactor.stop() #shutdown the reactor if Ctrl-C enter.

signal.signal(signal.SIGINT, shutdown)
reator.run(installSignalHandlers=0)

twisted.web2 auto reconnecting browser

Making use web2 Broker Class from http://code.google.com/p/twisted-web2-client

class Browser:
    #Init Channel
    #Authenticate Channel
    #Send Channel
    #Recieve Channel
    def __init__(self, timeout=180 ):
        self.timeout = timeout
        self.conn = Broker()
        self.headers = {}
        self.response_headers = None

    def parse_response_headers(self, headers):
        self.response_headers = headers
        self.raw_headers = headers._raw_headers
        if headers.hasHeader('set-cookie'):
            parseSetCookies = http_headers.parseSetCookie(headers.getRawHeaders('set-cookie'))
            self.headers['Cookie']= parseSetCookies

    def getResponseBody(self, resp):
        d = defer.Deferred()
        self.parse_response_headers(resp.headers)
        stack = []
        stream_mod.readStream(resp.stream, stack.append).addCallback(lambda x: ''.join(stack)).addCallbacks(d.callback, d.errback)
        return d

    def connect(self, uri, postdata=None, method='GET', *args, **kwargs):
        d = self.conn.ask(uri, method, self.headers, postdata,timeout=self.timeout, *args, **kwargs).addCallback(self.getResponseBody)
        return d

httpclient for twisted web2

Here is excellent tutorial for twisted web2 client.
http://code.google.com/p/twisted-web2-client/

Thanks to the author, I've spend quite sometime to google, but no luck!

install and config weblizer on ubuntu

http://fontignie.blogspot.com/2006/04/install-and-configure-webalizer-on.html

Install and configure webalizer on Ubuntu

1. install webalizer
   

   sudo apt-get install webalizer
   

   If webalizer is not found, you have to add sources in /etc/apt/sources.list: can comment the universe sources.
   


2. Enable the apache2 hostname resolution: go into /etc/apache2/apache2.conf: Change
   

   HostnameLookups Off
   

   into
   

   HostnameLookups On
   



3. By default webalizer is not well configured: it does not check in the good log. In /etc/webalizer.conf: Change
   

   
   LogFile         /var/log/apache2/access.log.1
   

   to
   

   LogFile         /var/log/apache2/access.log
   



4. Test webalizer:
   

   
   sudo webalizer
   

   
   If you get a warning like warning: Truncating ...
   It is because you did not put the hostnameLookup. If afterward, you get this message, it can be because you are attacked by a virus...
   


5. Run webalizer as a cronjob. This has to be run as root: Edit the root cronjobs by running the command:
   

   sudo crontab -e
   

   and add the line:
   

   0 * * * * webalizer
   

   With that line, every hour webalizer is run
   

install and config weblizer on ubuntu

http://fontignie.blogspot.com/2006/04/install-and-configure-webalizer-on.html

Install and configure webalizer on Ubuntu

1. install webalizer
   

   sudo apt-get install webalizer
   

   If webalizer is not found, you have to add sources in /etc/apt/sources.list: can comment the universe sources.
   


2. Enable the apache2 hostname resolution: go into /etc/apache2/apache2.conf: Change
   

   HostnameLookups Off
   

   into
   

   HostnameLookups On
   



3. By default webalizer is not well configured: it does not check in the good log. In /etc/webalizer.conf: Change
   

   
   LogFile         /var/log/apache2/access.log.1
   

   to
   

   LogFile         /var/log/apache2/access.log
   



4. Test webalizer:
   

   
   sudo webalizer
   

   
   If you get a warning like warning: Truncating ...
   It is because you did not put the hostnameLookup. If afterward, you get this message, it can be because you are attacked by a virus...
   


5. Run webalizer as a cronjob. This has to be run as root: Edit the root cronjobs by running the command:
   

   sudo crontab -e
   

   and add the line:
   

   0 * * * * webalizer
   

   With that line, every hour webalizer is run
   

squid log analysis

http://www.safesquid.com/html/viewtopic.php?t=2395
http://cord.de/tools/squid/calamaris/
http://www.squid-cache.org/Scripts/

Good Learning material for twisted and asynchronous programming

http://krondo.com/blog/?p=1209

Excellent documentation

Configure network ubuntu/redhat

Redhat:
manually eth0: edit /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:e0:81:26:1e:9e
ONBOOT=no
BOOTPROTO=none
TYPE=Ethernet
NETMASK=255.255.255.0
IPADDR=192.168.1.214
GATEWAY=192.168.1.1

Using Commands: None-Gui command (system-config-network-tui)
GUI command: system-config-network &
Reference : http://www.cyberciti.biz/faq/rhel-centos-fedoracore-linux-network-card-configuration/

Ubuntu:
edit file: /etc/network/interfaces
iface eth0 inet static
address 192.168.1.100
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.254

Restart the network.
sudo /etc/init.d/networking restart

Reference: http://www.cyberciti.biz/tips/howto-ubuntu-linux-convert-dhcp-network-configuration-to-static-ip-configuration.html

created a defered function

def Normal(args):
def startMessages():
for key in sendMSGSequence:
self.deferQueue.put(key)
sleep(1) # simulate like a real GUI serial Connection.
#print key
reactor.callLater(1, startMessages)

The inner function will execute after 1 second callback from the reactor function. The Reactor is continue is execution with being block.