python twisted Throttling with Cooperator

Example 8: Throttling with Cooperator


from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.internet import defer, task

maxRun = 2

urls = [
'http://twistedmatrix.com',
'http://twistedsoftwarefoundation.org',
'http://yahoo.com',
'http://www.google.com',
]

def pageCallback(result):
print len(result)
return result

def doWork():
for url in urls:
d = getPage(url)
d.addCallback(pageCallback)
yield d

def finish(ign):
reactor.stop()

def test():
deferreds = []
coop = task.Cooperator()
work = doWork()
for i in xrange(maxRun):
d = coop.coiterate(work)
deferreds.append(d)
dl = defer.DeferredList(deferreds)
dl.addCallback(finish)

test()
reactor.run()

This is the last example for this post, and it's is probably the most arcane :-) This example is taken from JP's blog post from a couple years ago. Our observation in the previous example about the way that the deferreds were created in the for loop and how they were run is now our counter example. What if we want to limit when the deferreds are created? What if we're using deferred semaphore to create 1000 deferreds (but only running them 50 at a time), but running out of file descriptors? Cooperator to the rescue.

This one is going to require a little more explanation :-) Let's see if we can move through the justifications for the strangeness clearly:

1. We need the deferreds to be yielded so that the callback is not created until it's actually needed (as opposed to the situation in the deferred semaphore example where all the deferreds were created at once).
2. We need to call doWork before the for loop so that the generator is created outside the loop. thus making our way through the URLs (calling it inside the loop would give us all four URLs every iteration).
3. We removed the result-processing callback on the deferred list because coop.coiterate swallows our results; if we need to process, we have to do it with pageCallback.
4. We still use a deferred list as the means to determine when all the batches have finished.

This example could have been written much more concisely: the doWork function could have been left in test as a generator expression and test's for loop could have been a list comprehension. However, the point is to show very clearly what is going on.

I hope these examples were informative and provide some practical insight on working with deferreds in your Twisted projects :-)
Source

twisted task and schedule handler

from twisted.internet import task, reactor
import time
def timer():
print time.ctime()
loop = task.LoopingCall(timer)
loop.start(1, now=True)
reactor.run()

python urllib2 handle cookies

To handle python cookies in urllib2, it's already provide with new urllib2.

Here is a example to deal with cookies return from http server.
opener = urllib2.build_opener(HTTPCookieProcessor(),HTTPBasicAuthHandler(),HTTPDefaultErrorHandler( ))
opener.addheaders = [('User-agent', 'Mozilla/5.0'),("Content-type", "text/xml; charset=\"UTF-8\"")]
opener.open(url)
urllib2.install_opener(opener)
urllib2.open(new_url)

We have register the opener into urllib2, afterward; we can use urllib2.open() without worry about handling cookies return.

python conversion unicode to ascii

unicodeStr = something
ascii = unicodeStr.encode('ascii','ignore')

Usage Python twisted

For new page usage:
from twisted.internet import reactor
from twisted.internet.defer import DeferredList

data = "some data for http body"

def responseData(result, factory)
print "Content Length=", len(result)
print "Cookies =", factory.cookies

def failurePage(error)
print 'msg=', error.getErrorMessage()
print 'err=', error

def finished(ign):
reactor.stop()

d1, factory = getNewPage(url, postdata=data)

dl = DeferredList([d1])
dl.addCallback(responseData, factory)
dl.addErrback(failurePage)
dl.addCallback(finished)

reactor.run()

Đức Đạt Lai Lạt Ma viếng thăm Hoa Kỳ

Đức Đạt Lai Lạt Ma viếng thăm Hoa Kỳ
Đăng ngày: 19-02-2010 lúc 5:26 PM Font size:



Đức Đạt Lai Lạt Ma đã đến Washington hôm thứ Tư, trước một cuộc họp với Tổng thống Hoa Kỳ Barack Obama.

Chế độ cộng sản Trung Quốc đã thúc giục chính quyền của tổng thống Obama hủy bỏ cuộc họp. Họ coi Đức Đạt Lai Lạt Ma như một phần tử ly khai.

Lưu tâm tới sự nhạy cảm của chế độ Trung Quốc, ông Obama đã trì hoãn cuộc họp với Đức Đạt Lai Lạt Ma cho đến sau khi ông gặp những người lãnh đạo Trung Quốc trước trong chuyến công du Châu Á tháng 11 năm ngoái.

Như một cử chỉ nhượng bộ nữa trước Trung Quốc, ông Obama sẽ không tiếp Đức Đạt Lai Lạt Ma tại Phòng Bầu dục của Nhà trắng. Thay vào đó, ông sẽ đón tiếp Ngài ở Phòng Bản đồ, nơi thường được dùng cho các cuộc gặp gỡ mang tính xã hội.

Tin tức Tân Đường Nhân, ngày 18 tháng 02 năm 2010.

Tags : Tin nổi bật
Chuyên mục: Tin thế giới
from: vietsoh.com

twisted framework

Modified getPage () to return a factory and deferred, so that we when extract the return cookies.


from twisted.web import client
def getNewPage(url, contextFactory=None, *args, **kwargs):
"""
Download a web page as a string and return clientfactory.
Download a page. Return a deferred, which will callback with a
page (as a string) or errback with a description of the error.

See HTTPClientFactory to see what extra args can be passed.
"""
factory = client._makeGetterFactory(url,client.HTTPClientFactory, contextFactory=contextFactory, *args, **kwargs)
return factory.deferred, factory