I have written a versatile web-crawler in Python.
Ruya http://ruya.sourceforge.net/
It features all aspects of a well-behaved, decent web-crawler for the Internet. Check out the website for detailed documentation.
I have written a versatile web-crawler in Python.
Ruya http://ruya.sourceforge.net/
It features all aspects of a well-behaved, decent web-crawler for the Internet. Check out the website for detailed documentation.