Simple collection of crawlers
# 1. I only made a function to collect URLs. If you need to store the data into the database, you can write the processing code in the parseData function.
# 2. Requires sqlite3 or pysqlite support
# 3. Can run on DreamHost.com space
# 4. User-Agent can be modified to pretend to be a search engine spider
# 5. You can set the pause time and control the collection speed.
Expand
