I have an architecture which is basically a queue with url addresses and some classes to process the content of those url addresses. At the moment the code works good, but it is slow to sequentially pull a url out of the queue, send it to the correspondent class, download the url content and finally process it.
It would be faster and make proper use of resources if for example it could read
n urls out of the queue and then shoot
n processes or threads to handle the downloading and processing.
I would appreciate if you could help me with these:
- What packages could be used to solve this problem ?
- What other approach can you think of ?