Intelligent download manager for the ports collection
Project description
Ports collection, as of now, simply uses fetch(1) to download distfiles. There are a few drawback to this approach:
- Duplicate downloads
- Picking a mirror to download is a problem
- Cannot cap download speed or number of connections
The requirements are clearly put down on the ideas page at IdeasPage/IDMS
Implementation
Client:
The client will be a replacement of the current "fetch" program used by the ports collection. It will be used in the exact same way as fetch, but instead of actually fetching the files, it requests the Download Manager Service to download the files. It will be blocking on the service till the download is complete. It can frequently request the service for status updates.
Service:
The service assigns a "worker" thread to handle each request. If the request is for a file which is already being downloaded, the same download will be used to serve both requests. The service will also maintain a record of the download speeds from each mirror site. This record will be used to assign priority to the mirror site and the workers attempt to get a requested file from a site with higher priority. The service also updates any client with the status of the download upon request.
Deliverables
1. DMS client
2. DMS service (basic)
3. DMS service (improved - status updates, duplicate requests, checksum verification)
4. Site profiling
5. Job migration
6. Command line status dump tool
Schedule
(Jun 17 - Jun 23) DMS Client
(Jun 24 - July 7) DMS Service (Basic)
(July 8 - July 21) DMS Service (Improved)
(July 22 - July 28) Buffer
(July 29 - Aug 11) Site profiling
(Aug 12 - Aug 18) Buffer
(Aug 19 - Sep 1) Job Migration
(Sep 2 - Sep 8) Buffer
(Sep 9 - Sep 15) Command line dump tool
(Sep 16 - Sep 22) Buffer