A lazy strategy for an FTP and HTTP mirroring system that optimizes use of resources.
Carmen Lima
MSc in Computer Science
Department of Computer Science
Faculty of Science, University of Porto
October 1999
Abstract
Ftp is one of the most used Internet services. Some of the ftp servers and
the files they store are very popular which leads to both overloading of
the servers and of the lines used to access them. In order to minimize
this problem many local administrators maintain local
mirror sites
of the more common ftp sites.
Mirror sites are ftp servers that maintain an exact copy of the files
from one or more ftp servers. This help reduce both long distance traffic
and download time for local users. There are however some inconvenients
with this approach. The copies are made on a periodical basis,
and so they are not always in tune with the originals.
There are however some problems with this strategy which may occur when
the mirrored site changes frequently and when only some subtrees of the
original site are accessed locally. By copying the entire sites frequently
one risks generating the unnecessary long distance traffic which one was
trying to avoid.
One more efficient solution is based on a lazy evaluation model. On that model the transfer of files is made on demand, if they don't already exist on the
local server. Users can access all files without knowing that some of them do
not exist on the server, but are automatically and concurrently transferred
when they are requested.
This kind of server allows the national network parameterization, promoting
connections to central servers and discouraging connections to other ftp servers.
This process can also be extended to the http protocol.
We can keep in the local server, a copy of the pages maintained by other http
servers.
