a small and efficient HTTP crawler
external] [url ...]
crawl utility starts a depth-first traversal of the
web at the specified URLs. It stores all JPEG images that match the configured
The options are as follows:
- The verbosity level of
crawl in regards to
printing information about URL processing. The default is 1.
expression that all URLs that should be included in the traversal have to
expression that determines which URLs will be excluded from the
expression that all image URLs have to match in order to be stored on
expression that determines the images that will not be stored.
- Specifies the directory under which the images will be stored.
- Specifies the maximum depth of the traversal. A 0 means that only the URLs
specified on the command line will be retrieved. A -1 stands for unlimited
traversal and should be used with caution.
- Continues a traversal that was interrupted previosly. The remaining URLs
with be read from the file state.
- Specifies the time in seconds that needs to pass between successive access
of a single host. The parameter is a float. The default is five
- Specifies the agent string that will be included in all HTTP
- Specifies that the crawler should ignore the
- Specifies an external filter program that can refine which URLs are to be
included in the traversal. The filter program reads the URLs on
stdin and outputs a single character on
stdout. An output of
y’ indicates that the URL may be
n’ means that the URL
should be excluded.
The source code for existing web crawlers tend to be very
crawl is a very simple design with
pretty simple source code.
A configuration file can be used instead of the command line
arguments. The configuration file contains the MIME-type that is being used.
To download other objects besides images the MIME-type needs to be adjusted
accordingly. For more information, see
crawl -m 0 http://www.w3.org/
Searches for images in the index page of the web consortium
without following any other links.
This product includes software developed by Ericsson Radio Systems.
This product includes software developed by the University of
California, Berkeley and its contributors.
crawl utility has been developed by Niels Provos.