features
- website crawling functionality
- site indexing : unique site URLs can be stored
- load testing : scale up processes or reduce the crawl interval to simulate different loads
- user-agent : manage user-agent setting, using the default, varying from a list in docs/ua.txt, or one specified with the `--ua` argument
- _experimental_ `--gz` and `--robots` are supported experimentally, in this release they _will_ raise errors until error handling around these requests is improved
bugs
- KeyboardInterrupt events do not exit subprocesses cleanly, required forced killing of the parent process
- various formatting issus
- Queue.Empty events are occasionally raised unexpectedly