Subject: Re: see files fetched repeatedly recently
Newsgroups: gmane.network.wwwoffle.user
Date: 2003-06-30 08:32:43 GMT (4 years, 46 weeks and 7 hours ago)
Dan Jacobson <jidanni@...> writes: > To see what files have been fetched repeatedly recently, try > cd /var/cache/wwwoffle && find *time* -name U\*|xargs sed :|sort|uniq -c|sort -nr > In particular, some sites you visit often might have broken graphics > that get refetched each visit, even though unchanged, or something. I have taken the opportunity to examine the command that you propose and have added to it. It now provides also the name of the file that contains the data for the URL. This is useful for the second script which examines all of the versions of the cached file and prints out the duplicated headers. The point of looking at the cached headers is to see why the file was fetched more than once. The sort of things that might be seen are: Different 'Expires' headers for recent dates WWWOFFLE is re-fetching the page because the headers say that the page has expired. You can stop WWWOFFLE doing this by setting 'request-expired = no' in the OnlineOptions section of the wwwoffle.conf file. A 'Cache-Control: no-cache' or 'Pragma: no-cache' header WWWOFFLE is re-fetching the page because it contains a header that says that it is not to be cached. You can stop WWWOFFLE doing this by setting 'reqeust-no-cache = no' in the OnlineOptions section of the wwwoffle.conf file. A 'Cache-Control: max-age=...' header and a 'Date' header The specified age in seconds needs to be added to the date, this sets an expiry time for the page and is treated the same way as an 'Expires' header. Different 'ETag' headers WWWOFFLE is re-fetching the page because when it asks the server if the page has changed (a different Etag) the server says that it has (even though it may not have changed). Disabling this will become an option in the next version of WWWOFFLE. The scripts are below along with examples of their use. -------------------- find-duplicate-urls.sh -------------------- #!/bin/sh cd /var/spool/wwwoffle find *time* -name U\* -exec sh -c "echo -n {} ; echo -n ' ' ; cat {} ; echo" \; |\ sed 's%[a-z0-9]*/U%D%g' |\ sort -k 2 |\ uniq -c -f 1 |\ sort -------------------- find-duplicate-urls.sh -------------------- For example: $ ./find-duplicate-urls.sh ... 2 DikktDk-E6-RJzjBO0gLYTw http://images.slashdot.org/topics/topicgamesportable.gif ... -------------------- find-duplicate-headers.sh -------------------- #!/bin/sh if [ "$1" = "" ]; then echo Specify a filename exit fi cd /var/spool/wwwoffle echo *time*/$1 |\ xargs -n 1 awk '/^\r$/ {finished=1} {if(!finished) print}' |\ sort |\ uniq -c -------------------- find-duplicate-headers.sh -------------------- For example: $ ./find-duplicate-headers.sh DikktDk-E6-RJzjBO0gLYTw 2 Accept-Ranges: bytes 2 Cache-Control: max-age=604800 2 Connection: close 2 Content-Length: 1278 2 Content-Type: image/gif 1 Date: Fri, 27 Jun 2003 05:16:13 GMT 1 Date: Sat, 28 Jun 2003 05:22:10 GMT 1 ETag: "1b408f-4fe-b7d3a080" 1 ETag: "2de1c-4fe-b7d3a080" 1 Expires: Fri, 04 Jul 2003 05:16:13 GMT 1 Expires: Sat, 05 Jul 2003 05:22:10 GMT 2 HTTP/1.1 200 OK 2 Last-Modified: Fri, 27 Jun 2003 03:41:38 GMT 2 Server: Apache/2.0.46 (Unix) mod_ssl/2.0.46 OpenSSL/0.9.6c This has probably been refetched because of the Etag header. -- Andrew. ---------------------------------------------------------------------- Andrew M. Bishop amb@... http://www.gedanken.demon.co.uk/ WWWOFFLE users page: http://www.gedanken.demon.co.uk/wwwoffle/version-2.7/user.html