http://blog.devonlinux.net ICT solutions that work Tue, 02 Feb 2010 23:54:34 +0000 en hourly 1 http://wordpress.org/?v=3.0 http://blog.devonlinux.net/2010/01/12/simple-map-site/ http://blog.devonlinux.net/2010/01/12/simple-map-site/#comments Tue, 12 Jan 2010 12:14:50 +0000 neuromancer http://blog.devonlinux.net/?p=198 &1 |grep $1 |cut -d " " -f 4 |cut -d "?" -f 1 | sort | uniq > $2 rm -rf ${1#http://} [...]]]> A little shell script that I’ve used to generate the list of page composing a given site

#!/bin/bash
if [ "$#" -eq 2 ]
then
wget -erobots=off --mirror --delete-after --reject .jpg,.png,.gif,.swf,.css,.js,.txt,.pdf,.rtf,.odt,.doc $1 2>&1 |grep $1 |cut -d " " -f 4 |cut -d "?" -f 1 | sort | uniq > $2
rm -rf ${1#http://}
else
echo "Usage: ./scriptname http://testsite outputfile"
fi

Save with a desired file name and give it a chmod +x and then just launch from shell:
./scriptname http://testsite outputfile

]]> http://blog.devonlinux.net/2010/01/12/simple-map-site/feed/ 0