61ba870458
- Several new wordlist entries, courtesy of Glastopf Honeypot:
http://glastopf.org/index.php
- A tweak to path mapping detection logic to detect certain path mappings.
- Makefile now honors external LDFLAGS, CFLAGS.
- Some more documentation tweaks.
- PUT detection logic.
196 lines
8.7 KiB
Plaintext
196 lines
8.7 KiB
Plaintext
This directory contains four alternative, hand-picked Skipfish dictionaries.
|
|
|
|
Before you pick one, you should understand several basic concepts related to
|
|
dictionary management in this scanner, as this topic is of critical importance
|
|
to the quality of your scans.
|
|
|
|
-----------------------------
|
|
Dictionary management basics:
|
|
-----------------------------
|
|
|
|
1) Each dictionary may consist of a number of extensions, and a number of
|
|
"regular" keywords. Extensions are considered just a special subset of
|
|
the keyword list.
|
|
|
|
2) Use -W to specify the dictionary file to use. The dictionary may be
|
|
custom, but must conform to the following format:
|
|
|
|
type hits total_age last_age keyword
|
|
|
|
...where 'type' is either 'e' or 'w' (extension or wordlist); 'hits'
|
|
is the total number of times this keyword resulted in a non-404 hit
|
|
in all previous scans; 'total_age' is the number of scan cycles this
|
|
word is in the dictionary; 'last_age' is the number of scan cycles
|
|
since the last 'hit'; and 'keyword' is the actual keyword.
|
|
|
|
Do not duplicate extensions as keywords - if you already have 'html' as
|
|
an 'e' entry, there is no need to also create a 'w' one.
|
|
|
|
There must be no empty or malformed lines, comments in the wordlist
|
|
file. Extension keywords must have no leading dot (e.g., 'exe', not '.exe'),
|
|
and all keywords should be NOT url-encoded (e.g., 'Program Files', not
|
|
'Program%20Files'). No keyword should exceed 64 characters.
|
|
|
|
If you omit -W in the command line, 'skipfish.wl' is assumed. This
|
|
file does not exist by default; this is by design.
|
|
|
|
3) The scanner will automatically learn new keywords and extensions based on
|
|
any links discovered during the scan; and will also analyze pages and
|
|
extract words to use as keyword candidates.
|
|
|
|
A capped number of candidates is kept in memory (you can set the jar size
|
|
with the -G option) in FIFO mode, and are used for brute-force attacks.
|
|
When a particular candidate results in a non-404 hit, it is promoted to
|
|
the "real" dictionary; other candidates are discarded at the end of the
|
|
scan.
|
|
|
|
You can inhibit this auto-learning behavior by specifying -L in the
|
|
command line.
|
|
|
|
4) Keyword hit counts and age information will be updated at the end of the
|
|
scan. This can be prevented with -V.
|
|
|
|
5) Old dictionary entries with no hits for a specified number of scans can
|
|
be purged by specifying the -R <cnt> option.
|
|
|
|
----------------------------------------------
|
|
Dictionaries are used for the following tasks:
|
|
----------------------------------------------
|
|
|
|
1) When a new directory, or a file-like query or POST parameter is discovered,
|
|
the scanner attempts passing all possible <keyword> values to discover new
|
|
files, directories, etc.
|
|
|
|
2) The scanner also tests all possible <keyword>.<extension> pairs. Note that
|
|
this results in several orders of magnitude more requests, but is the only
|
|
way to discover files such as 'backup.tar.gz', 'database.csv', etc.
|
|
|
|
In some cases, you might want to inhibit this step. This can be achieved
|
|
with the -Y switch.
|
|
|
|
3) For any non-404 file or directory discovered by any other means, the scanner
|
|
also attempts all <node_filename>.<extension> combinations, to discover,
|
|
for example, entries such as 'index.php.old'. This behavior is independent
|
|
of the -Y option, since it is much less request-intensive.
|
|
|
|
----------------------
|
|
Supplied dictionaries:
|
|
----------------------
|
|
|
|
1) Empty dictionary (-).
|
|
|
|
Simply create an empty file, then load it via -W. If you use this option
|
|
in conjunction with -L, this essentially inhibits all brute-force testing,
|
|
and results in an orderly, link-based crawl.
|
|
|
|
If -L is not used, the crawler will still attempt brute-force, but only
|
|
based on the keywords and extensions discovered when crawling the site.
|
|
This means it will likely learn keywords such as 'index' or extensions
|
|
such as 'html' - but may never attempt probing for 'log', 'old', 'bak', etc.
|
|
|
|
Both these variants are very useful for lightweight scans, but are not
|
|
particularly exhaustive.
|
|
|
|
2) Extension-only dictionary (extensions-only.wl).
|
|
|
|
This dictionary contains about 90 common file extensions, and no other
|
|
keywords. It must be used in conjunction with -Y (otherwise, it will not
|
|
behave as expected).
|
|
|
|
This is often a better alternative to a null dictionary: the scanner will
|
|
still limit brute-force primarily to file names learned on the site, but
|
|
will know about extensions such as 'log' or 'old', and will test for them
|
|
accordingly.
|
|
|
|
3) Basic extensions dictionary (minimal.wl).
|
|
|
|
This dictionary contains about 25 extensions, focusing on common entries
|
|
most likely to spell trouble (.bak, .old, .conf, .zip, etc); and about 1,700
|
|
hand-picked keywords.
|
|
|
|
This is useful for quick assessments where no obscure technologies are used.
|
|
The principal scan cost is about 42,000 requests per each fuzzed directory.
|
|
|
|
Using it without -L is recommended, as the list of extensions does not
|
|
include standard framework-specific cases (.asp, .jsp, .php, etc), and
|
|
these are best learned on the fly.
|
|
|
|
** This dictionary is strongly recommended for your first experiments with
|
|
** skipfish, as it's reasonably lightweight.
|
|
|
|
You can also use this dictionary with -Y option enabled, approximating the
|
|
behavior of most other security scanners; in this case, it will send only
|
|
about 1,700 requests per directory, and will look for 25 secondary extensions
|
|
only on otherwise discovered resources.
|
|
|
|
3) Standard extensions dictionary (default.wl).
|
|
|
|
This dictionary contains about 60 common extensions, plus the same set of
|
|
1,700 keywords. The extensions cover most of the common, interesting web
|
|
resources.
|
|
|
|
This is a good starting point for assessments where scan times are not
|
|
a critical factor; the cost is about 100,000 requests per each fuzzed
|
|
directory.
|
|
|
|
In -Y mode, it behaves nearly identical to minimal.wl, but will test a
|
|
greater set of extensions on otherwise discovered resources at a relatively
|
|
minor expense.
|
|
|
|
4) Complete extensions dictionary (complete.wl).
|
|
|
|
Contains about 90 common extensions and 1,700 keywords. These extensions
|
|
cover a broader range of media types, including some less common programming
|
|
languages, image and video formats, etc.
|
|
|
|
Useful for comprehensive assessments, over 150,000 requests per each fuzzed
|
|
directory.
|
|
|
|
In -Y mode, this dictionary offers the best coverage of all three wordlists
|
|
at a relatively low cost.
|
|
|
|
Of course, you can customize these dictionaries as seen fit. It might be, for
|
|
example, a good idea to downgrade file extensions not likely to occur given
|
|
the technologies used by your target host to regular 'w' records.
|
|
|
|
Whichever option you choose, be sure to make a *copy* of this dictionary, and
|
|
load that copy, not the original, via -W. The specified file will be overwritten
|
|
with site-specific information unless -V used - and you probably want to keep
|
|
the original around.
|
|
|
|
----------------------------------
|
|
Bah, these dictionaries are small!
|
|
----------------------------------
|
|
|
|
Keep in mind that web crawling is not password guessing; it is exceedingly
|
|
unlikely for web servers to have directories or files named 'henceforth',
|
|
'abating', or 'witlessly'. Because of this, using 200,000+ entry English
|
|
wordlists, or similar data sets, is largely pointless.
|
|
|
|
More importantly, doing so often leads to reduced coverage or unacceptable
|
|
scan times; with a 200k wordlist and 80 extensions, trying all combinations
|
|
for a single directory would take 30-40 hours against a slow server; and even
|
|
with a fast one, at least 5 hours is to be expected.
|
|
|
|
DirBuster uses a unique approach that seems promising at first sight - to
|
|
base their wordlists on how often a particular keyword appeared in URLs seen on
|
|
the Internet. This is interesting, but comes with two gotchas:
|
|
|
|
- Keywords related to popular websites and brands are heavily
|
|
overrepresented; DirBuster wordlists have 'bbc_news_24', 'beebie_bunny',
|
|
and 'koalabrothers' near the top of their list, but it is pretty unlikely
|
|
these keywords would be of any use in real-world assessments of a typical
|
|
site, unless it happens to be BBC or Disney.
|
|
|
|
- Some of the most interesting security-related keywords are not commonly
|
|
indexed, and may appear, say, on no more than few dozen or few thousand
|
|
crawled websites in Google index. But, that does not make 'AggreSpy' or
|
|
'.ssh/authorized_keys' any less interesting - in fact, you might care
|
|
about them a whole lot more.
|
|
|
|
Bottom line is, tread carefully; poor wordlists are one of the reasons why some
|
|
web security scanners perform worse than expected. You will almost always be
|
|
better off narrowing down or selectively extending the supplied set (and
|
|
possibly contributing back your changes upstream!), than importing a giant
|
|
wordlist scored elsewhere.
|