Tuesday, July 2, 2013

Burp Extension: Directory and File Listing Parser - Burp Site Map Importer

Click Here to Download Source Code

Penetration testers, rejoice! While conducting application penetration tests it’s sometimes necessary to request specific information from the application owner or client. As a pen tester it can be extremely beneficial to perform a test with a full directory and file listing of the application, which sometimes can be difficult to acquire.

So let’s assume all clients are perfect and provide a full directory and file listing of their application (funny, I know) but what do we do with it? My process usually involves manually looking over everything trying to find keywords which jump out… I just might want to take a look at adminpassword.txt. Depending on the size of the application I may attempt to reach every file but usually this is not an efficient use of time. I wanted to create a quick and easy process for dealing with directory and file listings so I created a Burp Suite extension which will do a lot of the work for me.

My Burp extension contains two main features. The first feature is the ability to parse the listing file and generate a list of valid URLs to request each resource. The second feature is generating a request for each URL and importing the valid request/response pairs into Burp’s Target Site Map. Why is having a full site map helpful? We now have the ability to see the entire structure of the application, search within all valid responses, conduct manual testing or an active scan on ALL accessible resources, and much more. The process flow looks like this:

The Burp extension is written in python so a standalone jython jar will be needed to run it: Currently the extension is only tested and working with jython-standalone 2.5.3

After loading the extension you will have an option in the context menu to “Import Directory Listing”:

A GUI will appear for the extension. Fields such as hostname, SSL, and port will automatically populate depending on the request or response the menu option was originally invoked from. Cookies will also be displayed and used in any requests the extension makes. This feature makes it easy to compare site maps of two application user roles (based on varying session information such as cookies) to determine if each role has the correct access.

In this example I have selected the “Import Directory Listing” menu option on the DVWA web application which is running on my local machine. Now we must fill out all options on the left side of the GUI including hostname, full directory path (windows only, but CAN be used to modify URLs from a linux listing type) which specifies where the root of the web application sits, SSL, port, file listing type, and path to listing file.

On a Windows XP machine, I used the ‘dir /s’ command in cmd.exe which displays all files from the current directory and all sub directories. If the application is sitting on a Windows platform this is a very common command used for directory and file listings. A partial output of a directory and file listing for the DVWA web application (selected in the above image as C:\dvwa-listing.txt) looks like this:

dir /s:
 Volume in drive C has no label.
 Volume Serial Number is 5033-AA99

 Directory of C:\xampp\htdocs\dvwa

09/08/2010  09:50 PM    <DIR>          .
09/08/2010  09:50 PM    <DIR>          ..
09/08/2010  09:49 PM               497 .htaccess
08/26/2010  11:15 AM             2,792 about.php
06/06/2010  07:55 PM             5,066 CHANGELOG.txt
09/08/2010  09:50 PM    <DIR>          config
03/16/2010  12:56 AM            33,107 COPYING.txt
09/08/2010  09:50 PM    <DIR>          docs
09/08/2010  09:50 PM    <DIR>          dvwa
03/16/2010  12:56 AM               883 ids_log.php
06/06/2010  07:52 PM             1,878 index.php
03/16/2010  12:56 AM             1,761 instructions.php
08/26/2010  11:18 AM             2,580 login.php
03/16/2010  12:56 AM             2,738 security.php
06/06/2010  10:58 PM             1,350 setup.php
09/08/2010  09:50 PM    <DIR>          vulnerabilities
              16 File(s)         59,772 bytes

 Directory of C:\xampp\htdocs\dvwa\config

09/08/2010  09:50 PM    <DIR>          .
09/08/2010  09:50 PM    <DIR>          ..
08/26/2010  10:32 AM               576 config.inc.php
08/26/2010  11:06 AM               576 config.inc.php~
               2 File(s)          1,152 bytes

 Directory of C:\xampp\htdocs\dvwa\docs

09/08/2010  09:50 PM    <DIR>          .
09/08/2010  09:50 PM    <DIR>          ..
08/26/2010  10:32 AM           526,043 DVWA-Documentation.pdf
               1 File(s)        526,043 bytes

The parser is also capable of parsing directory listing files from various linux commands such as ‘ls -lR’ and ‘ls –R’. Examples of the format are as follows:

ls -lR:
total 124
-rw-rw-r--  1 user user  2792 Aug 26  2010 about.php
-rw-rw-r--  1 user user  5066 Jun  6  2010 CHANGELOG.txt
drwxrwxr-x  2 user user  4096 Jul  1 16:52 config
-rw-rw-r--  1 user user 33107 Mar 16  2010 COPYING.txt
drwxrwxr-x  2 user user  4096 Jul  1 16:52 docs
drwxrwxr-x  6 user user  4096 Jul  1 16:52 dvwa
-rw-rw-r--  1 user user   883 Mar 16  2010 ids_log.php
-rw-rw-r--  1 user user  1878 Jun  6  2010 index.php
-rw-rw-r--  1 user user  1761 Mar 16  2010 instructions.php
-rw-rw-r--  1 user user  2580 Aug 26  2010 login.php
-rw-rw-r--  1 user user   413 Mar 16  2010 logout.php
-rw-rw-r--  1 user user  2738 Mar 16  2010 security.php
-rw-rw-r--  1 user user  1350 Jun  6  2010 setup.php
drwxrwxr-x 11 user user  4096 Jul  1 16:52 vulnerabilities

total 8
-rw-rw-r-- 1 user user 576 Aug 26  2010 config.inc.php
-rw-rw-r-- 1 user user 576 Aug 26  2010 config.inc.php~

total 516
-rw-rw-r-- 1 user user 526043 Aug 26  2010 DVWA-Documentation.pdf

total 16
drwxrwxr-x 2 user user 4096 Jul  1 16:52 css
drwxrwxr-x 2 user user 4096 Jul  1 16:52 images
drwxrwxr-x 3 user user 4096 Jul  1 16:52 includes
drwxrwxr-x 2 user user 4096 Jul  1 16:52 js
ls -R:




Now that we have a directory listing for our application, let’s parse out all of the directories and files and create valid URLs. Click the “Generate URL List” button to start the parsing. Tip: If the generated URLs don’t look correct you can modify the fields on the left of the GUI and regenerate the list or copy the list and use your own text editor to make changes. (Please e-mail SmeegeSec@gmail.com with any parsing issues or suggestions)

A text area is populated with the URLs and the total count of directories and files processed.

At this point we have a few options. We can take our list of URLs and use them in Burp’s Intruder. To do this it would be very easy, all that needs to be done is remove the protocol, hostname, and port from each URL within a text editor. From there we take the path of each resource as a payload in a GET request in Intruder. We could then look to see which resources we are able to reach by analyzing status codes and content length. A second option is built into the extension via the “Import URL List to Burp Site Map” button. This button makes a request to each URL in the list (with cookie information, if it was found) and if a valid response is returned, will add the request and response to Burp’s Site Map. Requests with keywords such as logout, logoff, etc. are skipped to avoid ending sessions. The import to site map functionality was one of the main features I wanted to implement.

Warning: Actual requests are being made. Remove any resources you don't want being made, such as delete_database.php!! Regex to remove resources will be added in updates.

Done! A message dialog tells the user how many URLs were valid and imported into the site map. In the above image you can see a full site map and proxy history which was not found by spidering or brute forcing directories/files, but rather a simple directory and file listing of the application. With a full site map we are now ahead of the game. If you have multiple testers testing an application you can save the state in Burp and distribute it to save time, almost completely bypassing the discovery phase.

Note: So far the parsing does not consider virtual directories or different URL mappings from different web frameworks. Future updates may include parsing of mapping files such as ASP.NET’s web.config and Java’s web.xml.

Tip: Running a plugin multiple times or multiple plugins at a time may require increased PermGen, an example to increase the max when launching Burp would be:

java -XX:MaxPermSize=1G -jar burp.jar

Also, please provide feedback if you use this extension. With many different output formats for directory and file listings it can be difficult to write a dynamic parser which works for every format. If you have a listing file which is not being properly parsed please contact me so I can include it in an update. Thanks!

Click Here to Download Source Code


  1. Nice work! I'm anxious to try this out.

  2. seems nice! But why not doing the directory listing from Jython? It would be cross-platform + no parsing needed

    1. I wanted to accept multiple directory listing types because web apps sit on all sorts of different machines and a listing from Jython is not always the easiest to get from a customer when pen testing their application. Sometimes it's just easier for them to open up cmd.exe and run a 'dir /s' command or something similar on their respective OS.

  3. The python script dont work.

    Traceback (most recent call last):
    File "C:\Users\XXX\Desktop\Site Map Importer.py", line 19, in
    import os, shlex
    ImportError: No module named os

    at org.python.core.Py.ImportError(Py.java:296)
    at org.python.core.imp.import_first(imp.java:774)
    at org.python.core.imp.import_module_level(imp.java:856)
    at org.python.core.imp.importName(imp.java:936)
    at org.python.core.ImportFunction.__call__(__builtin__.java:1278)
    at org.python.core.PyObject.__call__(PyObject.java:373)
    at org.python.core.__builtin__.__import__(__builtin__.java:1231)
    at org.python.core.imp.importOne(imp.java:955)
    at org.python.pycode._pyx1.f$0(C:\Users\Able001\Desktop\Site Map Importer.py:267)
    at org.python.pycode._pyx1.call_function(C:\Users\Able001\Desktop\Site Map Importer.py)
    at org.python.core.PyTableCode.call(PyTableCode.java:165)
    at org.python.core.PyCode.call(PyCode.java:18)
    at org.python.core.Py.runCode(Py.java:1302)
    at org.python.core.__builtin__.execfile_flags(__builtin__.java:518)
    at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:225)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at burp.chd.(Unknown Source)
    at burp.hwc.a(Unknown Source)
    at burp.gud.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

    1. Hey Victor, how are you loading the extension? The import error seems like it would come from not running the extension properly. It should only be used through Burp's extension tab (Burp Suite Professional) with a standalone Jython jar loaded in the Extender>Options tab (this is the one I used: http://search.maven.org/remotecontent?filepath=org/python/jython-standalone/2.5.3/jython-standalone-2.5.3.jar). Let me know if you have any further issues.

  4. Directory and file listing plugin won't load in Burp Pro 1.5.14 It errors out with a lot of Python errors. I'm using jython-stanalone version 2.7-b1

  5. I fixed above problem by using an older version of Standalone Jython ie 2.5.3

    1. Thanks a lot for the feedback, I'll be looking into this and try to get it working with all jython versions.

  6. very nice tool, i do this generally with using notepad++ and some regex or tool named inspath :). You should also add web.xml and web.config parsing support. Also importing urls from google would be nice feature. Analyzing javascripts for finding endpoint applications and urls also should be considired as new features :)


    1. Hey thanks for your input, the inspath tool looks interesting. When I have time I would definitely like to add more parsing support. Care to elaborate on importing urls from google? Sounds like it could almost warrant its own extension.

  7. Thanks a lot for such a great piece of code! I really like it! This addon is really really cool!