Crawling

With Burp

Add a domain to the target scope.

Then set up filtering to only look at the scope:

Next Steps:

Then go to the spider options tab and add whatever settings you want. Note the form submission part. This is where we can determine what we want to enter into various submission areas, set up cookies, or we can set up actions to take on various http codes. We can also have it prompt us for questions.

Finally hit enable then request the page you want to start the spider from. Output of the spider will be in the target->sitemap area