Update README to explain the behavior of the recursive scans

README.md

@@ -90,6 +90,8 @@ By default `rwalk` will use a recursive-like scan. You can change the depth of t
 ```bash
 rwalk https://example.com wordlist.txt -d 3
 ```
+
+The recursive mode only scans urls [identified as directories](src/runner/filters.rs#L290). If you are not happy with the default behavior, you can use the `--force-recursion` (`--fr`) flag to force the recursion on all found urls.
 <!-- omit in toc -->
 #### Classic scan
 
@@ -436,8 +438,11 @@ Please take these results with a grain of salt.
 
 ## Contributing
 
+*Contributions are welcome! I am always looking for new ideas and improvements.*
+
 If you want to contribute to rwalk, please read the [CONTRIBUTING.md](CONTRIBUTING.md) file.
 
+
 ## License
 
 Licensed under the [MIT License](LICENSE).

src/runner/filters.rs

@@ -318,7 +318,12 @@ pub fn is_directory(response: &reqwest::Response) -> bool {
                 return false;
             }
         }
-    } else if response.status().is_success() || matches!(response.status(), StatusCode::FORBIDDEN) {
+    } else if response.status().is_success()
+        || matches!(
+            response.status(),
+            StatusCode::FORBIDDEN | StatusCode::UNAUTHORIZED // 403, 401 ; a little bit of a hack but it works most of the time
+        )
+    {
         // status code is 2xx or 403, need to check if it ends in /
 
         if response.url().as_str().ends_with('/') {