crawler: fix redirect issue
There's one capsule that kills the crawler due to a redirect overflow...it think it's the best approach to limit redirects somehow.
tentative fix: set
max_crawl_depth = 50 in
max_crawl_depth to 100, todays crawl did not fail. This is included in
02d00f10ed, but unfortunately this commit is a bit of a mess as it includes changes in
EXCLUDED_URL_PREFIXES and won't apply upstream.
In the long run i think
EXCLUDED_URL_PREFIXES and similar configs should move to constants or an own file much like "seed-requests.txt".
Deleting a branch is permanent. It CANNOT be undone. Continue?