Browse Source

limit max_crawl_depth to 100 for normal crawl

There are capsules out there that kill the crawler due
to a recursion exceeding the limits of python.
Python limit seems to be around 1000, so the value
can be increased if needed, but i don't think we
miss anything with the current value.

Signed-off-by: Natalie Pendragon <natpen@natpen.net>
master
René Wagner 8 months ago
committed by Natalie Pendragon
parent
commit
569baa0637
  1. 2
      gus/crawl.py

2
gus/crawl.py

@ -784,7 +784,7 @@ def run_crawl(should_run_destructive=False, seed_urls=[]):
global domain_hit_timings
domain_hit_timings = {}
global max_crawl_depth
max_crawl_depth = -1
max_crawl_depth = 100
expired_resources = [GeminiResource(url) for url in load_expired_urls()]
for resource in expired_resources:

Loading…
Cancel
Save