You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Reimplement the can_fetch() function of RobotFileParser such that it prioritizes multiple user-agents. Add unit test for said functionality and set the user-agents this crawler uses to ["gus", "indexer", "*"] (as they were in the past, though with bugs). This was heavily inspired by the earlier discussion at https://lists.sr.ht/~natpen/gus/%3C20210212070534.14511-1-rwagner%40rw-net.de%3E
|3 months ago|
|lib||3 months ago|
|test_crawl.py||11 months ago|