update handling of "crawl_delay" #14

Closed
opened 10 months ago by René Wagner · 1 comments
Owner

similar to handling of robots.txt disallow rule

currently only the crawl-delay specified for "gus" will be taken into account, which is probably seldom used

related to #7

similar to handling of robots.txt disallow rule currently only the crawl-delay specified for "gus" will be taken into account, which is probably seldom used related to #7
René Wagner added the
bug
label 10 months ago
Poster
Owner
https://lists.sr.ht/~natpen/gus/patches/20446
René Wagner closed this issue 9 months ago
Sign in to join this conversation.
No Milestone
No Assignees
1 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.