crawl: set a timeout for every fetch #15

Closed
opened 9 months ago by René Wagner · 2 comments
Owner

we should set a timeout for every page fetch.
Timeouts are sometimes 30 seconds, sometimes a few minutes...

I think 30 seconds would be ok to start with

we should set a timeout for every page fetch. Timeouts are sometimes 30 seconds, sometimes a few minutes... I think 30 seconds would be ok to start with
René Wagner added the
enhancement
label 9 months ago

I've never used the Python standard ssl (sic) library but it seems timeouts are possible with socket.settimeout(value) (with the PyOpenSSL, timeouts don't really work).

I've never used the Python standard `ssl` (sic) library but it seems [timeouts are possible](https://docs.python.org/3/library/socket.html#socket.socket.settimeout) with `socket.settimeout(value)` (with the PyOpenSSL, timeouts don't really work).
René Wagner changed title from set a timeout for every fetch to crawl: set a timeout for every fetch 9 months ago
Poster
Owner

Timeout is already implemented in gusmobile and currently set to 15 seconds.

Timeout is already implemented in [gusmobile](https://src.clttr.info/rwa/gusmobile/src/branch/main/gusmobile/client.py#L216) and currently set to 15 seconds.
René Wagner closed this issue 5 months ago
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.