6 Commits

Author SHA1 Message Date
Arseniy Zaostrovnykh
acd760f75c Fix the indentation (big deal in Python) 2021-05-06 11:22:12 +02:00
Arseniy Zaostrovnykh
b021d34a09 Use "requests" instead of urllib; improve stability of link probing
"requests" library supports HTTP2, HTTPS, coockies by default and
connection:keep-alive header, which increase the number of positive responses
from the probed servers.

Nevertheless, due to unreliable nature of the Internet and flaky performance of
web servers some links occasionaly fail, and with this many links to
probe (1.3k+) it becomes very likely that at least one of them fails.
To mitigate these failures, the script retries the failed links once more.
2021-05-06 11:01:40 +02:00
Arseniy Zaostrovnykh
e42172f8fa Skip deprecated and superseded rules when checking URLs 2021-04-29 17:58:21 +02:00
Arseniy Zaostrovnykh
034ea0f942 Accept cookies when checking links to better emulate a browser
And fix some "infinite redirect" issues for websites, such as
developer.android.com that need cookies
2021-04-29 16:34:59 +02:00
Nicolas Harraudeau
a84840c5b4 Accept local links without checking them. 2021-02-25 16:21:19 +01:00
tomverin
e8530167f5
Check links embeded in adocs files (#32) 2021-02-12 15:18:24 +01:00