74c65d993d
timeout if crawling takes too long
...
default is 5 minutes before the task gets killed, in the future we
should probably have a more graceful shutdown method
2025-03-16 16:07:47 -07:00
2463986e8d
await ack
2025-03-14 18:49:27 -07:00
38ef5a45f6
use suggested route
2025-03-14 18:38:43 -07:00
0f6f0deb9c
linkrelstore + site suggestion cli
...
logs href relations in database, allowing for pagerank calculation in
the future.
sites can now be added to the queue by running ./vorebot <url> [damping]
default damping is 0.45 for suggested sites
2025-03-14 18:20:15 -07:00
d341c66390
prioritize english
2025-03-14 14:29:46 -07:00
c754338bf4
favor newer hostnames
2025-03-14 12:13:45 -07:00
384464bdbc
preemptive fix
2025-03-13 19:45:46 -07:00
d45f13f030
add proxy support to vorebot
2025-03-13 19:23:59 -07:00
17054c0a9c
add password support to vorebot
2025-03-13 19:10:55 -07:00
1ad6dad9eb
fix foundationdb dependency
2025-03-12 21:09:35 -07:00
906c2ed8df
bring over new crawler and lyphedb
2025-03-12 12:52:24 -07:00
