diff --git a/crawler.html b/crawler.html index 65ae05d..b5b967e 100644 --- a/crawler.html +++ b/crawler.html @@ -24,6 +24,11 @@ that software such as search engines can use to help find specific websites.

+ our web crawler is specifically used for indexing for the work-in-progress search engine askLyphe, + which aims to not rely on the results of other search engines and as such needs its own web crawler to function. + we do not use our indexes to train neural networks, and currently do not store full pages in their entirety whatsoever. +
+
our web crawler attempts to respect standard robots.txt files, and should also respect robots.txt blocks for googlebot (unless you specifically allow vorebot); however, no one is a perfect programmer and we may have made a mistake. @@ -51,6 +56,12 @@ so on.

+ Our web crawler is specifically used for indexing for the search engine askLyphe, + which is currently in development and not available to the public. Our design goal is to not rely on other search engines for our results, + thus we must run our own web crawler. + We do not use our indexes to train neural networks, and currently do not store full pages in their entirety. +
+
Our web crawler attempts to respect "robots.txt" files (https://en.wikipedia.org/wiki/Robots.txt) and will also respect blocks on "googlebot" (unless you specifically allow "vorebot"). However, our program may make errors. If our program has made an error, please email us at devnull@voremicrocomputers.com