diff --git a/crawler.html b/crawler.html
index 65ae05d..b5b967e 100644
--- a/crawler.html
+++ b/crawler.html
@@ -24,6 +24,11 @@
that software such as search engines can use to help find specific websites.
+ our web crawler is specifically used for indexing for the work-in-progress search engine askLyphe,
+ which aims to not rely on the results of other search engines and as such needs its own web crawler to function.
+ we do not use our indexes to train neural networks, and currently do not store full pages in their entirety whatsoever.
+
+
our web crawler attempts to respect standard robots.txt files,
and should also respect robots.txt blocks for googlebot (unless you specifically allow vorebot);
however, no one is a perfect programmer and we may have made a mistake.
@@ -51,6 +56,12 @@
so on.
+ Our web crawler is specifically used for indexing for the search engine askLyphe,
+ which is currently in development and not available to the public. Our design goal is to not rely on other search engines for our results,
+ thus we must run our own web crawler.
+ We do not use our indexes to train neural networks, and currently do not store full pages in their entirety.
+
+
Our web crawler attempts to respect "robots.txt" files (https://en.wikipedia.org/wiki/Robots.txt)
and will also respect blocks on "googlebot" (unless you specifically allow "vorebot"). However, our
program may make errors. If our program has made an error, please email us at devnull@voremicrocomputers.com