NishiBot
NishiBot is the web crawler behind nishifamily.com/search, an independent search engine with its own index. It is not an AI training crawler and it does not resell or license the pages it fetches.
How to identify it
Requests carry exactly one User-Agent header:
User-Agent: NishiBot/1.0 (+https://nishifamily.com/bot)
How to control it
NishiBot obeys the Robots Exclusion Protocol (RFC 9309), including the rules most crawlers get wrong:
- A
4xx on /robots.txt means no rules exist, and we may crawl. - A
5xx or a network failure means the file is unreachable, and we assume a complete disallow for that host — RFC 9309 §2.3.1.4. Crawl-delay is honoured as published. We do not clamp it down to a value that suits us; if the delay is longer than a single fetch can wait, we defer the host and come back later.- We follow up to five consecutive redirects on
/robots.txt, per §2.3.1.2.
To exclude us entirely:
User-agent: nishibot
Disallow: /
To slow us down instead:
User-agent: nishibot
Crawl-delay: 30
What it fetches
Static HTML only. NishiBot does not execute JavaScript, does not submit forms, does not attempt to log in, and does not fetch media it cannot index. It caps how many pages it takes from any one host per pass so that no single site absorbs the crawl.
Contact
If NishiBot is behaving badly on your site — crawling too fast, ignoring a rule, or fetching something it should not — write to elderwesto@gmail.com and it will be corrected. A crawler that cannot be complained to is not a good citizen of the web.
Why an independent crawler
Nishi’s index is built from its own crawl rather than resold from a major provider. That is the whole point: an independent index is the only way a search result can be genuinely different from everyone else’s. It also means the crawling is ours to answer for, which is what this page is.