This page is primarily intended for two audiences:
Momiji is a user-facing desktop application which provides free, private and anonymous web search functionality.
MomijiSlowCrawler is a bot/crawler/spider which indexes web page content for potential later inclusion within Momiji.
At the time of this writing, there are many bots which do not act in good faith and such has naturally given a bad name to bots in general. In contrast, MomijiSlowCrawler is designed to act in good faith, and it expects to met in kind by the web sites it visits.
MomijiSlowCrawler will announce itself via the User-Agent request header as:MomijiSlowCrawler (Public Search Engine; ai=no, ml=no, scraping=no, indexing=yes; url=https://www.path10.net/momiji)
Prior to public release of Momiji, and during the development and testing of this crawler, it announced itself simply as:MMJSlowCrawler
The reasons are twofold:
MomijiSlowCrawler is only interested in indexing the following items from each web page it is permitted to crawl:
Whole sentences/paragraphs (apart from the title), images, videos, audio files, fonts, style sheets and other web page elements are NEVER scraped, NOR cached. There is simply no room for them in the index, which is distributed to each Momiji user.
In the future, proper names of places, people, animals, etc... appearing in the page content may be indexed and treated as tags to render content more surfaceable in search.
Momiji exists to provide users with private and anonymous web search and, like any web search engine, it displays search results relevant to user queries.
At ALL times, the URL of each web page is displayed in each individual search result (where said search result references a web page and not other search result types like location coordinates). Additionally, the site portion of said URL has its text intentionally bolded to draw special attention to the provider of the content.
Example: https://www.path10.net/momiji
Where available, unambiguous and recognized, the copyright or license information (usually open licenses such as those offered by Creative Commons) may be displayed both in the search result list as well as in an "info panel" on the right side of the Momiji application window. More licenses are being added over time as well as mechanisms for blanket licenses for web sites which do not make specific mention on a per-page basis.
Momiji NEVER does any of the following:
MomijiSlowCrawler DOES remove irrelevant query parameters (including but not limited to those of affiliate links and/or user tracking mechanisms) from URLs since these are wholly inappropriate for general purpose search engine results, not to mention muddy the data within dashboards of site operators since all users would consequently appear to be belonging to a singular campaign or cohort.
MomijiSlowCrawler may also disregard some tags that were available on the web page (to prevent keyword stuffing), and Path10 my occasionally add specific and relevant tags, both automatically and manually, in the interest of making relevant results more generally surfaceable to users.
Although mentioned specifically above, it will be mentioned once more that Momiji DOES NOT DISPLAY web page content within its user interface.
When a user double clicks a search result, or presses a dedicated "Open" button in the right-side info panel, the user's preferred web browser (such as Chrome, Firefox or other) will open the corresponding URL in a new browser tab and this will be the moment the web content will first become visible.
Momiji is unaware of what the user will see as well as how the user will proceed to interact with said site from this moment forward. Momiji, being a desktop application, will however remain open for the next time the user wishes to use it.
Path10 will NEVER do any of the following with the content indexed by MomijiSlowCrawler:
To be clear, a non-top per-user page means a URL which is NOT a top page and indicates that it pertains to a given user.
Example non-top per-user page (which MomijiSlowCrawler will strive NOT to index): https://www.path10.net/users/msmith
Example top page URL (which MomijiSlowCrawler WILL index, and ONLY the top page, unless permitted to index beyond): https://msmith.path10.net/
More information about how permissions are determined is described in detail below.
Prior to accessing any web page on a site which was not accessed in the past 24 hours, MomijiSlowCrawler will first attempt to consult the site's robots.txt file.
Path10 and MomijiSlowCrawler take the robots.txt standard LITERALLY and considerable effort to be in compliance with the standard has been taken by Path10.
RFC-9309 is a longstanding (30+ years) and globally recognized convention to be followed, in good faith, BOTH by crawlers AND web sites, and it grants the full opportunity to web site operators to communicate their wishes to crawlers. MomijiSlowCrawler is listening for and will do everything reasonable to abide by these wishes.
Path10 however recognizes the well-established default, that THERE ARE NO RULES for crawlers to follow unless such rules are openly specified. It is NOT that crawlers have no abilities until permitted. If this doesn't sound right to you, please give RFC-9309 a good reading.
In accordance with RFC-9309 section 2.4, the robots.txt file may be cached for a period not exceeding 24 hours, or longer should the robots.txt file be temporarily unavailable. Accordingly, MomijiSlowCrawler may temporarily rely on a cached copy of this file.
If a website does not have a robots.txt file, placed at the root directory of the site (from the perspective of an HTTP client), then it will be considered by MomijiSlowCrawler as a lack of rules to follow.
Example root directory placement of robots.txt: https://www.path10.net/robots.txt
If a website does have a robots.txt file but is unable or unwilling to offer it to MomijiSlowCrawler (observed as a HTTP 4xx response), then in accordance with RFC-9309 section 2.3.1.3, the robots.txt file may be ignored and MomijiSlowCrawler will accordingly consider this a lack of rules to follow.
A special note about HTTP 429 - Too Many Requests... An HTTP 429 response will not trigger any slower behavior on the part of MomijiSlowCrawler, and such may even be interpreted as the site acting in bad faith. The site may lack the ability to distinguish the clients behind a single global IP address and may be experiencing a legitimate firehose of requests from said IP address through which MomijiSlowCrawler and other clients are appearing. Path10 understands this. But the site can always avail itself of the User-Agent HTTP request header to see that any firehose will NEVER be due to MomijiSlowCrawler. Please also understand that MomijiSlowCrawler is already going very slow to begin with, and does NOT make multiple requests in succession. Furthermore, there is little to no recourse on MomijiSlowCrawler's side. Just how slow MomijiSlowCrawler actually goes will be described in more detail below.
Currently MomijiSlowCrawler does not have special handling for HTTP 5xx responses and, for the time being, will treat such as server errors impeding further immediate communication with the site. Special consideration will be given in the future when designing the implementation for such errors.
MomijiSlowCrawler will apply an overly-generous 60 seconds timeout when requesting the robots.txt file. This means, from the time MomijiSlowCrawler issued the request, if 60 seconds should elapse before the robots.txt file is received, the connection will be cut from MomijiSlowCrawler's side. If this occurs, it will be interpreted as the scenario outlined in RFC-9309 section 2.3.1.3, possibly by a site attempting to hold MomijiSlowCrawler in limbo, in bad faith, and MomijiSlowCrawler will consider this a lack of rules to follow.
The reason for this above interpretation is that robots.txt is a small, static text file which seldomly undergoes changes and even with rudimentary equipment from the 1990s would require next-to-no computing resources to provide to any HTTP client, even WITHOUT caching mechanisms in place. Even 15 seconds would be more than generous enough a timeout, especially in the year 2026.
When made available to MomijiSlowCrawler, ALL directives, except for one, within the robots.txt file will be followed TO THE LETTER, in good faith.
The only directive which may be interpreted more loosely is Crawl-Delay, as it is not an official directive within RFC-9309, and only in the event that it is set to a value exceeding 600 seconds (10 minutes) per request, which would be considered by MomijiSlowCrawler as the site acting in bad faith. In such case, the delay observed by MomijiSlowCrawler will revert back to 600 seconds. In the future, other remedies may be considered for dealing with such sites.
MomijiSlowCrawler will never make multiple requests to a given site any more frequently than every 30 seconds, even if the robots.txt file indicates that it can accept more frequent requests or omits the Crawl-Delay directive altogether. If the Crawl-Delay is between 30 and 600 seconds, the directive will be followed as per the site's wishes.
Although not a strict standard, and not part of RFC-9309, the Content-Signal directive spearheaded by Cloudflare will be obeyed by MomijiSlowCrawler. This means that if public search engines are explicitly welcomed by this directive, MomijiSlowCrawler will accept the warm welcome.
As both a consequence of legal considerations and as a courtesy to web sites, MomijiSlowCrawler takes what would mostly be considered a "breadth-first" approach to crawling web sites and, absent any special rules internal to MomijiSlowCrawler, will only index the top page of a given site, and again, only if not prohibited by robots.txt.
A top page (sometimes called a homepage) is defined here as a page accessible by the root URL wtithout any mention of filename, or any page accessible by the root URL and a filename typical of a top page not nested within a directory/folder visible in the URL. Below is the list of such filenames that MomijiSlowCrawler currently recognizes as "typical top pages", for convenience.
An example top page URL: https://www.path10.net/index.html
Since it is impossible, from the outside, to know how a web site maps URLs to servers and/or file system directories, MomijiSlowCrawler will treat subdomains without filenames (or with recognized top page filenames) as top pages of distinct sites also.
An example sub-domain top page URL: https://wiki.path10.net/index.html
While not currently implemented, in the future, MomijiSlowCrawler may recognize pages within specific language directories as top pages also.
Specific language top page URL: https://www.path10.net/ja/index.html
Note that if requesting a top page URL results in a redirection by the server, MomijiSlowCrawler will NOT record the redirected URL in its database. This is a legal consideration, since the site may at a later time reconfigure its redirection mechanisms, leading to a potential future access to a non-top page.
To crawl deeper than the top page, MomijiSlowCrawler, by design, requires a special per-site entry in its "unrestricted list". Adding such an entry is not something a crawler can do on its own, and such requires human eyes to vet the site's terms. (discussed in detail below)
It should be noted that Path10 believes in open and free exchange of publicly available resources via deep-linking, and MomijiSlowCrawler's current deep-linking logic design is an unfortunate contrivance in order to be in compliance with a world that doesn't always understand how the web was meant to work.
Web pages contain links to other pages both internal and external to the hosting site and each valid link discovered by MomijiSlowCrawler in the course of crawling, and not listed in its "restricted list", is persisted to a database if it was not already, for later indexing.
There is no preferential treatment toward any one web site or page over another when deciding which persisted link to index next within MomijiSlowCrawler's logic. Each link is randomly retrieved from the database and filtered from a set of candidates which warrant indexing or re-indexing.
To assist with this filtration, MomijiSlowCrawler manages a per-web page indexing frequency value, defaulting to every 14 days for yet-to-be-indexed pages.
If, after indexing a given web page, MomijiSlowCrawler determines that it has not undergone any content changes since the previous indexing, it will double the frequency until the next indexing. For instance, if the original frequency was 14 days, this doubling will cause it to become 28 days until the next indexing. This can be doubled again and again, with the maximum indexing frequency being every 180 days.
If, on the other hand, a content change did occur (this is coincidentally the case for each page upon first indexing), the indexing frequency will be halved. The minimum frequency will be 24 hours. Note that this does not mean the next indexing will occur exactly 24 hours later, it only means that the earliest it could occur is 24 hours later.
In the case of web pages, content is judged to have changed when a web page's title or tag words are different from that of the previous indexing.
Larger web page content is NEVER cached by Path10 and the above mechanisms needn't use full content information to judge changes.
⚠️ Note that artificially changing a web page's content to encourage a shorter indexing frequency will NOT have a measurable or direct impact upon the sort order of search results in Momiji (should the user elect to specify "Newest" as a sort aspect of higher importance). This is because the Newest sort aspect acts upon the timestamp that Path10 first processed the web page's entry and not the date it was last updated within the site.
While MomijiSlowCrawler is completely unbiased in its indexing activities, a separate peripheral tool for manual curation (aptly named MomijiCurator) used internally by Path10 is NOT, and allows for specific indexing and even the adding of manually chosen URLs at the sole discretion of Path10. This tool is outside the scope of this document but is briefly mentioned here in the spirit of transparency.
Sites that offer RSS, XML or JSON feeds may have one of these be regularly indexed in order to discover web pages in a more lightweight fashion which is respectful of the site's infrastructure. The feed URL itself will not be added to the index distributed to Momiji users.
The only exception to the usability of a feed is if said feed is made available via another site at a different URL and said site's Terms of Service are not amenable to crawling.
Below is a list of currently recognized, supported and unsupported feeds platforms.
Supported Not Supported feedburner feedpress.com
The indexing frequency rules mentioned above for web pages also apply to feeds, however the default frequency is 24 hours, the minimum is 12 hours and the maximum is 30 days.
To judge content change in the case of feeds, each downloaded feed is cryptographically one-way hashed and only its short-form (less than 30 characters) digest is then stored. The feed is judged to have changed if this digest is different from the one generated upon the previous indexing.
Although not currently supported, in the future, MomijiSlowCrawler may support crawling of sitemap.xml files.
Indexing occurs upon the static HTML content of a web page and NOT the document object model (DOM) which may have been altered by scripting, post loading. The reasons for this are inordinate and too much to go into here.
It should be noted that the static web page content currently needs to have the majority of its text in either English or Japanese and that the text will be subject to an "objectionable content" scan. This entails looking for terms which relate to, among other things, sex, drugs, and violence.
Sites renown for maintaining a high standard of content moderation may be afforded custom thresholds for objectionable content, at the sole discretion of Path10.
Pages with minimal objectionable content will be eligible for auto-curation. Yes, this is an oxymoron, but essentially this means the page needn't be reviewed by human eyes and will be automatically slated for inclusion in the next Momiji Index update. Pages which do not fall under the objectionable content thresholds may still be curated at a later stage but will require human review to make such a determination.
Path10 believes that there IS a time and place to drop an F-bomb and Momiji is not about trying to control anybody's voice. At the same time however, Momiji strives to give control to users over what they see and Path10 is always calibrating and fine-tuning this content scan to best describe the nature of content to users.
Somewhat related is the Path10 stance on sexual content. Momiji will NOT EVER knowingly include gratuitously explicit sexual content (pornography) but will set the upper threshold to "gravure idol"-level content and will strive to rate content as such for users.
In addition to the aforementioned objectionable content scanning, MomijiSlowCrawler recognizes the below two meta-tags for sites which responsibly self-describe their page content as being for adults only.
<meta name="RATING" content="adult"/><meta name="RATING" content="RTA-5042-1996-1400-1577-RTA"/>
Content related to advocacy of suicide and self harm is included in the objectionable content scan and will NOT knowingly be curated, AND will be actively removed if found, and may result in the hosting site being placed in the restricted list so as not to be curated going forward.
Preventative and corrective measures are currently in place to help thwart the curation of web pages hosting all types of explicitly objectionable content and these will be further bolstered over time.
When Path10 determines that a site's larger body of content is a potentially good fit for Momiji, it will strive to find said site's Terms of Service. These terms may go by a different name such as Terms of Use, Site Terms, Terms and Conditions, or even Ts & Cs.
1. Typically, there is a link to the Terms of Service reachable from a site's top page, and this is where the search will begin. (Note that many sites place the link near or at the very bottom of the top page and some sites dynamically load more content when a user scrolls to the bottom, causing the terms link to fall further down, completely outside the viewport of the browser window, and, as a consequence, out of reach for anyone to click upon. Path10 may elect to omit such sites from inclusion within Momiji, at its sole discretion.)
2. Failing to find such a link, Path10 will look into the site's "About Us" page since some sites stipulate their rules there.
3. Failing this, Path10 will look into the robots.txt file since some sites use comments in this file
to link to such a page or clearly indicate their comfort level with regard to bots crawling the site.
If the robots.txt file uses the Content-Signal directive and it specifically
states search=yes for a user agent rule which applies to MomijiSlowCrawler
then said site may be eligible for immediate indexing and it may be placed in the unrestricted list.
The only exception to this is if any terms in the Terms of Service contradict the
Content-Signal directive, in which case the site operator/owner may be
contacted for clarification.
4. Failing this, if the site offers a search feature, a search for "terms" (and 「利用規約」, 「リンクポリシー」 in the case of Japanese sites) will be performed.
5. Failing this, the paths /terms, /termsofuse and /tos, and potentially others, will be individually appended to the host URL's root, in case they yield results.
6. Failing this, if the site has a parent company, hosting company (typical of blogging platforms), or third party licensing partner, such company site may offer an umbrella Terms of Service for it and other subsidiaries/partners, and such will be searched.
If, ultimately, there are no obvious Terms of Service, it will be considered by Path10 that the site understands the open web, and that the contents of the robots.txt file, if any, are the only rules to follow.
If a link to the Terms of Service exists but results in a HTTP 404 or similar response, the site may be contacted by Path10 to notify of such, and Path10 will refrain from adding the site to MomijiSlowCrawler's unrestricted list until such time as the link is repaired.
When the Terms of Service do exist and are openly available, they will be vetted. The style and tone of each document varies and what Path10 is looking for is an understanding of the open web.
Stipulations about licensing agreements and expressly written permissions, ambiguous definitions about "commercial use", and other such impediments will most likely render the site unindexable by MomijiSlowCrawler.
The terms, as they are, either...
In the event that a site's Terms of Service are deemed not amenable to crawling/indexing/linking, Path10 will make a record of the examination. As a general policy, the site will be re-examined no sooner than one year later (same month and date of the following year). Upon each examination/re-examination, another year from the current date will be allowed to elapse, until the terms are rendered amenable. If re-examination occurs many times, the site may be deemed not worth future re-examination. This policy is to give preferential treatment to other sites which are open to crawling/indexing/linking.
Keep in mind that, in the case of unfavorable terms, since the Terms of Service themselves nearly always prohibit automated crawling, this re-examination CANNOT be automated. Hence the need to manually examine the terms.
Contacting Path10 to inform of updates to the Terms of Service (as in: "Oh hey, remember us? Last year we instructed you not to index our site. Well, I'm thrilled to inform you that we've come around to opening up, so please index our site now!") will NOT result in expedited re-examination, and in the case of the site having been previously deemed not worth re-examination, will NOT reverse such decision.
Also note that general solicitations by sites wishing to be indexed are NOT accepted. Inquiries regarding when a site’s Terms of Service were last examined by Path10 will also NOT be accepted. Lastly, inquiries regarding whether a site has been been deemed not worth re-examination will NOT be accepted.
Path10 is busy managing other software development in addition to Momiji (and peripheral systems), as well as content management, among other duties, and simply cannot accommodate such inquiries.
The only solicitations entertained will be by charities interested in the zero-cost Charity Tier of the Calling Cards feature, once released.
Path10's advice to site owners/operators is THIS → Take the time to weigh the trade-offs of being open to public search engines, and act early should you decide to open up.
It is entirely possible to protect one's copyrights and trademarks and still invite public search engines to minimally crawl and index your site in the interest of connecting users with great content, and many web sites are already accomplishing such with their Terms of Service.
Keeping your Terms of Service the way they are, and adding a section like the one below is often times sufficient...
You are not allowed to crawl the site, unless:
- You develop and/or operate a public search engine,
- And only to the extent of minimal indexing of content,
- And only if obeying the rules stipulated in the site’s robots.txt file,
- And without placing an undue burden on site’s network infrastructure (typically no more requests than a human user would incur in 30 seconds),
- And not for the purposes of training AI or ML.
You are not allowed to deep-link into the site, unless:
- You develop and/or operate a public search engine,
- And only if it is made clear that <OUR COMPANY NAME> is the provider of the content.
For Calling Card users, the links you submit may not be amenable to automated examination during the URL Claim Verification phase and manual verification may be needed. Momiji will have some predetermined knowledge of some unsupported sites and more will be added over time. Links may be removed later if any site's Terms of Service undergo changes and/or new sites Terms of Service are recognized.
For bloggers, if your blog is hosted on a blogging platform, and such is visibly recognizable (the URL contains the name of the platform, or the platform is mentioned clearly on the web page) then Path10 must abide by said platform's Terms of Service, and this may prevent indexing. If the platform allows for self-hosting (bloggers obtaining the software and hosting their own instances on their own sites) then the blog MAY be indexable by MomijiSlowCrawler.
Below is a list of currently recognized, supported and unsupported blogging/hosting platforms.
Supported Not Supported blog.fc2.com www.ghost.com (anti-crawler) micro.blog www.livedoor.jp (anti-crawler) seesaa.net www.medium.com (anti-crawler) www.cocolog-nifty.com www.squarespace.com (anti-crawler) www.pressable.com www.substack.com (anti-crawler) www.wordpress.org
To be supported, the host must be accepting of bots crawling their site as well as not have any stipulations about linking to any page within. Bloggers interested in having their blogs indexed in Momiji are encouraged to self-host when possible (this is also the best way to have your true voice online!) or choose a hosting platform more open to public search engines.
For podcasters and musicians/performers, if your content is hosted on a media platform, and such is visibly recognizable (the URL contains the name of the platform, or the platform is mentioned clearly on the web page) then Path10 must abide by said platform's Terms of Service, and this may prevent indexing.
Below is a list of currently recognized, supported and unsupported media platforms.
Supported Not Supported www.vimeo.com bandcamp.com (anti-open web/indexing) www.youtube.com www.rumble.com (anti-crawler) www.soundcloud.com (anti-crawler) www.spotify.com (anti-crawler)
Path10 is based in and operates within Japan, and abides by local laws with regards to upholding copyrights and trademarks, as well as in its crawling and indexing activities.
The following is offered as a courtesy and is NOT to be considered legal notice nor legal advice, and the reader is both implored to familiarize himself/herself with Japanese copyright law, as well as consult with legal counsel regarding the below statements.
Japanese copyright law is stringent but allows for special exceptions in an effort to reasonably balance the rights of copyright holders with those of the general public.
Path10 operates under Japanese Copyright Law Article 47-5 (Minor Exploitation Incidental to Computerized Data Processing and the Provision of the Results Thereof) paragraph (1) item (i) and item (iii), and paragraph (2), Article 48 (Indication of Source) paragraph (1) item (iii), and paragraph (3) item (i), and, operating as a software company is afforded, under the law, the ability to offer a freely-accessible public search engine provided that it:
⚠️ Know that both MomijiSlowCrawler and Momiji, in their implementations, were carefully designed and developed, from day one, to be IN FULL AND EVIDENT COMPLIANCE with each and every legal requirement above.
Although largely considered unenforceable under Japanese law, stipulations against indexing even the top page of a site, where said top page is publicly accessible and not behind an authentication/authorization mechanism, will nonetheless be followed by Path10 (once such is first understood by Path10 via human review), and said top page will NOT be indexed, or it will be removed from the index if previously added. This is perhaps less of a legal consideration and more out of wanting to award preferential treatment to web sites which are open over those which are not.
As an aside, in many jurisdictions around the world, a site's Terms of Service are only legally enforceable when the user is given the opportunity to easily locate, read and accept/reject the terms prior to consuming a site's larger content. Many sites do not prompt users adequately and such terms may not be legally enforceable as a result.
Many web sites have stipulations pertaining to the "commercial use" of content. There are also nearly as many definitions of "commercial" as there are such sites. It warrants discussing whether Momiji is a commercial product or not.
Path10 does NOT consider its activities with regard to indexing content for private/anonymous search as being commercial in nature. Not a single yen is earned from indexing content, nor from forwarding users to any search result's web page, nor from any affiliate links (there are in fact none in Momiji), nor from advertising within Momiji (there are no ads). If anything, these activities are costing Path10 money.
Path10 offers Momiji search functionality completely for free and this will NOT EVER change. There are donation links in Momiji but donations are NOT compulsory and there is NO difference in user experience whether a user donates or does not donate to Path10.
Momiji is an independent passion project and will continue to exist and be further developed regardless of potential profitability.
Having said all this, Path10's view is not the only one to consider and Path10 DOES make an effort to put itself in the shoes of web site operators each time a site's Terms of Service are vetted. If said terms' views about what constitutes "commercial" are clear and overwhelmingly not inline with Path10's views then the site may be omitted from indexing.
There is one aspect of Momiji which is partially commercial: The at-cost Calling Cards functionality (excluding the Free Tier and Charity Tier of the service). This is offered with small businesses and entrepreneurs in mind, to help accelerate the development and provisioning of Momiji for all users, while simultaneously helping said small businesses and entrepreneurs gain exposure for the value they create.
It should be noted however that the Calling Cards business model is mostly based on the inclusion of and linking associations of content defined and provided by Calling Cards users themselves, and less so the inclusion of content already indexed elsewhere by MomijiSlowCrawler. For what it is worth, the logic of both MomijiSlowCrawler and Calling Cards are NOT in ANY way integrated, and neither feature is even aware of the other in ANY way.
Also worth noting is that at-cost Calling Cards will NOT EVER be allowed to comprise more than 5% of the overall Momiji Index in terms of total curated result record count. To be clear, this means that for every 1 million results made available to Momiji users by Path10, no more than 50,000 of these can ever be Full Tier Calling Cards. Free Tier and Charity Tier Calling Cards, being available at no-cost, are considered normal search results in this scenario.
While money is nice, Momiji = MISSION > money. As such, Path10 believes that Momiji is for the most part NOT a commercial product.
For legal reasons, Path10 retains backups of ALL logs output by MomijiSlowCrawler for no less than 366 days. These logs record the timestamped requests for all sites/pages undergoing indexing and, in many cases, the received response codes and other relevant fields.
If you are site owner/operator and have questions regarding this transparency page you can contact legal@path10.net.