StarpathDirectoryBot

If you run the website of a school, kindergarten, co-op or shop, you may see StarpathDirectoryBot in your logs. It's the crawler behind two of Starpath Learning's free directories: the Waldorf Map and the homeschool co-op directory.

How to recognise it

Every request it makes carries this user agent:

It never sends any other name, and it never pretends to be a browser.

What it reads

Your home page, and the pages it links to that a family would look for: about, contact, admissions, tuition, events and calendar pages, and a fee schedule when it's a PDF. It reads the text and the links. It doesn't log in, fill in forms or post anything.

From that text we take the facts a family needs to find you: your name, town, ages, open days and fairs, and tuition when it's printed with its school year. Each listing links to your own site and shows the day we read it. How we check every listing.

How often, and how fast

We read each Waldorf Map place about once a year and each co-op listing about once a quarter, with an extra visit now and then to see whether a site has moved or closed. A visit is a few pages, one at a time, and the crawler asks for at most one page a second.

How to block it

The crawler reads your robots.txt before anything else. To keep it off your whole site, add these two lines:

It follows the Disallow lines in a group for StarpathDirectoryBot or for every crawler (User-agent: *). It doesn't read Allow lines, so to keep it out of part of your site, list each path to block.

If your site answers with a captcha or a “checking your browser” page, the crawler takes that as a no and leaves. It doesn't try again under another name.

Write to us

To correct a listing, or to ask us anything about the crawler, write to hello@starpathlearning.com.