diff --git a/content/policies/crawlers.mdx b/content/policies/crawlers.mdx index ea48de1f245..ea20ae8b183 100644 --- a/content/policies/crawlers.mdx +++ b/content/policies/crawlers.mdx @@ -10,3 +10,9 @@ npm's [website](https://www.npmjs.com) also has package metadata available. We a If you do not wish to install CouchDB to manage replication, we provide [open source software](https://github.com/npm/concurrent-couch-follower) that makes it easy to sync to the registry's public feed. If you attempt to access package metadata by high-velocity crawling of the npm website, we reserve the right to rate-limit or ban your IP, user-agent or both. + +## Website no-indexing rules for new packages + +The npm website uses the `X-Robots-Tag: noindex` tag in HTTP response header for new packages to avoid spam. The no-index header eventually goes away when the packages get a certain level of adoption. + +We understand this might not be desirable for newly published and well intended packages, but this is a measure we've had to take due to active abuse campaigns against the registry.