Skip to content

Search for and respect robots.txt when retrieving content #79

@leozqin

Description

@leozqin

Please 👍 this issue if you feel that we should implement a robots.txt check for Precis before executing content retrieval. I personalIy do not feel that Precis is a "web robot" in the sense that robots.txt is meant to address, but if a substantial number (as defined by me/vibes) of people feel that we should do this, I will implement such a feature.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions