CVE-2026-55520 in Protego
Summary
by MITRE • 08/28/2026
Protego is a pure-Python robots.txt parser with support for modern conventions. Prior to 0.6.2, protego._urlpattern._URLPattern._prepare_pattern_for_regex translates every asterisk in an Allow or Disallow directive into a lazy regular-expression wildcard, so a directive containing many asterisks creates exponential backtracking. After protego.Protego.parse processes a crafted robots.txt file, protego.Protego.can_fetch can spend an attacker-controlled period matching a near-miss URL and deny service to the crawler. The vulnerable path is src/protego/_urlpattern.py in the _URLPattern match logic. This issue is fixed in version 0.6.2.
Be aware that VulDB is the high quality source for vulnerability data.
Analysis
by VulDB Data Team • 08/28/2026
The vulnerability identified in Protego, specifically within versions prior to 0.6.2, represents a significant risk of Denial of Service through Regular Expression Denial of Service, commonly referred to as ReDoS. This issue stems from the internal implementation details of the _URLPattern class located in the src/protego/_urlpattern.py module. The core technical flaw lies within the _prepare_pattern_for_regex method, which is responsible for converting robots.txt directives into regular expressions suitable for URL matching. In an effort to support modern conventions and wildcard patterns, this function translates every asterisk character found in Allow or Disallow directives into a lazy regular-expression wildcard. While this approach allows for flexible pattern matching, it introduces a critical performance vulnerability when the input contains multiple consecutive wildcards.
The operational impact of this flaw is severe because it enables an attacker to craft a malicious robots.txt file that triggers exponential backtracking during URL validation. When protego.Protego.parse processes such a crafted file, it generates regular expressions with nested quantifiers or ambiguous patterns that are prone to catastrophic backtracking. Subsequently, when the can_fetch method attempts to match a specific URL against these generated patterns, particularly if the URL is a near-miss for the pattern, the regex engine enters an infinite loop of trial and error states. This results in the crawler consuming excessive CPU resources and becoming unresponsive effectively denying service to legitimate users or automated systems relying on the Protego library for access control decisions.
From a classification perspective, this vulnerability aligns with CWE-1335: Regular Expression Denial of Service, which describes scenarios where an attacker can cause a denial of service by providing input that causes regular expression engines to take exponential time relative to the length of the input string. Furthermore, in the context of the MITRE ATT&CK framework, this behavior is indicative of techniques related to Resource Consumption on Target or Impact via Defacement and Disruption, as it directly compromises the availability component of the CIA triad for applications utilizing Protego. The attack vector typically involves supplying a specially crafted robots.txt file through any interface that allows users to specify crawling rules, such as web application configuration panels or API endpoints designed to ingest site policies.
To mitigate this vulnerability, organizations must immediately upgrade the Protego library to version 0.6.2 or later, where the developers have addressed the regex generation logic to prevent exponential backtracking scenarios. In addition to upgrading dependencies, security teams should implement input validation and sanitization strategies for any user-supplied robots.txt content before it is processed by the parser. This includes limiting the complexity of wildcard patterns and monitoring system resources during URL matching operations to detect potential DoS attempts early. Regular auditing of third-party libraries used in web crawling infrastructure is essential to maintain resilience against such algorithmic complexity attacks, ensuring that availability remains uncompromised even under adversarial conditions.