UrlRecognizer detects many false positives when analyzing code snippets

[gfrebello]

Describe the bug
The BASE_URL_REGEX pattern of presidio_analyzer.predefined_recognizers.url_recognizer.UrlRecognizer generates several false positives when analyzing text that contains code snippets, filenames, or any non-URL token in the format <foo>.<bar>. In particular, regex patterns like (?:sy), (?:mt), (?:py) etc. match with os.system (Python module), zeus.mtia.local (hostname), rpc.py (Python file) and others, causing unintended false positives like os.sy, zeus.mt, rpc.py, etc.

To Reproduce
Install presidio_analyzer and run this:

from presidio_analyzer import AnalyzerEngine

text = '''
# Exploit Title: - rpc.py: Remote Code Execution (RCE)
# Google Dork: N/A
# Date: 2022-07-12
# Exploit Author: Elias Hohl
# Vendor Homepage: https://github.com/abersheeran
# Software Link: https://github.com/abersheeran/rpc.py
# Version: v0.4.2 - v0.6.0
# Tested on: Debian 11, Ubuntu 20.04
# CVE : CVE-2022-35411
import requests
import pickle
HOST = "zeus.mtia.local:65432"
HEADERS = {
"serializer": "pickle"
}
def generate_payload(cmd):
  class PickleRce(object):
    def __reduce__(self):
      import os
      return os.system, (cmd,)
  payload = pickle.dumps(PickleRce())
  print(payload)
  return payload
'''

analyzer = AnalyzerEngine()
results = analyzer.analyze(text=text, entities=['URL'], language='en')
piis = []
for result in results:
    pii = text[result.start:result.end]
    piis.append(pii)

print(piis)

Expected behavior
['https://github.com/abersheeran/rpc.py', 'https://github.com/abersheeran']

Actual behavior
['https://github.com/abersheeran/rpc.py', 'https://github.com/abersheeran', 'rpc.py', 'zeus.mt', 'os.sy']

Screenshots
UrlRecognizer.BASE_URL_REGEX:

Additional context
If this behavior is intended, adding (?![a-z]) to the expression at least prevents false positives like os.system (see the screenshots below). Cases like rpc.py (Python file) and rpc.py (valid Paraguayan URL) don't seem to have a simple solution though.

[omri374]

Thanks for raising this issue. Agree this isn't straightforward as website.ru is a valid URL which is similar to website.py.

One way to overcome this is to set the confidence threshold above 0.5, which is the current base URL pattern score. In case of a real URL, context words around the url text could help boost the score (but not in all cases).

If you have a suggestion for improvement, we'd be happy to review it.