apunkt/trustgraph
1
0
Fork 0
mirror of https://github.com/trustgraph-ai/trustgraph.git synced 2026-07-14 15:52:11 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

fix: preserve non-ASCII entity names in ontology URI normalization (#1036)

Browse source 
The entity URI normalizer stripped every non-ASCII character via the
ASCII-only patterns `[^a-z0-9\-.]` and `[^a-z0-9\-]`. For non-Latin
entity names (e.g. Chinese), this deleted the entire name, so distinct
entities collapsed onto the same URI (".../<ontology>/-"), silently
merging unrelated nodes and breaking OntoRAG for non-English content.

Switch both filters to `[^\w\-.]` / `[^\w\-]`. In Python 3, `\w` is
Unicode-aware for str patterns, so non-ASCII letters (CJK, etc.) are
preserved while punctuation and symbols are still removed. ASCII
behaviour is unchanged: names are lowercased and underscores converted
to hyphens before the filter runs, so for ASCII input the patterns are
equivalent to the originals.

Adds tests covering ASCII regression guards and non-ASCII names/types,
including the core case that two distinct Chinese entities must not
collapse onto the same URI.

Co-authored-by: arthurxuwei <5179840+arthurxuwei@users.noreply.github.com>
This commit is contained in:
Wei Xu 2026-07-09 07:05:35 +08:00 • committed by GitHub
parent e9c6a850ad
commit f106ae2103
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 88 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

13
trustgraph-flow/trustgraph/extract/kg/ontology/entity_normalizer.py
View file

@ -23,8 +23,11 @@ def normalize_entity_name(entity_name: str) -> str:
# Replace spaces and underscores with hyphens
normalized = re.sub(r'[\s_]+', '-', normalized)
# Remove any characters that aren't alphanumeric, hyphens, or periods
normalized = re.sub(r'[^a-z0-9\-.]', '', normalized)
# Remove any characters that aren't word characters (incl. non-ASCII
# letters such as CJK), hyphens, or periods. \w is Unicode-aware for
# str patterns in Python 3, so non-ASCII entity names are preserved
# rather than stripped (which would collapse them onto a single URI).
normalized = re.sub(r'[^\w\-.]', '', normalized)
# Remove leading/trailing hyphens
normalized = normalized.strip('-')
@ -52,8 +55,10 @@ def normalize_type_identifier(type_id: str) -> str:
# Replace slashes, colons, and spaces with hyphens
normalized = re.sub(r'[/:.\s_]+', '-', normalized)
# Remove any remaining non-alphanumeric characters except hyphens
normalized = re.sub(r'[^a-z0-9\-]', '', normalized)
# Remove any remaining characters that aren't word characters (incl.
# non-ASCII letters such as CJK) or hyphens. \w is Unicode-aware for
# str patterns in Python 3, preserving non-ASCII type identifiers.
normalized = re.sub(r'[^\w\-]', '', normalized)
# Remove leading/trailing hyphens
normalized = normalized.strip('-')