alnum_utils
Shared alphanumeric utilities for text normalization and cursor advancement.
- pipecat.utils.text.alnum_utils.strip_trailing_punctuation(text: str) str[source]
Remove punctuation only at the very end of text.
- Parameters:
text – Input text to trim.
- Returns:
text with any trailing run of Unicode punctuation characters removed.
- pipecat.utils.text.alnum_utils.fold_typography(text: str) str[source]
Replace typographic punctuation variants with their ASCII equivalents.
- Parameters:
text – Input text to fold.
- Returns:
text with typographic quotes and dashes replaced by ASCII; same length as text.
- pipecat.utils.text.alnum_utils.fold_for_matching(text: str) str[source]
Fold away surface variation between two spellings of the same text, 1:1.
Unlike
alnum_only(), this never removes or merges characters – punctuation, spaces, and markup are passed through unchanged, and each output character corresponds to exactly the same-index input character. A raw offset computed against the folded text therefore applies unchanged to the original, so callers can use it as a drop-in transform before a position-based literal comparison, without the risk of a fully-normalized (whitespace/punctuation-stripped) comparison matching across a boundary that wasn’t already a candidate in the untransformed comparison.Folds case, accents, and typographic punctuation (
’->',–->-) – the variations a TTS service may introduce between the text it was sent and the words it reports back. Deliberately narrow: each folded character is listed in_TYPOGRAPHIC_FOLD, rather than applying a blanket Unicode compatibility normalization, which would silently fold thousands of characters (CJK compatibility ideographs, halfwidth katakana, math alphanumerics) that no service is known to substitute.- Parameters:
text – Input text to fold.
- Returns:
text with those variations folded away; same length as text.
- pipecat.utils.text.alnum_utils.alnum_only(text: str) str[source]
Strip XML/HTML tags then keep only lowercase alphanumeric characters.
Accented letters (e.g. ã, é) are reduced to their base letter so TTS output can be matched against LLM text even when the provider strips diacritics. Non-Latin scripts (CJK, Hangul) are kept as-is — each original character contributes exactly one char to the result, keeping normalized length in sync with raw alnum counts used by advance_by_alnums.
- Parameters:
text – Input text to reduce.
- Returns:
Lowercase alphanumeric-only string with tags stripped.
- pipecat.utils.text.alnum_utils.has_alnum(text: str) bool[source]
Return True if text holds anything alphanumeric once markup is stripped.
The predicate form of
alnum_only(), for the question the callers actually ask: is there anything left to speak here? A tag’s letters do not count –"<break/>"is empty by this measure.
- pipecat.utils.text.alnum_utils.advance_by_chars(text: str, start_pos: int, n: int) int[source]
Return the position in text after advancing past n characters.
XML/HTML tags (
<...>) are crossed without counting against the budget, so the span returned includes any tag met on the way. Everything else counts, punctuation and spaces included.This keeps a cursor in step with one walking the same content written without the tags: both move over the same characters, so neither reaches past what the other has. Where the budget is a count of letters and digits instead, use
advance_by_alnums().- Parameters:
text – The source text to scan.
start_pos – Starting position in text.
n – Number of characters to consume.
- Returns:
New position in text after consuming n characters.
- pipecat.utils.text.alnum_utils.advance_by_alnums(text: str, start_pos: int, n: int) int[source]
Return the position in text after advancing past n alphanumeric chars.
Moves through the text one character at a time, counting only alphanumeric characters. XML/HTML tags (
<...>) are skipped entirely — their content is not counted against the budget, so the returned span includes the full tag. Other non-alphanumeric characters (spaces, punctuation) are also passed over without decrementing the budget.After the n alnum chars are consumed, advances further past any immediately following punctuation or tags (e.g. the
,in"questions,"or the</card>.in"4111</card>."), stopping before the next space or alnum char.- Parameters:
text – The source text to scan.
start_pos – Starting position in text.
n – Number of alphanumeric characters to consume.
- Returns:
New position in text after consuming n alnum chars and what follows them.