sanitize

sanitize(source: text) 🡒 text, pure

There are cases where the text that the user sees is different from the text that user expects when working with it or processing it: The text might contain so-called invisible characters that can appear like regular space characters on the screen or they do not appear at all, hence truly becoming invisible. Such invisible characters can for example be tabulators, no-break spaces or Hangul Fillers. A list of invisible characters that are supported by the sanitize function will be provided in the next section. For further study, an exhaustive list of invisible Unicode characters can be found on https://invisible-characters.com.

Envision’s escape function is very useful in making such characters visible by replacing the character itself with its Unicode notation, but it can become cumbersome to clean up the text and remove or replace those characters for safer processing because that can require several replace operations for each of the invisible characters contained in the source text.

The sanitize function removes all supported invisible characters from the source argument and returns a cleaned-up version of the source text.

Example

This example illustrates using the function with accepted characters:

t = "Sanitizing a no-break-space\u00A0, a Hangul Filler\u3164, and a tab\t."

l=strlen(t)
show scalar "The untouched text source. \{l} Characters." a1e1 with t

show scalar "Revealing the invisible characters. \{l} Characters." a2e2 with escape(t)

t = sanitize(t)
l = strlen(t)

{ textBold: true }
show scalar "The sanitized text. \{l} Characters." a4e4 with t

Supported Invisible Characters

The sanitize function will scan for and remove invisible characters from the following list:

Valid source text

The function only accepts text containing characters from the following Unicode blocks:

Invalid source text

The sanitize function restricts the text that it processes to supported languages and symbols that can be expected in supply chain forms.

Characters from the following Unicode ranges are not accepted by the function:

If any such characters are contained in the source argument, Envision will respond with a message using this format:

sanitize(): "<source>" has invalid character \u<character value> '<Unicode character>'.

For example, when trying

t = sanitize("Text with Dingbats ✂")

Envision will report the following:

sanitize(): "Text with Dingbats ✂" has invalid character \u2702 '✂'.

Passing a source text with emojis will produce a similar respone:

t = sanitize("Text with emoji 🙂")

Envision will then report:

sanitize(): "Text with emoji 🙂" has invalid character \uD83D\uDE42 '🙂'.

See also

• escape
• replace