Internationalization and localization tools


Locale-Sensitive C/C++ String Operation Function

char *strtok_r(char *newstring, const char *delimiters, char **save_ptr);

Internationalization (I18n) Function Overview

Just like strtok, this function splits the string into several tokens which can be accessed by successive calls to strtok_r. The difference is that the information about the next token is stored in the space pointed to by the third argument, save_ptr, which is a pointer to a string pointer. Calling strtok_r with a null pointer for newstring and leaving save_ptr between the calls unchanged does the job without hindering reentrancy.

I18n Issues

There is no ANSI multibyte UTF-8 version of this function, and strtok_r does not work correctly with multibyte UTF-8 characters. (In a string containing multibyte UTF-8 characters, characters consisting of more than one byte are not treated by strtok_r as an entity, each byte is treated separately.)

Additionally, there is no ANSI wide character version of this function.

These two issues combined mean that there is no suitable substitution for strtok_r that is readily available which supports both re-entrancy and either multibyte or wide character strings.

Recommended Replacements*

*If you're already using the recommended function, see I18n Issues for other reasons why Globalyzer is detecting the function.

Locale-Sensitive C/C++ String Operation Functions

 

Lingoport internationalization and localization services and software