Locale-Sensitive C/C++ String Operation Function

char *strtok(char *strToken, const char *strDelimit);

wchar_t *wcstok(wchar_t *strToken, const wchar_t *strDelimit);

unsigned char *_mbstok(unsigned char *strToken, const unsigned char *strDelimit);

_TXCHAR *_tcstok(_TXCHAR *strToken, _TXCHAR *strDelimit);

Internationalization (I18n) Function Overview

The strtok function returns a pointer to the next token found in the string strToken, and returns NULL when no more tokens are found. Each call to the function also modifies strToken by substituting a NULL character for the encountered delimiter, specified by strDelimit.

wcstok is the wide character equivalent; its parameters and return are wide-character strings.

_mbstok is supported on Windows platforms only and is the multibyte equivalent; its parameters and return are multibyte strings.

_tcstok is the Windows-only Generic version of the function; with the _MBCS or _UNICODE compiler flags determining its mapping to either _mbstok or wcstok.

I18n Issues

There is no ANSI multibyte UTF-8 version of this function, and strtok does not work correctly with multibyte UTF-8 characters. (In a string containing multibyte UTF-8 characters, characters consisting of more than one byte are not treated by strtok as an entity, each byte is treated separately.)

Therefore, if any function parameters may possibly contain multibyte UTF-8 characters, the parameters will have to be converted to wide characters (wchar_t) and then the wide function wcstok can be used.

For Windows MBCS platforms, ensure that the multibyte code page is set properly, as _mbstok depends on it. By default, the multibyte code page is set to the system-default ANSI code page obtained from the operating system at program startup. Use _getmbcp and _setmbcp to query or change the current multibyte code page, respectively.

Recommended Replacements*

ANSI UTF-8	convert and use wcstok
	(See "I18n Issues" above)
ANSI UTF-16	wcstok
Microsoft Windows MBCS	_mbstok
Microsoft Windows Unicode	wcstok
Microsoft Windows Generic	_tcstok

Locale-Sensitive C/C++ String Operation Functions