Strings are complicated because they can represent text in a variety of different ways. The most common encoding is UTF-8, which can represent any character in any language. However, there are many other encodings that can be used, such as ASCII, Latin-1, and ISO-8859-1.
Different programming languages make different choices about how to present this complexity to the programmer. Some languages, such as Python, hide the complexity of strings from the programmer by automatically converting strings to UTF-8. Other languages, such as Java, require the programmer to explicitly specify the encoding of their strings.
Rust has chosen to make the correct handling of String data the default behavior for all Rust programs. This means that Rust programmers have to put more thought into handling UTF-8 data upfront. However, this trade-off exposes more of the complexity of strings than is apparent in other programming languages, but it prevents you from having to handle errors involving non-ASCII characters later in your development life cycle.
Here are some examples of the complexity of strings:
- Invalid characters: Some encodings, such as ASCII, do not support all characters in all languages. If you try to encode a character that is not supported by the encoding, the resulting string will be invalid.
- Overlapping characters: Some characters, such as diacritics, can be represented by multiple characters in different encodings. This can lead to overlapping characters in a string, which can cause problems for some programs.
- Unicode characters: Unicode is a standard that defines a unique code point for every character in every language. However, not all programs and systems support Unicode. If you try to use a Unicode character in a program that does not support Unicode, the character may not be displayed correctly.
The good news is that the Rust standard library offers a lot of functionality built off the String and &str types to help handle these complex situations correctly. For example, the contains() method can be used to search for a substring in a string without having to worry about the encoding of the string. The replace() method can be used to substitute parts of a string with another string without having to worry about overlapping characters.
