3. Count words in a text

The task

Count how often each word occurs in a text, without regard to case. This is the “hello world” of maps, and agents get it wrong in small ways that add up.

The versions

1. WorseSplits on single spaces, then contains_key, get().unwrap().clone(), and insert: three lookups and two copies of each key.

   pub fn count_words(text: String) -> HashMap<String, i32> {
       let mut counts: HashMap<String, i32> = HashMap::new();
       for word in text.split(" ") {
           let word = word.to_lowercase();
           if counts.contains_key(&word) {
               let count = counts.get(&word).unwrap().clone();
               counts.insert(word.to_string(), count + 1);
           } else {
               counts.insert(word.to_string(), 1);
           }
       }
       counts
   }

2. OKSplits on whitespace, and uses get_mut with a fallback insert. Two lookups for a new word, no unwrap.

   pub fn count_words(text: &str) -> HashMap<String, usize> {
       let mut counts = HashMap::new();
       for word in text.split_whitespace() {
           let word = word.to_lowercase();
           if let Some(count) = counts.get_mut(&word) {
               *count += 1;
           } else {
               counts.insert(word, 1);
           }
       }
       counts
   }

3. BetterTrims punctuation, uses one entry call per word, and returns a sorted BTreeMap.

   /// Words are split on whitespace, with punctuation at either end removed.
   pub fn count_words(text: &str) -> BTreeMap<String, usize> {
       let mut counts = BTreeMap::new();
       for word in text.split_whitespace() {
           let word = word.trim_matches(|c: char| !c.is_alphanumeric());
           if word.is_empty() {
               continue;
           }
           *counts.entry(word.to_lowercase()).or_insert(0) += 1;
       }
       counts
   }

The “Worse” version has two bugs and several costs:

• split(" ") gives empty words. Two spaces in a row, a tab, or a line break give "" or words with \n in them. The test worse_counts_empty_words_and_punctuation shows an empty word with a count of 1.
• Punctuation is part of the word. "cat." and "cat" are counted separately.
• Three lookups per word: contains_key, get, and insert. The unwrap() is safe here, but a reader must check that.
• Copies for nothing. word.to_string() copies a String that is already owned. .clone() on an i32 is a copy that Clippy reports (clone_on_copy). The function takes String by value but only reads it. See S05: Owned parameters (String, Vec<T>) where a borrow is sufficient.

The “OK” version fixes the split and the extra copies. get_mut returns Option<&mut usize>, so an existing count changes in place. A new word still needs a second lookup for insert.

The “Better” version uses the entry API: *counts.entry(word).or_insert(0) += 1 is one lookup, like Java’s merge(word, 1, Integer::sum). It also decides what a “word” is (trim_matches removes punctuation at both ends), and it returns a BTreeMap, so the output is sorted and the same on every run.

Java habit

In Java, text.split(" ") takes a regular expression, and split("\\s+") is a common fix. In Rust, split(" ") splits on the exact string. split_whitespace() is the method that splits on any whitespace and drops empty parts. See Strings.

Verdict

Each version compared with the Worse version

Criterion	OK	Better
Correctness	▲ Better	▲ Better
"Worse" counts empty words. "OK" still counts "cat." and "cat" separately. "Better" defines a word and tests it.
Failure behavior	▲ Better	▲ Better
"Worse" has an unwrap() that is safe only because of the check before it. The others have none.
Performance	▲ Better	▲ Better
"Worse" does three lookups and copies each word twice. "Better" does one lookup. A BTreeMap is slower than a HashMap for very large maps.
Readability	▲ Better	▲ Better
The entry line is idiomatic and short, once you know the API.
API ergonomics	▲ Better	▲ Better
Both take &str. "Better" returns a sorted map, which is easy to print and to compare in tests.

Trade-offs

• HashMap is fine when the order does not matter, for example when the next step looks up single words. It is faster for large maps. Then the “Better” code is the same with HashMap::new().
• What a word is depends on the task. trim_matches keeps "don't" as one word and drops quotes around words. For real text in many languages, a crate such as unicode-segmentation splits words by the Unicode rules. Check it on crates.io before you accept it.
• to_lowercase() allocates a new String for each word. That is necessary, because the map stores the key. To count very large texts, the next step would be to avoid it for words that are already lowercase.

Related

• Smells: S05: Owned parameters (String, Vec<T>) where a borrow is sufficient, S01: clone() to satisfy the borrow checker, S02: unwrap() / expect() in non-test code.
• Concepts: Collections, Strings, Iterators vs streams.