Shared task
FIX a bug in feature_extraction.text.strip_accents_unicode
PR #15100 ↗ · scikit-learn/scikit-learn · · merged Oct 3, 2019 · +25 −3 · base af8a6e592a1a
what a new run launched now would send
strip_accents_unicode fails to strip accents from strings that are already in NFKD form
<!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->
<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->
#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
The `strip_accents="unicode"` feature of `CountVectorizer` and related does not work as expected when it processes strings that contain accents, if those strings are already in NFKD form.
#### Steps/Code to Reproduce
```python
from sklearn.feature_extraction.text import strip_accents_unicode
# This string contains one code point, "LATIN SMALL LETTER N WITH TILDE"
s1 = chr(241)
# This string contains two code points, "LATIN SMALL LETTER N" followed by "COMBINING TILDE"
s2 = chr(110) + chr(771)
# They are visually identical, as expected
print(s1) # => ñ
print(s2) # => ñ
# The tilde is removed from s1, as expected
print(strip_accents_unicode(s1)) # => n
# But strip_accents_unicode returns s2 unchanged
print(strip_accents_unicode(s2) == s2) # => True
```
#### Expected Results
`s1` and `s2` should both be normalized to the same string, `"n"`.
#### Actual Results
`s2` is not changed, because `strip_accent_unicode` does nothing if the string is already in NFKD form.
#### Versions
```
System:
python: 3.7.4 (default, Jul 9 2019, 15:11:16) [GCC 7.4.0]
executable: /home/dgrady/.local/share/virtualenvs/profiling-data-exploration--DO1bU6C/bin/python3.7
machine: Linux-4.4.0-17763-Microsoft-x86_64-with-Ubuntu-18.04-bionic
Python deps:
pip: 19.2.2
setuptools: 41.2.0
sklearn: 0.21.3
numpy: 1.17.2
scipy: 1.3.1
Cython: None
pandas: 0.25.1
```
Work only inside this repository checkout. Make the code change the task
describes, keeping the diff focused — no drive-by refactors.
When you are done, leave your changes committed or in the working tree;
they are collected automatically.
Stay on this snapshot checkout (`task/ycb_scikit-learn_pr15100`). Never checkout, pull, or rebase onto `main`. That branch is a README-only orphan.
Stay on this HEAD. Do not fetch another default branch. Push only on the Cursor-created `crazy-cursor/…` side branch from this HEAD.Some past runs of this task were launched with a different prompt (the prompt template changed since, or those runs predate this benchmark's stored prompt). Each run persists the exact prompt it sent at launch — that per-launch record is the audit trail; this page shows only the current one.
| Run | Model | Verdict |
|---|---|---|
| Aug 21, 12:31 UTC · completed | composer-2.5 | PASS |
| Aug 21, 12:31 UTC · completed | grok-4.6 | PASS |
| Aug 21, 12:31 UTC · completed | grok-4.6-low |
Powered by YourCodingBench — benchmark coding models on your own repo's commits. sign in
| Aug 21, 12:31 UTC · completed | grok-4.6-medium | PASS |
| Aug 21, 12:31 UTC · completed | grok-4.6-xhigh | PASS |
Binary verdicts from the pinned judge (D27). Full attempt detail — candidate diff, transcript, timings — lives on each run page's score matrix.