For programmers: cleaning up user (display) names.

append delete 26 Dec ’12 · 17:10 Martijn

Hello everyone, here’s an interesting headscratcher for you.

I am working on a website that should allow for users to register a login name but that will allow them to pick any display name they want for the forum. It has been decided that we do not want to limit them too much in what they can choose. If they want to be called _Holy Zarquon's Singing Fish_ [1] I do not want to get in their way.

However, how do you keep them from spoofing eachother when you open-up the unicode bucket? In other words, how should names be cleaned up before using them in our system?

Some of the things I think will be important:
1. Whitespace should be removed from the beginning and end of the name.
2. Whitespace in the middle of the name should be normalised to normal spaces only.
3. Break ligatures into separate letters so `ij` can’t be spoofed with `ĳ`.
4. Replace ‘letterlike symbols’, e.g. replace `K` with `K`.

How would you do this in PHP? See the first reply in this thread which illustrates a problem NoNonsense Forum has with this.

---

[1]: http://www.metafilter.com/user/130801

Replies

append delete 26 Dec ’12 · 17:10 #1. 　

append delete 26 Dec ’12 · 17:21 #2. Kroc

If you have PHP5.4, then the transliterator can ascii-ize almost anything. The recipe I use also breaks unicode characters apart, such as "ij", and replaces letter-like symbols. Basically, using PHP5.4's transliterator will cover 90% of the conversion, and regex can clean up the rest.

You can normalise complex whitespace by using Unicode regex matching, i.e. `\p{z}` (http://www.regular-expressions.info/unicode.html#prop) to match all whitespace and replace with the common space (though I believe the transliterator will do this for you, already).

Or, you could just limit to A-Z0-9 :P

(BTW, another way to kind of stop spoofing, when it comes to important users, is to mark their name differently in the interface. That way an ideograph spoofed named will not be highlighted the same)

append delete 26 Dec ’12 · 19:29 #3. Martijn

The problem with transliterations is that they will also take out accented characters. I don’t want to force people away from using their own name. Names like Björk Guðmundsdóttir, 毛泽东, or Борис Николаевич Ельцин [1]. Those would not pass any transliteration system.

That’s what makes it a headscratcher. I don’t want it easy for people to spoof things, so whitespace and letterlikes have to be normalised, but that doesn’t mean they should be limited to ASCII (or Latin-1).

---

[1]: Names taken from the W3C’s article on personal names around the world, http://www.w3.org/International/questions/qa-personal-names

append delete 26 Dec ’12 · 19:53 #4. Kroc

Could you perhaps look at the IDN spec to see what code-points / ranges they've excluded? There may also be libraries out there that process IDNs, and therefore include anti-spoofing. I'm not quite sure how one would make something accurate without having to paint a broad brush across the unicode space because it's unreasonable for anybody to have to go through the whole unicode space to find untrustworthy symbols.

append delete 23 Apr ’13 · 09:58 #5. Akerhielm

"
Could you perhaps look at the IDN spec to see what code-points / ranges they've excluded? [...]
"
This will rather unhelpful since the IDN spec is very lenient, only forbidding code points with property "Separator" (Z) and "Other" as you may remember from this pull request to NNF
https://github.com/Kroc/NoNonsenseForum/pull/118

"
[...]There may also be libraries out there that process IDNs, and therefore include anti-spoofing. I'm not quite sure how one would make something accurate without having to paint a broad brush across the unicode space because it's unreasonable for anybody to have to go through the whole unicode space to find untrustworthy symbols.
"
TLDs that are deemed "trustworthy" by browsers, to have their domainnames displayed as unicode all have rather short, individual white-lists.
For example the Swedish TLD ".se" allow "0-9,a-z,å,ä,ö" if I recall correctly.

:: @Akerhielm added on 23 Apr ’13 · 09:59

Turns out I'm an idiot and forgot how to quote.

append delete 24 Apr ’13 · 15:48 #6. Martijn

Hi @Akerhielm! Thanks for dropping in on this conversation. You might want to check out the recently sparked discussion after I spoofed Kroc’s name here on the forum: http://forum.camendesign.com/shuting-down-website-for-a-month+1#1to6h29w2xol

There is now a NoNonsenseForum issue on points 1 and 2, whitespace, we would appreciate all input: https://github.com/Kroc/NoNonsenseForum/issues/175 (Normalising space and invisible characters in names.)

If you know your way around the Unicode specification also check out the issue on validating bytes: https://github.com/Kroc/NoNonsenseForum/issues/176

It is no longer a theoretical problem I am having, but also something we would love to land in the NNF core.

append delete 27 Apr ’13 · 15:24 #7. Kroc

With a baby coming in a couple of months, there's not much time for me to be working on this, but I'm itching to get on to it.

append delete 28 Apr ’13 · 03:41 #8. theraje

Why is Reply #1 completely blank? No user name, no "this post was deleted by a moderator", just a timestamp and an empty space where content would normally be...

append delete 28 Apr ’13 · 09:38 #9. Kroc

It's the very problem we're discussing :) -- abuse of Unicode to fake posts and names!

append delete 28 Apr ’13 · 16:50 #10. Martijn

@theraje:

And hopefully once [1] is done this particular Unicode abuse will no longer be possible. The character used has the ‘White_Space’ property and should just be replaced with a normal space ( ). At least that’s what we are currently looking at.

The problem is size. Unicode 6.2 contains 110 182 different characters. How are we to know what to censor and what not? And when?

For instance I used a thin space (U+2009) as a thousands separator in that number. Should this be allowed or not? Should it be removed or should it be changed to a full space? Or may it is allowed in text but not in names? Many decisions, few experts.

[1]: https://github.com/Kroc/NoNonsenseForum/issues/175

append delete 28 Apr ’13 · 20:10 #11. theraje

Sorry, didn't realize that. I saw the OP, then a blank reply, then Kroc replying to the OP. It gave me the impression that something happened recently to make Reply #1 disappear somehow. (I didn't relate the following posts to the invisible post issue. I'm cool like that.)