Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Perhaps it's time to agree upon how to Unicode in identifiers? The normalization, unprintable characters, confusing characters with same glyphs, etc. It's obviously problematic when everyone is doing it on their own.


The first bar to clear is "The Turkish Test"[0], then we can talk about Unicode. It'll smooth the rest of the process a lot.

You can't guess how many workarounds I implement to make sure that a stray application doesn't get "ı" or "İ" in their naive codepaths, and start burning mid-flight (e.g.: Kodi, Pagico, some old Java programs, oh my...).

[0]: https://blog.codinghorror.com/whats-wrong-with-turkey/


The date format part is ridiculous. Americans are almost unique in using mm/dd/yyyy, so an assumption of that would be plain wrong.


Localization libraries handle these parts well, since date is same with Europe (and generally stored as time-date objects rather than pure strings). None of the number shenanigans cause problems since these numbers are always stored as IEEE754 or other decimal formats. Money is no problem as well.

However, when you go through an upper() or lower() or anything which plays with capitalization, and if that data is being fed to a hash algorithm or anything which mucks with strings, boy, oh boy...

The easiest way is to sanitize these programmatic parts with forced locale of en_US or plain old "C". If the strings is not facing to the user and never localized, just force its locale. It's the only sane way.


> since date is same with Europe

Do you mean MM-DD-YYYY? No, the vast majority of Europe does DD-MM-YYYY in some form or another.


No, I mean DD-MM-YYYY. We use the same format with the vast majority of Europe.


I'm confused, are you talking about the US? The US for sure does not default to DD-MM-YYYY.


I'm talking about the Turkish Language, its locale and its peculiarities since it has letters "ı" and "İ".

So, I'm talking about Turkish date format. Turkey uses DD-MM-YYYY format, like the most of the Europe.


I have switched to yyyymmdd for everything - it is usually obvious to everyone what date I mean.


I also use the same format while naming my files, or in changelogs or whatnot, but not all documents are suitable for that, and in the presentation layer you need to match the country standards.

However, date is mostly presentation and internal storage of these are vastly different than what we see generally.


I don't match country standards. That is the point.


It depends on what you're doing, though. If you're helping people fill out documents (even non-government documents), then you really need to match the country standard.

Localization is important; some countries outright require it if you're going to do business within their borders. But even where it's not required, you will lose customers if your website/application/product feels "foreign". I'm not sure date ordering is a big enough deal to trigger that feeling in anyone, but unless it's a huge burden to format things the way people expect, I would do so for the UX benefits.


https://xkcd.com/1179/ I heard the US and A are moving to the hissing cat date format shown here.


I kind of like the one using roman numerals for month. Reasonable people would figure out that other reasonable people would not use roman numerals for days, so the order can be implicit. I like implicit ordering, it always makes things more interesting.


Unicode has provided a specification for Unicode identifiers since 2005: https://www.unicode.org/reports/tr31/


Great! Is there a library for their validation? ICU seems to have only spoof checker for confusables.


ICU: International Components for Unicode: https://en.wikipedia.org/wiki/International_Components_for_U...

unicode-org/icu: https://github.com/unicode-org/icu

Microsoft/ICU: https://github.com/microsoft/icu

IDN: Internationalized domain name: https://en.wikipedia.org/wiki/Internationalized_domain_name

Punycode: https://en.wikipedia.org/wiki/Punycode

IDN homograph attack: https://en.wikipedia.org/wiki/IDN_homograph_attack

CWE-1007: Insufficient Visual Distinction of Homoglyphs Presented to User: https://cwe.mitre.org/data/definitions/1007.html

GNU libidn/libidn2: https://gitlab.com/libidn/libidn2

Comparison of regular expression engines > Language features > Part 2; Unicode support: https://en.wikipedia.org/wiki/Comparison_of_regular_expressi...


libu8ident


rurban/libu8ident : https://github.com/rurban/libu8ident :

> unicode security guidelines for identifiers


As long as I can enter my Zalgo[1] username, I'm fine with your suggestion.

[1]: https://en.wikipedia.org/wiki/Zalgo_text


> Perhaps it's time to agree upon how to Unicode in identifiers?

And then update all data structures that refer to them (like last and w/who, also NFS), as well as file formats (like cpio, tar, and pax which encodes ownership).


Yes. Those formats have had 20 years since Unicode was standardised, and things like my terminal still routinely break when given “unexpected” inputs. Practically every other application can handle it.


Good luck bringing everyone together. There's still a ton of Microsoft software that relies on the presence of the BOM [1], despite practically everyone else not using it. And bidirectional rsync between practically everything else and a Mac still requires `--iconv=utf-8,utf-8-mac` to avoid problems because of homographs.

[1] https://en.wikipedia.org/wiki/Byte_order_mark


Would punycode be suitable?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: