1028988 - Sort order changed in Firefox 30

Status: RESOLVED WONTFIX

(Firefox :: Untriaged, defect)

Description

[web_mas]

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0 (Beta/Release)
Build ID: 20140605174243



Actual results:

In the 30 FF changed the sort order.
Should be so: digits --> English --> at the end of the Cyrillic
Became so: digits --> Cyrillic --> then English.
Check this please.

Sorry my bad English.

Comment 1

[:Gijs (he/him)]

Simon, do you know if there was an intentional change here as part of some of the i18n integration stuff that happened for Firefox 29/30?

Reporter, can you clarify when you last saw this working, and what exact sort order you mean? The default JS Array.sort(), or something else?

Comment 2

[Infocatcher]

Looks like this is about following:

"abcdABCDабвгАБВГ0123".split("").sort(function(a, b) {
	return a.localeCompare(b);
}).join("");
// In Firefox 24: 0123aAbBcCdDаАбБвВгГ - numbers - English/Latin - Cyrillic
// In Firefox 30: 0123аАбБвВгГaAbBcCdD - numbers - Cyrillic - English/Latin

Comment 3

[web_mas]

Attachment: img

Comment 4

[web_mas]

(In reply to :Gijs Kruitbosch from comment #1)
> or something else?
https://bug1028988.bugzilla.mozilla.org/attachment.cgi?id=8444522

Comment 5

[Simon Montagu :smontagu]

This would be caused by bug 769871. Waldo will know better than I whether the changed behaviour is correct by spec.

Comment 6

[Simon Montagu :smontagu]

That said, I can't reproduce the bug with the example from comment #2

> "abcdABCDабвгАБВГ0123".split("").sort(function(a, b) {
> 	return a.localeCompare(b);
> }).join("");
> // In Firefox 24: 0123aAbBcCdDаАбБвВгГ - numbers - English/Latin - Cyrillic
> // In Firefox 30: 0123аАбБвВгГaAbBcCdD - numbers - Cyrillic - English/Latin

On OSX mozilla-central nightly I get 0123aAbBcCdDаАбБвВгГ (and the same in Chrome and Safari FTR)

Comment 7

[:Gijs (he/him)]

(In reply to Simon Montagu :smontagu from comment #6)
> That said, I can't reproduce the bug with the example from comment #2
> 
> > "abcdABCDабвгАБВГ0123".split("").sort(function(a, b) {
> > 	return a.localeCompare(b);
> > }).join("");
> > // In Firefox 24: 0123aAbBcCdDаАбБвВгГ - numbers - English/Latin - Cyrillic
> > // In Firefox 30: 0123аАбБвВгГaAbBcCdD - numbers - Cyrillic - English/Latin
> 
> On OSX mozilla-central nightly I get 0123aAbBcCdDаАбБвВгГ (and the same in
> Chrome and Safari FTR)

Ditto here with Firefox 30 (in hebrew, for reasons unrelated to this bug) on OS X. I wonder if this has to do with the OS (language)...

Comment 8

[Infocatcher]

Also the same (https://bugzilla.mozilla.org/show_bug.cgi?id=1028988#c2) on latest Firefox 33.0a1 (2014-06-23).
This is under Windows 7 x86 with Russian localization (even on non-localized Firefox Nightly).

Comment 9

[Masatoshi Kimura [:emk]]

Works for me with Japanese locale. Specific to Russian locale?

Comment 10

[Simon Montagu :smontagu]

(In reply to Masatoshi Kimura [:emk] from comment #9)
> Works for me with Japanese locale. Specific to Russian locale?

That seems likely:

"abcdABCDабвгАБВГ0123".split("").sort(function(a, b) {
	return a.localeCompare(b, "ru");
}).join("");

gives me 0123аАбБвВгГaAbBcCdD

This is much the same as bug 999003, and I still find it surprising that the ES i18n API is not backwards compatible, i.e. that functions like localeCompare and toLocaleString with undefined locale have changed behaviour.

Comment 11

[Jeff Walden [:Waldo]]

(In reply to :Gijs Kruitbosch from comment #7)
> I wonder if this has to do with the OS (language)...

Yes.  The change here basically makes strX.localeCompare(strY) consistent with the behavior of new Intl.Collator(undefined, undefined).compare(strX, strY), which by passing in undefined for the locale list (first argument) calls intrinsic_RuntimeDefaultLocale, which consults setlocale(LC_ALL, nullptr) if it's available or LANG if not.  If the CLDR data says the locale that gets used, sorts differently than expected, you'd get unexpected sort results.

I can't intelligently comment on whether the sort order for Hebrew, or Russian, or Japanese, or whatever, is right or wrong.  That's a CLDR issue.  Given that CLDR doubtless gets far more attention than the nsICollation created from nsILocaleService likely did, I'd be inclined to trust CLDR as more right, and our old thing as more wrong.  Simon can comment far more intelligently on that, I expect, than I can.

Comment 12

[Phobos]

Hello. Is there any chance, that this issue will be resolved in a future versions of FF (Russian locale)?
Thanks.

Comment 13

[:Gijs (he/him)]

(In reply to phobos from comment #12)
> Hello. Is there any chance, that this issue will be resolved in a future
> versions of FF (Russian locale)?
> Thanks.

I think so far, we're not even sure this is actually a bug... if you can provide more details as to why this behaviour is wrong per se, that may be helpful. Also needinfo'ing Simon per comment #11...

Comment 14

[web_mas]

> I think so far, we're not even sure this is actually a bug...

Of course it was a bug. And it needs to be fixed!

Comment 15

[Simon Montagu :smontagu]

(In reply to Jeff Walden [:Waldo] (remove +bmo to email) from comment #11)

> I can't intelligently comment on whether the sort order for Hebrew, or
> Russian, or Japanese, or whatever, is right or wrong.  That's a CLDR issue. 
> Given that CLDR doubtless gets far more attention than the nsICollation
> created from nsILocaleService likely did, I'd be inclined to trust CLDR as
> more right, and our old thing as more wrong.  Simon can comment far more
> intelligently on that, I expect, than I can.

I don't think this is a binary "right" or "wrong" issue. The sort order digits > Cyrillic script > Latin script is as specified by CLDR as the default ordering for the ru locale:

collation/ru.xml (in http://unicode.org/Public/cldr/25/core.zip) includes the line:
 <settings reorder="Cyrl"/>

and following the rules at http://www.unicode.org/reports/tr35/tr35-collation.html#Interpretation_reordering gives us the "core codes" space, punct, symbol, currency, and digit before Cyrillic, and all other scripts after it.

All that said, if this behaviour isn't what users expect, maybe we should be overriding it? I have no idea how to assess that in general. On the face of it, it certainly seems logical to have "native" script sorted before "foreign" scripts, and I assume that is why CLDR defines it that way.

(In reply to web_mas from comment #14)
> Of course it was a bug. And it needs to be fixed!

As I said above, it isn't clear to me that this is "of course" a bug. It's certainly a change, but why is the old behaviour better than the new?

Comment 16

[web_mas]

> it isn't clear to me that this is "of course" a bug. It's
> certainly a change, but why is the old behaviour better than the new?

Because this sort order the de facto standard. In all programs in all operating systems.
It's like a traffic light green place in the middle - it's stupid and wrong.

Comment 17

[Jeff Walden [:Waldo]]

The old sort order is not a standard.  Nor, really, is the new one a standard.  The Internationalization spec doesn't specify a particular ordering for this stuff, because languages change over time, and it's impossible to reasonably specify a consistent ordering that will accord with actual use over time.

To the extent there's a standard, that standard is CLDR.  It *is* used in all programs in all operating systems and has no serious competitor (to the best of my understanding).  If you have complaints about its orderings, complain to them.  Further research prompted by comment 15 indicates this was a deliberate change, confirmed by Russian localizer experts:

http://unicode.org/cldr/trac/ticket/5577#comment:3
http://unicode.org/cldr/trac/changeset/8059/trunk/common/collation/ru.xml

If you have concerns, please take it up with the CLDR folks in their bug database.

My understanding is that Chrome, IE, Safari (?), and everyone now uses CLDR data to determine sort order.  Some may just be using older CLDR data than we are.  Note that Chrome has upgraded to newer CLDR, and their bleeding-edge builds (per https://code.google.com/p/chromium/issues/detail?id=132145 as of ~July 18) have the same behavior as we do.

At this point, barring the CLDR change being reverted, I think this is WONTFIX.

Comment 18

[Phobos]

(In reply to :Gijs Kruitbosch from comment #13)
> ... if you can provide more details...

There're a couple of examples, if you don't mind.

   1) I've checked this issue in IE 11 (Windows 7 x64). When I select Ordering by Name in Favorites, sorting is done as before (English-->Cyrillic). In Windows Explorer all the folders and files are sorted the same. So it was a long time in Windows (Russian locale).
   2) Mac OS X has the option named "Order for sorted lists" (pop-up menu in System Preferences - Language and Text). I can choose Russian or Universal. Universal item resolve the problem with sorting. It may be possible to add a similar option in FF?
   Thanks for attention.