UTF-8 in .NET, revisited

A couple of months ago I described how I managed to get Eiffel for .NET to work with UTF-8 Unicode strings. The solution given in UTF-8 Unicode in Eiffel for .NET seems to be working fine, but we've noticed that our VB application is running more slowly than before. Well I did say, "This implementation is no doubt inefficient..."

Another problem with that solution is that it creates a dependency on Gobo. This wasn't a problem for us, but it might bother others.

Because we had a performance problem, I started profiling my application. EiffelStudio's built-in profiler doesn't work in .NET (despite the fact that Project Settings misleadingly offers this as an option in .NET projects), so I used NProf, a free profiler for .NET. I tried NProf 0.10, the latest, but I found it strangely minimalist. Then I tried NProf 0.9.1, and it was much better, because it gives more options for viewing the results of a profiling run. (I wonder whether NProf is being rewritten from scratch.)

NProf showed me that, in a common usage scenario, our application was spending 18.43% of its time in STRING.make_from_cil! Most of this was spent in one of the routines whose implementation I had overridden: 16.25% of the total time was in SYSTEM_STRING_FACTORY.read_string_into. To quantify just how bad my UTF-8 implementation was, I removed it and profiled again: NProf showed 2.66% and 0.47% respectively.

I saw that make_from_cil and read_string_into were being called 39,000 times, and that most of these calls were from a particular STRING function of ours that concatenates strings and string constants. This function was amenable to optimisation by caching its STRING result in an attribute. This worked: it noticeably improved performance. According to NProf, there were now only 9,738 calls, reducing the respective percentages of total time to 11.47% and 10.85%. Better, but still bad. Could I optimise SYSTEM_STRING_FACTORY itself?

My override of SYSTEM_STRING_FACTORY was using Gobo's UC_UTF8_STRING to convert UTF-8 bytes to characters. In .NET, however, there is an obvious alternative: the System.Text.UTF8Encoding class. This class has various methods for encoding and decoding between .NET String objects and character arrays, on the one hand, and .NET Byte arrays on the other. The strings and character arrays are encoded in UTF-16; the byte arrays are encoded in UTF-8.

The following rewrite of SYSTEM_STRING_FACTORY.read_string_into takes only 1.10% of the application's total time, a dramatic improvement which of course is reflected in STRING.make_from_cil, which now takes only 1.81%. The application runs noticeably faster too!

local i, nb: INTEGER l_str8: STRING bytes: NATIVE_ARRAY [NATURAL_8] do if a_result.is_string_8 then i := a_str.length create bytes.make ({ENCODING}.utf8.get_max_byte_count (i)) i := {ENCODING}.utf8.get_bytes (a_str, 0, i, bytes, 0) l_str8 ?= a_result l_str8.make (i) l_str8.set_count (i) {SYSTEM_ARRAY}.copy (bytes, l_str8.area.native_array, i) else

This new implementation creates a .NET Byte array (a NATIVE_ARRAY [NATURAL_8], in Eiffel-speak) large enough to hold the biggest possible UTF-8 encoding of the .NET String (a SYSTEM_STRING in Eiffel-speak). It then calls UTF8Encoding.GetBytes() to encode the UTF-16 characters in the String as UTF-8.

Finally, it copies these bytes straight into the STRING result's native_array. This part is tricky; it took me quite a while to understand what I needed to do. The Eiffel STRING's native_array is a .NET array of .NET characters (a NATIVE_ARRAY [CHARACTER_8], in Eiffel-speak). Because .NET characters are UTF-16, you might expect that the native_array would be UTF-16 too. I sure did. But it isn't; it's UTF-8. Only the least significant of each character's two bytes is used by normal Eiffel code. This can get really confusing, because it is possible, via .NET classes, to stuff native_array with UTF-16 characters; this can produce weird logic errors, such as when the EiffelStudio 5.7 debugger told me that a particular character a_char had the ordinal value 45, and a debugger watch expression told me that a_char <= 127 was True, but the running program evaluated a_char <= 127 as False. Weird! After many hours, I figured out that the character's ordinal value was actually not 45, but that it had something in the high byte due to UTF-16 encoding. Once I understood this important point, I realised that I simply needed to copy the UTF-8 bytes straight into the native_array. Simple!

This dealt with the worst inefficiency, but I decided to tackle SYSTEM_STRING_FACTORY.from_string_to_system_string too.

nb := a_str.count create bytes.make (nb) from i := 1 until i > nb loop bytes.put (i - 1, a_str.code (i).to_natural_8) i := i + 1 end Result := {ENCODING}.utf8.get_string (bytes)

UTF8Encoding.GetString() helps out, by decoding the bytes in the STRING's native_array to create a .NET SYSTEM_STRING. That's all there is to it. The only complication is the loop, which converts the character array native_array into the byte array bytes.

My new override of SYSTEM_STRING_FACTORY is attached. Like the old implementation, it assumes that we are working with UTF-8 strings.