One of the more interesting technical support incidents of the past week comes from Mohit Kumar Sethi at 3i Infotech Consumer Services Ltd.
Mohit is using NowSMS Lite to send binary SMS messages to a Java application that his company is developing.
He noticed that when sending SMS binary messages longer than 140 bytes, the messages were received correctly by Nokia and SonyEricsson devices, however many Motorola devices received corrupted messages.
In GSM environments, a single SMS message can be no longer than 140 bytes.
When you send a text message, as long as the text only contains characters that are included in the GSM 7-bit character set , 160 7-bit characters are compressed into 140 8-bit bytes to produce the 160 character limit that we are so familiar with. (Note: 160 * 7 = 140 * 8)
There's more general discussion of SMS message size limits in the article at http://blog.nowsms.com/2007/06/long-sms-text-messages-and-160.html.
But back to Mohit's problem...
When sending a binary SMS message longer than 140 bytes, it has to be segmented into multiple SMS messages. A special header is added to each physical SMS message so that the receiving client knows that it is a multipart SMS message that must be reassembled by the client. These headers are known as segmentation or concatenation headers. 6 bytes (8-bits each) are required for these concatenation headers in each physical SMS message. These headers are placed in the User Data Header (UDH) field of the message, but they do count against the overall size limit of the message.
As a result, a long binary SMS message will be divided into 134 byte segments. Or 153 characters per segment for standard text messages. Or 67 Unicode characters per segment for Unicode text messages.
Mohit observed that with various Motorola RAZR and SLVR models, the received message would lose one byte for each of these segments.
He had his applications on the phones send long binary SMS messages back in to NowSMS.
When his application on a Nokia or SonyEricsson device sent a long binary SMS message, it generated a User Data Header like this: 0B0504232923290003070301
0B is the length of the user data header.
05 indicates that source and destination port information is present in the message.
04 indicates the length of the source and destination port information.
2329 2329 indicates source and destination ports of 2329.
00 indicates that the message is segmented using an 8-bit reference number.
03 indicates the length of the segmentation headers.
07 is the 8-bit reference number for the segmented message.
03 is the number of segments in the overall message.
01 is the number of the current segment (i.e,. 1 of 3).
Long binary messages sent by the Motorola devices were different. A typical user data header looked like this: 0C080400460301050423292329
0C is the length of the user data header.
08 indicates that the message is segmented using a 16-bit reference number.
04 indicates the length of the segmentation headers.
0046 is the 16-bit reference number for the segmented message.
03 is the number of segments in the overall message.
01 is the number of the current segment (i.e,. 1 of 3).
05 indicates that source and destination port information is present in the message.
04 indicates the length of the source and destination port information.
2329 2329 indicates source and destination ports of 2329.
Whether the segmentation headers or port information headers was not significant. But what was significant was that the Motorola device was generating messages that used 16-bit reference numbers.
Mohit determined that when the Motorola device received a long binary message using an 8-bit reference number, it converted this to a 16-bit reference number. If the overall message length including UDH was 140 before this conversion, the conversion would cause the last byte of data in that segment to be lost.
The Motorola device didn't seem to have any problems with standard long text messages being sent to the SMS client on the device, or with long binary WAP push messages. The bug seems to be in the part of the device code that routes messages to Java applications.
To make a long story short, this is an annoying bug. And if you're sending messages to Java applications, you may encounter it.
To deal with this scenario, we added a configuration option to NowSMS and NowSMS Lite, which would cause NowSMS to use 16-bit reference numbers when generating concatenated multipart messages. By default, NowSMS uses 8-bit reference numbers, which allows the most characters to be fit in a single SMS.
When 16-bit reference numbers are used, message length per segment rules change. For long text messages, instead of 153 characters per segment, only 152 characters per segment can be achieved.
For Unicode messages, instead of 67 characters per segment, only 66 characters per segment can be achieved.
For binary messages, 133 bytes of binary data can be included per segment, instead of 134 bytes.
This 16-bit reference number support was added in the 2009.11.04 release of NowSMS. This support will be carried through to future releases. For now v2009.11.04 has been made available as a special release at http://www.nowsms.com/download/nowsms20091104.zip (standard version) and http://www.nowsms.com/download/lite20091104.zip (NowSMS Lite).
To enable 16-bit reference numbers, edit SMSGW.INI, and under the [SMSGW] section header, add ConcatRef16Bit=Yes.
Note that when this setting is applied, 16-bit reference numbers are only used when NowSMS performs the segmentation (message submitted either via HTTP, or via SMPP using a single message_payload submission) ... NowSMS does not resegment messages that have already been segmented by the submitting client.
For more about sending SMS messages to J2ME apps, see http://blog.nowsms.com/2007/06/send-sms-to-java-appmidlet.html.
