XML Status Query for SMSC Connection Status and Statistics

We posted details about an XML-based NowSMS status query in an earlier posting: http://blog.nowsms.com/2009/02/xml-status-query-for-smsc-connection.html

This query reports information similar to what is reported on the "Status" page of the NowSMS configuration dialog. The query results include information about SMSC connection status, the number of messages processed via the different connections, and the number of messages pending in the queues, among other information.

The format of the XML data given in the example is relatively straight-forward, but in some environments it may make it easier to have an XSD file to document the format. An XSD file can be downloaded at the following link http://www.nowsms.com/download/xmlstatus.xsd.txt.

For additional information on the XML status query, please refer to the earlier posting: http://blog.nowsms.com/2009/02/xml-status-query-for-smsc-connection.html

Now SMS Group Text Messaging

The NowSMS Group Text feature is designed to facilitate group communications over SMS.

At it's core, it's about sending a single text message, and having that text message automatically forwarded to a group of people.

NowSMS Group Text is a free add-on for the Now SMS/MMS Gateway and NowSMS Lite.

NowSMS Group Text will only work with NowSMS or NowSMS Lite versions 2008.06.03 or later. It will not work with earlier versions of NowSMS.

NowSMS Group Text is a work in progress. Before making the first official release available, we wanted to make a pre-release available for download, and get feedback from existing NowSMS customers.

The best way to explain NowSMS Group Text is to describe some of the different possible usage scenarios.

Smaller teams and workgroups can use NowSMS Group Text to easily share information and communicate as a group. Any member of a group can send a text message that is automatically forwarded and rebroadcast to the entire group.

Larger teams and workgroups can limit which group members are allowed to send a broadcast message to the entire group. Authorised users can send broadcast messages to the group, while standard users only receive these broadcast messages.

It is also possible for end users to opt-in and opt-out of group communications by sending text commands to the NowSMS Group Text server via SMS. Group managers and/or event organisers can create groups and invite users to subscribe to group broadcasts by sending a text message to the NowSMS Group Text Server, freeing them of the need to manually manage subscriber lists.

NowSMS Group Text is implemented using the 2-way command interface of NowSMS, and is compatible with any SMSC interface that supports both sending and receiving SMS messages (GSM modems, SMPP, UCP/EMI, CIMD2). It is also possible to receive SMS message using one SMSC interface, such as a GSM modem, while sending outbound messages via a different SMSC connection, such as an SMPP or HTTP based provider.

Installing NowSMS Group Text

NowSMS Group Text will only work with NowSMS or NowSMS Lite versions 2008.06.03 or later. It will not work with earlier versions of NowSMS.

The Group Text add-on can be downloaded at http://www.nowsms.com/download/grouptext.zip.

For product support, please visit the discussion forum at http://www.nowsms.com/grouptext.

The current pre-release version of the NowSMS Group Text feature requires the manual copying of 2 files to the NowSMS program directory (usually C:\Program Files\NowSMS or C:\Program Files (x86)\NowSMS on 64-bit versions of Windows): SMSGROUPER.EXE and SMSGROUPCMD.EXE.

In NowSMS, on the “2-way” page of the configuration dialog, SMSGROUPER.EXE must be installed as a 2-way command.

To define SMSGROUPER.EXE as a 2-way command, use the following settings:

• SMS Command Prefix = *
• Command To Execute = "C:\Program Files\NowSMS\smsgrouper.exe" /s @@SENDER@@ /r @@RECIP@@ /t @@FULLSMS@@
• Command returns response text = checked

For most installations, “Receive Phone Number(s)” can be left blank. Leaving this field blank means that the SMSGROUPER command will process all received SMS messages. If you have multiple GSM modems or are receiving SMS messages for multiple phone numbers or short codes, then you should specify the phone number or short code that is to be allocated for Group Text in this field.

Creating a Group

The current user interface for NowSMS Group Text is designed to allow groups to be created and maintained via SMS.

A command line utility is also provided so that admin commands can be issued from the computer running the NowSMS Group Text server. All of the administrative commands supported by the SMS interface are supported from the command line interface by running SMSGROUPCMD.EXE and including the command on the command line after SMSGROUPCMD.

Before any groups can be created, a special user account must be created on the NowSMS server in the “SMS Users” area. The user name give to this user account should be the phone number of the owner (or administrator) of the group. This phone number is not the phone number allocated for sending/receiving group text messages. This is a phone number that is allowed to send administrative commands into the group text server for group creation and/or maintenance. In most cases, this phone number should be defined in full international format starting with a “+” (e.g., +4477777777777).

Once this user account is created, it is possible to send a message from the group owner phone number into the group text server to create a group.

This can be done by simply sending a text message that says: CREATE groupname (where groupname is the name of the group to be created).

Note: When using the command line interface, it is necessary to include the owner phone number following the group name. (For example: SMSGROUPCMD CREATE groupname ownerPhoneNumber)

Several parameters can be specified in the CREATE command to specify group attributes. These parameters can be added to the end of the CREATE command.

PUBLIC or PRIVATE – If the group is PUBLIC (default), then anyone will be allowed to join the group. If the group is PRIVATE, any join requests will be forwarded to the group owner and/or administrator for approval before joining.

MOD, UNMOD or BROADCAST – This specifies whether or not group members are allowed to submit messages that are forwarded/rebroadcast to the entire group. An UNMODerated (default) group means that and any group member can send a message to the group. A MODerated group means that specially authorised group members can send a message to the group, messages sent by regular group members will be forwarded to the group owner and/or administrator for approval before being sent to the group. A BROADCAST group means that only specially authorised group members can send a message to the group.

NOTIFY or NONOTIFY – This specifies whether or not the group owner and/or administrators are notified with a text message whenever a member joins or leaves the group. NONOTIFY (default) means that the group owner is not notified. NOTIFY means that the group owner is notified whenever a new member joins the group, or an existing member leaves the group.

Modifying Group Properties

These group attributes can be changed after a group has been created by sending the command ADMIN groupname PROP followed by any of the attributes described in the previous section.

Authorised Members for Broadcast or Moderated Groups

For broadcast or moderated groups, members that are allowed to send messages to the group without moderator approval can be managed via the ADMIN groupname AUTH command.

To add a member to the authorised list, use ADMIN groupname AUTH ADD phonenumber

To remove a member from the authorised list, use ADMIN groupname AUTH REMOVE phonenumber

The command line interface also supports an additional parameter to return a list of currently authorised members: ADMIN groupname AUTH LIST

Specifying Alternative Administrators

For moderated and private groups, administrators must approve certain transactions (new members and/or message postings) before they are processed. By default, the group owner is an administrator. It is also possible to assign additional administers.

To add an administrator, use ADMIN groupname ADMIN ADD phonenumber

To remove an administrator, use ADMIN groupname ADMIN REMOVE phonenumber

The command line interface also supports an additional parameter to return a list of administrators: ADMIN groupname ADMIN LIST

Adding Members

Group owners and administrators are allowed to add new members to a group without requiring the member to first send a message asking to join.

To add a member, use the command ADMIN groupname ADD phonenumber username

Removing and/or Blocking Members

Group owners and administrators are allowed to remove members from a group, and optionally block the member from rejoining the group.

To remove a member, use the command ADMIN groupname REMOVE phonenumber or username

To remove a member and block them from rejoining the group, use the command ADMIN groupname BLOCK phonenumber or username

Listing Group Members

The command line interface supports a command to return a list of all group members. To return this member list, using the command ADMIN groupname LIST

Posting a Message to The Group

The command line interface supports a command to post a message to the group. The format of this command is: ADMIN groupname POST text to post.

To post group messages via SMS, please refer to the next section “Joining Groups and Posting Messages to a Group”.

Joining Groups and Posting Messages to a Group

Members can join a group by sending a message to the group text server with the text JOIN groupname username

Groupname is the name of the group that they are asking to join. Username is a user handle that is used when they post messages to the group.

Members can leave a group by sending a message to the group text server with the text LEAVE groupname

Members can also leave all groups by sending a message to the group text server with the text LEAVE or STOP, without specifying a group name.

Members can post a message to a group by sending a message to the group text server that starts with the text @groupname, where groupname is the name of the group. For example, if the group is named simply “B”.

@B This is a message to the entire group.

If the group is moderated, the message will be first forwarded to the group administrator for approval before being posted to the entire group.

Messages sent through the group text server will be prefaced with the text @groupname:username.

It is also possible for members to send a text message directly to another member of the group without sending it to the entire group by sending a message to the group text server that starts with the text @groupname:username. For example, to send to a member “mike” on a group named “B”, the following text message could be sent:

@B:mike Hey Mike, let's get out of here!

Bug Sending Long Binary SMS Messages to Java Apps on Motorola Devices

One of the more interesting technical support incidents of the past week comes from Mohit Kumar Sethi at 3i Infotech Consumer Services Ltd.

Mohit is using NowSMS Lite to send binary SMS messages to a Java application that his company is developing.

He noticed that when sending SMS binary messages longer than 140 bytes, the messages were received correctly by Nokia and SonyEricsson devices, however many Motorola devices received corrupted messages.

In GSM environments, a single SMS message can be no longer than 140 bytes.

When you send a text message, as long as the text only contains characters that are included in the GSM 7-bit character set , 160 7-bit characters are compressed into 140 8-bit bytes to produce the 160 character limit that we are so familiar with. (Note: 160 * 7 = 140 * 8)

There's more general discussion of SMS message size limits in the article at http://blog.nowsms.com/2007/06/long-sms-text-messages-and-160.html.

But back to Mohit's problem...

When sending a binary SMS message longer than 140 bytes, it has to be segmented into multiple SMS messages. A special header is added to each physical SMS message so that the receiving client knows that it is a multipart SMS message that must be reassembled by the client. These headers are known as segmentation or concatenation headers. 6 bytes (8-bits each) are required for these concatenation headers in each physical SMS message. These headers are placed in the User Data Header (UDH) field of the message, but they do count against the overall size limit of the message.

As a result, a long binary SMS message will be divided into 134 byte segments. Or 153 characters per segment for standard text messages. Or 67 Unicode characters per segment for Unicode text messages.

Mohit observed that with various Motorola RAZR and SLVR models, the received message would lose one byte for each of these segments.

He had his applications on the phones send long binary SMS messages back in to NowSMS.

When his application on a Nokia or SonyEricsson device sent a long binary SMS message, it generated a User Data Header like this: 0B0504232923290003070301

0B is the length of the user data header.

05 indicates that source and destination port information is present in the message.

04 indicates the length of the source and destination port information.

2329 2329 indicates source and destination ports of 2329.

00 indicates that the message is segmented using an 8-bit reference number.

03 indicates the length of the segmentation headers.

07 is the 8-bit reference number for the segmented message.

03 is the number of segments in the overall message.

01 is the number of the current segment (i.e,. 1 of 3).

Long binary messages sent by the Motorola devices were different. A typical user data header looked like this: 0C080400460301050423292329

0C is the length of the user data header.

08 indicates that the message is segmented using a 16-bit reference number.

04 indicates the length of the segmentation headers.

0046 is the 16-bit reference number for the segmented message.

03 is the number of segments in the overall message.

01 is the number of the current segment (i.e,. 1 of 3).

05 indicates that source and destination port information is present in the message.

04 indicates the length of the source and destination port information.

2329 2329 indicates source and destination ports of 2329.

Whether the segmentation headers or port information headers was not significant. But what was significant was that the Motorola device was generating messages that used 16-bit reference numbers.

Mohit determined that when the Motorola device received a long binary message using an 8-bit reference number, it converted this to a 16-bit reference number. If the overall message length including UDH was 140 before this conversion, the conversion would cause the last byte of data in that segment to be lost.

The Motorola device didn't seem to have any problems with standard long text messages being sent to the SMS client on the device, or with long binary WAP push messages. The bug seems to be in the part of the device code that routes messages to Java applications.

To make a long story short, this is an annoying bug. And if you're sending messages to Java applications, you may encounter it.

To deal with this scenario, we added a configuration option to NowSMS and NowSMS Lite, which would cause NowSMS to use 16-bit reference numbers when generating concatenated multipart messages. By default, NowSMS uses 8-bit reference numbers, which allows the most characters to be fit in a single SMS.

When 16-bit reference numbers are used, message length per segment rules change. For long text messages, instead of 153 characters per segment, only 152 characters per segment can be achieved.

For Unicode messages, instead of 67 characters per segment, only 66 characters per segment can be achieved.

For binary messages, 133 bytes of binary data can be included per segment, instead of 134 bytes.

This 16-bit reference number support was added in the 2009.11.04 release of NowSMS. This support will be carried through to future releases. For now v2009.11.04 has been made available as a special release at http://www.nowsms.com/download/nowsms20091104.zip (standard version) and http://www.nowsms.com/download/lite20091104.zip (NowSMS Lite).

To enable 16-bit reference numbers, edit SMSGW.INI, and under the [SMSGW] section header, add ConcatRef16Bit=Yes.

Note that when this setting is applied, 16-bit reference numbers are only used when NowSMS performs the segmentation (message submitted either via HTTP, or via SMPP using a single message_payload submission) ... NowSMS does not resegment messages that have already been segmented by the submitting client.

For more about sending SMS messages to J2ME apps, see http://blog.nowsms.com/2007/06/send-sms-to-java-appmidlet.html.

SMS and MMS with the Novatel Merlin XU870 ExpressCard Modem

If you are using NowSMS in a laptop with an available ExpressCard slot, the Novatel Wireless Merlin XU870 ExpressCard Modem is a good modem to use with NowSMS to send and receive SMS and MMS messages.

This modem supports HSDPA up to 7.2 Mbps (although locating the appropriate firmware driver update seems to be a challenge, without it, it is limited to 3.6 Mbps). That is good for receiving MMS.

However, the modem does not support HSUPA, limiting it to 384 Kbps for uploads and sending MMS.

Although we have yet to test it, the Merlin X950D would likely be a better choice for MMS, as it claims to support HSUPA up to 2.1 Mbps (Note: the X950D requires a firmware update to enable HSUPA and there may be some difficulty locating the firmware update). SMS speeds should not be a difference between the two modems.

Drivers for this modem can be found on the Novatel Wireless web site at http://www.novatelwireless.com/index.php?option=com_content&view=article&id=220&Itemid=272. (If this direct link is broken, navigate from http://www.novatelwireless.com/support/.)

There are no special considerations for using the Novatel XU870 modem with NowSMS. However, it should be noted that this modem does NOT support delivery receipts.

For general information on configuring NowSMS to send and receive SMS and MMS messages with a GSM modem, such as the Novatel Wireless Merlin XU870, please see the NowSMS Quick Start Guide at http://www.nowsms.com/documentation/quickstartguide/.

Frequencies supported by the Novatel Merlin XU870:

3G/UMTS/HSDPA/HSUPA - 850Mhz/1900Mhz/2100Mhz (Good for most of the world, plus AT&T USA)

EDGE/GPRS - 850Mhz/900Mhz/1800Mhz/1900Mhz (Global)

HSDPA Max Performance: Download speed - 7.2 Mbps; Upload speed - 384 Kbps

Using NowSMS as a WAP Push Proxy Gateway (PPG)

NowSMS supports the Open Mobile Alliance (OMA) Push Access Protocol (PAP), which allows other applications to use NowSMS as a WAP Push Proxy Gateway (PPG).

Even though NowSMS has its own MMSC capability, we do have customers who are using NowSMS as a push proxy gateway for another vendor's MMSC.

In addition to providing PPG support for MMS, NowSMS can be used as a push proxy gateway for other types of push messages, including, but not limited to SyncML DM, SyncML DS, Wireless Village/IMPS, OMA Digital Rights Management, OMA E-Mail Notification (EMN), Service Indication and Service Load.

To post a push message via the PAP interface, perform an HTTP POST to the "web" port configured for the NowSMS web interface (not the MMSC), posting to a URL of "/pap". If user authentication is enabled in NowSMS, user identification can be included as URL parameters (e.g., "/pap?user=username&password=password"), or via an HTTP Basic Authentication "Authorization:" header.

A PAP post is a multipart MIME request, with the first MIME part being the PAP control document that specifies message recipient(s). The second MIME part is the data to be pushed.

An example of a service indication push being sent via PAP is shown below:

-----------------BEGIN EXAMPLE-----------------

POST /pap?user=username&password=password HTTP/1.0

Content-Type: multipart/related; boundary=mime-boundary; type="application/xml"

Content-Length: xxxxx

--mime-boundary

Content-Type: application/xml

--mime-boundary

Content-Type: application/xml

<?xml version="1.0"?>

<!DOCTYPE pap PUBLIC "-//WAPFORUM//DTD PAP 2.0//EN" "http://www.wapforum.org/DTD/pap_2.0.dtd"

[<?wap-pap-ver supported-versions="2.0,1.*"?>]>

<pap>

<push-message push-id="9fjeo39jf084@pi.com">

<address address-value="wappush=xxxxxxxxx/type=user@ppg.operator.com"></address>

</push-message>

</pap>

--mime-boundary

X-Wap-Application-Id: x-wap-application:wml.ua

Content-Type: text/vnd.wap.si

<?xml version="1.0"?>

<!DOCTYPE si PUBLIC "-//WAPFORUM//DTD SI 1.0//EN"

"http://www.wapforum.org/DTD/si.dtd">

<si>

<indication href="http://www.xyz.com/email/123/abc.wml" created="2009-10-25T15:23:15Z"

si-expires="2009-11-03T00:00:00Z">

You have 4 new emails

</indication>

</si>

--mime-boundary--

-----------------END EXAMPLE-----------------

Note that the "/type=user@ppg.operator.com" included in the WAP Push address specification is optional for NowSMS, and will be ignored, as only the relevant destination phone number is parsed from the request.

NowSMS will perform XML to WBXML conversions for WAP push of the following content types:

• text/vnd.wap.si
• text/vnd.wap.sl
• application/vnd.oma.drm.rights+xml
• text/vnd.wap.emn+xml (or application/vnd.wap.emn+xml)
• text/vnd.wap.co
• text/vnd.wap.connectivity-xml

Other types (including non WBXML based encodings such as "application/vnd.wap.mms-message" for MMS or "application/vnd.syncml.ds.notification" for SyncML SAN/DS) must be submitted via the appropriate binary encoding.

NowSMS supports the following text values for "X-Wap-Application-id":

• x-wap-application:push.sia
• x-wap-application:wml.ua
• x-wap-application:wta.ua
• x-wap-application:mms.ua
• x-wap-application:push.syncml
• x-wap-application:loc.ua
• x-wap-application:syncml.dm
• x-wap-application:drm.ua
• x-wap-application:enm.ua
• x-wap-application:wv.ua

Other "X-Wap-Application-id" header values must be submitted to NowSMS using their assigned decimal or hexadecimal value rather than the text name. (If using a hexadecimal value, preface the value with "0x" to indicate hexadecimal format.)

Further information on the WAP Push Access Protocol can be found from by downloading the WAP-247-PAP specification from http://www.openmobilealliance.org/tech/affiliates/wap/wapindex.html

Note that NowSMS does not support guaranteed delivery or delivery notifications.

NowSMS does not support base64 encoding for PAP content, binary content should be posted as 8-bit data using Content-transport-encoding: binary. We further recommend that the MIME part that includes the binary content have a "Content-Length:" header to avoid the potential of extra bytes (such as blank lines) being interpreted as part of the binary stream.

NowSMS sends WAP Push messages using connection-less WDP (Wireless Datagram Protocol) over SMS. SMSC connection from the NowSMS Push Proxy Gateway can use either SMPP, UCP/EMI or CIMD2. GSM modem connections can also be used as the SMSC connection for smaller test environments.

Special considerations exist for WAP Push in CDMA and CDMA2000 environments, namely the SMSC needs to support WDP Adaptation for SMPP. This is described in more detail at http://www.nowsms.com/support/bulletins/tb-nowsms-010.htm.