XMPP Push Notification Test (iOS)

Axel Reimer

Preamble: This notification test was executed in June/July 2022. Meanwhile, new versions of Siskin IM and Monal were released. The developers introduced new features which are now mentioned in the pro- and contra-lists of the apps. I did not repeat the tests for the new versions since I do not think that the push notification test results would be any different (I did not test it because it is very time-consuming). I considered the following updates in this article.

  • Update 2024-04-13: Monal supports encrypted video-calls since version 6.1 and group creation and management since version 6.2.0. This has been changed in the text.
  • Update 2023-12-12: Monal supports encrypted audio-calls since version 6.0. This has been changed in the text.
  • Update 2023-10-04: Monal now supports encrypted audio-calls in the current public Beta.
  • Update 2022-09-27: sure.im has been patched. It is now possible to successfully invite Conversations users. Additionally, sure.im is now a Category A server.
  • Update 2022-08-25: Monal now enables encryption by default for new contacts and new group chats.
  • Update 2022-08-24: You can now see the participants of a group chat in Monal.
  • Update 2022-08-23: Monal now supports animated GIFs and allows recording voice messages without permanently touching the screen.
  • Update 2022-08-09: Siskin IM is now available in different languages.

Introduction

I often recommend XMPP as an alternative to other instant messaging apps. But sometimes (especially iOS) users complain that they did not receive a notification for a new message. For that reason I decided to do a quick notification test for three popular iOS XMPP apps:

  • Monal 5.2.0 (796)
  • Siskin IM 7.2
  • Snikket 1.2 (Build 10)

I thought that it would be nice to evaluate those apps on different server implementations (ejabberd, prosody, Tigase, Snikket). I took a look at (xmpp.net providers (external)) and selected the following providers

  • magicbroccoli.de - it’s category A and is already updated to ejabberd 22.05
  • 5222.de - it’s Category B for only offering registration via web, but it is already updated to prosody 0.12.1
  • sure.im - it’s Category C. But it is the first Tigase server I found in the list (Update 2022-09-27: sure.im is now Category A)
  • self-hosted - no Category. Snikket is invitation based. Since I had not invitation, I self-hosted a server for this test

Important Notes

Please consider the following when reading this blog post.

  1. This is not a scientific study. This is just a report of a bunch of test results I did in my leisure time. Please do not hesitate to contact me via mail or XMPP if you encounter mistakes or if your experiences are different.

  2. Please keep in mind that there are currently (historical reasons) different technical ways to enable notifications in XMPP group chats (MUCs). The developers (server software and apps) use different solutions to handle notifications in MUCs. This means: You cannot say who actually uses the right solution for this task. Maybe this problem will be solved as soon as a standardised XEP is available? Maybe a transition to group chats based on MIX (external) will solve the problem? We will see.

  3. These notification tests were done in July 2022. The behaviour of the apps and servers might differ in the future.

Testing Environment

The testing device was an iPhone 8 running iOS 15.5. I registered four accounts (one on each of the selected servers) which act as communication partners for the apps under test. I used the Android app Conversations for sending messages to the iOS apps. I decided to register the following accounts (chat.example.com represents my self-hosted Snikket server):

  • conversationspush@[magicbroccoli.de | 5222.de | sure.im | chat.example.com]

Afterwards, I registered a separate account for each app under test on each of the tested providers via web registration. Please keep in mind that the Snikket app is only specified to be used with a Snikket server.

  • monalpush@[magicbroccoli.de | 5222.de | sure.im | chat.example.com]
  • siskinpush@[magicbroccoli.de | 5222.de | sure.im | chat.example.com]
  • snikketpush@chat.example.com (self-hosted Snikket server)

Then I installed Monal, Siskin IM and Snikket and connected each app to the matching account on the same server (i.e. for the magicbroccoli.de test I assigned Monal to monalpush@magicbroccoli.de and Siskin IM to siskinpush@magicbroccoli.de). I added the conversationspush XMPP adresses for this provider to the contacts of Monal and Siskin IM, added the monalpush and siskinpush adresses to each of the Conversations contact lists (for creating group chats on different servers), activated OMEMO encryption and tested if everything works fine (sending and receiving messages to each other).

In Siskin IM I set additional options to allow notifications even when Siskin IM is in “Do not Disturb” mode and to allow notifications in group chats (whenever Siskin IM allowed me to). Now I was able to execute the tests. The tests were not automated. This means I had to create the group chats manually (using my Android device) and I had to send the messages manually.

After finishing the tests for one server, I removed the accounts from the app and deleted the apps from the iPhone. I restarted the iPhone before installing the apps again for the next server.

I tested a lot of things but there were also things I did not test, e.g.:

  • Notifications for group chats (MUCs) which are hosted on a different server using the same server software (e.g. account is on magicbroccoli.de (ejabberd) and MUC is on trashserver.net (ejabberd))
  • Notifications when the the app has not been used for a long time
  • Notifications when having an unstable connection to the internet

For each app I ran at least once into a situation where no notifications were triggered anymore. I could not reproduce this behaviour and I am not sure if there was a problem with iOS, my internet connection or whatever.

Results

Please keep in mind that the tests were done manually and it is possible that I made some mistakes. The results were made accessible for the app developers before publishing them. This was a good idea since the developers could explain some failed tests (e.g. at first I did not recognise that Siskin IM only activates notifications for MUCs when mentioned. The developer showed me this option and I had to retest some test cases).

I organised my test cases in Kanban boards. Lists are used to express the test results (successful, failed, …). Swimlanes are used to differentiate between the servers. You can select each card and you will find a short description of the according test case. Some test cases are labeled as “Federation”. Those tests needed a working federation between servers because the MUC was hosted on a different server. At the bottom of each Kanban board you will find a swimlane called general information. This swimlane contains cards in which I collected some additional notes which might be of interest.

The test results are available at kanban.adminforge.de:

When you look at the test results you will recognise that I did not test group chats (MUC/MIX) on sure.im. My Android app Conversations was not able to create a group chat on sure.im. When I created a group chat using Siskin IM, my Android client showed this group chat but directly left it. I do not know if maybe I did something wrong but I was not able to create a working group chat for the clients. Additionally, Monal was not able to use a registered account on sure.im.

Update 2022-09-27: The sure.im server received a patch and it is now possible to create group chats using Conversations. Additionally, Conversations users can be invited to sure.im group chats.

The tests and results helped me to understand the problems iOS users are encountering when using XMPP. But what is my recommendation for iOS users? Let us have a look at each app.

Siskin IM

Siskin IM performs well for 1:1 messages but it does not when group chats (MUCs) are hosted on other servers. This has to be taken into account when recommending Siskin IM to iOS users. Siskin IM seems to be optimised for Tigase servers. Some options in the app can only be activated when being used with a Tigase server. I believe (but I could not test it) Siskin IM will work great if group chats are hosted on Tigase servers (MUC/MIX).

  • Pros
    • Creating group chats (MUC/MIX)
    • Encrypted audio-/videocalls
    • Recording voice messages
    • Sending geolocations
    • Supports animated GIFs
    • Update 2022-08-09: User interface is available in different languages (since Siskin IM 7.3).
  • Cons
    • Notifications for group chats do not work well in the federated XMPP network
    • Notifications usually appear two times (once when received and a second time when opening the app)
    • No notification preview
    • End-to-end encryption is not activated by default (but there is an option to do so)
    • Group chat invitations need to be joined manually and can easily be missed
    • App is not easy to handle for new users

In my opinion this app addresses the professional user who knows what XMPP is and how it works. There are many options which are not set intuitive for a modern mobile messenger (notifications for MUC, encryption is deactivated as default, …). This does not mean that the options do not make sense but I do not think that a normal mobile user expects them as they are.

Monal

Monal is the app which performed best in the notification test (even for federated group chats (MUCs)) and it was the only app showing the content of encrypted messages in the notification area. Additionally, it is available in different languages.

  • Pros
    • Monal outperformed the other apps in the notification test (even federated group chats worked flawlessly)
    • Update 2024-04-13: Encrypted audio-calls (since Monal 6.0) and video-calls (since Monal 6.1) are available (not available in China).
    • Users can see the content of a message in the notification preview (even when it is an encrypted message)
    • Update 2022-08-23: Recording voice messages (Since Monal 5.3.0 it is possible to record voice messages without permanently touching the screen)
    • Sending geolocations
    • User interface is available in different languages
    • App is good to handle for new users
    • Monal claims to have redundant push servers in different locations which cannot see your JID (Link (external))
    • Update 2022-08-25: End-to-end encryption (OMEMO) is activated by default for new contacts (since Monal 5.3.0) and for new private group chats (since Monal 5.3.1).
    • Update 2022-08-23: Monal 5.3.0 now supports animated GIFs
    • Update 2022-04-13: It is possible to create or manage group chats (since Monal 6.2.0).
  • Cons
    • There are no known cons. The cons which were problematic have been addressed and fixed by the developers.

Snikket

This app is a fork of Siskin IM and has been customized for Snikket servers. It is very easy to use. The default settings are set as I would expect them for a modern mobile instant messenger. The app has a very good notification behaviour on Snikket servers. But it currently encounters problems with group chats (MUCs) hosted on other servers. Since Snikket is a fork of Siskin IM it currently has nearly the same featureset as Siskin IM (but additionally activates encryption by default).

  • Pros
    • Creating group chats (MUC)
    • Encrypted audio-/videocalls
    • Recording voice messages
    • Sending geolocations
    • End-to-end encryption (OMEMO) is activated by default
    • User interface is available in different languages
    • Supports animated GIFs
    • App is good to handle for new users
  • Cons
    • Notifications for group chats do not work well in the federated XMPP network
    • Notifications usually appear two times (once when received and a second time when opening the app)
    • No notification preview

Keep in mind that Snikket is especially designed for Snikket servers. You might encounter problems when using the app with other servers. Snikket is a new project and currently under very active development.

Conclusion

Currently there is not the perfect XMPP app for iOS. If you recommend XMPP to an iOS user, you should check the requirements and the technical skills of this person. Technically advanced people have the option to use a combination of Monal and Siskin IM on their Apple device. Siskin IM can be used as the group chat administration and videocall application which is only connected to the server if you need to create/change a group chat or you want make a videocall. Monal can be used as the default instant messenger for everyday use.

If you can invite new users to a Snikket server you have to decide whether to recommend Monal or Snikket. Both are performing great. Snikket is a good choice if you know that the user is not really using federated group chats at the moment. In this case, the application outperforms Monal because of the additional features. When federated group chats are needed, the user can use Monal or a combination of both apps. Whatever you decide: I think you can be sure that both applications will be enhanced in the future. So you should always have a look at the App store and see what’s new in Snikket or Monal.

Personally, I will currently not recommend to register an XMPP account at sure.im. The reasons are that Conversations was not able to join sure.im group chats and Monal was not able to connect to a sure.im account. I do not know if this is really a sure.im problem but it is a problem for me communicating with my friends.

Update 2022-09-27: The sure.im server received a patch and it is now possible to create group chats using Conversations. Additionally, Conversations users can be invited to sure.im group chats.

Additional Notes

  • I know that some people might say that it is not fair to simply choose providers by looking at xmpp.net providers (external). It might be better to have a deeper look and see which server has activated special extensions like mod_muc_offline_delivery (external) and mod_muc_cloud_notify (external). But I do not think that a normal user reads (and understands) the server specifications. And if he/she does: Sooner or later this user might be added to a MUC which is hosted on a server which does not have this extension.
  • I also tested the Snikket (iOS) app with ejabberd, prosody and Tigase. But I decided to not publish the results. Why? The description in the App Store (external) clearly states that you need an invite to a Snikket server to use this app. Using this app with another server software is a use out-of-specification. In conclusion, you might ask why I tested whether Snikket works with group chats (MUCs) on other servers. The description in the App stores states that you join a worldwide communication network. I think this refers to the XMPP network and not to the Snikket network (since Snikket offers server-to-server communication to other XMPP servers).