Load test T-Mobile IoT network together with SODAQ

JasperP

Introduction

Are you also curious about how our IoT network performs? So are we! Therefore, we have been load testing our network in the last few weeks. And for that we've asked SODAQ to give us a hand.

We have tested 3 scenarios, in which we have put one cell under stress to investigate how it performs. In these scenarios the frequency of the number of messages sent was chosen to be much higher than the maximum of 120 messages per day per device. This was done to simulate with a relatively low number of devices that are sending messages at high frequency the load of thousands of devices sending their messages to one cell with this lower frequency.

Scenario 1: a test with 10 devices sending uplink messages every 1 second
Scenario 2: a test with 30 devices sending uplink message every 30 seconds
Scenario 3: a test with 105 devices sending uplink messages varying from every 90 seconds to every 1 second

The results of these tests will give an indication of the capacity of the cell and network related to the number of devices that can be supported by this cell in a real-life scenario.

Test setup & conditions

The test setup and conditions for the 3 test scenarios are described below.

Scenario1: Test with 10 devices

10 devices (SODAQ U-Blox SARA N211)
UDP messages of 250 bytes
Time period between every message was 1 second
The 10 devices were turned on at the same time, did the attach and started sending the data.
The test was done in the T-Mobile office in The Hague

Scenario 2: Test with 30 devices

30 devices (SODAQ U-Blox SARA N211)
UDP and CoAP messages of 250 bytes
Time period between every message was 60, 30 and 15 seconds
The devices were turned on per 10 at the same time, did the attach and started sending the data
The test was done in the SODAQ office in Hilversum

Scenario 3: Test with 105 devices

SODAQ prepared a nice box for 100 of the 105 devices, so that these were easy to transport. The test was performed at the SODAQ office in Hilversum. The devices were located at about ± 1 km from the cell and were indoors in one of their rooms at the 2nd level. To make sure all devices attached to the same cell we turned off all the neighbouring cells.

The boards are powered by USB and/or battery power. The batteries can be recharged when the box is connected to mains power. We used 10 USB powered hubs with each port to connect all 10 boards.

There were 10 batches of 10 devices. This allowed us to turn on 10 devices at the same time. Each batch was placed at a distance of several meters between the other batches. The distance between the 10 devices (antenna's) per batch was about 10cm from each other. The test was performed with CoAP. This allowed us to immediately see whether the message was successfully sent. If CoAP works well, the less heavy UDP protocol will also be successful.

While the test was performed, all the different components in the network were monitored by the component experts. For several devices the transitions were monitored and the power output of the radio module. All devices also showed their status via different color LED's. This showed immediately whether the devices had a proper attach and delivered their message successfully. The RAN and CORE were monitored on their throughput and capacity. The data sent by the devices were stored in a database on a server.

Data stored:

Current Time (EPOCH)
Packet counter
Output of the NEUSTATS command: CE Level, SNR, Signal strength, …

Conditions:

105 devices (SODAQ U-Blox SARA N211)
CoAP messages of 250 bytes
40 devices sending a message every 90 seconds
40 devices sending a message every 30 seconds
20 devices sending a message every 10 seconds
5 devices sending a message every 1 second
The devices were turned on per 10 at the same time, did the attach and started sending the data
The test was done in the SODAQ office in Hilversum

Results

Scenario 1: Test with 10 devices

9 devices attached on 1 cell and 1 device attached on a second cell
85% of the uplink messages were sent in CE Level 0, 15% of the messages were sent in CE Level 1
There was 0% packet loss
The average latency was 2.2 seconds

Scenario 2: Test with 30 devices

30 devices sending first every 60 seconds, 30 seconds and 15 seconds, first UDP and then CoAP
All devices attached on 1 cell, since the others were turned off
100% of the uplink messages were sent in CE Level 0
There was close to 0% packet loss
The average latency was 3 seconds

Scenario 3: Test with 105 devices

40 devices sending 250 bytes every 90 seconds, 40 devices sending every 30 seconds, 20 devices sending every 10 seconds and 5 devices sending every 1 second
All devices attached on 1 cell, since the others were turned off.
100% of the uplink messages were sent in CE Level 0.
There was 0.95% packet loss
The average latency was 5 seconds

Conclusions & learnings

Based on the results we have seen that the cell and therefore our IoT network is capable of supporting many devices under these set conditions. We have seen data throughput of 30 kbps and more, which means we still have more capacity left. However, we have seen some packet loss in the last test of 0.95%. Although the packet loss is low, we will be further investigating this to see what is happening and further reduce it.

The tests were performed in CE Level 0. In CE Level 1 and 2 the capacity is expected to be lower. Future tests should provide more insights in the performance when the devices are put deep indoors (e.g. in a cellar).

The tests show that the network is ready to go! We now tested one cell. One cell actually contains on average 3 NB-IoT antenna's in 3 different directions to cover the area around the cell. In the Netherlands T-Mobile has 4700 sites/cells for NB-IoT with on average 3 antenna's. This means that the test already shows that we can support 105 devices x 3 antenna's x 4700 sites = 1.5 million devices. Since the message rate in real-life cases will be much lower, maximum 5 per hour, and the message size will also be much lower than the 250 bytes used in the test, we will be able to support many more devices before we will reach the same load as we have seen during the tests.

Furthermore, the 3GPP is working on further improvements of the NB-IoT standard, which will contain new features and performance improvements. This will also have positive effects on the network capacity. The tests we have performed show us that we have a good baseline to start with. We will continue our testing together with our customers and continuously improve our IoT network.

Some learnings and recommendations to conclude with:

Devices that are doing an attach generate quite some load. Therefore, it is better to attach the devices sequentially. The device only needs to do an attach once, so once it is attached you do not need to attach it anymore.
We have some flood control mechanisms in our network that prevent devices to send messages in a very high frequency. These very high frequencies are not expected for NB-IoT and could (unintentionally) harm the network. The flood control mechanism helps us to prevent this from happening.
100 devices in a small box cause interference. Therefore, we have taken the devices out of the box to prevent interference having a big influence on the performance.
250 bytes is quite a lot for 1 message. Most use cases only require about 50 bytes max.
Devices should send max. 120 messages per day, preferably less. Also, for battery lifetime this has quite a big impact (see our post on the forum about our battery lifetime test we have done).