Wireless alliances like Wi-Fi and ZigBee certify wireless devices for interoperability which to the end user provide a confidence level of smooth operation between these devices from different manufacturers. For instance this level of interoperability is necessary (but not sufficient) to ensure communications on WLAN Wi-Fi segments containing an Access Point (AP) and a number of wireless client cards. The same concept applies to ZigBee networks where interoperability gives a confidence level for ensuring operation of devices such as a light bulb and a switch, an RF4CE remote control and paired TV, DVD player or set top box, but also insufficient to assure seamless operation of the network over time and under variable environmental conditions especially in demand-response time critical scenarios.
The bandwidth demanding applications or operations of the time sensitive controllable devices on wireless networks at large require more than just a confidence level of interoperability that in some situations may not be satisfied. For instance, on a Wi-Fi segment, playing a movie and streaming it wirelessly in real time require higher level of operational assurance than interoperability between the source and destination on the wireless network that include an AP in an infrastructure mode or directly between wireless devices (cards) in an ad-hoc mode. This higher level comes from satisfaction of performance metrics that ensure maximum throughput, minimum latency (and consequently jitter), and minimum (or zero) packet loss for all streamed wireless data. On the ZigBee front where the network is by definition a meshed network such that communication between two end points may be accomplished on more than one route, rerouting the necessary wireless information through redundant links will require fast performing devices to maintain the network operation when one of the network elements fail. The time to respond and survive a noisy environment will be measures of performance for such networks.
In all cases there will be a need to characterize performance of individual wireless devices on their merit and to conduct comparative analyses to assess such performance with respect to other devices in the same class. The need extends to characterizing performance of the operational network as well. This performance characterization helps greatly not only in assuring smooth operation of the wireless network, but also to avoid unexpected failures should the application demand more than what the combined network can provide. And, as important, to avoid heavy expenditure of budgets for projects of large scale if the wireless components were selected based on interoperability only while overlooking performance measures.
The need for high bandwidth is growing to meet demand of real time applications that use wireless data transmission. For instance to play a movie in real time and to have a stable display on each frame the wireless network has to support 30-35 frames per second of full color video and the corresponding audio. This depends on the ability of the wireless nodes (APs and client cards –also known as wireless stations) to provide the bandwidth while being interoperable since these nodes usually are manufactured differently and from different chipmakers or integrators. Thus far attention has always been paid to interoperability and rightly so, but the real measure that assures steady operation of the wireless networks while carrying traffic of this kind is performance that includes critical components to be characterized. These are throughput, latency, and jitter and packet loss in addition to others that are not in the scope of this article.
Throughput as defined by the IETF (Internet Engineering Task Force) RFCs (e.g. RFc2544) is the maximum rate in Mb/Sec or packets/Sec a buffering device can transmit without packet loss. Therefore knowing the number of frames per second in a video transmission, its audio, and the transmission overheads a simple calculation can lead to the minimum throughput needed for the buffering device in order to transmit the wireless data. This is not enough yet as it only shows the ability to transmit the wireless data between the source and destination. But in such video/audio transmission another critical performance measure must also be satisfied; that is latency (and consequently jitters). Latency is the time delay taken by the buffering device to transmit a predefined packet size between the input and the output buffers. The value of latency should be minimum to the level the end user may not see freezing frames or blank frames during the video streaming. The latency is usually characterized by its average value since it is near impossible to have perfect constant values between different frames. So considering only the average latency, its absolute value may not be of critical deteriorating effect if that value is almost constant which means the transmission is just shifted with respect to time between the source and destination. Thus the jitter becomes critically important since it is the rate of change of latency. A clear and stable video/audio transmission requires a minimum value of jitter to be satisfied (ideally should be zero but almost impossible).
The packet loss even though it is also a very important measure of performance but the value sought for approving a wireless buffering device for the video/audio transmission must be zero. So during testing that leads to characterizing the performance of the wireless device, values of packet loss at different rates will be measured and the rate at which packet loss occurred will be determined as the throughput value. The average latency at this rate (the throughput value) will be the associated value. Jitter will be calculated over the time period over which throughput and latency are obtained and packet loss must be the zero value.
These values (throughput, latency/jitter, and packet loss) are known as the performance metrics that can distinguish one wireless device from another in the same class. A better device is the one with higher throughput, lower latency/jitter values at the zero packet loss at which these metrics are measured/calculated.
Measurement of performance metrics require definition of how the network is constructed. A wireless network that includes AP and stations is known as “infrastructure” network while in the absence of the AP i.e. with wireless stations only the network is known as “ad-hoc”.
Direction of the wireless transmission is also to be defined; unidirectional (between a source and a destination one way), or bidirectional between the source and the destination.
The network can be partially meshed or fully meshed based on the number and the directions of its transmission.
Throughput, latency/jitter and packet loss are usually measured/calculated for a fully meshed network that exercises the maximum rate of transmission in the network.
Sufficiency of Interoperability Tests
While interoperability is necessary to assure the ability of the wireless devices to connect, authenticate and communicate with each other in the wireless network, satisfaction of interoperability conditions are not sufficient to ensure performance. It is necessary for two wireless devices to negotiate the OTA (On The Air) information in form of beacons to extract important data like the supported rates, the SSID (name of the network), QoS (Quality of Service) parameters, etc. These beside the association and authentication data whether in a form of passphrase or through an authentication server in a process that requires verification of pre-established certificates are conditions of interoperability. Furthermore satisfaction of predetermined (reference) rates in Mb/S is required as other interoperability conditions. These conditions are necessary to satisfy interoperability which in Wi-Fi interpretation means that the wireless devices certified for interoperability are most likely to interoperate without offering a guarantee condition.
This shows that interoperability is necessary but insufficient for meeting performance measures. To the contrary, satisfaction of interoperability conditions (according to some Wi-Fi test plans) is squarely against performance and sustaining stability. An AP that supports mixed rates of 802.11b and 802.11g dual band operation mode is supposed to switch to the lower 802.11b rate if it detects a legacy wireless client that has only an 802.11b radio according to a protection mechanism under some conditions. In this scenario meeting such interoperability condition results in reducing the transmission rate which affects performance negatively. Another interoperability condition requires the wireless device upon detection three times of what is known as MIC attack (bad packet) to de-associate all other wireless devices it is connected to for 60 seconds. This again may have a negative effect on performance in applications that require sustaining stability.
The Need for Performance Tests
Performance tests where checking for satisfaction of performance metrics are needed to ensure integrity and stability of wireless networks operations. Test labs that provide performance testing assess the ability of the Device Under Test (DUT) to transmit the wireless data at rates no less than a predefined set that suit every application and ranks the DUT’s performance with respect to other wireless devices in the same class. Similarly for latency/jitter and packet loss, the tests are conducted to characterize these metrics and to make sure that the DUT will possess the required data satisfying the limits of each.
Types of Tests
These tests utilize the APs and Stations in meshed configurations where every unit sends traffic to every other unit bi-directionally via an AP in an Infrastructure configuration of which there are two types (Star-Meshed and Fully-Meshed) or directly in an Ad-Hoc configuration.
Star-Meshed Infrastructure
In this test configuration an AP establishes a network and all other Stations connect to this network and associate with the AP. In the case of four Stations and one AP, eight streams of transmission will be established as shown in Figure 1.
Figure 1: Star-Meshed Infrastructure Test Diagram
Fully-Meshed Infrastructure
In this test configuration an AP establishes a network and all other stations connect to this network and associate with the AP. In the case of four stations and one AP, twenty streams of transmission will be established as shown in Figure 2. Each station will send and receive to/from the test console via the AP, and will send and receive to/from every other station as connected via the AP.
Figure 2: Fully-Meshed Infrastructure Test Diagram
Fully-Meshed Ad-Hoc
In this test configuration a station creates a network and all other stations connect to this network. In the case of four stations (including the creator) there will be twelve streams of
transmission between each station and each other to send and receive as shown in Figure 3.
Figure 3: Fully-Meshed Ad-Hoc Test Diagram
Test results and conclusions
The scope of this article does not include a provision to show specific test results which is a topic of a subsequent article. But multiple test results from various tests show the following:
- The throughput values depend on test configurations; number of streams; mesh type, type of the wireless network, packet sizes, and many other variables.
- In all cases the throughput values should not be confused with the rate limits that decide pass or fail of different tests for interoperability within the Wi-Fi framework.
- Performance metrics as discussed must be followed carefully and must be referenced to the IETF-BMWG RFCs as the standard reference for characterizing performance.
- Performance measures should be used to characterize wireless devices (and wireless networks) and should be used for certifying them to handle bandwidth demanding real time applications in addition to satisfying interoperability that must be considered as prerequisite for performance certification.
Dr. Farouk Zanaty has more than 15 years experience in network engineering, protocols, and real time communication systems. Prior to being a cofounder of Wi-PerforMax, Farouk was the leader for Wi-Fi Interoperability Certification programs for all Agilent Labs during a period Agilent was the exclusive test house for Wi-Fi. He joined Conexant systems to automate all the in-house wi-fi testing programs and achieved multiple golden reference certifications for Conexant. He then joined TUV Rheinland of North America as the Wireless Development Manager for the Telecom divisions internationally in 4 countries. He conducted certification tests for all Wi-Fi test plans (802.11a/b/g/n and others) on products from most Wi-Fi member companies while automating Agilent Wi-Fi test beds worldwide. He established the very first Wi-Fi official certification test beds for WPA2 and WMM test plans and exclusively conducted these tests for the first few months. He is the inventor of multiple patents (awarded and pending) in IP Transaction Detail Records, Wireless Networks, C-Chip for cell phone standards, Interlaced Monitors, and SS7 full duplex transmitters. Farouk received his B.Sc. degree in Aeronautical Engineering from Cairo University (Egypt 1974), his first M.Sc. degree in Control Engineering from Hatfield Polytechnic (England 1979), his second M.Sc. degree in Aeronautical Engineering from Cairo University (Egypt 1981) and his Ph.D. in Control Systems of Space Robotics from Oakland University (Michigan USA 1993).
