The System Architecture and Data Flow

The System Architecture and Data Flow

The System Architecture and Data Flow

System elements overview

Figure 14 shows an overview of the EUMETCast hardware and software elements of a typical reception station, and the internal and external data flows. The functions of these elements and the data flows will be described in this section.

The EUMETCast broadcast is received at the antenna/LNB front end and down-converted into an L-Band signal. The L-Band signal is routed via a co-axial cable, which acts as the input to the DVB receiver. The signal reception can be accomplished in different ways, either:

  • The built-in PCI DVB card, or;

  • An external USB DVB receiver

  • An external DVB router and a second network interface card (NIC) in the receiving computer.

Actually, all DVB reception devices appear to the operating system in the same way: as a network interface. The DVB card driver or DVB router or firmware decodes the data and filters the IP packets according to the configured parameters, such as MODCODs and Packet Identifiers (PIDs), to select which IP multicast packets are passed through to the network interface. 

The TelliCast client software interfaces to the DVB network interface via the TCP/IP stack by subscribing/un-subscribing to multicast data streams. In detail, when the TelliCast client is started, it sends a “join” message to the selected network interface specifying the Announcement Channel multicast (group) address. The network interface will forward (signal) this join message to the next network router (if there is one) and transmits any received multicast packet matching that group back to the requesting client. 

The TelliCast client then gets a continuous stream of announcement data - the announcement channel is “opened”. 

The TelliCast client uses its username to identify the subset of data it is configured to receive. In addition to the username, a password and an EKU (EUMETCast Key Unit) are necessary to decrypt the data. A given username works only with the matching password/EKU combination. For each transmission a data channel is opened using the same concept as above to “join” a multicast data channel. At the end of the transmission the data channel is closed with a “leave” request to the network interface.

The reception behaviour using the EUMETCast default configuration is as follows: During reception, the file fragments are immediately written to the temporary location on the disk using a temporary file name and control information is held in a database, which is resident in memory – the file database. Once a file is completely received, it is moved from the temporary to the target location on the disk and renamed to the original filename, and the timestamp of the file is set to the original time in the EUMETCast platform. It is important that the temporary location and target location are on the same file system, otherwise the move operation will not be possible (in Linux) or result in a copy operation (Windows) which takes more time and resources. The move is an “atomic” and fast operation because it consists of just a change in the file allocation table - the file itself is not touched.

Subsequent data processing, data replication, data distribution and housekeeping of files and directories are controlled by other applications.

Distribution of received and/or processed data may be accomplished in three ways. Firstly, file-distribution software may be configured to deliver – FTP push – files to remote locations via the local network. Secondly, the files may be pulled from the EUMETCast station via FTP by the end-user equipment. A standard FTP server on the station provides the interface for this functionality. Lastly, the local directories of the reception station may be mounted over the network using the Samba protocol or an equivalent network file sharing method.

The integration of processing, file distribution and display software must be carefully done to avoid any bad impact on the reception. The following sections will give some information about the most critical elements.

 

 

 

 

 

 

Figure 14: Reception Station Architecture


Key Elements for Performance

The TelliCast client is real-time software that has to be able to process incoming data, access information and outgoing data at the speed of the incoming information. This makes it much different from unicast transfer protocols (ftp, etc.) where lost data will be requested and retransmitted. The salient points for the TelliCast performance are the interfaces - networking for incoming data, file system for file fragments, data and logs, and USB bus to EKU for decryption. Any bottlenecks in these interfaces will have an impact in performance.

Networking: The multicast stream consists of UDP packets and is optimised for best bandwidth usage. The timing between announcement channel and the data channels is very important to achieve high efficiency on the Satellite links. That means that the active network elements including DVB receivers should not introduce jitter of more than 100 msec until the traffic reaches the TelliCast client. In terms of routing the network elements (routers or switches) between the DVB receiver or the terrestrial border router and the computer must be configured to either react very fast for join and leave requests, or must be configured to pass all multicast traffic from the DVB receiver (static routing). Any interruptions of the traffic must be avoided because of the unidirectional data flow, i.e. there is no retransmission in case of lost packets. Special attention must be placed on

  • Stable Ethernet link between network ports, avoid repeated auto negotiation or speed switches

  • Multicast signalling should be static, i.e. all traffic statically routed from DVB receiver or border routers to computer(s)

  • Sufficient bandwidth for multicast, i.e. use dedicated LANs or VLANs for the multicast, and avoid routing of unicast and multicast together on the same LAN

  • Avoiding buffering, i.e. buffering will lead to jitter which may become unacceptable. Buffering can also occur in the tcpip stack, DVB devices, or other interfaces (PCIe, USB, etc.).

  • Interrupt servicing of network interfaces by multiple CPUs or cores. Static assignment of interrupts allows better performance compared to dynamically changing assignments (see irqbalance in Linux systems). The effects can result in buffering and packet drops (overruns) in the kernel.

Networking: The multicast stream consists of UDP packets and is optimised for best bandwidth usage. The timing between announcement channel and the data channels is very important to achieve high efficiency on the Satellite links. That means that the active network elements including DVB receivers should not introduce jitter of more than 100 msec until the traffic reaches the TelliCast client. In terms of routing the network elements (routers or switches) between the DVB receiver or the terrestrial border router and the computer must be configured to either react very fast for join and leave requests, or must be configured to pass all multicast traffic from the DVB receiver (static routing). Any interruptions of the traffic must be avoided because of the unidirectional data flow, i.e. there is no retransmission in case of lost packets. Special attention must be placed on

  • Stable Ethernet link between network ports, avoid repeated auto negotiation or speed switches

  • Multicast signalling should be static, i.e. all traffic statically routed from DVB receiver or border routers to computer(s)

  • Sufficient bandwidth for multicast, i.e. use dedicated LANs or VLANs for the multicast, and avoid routing of unicast and multicast together on the same LAN

  • Avoiding buffering, i.e. buffering will lead to jitter which may become unacceptable. Buffering can also occur in the tcpip stack, DVB devices, or other interfaces (PCIe, USB, etc.).

  • Interrupt servicing of network interfaces by multiple CPUs or cores. Static assignment of interrupts allows better performance compared to dynamically changing assignments (see irqbalance in Linux systems). The effects can result in buffering and packet drops (overruns) in the kernel.

File System Performance: In general, all file systems must have sufficient performance to avoid blocking I/O situations in the TelliCast application. Insufficient file system performance is the top 3 of problems reported by users. The file database contains the control information for all transmissions in progress including received and missing fragments. Since transmissions are going on in parallel on many channels and the activities are very dynamic, the access to the control information is very time critical. Any delay to this control information will cause dropped packages or dropped control information. In order to achieve this, it is mandatory that the file database location is in memory, as in the default configuration of the TelliCast clients. When the file database becomes full, it will discard data and considerably slow down due to internal housekeeping operations, even when it is in memory, and therefore cause losses.

The next equally important interface is the files system for data storage. In the default configuration it contains the temporary and target location where the file fragments are written to and, after the file is complete, the data will be made available. Different locations can be configured per EUMETCast channel, however it is essential that the temp and target directory for a specific channel resides on the same file system. Recommended solutions for this files system are:

  • RAM disks provide the best possible performance in terms of random access and throughput. However the limitation is the maximum size which depends on the operating system and the hardware. The RAM disk must always have sufficient space left to allow temporary peaks and temporary file fragments to be stored. Successfully received files must be moved as fast as possible to persistent storage system or discarded after processing. Because of the limitation in RAM this solution is not recommended on a production system unless it is properly designed for its purpose and data profile under worst case reception conditions. 

  • SSDs provide much better random access and throughput performance compared to what is required by the TelliCast client. In addition, the persistent SSD storage is an order of magnitude higher and costs much lower compared to RAM.  So far no general IO bottleneck has been observed on SSD systems, in particular when several GB of RAM are available as file cache. As much as possible, separate data disks should be used, in order to avoid impact from system and possible memory swap activities.

  • Spinning disks are still widely used and provide persistent storage at low cost. However, in particular the random access performance is too low to support the full EUMETCast stream. It works only satisfactorily when a high amount of RAM is available as file cache and if a state of the art disk interface (SCSI, SAS, SATA) in connection with hard-disk cache is used. Otherwise just random read access to the data will cause reception losses.

File System Performance: In general, all file systems must have sufficient performance to avoid blocking I/O situations in the TelliCast application. Insufficient file system performance is the top 3 of problems reported by users. The file database contains the control information for all transmissions in progress including received and missing fragments. Since transmissions are going on in parallel on many channels and the activities are very dynamic, the access to the control information is very time critical. Any delay to this control information will cause dropped packages or dropped control information. In order to achieve this, it is mandatory that the file database location is in memory, as in the default configuration of the TelliCast clients. When the file database becomes full, it will discard data and considerably slow down due to internal housekeeping operations, even when it is in memory, and therefore cause losses.

The next equally important interface is the files system for data storage. In the default configuration it contains the temporary and target location where the file fragments are written to and, after the file is complete, the data will be made available. Different locations can be configured per EUMETCast channel, however it is essential that the temp and target directory for a specific channel resides on the same file system. Recommended solutions for this files system are:

  • RAM disks provide the best possible performance in terms of random access and throughput. However the limitation is the maximum size which depends on the operating system and the hardware. The RAM disk must always have sufficient space left to allow temporary peaks and temporary file fragments to be stored. Successfully received files must be moved as fast as possible to persistent storage system or discarded after processing. Because of the limitation in RAM this solution is not recommended on a production system unless it is properly designed for its purpose and data profile under worst case reception conditions. 

  • SSDs provide much better random access and throughput performance compared to what is required by the TelliCast client. In addition, the persistent SSD storage is an order of magnitude higher and costs much lower compared to RAM.  So far no general IO bottleneck has been observed on SSD systems, in particular when several GB of RAM are available as file cache. As much as possible, separate data disks should be used, in order to avoid impact from system and possible memory swap activities.

  • Spinning disks are still widely used and provide persistent storage at low cost. However, in particular the random access performance is too low to support the full EUMETCast stream. It works only satisfactorily when a high amount of RAM is available as file cache and if a state of the art disk interface (SCSI, SAS, SATA) in connection with hard-disk cache is used. Otherwise just random read access to the data will cause reception losses.

EKU communication: The EKU uses the USB bus as its interface. Delays affecting the USB response might cause timeouts in the TelliCast client. A 10 s response delay will be interpreted as an EKU failure and the TelliCast client is restarted. If immediately after the restart another timeout occurs then the EKU is disabled and encrypted channels cannot be received. Jumps in the system time (due to non-optimal time synchronisation) can falsely indicate EKU timeouts and lead to the mentioned problems. Also the EKU itself can experience communication “hang-ups”. A soft hang-up can be recovered with a manual TelliCast restart. Therefore, it is advised to install an EKU monitor tool to perform this recovery.

The EKU can also experience a “hard hang-up”, indicated by the LED going permanently off. This state can only be recovered by a power cycle of the EKU (unplug, replug EKU, or unload reload the USB module). For this case, contact EUMETSAT Helpdesk to replace the EKU.

EKU communication: The EKU uses the USB bus as its interface. Delays affecting the USB response might cause timeouts in the TelliCast client. A 10 s response delay will be interpreted as an EKU failure and the TelliCast client is restarted. If immediately after the restart another timeout occurs then the EKU is disabled and encrypted channels cannot be received. Jumps in the system time (due to non-optimal time synchronisation) can falsely indicate EKU timeouts and lead to the mentioned problems. Also the EKU itself can experience communication “hang-ups”. A soft hang-up can be recovered with a manual TelliCast restart. Therefore, it is advised to install an EKU monitor tool to perform this recovery.

The EKU can also experience a “hard hang-up”, indicated by the LED going permanently off. This state can only be recovered by a power cycle of the EKU (unplug, replug EKU, or unload reload the USB module). For this case, contact EUMETSAT Helpdesk to replace the EKU.

CPU Performance: The CPU requirements of TelliCast are not very high. Normal state-of-the-art CPUs for Desktops are sufficient. Due to the single thread main working process of Tellicast, there should be at least 2 more cores available than Tellicast services are running.

CPU Performance: The CPU requirements of TelliCast are not very high. Normal state-of-the-art CPUs for Desktops are sufficient. Due to the single thread main working process of Tellicast, there should be at least 2 more cores available than Tellicast services are running.