75 GB/s Data Acquisition to Disk
A cost-optimized solution with ADQ35, leveraging libads.
www.spdevices.com
Teledyne SP Devices’ ADQ3 series digitizers offer high data throughput and an SDK designed for easy application development. In this article, we present a disk streaming system that sustains 75 GB/s write throughput across 6 channels for over 50 seconds, capturing 4 TB of data. The system is built on the DPS7-PCIe platform with six ADQ35 digitizers, each acquiring at 10 GSPS (12.5 GB/s per channel). Customers use this type of system in applications such as satellite monitoring and lidar.
Low-cost consumer NVMe drives are configured in single-level cell (SLC) mode to deliver over 12.5 GB/s per drive. Combined with Teledyne SP Devices’ libads disk streaming library, which bypasses the file system and offloads the CPU, the result is a cost-effective solution with minimal system overhead. Lower-cost setups can cost as little as 5-10% of more expensive enterprise solutions, making them well worth considering if the application does not require the endurance of enterprise solutions.
System Overview
Built on the Teledyne SP Devices DPS7-PCIe platform in a compact 4U rackmount/desktop form-factor (shown in Figure 1), this system acquires and stores data from 6 channels at 10 GSPS or 12 channels at 5 GSPS. Continuous recording for over 50 seconds yields a total data volume of 4 TB. The system is fully self-contained, with integrated clock generation, distribution, and trigger synchronization across all channels.
- DPS7-PCIe, configurable server-grade computer platform.
- 6 × ADQ35 digitizers, each configured to acquire data at 12.5 GB/s.
- 6 × Kingston Fury Renegade G5 2 TB, NVMe SSDs (configured for 670 GB effective size).
- libads, Teledyne SP Devices disk streaming library.
- Ubuntu 24.04 LTS, operating system.
- Teledyne SP Devices clock fanout board, for reference clock generation and distribution.
A 10 MHz clock reference generated by the clock fanout board was distributed to all digitizers. The trigger ports of the digitizers were connected in a daisy chain, with one digitizer configured as master. This allows for synchronized acquisition start across all digitizers.
Figure 1: Teledyne SP Devices DPS7-PCIe in 4U rackmount/desktop form-factor, example photo with 2 digitizers installed.
About Libads
Libads is a high-performance disk streaming library designed for the ADQ3 series digitizers. It is closely integrated with the digitizer API for easy application development. The highly parameterized bundled examples cover most use cases, and can be used as a starting point for advanced applications. A custom file system is used to remove CPU overhead and allow for easy identification of stored datasets.
The described solution uses one drive per digitizer, but the library can support multiple digitizers per drive, and multiple drives per digitizer (RAID). The optional NVIDIA GPUDirect Storage support enables real-time GPU processing before storage.
The library is optimized for high throughput and low system load. Data is written asynchronously to NVMe SSDs, bypassing the operating system’s file system and caching mechanisms. This approach offloads the CPU, reducing the critical components to PCIe, disk, and DRAM (or GPU). The data path is shown in Figure 2. The acquired data is written from the digitizer to DRAM buffers, where it is fetched by the disks for storage, while the CPU is only involved in control and monitoring.
Figure 2: Flowchart showing the high bandwidth data path in green, and the low bandwidth control and monitor path in red.
System Component Selection
The application requirements affecting the system components are listed below.
- Throughput: 6 channels at 10 GSPS or 12 channels at 5 GSPS with 10-bit compression -> 6 × 12.5 GB/s or 12 × 6.25 GB/s = 75 GB/s. The 10-bit compression is used to reduce the data volume with minimal impact on the signal quality.
- Storage capacity: continuous recording for at least 50 seconds -> 75 GB/s × 50 s = 3.75 TB.
- Processing capabilities: no real-time processing required.
The specific component considerations for this application are described below.
- Digitizers: The ADQ35 digitizer was selected for its high sampling rate and throughput. Each digitizer can acquire 1 channel at 10 GSPS or 2 channels at 5 GSPS and transfer data at over 14 GB/s.
- Storage: The storage configuration is mainly determined by the required capacity and throughput. Endurance, i.e. the total amount of data that can be written to the drive before it wears out, can be a concern for systems designed for heavy use. For this application, a low-cost 2 TB drive was chosen, which was configured to support over 12.5 GB/s throughput per drive by reducing the effective capacity to 670 GB per drive.
The system is based on the DPS7-PCIe server-grade platform, which can be scaled to suit most applications through component selection. The detailed system specification is shown in the table below. The digitizers and NVMe drives are shown in Figure 3. The clock fanout board is shown in Figure 4, here configured for PCIe slot mounting.
Figure 3: Digitizers and NVMe drives installed in the DPS7-PCIe system.
Figure 4: Teledyne SP Devices clock fanout board configured for PCIe slot mounting.
Results
The continuous acquisition was successfully recorded at 75 GB/s sustained throughput for 6 channels (12.5 GB/s per channel), for 53 seconds (4 TB). In total, 4,020 GB were written at 74.989 GB/s. The minor deviation from 75 GB/s is due to the timer including the startup and shutdown of the acquisition.
Once completed, development functions were used to verify the integrity of the acquired data and metadata. The acquisition was repeated multiple times, with the drives being formatted between each run since the whole capacity was used.
To determine the maximum sustained throughput of the drives, additional acquisitions were run with discrete records of 2048 samples in 16-bit format. The discrete records were successfully recorded with a sustained throughput of 75.9 GB/s. This is the maximum sustained throughput of this system configuration.
Scalability & Future Expansion
Disk streaming systems can be scaled to meet most requirements. The digitizer firmware options also offer a path for future upgrades. For example, a 4-channel system can be built with 2 × ADQ35 running 2-channel firmware at 5 GSPS, and later be expanded to 4 × ADQ35 running 1-channel firmware at 10 GSPS. The availableoptions for disk streaming systems include:
- Petabyte-scale storage capacity and 80+ GB/s sustained write throughput: Utilizing enterprise-grade NVMe drives and drive carriers.
- High endurance drives: Each rated for over 70,000 TB written (TBW).
- Synchronized multi-chassis configurations for high channel count: The system can be expanded with our 16-slot expansion chassis DPS16-PCIe. Alternatively, multiple DPS7-PCIe systems can be used. The digitizers’ clock and trigger signals can be synchronized across all chassis.
- Real-time GPU processing before storage: By performing data processing on a GPU to reduce the amount of data to be stored, the required storage capacity and throughput can be reduced. The libads library supports NVIDIA GPUDirect Storage for direct transfer of data from digitizer to GPU to disk.
- Real-time FPGA processing before storage: Data can be processed and reduced in the FPGA with available firmware options for the ADQ3 series digitizers, or by custom logic implemented using the development kit option.
- Form-factor: PXIe, desktop and rackmount form-factors are available.
Conclusion
The presented disk streaming solution reduces the cost of high-speed disk streaming by utilizing Teledyne SP Devices’ libads library and drives operating in SLC mode. The total cost of storage is reduced significantly. As fewer PCIe slots are consumed by storage devices, a higher channel count per chassis can be achieved.
A cost-effective disk streaming solution can be tailored to meet almost any requirement. We offer both complete systems and guidance for building or upgrading your own system.
About Teledyne SP Devices
Teledyne SP Devices is a pioneer in high-speed data acquisition, delivering advanced digitizers and signal processing solutions for demanding applications worldwide. With a long history of innovation, including early leadership in 10 GSPS digitizer technology, we help customers capture and process data faster, more accurately, and more efficiently.
Learn more at: https://www.spdevices.com/en-us?utm_source=mepax
