In response to discussions regarding the max transfer speed of 22.6 MB/s cited in Guccifer 2.0 NGP/VAN Metadata Analysis, the Forensicator went back and took another look at the metadata and found strong evidence of peak transfer rates of approximately 38 MB/s. Although this higher peak transfer speed might not completely refute the counter-claims made by various critics (regarding transfer speeds that can be achieved over the Internet), it certainly raises the bar.
Overall Transfer Speeds are Impacted by a Single Directory, “CNBC/Portfolio”, which has Many Small Files
The NGP VAN files have a directory called “CNBC/Portfolio”, which has 1900 small files, all 28 bytes in size. The measured copy speed for that directory is noticeably slow because there is noticeable overhead when creating many small files. Two possible reasons for this slow down when copying small files are: (1) there is overhead (esp. with USB-2 devices) when having to “turn around” communication with the device where reads and updates of directory entries are interleaved with writes to the file and (2) the operating system takes special actions (which require more writes to the device) to ensure that the file system remains in tact across a system crash.
The fact that this “CNBC/Portfolio” directory has a couple thousand tiny files explains why the calculated overall 23 MB/s average rate is slower than might be seen with a more typical distribution of file sizes. There are approximately 2300 files in the NGP VAN collection. The 1900 files in the “CNBC/Portfolio” directory are a big chunk (80%) of all the files by count, but a tiny percentage (0.02%) of the total file size. Yet, the transfer time for that directory is 14% of the total transfer time.
The observations made above highlight the impact that a single directory had on overall (average) transfer times. The fact that this directory with many small files had a significant impact on overall transfer speed underlines the importance of running tests with the actual files in the NGP VAN collection in order to derive accurate results.
Exclusion of this “CNBC/Portfolio” Directory with Many Small Files Brings the Overall Transfer Speed up to 28 MB/s.
The table below isolates the estimated transfer speed calculated for a large directory called “FEC”. Note that since all the files are under one top-level directory, there are no “time gaps”, which complicate the transfer speed estimate for other parts of the collection.
As we can see this “FEC” directory is over one third of the total NGP VAN collection by file size; it represents a large sample of 744 Megabytes. The average transfer rate is 28 MB/s.
Peak Transfer Speed: 38 MB/s
The following table isolates a sequence of 20 files in the FEC directory. That sequence has two large files (108 MB and 228 MB) which will contribute to a much higher estimated transfer rate. (Note: some rows are hidden to improve presentation; the total size shown is for the entire group of files, which is almost 400 MB).
This peak rate of 38 MB/s is much closer to the practical maximum transfer speed of a USB-2 storage device. The theoretical maximum speed for USB-2 is 60 MB/s, but for various technical reasons, it is closer to 55 MB/s. The practical maximum, due to various implementation issues is approximately 45 MB/s. In practice, realized speeds are noticeably lower than that.
The Forensicator 