This article is related to my previous post “Run commands simultaneously on different servers”. Once I figured out how to run the command on the different machines, I want to aggregate my load testing tool results to get an aggregated report.
I use two standard Linux command-line utilities, parallel and SCP, to solve my problem.
First, you create a file with all the “commands” you want to execute in parallel like
scp [email protected]:"/root/results.*.bin" .
scp [email protected]:"/root/results.*.bin" .
scp [email protected]:"/root/results.*.bin" .
scp [email protected]:"/root/results.*.bin" .
scp [email protected]:"/root/results.*.bin" .
cp "/root/results.loadtesting.bin" /root/reports/Next, you run the parallel command and pipe the file with the command instructions into it.
parallel -j 6 < jobsWith the -j flag, you configure the number of jobs ran in parallel. I tend to configure it equally to the number of
commands. If you change those, often you can use wc -l and awk to configure it dynamically for you, like
parallel -j "$(wc -l jobs | awk '{print $1}')" < jobs“Faster” downloads
Initially, I downloaded the files to my local workstation to aggregate the files, but this process is relatively slow. Therefore I recommend using one of the load testing instances inside your cloud/datacenter to speed up collecting the files. On some occasions, the files were several hundred MB’s in size, so you want to leverage the network speeds within your cloud and eliminate waiting on downloading them to your local machine.
Make sure you enable forwarding of the authentication agent connection using the -A ssh option when you ssh to the
instance you want to aggregate the files on. Forwarding the authentication agent makes it possible for the SCP sessions
you start using parallel to authenticate.
Ps: when you aggregate the files from one of the load generating instances, you might want to replace a scp instruction with an mv command instead. The parallel command isn’t restricted to the SCP command. 😉
Even though I use SCP to accomplish the tasks I read this week, SCP is considered deprecated 😅 , so I recommend using rsync or sftp instead. You can read more about the details on LWN.net.
