Parallel SSH / PSSH: Manage Multiple Presearch Nodes at Once
OK, so you wanna run multiple Presearch nodes. Let’s see how we can make it a bit easier.
Sometimes, you just have to run a manual update across all of your nodes. Or maybe you just wanna see the docker logs output across all of them at once. Parallel SSH / PSSH is your friend to help make this happen.
Windows users: I highly recommend installing the Windows Subsystem for Linux (WSL) to utilize the benefits of Parallel SSH directly. If you install the default WSL, follow the Ubuntu directions below.
Install the Parallel SSH package:
Ubuntu:
apt-get install pssh
CentOS:
yum install pssh
macOS:
brew install pssh
Ubuntu users: this installs the command parallel-ssh, so to make things easier, let’s just symlink that to pssh so that all of the commands are the same:
ln -s /usr/bin/parallel-ssh /usr/local/bin/pssh
OK, so sign up for all of your VPSes/servers/whathaveyou. Once you’ve purchased the new servers, you should be able to use the dashboard provided by that service to get the IP address, SSH port, and root password for each one. For each server, you’ll need to setup SSH properly for easy and secure use with Parallel SSH.
Here, we get our local SSH credentials setup on each remote server. (If you haven’t ever signed into another server with certificate-authed SSH before, you’ll need to generate some local keys.)
Replace {port} in each command with the Port provided in the dashboard.
Replace {ip} in each command with the IP Address provided in the dashboard.
ssh-copy-id -p {port} root@{ip}
ssh -p {port} root@{ip}
passwd
Set a new root password so you don’t still have the one generated by your service.
exit
Repeat the above steps for each server you are going to be using.
OK, when you are done, you’ll be back at your local shell. Let’s setup the PSSH file that will have the config for all of your servers.
nano nodes.txt
This will open a text file in your terminal. Just use the keyboard to enter the following data for each server:
{ip1}:{port1}
{ip2}:{port2}
{ip3}:{port3}
...
Once you’ve got all of them entered, exit nano by pressing ctrl-x, then y to save the changes, then press enter to keep the filename nodes.txt.
Next up, we’re going to configure SSH on every server to only allow certificate authentication:
pssh -i -t 0 -h nodes.txt -l root 'sed -i -E "s/^PasswordAuthentication=yes$/PasswordAuthentication=no/" /etc/ssh/sshd_config && service sshd restart'
Awesome, we are now setup to use Parallel SSH with these servers going forward!
So let’s setup all of these servers to run Presearch nodes. We’re going to tell them to install Docker and dig, then run the Presearch Node install script, with a little tweak to include the IP address and Hostname of each server in the node’s description.
Install Docker (choose the line for the OS that’s running on your remote servers, not what you are running locally):
Ubuntu:
pssh -i -t 0 -h nodes.txt -l root 'apt-get remove docker docker-engine docker.io containerd runc ; apt-get update && apt-get install apt-transport-https ca-certificates curl gnupg lsb-release && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && apt-get update && apt-get install docker-ce docker-ce-cli containerd.io'
CentOS:
pssh -i -t 0 -h nodes.txt -l root 'yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine ; yum install -y yum-utils && yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && yum install docker-ce docker-ce-cli containerd.io && systemctl start docker'
Let’s also install dig, a tool to easily let us get our public IP address:
Ubuntu:
pssh -i -t 0 -h nodes.txt -l root 'apt-get install dnsutils'
CentOS:
pssh -i -t 0 -h nodes.txt -l root 'yum install dnsutils'
And finally, get Presearch up and running:
pssh -i -t 0 -h nodes.txt -l root 'REGCODE="{PUT YOUR REGISTRATION CODE HERE}" IP="$(dig +short myip.opendns.com @resolver1.opendns.com)" HOST="$(hostname)" ; docker stop presearch-node ; docker rm presearch-node ; docker stop presearch-auto-updater ; docker rm presearch-auto-updater ; docker run -d --name presearch-auto-updater --restart=unless-stopped -v /var/run/docker.sock:/var/run/docker.sock presearch/auto-updater --cleanup --interval 900 presearch-auto-updater presearch-node && docker pull presearch/node && docker run -dt --name presearch-node --restart=unless-stopped -v presearch-node-storage:/app/node -e REGISTRATION_CODE=${REGCODE} -e DESCRIPTION="${HOST} ${IP}" presearch/node && sleep 10 && docker logs presearch-node'
Awesome, you should see the Presearch logo output for each server onto your screen. You did it!
OK, so now what? Why all this setup? Well, now you can just check in on your nodes whenever you like: Are my Docker Presearch instances running?
pssh -i -t 0 -h nodes.txt -l root 'docker ps'
What are the latest logs?
pssh -i -t 0 -h nodes.txt -l root 'docker logs --tail 10 presearch-node'
Want to just manually restart them all?
pssh -i -t 0 -h nodes.txt -l root 'REGCODE="{PUT YOUR REGISTRATION CODE HERE}" IP="$(dig +short myip.opendns.com @resolver1.opendns.com)" HOST="$(hostname)" ; docker stop presearch-node ; docker rm presearch-node ; docker stop presearch-auto-updater ; docker rm presearch-auto-updater ; docker run -d --name presearch-auto-updater --restart=unless-stopped -v /var/run/docker.sock:/var/run/docker.sock presearch/auto-updater --cleanup --interval 900 presearch-auto-updater presearch-node && docker pull presearch/node && docker run -dt --name presearch-node --restart=unless-stopped -v presearch-node-storage:/app/node -e REGISTRATION_CODE=${REGCODE} -e DESCRIPTION="${HOST} ${IP}" presearch/node && sleep 10 && docker logs presearch-node'
What if you decide to add more servers later? Follow the same steps as above to setup new servers, just create a new text file, e.g., nodes-new.txt to put their config into until they are all setup. Then, just add them to your original nodes.txt to keep all of your servers together going forward.