2023-03-29

Send Docker container traffic through second VNIC on Oracle Cloud Compute Instances

I have a need to route all traffic to a running Docker conainer through a second network interface card (NIC) on Oracle Cloud. Oracle Cloud calls these Virtual NICs as they are attached to a virtualized instance.

This guide assumes you are running Ubuntu and have Docker installed already.

On the Oracle Cloud Dashboard:

Create a new Subnet for your second VNICs
- From the hamburger menu, go to Networking / Virtual Cloud Networks then Open your default VCN (you may have to choose your Compartment on the left first)
- Click the ‘Create Subnet’ button
- For our example, we’re going to use 10.0.13.x as our subnet
- For any options not specified below, leave them on their default setting
- For ‘Name’, enter subnet-10-0-13-x
- For the ‘IPv4 CIDR block’, enter 10.0.13.0/16
- For ‘Route Table Compartment’, choose your default route table
- Choose Public Subnet for ‘Subnet access’
- For ‘DHCP Options Compartment’ choose your default options
- For ‘Security Lists’, choose your default security list
- Click the ‘Create Subnet’ button
- You should now see two subnets in the Subnet table
Create a new VNIC on your Ampere instance using the new subnet
- From the hamburger menu, go to Computer / Instances then open your Ampere instance
- Scoll down to find Attached VNICs on the left and click it
- Click the ‘Create New VNIC’ button
- For any options not specified below, leave them on their default setting
- For ‘Name’, enter anything you’d like, e.g., instancename-vnic-2
- ‘Select a virtual cloud network’ should be your default network
- For ‘Select a subnet’, choose the newly created subnet-10-0-13-x
- Check the box for ‘Skip source/destination check’
- Check the box for ‘Assign a public IPv4 address’
- Click the ‘Save changes’ button

Everything is now configured in the Oracle Dashboard, now we have to set everything up on the instance itself so ssh into it to complete the following steps.

Download the Oracle Cloud secondary VNIC configuration script, move it to /usr/local/sbin, make it executable, and run it:

wget https://docs.oracle.com/en-us/iaas/Content/Resources/Assets/secondary_vnic_all_configure.sh && \
chmod 755 ./secondary_vnic_all_configure.sh && \
sudo chown root:root ./secondary_vnic_all_configure.sh && \
sudo mv ./secondary_vnic_all_configure.sh /usr/local/sbin/ && \
sudo /usr/local/sbin/secondary_vnic_all_configure.sh -c

Create a custom Docker network to use the second VNIC. In the example here, I’ve named the network bridge-ps-2 and use that name throughout the following examples.

docker network create --subnet 172.22.0.0/16 --gateway=172.22.0.1 \
--attachable --opt com.docker.network.bridge.name=bridge-ps-2 \
--opt com.docker.network.bridge.enable_ip_masquerade=true \
--opt com.docker.network.bridge.host_binding_ipv4=$(hostname -I | awk '{print $2}') \
bridge-ps-2

Create a custom network in your route table:
```
sudo nano /etc/iproute2/rt_tables
```
- Use the arrow keys to get to the last line, then paste in
```
22     bridge-ps-2
```

Create a new script that will automatically run the necessary commands to route the secondary VNIC traffic via the network you just added to the route table:

sudo nano /usr/local/sbin/secondary_vnic_docker_routes.sh

Paste in the following:

#!/usr/bin/env bash
# Author: nabeards, copied from https://nabeards.com/send-docker-container-traffic-through-second-vnic-on-oracle-cloud-compute-instances/

# This must match the gateway address for the subnet you chose to create
declare LANGATEWAY="10.0.13.1"

ip rule add iif bridge-ps-2 table bridge-ps-2
ip route add table bridge-ps-2 172.22.0.0/16 dev bridge-ps-2 src 172.22.0.1
ip route add table bridge-ps-2 default via ${LANGATEWAY} dev enp1s0
ip rule add oif enp1s0 table bridge-ps-2

Configure the script to be executable and then run it:

sudo chmod 755 /usr/local/sbin/secondary_vnic_docker_routes.sh && \
sudo /usr/local/sbin/secondary_vnic_docker_routes.sh

Ensure the secondary VNIC’s Public IP address is now being properly routed:
- Get the second public IP:
```
curl --interface enp1s0 https://api.ipify.org
```
- Then get the IP the bridge-ps-2 network is bound to in Docker:
```
docker run --rm --network bridge-ps-2 byrnedo/alpine-curl https://api.ipify.org 2>/dev/null
```
- If the two IPs match, you are good to move forward!
To ensure the seconary VNIC works after a the instance is restarted, you have to configure the Oracle-provided script and the routing table script to run at reboot using cron:
```
sudo nano /etc/crontab
```
- Paste in at the bottom:
```
@reboot         root    /usr/local/sbin/secondary_vnic_all_configure.sh -c
@reboot         root    sleep 30 && /usr/local/sbin/secondary_vnic_docker_routes.sh
```

Now, to launch your Docker intance using this network, simply add the --network bridge-ps-2 switch to your docker launch command as shown in Step 8.

If you want this docker instance to use the secondary network after a reboot, you have to manually configure a script to do so. You cannot set the docker instance to restart automatically, i.e., you cannot use --restart=unless-stopped, because the docker instance will default to the primary VNIC instead of the secondary.

Instead, you’ll need to create a script to launch your docker instance directly. Below is an example script to launch a Presearch node.

sudo nano /usr/local/sbin/docker_start_node2.sh

#!/usr/bin/env bash
# Author: nabeards, copied from https://nabeards.com/send-docker-container-traffic-through-second-vnic-on-oracle-cloud-compute-instances/

REGCODE="{your registration code}"
INSTANCE=2
IP="$(dig +short myip.opendns.com @resolver1.opendns.com)"
HOST="$(hostname | cut -d"." -f1)"
ID=''
if [ ${#INSTANCE} -eq 0 ]; then ID=1; else ID=${INSTANCE}; fi; 

docker stop presearch-node${INSTANCE}
docker rm presearch-node${INSTANCE}
docker pull presearch/node
docker run -dt --network bridge-ps-${ID} --name presearch-node${INSTANCE} --restart=no -v presearch-node-storage${INSTANCE}:/app/node -e REGISTRATION_CODE=${REGCODE} -e DESCRIPTION="${HOST}-${ID} ${IP}" presearch/node

Set the script to be executable:

sudo chmod 755 /usr/local/sbin/docker_start_node2.sh

Run the script to ensure your node will launch properly.

sudo /usr/local/sbin/docker_start_node2.sh

Check the Nodes Dashboard to verify the new node launched successfully.

Then add the script to your crontab:

crontab -e

Paste in at the bottom:

@reboot              sleep 60 && /usr/local/sbin/docker_start_node2.sh

Now, on a reboot, your system will run the secondary VNIC configuration script, then add the necessary entries to the route table, and finally will launch your docker instance using the secondary VNIC.

One final note specific to Presearch nodes: you will want to update your watchtower tool so that it’s aware of both nodes running on the instance. To do so, you can run the following:

TOTAL=`docker ps | grep presearch-node | wc -l` NODES="presearch-node" ; \
for (( c=2; c<=$TOTAL; c++ )); do NODES="${NODES} presearch-node${c}"; done ; \
docker stop presearch-auto-updater ; docker rm presearch-auto-updater ; \
docker run -d --name presearch-auto-updater --restart=unless-stopped \
-v /var/run/docker.sock:/var/run/docker.sock presearch/auto-updater \
--log-opt max-size=10m --cleanup --interval 900 presearch-auto-updater ${NODES}

And, finally, to make sure it’s all working, restart your Instance and make sure everything comes back up!

sudo reboot