27 Jan How to Survive a Brutal VCF 9 DoD Deployment: The Depot Part 2

Posted at 22:17h in VCF by

In my previous post, How to Survive a Brutal VCF 9 DoD Deployment: The Depot, I mentioned that we ran into some significant issues with NFS. I promised a deep dive into our NFS obstacles. However, I first want to provide a guide for the Offline Depot. It took significant effort to get this working correctly.

The official documentation often leaves gaps. These omissions made our process much more difficult. I hope that sharing our experience helps you navigate these same hurdles. You can now get your secure environment running without the trial and error we faced.

Deploying the Photon OVA

We began by following the official Broadcom technical documentation. Since security protocols prevented us from downloading the Photon OVA directly from GitHub, we had to get a bit creative to bring that file into our secure environment. Aside from choosing a custom hostname for our Offline Depot, we followed every other step exactly as the guide outlined.

Solving the Apache Installation Errors

In step 7, the guide instructs you to run the tdnf command to download the Apache HTTP Server and update the OS. If you’re working in a hardened environment, this is where things usually get complicated. You might remember the SSL Inspection issue I mentioned earlier. Because of that interception, we encountered the following error:

tdnf ssl error

This error effectively stops the installation in its tracks. The system cannot verify the certificate chain from the inspection device. Consequently, it refuses to pull the Apache binaries from the remote repository.

You may see a “Certificate Verify Failed” message. You might also see “Peer certificate cannot be authenticated.” If so, you are likely dealing with a security appliance.

Bypassing SSL for tdnf

We did not waste time fixing the complex tdnf configuration. Instead, we used a quick workaround to bypass SSL verification. This allowed us to move forward.

I always recommend a proper certificate fix for long-term stability. However, sometimes you must hit your project milestones. In those cases, getting the binaries down is the priority.

The quick fix is to edit the photon-updates.repo file. You must add sslverify=0 to the configuration. Use the following one-liner to open the file with the VI editor and make your changes:

vi /etc/yum.repos.d/photon-updates.repo

Keep in mind that this is only a temporary fix. I do not recommend leaving this setting active in a production environment. Once you finish the installation, you should address the root certificate issues.

Correcting the Apache DocumentRoot

The official documentation does not perfectly match Photon OS v. 5 (Hardware Version 15). If you use this version, the directory paths will look different than the guide suggests.

Step 12-b tells you to search for DocumentRoot "/var/www/html". However, in this version of Photon, the default path is actually: DocumentRoot "/etc/httpd/html"

You must find this line to proceed with your configuration.

DocumentRoot "/ect/httpd/html"
<Directory "/etc/httpd/html">

This is a critical step because if the pathing doesn’t match, the installer won’t be able to find your files. You must edit this section of the file so it points to /var/www/html. Once you finish your edits, the configuration should look like this:

DocumentRoot "/var/www/html"
<Directory "/var/www/html">

In our case, the Require all granted setting was already in place, but you should double-check your own configuration to be certain. If your file shows anything else, go ahead and make that change now. Once you have verified that line, you can proceed with the remaining steps in the Broadcom documentation.

Filling the Documentation Gaps

At this point, you might assume you are ready to move forward. However, we must fill some gaps in the official documentation. Because it just stops. No Next Steps, no link to another guide to move you along.

Experienced consultants likely know these next steps by heart. Still, these omissions can frustrate a novice user. It is often overwhelming to determine the correct path forward. I’ll walk you through those missing pieces so you can get this done.

Preparing the Directory Permissions

Every environment is unique. These specific steps worked for us, but your results may vary. Keep your specific security constraints in mind as you proceed.

First, we changed the permissions on three key directories. These include /var, /var/www, and /var/www/html. We then restarted the Apache service. This allowed us to view the directory structure via the FQDN. You can use the following commands to achieve this:

chmod 755 /var
chmod 755 /var/www
chmod 755 /var/www/html
systemctl restart httpd

Transferring the VCF Download Tool

Next, you need to get the VCF Download Tool onto the Offline Depot. First, download the tool from the Broadcom Support Page. You can then use a tool like WinSCP to place the .tar.gz file into the /tmp directory.

Once the file is in place, use an SSH client like PuTTY to run the following commands. The first command creates a new directory called vcf-download-tool in the /root folder. The second command unzips the tool into that new directory. Finally, you will change to the /bin directory. This is where you will run the VCF Download Tool. Those commands are below:

mkdir -p /root/vcf-download-tool
tar -xzvf /tmp/vcf-download-tool-version.tar.gz -C /root/vcf-download-tool
cd /root/vcf-download-tool/bin

Setting Up Your Download Token

You will now need your download token. Reach out to your Broadcom Technical Adoption Manager (TAM) if you do not have it. They can provide this for you.

We must create a .txt file with your token. The VCF Download Tool will reference this file in future commands. You can do this entirely within PuTTY.

Ensure you are in the /vcf-download-tool/bin directory. If you are not there yet, change to that directory now. Copy your token to your clipboard and run the following command:

vi token.txt

To edit in VI, you must press the “i” key. After you paste your token, press ESC. Then, type :wq to save and close the file.

You can check that the file exists by typing ls, and should see the token.txt file listed in the directory. You can also verify the file contents by typing cat token.txt. This command prints the token on your screen.

I have redacted the download token in the screenshot below. However, you can see exactly how these commands should look in your terminal.

Commands to show download token file

Diagnosing the Connection Failure

We tried to reach out to the Broadcom Depot to download the files. However, we immediately hit the following error:

“Internal error while validating credentials. Network is unreachable”

At first, we assumed this was a firewall issue. We pursued that lead while the network team investigated. After digging into the logs, we found the following:

[ERROR] Internal error while validating credentials: Network is unreachable java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at com.vmware.vcf.download.util.DepotClient.validateCredentials(DepotClient.java:142) ... Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

This log changed everything. I realized the “Network is unreachable” error was just a generic message. The real issue was an SSL/TLS handshake failure. This was caused by Deep Packet Inspection (DPI) or SSL Interception.

The “PKIX path building failed” error is very specific. It means the Java TrustStore does not recognize the certificates from the intercepting proxy. This was our root cause. Now I’ll show you exactly how we fixed it.

VCF Certificate Patching Script

This script fixes the “Network is unreachable” error. It allows Java to trust your security appliance.

When the script connects to Broadcom servers, the security appliance intercepts the traffic. It then presents its own certificates. This script captures that entire certificate chain.

The script uses csplit to break the certificate bundle into individual files. Java requires this specific format to process them. It then injects these certificates into two critical locations. These include the standard system Java keystore and the private environment for the VCF Download Tool.

By adding these certificates to the cacerts trust stores, the tool recognizes the security appliance as a trusted entity. It no longer views it as a security threat. This finally allows the credential validation and download process to finish.

#!/bin/bash

# --- CONFIGURATION ---
TARGET_URL="dl.broadcom.com:443"
KEYTOOL_CMD="/root/vcf-download-tool/jre/lin64/bin/keytool"
KEYSTORE="/root/vcf-download-tool/jre/lin64/lib/security/cacerts"

# --- CHECK REQUIREMENTS ---
if [ ! -f "$KEYTOOL_CMD" ]; then
    echo ""
    echo "CRITICAL ERROR: keytool not found at $KEYTOOL_CMD"
    exit 1
fi

# --- STEP 1: DOWNLOAD ---
echo "========================================"
echo "Step 1: Downloading Chain from $TARGET_URL..."
echo "========================================"

openssl s_client -showcerts -connect $TARGET_URL </dev/null 2>/dev/null | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > /tmp/fullchain.pem

# Check if file downloaded and has content
if [ ! -s /tmp/fullchain.pem ]; then
    echo "ERROR: Download failed. Check network/firewall."
    exit 1
fi

# Split certs (remove old ones first)
rm -f /tmp/cert-*
csplit -f /tmp/cert- /tmp/fullchain.pem '/BEGIN CERTIFICATE/' '{*}' -s

# --- STEP 2: IMPORT ---
echo "========================================"
echo "Step 2: Importing Certificates"
echo "========================================"

for file in /tmp/cert-*; do
    if [ -s "$file" ] && grep -q "BEGIN CERTIFICATE" "$file"; then
        ALIAS="dod-proxy-cert-$(basename $file)"
        
        echo "Processing $ALIAS..."
        
        # 1. Delete alias if it exists (Force Update)
        "$KEYTOOL_CMD" -delete -alias "$ALIAS" -keystore "$KEYSTORE" -storepass changeit 2>/dev/null
        
        # 2. Import new cert
        "$KEYTOOL_CMD" -import -alias "$ALIAS" -file "$file" -keystore "$KEYSTORE" -storepass changeit -noprompt
            
        if [ $? -eq 0 ]; then
            echo "SUCCESS: Imported $(basename $file)"
        else
            echo "FAILED: Could not import $(basename $file). See error above."
        fi
    fi
done

echo "========================================"
echo "DONE. All Java environments are now patched."
echo "========================================"

Initiating the Download

We can finally run the command to start the download. However, we were still in trial and error mode. We encountered a recurring issue with the PROD directory location.

The system should create this directory in the /var/www/html directory. Instead, it kept appearing in the /root directory. We never discovered the specific cause, but the issue was repeatable.

To bypass this, we created a symbolic link. This “tricked” the Download Tool into placing the directory in the correct spot. You can follow our process below.

First, you must run the command to initiate the download. It looks like this:

./vcf-download-tool binaries download --depot-store /var/www/html --vcf-version 9.0.1 --depot-download-token-file /root/vcf-download-tool/bin/token.txt --automated-install --type INSTALL

Wait for the prompt asking you to join CIEP. At this point, stop and use WinSCP to check the /root directory.

Look for a folder named /PROD. If you see it there, you must apply this fix. This means the tool is placing files in the wrong location.

First we’ll remove the current PROD directory in /root. Then we will make a new PROD directory in the /var/www/html directory, where it should be. Finally we create the symbolic link. If you need to break out of the Download Tool and get back to the Shell just press Crtl+C.

rm -rf /root/PROD
mkdir -p /var/www/html/PROD
ln -s /var/www/html/PROD /root/PROD

Now we can use the up arrow to get back to the command to start the download. Say no to joining CEIP then sit back and wait for the downloads to complete. Once they are done you’ll want to change ownership and permissions on the newly downloaded files by running these commands.

chown -R apache:apache /var/www/html
find /var/www/html -type d -exec chmod 755
find /var/www/html -type f -exec chmod 755

Bash Script for Full Download

This script can be used to do a full download of binaries and metadata as well as applying the permission/ownership fixes once the files are downloaded.

#!/bin/bash

# Define variables for better readability/maintenance
BIN_PATH="/root/vcf-download-tool/bin/vcf-download-tool"
DEPOT_PATH="/var/www/html"
TOKEN_FILE="/root/vcf-download-tool/bin/token.txt"

echo "========================================"
echo "Starting VCF 9.0.1 Binary Download..."
echo "========================================"

# Use && to ensure we only fix permissions if the download actually succeeds
$BIN_PATH binaries download \
  --vcf-version 9.0.1 \
  --depot-store "$DEPOT_PATH" \
  --depot-download-token-file "$TOKEN_FILE" \
  --automated-install \
  --type INSTALL && {

    echo "----------------------------------------"
    echo "Download Successful. Applying Permissions..."
    echo "----------------------------------------"

    # Change ownership to the web server user
    chown -R apache:apache "$DEPOT_PATH"

    # Optimized permission fixes using '+' for speed
    find "$DEPOT_PATH" -type d -exec chmod 755 {} +
    find "$DEPOT_PATH" -type f -exec chmod 755 {} +

    echo "SUCCESS: Download and Permissions Fix Complete."
} || {
    echo "CRITICAL ERROR: Download failed. Permissions were not modified."
    exit 1
}

Bash Script for Updates Only

Since full binaries are typically for the initial deployment, most times we will only need to download updates. Here’s the script for that.

#!/bin/bash

# Define Variables
BIN_TOOL="/root/vcf-download-tool/bin/vcf-download-tool"
TARGET_DIR="/var/www/html"
TOKEN="/root/vcf-download-tool/bin/token.txt"

echo "Checking for VCF 9.0.1 Upgrade Binaries..."

$BIN_TOOL binaries download \
  --vcf-version 9.0.1 \
  --depot-store "$TARGET_DIR" \
  --depot-download-token-file "$TOKEN" \
  --type UPGRADE && {
    
    echo "Download successful. Updating permissions for Apache..."
    
    # Standardize ownership
    chown -R apache:apache "$TARGET_DIR"

    # Optimized permission changes: '+' handles multiple files per process
    find "$TARGET_DIR" -type d -exec chmod 755 {} +
    find "$TARGET_DIR" -type f -exec chmod 755 {} +

    echo "DONE: Upgrade binaries are ready for SDDC Manager."
} || {
    echo "ERROR: Upgrade download failed. Check /root/vcf-download-tool/logs/ for details."
    exit 1
}

Connecting the Installer to the Depot

Now we have downloaded the files and fixed the permissions. It is time to connect everything to our Installer Appliance.

Return to the Installer and click on Depot Settings. Next, select Binary Management. Click Configure under the Offline Depot section.

Enter the specific details for your Offline Depot here. Once finished, click Configure.

Secure protocol communication error connecting Offline Depot to VCF 9 Installer Appliance

Resolving Trust Issues Between the Installer and Depot

After all the trial and error, we were confident the deployment was finally starting. Unfortunately, we hit another roadblock with a connection error. I reviewed the logs and immediately confirmed this was an SSL certificate issue.

The VCF Installer Appliance used a self-signed certificate. Meanwhile, the Offline Depot used a custom CA-signed certificate. Since they did not trust each other, the handshake failed. I decided the best fix was to add the Offline Depot certificate chain to the Installer’s trust store.

I initially followed the Broadcom KB to import the certificates. However, the fix still would not work. I was nearly ready to open a Support Request (SR) when I found an article by William Lam.

William addressed this exact issue using similar commands. The breakthrough was the location of the keystore. The Broadcom KB pointed to one path, but William identified the correct one for this version. We pointed the commands to the right keystore, and the connection worked immediately.

Automating Certificate Trust with a One-Liner

Instead of asking the customer to manually collect the Offline Depot, intermediate, and root certificates to create a .pem file, I decided to use a one-liner. This script automates the entire process. It queries your Depot by IP and pulls the full certificate chain into a file named depot.pem. Finally, it imports that file into the correct keystore automatically. Just be sure to edit the IP address to reflect the IP of your Offline Depot VM and change the -alias to reference your Offline Depot hostname or FQDN.

Also, incase you don’t know, when you SSH into the Installer Appliance to run this, login with username vcf and your password. Then use su – to elevate to root before running the command.

openssl s_client -connect 192.168.10.51:443 </dev/null 2>/dev/null | \
sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > /tmp/depot.pem && \
keytool -import -trustcacerts -alias pl-vcfsddc -file /tmp/depot.pem \
-keystore /usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts \
-storepass changeit -noprompt

With the chain now imported we need to restart the VCF Installer services. 

echo 'y' | /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Now we can refresh our browser and add in the info for our Offline Depot again.

Offline Depot Credentials

Once connected you’ll see all the components show up under Binary Management. Click the drop down next to Version and select the right version, then check the box to select all components, and click Download. Now wait for the files to be downloaded then you’ll be able to start your deployment.

VCF component download

There is one more caveat to the VCF Download Tool. You must configure ESXi downloads separately. This process uses UMDS inside the tool.

Since this post is already long, I will cover that setup next. We will need to run another certificate script for that process. This is necessary because UMDS uses a different Java TrustStore than the rest of the system.

Join the Conversation

Have you run into these documentation gaps during your own VCF deployments? Secure environments often throw unexpected curveballs like these.

I would love to hear how you handled these SSL or directory issues. Drop a comment below or find me on LinkedIn to share your experience. Let’s help each other build better, more secure environments!

Continue the Journey

This post is part of a series dedicated to navigating the complexities of VMware Cloud Foundation 9.

Tags:
, , , , ,
Print page
0 Likes
Chris Pope
[email protected]

Certified Senior Virtualization Engineer with over 13 years of experience designing, deploying, and optimizing VMware and Omnissa environments across secure DoD and NATO systems. Adept at streamlining hybrid cloud operations, executing complex P2V migrations, and enhancing disaster recovery. Skilled communicator who simplifies complex technology for users and teams.

No Comments

Post A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.