27 Jan How to Survive a Brutal VCF 9 DoD Deployment: The Depot

In our last installment, How to Survive a Brutal VCF 9 DoD Deployment: ESX Hosts, we covered how to properly configure your hosts to set the stage for success. Once your hosts are ready, your next big step is deploying the VCF Installer Appliance.

This post, and perhaps the next one, depending on how much ground we need to cover, focuses on the VCF 9 Offline Depot. While the appliance deployment is your immediate task, setting up the depot correctly is what actually fuels the installation in a secure environment.

While the Installer Appliance deployment process itself is relatively straightforward, a few strategic details can save you from a world of frustration if things don’t go perfectly on the first try.

The Evolution of the SDDC Manager

It’s important to understand where the architecture is heading. As of VCF 9.0.1, the SDDC Manager is still a core component. However, Broadcom plans to transition away from the SDDC Manager in favor of VCF Operations in an upcoming major release.

For now, you have a choice to make during the initial setup that affects your recovery options:

• Option A: If you install the appliance on a host destined for your Management Domain, that appliance actually becomes the SDDC Manager during the deployment.
• Option B: If you run the appliance from a jump box or a host outside the target Management Domain, the process deploys a brand-new, separate SDDC Manager.

Protecting Your Progress: Why Placement Matters

In a perfect world, every deployment would work the first time. But as we know, in complex DoD environments, failure is a real possibility. If your deployment hits a snag, you’ll likely have to wipe and rebuild your hosts.

If you chose Option A and put your Installer Appliance on one of those hosts, you’ll lose the appliance when you reset the host. That means you’ll have to redeploy the appliance and re-configure your Depot settings from scratch.

While that might only take a few extra minutes, those minutes feel like hours when leadership is breathing down your neck to meet a milestone. To avoid unnecessary rework, I recommend deploying the Installer Appliance in a “safe” location—somewhere outside your target management cluster. This keeps your installer intact even if you have to start the host imaging process over again.

The “Offline Depot” Obstacle Course

In a standard environment, you would simply follow the Broadcom guide to build your depot. However, in a DoD setting, that guide often falls short. For example, the standard method requires downloading an OS directly from GitHub, a move that most security protocols ban immediately.

Since we couldn’t follow the standard script, the local team built a Red Hat Enterprise Linux (RHEL) server for us to use as our Offline Depot. We hit a wall almost immediately. When we tried to run simple commands like sudo dnf install httpd to install Apache, it failed. The network blocked access to the external repositories, meaning we couldn’t even fetch the Apache binaries.

We went back to the drawing board and requested a new build from the Linux admin, this time with Apache pre-installed. With a working web server finally in place, we moved to the next step: installing the VCF Download Tool to pull the VCF 9 files.

The “Network is Unreachable” Mystery

As soon as we initiated the download from the Broadcom repository, we were blocked again. The tool returned a blunt error: “Network is unreachable.” When we checked with the Network team, they insisted there were no firewall rules blocking our path to Broadcom. However, a deeper look into the logs revealed the real culprit:

sun.security.validator.ValidatorException: PKIX path building failed: unable to find valid certification path to requested target

That specific error is a “smoking gun” for SSL/TLS Inspection or Deep Packet Inspection (DPI). It happens when a security appliance intercepts the connection and presents its own certificate, which the VCF tool doesn’t recognize or trust.

Even though the local network team assured us they weren’t doing this on their hardware, the log didn’t lie. Something upstream was intercepting our traffic. We were left with two choices: hunt down the specific device to get the correct certificate chain or find a different way forward. To keep the project moving, we decided to shift gears while the team investigated the upstream blockers.

The Third Option: The VCF Download Tool

With an Online Depot off the table and the Offline Depot giving us trouble, we found there’s a third option, so we pursued this option in an effort to keep forward progression. On the Depot Settings page of the Installer Appliance there’s a note as you can see in the following screenshot.

This references this guide from Broadcom that gives you another option if you can’t connect the Installer Appliance to an Online or Offline Depot.

To download your binaries using the VCF Download Tool, you can follow these four steps:

1. Download the tool: Grab the latest version of the VMware Cloud Foundation Download Tool from the Broadcom Support Portal.
2. Retrieve the binaries: Use the tool on a computer with internet access to download the metadata and binaries for your specific VCF version.
3. Transfer the data: Copy those downloaded files directly onto your VCF Installer appliance.
4. Upload to the repository: Run the tool again on the VCF Installer appliance to move the files into the internal repository.

Moving Toward a Successful Deployment

Once my customer used the VCF Download Tool to grab the necessary binaries and metadata, we moved the files into their environment using a jump drive. From there, we simply transferred those files to the Installer Appliance and ran the tool again to populate the internal repository.

With the repository ready, we began navigating the deployment wizard based on the data in our Deployment Workbook. While this allowed us to move forward, the road was not entirely smooth. We encountered significant issues using NFS as the default datastore for the Management Domain, which I will cover in detail in a future post.

Confronting the Certificate Crisis

Fast-forwarding to our successful VCF deployment, we still faced a major hurdle. Even with VCF up and running, we needed a permanent Offline Depot to handle future updates and security patches.

Our previous attempts to trace the various pieces of DoD networking equipment to find a full certificate chain had failed. We knew we had to address the SSL Inspection and certificate issues head-on to find a permanent fix. I will detail exactly how we finally conquered those “Network unreachable” errors and fixed the certificate chain in my next post.

Join the Conversation

I’d love to hear about your experience. Have you run into a specific roadblock while setting up your Offline Depot, or perhaps a “Network Unreachable” error that left you scratching your head? Whether you found a creative workaround or you’re currently stuck on a configuration step, leave a comment below. Sharing these lessons helps the entire community move past the “vanilla” documentation and get these complex systems across the finish line.

Continue the Journey

This post is part of a series dedicated to navigating the complexities of VMware Cloud Foundation 9.