Paige Liu's Posts

Lessons learned from setting up Azure IoT Nested Edge with X.509 Certificates

Manufacturing customers often run their workload on the factory floor in networks that follow ISA-95 standard - machines on the lower layer don’t have Internet access and can only communicate with machines a layer directly above. Additionally, manufacturing customers often have strict security requirements that only allow for certificate based authentication. Azure IoT Edge supports connecting a hierarchy of edge devices with the lower layer devices in an isolated network connecting to a top layer edge device acting as a gateway.

Tutorial: Create a hierarchy of IoT Edge devices uses the iotedge-config tool to automatically provision and configure the devices. Not being an IoT expert, I wanted to manually provision and configure a nested edge environment, using certificates for authentication where possible, so I can better understand how things work. I ran into a few issues where no readily available answers can be found on the Internet. In hindsight, they are mostly due to some confusing certificate related terminologies and my lack of understanding of how Azure IoT Hub and Edge work when it comes to authentication. Hope this blog will help others on the same journey.

The scenario to build

The goal is to create 2 edge devices, one in a subnet with Internet access (the parent edge device), and another in an isolated subnet that doesn’t have Internet access (the child edge device). Configure them such that we can see their status as if they are both connected to Azure IoT Hub, and the messages sent from a module on the child edge can go through the parent edge to IoT Hub.

We can build this out in Azure with VNET and VMs. Here are the main steps:

  1. Create a VNET with 2 subnets, layer3 and layer4.
  2. Create 2 Ubuntu VMs, edge3 in layer3 subnet and edge4 in layer4 subnet. Install Azure IoT Edge on both machines. Note that here we only install the bits, not yet register the devices to Azure.
  3. Add a Network Security Group rule to layer3 subnet that denies all inbound and outbound Internet connection. Verify that edge3 can no longer curl any public web site, for example, curl https://www.google.com will fail.
  4. Generate demo certificates. Note that for nested edge to work, the edge hostname used for registering devices must be FQDN or IP. The CN in the certificates must match the hostnames. So we’ll pass FQDN to the certificate generation commands.
  5. Register both the parent edge edge4 and the child edge edge3 to IoT Hub with the generated certificates.
  6. Configure the parent edge as a gateway, and connect edge3 as a child of edge4.
  7. Deploy the simulated temperature sensor to edge3 and verify the messages are received in IoT Hub.

Lessons learned

Most of the above steps are straightforward by following Azure documentations. The confusion is mostly around step 5. The document Configure gateways for IoT Edge devices assumes both parent and child edges are already registered, and therefore focuses on configuring the parent-child relatioship. But how do I register the edge devices with CA signed certificates, and when the child edge has no Internet connection?

Self-signed vs. (private or enterpise) CA signed certificates

My understanding is that a self-signed certificate is not signed with any authority, which is different from a certificate signed by a private or enterprise authority. The latter is still a CA-signed certificate even though the CA is not a trusted commericial certificate authority such as Baltimore, Verisign, Digicert, or GlobalSign. CA signed certificate is verified by tracing the signing chain and checking if the root authority is trusted. Self signed certificate doesn’t have an authority so the certificate is verified by comparing thumbprints in IoT Hub.

Since most of our manufacturing customers have their own enterprise certificate authority (CA), I wanted to understand how CA signed certificates work. Generate demo certificates tutorial create demo CA signed certificates.

Here comes the confusion, to manually create an Azure IoT Edge device, only two authentication options are supported - symmetric key and self-signed certificates. So how do I create an edge device with CA signed certificates? Azure IoT Device Provisioning Service (DPS) supports CA signed certificate. But the device needs to connect to DPS to be registered, which is fine with the parent edge, but how do I register the child edge without Internet connectivity? Authenticate a downstream device to Azure IoT Hub only seems to apply to non-edge downstream IoT devices.

Turns out for manual registration of an edge device, CA signed certificates can be treated the same way as self signed certificates. Just retrieve the thumbprint of the CA signed certificate and use it for both the primary and secondary thumbprints.

Create the parent edge

To provision the parent edge with DPS, it’s important to note -

With the parent edge provisioned and registered, the next step is to configure it as a gateway. We must deploy the necessary modules to the parent edge so it can support the child edge. Verify the following is working from the child edge machine before proceeding further -

Create the child edge

For the child edge to successfully communicate with the parent edge -

sudo cp <path>/<root ca certificate>.pem /usr/local/share/ca-certificates/<root ca certificate>.pem.crt
sudo update-ca-certificates

hostname = "<this-device-FQDN>"
parent_hostname = "<parent-FQDN>"

## Manual provisioning with X.509 certificate
[provisioning]
source = "manual"
iothub_hostname = "<myhub>.azure-devices.net"
device_id = "<this-device-FQDN>"

[provisioning.authentication]
method = "x509"

## identity certificate private key
identity_pk = "file:///path/to/iot-edge-device-identity-<name>.key.pem"

## identity certificate
identity_cert = "file:///path/to/iot-edge-device-identity-<name>-full-chain.cert.pem"

##
[agent]
name = "edgeAgent"
type = "docker"

[agent.config]
image: "<parent-FQDN>:443/azureiotedge-agent:<version 1.2 or above>"

Apply the configuration by running sudo iotedge config apply. Verify the status of the child edge is now available in IoT Hub.

FQDN and name resolution in docker container

The FQDN for the edge hostname must be resolvable not just from the host edge machines, but also from inside docker containers. This means if the hostnames are added to the host machine’s /etc/hosts, then they must be added as docker containers’ create options. In /etc/aziot/config.toml, this means the edge agent must be configured as following:

[agent.config]
image: "<parent-FQDN>:443/azureiotedge-agent:<version 1.2 or above>"
createOptions = { HostConfig = { ExtraHosts=["<FQDN_child>:<IP_child>", "<FQDN_parent>:<IP_parent>"], Binds = ["/iotedge/storage:/iotedge/storage"] } }

If we initially run the edge agent without this setting on the child edge and add it later, it won’t be updated unless we docker rm the existing edge agent container and rerun iotedge config apply. We can use docker inspect edgeAgent to verify it does have the ExtraHosts set correctly.

In IoT Hub, the deployment manifest should include the following in the create options for edgeAgent and edgeHub:

{
    "HostConfig": {
        "ExtraHosts": [
            "<FQDN_child>:<IP_child>",
            "<FQDN_parent>:<IP_parent>"
        ],
        "Binds": [
            "/iotedge/storage:/iotedge/storage"
        ]
    }
}

Summary of required certificates and their respective use

How are certificateificates used in authentication?