Skip to content

Azure INFRA and WORKLOAD machine creation failed #148

@qiliRedHat

Description

@qiliRedHat

https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/scale-ci/job/e2e-benchmarking-multibranch-pipeline/job/cluster-workers-scaling/708/console

% oc get machineset -A
NAMESPACE               NAME                                               DESIRED   CURRENT   READY   AVAILABLE   AGE
openshift-machine-api   infra-northcentralus2                              1         1                             3h5m
openshift-machine-api   infra-northcentralus3                              1         1                             3h5m
openshift-machine-api   infra-qili-preserve-az0516-sr44j1                  1         1                             3h5m
openshift-machine-api   qili-preserve-az0516-sr44j-worker-northcentralus   3         3         3       3           3h55m
openshift-machine-api   workload-qili-preserve-az0516-sr44j                1         1                             3h5m
% oc get machines -A | grep infra
openshift-machine-api   infra-northcentralus2-82z2h                              Failed                                              3h8m
openshift-machine-api   infra-northcentralus3-2klrt                              Failed                                              3h8m
openshift-machine-api   infra-qili-preserve-az0516-sr44j1-lbhxq                  Failed                                              3h8m

Describing the machine, machine creation failed for Please make sure that the referenced resource exists, and that both resources are in the same region

  Error Message:           failed to reconcile machine "infra-northcentralus2-82z2h": network.InterfacesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidResourceReference" Message="Resource /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/qili-preserve-az0516-sr44j-rg/providers/Microsoft.Network/virtualNetworks/qili-preserve-az0516-sr44j-vnet/subnets/qili-preserve-az0516-sr44j-worker-subnet referenced by resource /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/qili-preserve-az0516-sr44j-rg/providers/Microsoft.Network/networkInterfaces/infra-northcentralus2-82z2h-nic was not found. Please make sure that the referenced resource exists, and that both resources are in the same region." Details=[]

Check the infra machineset yaml, location is centralus.

% oc get machinesets/infra-northcentralus2 -n openshift-machine-api -o yaml
...
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: qili-preserve-az0516-sr44j
      machine.openshift.io/cluster-api-machineset: infra-northcentralus2
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: qili-preserve-az0516-sr44j
        machine.openshift.io/cluster-api-machine-role: infra
        machine.openshift.io/cluster-api-machine-type: infra
        machine.openshift.io/cluster-api-machineset: infra-northcentralus2
    spec:
      lifecycleHooks: {}
      metadata:
        labels:
          node-role.kubernetes.io/infra: ""
      providerSpec:
        value:
          apiVersion: azureproviderconfig.openshift.io/v1beta1
          credentialsSecret:
            name: azure-cloud-credentials
            namespace: openshift-machine-api
          image:
            offer: ""
            publisher: ""
            resourceID: /resourceGroups/qili-preserve-az0516-sr44j-rg/providers/Microsoft.Compute/images/qili-preserve-az0516-sr44j
            sku: ""
            version: ""
          kind: AzureMachineProviderSpec
          location: centralus
          managedIdentity: qili-preserve-az0516-sr44j-identity
          metadata:
            creationTimestamp: null
          osDisk:
            diskSettings: {}
            diskSizeGB: 128
            managedDisk:
              storageAccountType: Premium_LRS
            osType: Linux
          publicIP: false
          resourceGroup: qili-preserve-az0516-sr44j-rg
          subnet: qili-preserve-az0516-sr44j-worker-subnet
          userDataSecret:
            name: worker-user-data
          vmSize: Standard_D48s_v3
          vnet: qili-preserve-az0516-sr44j-vnet
          zone: "2"

Checking code
https://github.com/openshift-qe/ocp-qe-perfscale-ci/blob/8f3eb8793de587d58a8322db57d77dc59f070750/Jenkinsfile#L297

But the worker node machinesets is actually on 'northcentralus'

 % oc get machineset/qili-preserve-az0516-sr44j-worker-northcentralus -n openshift-machine-api -o yaml
...
spec:
  replicas: 3
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: qili-preserve-az0516-sr44j
      machine.openshift.io/cluster-api-machineset: qili-preserve-az0516-sr44j-worker-northcentralus
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: qili-preserve-az0516-sr44j
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: qili-preserve-az0516-sr44j-worker-northcentralus
    spec:
      lifecycleHooks: {}
      metadata: {}
      providerSpec:
        value:
          acceleratedNetworking: true
          apiVersion: machine.openshift.io/v1beta1
          credentialsSecret:
            name: azure-cloud-credentials
            namespace: openshift-machine-api
          image:
            offer: ""
            publisher: ""
            resourceID: /resourceGroups/qili-preserve-az0516-sr44j-rg/providers/Microsoft.Compute/images/qili-preserve-az0516-sr44j-gen2
            sku: ""
            version: ""
          kind: AzureMachineProviderSpec
          location: northcentralus

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions