DataShield - Run your Docker images in Software Guard Extensions Enclaves and Protect your Application at Runtime

Install the IBM CLI

  curl -sL https://ibm.biz/idt-installer | bash

The above will install IBMCloud together with a couple of CLI useful for every developer. In particular it will install:

  - Homebrew (Mac only)
  - Git
  - Docker
  - Helm
  - kubectl
  - curl (Linux™ only)
  - IBM Cloud Developer Tools plug-in
  - IBM Cloud Functions plug-in
  - IBM Cloud Object Storage plug-in
  - IBM Cloud Container Registry plug-in
  - IBM Cloud Kubernetes Service plug-in

Another option will be the one of installing the necessary tools:

• IBM Cloud
• Kubernetes
• Docker
• Helm
• Kubernetes Service (ibmcloud plugin)
• Container Registry (ibmcloud plugin)

individually.

A good place to get started in this sense is to login in the IBM cloud via CLI through

  ibmcloud login [-a API_ENDPOINT] [--sso] [-u USERNAME] [-p PASSWORD] [--apikey KEY | @KEY_FILE] [--no-iam] [-c (ACCOUNT_ID | ACCOUNT_OWNER_USER_ID) | --no-account] [-g (RESOURCE_GROUP_NAME | RESOURCE_GROUP_ID)] [-r REGION | --no-region] [-o ORG] [-s SPACE]

and look at the available plug-ins, the installed ones and the available updates:

  ibmcloud plugin repo-plugins -r "IBM Cloud"

You can then install ibmcloud plugins through

  ibmcloud plugin install container-registry

  ibmcloud plugin install kubernetes-service

Moreover, you can install the necessary Kubernetes, Docker and Helm CLI from the internet should you not be willing of running the bash script above.

Notice: In order to leverage the IBM Data Shield service, it will be necessary for you to deploy a bare metal Kubernetes cluster running on SGX enabled processors. Moreover, to the present stage you will have to select a cluster running on the Ubuntu 16 OS, i.e. either the mb2c.4x32 cluster as done in this demo, or the mb2c.4x32.1.9tb.ssd model.

After provisioning the above cluster you should receive an E-mail a couple of hours later, when your cluster is up and running.

You should be then be able to find your cluster via the IBM kubernetes CLI plug-in.

 ibmcloud ks cluster ls

 OK
 Name                              ID                     State    Created       Workers   Location    Version        Resource Group Name   Provider   
 mycluster-fra02-mb2c.Datashield   bti5pcef0j3046qt1ks0   normal   2 weeks ago   1         Frankfurt   1.17.12_1540   Default               classic   

You can read from the above the ID of your cluster as well as it's state. This should be in the normal state as in the above case.

Through the above cluster-ID it is then possible to connect your local kubernetes CLI with the created cluster. In such a way you will be able to operate on your cluster via the kubernetes CLI locally.

In order to do that, you can run the following command:

 ibmcloud ks cluster config --cluster bti5pcef0j3046qt1ks0                                                                                        <

 OK
 The configuration for bti5pcef0j3046qt1ks0 was downloaded successfully.

 Added context for bti5pcef0j3046qt1ks0 to the current kubeconfig file.
 You can now execute 'kubectl' commands against your cluster. For example, run 'kubectl get nodes'.

You can then verify that your CLI is correctly configured with your cluster by running

 kubectl cluster-info

 Kubernetes master is running at https://c106.eu-de.containers.cloud.ibm.com:31398
 CoreDNS is running at https://c106.eu-de.containers.cloud.ibm.com:31398/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
 kubernetes-dashboard is running at https://c106.eu-de.containers.cloud.ibm.com:31398/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy                          
 Metrics-server is running at https://c106.eu-de.containers.cloud.ibm.com:31398/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
 NodeLocalDNS is running at https://c106.eu-de.containers.cloud.ibm.com:31398/api/v1/namespaces/kube-system/services/node-local-dns:dns/proxy

 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

You can then compare this information with the one obtained through the ibmcloud kubernetes CLI plug-in

ibmcloud ks cluster get --cluster bti5pcef0j3046qt1ks0                                                                                          

Retrieving cluster bti5pcef0j3046qt1ks0...
OK
                                   
Name:                           mycluster-fra02-mb2c.Datashield   
ID:                             bti5pcef0j3046qt1ks0   
State:                          normal   
Created:                        2020-09-18T07:07:29+0000   
Location:                       fra02   
Master URL:                     https://c106.eu-de.containers.cloud.ibm.com:31398   
Public Service Endpoint URL:    https://c106.eu-de.containers.cloud.ibm.com:31398   
Private Service Endpoint URL:   -   
Master Location:                Frankfurt   
Master Status:                  Ready (1 week ago)   
Master State:                   deployed   
Master Health:                  normal   
Ingress Subdomain:              mycluster-fra02-mb2c-da-f9bf7c7800de9856ddeffe14aad922a5-0000.eu-de.containers.appdomain.cloud   
Ingress Secret:                 mycluster-fra02-mb2c-da-f9bf7c7800de9856ddeffe14aad922a5-0000   
Workers:                        1   
Worker Zones:                   fra02   
Version:                        1.17.12_1540   
Creator:                        -   
Monitoring Dashboard:           -   
Resource Group ID:              f274228d8aa542fb8d3d120e7215071d   
Resource Group Name:            Default   

For running datashield, you will then be asked to install a couple of helm charts on your cluster. These will be necessary for the proper configuration of your service.

Helm charts, are kubernetes resources which are packaged together in an easy to deploy way for developers. You can think them in terms of libraries that trigger kubernetes configuration files.

In order to do that by following this documentation make sure that you are using the helm client in it's third version

helm version

version.BuildInfo{Version:"v3.2.3", GitCommit:"8f832046e258e2cb800894579b1b3b50c2d83492", GitTreeState:"clean", GoVersion:"go1.13.12"}

You will then be able to add to the helm client the IBM kubernetes helm charts.

helm repo add iks-charts https://icr.io/helm/iks-charts

You can finally add the IBM kubernetes helm charts to your repository

helm repo add iks-charts https://icr.io/helm/iks-charts

And finally install the cert-manager chart to issue certificates in your cluster.

kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/v0.10.1/deploy/manifests/00-crds.yaml

Create a namespace on your cluster where to deploy your cert-manager chart issuing the necessary certificates. Finally give it a label to reference it at later points.

kubectl create namespace cert-manager
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

Add the jetstack charts to your repository

Install the cert-manager chart in the cert-manager namespace

helm repo update && helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.10.1 --set extraArgs[0]="--enable-certificate-owner-ref=true" --set webhook.enabled=false

You should then be able to observe the deployed pods from the helm chart in the cert-manager namespace

kubectl get pods --namespace=cert-manager

| NAME                                    | READY | STATUS  | RESTARTS | AGE |
| cert-manager-87f84bbbb-m2ghd            | 1/1   | Running |        0 | 19d |
| cert-manager-cainjector-98fb4f86c-brdd5 | 1/1   | Running |        1 | 19d |

So far so good.

In this section we will create an IBM container registry where you can upload your docker images.

This will be a private registry so that you can keep your code proprietary.

Again you will need the container-registry ibmcloud CLI plug-in.

ibmcloud cr login

You can then select a region where to host your registry. This you will need to pass at a later stage when creating a kubernetes secret in order to interact with the registry.

ibmcloud cr region-set

1. ap-north ('jp.icr.io')
2. ap-south ('au.icr.io')
3. eu-central ('de.icr.io')
4. global ('icr.io')
5. uk-south ('uk.icr.io')
6. us-south ('us.icr.io')
Enter a number ()> 3
The region is set to 'eu-central', the registry is 'de.icr.io'.

OK

You will then add a namespace to the registry endpoint. You will have access to such namespace and manage your docker images from there.

ibmcloud cr namespace-add datashield_mahass

Check the successful creation of the namespace

ibmcloud cr namespace-list

Listing namespaces for account 'IBM' in registry 'de.icr.io'...

Namespace   
datashield_mahass   

OK

You can now add images to your namespace and check what is available on your registry

ibmcloud cr image-list

In this section you will create a IAM service-id and give it a reader and writer permission to operate on your cluster. You will then use the created API key when issuing the commands to convert existing docker images into SGX-compliant images. In fact, you will pass the API key to a kubernetes secret that you will use when converting images.

Important! It is essential to notice that you should create such kubernetes secret before installing the IBM data shield helm chart. This will refer in fact to the created kubernetes service.²

Create a IAM service to interact with your cluster

  ibmcloud iam service-id-create data-shield-container-converter -d 'IBM Cloud Data Shield Container Converter'

Create an Api-key for that service

  ibmcloud iam service-api-key-create 'IBM Cloud Data Shield Container Converter' data-shield-container-converter

Creating API key Data Shield Container Converter of service ID data-shield-container-converter under account <XXX> as Marco.Hassan@ibm.com...                                                                                                                             
OK
Service ID API key Data Shield Container Converter is created

Please preserve the API key! It cannot be retrieved after it's created.
                 
ID            ApiKey-1eaab4fb-3d2e-4937-b4e4-0684f1d4a91c   
Name          Data Shield Container Converter   
Description      
Created At    2020-09-24T12:46+0000   
API Key       <READ YOUR API KEY FROM HERE>   
Locked        false   

Notice that the Api key was removed from the above for security reasons. You will find it at the <READ YOUR API KEY FROM HERE> above. Make a note of it as you will need it shortly when creating the kubernetes secret.

Create a IAM policy specifying the policy for which you can interact to your cluster with the IAM service

  ibmcloud iam service-policy-create data-shield-container-converter --roles Reader,Writer --service-name container-registry

Create a Kubernetes secret to be used for future conversions. Notice that you will have to replace the <ENTER YOUR CONTAINER REGISTRY REGION HERE> and the <ENTER HERE YOUR API KEY FOR THE IAM SERVICE ABOVE> strings from the snippet below.

  (echo -n '{"auths":{"<ENTER YOUR CONTAINER REGISTRY REGION HERE>.icr.io":{"auth":"'; echo -n 'iamapikey:<ENTER HERE YOUR API KEY FOR THE IAM SERVICE ABOVE>' | openssl base64 -A;  echo '"}}}') | kubectl create secret generic converter-docker-config --from-file=.dockerconfigjson=/dev/stdin

Where the container registry endpoint is the region you selected for your image repository in the previous section. In my case hence de, such that the above becomes de.icr.io and the API key is the one you generated right above.

You can then verify the existence of your kubernetes secret by running

  kubectl get secret -ojson converter-docker-config

{                                                                                                                                               
    "apiVersion": "v1",                                                                                                                         
    "data": {
        ".dockerconfigjson": <base-64 authorization string passed above>
    },
    "kind": "Secret",
    "metadata": {
        "creationTimestamp": "2020-10-01T14:13:50Z",
        "name": "converter-docker-config",
        "namespace": "default",
        "resourceVersion": "3918214",
        "selfLink": "/api/v1/namespaces/default/secrets/converter-docker-config",
        "uid": "661fd885-e0c9-4212-93a7-849681cd504b"
    },
    "type": "Opaque"
}

Good, you are all set. You can now get the necessary information about your cluster and install the Data Shield Chart on your Cluster.

First of all get your IBM account HEX number

ibmcloud account show

Retrieving account IBM of Marco.Hassan@ibm.com...
OK
                                       
Account Name:                       IBM   
Account ID:                         <HERE YOU FIND THE HEX ACCOUNT NUMBER>
Account Owner:                      marco.hassan@ibm.com   
Account Type:                       PAYG   
Account Status:                     ACTIVE   
Linked Softlayer Account:           xxx   
VRF Enabled:                        false   
Service Endpoint Enabled:           false   
EU Supported                        false   
PoC (Commercial Proof of Concept)   false   
HIPAA Supported                     false   

Secondly get the Ingress Subdomain of your cluster, by running again the command

ibmcloud ks cluster get --cluster bti5pcef0j3046qt1ks0

......

Ingress Subdomain:              mycluster-fra02-mb2c-da-f9bf7c7800de9856ddeffe14aad922a5-0000.eu-de.containers.appdomain.cloud   

......

Make not of the ingress subdomain or enter it into a variable as you will use it extensively throughout the next sections.

  subdomain_ingress="mycluster-fra02-mb2c-da-f9bf7c7800de9856ddeffe14aad922a5-0000.eu-de.containers.appdomain.cloud"

  echo $subdomain_ingress

mycluster-fra02-mb2c-da-f9bf7c7800de9856ddeffe14aad922a5-0000.eu-de.containers.appdomain.cloud

Finally install the datashield chart in the default namespace of your cluster (recall to insert the HEX number of the account in the below snippet)

helm install myshield iks-charts/ibmcloud-data-shield --set enclaveos-chart.Manager.AdminEmail=Marco.Hassan@ibm.com --set enclaveos-chart.Manager.AdminName=Marco.Hassan@ibm.com --set enclaveos-chart.Manager.AdminIBMAccountId=<ENTER PREVIOUSLY OBTAINED HEX> --set global.IngressDomain=$subdomain_ingress --set converter-chart.Converter.DockerConfigSecret=converter-docker-config

Fantastic, if that worked you are good to go.

Check that all of the pods are running correctly

kubectl get pods

NAME                                                            READY   STATUS      RESTARTS   AGE
data-shield-setup-q4rng                                         0/1     Completed   0          26m
pmyshield-enclaveos-agent-mhn55                                  2/2     Running     2          26m
myshield-enclaveos-cockroachdb-0                                1/1     Running     0          26m
myshield-enclaveos-cockroachdb-init-wb8cm                       0/1     Completed   0          26m
myshield-enclaveos-cockroachdb-status-checker-86bdff5ff-7rz29   1/1     Running     0          26m
myshield-enclaveos-converter-76947448b7-cmvrn                   2/2     Running     0          26m
myshield-enclaveos-dkms-z4xll                                   1/1     Running     2          26m
myshield-enclaveos-frontend-868b5dbc5c-68ww6                    1/1     Running     0          26m
myshield-enclaveos-manager-0                                    2/2     Running     1          26m
myshield-sgx-6j87h                                              1/1     Running     1          26m
myshield-sgx-device-plugin-hp2rf                                1/1     Running     0          26m

Moreover you can get the link to your datashield UI by:

helm status myshield                                                                                                                            

NAME: myshield
LAST DEPLOYED: Fri Sep 18 14:42:52 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing ibmcloud-data-shield.

Your release is named myshield.

To learn more about the release, try:

  $ helm status myshield
  $ helm get myshield

You can access the Enclave Manager web console at https://enclave-manager.mycluster-fra02-mb2c-da-f9bf7c7800de9856ddeffe14aad922a5-0000.eu-de.containers.appdomain.cloud

Documentation may be found at https://console.bluemix.net/docs/services/data-shield/index.html

In order to convert the images via CLI you can pass the necessary parameters together with an authentication token to the converter API endpoint.

In order to do that you first generate the access token via:

export token=`ibmcloud iam oauth-tokens | awk -F"Bearer " '{print $NF}'`
echo $token

eyJraWQiOiIyMDIwMDkyMjE4MzMiLCJhbGciOiJSUzI1NiJ9.eyJpYW1faWQiOiJJQk1pZC01NTAwMDQ0SzdBIiwiaWQiOiJJQk1pZC01NTAwMDQ0SzdBIiwicmVhbG1pZCI6IklCTWlkIiwianRpIjoiOTgxZGViYmQtOGY1My00ZmUzLWJlYTMtMzA3ZmI5YTlhYTk3IiwiaWRlbnRpZmllciI6IjU1MDAwNDRLN0EiLCJnaXZlbl9uYW1lIjoiTWFyY28iLCJmYW1pbHlfbmFtZSI6Ikhhc3NhbiIsIm5hbWUiOiJNYXJjbyBIYXNzYW4iLCJlbWFpbCI6Ik1hcmNvLkhhc3NhbkBpYm0uY29tIiwic3ViIjoiTWFyY28uSGFzc2FuQGlibS5jb20iLCJhY2NvdW50Ijp7InZhbGlkIjp0cnVlLCJic3MiOiI3NWM0YzRlYmJhZmY0MGEzOTM1ZjA2OWJmY2VjNjVhOSIsImltc191c2VyX2lkIjoiNzg2ODk5NCIsImltcyI6IjE5OTU0MjIifSwiaWF0IjoxNjAyMTU2MTM4LCJleHAiOjE2MDIxNTk3MzgsImlzcyI6Imh0dHBzOi8vaWFtLmNsb3VkLmlibS5jb20vaWRlbnRpdHkiLCJncmFudF90eXBlIjoidXJuOmlibTpwYXJhbXM6b2F1dGg6Z3JhbnQtdHlwZTpwYXNzY29kZSIsInNjb3BlIjoiaWJtIG9wZW5pZCIsImNsaWVudF9pZCI6ImJ4IiwiYWNyIjoxLCJhbXIiOlsicHdkIl19.dpy_LEsdh7UnFPsyPdi_m0rNYrvj2p10w9xZdWHfZOIR_GaqQqN3O_s4TjqTXgxyTowPOjAZZZ_wLp-bsgUgT32MfFyva9PK3yH1mwmdvyCh_3eKqHqmZSWsMFMJC5s7wQpu2PLWXB6S9hTgoka-t0Cvk-eM2V-txcpsTxDiThWFtNSId0RpOsrrfeIdS3Wi9GYRzbNl9O9Wyt-gtE_HK4nkrgQzRk0nHTERE7B1SnILFGF1iWZyvZeDs_cj04e9wsPkSWgcEvII-vDvlqMeduVZGQsQDwKVcBT2It_XHmisag_G8-ddmFzxVPaPc8to3ErPpBT1QyZhAGOSfTO6yg

Then you pass it together with the image name to be converted, the image output name as well as the cluster ingress point + api route (notice that the snippet below uses the $subdomain_ingress variable previously defined):

curl -H 'Content-Type: application/json' -d '{"inputImageName": "de.icr.io/datashield_mahass/flask_dashboard:v1", "outputImageName": "de.icr.io/datashield_mahass/sgx_flask_dashboard:latest"}'  -H "Authorization: Basic $token"  https://enclave-manager.$subdomain_ingress/api/v1/tools/converter/convert-app

{"isvsvn": 0, "newImage": "de.icr.io/datashield_mahass/sgx_flask_dashboard:latest", "mrenclave": "5a4db6b8730cc48172edbf230ef9e30d49f2021c82cf94bd22824b792d56d850", "imageSize": 1144509494, "imageSHA": "6ba65e6d89fb", "isvprodid": 0, "mrsigner": "83a625398a0ff03a60c31635d3116021de498fe77ef25cb29cb0f0afc98dbd52"}

You can finally run the image on your cluster by running the deployment and services yaml files in the repository. Notice, though, that you will have to change the url of the image referenced in such files.

kubectl get pods -o wide                                                                                                                                             

NAME                                                            READY   STATUS      RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES
data-shield-setup-q4rng                                         0/1     Completed   0          20d     172.30.128.18   10.215.171.81   <none>           <none>
java-hello-world                                                0/1     Completed   0          14d     172.30.128.34   10.215.171.81   <none>           <none>
myflaskapp-deploy-84dffc55df-jdrbc                              1/1     Running     0          167m    172.30.128.49   10.215.171.81   <none>           <none>
myshield-enclaveos-agent-mhn55                                  2/2     Running     2          20d     10.215.171.81   10.215.171.81   <none>           <none>
myshield-enclaveos-cockroachdb-0                                1/1     Running     0          20d     172.30.128.24   10.215.171.81   <none>           <none>
myshield-enclaveos-cockroachdb-init-wb8cm                       0/1     Completed   0          20d     172.30.128.22   10.215.171.81   <none>           <none>
myshield-enclaveos-cockroachdb-status-checker-86bdff5ff-7rz29   1/1     Running     0          20d     172.30.128.21   10.215.171.81   <none>           <none>
myshield-enclaveos-converter-76947448b7-7b2vf                   2/2     Running     0          6d22h   172.30.128.41   10.215.171.81   <none>           <none>
myshield-enclaveos-dkms-z4xll                                   1/1     Running     2          20d     10.215.171.81   10.215.171.81   <none>           <none>
myshield-enclaveos-frontend-868b5dbc5c-68ww6                    1/1     Running     0          20d     172.30.128.19   10.215.171.81   <none>           <none>
myshield-enclaveos-manager-0                                    2/2     Running     1          20d     172.30.128.25   10.215.171.81   <none>           <none>
myshield-sgx-6j87h                                              1/1     Running     1          20d     10.215.171.81   10.215.171.81   <none>           <none>
myshield-sgx-device-plugin-hp2rf                                1/1     Running     0          20d     172.30.128.20   10.215.171.81   <none>           <none>
sgx-flask-dashboard-5c968f5c5b-krvbd                            1/1     Running     0          7s      172.30.128.55   10.215.171.81   <none>           <none>

kubectl get deployments                                                                                                                                              

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
myflaskapp-deploy                               1/1     1            1           167m
myshield-enclaveos-cockroachdb-status-checker   1/1     1            1           20d
myshield-enclaveos-converter                    1/1     1            1           20d
myshield-enclaveos-frontend                     1/1     1            1           20d
sgx-flask-dashboard                             1/1     1            1           12s