1. Overview

Cloudentity Secure Service Mesh is a platform to secure microservices. It goes beyond the plain channel security not only by augmenting the mesh with the application level service-to-service authentication but also with an extensive policy validation. In other words, it gives a real identity to services.

Thanks to handling of end-user to service authentication and session translation at the API gateway it provides each microservice in a service mesh with the verifiable user identity context. This combination of user and microservice identities lets you apply effective security controls at the end-service level as well as at each microservice level that is part of the mesh. In addition, you can consume the injected user context in your microservices keeping them separate from the end-user session type.

It all happens without requiring any changes in the microservices code and is irrelevant to the programming language the microservices are implemented in. It’s thanks to the Cloudentity Security Sidecar that gets automatically installed along with each microservice container. The Security Sidecar manages signing/verification of keys, policy synchronization and does authentication, authorization of all incoming requests as well as a transparent generation of security tokens for outgoing requests. The token signing & verification, policy enforcement and policy decisions are happening in a stateless fashion close to your microservice that introduces minimal latency.

The microservice mesh security plane described above co-exists and augments the microservice mesh data and control planes. It also it bridges the identity, API management and microservice mesh worlds. It gives you an option to build truly secure distributed microservices environment with proper controls where you need them and end-user identity context passed to your microservices.

1.1. Architecture

Cloudentity Secure Mesh consists of the following components:

  • Security Plane - authorization policies synchronization

  • Control Plane - proxy configuration for route traffic

  • Data Plane - control of communication between microservices (only with Istio)

  • API Gateway - end-user to service authentication and user context injection

  • Cloudentity IDaaS - microservices, OAuth applications and users management platform

The diagram below shows how the components are related:

architecture

This is simplified diagram is sufficient at this stage. Post going through the instruction you should have a look at the more comprehensive one in one of the subsequent chapters.

1.1.1. Cloudentity IDaaS

Multi-tenant identity, authorization, authentication, oauth, application management platform with delegate administration features.

Available as a service or installable in the cloud or on-prem to give you and your teams control over Security Plane in all microservice environments.

It provides comprehensive set of features a modern consumer identity management platform should have.

Makes possible a transparent, gradual migration to the microservice architectural style.

1.1.2. Security Plane

Certificate Authority (CA)

Vault is used as a CA to read and generate dynamic X.509 certificates.

Sync

Cloudentity Mesh Config Sync is used to register microservices in Cloudentity IDaaS and synchronize policies and APIs configuration.

Initializer

Cloudentity Kubernetes Initializer automatically injects security sidecar and proxy to a pod and generates a new X.509 certificate upon deployment.

1.1.3. Control Plane

Control plane consists of Istio Pilot and Istio Mixer. It is responsible for managing and configuring proxies to route traffic.

1.1.4. Data Plane

Proxy

Envoy or Linkerd is used as an intelligent HTTP proxy which provides dynamic service discovery, load balancing, circuit breakers, traffic routing, metrics and more. Both Envoy and Linkerd are extended with a plugin which calls security sidecar to validate and transform requests.

Security Sidecar

Security Sidecar allows in place validation of policies (enforcement and decisions) for incoming requests and transformation of outgoing requests.

In default scenario, it validates if incoming requests were signed using a key generated by internal CA and injects JWT signed using a private key of a microservice. The user context is propagated between microservices as a JWT claim.

1.1.5. API Gateway

API Gateway is a entry point to the service mesh.

It exposes public endpoints using OAuth token for authentication and injects JWT to proxied requests signed using a private key issued exclusively for the API Gateway by internal CA. The JWT contains User Context as a JWT claim.

API Gateway can enforce rich policies that can be created in Cloudentity IDaaS involving user attributes, roles, relationships, session attributes, a location, device information, authentication methods used and risk factor of a transaction user or a device.

1.2. Integrations

Cloudentity Secure Service Mesh integrates with major service mesh proxies and microservices orchestration platforms.

Traffic management
  • Istio + Envoy

  • Linkerd

Deployment
  • Kubernetes

  • Docker

2. Getting Started

2.1. Registering a trial account

First, you’ll need to register a trial account so you can install Cloudentity Mesh.

Go to https://cloudentity.com/demo/ and follow the instructions.

2.2. Downloading config files

On the dashboard site you’ll find a command which downloads personalized configuration files describing your local environment.

2.3. Running on Kubernetes

Once you have the config files you can proceed to run Cloudentity Mesh on Kubernetes cluster.

2.3.1. Prerequisites

You’ll need kubectl command line interface and a Kubernetes cluster.

  1. Install kubectl v1.9.x

  2. Install minikube v.0.25.x (You need to have Virtualbox installed).

    # on osx
    curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.25.2/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
    # on linux
    curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.25.2/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
  3. Delete a VM if it already exists.

    minikube delete
  4. Start minikube

    minikube start --kubernetes-version v1.9.0 --memory=3072 --extra-config=apiserver.Admission.PluginNames="Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota"
    Note

    The Kubernetes cluster needs to have alpha features enabled to support automatic injection of sidecars.

2.3.2. Installation

After setting up the Kubernetes cluster you are ready to install Cloudentity Mesh.

  1. Enter the directory with config files (obtained in section Downloading config files).

    cd <YOUR_CONFIG_FILES_DIRECTORY>/istio
  2. Run watch -n 3 ./status.sh for simple Kubernetes cluster monitoring.

    Note

    Install watch if necessary.

    # on osx
    brew install watch
    
    # on ubuntu
    apt-get install watch
  3. Deploy Cloudentity Mesh

    kubectl apply -f mesh
  4. Wait until Cloudentity Mesh is deployed

    ./wait.sh
    Note

    The command above checks the status of deployments and finishes when all pods are ready.

    It might take several minutes to download all the containers.

    Be patient.

  5. Deploy Security Initializer

    kubectl apply -f initializer
  6. Wait until Security Initializer is deployed

    ./wait.sh

From here you can go to:

Deployment of sample microservices (HTTP/1.1)

If you want to use microservices prepared by us.

Or

Building and securing microservices (HTTP/1.1)

If you prefer to build a simple microservice from scratch.

2.3.3. Deployment of sample microservices (HTTP/1.1)

  1. Deploy the sample microservices using

    kubectl apply -f services
  2. Make sure to wait until all services are ready.

    ./wait.sh
    Note

    Downloading services may take up to several minutes…​ Go on and use this time to explore other tabs that are in the Cloudentity IDaaS. On the side-panel on the left, you can see Users, Applications and Microservices tabs.

    Have a look at Microservices. It’s empty since we haven’t discovered any microservices yet. It’ll fill up when the services are up and ready in your cluster. You’ll be able to see all policies and APIs pertaining to these microservices.

    The Application tab contains all client applications, resource servers and API sets protected by the gateway. When your services start, you’ll see there the client application that you’ll use and microservice APIs exposed through the API gateway. We’ll set it up for you when your environment is ready.

    In the Users tab, you can see and manage all users in this organization that will be able to use these services apart from you.

  3. Expose financial-frontend on localhost

    kubectl port-forward $(kubectl get pod -l app=financial-frontend -o jsonpath='{.items[0].metadata.name}') 4000:80
  4. Enter http://localhost:4000 and click Login with Cloudentity button to authenticate.

  5. Click Active or Deactive buttons to manage credit cards.

Great job! You have set up two sample microservices and you have secured them with the Cloudentity Secure Mesh.

Note
Now go to the Microservices tab on the admin panel. You can see the services that were discovered in your environment. Have a look at their APIs and policy configuration. Also have a look at Applications tab to see what applications we have preconfigured for you.

From here you can go to:

Applications, microservices and policy management

To learn more about the applications, microservices and policies management

Or

Understanding what happened

To find out what Cloudentity Secure Mesh does when you active or deactivate cards.

2.3.4. Building and securing microservices (HTTP/1.1)

  1. Create your Node.js application

    server.js
    var http = require('http');
    
    var handleRequest = function(request, response) {
      console.log('Received request for URL: ' + request.url);
      response.writeHead(200);
      response.end('Hello World!');
    };
    
    var www = http.createServer(handleRequest);
    www.listen(8080);

    If you run the app node server.js and call curl localhost:8080 you should get Hello Wordl!.

  2. Create a Dockerfile and build Docker container

    Dockerfile
    FROM node:6.9.2
    EXPOSE 8080
    COPY server.js .
    CMD node server.js
    eval $(minikube docker-env)
    docker build -t hello-node:v1 .
  3. Create a Deployment and Service description.

    hello-node.yaml
    apiVersion: apps/v1beta1
    kind: Deployment
    metadata:
      annotations:
        initializer.cloudentity.com/security: "true"
      name: hello-node-deployment
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: hello-node
        spec:
          containers:
          - name: hello-node
            image: hello-node:v1
            ports:
            - containerPort: 8080
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: hello-node
      annotations:
        initializer.cloudentity.com/security: "true"
    spec:
      ports:
      - name: http
        port: 8080
        targetPort: 8080
        protocol: TCP
      selector:
        app: hello-node
      type: LoadBalancer
  4. Deploy hello-node microservice.

    kubectl apply -f hello-node.yaml

    Notice that you get 401 Unauthorized if you call the service now directly.

    $ curl -v $(minikube service hello-node --url)
    * Rebuilt URL to: http://192.168.99.100:32342/
    *   Trying 192.168.99.100...
    * TCP_NODELAY set
    * Connected to 192.168.99.100 (192.168.99.100) port 32342 (#0)
    > GET / HTTP/1.1
    > Host: 192.168.99.100:32342
    > User-Agent: curl/7.55.1
    > Accept: */*
    >
    < HTTP/1.1 401 Unauthorized
    < Content-Length: 0
    <
    * Connection #0 to host 192.168.99.100 left intact
  5. Deploy edge-api-gateway.

    kubectl apply -f services/edge-api-gateway.yaml
  6. Wait until all services are ready.

    ./wait.sh
  7. Get an OAuth Access Token.

    1. Expose OAuth Sandbox locally

      kubectl port-forward -n cloudentity-system $(kubectl get pod -l app=oauth-sandbox -n cloudentity-system -o jsonpath='{.items[0].metadata.name}') 5000:80
    2. Enter http://localhost:5000 and click Login with Cloudentity button to authenticate.

    3. After authentication copy Access Token.

  8. Call hello-node app using edge-api-gateway as a proxy. Replace <ACCESS_TOKEN> with your token.

    $ export http_proxy=$(minikube service edge-api-gateway --url)
    $ curl -H "Authorization: Bearer <ACCESS_TOKEN>" http://hello-node:8080
    Hello Wordl!

2.3.5. Uninstalling

To uninstall the Cloudentity Service Mesh run the following commands.

kubectl delete -f services
kubectl delete -f mesh
kubectl delete secret vaulttokens

2.4. Understanding what happened

Before we get to the actual flow initiated with each button click. Let’s see what you’ve deployed so far.

cloudentity secure servcie mesh sample architecture

The diagram above shows the client application that you had a chance to see in your browser and two sample microservices deployed in the mesh that you have seen in the output of the status command.

Although there are only two microservices on the diagram, imagine that this is a just small piece of all microservices in this mesh and that the Financial Service is exposing many types of APIs and it is connecting to many other microservices, one of which is the Credit Card service responsible for entire credit card related flows.

On the other hand the financial service APIs are exposed through the API gateway, however the API gateway may not be the only consumer of these APIs in the mesh.

2.4.1. The end-user-to-service flow

Each request that is triggered by a button click in the client application contains a payload and the access_token in the header. The API gateway authenticates the call and performs authorization not only checking the scopes but also invoking all other policies associated with this API. Upon successful authentication and authorization the request is proxied to the appropriate microservice in the mesh with added JWT token with enriched user claims. Since the API gateway is a microservice itself it’s aware of other microservices in the network and it signs the JWT with its individual private key generated with Vault so that other microservices can verify it and identify this microservice.

2.4.2. The service-to-service flow

Each microservice is secured by Cloudentity Secure Mesh Sidecar. Each incoming and outgoing request goes through proxy before it reaches target microservice or goes outside of a microservice. Proxy uses Cloudentity Secure Mesh Plugin to validate each incoming request and to transform each outgoing request. In case of Istio the Envoy Proxy is being automatically configured by Istio Pilot.

Incoming request
  1. Incoming request hits Proxy.

  2. Proxy calls Security Sidecar to validate a request.

  3. Security Sidecar looks for a policy rule that matches given request.

  4. Security Sidecar validates policies for given request.

    Note

    By default, it checks if incoming JWT is valid and signed using the key that was issued in Certificate Authority (Vault).

  5. Proxy evaluates validation result.

    1. When authorized, Proxy forwards the request to the target microservice.

    2. When unauthorized, Proxy returns 401 HTTP status.

Outgoing request
  1. Outgoing request is goes through Proxy .

  2. Proxy calls Security Sidecar to transform request.

  3. Security Sidecar executes a chain of Http Request Transformers.

    Note

    By default, it injects JWT signed using the private key that is assigned to the target microservice as an HTTP header.

  4. Proxy modifies outgoing request.

2.4.3. Distributed tracing

You can easily track the flow using Jaeger which is built into Cloudentity Mesh.

Execute the following command to get the URL of Jaeger UI and enter it.

minikube service jaeger-query -n cloudentity-system --url

Select financial-service as a Service, put span.kind=server as a Tags filter, click Find Traces and click on a trace to see the details.

jaeger
An example trace of unauthorized request
Note

In this example the request was rejected on card-service because the user does not have list_cards scope.

2.5. Applications, microservices and policy management

Although you’re playing with the local Kubernetes cluster, the application, microservice, policy and API management is happening in the Cloudentity IDaaS demo. Thanks to its multi-tenancy and delegated administration it controls multiple meshes and only users with certain permissions are able to have a control over it. If you have a look at "Microservices" and "Applications" tabs in the panel on the left side, you’ll see now that Cloudentity IDaaS discovered your microservices. For the sake of this demo, we have also preconfigured applications and policies for your applications and microservices.

Before you go to the screenshots it’s important for you to learn about the applications and their capabilities. In Cloudentity applications may have multiple capabilities. The diagram below will let you understand this convention.

application capabilities

We have preconfigured the Financial Fronted and the Financial Portal Resource Server Applications.

applications list

When you start exploring them you’ll see that the Financial Frontend is the application with the OAuth2 Client capability that consumes resources of the Financial Portal Resource Server that is an OAuth2 resource server exposing certain resources that you’ve been asked for access to while you were accessing the application initially.

financial portal resources

Since it’s a gateway protected application at the same time the gateway also exposes APIs that you can manage. We’ve preconfigured these APIs for you, but typically it’s happening through a swagger import. Since the APIs are exposed by one of the microservices you need to select the target microservice to which requests will be proxied. Thanks to that API gateway being a microservice itself and aware of other microservices you just need to select the target microservice from the list of all microservices using the selection field above the APIs.

Each API also has the option to associate a policy with. When the first API is invoked, the policy selected for it is verified. The policy can be managed in the policies tab on this resource server.

financial portal apis

We have also detected the two microservices that have appeared in your local service discovery so that you can manage them.

microservices

Same like for applications you can see their APIs. These are the ones of the financial service APIs along with their policy assignment.

financial service apis

These are the card service APIs with and their policy assignment.

cc service apis

And this is the configuration of a sample policy at credit card service. The policy verifies whether the request is coming from the financial service and whether application that made the call to the API gateway on user’s behalf has been given by this user access to the resources i.e. a certain scope and by that whether a user had been earlier assigned the permission to be able to grant it to the application. The entire policy verification is happening within milliseconds in the sidecar that intercepts the calls.

cc service policy

2.6. Next steps

The diagram below shows the more detailed demo architecture that will give you more answers on the relation between Cloud IDaaS and Cloudentity Secure Mesh components. All blue components are Cloudentity stack components. The diagram should be self-explanatory, but if you’re interested in more details contact us http://www.cloudentity.com/talk-to-us/

cloudentity secure microservice mesh

3. Installation

3.1. AWS

TODO

3.2. Google Cloud

TODO

4. Features

4.1. Proxy

TODO

4.2. Security

TODO

4.3. Policies

TODO

5. Reference

5.1. API

5.1.1. Sidecar

gRPC
Method Name Request Type Response Type Description

validate

ValidateRequest

ValidateResponse

Validate policies for incoming HTTP request

transform

HttpRequest

HttpRequest

Used to transform outgoing HTTP request (for instance inject JWT)

ValidateRequest
Field Type Label Description

url

string

required

HTTP url

method

Method

required

HTTP Method

headers

Header

repeated

List of HTTP headers

body

bytes

optional

HTTP body

ValidateResponse
Field Type Label Description

result

ValidateResponse.Result

required

The result of policies validation

external

bool

required

Determines if policies were validated offline or online

failures

ValidateResponse.ValidatorFailure

repeated

List of failures if ValidateResponse.Result is not AUTHORIZED

HttpRequest
Field Type Label Description

url

string

required

HTTP request

method

Method

required

HTTP method

headers

Header

repeated

List of HTTP headers

ValidateResponse.ValidatorFailure
Field Type Label Description

validator

string

required

Name of failed validator

message

string

required

Error message

details

string

optional

Details of the error

Header
Field Type Label Description

name

string

required

HTTP header name

value

string

required

HTTP header value

Method
Name Number

UNKNOWN

0

GET

1

HEAD

2

POST

3

PUT

4

DELETE

5

CONNECT

6

OPTIONS

7

TRACE

8

PATCH

9

ValidateResponse.Result
Name Number Description

UNKNOWN

0

Unknown result

AUTHORIZED

1

Successful validation for all policies

NOT_AUTHORIZED

2

Failure during policy validation

NO_RULE

3

Could not find matching rule for given HTTP method and url

NO_POLICY

4

Could not find policy for a matched rule

NO_VALIDATOR

5

Could not find validator for a policy

ERROR

6

Internal error

5.2. Configuration

5.2.1. Sidecar

core:
  port: 3000
  httpPort: 3001
  workQueue: 100
  workers: 10

logging:
  level: info
  colors: true

instances:
  jwtService: symmetric
  publicKeyReader: vault

pipeline: [JwtInjector]

prePolicies: [JWT]

rules:
  - method: GET
    path: .*
    policyName: JWT

policies:
  - policyName: JWT
    validators:
      - name: jwt
        conf: {}

services:
  symmetricJwt:
    secret: secret
    root: content
    algorithm: HS256
    expiresIn: 1h
  asymmetricJwt:
    algorithm: RS256
    root: content
    expiresIn: 1h
    keystore:
      path: "./keystore.p12"
      password: "password"
    params:
      serviceId: "sample-service"
      serialNumber: "abcdef"
      instanceId: "1"
  authz:
    host: http://localhost
    port: 4000
    basePath: /authz
    timeout: 10
  vault:
    host: http://localhost
    port: 8200
    ssl: false
    timeout: 10
  publicKeyCache:
    expirationInterval: "15m"
    cleanupInterval: "20m"

tracing:
  sampler:
    type: "const"
    param: 1
  reporter:
    logSpans: true
    bufferFlushInterval: "1s"
Core
Field Type Description

port

int

gRPC port

workQueue

int

Size of external validation work queue

workers

int

Number of external validation workers

name

string

name used for tracing and metrics

Logging
Field Type Description

level

string

Logging level (debug, info, warn, error, fatal or panic)

colors

boolean

Logging with or without colors

Instances
Field Type Description

jwtService

string

Implementation of JwtService (symmetric or asymmetric)

publicKeyReader

string

Implementation of PublicKeyReader (vault or keystore)

Root
Field Type Description

pipeline

array of strings

List of HTTP request transformers (JwtInjector)

prePolicies

array of strings

List of policies that should be validated for all requests regardless of rule configuration

rules

array of Rules

List of rules

policies

array of Policies

List of policies

Rule
Field Type Description

method

string

HTTP method (GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE or PATCH)

path

string

URL path regex (for example: /books/.+)

policyName

string

Optional name of policy to validate

Policy
Field Type Description

policyName

string

Name of policy

validators

array of Validators

List of validators (true, false, jwt, embedded, session, user, jwt-content, jwt-claims or conditional)

Validator
Field Type Description

name

string

Name of a validator

conf

object

Validator configuration

Services.SymmetricJwt
Field Type Description

secret

string

Key for signing JWT

root

string

Name of json object with content

algorithm

string

Symmetric algorithm (HS256, HS384 or HS512)

expiresIn

string

A Duration

Services.AsymmetricJwt
Field Type Description

algorithm

string

Asymmetric algorithm (RS256, RS384 or RS512)

keystore

AsymmetricJwtKeystore

params

AsymmetricJwtParams

root

string

Name of json object with content

expiresIn

string

A Duration

Services.AsymmetricJwt.Keystore
Field Type Description

path

string

Path to the pkcs12 keystore

password

string

Password for pkcs12 keystore

Services.AsymmetricJwt.Params
Field Type Description

serviceId

string

iss

serialNumber

string

kid

instanceId

string

iid

5.2.2. LinkerD Plugin

TODO specification of Cloudentity Mesh LinkerD plugin.

5.2.3. Kubernetes Initializer

Core
Argument Default Description

--annotation

initializer.cloudentity.com/security

The annotation to trigger initialization

--skip-sidecars-annotation

initializer.cloudentity.com/skip-sidecars

The annotation to skip sidecars injection

--initializer-name

security.initializer.cloudentity.com

The name of initializer

--configmap

security-initializer

The initializer configuration configmap

--namespace

default

The configuration namespace

--label

app

The label with service name

--domain

cloudentity.com

The domain for services in CA

--sidecar-name

s5d

The name of security sidecar container

--sidecar-configurator-name

s5d-configurator

The name of security sidecar configurator container

--keystore-factory-name

keystore-factory

The name of keystore factory container

Service Registration (consul)
Argument Default Description

--consul-registration

false

Enables service registration in consul

--consul-tags

cloudentity_microservice

The tags for consul service registration

Certificate Authority (vault)
Argument Default Description

--vault-registration

false

Enables certificate authority management in vault

--vault-token-env-name

VAULT_TOKEN

The name of environment variable with vault token

--vault-role

cloudentity

The name of the role to generate certificates

5.2.4. Vault Initializer

TODO specification of Cloudentity Mesh Vault initializer.

5.2.5. Vault unsealer

TODO specification of Cloudentity Mesh Vault unsealer.