Skip to content

Latest commit

 

History

History
289 lines (230 loc) · 6.06 KB

README.md

File metadata and controls

289 lines (230 loc) · 6.06 KB

OpenFaas Pipeline

Event driven pipeline using OpenFaaS, Minio and Tensorflow inception

OpenFaaS Pipeline

Kubernetes

For Kubernetes, first install Helm and Tiller

Install Helm

Install Helm CLI Client

  • On Linux and Mac/Darwin
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
  • On Mac Via Homebrew
brew install kubernetes-helm

Install Tiller

  • Create RBAC permissions for tiller
kubectl -n kube-system create sa tiller \
  && kubectl create clusterrolebinding tiller \
  --clusterrole cluster-admin \
  --serviceaccount=kube-system:tiller
  • Install the server-side Tiller component on your cluster
helm init --skip-refresh --upgrade --service-account tiller

Install OpenFaaS on Kubernetes

  • Create OpenFaaS namespace
kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml
  • Add OpenFaaS helm repository
helm repo add openfaas https://openfaas.github.io/faas-netes/
  • Create basic-auth credentials
PASSWORD=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)
kubectl -n openfaas create secret generic basic-auth \
--from-literal=basic-auth-user=admin \
--from-literal=basic-auth-password="$PASSWORD"
  • Install OpenFaaS on kubernetes cluster
helm repo update \
 && helm upgrade openfaas --install openfaas/openfaas \
    --namespace openfaas  \
    --set basic_auth=true \
    --set functionNamespace=openfaas-fn \
    --set operator.create=true

Install Minio Client

  • MacOS
brew install minio/stable/mc
  • Linux
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc

Install and Configure minio

  • Create OpenFaaS namespaces
kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml
  • Generate secrets for Minio
SECRET_KEY=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)
ACCESS_KEY=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)
  • Create Secrets in kubernetes
kubectl create secret generic -n openfaas-fn \
 s3-secret-key --from-literal s3-secret-key="$SECRET_KEY"

kubectl create secret generic -n openfaas-fn \
 s3-access-key --from-literal s3-access-key="$ACCESS_KEY"
  • Install minio with helm chart
helm install --name cloud-minio \
    --namespace openfaas \
    --set accessKey=$ACCESS_KEY,secretKey=$SECRET_KEY,replicas=1,persistence.enabled=false,service.port=9000,service.type=NodePort stable/minio
  • Get Minio NodePort
MINIO_PORT=$(kubectl get svc/cloud-minio -n openfaas --output=jsonpath='{.spec.ports[?(@.name=="service")].nodePort}')
  • Configure minio client
mc config host add minio-kube http://127.0.0.1:$MINIO_PORT $ACCESS_KEY $SECRET_KEY
  • Get minio config and edit the JSON to add webhook handler
mc admin config get minio-kube > myconfig.json

edit webhook section of myconfig.json and save it

"webhook":{
    "1":{
        "enable":true,
        "endpoint":"http://gateway.openfaas:8080/function/minio-webhook-hanlder"
        }
    }
}
  • Update minio config and restart mino server Update the mini config and restart minio server
mc admin config set minio-kube < myconfig.json
mc admin service restart minio-kube
  • Create buckets
mc mb minio-kube/images
mc mb minio-kube/images-thumbnail
mc mb minio-kube/inception
  • Add event for the webhook
mc event add minio-kube/images arn:minio:sqs::1:webhook --event put --suffix .jpg
  • Change images bucket policy to public so that inception function can download the image without secret
mc policy public minio-kube/images

Deploy Functions

  • Login with faas-cli
faas-cli login -u admin -p $PASSWORD -g http://127.0.0.1:31112
  • Deploy functions
faas-cli deploy -f stack.kubernetes.yml

Docker swarm

Install OpenFaaS

git clone https://www.github.com/openfaas/faas && \
        cd faas && ./deploy_stack.sh

Install and Configure minio

  • Create minio secret and access key
SECRET_KEY=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)
ACCESS_KEY=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)

echo -n "$SECRET_KEY" | docker secret create s3-secret-key -
echo -n "$ACCESS_KEY" | docker secret create s3-access-key -
  • Deploy minio to cluster
docker service create --constraint="node.role==manager" \
 --name minio \
 --mount type=bind,source=~/minio/data,target=/data \
 --mount type=bind,source=~/minio/config,target=/root/.minio \
 --detach=true --network func_functions \
 --secret s3-access-key \
 --secret s3-secret-key \
 --env MINIO_SECRET_KEY_FILE=s3-secret-key \
 --env MINIO_ACCESS_KEY_FILE=s3-access-key \
minio/minio:latest server /data
  • Publish port 9000
docker service update minio --publish-add 9000:9000
  • Configure minio client
mc config host add minio http://127.0.0.1:9000 $ACCESS_KEY $SECRET_KEY
  • Get minio config and edit the JSON to add webhook handler
mc admin config get minio > myconfig.json

edit webhook section of myconfig.json and save it

"webhook":{
    "1":{
        "enable":true,
        "endpoint":"http://gateway:8080/function/minio-webhook-hanlder"
        }
    }
}
  • Update minio config and restart mino server Update the mini config and restart minio server
mc admin config set minio < myconfig.json
mc admin service restart minio
  • Create required buckets
mc mb minio/images
mc mb minio/images-thumbnail
mc mb minio/inception
  • Add event for the webhook
mc event add minio/images arn:minio:sqs::1:webhook --event put --suffix .jpg
  • Change images bucket policy to public so that inception function can download the image without secret
mc policy public minio/images

Deploy Functions

  • Login with faas-cli
faas-cli login -u admin -p $PASSWORD -g http://127.0.0.1:31112
  • Deploy functions
faas-cli deploy -f stack.swarm.yml