Documentation

Mach5 Search Helmcharts

Installation instructions for all helmcharts required for setting up Mach5 Search.

Cluster Autoscaler For AWS EKS environment

Not needed for an Azure Kubernetes Service cluster deployment, and its native autoscaler can be used.

Run the following commands to install the cluster-autoscaler helm chart:

helm repo add autoscaler https://kubernetes.github.io/autoscaler

helm install m5-cas autoscaler/cluster-autoscaler \ --set 'autoDiscovery.clusterName'=<CLUSTER NAME> \
   -f values_ca.yaml

Contents of values_ca.yaml:

extraArgs:
    logtostderr: true
    stderrthreshold: info
    v: 4
    ignore-daemonsets-utilization: true
    skip-nodes-with-system-pods: false
    daemonset-eviction-for-empty-nodes: true
    daemonset-eviction-for-occupied-nodes: true
    skip-nodes-with-local-storage: false
    scale-down-delay-after-add: 2m
    scale-down-unneeded-time: 2m
    unremovable-node-recheck-timeout: 2m
    expander: priority
    max-node-provision-time: 2m

Mach5 Cache Proxy

Run the following commands to install the mach5 cache proxy helm chart:
(Note: reader-key.json file has the json key for the service account for accessing Mach5 Search Artifact Registry. Contact Mach5 Search Administrator to get a copy of your reader-key.json file)

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io --force-update
helm repo update
helm install cm jetstack/cert-manager --version v1.5.3 -n cert-manager -f values_cm.yaml

kubectl create namespace cache-proxy

cat reader-key.json | helm registry login https://us-central1-docker.pkg.dev -u _json_key
 --password-stdin
 helm pull oci://us-central1-docker.pkg.dev/mach5-dev/mach5-docker-registry/mach5-cache-proxy 
 --version 1.13.1
 helm install m5-cache mach5-cache-proxy-1.13.1.tgz -n cache-proxy -f values_cp.yaml

Contents of values_cm.yaml

prometheus:
  enabled: false
installCRDs: true

Contents of values_cp.yaml

mach5ImagePullSecret:
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value (image-pull-secret file)

kube-image-keeper:
  registry:
    env:
      - name: blobdescriptorsize
        value: 1000
    persistence:
      accessModes: ReadWriteOnce
      enabled: true
      storageClass: gp2
      size: 75Gi
    nodeSelector:
      mach5-main-role: "true"
  controllers:
    nodeSelector:
      mach5-main-role: "true"
  proxy:
    podAnnotations:
      "cluster-autoscaler.kubernetes.io/enable-ds-eviction": "true"
  fullnameOverride: "cache-proxy"

  • A license token is required to install Mach5. Please follow the instructions in this document: License Token Setup Guide and provide the requested details to the Mach5 Administrator to obtain your license.

  • Once you have the license token, create a mach5 namespace

kubectl create namespace mach5
  • Run the following commands to install the mach5-search helm chart:

(Note: reader-key.json file has the json key for the service account for accessing Mach5 Search Artifact Registry. Contact Mach5 Search Administrator to get a copy of your reader-key.json file and for the version of Mach5 to be used - $version)

  cat reader-key.json | helm registry login https://us-central1-docker.pkg.dev -u _json_key --password-stdin


  helm pull oci://us-central1-docker.pkg.dev/mach5-dev/mach5-docker-registry/mach5-search --version $version


  helm upgrade --install m5s $version -nmach5 -f values.yaml

The contents of the values.yaml file, tailored to the deployment environment, are provided in the section below.
Once the installation is complete, initialize Mach5 Search by following the instructions based on cloud providers:
AKS: Initialize Mach5 Search in AKS
EKS: Initialize Mach5 Search in EKS
GKE: Initialize Mach5 Search in GKE

AKS (Azure Kubernetes Service Cluster)

Contents of values.yaml: (Edit fields marked CHANGE_ME)

mach5ImagePullSecret:
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value (image-pull-secret file)

license:w
  metered: false
  token: "CHANGE_ME" # Contact Mach5 Search Administrator for this value 

metadatadb: #Azure Database for PostgreSQL flexible server details
  name: "postgres" #CHANGE_ME
  host: "postgreshost" #CHANGE_ME
  port: "5432" #CHANGE_ME
  sslmode: "disable" 
  user: "postgres" #CHANGE_ME
  password: "Password123" #CHANGE_ME
  externalPostgresdb: true
  pvc:
    storageclass: "managed"

pvc:
  storageclass: "managed"

mediator:
  replicaCount: 1
  existingClaim: false
  cstoreSegmentCache:
    enabled: true
  loglevel: info
  useGcpInstanceMetadata: false
  cstoreCachefs:
    rcachepvsize: "483183820800" #450GB
    pvsize: "485331304448" #452GB
  resourceLimit:
    enabled: false

cstorecacheserver:
  env:
    enableEviction: true

mediatorcompactorcontroller:
  cstoreCachefs:
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  compaction_resources:
    #enabled: true
    max_compaction_memory: 
      enabled: true
      value: "536870912" # 512M

mediatoringestorcontroller:
  cstoreCachefs:
    mountroot: /cachefs
    rcachepv: /ingest
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  ingestion_resources:
    limits:
      memory:
        enabled: true
        value: "4096Mi"

mediatorwarehousecontroller:
  customnode:
    enabled: true
    workerresource:
      mediator:
        memory: "59055800320" #55GB
    headresource:
      ir:
        memory: "1073741824" #1GB
      os:
        memory: "25769803776" #24GB
      osd:
        memory: "1073741824" #1GB

nginx:
  azure:
    enabled: true
    loadBalancerInternal: "true"

argo-workflows:
  # -- Keep CRDs on chart uninstall
  crds:
    install: true
    keep: false

prometheus:
  enabled: false

EKS

Contents of values.yaml: (Edit fields marked CHANGE_ME)

mach5ImagePullSecret:
  createSecretResources: true
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value
  
metadatadb:
  name: "postgres"
  host: "postgresdb"
  port: "5432"
  sslmode: "disable"
  user: "postgres"
  password: "" #CHANGE_ME
  externalPostgresdb: false
  pvc:
    storageclass: "gp2"

mediator:
  replicaCount: 1
  existingClaim: false
  cstoreSegmentCache:
    enabled: true
  loglevel: info
  useGcpInstanceMetadata: false
  cstoreCachefs:
    rcachepvsize: "483183820800" #450GB
    pvsize: "485331304448" #452GB
  resourceLimit:
    enabled: false

cstorecacheserver:
  env:
    enableEviction: true

pvc:
  storageclass: "gp2"

teleportcollector:
  useGcpInstanceMetadata: "false"

mediatorwarehousecontroller:
  customnode:
    enabled: true
    workerresource:
      mediator:
        memory: "59055800320" #55GB
    headresource:
      ir:
        memory: "1073741824" #1GB
      os:
        memory: "25769803776" #24GB
      osd:
        memory: "1073741824" #1GB

mediatorcompactorcontroller:
  cstoreCachefs:
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  compaction_resources:
    #enabled: true
    max_compaction_memory: 
      enabled: true
      value: "536870912" # 512M

mediatoringestorcontroller:
  cstoreCachefs:
    mountroot: /cachefs
    rcachepv: /ingest
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  ingestion_resources:
    limits:
      cpu:
        enabled: false
        value: "1000m"
      memory:
        enabled: true
        value: "4096Mi"
    requests:
      cpu:
        enabled: false
        value: "500m"
      memory:
        enabled: false
        value: "1024Mi"

nginx:
  loadBalancerInternal: "true"
  loadBalancerSourceRanges:
  - "0.0.0.0/0"
  awsElbHttps:
    enabled: false
    sslCertARN: arn:aws:iam::user:server-certificate/id # Add ACM's ARN

argo-workflows:
  # -- Keep CRDs on chart uninstall
  crds:
    install: true
    keep: false

prometheus:
  enabled: false
  server:
    persistentVolume:
      enabled: true
      size: 8Gi
    service:
      type: LoadBalancer
      loadBalancerSourceRanges: ["0.0.0.0/0"] # Change this to your CIDR
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

EKS Shared Cluster with Taint Configuration

  • Apply the following taint to your node to mark it as a Mach5-Search node:
    kubectl taint nodes \<nodename\> mach5\=true:NoSchedule

Contents of values.yaml: (Edit fields marked CHANGE_ME)

mach5ImagePullSecret:
  createSecretResources: true
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value
  
metadatadb:
  name: "postgres"
  host: "postgresdb"
  port: "5432"
  sslmode: "disable"
  user: "postgres"
  password: "" #CHANGE_ME
  externalPostgresdb: false
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  pvc:
    storageclass: "gp2"

mdserver:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

msearchserver:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

meteringserver:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

metricsserver:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

dataexplorer:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

mediator:
  replicaCount: 1
  existingClaim: false
  cstoreSegmentCache:
    enabled: true
  loglevel: info
  useGcpInstanceMetadata: false
  cstoreCachefs:
    rcachepvsize: "483183820800" #450GB
    pvsize: "485331304448" #452GB
  resourceLimit:
    enabled: false

cstorecacheserver:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-ccs-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  env:
    enableEviction: true

cstoretxnrelayserver:
  nodeassignment:
    enabled: true
    nodeSelector:
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

pvc:
  storageclass: "gp2"

teleportcollector:
  useGcpInstanceMetadata: "false"

mediatorwarehousecontroller:
  nodeassignment:
    enabled: true
    nodeSelector: 
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  headnodeselector:
    enabled: true
    nodeSelector:
      mach5-warehouse-head-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  workernodeselector:
    enabled: true
    nodeSelector:
      mach5-warehouse-worker-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  customnode:
    enabled: true
    workerresource:
      mediator:
        memory: "59055800320" #55GB
    headresource:
      ir:
        memory: "1073741824" #1GB
      os:
        memory: "25769803776" #24GB
      osd:
        memory: "1073741824" #1GB

mediatorcompactorcontroller:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  compactornodeselector:
    enabled: true
    nodeSelector:
      mach5-compactor-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  cstoreCachefs:
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  compaction_resources:
    #enabled: true
    max_compaction_memory: 
      enabled: true
      value: "536870912" # 512M

mediatormaterializedviewcontroller:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  ingestornodeselector:
    enabled: true
    nodeSelector:
      mach5-ingestor-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

mediatoringestorcontroller:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  ingestornodeselector:
    enabled: true
    nodeSelector:
      mach5-ingestor-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  cstoreCachefs:
    mountroot: /cachefs
    rcachepv: /ingest
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  ingestion_resources:
    limits:
      cpu:
        enabled: false
        value: "1000m"
      memory:
        enabled: true
        value: "4096Mi"
    requests:
      cpu:
        enabled: false
        value: "500m"
      memory:
        enabled: false
        value: "1024Mi"

nginx:
  nodeassignment:
    enabled: true
    nodeSelector:
      mach5-main-role: "true"
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:
  loadBalancerInternal: "true"
  loadBalancerSourceRanges:
  - "0.0.0.0/0"
  awsElbHttps:
    enabled: false
    sslCertARN: arn:aws:iam::user:server-certificate/id # Add ACM's ARN

argo-workflows:
  controller:
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  server:
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  # -- Keep CRDs on chart uninstall
  crds:
    install: true
    keep: false

otel:
  nodeassignment:
    enabled: true
    nodeSelector:
    tolerations:
    - key: "mach5"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    affinity:

prometheus:
  enabled: false
  server:
    persistentVolume:
      enabled: true
      size: 8Gi
    service:
      type: LoadBalancer
      loadBalancerSourceRanges: ["0.0.0.0/0"] # Change this to your CIDR
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

GKE

Contents of values.yaml: (Edit fields marked CHANGE_ME)

mach5ImagePullSecret:
  createSecretResources: true
  dockerconfigjson: "CHANGE_ME" # Contact Mach5 Search Administrator for this value

metadatadb:
  name: "postgres"
  host: "postgresdb"
  port: "5432"
  sslmode: "disable"
  user: "postgres"
  password: "" #CHANGE_ME
  externalPostgresdb: false
  pvc:
    storageclass: "standard"

mediator:
  replicaCount: 1
  existingClaim: false
  cstoreSegmentCache:
    enabled: true
  loglevel: info
  useGcpInstanceMetadata: false
  cstoreCachefs:
    rcachepvsize: "483183820800" #450GB
    pvsize: "485331304448" #452GB
  resourceLimit:
    enabled: false

cstorecacheserver:
  env:
    enableEviction: true

pvc:
  storageclass: "standard"

mediatorwarehousecontroller:
  customnode:
    enabled: true
    workerresource:
      mediator:
        memory: "59055800320" 
    headresource:
      ir:
        memory: "536870912" # in bytes
      os:
        memory: "25769803776" 
      osd:
        memory: "1073741824"

mediatorcompactorcontroller:
  cstoreCachefs:
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  compaction_resources:
    #enabled: true
    max_compaction_memory: 
      enabled: true
      value: "536870912" # 512M

mediatoringestorcontroller:
  cstoreCachefs:
    mountroot: /cachefs
    rcachepv: /ingest
    rcachepvsize: "10737418240" #10GB
    pvsize: "12884901888" #12GB
  ingestion_resources:
    limits:
      cpu:
        enabled: false
        value: "1000m"
      memory:
        enabled: true
        value: "4096Mi"
    requests:
      cpu:
        enabled: false
        value: "500m"
      memory:
        enabled: false
        value: "1024Mi"

nginx:
  loadBalancerInternal: "true"
  loadBalancerSourceRanges:
  - "0.0.0.0/0"

argo-workflows:
  # -- Keep CRDs on chart uninstall
  crds:
    install: true
    keep: false

resourcecontrol:
  enabled: false

prometheus:
  enabled: false