Materialized Views in Mach5

This document provides a step-by-step guide to creating a materialized view on an index in Mach5.

A materialized view (MV) is essentially a precomputed, stored result of a query. Instead of recalculating heavy transformations or aggregations every time, a materialized view provides a ready-to-use snapshot that is always kept up to date as the underlying index changes.

This makes queries much faster, because the work of computation has already been done. In short, a materialized view combines the convenience of a regular view with the performance benefits of pre-stored results.

Prerequisites

  • This document assumes that Mach5 is deployed and running successfully. Lets assume it’s running at http://localhost:8888/

  • Store, store route and warehouse are created successfully. Refer to Quickstart document for help

Source Index Details

Let’s create Mach5 index vpclogs with following mappings and data:

Source Index Mapping

PUT vpclogs
{
    "mappings": {
      "properties": {
        "version": {
          "type": "integer"
        },
        "account-id": {
          "type": "keyword"
        },
        "interface-id": {
          "type": "keyword"
        },
        "srcaddr": {
          "type": "ip"
        },
        "dstaddr": {
          "type": "ip"
        },
        "srcport": {
          "type": "integer"
        },
        "dstport": {
          "type": "integer"
        },
        "start": {
          "type": "date",
          "format": "epoch_second"
        },
        "action": {
          "type": "keyword"
        }
      }
    }
}

Source Index Data

Consider following data of 10 records for vpclogs

POST /vpclogs/_bulk
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789012", "interface-id": "eni-abc123", "srcaddr": "10.181.10.41", "dstaddr": "10.181.10.111", "srcport": 443,  "dstport": 1024, "action": "ACCEPT", "start": 1753746309 }
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789012", "interface-id": "eni-abc123", "srcaddr": "10.181.10.42", "dstaddr": "10.181.10.112", "srcport": 80,   "dstport": 2048, "action": "DENY",   "start": 1753746310 }
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789012", "interface-id": "eni-abc124", "srcaddr": "10.181.10.43", "dstaddr": "10.181.10.113", "srcport": 22,   "dstport": 3072, "action": "ACCEPT", "start": 1753746311 }
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789013", "interface-id": "eni-abc125", "srcaddr": "10.181.10.44", "dstaddr": "10.181.10.114", "srcport": 8080,"dstport": 4096, "action": "ACCEPT", "start": 1753746312 }
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789013", "interface-id": "eni-abc126", "srcaddr": "10.181.10.45", "dstaddr": "10.181.10.115", "srcport": 3306,"dstport": 5120, "action": "DENY",   "start": 1753746313 }
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789014", "interface-id": "eni-abc127", "srcaddr": "10.181.10.46", "dstaddr": "10.181.10.116", "srcport": 53,  "dstport": 6144, "action": "ACCEPT", "start": 1753746314 }
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789014", "interface-id": "eni-abc128", "srcaddr": "10.181.10.47", "dstaddr": "10.181.10.117", "srcport": 25,  "dstport": 7168, "action": "ACCEPT", "start": 1753746315 }
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789015", "interface-id": "eni-abc129", "srcaddr": "10.181.10.48", "dstaddr": "10.181.10.118", "srcport": 110, "dstport": 8192, "action": "DENY",   "start": 1753746316 }
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789015", "interface-id": "eni-abc130", "srcaddr": "10.181.10.49", "dstaddr": "10.181.10.119", "srcport": 143, "dstport": 9216, "action": "ACCEPT", "start": 1753746317 }
{ "index": { "_index": "vpclogs" } }
{ "version": 2, "account-id": "123456789016", "interface-id": "eni-abc131", "srcaddr": "10.181.10.50", "dstaddr": "10.181.10.120", "srcport": 443, "dstport": 10240,"action": "DENY",   "start": 1753746318 }

Run a search query to view the records in the vpclogs index in Mach5 Devtools

GET vpclogs/_search
{
  "size": 20,
  "_source": ["start", "srcaddr", "dstaddr", "action"], 
  "sort": [
    {
      "srcaddr": {
        "order": "desc"
      }
    },
    {
      "dstaddr": {
        "order": "desc"
      }
    },
    {
      "action": {
        "order": "desc"
      }
    }
  ]
}

pvclogsSearch

Materialized Views

Prerequisites

  • Scale down the materialized view controller deployment: This can be done by running the following command
kubectl scale deployment m5s-mediatormaterializedviewcontroller --replicas=0 -n mach5
  • Retrieve the index ID of the source index for the materialized view: Run the following from the Dev Tools page
GET _cat/indices

Sample output:
green open vpclogs  K7_wqoiKSBeNSdOkStTZEg 1 1

  • In this case, K7_wqoiKSBeNSdOkStTZEg would be the source index id

  • Run the KQL query given below in the Mach5 Search UI -> Notebooks

 vpclogs | extend event_time = startofday(start, 0) | summarize total = 
 count() by event_time, srcaddr=strcat(srcaddr), dstaddr=strcat(dstaddr), action

This aggregation query counts how many log entries occurred per day (event_time), grouped by each combination of source IP, destination IP, and action type.

NotebookOutput

When there are millions of raw log records, one would not to scan or visualize each one. Instead, it would be a good idea to start by summarizing at a higher level, then progressively zoom in (drill down) to narrower scopes. A materialized view enables exactly this, providing pre-aggregated insights that make deeper exploration faster and more efficient. Let’s see how to create this materialized view from the Mach5 UI.

Create the Materialized View

  • From the Mach5 Administrative UI, navigate to the Materialized Views page
  • Click on + icon on the top right corner to create a new materialized view MaterializedView
  • Fill in the following details:
    • Name: Name of the materialized view. E.g., vpcmv
    • Source Index: ID of the source index for the materialized view, as shown in the prerequisites section above.
    • View Function: Kql: KQL query that would be used to populate the data into the materialized view. Following is the sample KQL query for the vpclogs Mach5 index:
     ["$segment"] | extend event_time = startofday(start, 0) | summarize total = 
 count() by event_time, srcaddr=strcat(srcaddr), dstaddr=strcat(dstaddr), action
  • Click on Save.

MaterializedViewDetails

Enable the Materialized View

  • Update the mapping for the materialized view from the Dev Tools page by running the following:
PUT vpcmv/_mapping
   {
    "properties" : {
          "total" : {
            "type" : "long"
          },
          "dstaddr" : {
            "type" : "ip"
          },
          "srcaddr" : {
            "type" : "ip"
          },
          "action" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword"
              }
            }
          }
        }
    }
  • Enable the materialized view processing by running the following command:
 kubectl scale deployment m5s-mediatormaterializedviewcontroller --replicas=1 -n mach5

Verify the data in the materialized view

Run the following command from the Dev Tools page to verify the contents of the materialized view:

  GET vpcmv/_search

vpcmvSearch

In short, this guide showed how to set up a materialized view in Mach5 Search step by step. Starting from the source index, we created the view, updated its mapping, enabled it, and finally checked the results. With this setup, queries run much faster since the data is already precomputed and kept up to date.