Mach5 Search v7.1.0 - Migration Guide

This guide provides the upgrade steps and compatibility notes for moving an existing Mach5 Search deployment from v6.2.0 to v7.1.0. Review the v7.1.0 release notes before starting the upgrade.

If you are upgrading from v6.1.0 or an earlier release, review the v6.2.0 migration guide and the v4.7.0-to-v6.1.0 migration guide for requirements that continue to apply.

Prerequisites

• Confirm that the current deployment is healthy before upgrading.
• Export the current Helm values and rendered manifests.
• Back up deployment configuration and customer data according to your environment’s recovery plan.
• Ensure ingest pipelines are paused, or confirm that the upgrade window tolerates ingestion interruption.
• Review direct service calls, OpenSearch integrations, dashboards, saved queries, and automation for the compatibility changes below.
• Review the system warehouse sizing and storage requirements before enabling the v7.1 defaults.

OpenSearch Compatibility

The legacy Java OpenSearch deployment (opensearch-legacy) is removed in v7.1. The /opensearch endpoint is served by MDX’s OpenSearch-compatible API.

Before upgrading, identify integrations that depend on Java OpenSearch plugins or implementation-specific behavior. Search, bulk, scrolling, aliases, aggregations, analyzers, highlighting, dashboards, and saved searches may behave differently where the native compatibility layer and Java OpenSearch previously differed.

The compatibility layer supports additional administrative, search, update, analysis, statistics, and template APIs. Unsupported operations return clearer client errors instead of generic server errors. Test the OpenSearch operations used by your applications and dashboards before completing the production upgrade.

System Warehouse and Diagnostics

The system warehouse and diagnostics receiver are enabled by default in v7.1. Helm bootstrap creates or reconciles a warehouse in the fixed _system namespace. This creates additional system-managed data, telemetry, diagnostics, and storage activity.

The relevant values are:

systemWarehouse:
  enabled: true
  name: default
  config:
    resource:
      num_mediators: 1
    enabled: true
    memory_policy: legacy
diagnosticsreceiver:
  enabled: true

systemWarehouse.config is sent to the Warehouse API as JSON. Extend it with the warehouse sizing, storage, placement, engine, or other settings required by the customer environment. The bootstrap creates or reconciles the warehouse in _system; it does not create it in the customer default namespace.

To disable Helm management of the system warehouse and diagnostics receiver, use:

systemWarehouse:
  enabled: false
diagnosticsreceiver:
  enabled: false

If the system warehouse is disabled, confirm that the environment does not depend on system diagnostics, query auditing, metadata auditing, or runtime diagnostic data being collected by this deployment.

API and Query Compatibility

The following v6.2 requirements continue to apply after upgrading to v7.1.

Direct service authorization

Direct calls to backend services must include the X-Mach5-Authorization header. Missing authorization no longer implies administrator access.

For example:

curl -sS -X POST \
  "http://<warehouse-opensearch-service-host>:<opensearch-port>/<index>/_search" \
  -H 'Content-Type: application/json' \
  -H 'X-Mach5-Authorization: ["admin"]' \
  -d '{"size":10,"query":{"match_all":{}}}'

Audit scripts, probes, port-forwarded clients, tests, and integrations that call msearchserver, mdserver, weaveserver, mcpserver, warehouse OpenSearch, or other backend services directly. Calls through the normal nginx/authentication path do not require this direct-service change.

Multi-field projection

Multi-fields remain supported; however, after upgrading, SELECT * and KQL project-all no longer include them automatically. Multi-fields must be projected explicitly.

For an index containing manufacturer.keyword:

SELECT manufacturer, manufacturer.keyword
FROM products
LIMIT 10;

products
| project manufacturer, manufacturer.keyword
| limit 10

Update dashboards, saved queries, tests, and scripts that depended on multi-fields being present in SELECT * or project-all results. This also applies to nested multi-fields such as product.category.keyword.

Warehouses, Autoscaling, and Placement

v7.1 adds autoscaling, shared warehouses, namespace-level warehouse attachment, and placement controls. Review the following before upgrading:

• Workload distribution across shared warehouse capacity.
• Resource and admission priorities.
• Worker placement and node-selection behavior.
• Autoscaling limits, latency expectations, and infrastructure cost.
• Existing warehouse state and namespace attachments.

Existing warehouses are adopted where possible during startup instead of being recreated. Existing resource state therefore affects upgrade behavior. Compare rendered Helm values and the Warehouse API state before and after the upgrade.

Indexing, Mappings, and Partitioning

Mapping limits are enforced more consistently, and partition-aware indexing and query execution are expanded. Review indexes that use:

• Large numbers of fields or very long field names.
• Strict mappings or nested fields.
• Multi-valued fields.
• Custom routing or partitioning.

Documents or mappings that were previously accepted may now be rejected when they exceed configured limits. Validate representative writes and queries after upgrading.

Data Assets and Package Lifecycle

Data-asset mapping and package installation workflows preserve more lifecycle state across retries, restarts, updates, failures, and recovery operations.

Review installation, update, uninstallation, rollback, mapping-change, refresh, and synchronization procedures. The improved lifecycle state is more visible in operational behavior and metadata usage; update runbooks and monitoring expectations accordingly.

Kafka and Protobuf Ingestion

Kafka Protobuf ingestion is already supported in v6.2.0 and remains supported in v7.1. This is not a new 7.1 capability; validate existing Protobuf pipelines during the upgrade. Protobuf pipelines require:

• A compiled Protobuf FileDescriptorSet, not only a raw .proto file.
• The fully qualified top-level message type.
• Kafka record values that match the descriptor and selected message type.
• Explicit unwrap targets when bytes fields contain nested serialized Protobuf messages.

Review existing Kafka pipelines and test descriptor, decoder, field-mapping, and pipeline behavior after the upgrade. Existing non-Protobuf Kafka formats remain supported.

Security and Secret Handling

Authorization is stricter and more comprehensive, particularly for MCP/agentic tools and administrative OpenSearch operations. Review permissions for indexes, warehouses, data assets, dashboards, workflows, notebooks, scripts, and connections.

Cloud credentials in dynamically created workloads are protected through secret references rather than direct secret exposure. Check workload definitions, environment descriptions, and operational tooling for assumptions that credentials are present as plain values.

Compaction, Ingestion, and Backpressure

Compaction and ingestion provide stronger admission control, auditing, and operational diagnostics. Under resource pressure, write requests may be delayed or rejected with a backpressure response.

Review clients and monitoring for retry behavior, delayed writes, rejected writes, and compaction state. This behavior is intended to reduce system instability during sustained ingestion or compaction pressure.

Local and Single-Node Deployments

The local and single-node deployment model is more closely aligned with the Kubernetes deployment model. Behavior can still differ between local and Kubernetes environments for profiling, storage backends, external services, and controller-managed operations.

The local runtime has a smaller default storage-backend footprint than the distributed deployment. Validate local storage, external service dependencies, and any deployment-specific assumptions separately.

Upgrade Procedure

Authenticate to the OCI registry and pull the v7.1.0 chart:

cat reader-key.json | helm registry login https://us-central1-docker.pkg.dev -u _json_key --password-stdin

helm pull oci://us-central1-docker.pkg.dev/mach5-dev/mach5-docker-registry/mach5-search \
  --version $version

Render the chart with the production values before applying it:

helm template <mach5-release-name> mach5-search-7.1.0.tgz \
  -n <mach5-namespace> \
  -f values.yaml > rendered-mach5-7.1.0.yaml

Review the rendered output for OpenSearch deployment changes, system warehouse and diagnostics settings, warehouse configuration, secrets, and service endpoints.

Upgrade the deployment:

helm upgrade --install <mach5-release-name> mach5-search-7.1.0.tgz \
  -n <mach5-namespace> \
  -f values.yaml \
  --wait

Post-upgrade Validation

After the upgrade:

1. Confirm all expected pods and services are ready.
2. Confirm /opensearch responds through the MDX compatibility layer.
3. Verify the system warehouse in the _system namespace when enabled.
4. Confirm diagnostics and system records are being collected as expected.
5. Run representative OpenSearch, SQL, KQL, aggregation, and multi-field queries.
6. Verify direct backend calls include X-Mach5-Authorization.
7. Run representative Kafka pipelines, including Protobuf pipelines where used.
8. Check warehouse autoscaling, placement, namespace attachment, and admission behavior.
9. Verify dashboards, saved searches, notebooks, workflows, data assets, and package lifecycle operations.
10. Confirm ingestion, compaction, backpressure, retries, and alerting behave as expected.

Use the v7.1.0 Quickstart and Helm Charts documentation for deployment-specific checks.