v6.1.0 Release Notes

Migration Guide

General Updates

  • Upgrading from v5.9.x or earlier: follow the v5.10.0 migration guide before upgrading to v6.1.0. See migration guide from v4.7.0 to v6.1.0.
  • Rename orphanedFileGracePeriodDurationseconds → orphanedFileGracePeriodDurationSeconds in any custom config before upgrading.
  • Upgrade to the latest mach5-sdk-ts to use the new gRPC query API.

API Updates

  • Update the ingest pipeline API payloads for S3, Iceberg, and Kafka as:

S3 pipeline

Move the scan-tuning fields inside the connector config:

Previous
{
  "source_config": {
    "config": {
      "type": "s3"
    }
  },
  "scan_filter_mpl": 1,
  "scan_filter_batch_size": 8192,
  "segment_bin_capacity_bytes": 104857600,
  "max_ingest_workflows_limit": 4,
  "segment_cache_size": 134217728,
  "append_batch_size": 104857600,
  "ignore_mapping_errors": false,
  "max_files_per_ingestor": 100,
  "workflow_timeout_seconds": 7200
}
6.1 snippet
{
  "source_config": {
    "config": {
      "type": "s3",
      "scan_mode": "enumerated",
      "scan_filter_mpl": 1,
      "scan_filter_batch_size": 8192,
      "segment_bin_capacity_bytes": 104857600
    }
  },
  "max_ingest_workflows_limit": 4,
  "segment_cache_size": 134217728,
  "append_batch_size": 104857600,
  "ignore_mapping_errors": false,
  "workflow_timeout_seconds": 7200
}

Iceberg pipeline

Drop the scan tuning keys and max_files_per_ingestor entirely.

Previous
{
  "source_config": {
    "config": {
      "type": "iceberg"
    }
  },
  "scan_filter_mpl": 1,
  "scan_filter_batch_size": 8192,
  "segment_bin_capacity_bytes": 104857600,
  "max_ingest_workflows_limit": 4,
  "segment_cache_size": 134217728,
  "append_batch_size": 104857600,
  "ignore_mapping_errors": false,
  "workflow_timeout_seconds": 7200,
  "max_files_per_ingestor": 100
}
6.1 snippet
{
  "source_config": {
    "config": {
      "type": "iceberg"
    }
  },
  "max_ingest_workflows_limit": 4,
  "segment_cache_size": 134217728,
  "append_batch_size": 104857600,
  "ignore_mapping_errors": false,
  "workflow_timeout_seconds": 7200
}

Kafka pipeline

Remove the scan filters, segment_bin_capacity_bytes, and max_files_per_ingestor

Previous
{
  "source_config": {
    "config": {
      "type": "kafka"
    }
  },
  "scan_filter_mpl": 1,
  "scan_filter_batch_size": 8192,
  "segment_bin_capacity_bytes": 104857600,
  "max_ingest_workflows_limit": 4,
  "segment_cache_size": 134217728,
  "append_batch_size": 104857600,
  "ignore_mapping_errors": false,
  "workflow_timeout_seconds": 7200,
  "max_files_per_ingestor": 100
}
6.1 snippet
{
  "source_config": {
    "config": {
      "type": "kafka"
    }
  },
  "max_ingest_workflows_limit": 4,
  "segment_cache_size": 134217728,
  "append_batch_size": 104857600,
  "ignore_mapping_errors": false,
  "workflow_timeout_seconds": 7200
}

What’s Changed

Query Language & MDX Enhancements

  • typeof() and to-typed() add runtime type inspection and explicit casting for finer control over value interpretation across heterogeneous sources.
  • Composite aggregations surface missing-value buckets via missing_bucket; Terms aggregations now produce correct results for boolean and IP fields.
  • mdx-cli now accepts a query as a command-line argument, simplifying scripting without intermediate files.

SQL Connectivity

  • Mach5 now offers PostgreSQL compatibility, so you can connect using any PostgreSQL-compatible client or BI tool.
  • Trino connectivity reaches production readiness with stability and correctness fixes for distributed query workloads.
  • SQL jobs now carry owner identity and track lifecycle state end-to-end, enabling resource governance in multi-tenant environments.

Garbage Collection

  • Orphaned file cleanup is more robust: several edge cases leaving files uncollected are fixed, and the deletion grace period is increased to 7 days.
  • A task leak in the fdb-reconciler causing unbounded memory growth on retry loops has been resolved.

Performance, Reliability & Infrastructure

  • Full-text indexing throughput is improved via batching and parallelism at the segment and term level.
  • Memory estimates now include projected columns, and both FSM and ART structures have a reduced baseline footprint.
  • Hash joins can spill to disk under memory pressure, preventing OOM failures on large join workloads.
  • Several correctness fixes land in this release: disk cache hot path and cstore snapshot consistency
  • Dremel null handling across all read modes, and three concurrency issues (PostOffice race, MutationGapError, job cancellation deadlock).
  • nginx worker_connections is now configurable via values.yaml for high-concurrency deployments.

External Integrations

  • OpenSearch and Elasticsearch can be connected as external data sources with query pushdown support.
  • Azure Blob Storage is now a supported ingest target with full Iceberg table support. GCS users can ingest Iceberg tables via native BigLake bucket integration.
  • BigQuery SQL pushdown is corrected for several patterns that previously fell back to client-side execution.

UI (Dex) & API changes

  • Cell outputs are updated in-place as data arrives rather than being fully recreated, reducing flicker.
  • Browser sessions refresh more reliably, reducing stale-session errors in long-running notebooks.
  • Editor fixes: Ctrl/Cmd+A selects the full active cell, connection resolution works in the ingest pipeline edit view, and dynamic value rendering is consistent.
  • Ingest pipeline source configs must be updated to the new per-type proto format when creating through API.
  • When creating a warehouse, you can choose how memory is managed during query execution. This helps balance performance, reliability, and cost based on your workload.
  • Route indices to different stores using namespace-based regex patterns.