How to Validate & Benchmark Kubernetes Storage Options Using Kubestr
Running stateful workloads in the k8s clusters can be a balancing act. The dependability of your underlying storage determines the dependability of your applications, whether they be enormous AI data pipelines or mission-critical databases.
A major driver for pod restarts, cascading microservice delays, and over-inflated cloud bills is misconfigured or underperforming persistent volumes.
What will you do to ensure you have the right provisioning class for your production apps?
Enter Kubestr. Kubestr, a lightweight, open-source command-line solution, is developed by Kasten by Veeam and is designed to discover, validate, and benchmark your Kubernetes storage resources. Let's take a closer look at deployment, complex Container Storage Interface (CSI) architectures, and tweaking your benchmark methodology for today's enterprise workloads.
What is Kubestr and Why Do You Need It?
Kubernetes storage is much more dynamic than traditional Virtual Machine environments, where your storage target is fixed. Storage classes are mechanisms used to provision and attach block or file storage on-demand, via CSI drivers.
For this layout, Kubestr serves as an automated diagnostic tool. Kubestr can be run from your terminal without having to manually write and apply the testing YAML manifests, and it is secure enough to connect through your active kubeconfig file.
Kubernetes deployment architecture with Kubestr performing storage class discovery, CSI snapshot verification, and FIO testing pods.
Kubestr can address three operational challenges:
- Storage Discovery: It scans your cluster and properly describes all storage classes and volume provisioners available in your cluster.
- CSI Verification: Safely validates that your storage providers can easily create, monitor, and then restore volume snapshots.
- Performance Benchmarking: It helps you to track your storage IOPS, throughput, and latency boundaries accurately using automated ephemeral pods with FIO (Flexible I/O Tester).
Step-by-Step Guide: Validating Storage with Kubestr
There's no need to install permanent operators or heavy CRDs in your cluster to run Kubestr. It is a binary utility that is run on your administrative machine and creates short-duration testing pods within your environment.
Step 1: Install Kubestr Locally
The official open-source release is available on GitHub, and you can download the correct architecture version directly from there.
Bash
Step 2: Discover and Check Storage Health
Perform a baseline validation check before stressing the volumes. Make sure that your terminal is pointing to your target cluster:
Bash
This command provides a high-level summary of your cluster version, evaluates the permissions you have for the current user, and identifies your active storage classes. If the driver is misconfigured, then Kubestr identifies the specific driver that is producing errors.
Step 3: Run an FIO Storage Performance Benchmark
You need actual performance data to properly size workloads. Any storage class found can be directly accessed with Kubestr for benchmarking.
Bash
When you call this command, Kubestr does the following background actions:
- A 50GB (Persistent Volume Claim) PVC provisioned dynamically with the storage class standard-ssd-sc.
- Adoption of a specialized testing pod that features an FIO engine, which mounts to this volume.
- Logging of physical throughput limits through execution of a set of default benchmarking tests including random read/ write mixes.
- Clean removal and destruction of the temporary testing pod and PVC resources.
Storage Benchmarking: Critical Workload Profiles
Some services in the enterprise are better suited to certain types of tests over others when optimizing storage space. The table below shows a mapping between specific FIO profiling configurations and real application requirements:
Upgrade Storage Infrastructure Standards
Infrastructure teams are facing a shortage of general-purpose block testing because of the increasingly complex cluster design. They now require a verified training strategy that focuses on modern storage protocols.
1. Enabling NVMe-oF-Enabled CSI Drivers
Data-intensive processing workloads and high-throughput AI environments are quickly migrating from traditional iSCSI to NVMe-oF. When testing, make sure to upgrade local vendor CSI drivers to support direct NVMe mapping. To prove that your CSI adaptation layer is not causing soft IRQ or CPU bottlenecks that are artificially capping how much storage can be achieved on the raw hardware.
2. Customizing Localized LLMs and Benchmarks for Vector Databases
This is the high storage pressure when running local Large Language Models (LLMs) and vector databases. When performing commands, rely on sequential read bandwidth and raw read latencies, rather than on random write IOPS:
Bash
The higher the sequential speed, the faster your weights and large context fragments are pulled into the volatile GPU memory, decreasing cold-start times for your models.
3. Conducting Continuous Cost-Performance Audits
With cloud storage, you can set up storage tiers (like AWS GP3 or Azure Ultra Disk options) with the flexibility of configuring IOPS and throughput independently of volume size. Use Kubestr as an ongoing auditing tool, not a one-off script to validate. You can avoid oversubscription of premium provisioned performance ranges that your cluster's current node CPU limits or physical network fabrics are already limiting.
Conclusion
You no longer need to rely on speculation or hand-editing of unwieldy YAML when you need to verify your Kubernetes storage layer. Kubestr serves as the missing link between the underlying hardware and the Kubernetes API layer, taking the burden of executing micro-benchmarks and reducing it down to a few commands in the terminal.
By embedding Kubestr validations into your automated workflows to provision clusters, you can avoid performance issues as your stateful pods begin to run. Whether you are optimizing for high-performance modern NVMe over Fabrics infrastructures, or you are tuning your storage to serve data feeds into localizable AI models packed with vector data, automated profiling can save you from falling into either the pit of resource starvation, or the trap of overpaying your cloud provider.
FAQs:
Does Kubestr impact workloads running inside the active cluster?
Kubestr generates isolated testing pods and fresh Persistent Volume Claims (PVCs) on the fly to execute Flexible I/O (FIO) tests and thus should not impact your active cluster. Keep in mind, however, that running FIO tests in a shared physical storage tier will have an impact on other workloads’ available IOPS in that shared tier. Therefore, running high-throughput FIO tests during a maintenance window is advised, and Kubestr is generally not recommended to be executed in production clusters.
What is the cause of the “Snapshot Class Not Found” error?
It likely means that you do not have your Volume Snapshot CRDs configured, or that the CSI snapshot controller of your cloud provider has not yet been deployed. Kubestr performs end-to-end validations, so that means that, if it is unable to create a point-in-time snapshot, it will not be able to test that tier of storage, which assumes your provisioner will support the VolumeSnapshotClass type it attempts to create.
Why does my cloud provider give higher numbers than Kubestr?
These are usually related to your nodes and network levels. The typical compute tier that you pick in the cloud restricts both your network throughput and your storage IOPS to some maximum value for a virtual machine profile. If you get lower numbers than the maximum volume capacity, this probably indicates your node has reached its network or vCPU limits, and therefore could not test the full volume bandwidth.
Comments
Post a Comment