Skip to main content
Drives are the part of your machine that wears out on their own schedule. A failing drive rarely stops all at once. It starts logging errors, containers fail to start, the filesystem drops to read-only, and your machine deverifies while the drive is still technically working. Vast requires SSD storage, a drive of at least 200 GB dedicated to Docker container storage, and 20 GB free on the root partition. See the verification requirements. The steps are the same on Ubuntu Server 22.04 and 24.04 except in step 6. 22.04 ships smartmontools 7.2 and 24.04 ships 7.4, and NVMe self-tests need 7.4.
Steps 2 to 5 and step 7 report on the drive without changing it and are safe on a machine with instances running. The one exception is an optional counter reset in step 5, flagged where it appears. Step 1 installs a package and starts a service, step 6 competes with client I/O, and step 8 takes the machine down. Plan those three.

​
1. Install smartmontools

nvme-cli covers the NVMe self-test that smartmontools 7.2 cannot run, so install both.
Check which smartmontools you have. Two NVMe features, starting a self-test with -t and reading the result with -l selftest, exist only in 7.4.

​
2. Find the drives

TRAN tells you which rows are physical drives. sata, sas and nvme are drives. Rows with no TRAN value are not: loop devices come from snaps, which Ubuntu Server installs by default, and some systems add a zram device for compressed swap. A stock Ubuntu Server has the former and not the latter. ROTA is the next column to read. 0 means the drive is not rotational, so it is an SSD. 1 is a spinning disk, which does not meet the storage requirement for the storage the platform uses.
Do not filter this command by major number. Linux spreads SATA and SAS disks across majors 8, 65 to 71, and 128 to 135, so a filter that names only major 8 silently hides every drive past the sixteenth on a dense chassis.
Now map the mountpoints back to physical drives. Ubuntu Server’s guided installer uses LVM by default, often over LUKS, so the filesystem is several layers above the disk:
Read up the tree from the mountpoint to the disk row. That top-level name is what you pass to smartctl. Here / lives on a logical volume, inside LUKS, on a partition of nvme0n1, so the drive to check is /dev/nvme0. The machine above has full-disk encryption, which adds the crypt row. A stock Ubuntu Server install has the same shape without it: partition, then LVM, then the mountpoint. Either way you are reading upward to the disk row. Container storage matters most, because that is where client instances live:
The same filesystem on both lines, as above, means Docker is sharing the root partition rather than using a dedicated drive.
This page is about whether the drive is failing, not whether it is full. A drive that is out of space but healthy is a different problem with a different fix.

​
If your drives are behind a RAID controller

If your SSDs sit behind a hardware RAID controller, the operating system sees a single virtual disk, and every command on this page reads that virtual disk rather than the drives inside it. Your drives can be wearing out or failing while the checks come back clean. To confirm which controller you are working with:
A hit means the machine has a RAID-capable controller, though not necessarily an array: an Intel chipset in RAID mode matches this too. Confirm by asking smartctl what it makes of the device. -d test reports the type it detects, then exits without running anything else:
A directly attached NVMe or SATA drive names its own type, as above. A controller’s virtual disk opens as a plain scsi device, and the per-drive SMART data is not behind it. scsi on its own is not proof of an array. A directly attached SAS drive is a real SCSI device and reports scsi too, with SCSI-format SMART data of its own that the rest of this page still applies to. Read the answer against what lspci found: scsi on a machine with a RAID controller and no SAS drives is the array. Where both are possible, the controller’s own tool (storcli, ssacli, arcconf) settles it by listing the physical drives behind the virtual disk. To reach the physical drives, name the controller type and the drive number: N is the drive’s number on the controller: 0 to 127 for MegaRAID and 3ware, 0 to 15 for cciss, and 1 to 24 on an Areca SATA controller. An Areca SAS controller is addressed by slot and enclosure instead, where N is the channel (1 to 128) and E the enclosure (1 to 8), and it needs controller firmware 1.51 or later. Once the device type is right, steps 3 through 6 work as written, one drive at a time.
On Areca and HP Smart Array controllers, address the controller, not the array. smartmontools is explicit that for these you use the device nodes corresponding to the RAID controllers, not the nodes corresponding to logical drives: an Areca controller is reached through a SCSI generic node such as /dev/sg2, which is deliberately not the node the array reads and writes through. MegaRAID and 3ware are not addressed this way. They take the disk node (/dev/sda) or the 3ware node (/dev/twl0) as in the table above.
To work out which generic node belongs to which device:
This gets you per-drive health through the controller. Whether the array itself is healthy, degraded, or rebuilding is a separate question that SMART cannot answer: use zpool status for ZFS, /proc/mdstat for mdadm, or the controller’s own tool such as storcli, ssacli or arcconf.

​
3. Check the overall verdict

Start with the drive’s own one-line summary. Use the controller name for NVMe (/dev/nvme0), and the disk name for SATA and SAS (/dev/sda).
FAILED means the drive has already failed or predicts its own failure within the next 24 hours. Stop renting the machine: unlist it with vastai unlist machine, and if instances are running, schedule the window with vastai schedule maintenance. If the machine has already dropped off the platform, work through Machine Offline.
PASSED is a low bar, not a clean bill of health. On NVMe it reflects a single byte, the Critical Warning field. On SATA it is one boolean the firmware returns. A drive can report PASSED while it is visibly wearing out or throwing media errors, which is why steps 4 and 5 exist.

​
4. Read an NVMe drive

The section to read is the health log:
Four fields decide whether the drive stays in the machine: Critical Warning is the field the drive itself considers an alarm. Each bit is a separate warning: available spare below threshold, a temperature threshold crossed, NVM subsystem reliability degraded, all media placed in read-only mode, volatile memory backup failed, or the persistent memory region gone read-only. Anything other than 0x00 needs explaining before the machine takes another rental. Available Spare is the percentage of spare capacity left for the drive to remap failing blocks with. Available Spare Threshold is the level the drive manufacturer set as the point of concern, 10% in the output above. The drive raises the matching Critical Warning bit once spare has fallen below it. Media and Data Integrity Errors counts occurrences where the controller hit an unrecovered data integrity error, such as an uncorrectable ECC failure, a CRC checksum failure, or an LBA tag mismatch. On a drive holding client data, this number should be zero. A drive whose count is rising between maintenance windows is corrupting data, not aging gracefully.
Percentage Used is not a failure indicator. It is the manufacturer’s estimate of how much of the drive’s rated endurance has been consumed. A value of 100 means the rated endurance has been used up, which the NVMe specification says may not indicate a failure. The value is allowed to exceed 100, and anything above 254 is reported as 255. It also updates only once per power-on hour, so re-reading it minutes later tells you nothing. Replace on Critical Warning, spare capacity, and media errors; use Percentage Used to plan ahead.
Temperature is a composite figure the controller computes. Compare it with the thresholds printed further up in the information section:
Individual sensors can read hotter than the composite value, so a single high sensor reading is not automatically a problem. Warning Comp. Temperature Time climbing between checks is, and it usually means airflow rather than the drive. Both temperature-time fields accumulate over the life of the drive, so a non-zero value may be entirely historical. Unsafe Shutdowns counts power losses where the drive was not told to shut down first. It is a record of how the machine has been treated rather than a fault in the drive, but each one is a chance the filesystem was left inconsistent, which is what step 7 checks. A number that grows every maintenance window points at how the machine is being powered off.

​
5. Read a SATA or SAS SSD

Use -x rather than -a on SATA and SAS drives. -a does not enable the options that need 48-bit ATA commands, so it leaves out logs this page relies on. On NVMe the two are equivalent.
SATA drives report a table of vendor attributes instead of a fixed log:
The FAIL column is the drive’s own verdict, but it only works where there is a threshold to compare against. In -x output it reads NOW if the normalized VALUE has dropped to or below THRESH, Past if it did so earlier and recovered, and - otherwise.
A dash in FAIL does not mean the attribute is healthy. --- in THRESH means smartctl could not obtain a threshold for that attribute, and an attribute with no threshold can never report NOW or Past no matter how bad its raw value gets.In the table above, 5 Reallocated_Sector_Ct and 187 Reported_Uncorrect both show THRESH ---. A drive with thousands of reallocated sectors would print the same - in that column as this healthy one. Read the raw values of the attributes below directly; do not scan the FAIL column and conclude the drive is fine.
These are the attributes to read directly, whatever the FAIL column says: Not every drive reports every attribute. The sample above has no 197 or 198; a drive that does not report an attribute simply omits the row.
Attribute numbers are vendor-specific, and the names smartctl prints are only correct when the drive is in the smartmontools database. On an unrecognised drive the ID may be right while the name and raw interpretation are wrong.
The raw values themselves are not standardized either. The conversion from a raw value to anything physical is not defined by the SMART specification, and vendors use their own conventions. The table above is a healthy drive with four and a half years of power-on hours, and it shows the trap: 230 Media_Wearout_Indicator has a VALUE of 001, which reads like 1% of life remaining. Its raw value is 0x011b0046011b, a packed vendor field rather than a percentage, and the drive is fine. For wear specifically, prefer the standardized figure in the device statistics log over any vendor attribute:
A non-zero 199 UDMA_CRC_Error_Count, or non-zero ICRC errors under SATA Phy Event Counters, points at the SATA cable or connector rather than the drive. Reseat both ends before condemning a disk. Attribute 199 is a lifetime count, so what matters is whether it grows between maintenance windows. The Phy counters can be cleared with -l sataphy,reset if you want a clean baseline before a soak test. That resets the counters on the drive, so it is the one command in this step that writes.
SAS drives report differently. A SAS SSD returns SCSI-format output rather than the ATA attribute table above: look for the grown defect list, the non-medium error count, and the read/write error counters. The overall verdict in step 3 and the self-test in step 6 work the same way.

​
6. Run a self-test

The health log records what the drive noticed during normal use. A self-test makes it go and look.
A self-test runs on a machine in normal use, but it competes with client I/O and will slow the drive down while it runs. Run it during a maintenance window, not while instances are working.
A short test takes a couple of minutes and checks a sample of the media. A long test reads the whole surface, takes hours on a large drive, and is the one that finds media errors a short test walks past. Use -t long when you suspect the drive but a short test came back clean.

​
SATA and SAS

Works on both Ubuntu releases.
The drive reports how long to wait in the Short self-test routine recommended polling time line of its -x output. Check progress or read the result with:
While a test is running the log reports Self-test routine in progress with a percentage remaining. When it finishes, the result appears in the table under SMART Self-test log structure revision number 1. Completed without error is what you want; a read failure naming a block the drive could not read is a replacement, not a repair.
On SATA, -l selftest reads the standard self-test log, which keeps the last 21 results and reports the LBA of an error in 28 bits. The extended log keeps more results and reports a full 48-bit LBA, which is what you need to name a bad block on a large drive. Read it with smartctl -l xselftest /dev/sda, or take it from the SMART Extended Self-test Log section of -x. SAS drives have no extended log, so -l selftest is the one to read there.

​
NVMe

On Ubuntu 24.04, smartmontools handles it:
The drive keeps the last 20 results, so you can tell a new failure from an old one.
On Ubuntu 22.04, smartmontools 7.2 cannot run NVMe self-tests, and it fails silently. smartctl -t short /dev/nvme0 exits successfully with a message suggesting you run -a instead, so it looks like the test started when nothing did. Its -x output has no self-test section at all.
Use nvme-cli instead on 22.04, which does support them:
--self-test-code=1 is the short test; 2 is the extended test.

​
7. Check what the kernel saw

The drive’s own logs miss failures that happen between the drive and the rest of the machine. The kernel does not.
Anything here changes the priority. I/O errors, controller resets, and link retraining mean the machine is already losing operations, whatever SMART says. A filesystem that has dropped to read-only is the clearest signal of the set, and it fails every container start on that drive:
ro on a filesystem that is supposed to be writable means the kernel remounted it after an error. Take the machine out of service as described in step 3.

​
8. Replace a failing drive

Replacing a drive usually means powering the machine down, which stops every running instance on it. Instances are not destroyed, but the workloads inside them are interrupted. Wait until all active rental contracts have ended, or schedule the window with vastai schedule maintenance so renters are notified and can save their work.Hot-swap bays on a redundant array can take a replacement without powering down. Treat that as something you have confirmed for your hardware, not as the assumption you start from.
Before the window, decide what the drive was holding:
  • The Docker storage drive. Client instances live here. Replacing it means the machine comes back with no cached images, so the first renters after the swap will wait for a full image pull.
  • The root drive. This is a rebuild: reinstall Ubuntu Server, the NVIDIA driver, and the Vast host software, then re-verify.
  • One member of a redundant array. The array rebuilds onto the replacement and the data survives, so this is the one case where a swap does not cost you the contents of the drive. The rebuild competes with client I/O for as long as it runs, so let it finish before you relist. Check on it with zpool status, /proc/mdstat, or your controller’s tool.
After the swap, confirm the new drive presents correctly before relisting:
Check ROTA is 0 on the replacement, that the machine reports the storage you expect, and that the root partition still has its 20 GB free. Then run the platform’s own machine self-test, which checks the GPUs and network as well, before you relist.

​
Recovery

​
smartctl says SMART is disabled

Nothing in steps 3 to 6 will return useful data until you turn it on:

​
smartctl cannot work out the device type

If a command returns Unknown USB bridge or a type you did not expect, ask it what it detected:
Then name the type explicitly. -d sat covers most SATA drives behind a SAS or USB bridge, and the RAID pass-through types are in step 2.

​
A drive is missing from lsblk entirely

A drive that does not appear at all has dropped off the bus, and that is a stronger failure signal than anything SMART would have told you. Check the kernel log from step 7 for the device disappearing, then reseat its data and power cables. A drive that stays missing is a replacement.

​
Get told before it fails

Everything above is a point-in-time check. smartd watches continuously and logs when a drive starts failing, which is how you find out without logging in to look. Installing the package enables it, so this is usually a check rather than a change:
If it comes back inactive, start it:
The service refuses to start inside a VM. Both releases ship the unit with ConditionVirtualization=no, so if you run your GPUs through a VM with PCI passthrough, enable --now completes without starting anything and is-active still reports inactive. Override it with sudo systemctl edit smartmontools.service, adding:
smartd.service works as a name on both releases; the package ships it as an alias of smartmontools.service. Either name is fine.
Its configuration lives in /etc/smartd.conf. Both releases ship the same single active line, which scans every device it can see and mails root:
DEVICESCAN does not cover everything this page checks. On NVMe drives smartd alerts on the Critical Warning byte, temperature, and new error log entries only. It does not watch Available Spare, Percentage Used, or Media and Data Integrity Errors, which are three of the four fields step 4 uses to decide whether a drive stays in the machine. Drives behind a RAID controller need their own explicit lines with the -d type from step 2. Neither case is covered by the shipped default, so keep running steps 3 to 5 by hand each window.
To be told directly, change -m root to your own address and leave the rest of the line alone. Mail only reaches you if the machine has a working mail transport, which a stock Ubuntu Server install does not. Without one, the warnings still land in the journal:
You are done when every drive returns PASSED, NVMe drives show Critical Warning: 0x00 with spare capacity above the threshold and no media errors, SATA drives show no NOW or Past in the FAIL column and zero reallocated, pending, and uncorrectable sectors, the kernel log from step 7 is clean, and smartmontools.service is active.