I've run out of storage space

There are two tracked resources in the NeSI filesystem, disk space and inodes (number of files).

Trying to write to a filesystem over its inode or disk quota will cause an error (and probably kill your job).

Current file-count and disk space can be found using nn_storage_quota.

Quota_Location    AvailableGiB   UsedGiB     Use%
home_user123               20G    1.957G    9.79%
project_nesi99999           2T      798G   38.96%
nobackup_nesi99999      6.833T

Note

There is a delay between making changes to a filesystem and seeing the change in nn_storage_quota, immediate file count and disk space can be found using the commands du --inodes and du -h respectively.

There are a few ways to deal with file count problems

Use the nobackup filesystem

Your nobackup directories /nesi/nobackup/<projectcode> has a significantly disk space limits.

Delete unnecessary files

Some applications will generate a large number of files during runtime, using the command du --inodes -d 1 | sort -hr (for inodes) or du -h -d 1 | sort -hr for disk space.  You can then drill down into the directories with the largest file count deleting files as viable.

SquashFS archive

Many files can be compressed into a single SquashFS archive. We have written a utility, nn_archive_files, to help with this process. This utility can submit the work as a Slurm job, which is preferred. nn_archive_files can take, as trailing options, the same options as mksquashfs, including choice of compression algorithm; see man mksquashfs for more details.

nn_archive_files -p <project-code> -n <num-processors> -t <time-limit> --verify -- /path/containing/files /path2/containing/files destination.squash

Then when files need to be accessed again they can be extracted using,

/usr/sbin/unsquashfs destination.squash

You can do many other things with SquashFS archives, like quickly list the files in the archive, extract some but not all of the contents, and so on. See man unsquashfs for more details.

Tarball

Many files can be compressed into a single 'tarball'

tar -czf name.tar /path/containing/files/

Then when files need to be accessed again they can be un-tarred using,

tar -xzf tarname.tar

Contact Support

If you are following the recommendations here yet are still concerned about inodes or disk space, Contact our Support Team and we can raise the limit for you.