New high performance storage for scratch data¶
We are pleased to announce a new scratch storage array that is based on fast NvME based hardware. This will hopefully make I/O related tasks much faster
Scratch space is the recommended location for temporarily storing data produced by cluster jobs. We often see jobs failing on the cluster due to users filling their home directory quota with data from job outputs, instead of using scratch space. Some users may be unaware that a P.I. can request a lab share and get the initial 1TB at no cost.
This new service replaces the old legacy /data/scratch
service, which we
switched off a few weeks ago, in order to re-use the same file paths. By
reusing the old paths we hope to make the service as simple to use as
possible.
The automatic deletion of old files is based on the same concept we have used
in the auto-purging scratch space called /data/autoScratch/weekly/$USER
and
/data/autoScratch/monthly/$USER
. We will be looking to phase these
weekly/monthly services out within the next few weeks or months in favour of
the faster and simpler /data/scratch/$USER
file path. This is partly due to
the disks these service are based on reaching end-of-life. All users should
start changing their job scripts to use the new service instead. The long-term
storage (home directories and lab shares) remains unchanged.
Action required
Change your job scripts to use /data/scratch/$USER
path (e.g.
/data/scratch/abc123
) instead of /data/autoScratch
. Also check for
presence of any symbolic links that point to the autoScratch service.
Files placed in /data/scratch
are deleted 35 days after last
modification. You will be sent warning emails that files are about to be
deleted after 31 days (4 days before deletion). These numbers have been
selected based on available capacity and observed behaviour. The service has
loose capacity-based quotas of 1TB or 1 million files, and requests to extend
this will be considered on an individual basis.
Scratch is not backed up
Scratch data is not backed up and any important data should be moved to other locations as quickly as possible. Data stored here will be automatically deleted. An email is sent to you shortly before this occurs.
Please do not abuse this service by modifying timestamps of your files to make this a permanent location as it is unfair to other users. It is far better to ensure that files are correctly backed up in the right place.