Upgraded highmem nodes on Apocrita¶

The Apocrita highmem nodes have just been upgraded so that they contain newer CPUs with more modern instruction sets.

Previously the highmem nodes on Apocrita were named emf and contained Intel Xeon E5-2695 v2 CPUs. These CPUs were first launched in 2013, at a time before the newer AVX2 and AVX-512 AVX instruction sets were widely available.

We have now upgraded the highmem nodes as well as renaming them to smf. These nodes now contain Intel Xeon Gold 6126 CPUs, which were released in 2017 and are the same CPUs you can find in our sdv nodes. The Xeon Gold 6126 is much more performant compared to the older Xeon E5-2695 v2:

The Xeon Gold 6126 also supports AVX2 and AVX-512 instruction sets. In our internal benchmarks we have seen up to 50% improvement on the new smf nodes compared to the old emf nodes when running like-for-like benchmarks.

Taking advantage of newer instruction sets in R¶

Some R packages such as Seurat can take advantage of the newer AVX2 and AVX-512 instruction sets. However, because we now provide a default centralised Makevars for all R users on Apocrita, packages such as Seurat will automatically compile based on the CPU present at the time they are installed (-march=native). In the past, this meant if you installed Seurat into a personal library in an Apocrita session running on a ddy node (our most commonly offered non-highmem node), then it would be compiled against the CPU contained in those nodes, a CPU which supports AVX2 and AVX-512. This was fine if you then also ran all subsequent workloads on a node containing a CPU that also supported AVX2 and AVX-512.

The issue that used to arise is that the emf nodes didn't support AVX2 and AVX-512, so if you installed Seurat into a personal R library on a ddy node but then tried to execute R code using it on an emf node, you would get something along the lines of:

*** caught illegal operation ***
address 0x2b70b6710598, cause 'illegal operand'

This was because Seurat was trying to address instruction sets that weren't present.

The upgrade from emf to smf should now prevent this from happening.

Recreate your library

If you previously installed your personal library on an emf node to circumvent the above issues, we strongly advise you to clear out your existing library and then reinstall everything in a qlogin/ qsub session (not on a frontend node - these don't support AVX512) or in RStudio on Open OnDemand. This will ensure that packages such as Seurat take advantage of AVX2 and AVX-512.

If you have any questions regarding these changes, please contact us on our Slack channel (QMUL users only), or by sending an email to its-research-support@qmul.ac.uk which is handled directly by staff with relevant expertise.

Title image: Cris DiNoto on unsplash