[Translation] When the environment variable speeds up the process 40 times

[Translation] When the environment variable speeds up the process 40 times


Today we want to talk about some recent updates of the Sherlock system [this is a high-performance cluster of Stanford University - approx. per.], which significantly speeds up the listing of files in directories with a large number of entries.

Unlike regular articles, this is rather an insider report on how regular work on Sherlock happens to keep it in its best form for our users. We hope to publish more such articles in the future.

Listing many files takes time


It all started with a question in tech support from the user. He reported a problem that executing ls takes several minutes in the c directory of more than 15,000 entries in the $ SCRATCH [directory for temporary files - approx. trans.].

Thousands of files in the same directory usually cause problems for the file system and this is definitely not recommended. The user knew this and admitted that it was not good, but mentioned that the listing on his laptop is 1000 times faster than in Sherlock. Of course, it hurt us. So we looked deeper.

Because ls looks beautiful


We looked at what ls actually does when listing a directory, and why the process takes so long. In most modern distributions, ls defaults to ls --color = auto , because everyone likes colors.

But beautiful colors have their price: for each file, ls should get information about the file type, its permissions, flags, extended attributes, and the like, in order to choose the appropriate color.

One of the simple solutions to the problem is to turn off the color in ls altogether, but imagine the users perturbation. In no case can not take the color output, we are not monsters.

Therefore, we looked deeper. ls colors the entries in the LS_COLORS environment variable that dircolors (1) sets based on the dir_colors (5) configuration file . Yes, an executable file reads the configuration file to create an environment variable, which is then used by ls (if you don’t know about the door (do) files, then dir_colors will work , no matter what).

We will understand more


To determine which colorization scheme is causing a slowdown, we created an experimental environment:

  $ mkdir $ SCRATCH/dont
 $ touch $ SCRATCH/dont/{1..10000} # don't try this at home!
 $ time ls --color = always $ SCRATCH/dont |  wc -l
 10,000

 real 0m12.758s
 user 0m0.104s
 sys 0m0.699s  

12.7 seconds for 10,000 files is not very good.
By the way, the - color = always flag is needed: although it appeals to ls --color = auto , but ls detects when it is not connected to the terminal (for example, via a channel or with redirection of output) and disables coloring if set to auto . Smart guy.
So what takes so much time? We watched using strace :

  $ strace -c ls --color = always $ SCRATCH/dont |  wc -l
 10,000
 % time seconds usecs/call calls errors syscall
 ------ ----------- ----------- --------- --------- ----  ------------
  44.21 0.186617 19 10000 lstat
  42.60 0.179807 18 10000 10000 getxattr
  12.19 0.051438 5 10000 capget
  0.71 0.003002 38 80 getdents
  0.07 0.000305 10 30 mmap
  0.05 0.000217 12 18 mprotect
  0.03 0.000135 14 10 read
  0.03 0.000123 11 11 open
  0.02 0.000082 6 14 close
 [...]  

Wow: 10,000 calls of lstat () , 10,000 calls of getxattr () (which all fail because there are no attributes in our environment that ls is looking for) 10 000 calls of capget () .

Surely this can be optimized.

Attributes capabilities? Nope


Following the advice of a 10-year-old bug , we tried to disable the capabilities :

  $ eval $ (dircolors -b | sed s/ca = [^:] *:/ca =:/)
 $ time strace -c ls --color = always $ SCRATCH/dont |  wc -l
 10,000
 % time seconds usecs/call calls errors syscall
 ------ ----------- ----------- --------- --------- ----  ------------
  98.95 0.423443 42 10000 lstat
  0.78 0.003353 42 80 getdents
  0.04 0.000188 10 18 mprotect
  0.04 0.000181 6 30 mmap
  0.02 0.000085 9 10 read
  0.02 0.000084 28 3 mremap
  0.02 0.000077 7 11 open
  0.02 0.000066 5 14 close
 [...]
 ------ ----------- ----------- --------- --------- ----  ------------
 100.00 0.427920 10221 6 total

 real 0m8.160s
 user 0m0.115s
 sys 0m0.961s  

Wow, acceleration up to 8 seconds! We got rid of all these expensive getxattr () calls, and capget () calls also disappeared, great.

But there are still these annoying lstat () calls, although ...

How many colors do you need?


Therefore, we looked at LS_COLORS in more detail.

At first, just turned off this variable:

 $ echo $ LS_COLORS
 rs = 0: di = 01; 34: ln = 01; 36: mh = 00: pi = 40; 33: so = 01; 35: do = 01; 35: bd = 40; 33; 01: cd = 40;  33; 01: or = 40; 31; 01: su = 37; 41: sg = 30; 43: ca =: tw = 30; 42: ow = 34; 42: st = 37; 44: ex = 01; 32: *. tar = 01; 31: *. tgz = 01; 31: *. arc = 01; 31: *. arj = 01; 31: *. taz = 01; 31: *. lha = 01; 31: *  .lz4 = 01; 31: *. lzh = 01; 31: *. lzma = 01; 31: *. tlz = 01; 31: *. txz = 01; 31: *. tzo = 01; 31: *. t7z  = 01; 31: *. Zip = 01; 31: *. Z = 01; 31: *. Z = 01; 31: *. Dz = 01; 31: *. Gz = 01; 31: *. Lrz = 01  ; 31: *. Lz = 01; 31: *. Lzo = 01; 31: *. Xz = 01; 31: *. Bz2 = 01; 31: *. Bz = 01; 31: *. Tbz = 01; 31: *. tbz2 = 01; 31: *. tz = 01; 31: *. deb = 01; 31: *. rpm = 01; 31: *. jar = 01; 31: *. war = 01; 31: *  .ear = 01; 31: *. sar = 01; 31: *. rar = 01; 31: *. alz = 01; 31: *. ace = 01; 31: *. zoo = 01; 31: *. cpio  = 01; 31: *. 7z = 01; 31: *. Rz = 01; 31: *. Cab = 01; 31: *. Jpg = 01; 35: *. Jpeg = 01; 35: *. Gif = 01  ; 35: *. Bmp = 01; 35: *. Pbm = 01; 35: *. Pgm = 01; 35: *. Ppm = 01; 35: *. Tga = 01; 35: *. Xbm = 01; 35: *. xpm = 01; 35: *. tif = 01; 35: *. tiff = 01; 35: *. png = 01; 35: *. svg = 01; 35: *. svgz = 01; 35: *  .mng = 01; 35: *. pcx = 01; 35: *. mov = 01; 35: *. mpg = 01; 35: *. mpeg = 01; 35: *. m2v = 01; 35: *. mkv  = 01; 35: *. Webm = 01; 35: *. Ogm = 01; 35: *. Mp4 = 01; 35: *. M4v = 01; 35: *. Mp4v = 01; 35: *. Vob = 01  ; 35: *. Qt = 01; 35: *. Nuv = 01; 35: *.  wmv = 01; 35: *. asf = 01; 35: *. rm = 01; 35: *. rmvb = 01; 35: *. flc = 01; 35: *. avi = 01; 35: *. fli =  01; 35: *. Flv = 01; 35: *. Gl = 01; 35: *. Dl = 01; 35: *. Xcf = 01; 35: *. Xwd = 01; 35: *. Yuv = 01;  35: *. Cgm = 01; 35: *. Emf = 01; 35: *. Axv = 01; 35: *. Anx = 01; 35: *. Ogv = 01; 35: *. Ogx = 01; 35:  * .aac = 00; 36: *. au = 00; 36: *. flac = 00; 36: *. mid = 00; 36: *. midi = 00; 36: *. mka = 00; 36: *.  mp3 = 00; 36: *. mpc = 00; 36: *. ogg = 00; 36: *. ra = 00; 36: *. wav = 00; 36: *. axa = 00; 36: *. oga =  00; 36: *. Spx = 00; 36: *. Xspf = 00; 36:
 $ unset LS_COLORS
 $ echo $ LS_COLORS

 $ time ls --color = always $ SCRATCH/dont |  wc -l
 10,000

 real 0m13.037s
 user 0m0.077s
 sys 0m1.092s 

What!?! Still 13 seconds?

It turns out that when the LS_COLORS environment variable is not defined or only one of its & lt; type & gt; = color: elements is missing, it uses the built-in database by default and still uses colors. Therefore, if you want to turn off coloring for a specific file type, you need to override it with & lt; type & gt; =: or & lt; type & gt; 00 in the DIR_COLORS file.

After a lot of trial and error, we narrowed the search to:

  EXEC 00
 SETUID 00
 SETGID 00
 CAPABILITY 00  

what is written as

  LS_COLORS = 'ex = 00: su = 00: sg = 00: ca = 00:'  

This means: do not color the files by the capabilities , none of the setuid/setgid nor by the executable flag .

Accelerate ls


And if you don’t do any of these checks, then the lstat () calls disappear, and now it's another matter:

  $ export LS_COLORS = 'ex = 00: su = 00: sg = 00: ca = 00:'
 $ time strace -c ls --color = always $ SCRATCH/dont |  wc -l
 10,000
 % time seconds usecs/call calls errors syscall
 ------ ----------- ----------- --------- --------- ----  ------------
  63.02 0.002865 36 80 getdents
  8.10 0.000368 12 30 mmap
  5.72 0.000260 14 18 mprotect
  3.72 0.000169 15 11 open
  2.79 0.000127 13 10 read
 [...]
 ------ ----------- ----------- --------- --------- ----  ------------
 100.00 0.004546 221 6 total

 real 0m0.337s
 user 0m0.032s
 sys 0m0.029s  

0.3 seconds on the list of 10,000 files, a record.

Customize Sherlock


From 13 seconds with default settings up to 0.3 seconds with a small LS_COLORS setting means 40-fold acceleration due to the absence of setuid / setgid and colored executable files. Not such a big loss.

Of course, this is now configured in Sherlock for each user.

But if you want to return the coloring, you can simply return to the default settings:

  $ unset LS_COLORS  

But then on directories with a large number of files, be sure to make coffee while ls is running.

Source text: [Translation] When the environment variable speeds up the process 40 times