Quotas on irene
Contents
What's the problem?
Sometimes, on Irene Joliot-Curie (the last TGCC supercomputer), you will be confronted to the following error:
error writing ‘XXXX’: Disk quota exceeded
This problem is actually more complex than it seems. This goal of this page is to provide the explanation of this issue and how to solve it.
Groups
On Linux, a user is defined by a number, which is set in /etc/passwd. For exemple, on my machine, I am the user 1001:
> cat /etc/passwd | grep lartigue
lartigue:x:1001:1001::/home/lartigue:/bin/bash
As you can see, the number 1001 is printed two times on this line: the first one corresponds to my 'User ID', the second one corresponds to my 'Group ID': these numbers are not necessarily identical. This means that each user is associated to a group, known as its 'principal group'.
However, a user can be associated to many other groups (which are known as ‘secondary groups’). For exemple, on irene, my groups are:
> groups
coria avbp ra0381 gen6880 gch0011 ra1113 intelmpi gen7345 gen10161 ra3999 gch0305 cont005
The first group that appears on this line is my principal group, the others are my secondary groups.
Creating files/directories
On Linux, each file on the filesystem possesses some metadata: its size, its creation date, to which user and to which group it belongs and what are the associated permissions. When a user creates a file (via cp/scp/touch/rsync/vi/...) this file is automatically associated to this user and to its principal group.
For example when I create a file on irene:
> touch /tmp/toto
> ls -l /tmp/toto
-rw-r-----. 1 lartigug coria 0 Nov 5 08:38 /tmp/toto
This file is empty (size=0), it has been created on Nov. 5th at 8:38AM and is associated to me (user=lartigug) and to my principal group (group=coria).
Moreover, as my umask is set to 0027, its permissions are set to -rw-r-----
The real Issue
It is actually quite simple to understand: on every production machine, there are some quota associated to users and groups. This means that the amount of data that belongs to you or to one of your group can not go beyond a certain limit.
Now, if you transfer a lot of files (scp/rsync) or if you create a lot of file (during a simulation), all these files will belong to your principal group. And if there is a quota on this group, you will get the nasty error error writing ‘XXXX’: Disk quota exceeded.
How to solve this issue
Well, the idea is quite simple: you must create the files with a different group for which the quota is not exceeded…
For example on irene, the quota on group gen6880 is much larger than on the group coria.
But HOW?
Technical solution #1
If the files are already there but you want to change their group, just use the classical chgrp [-R] command (with -R to go recursive).
> mkdir /tmp/dir_test
> ls -la /tmp/dir_test
drwxr-x---. 2 lartigug coria 6 Nov 5 08:58 .
chgrp gen6880 /tmp/dir_test
drwxr-x---. 2 lartigug gen6880 6 Nov 5 08:58 .
Technical solution #2
You can change the default behaviour of directories so that new files are created with different ownership.
If I create a file in the previous directory /tmp/dir_test (group=gen6880), this file is created with the classical ownership lartigug:coria
> touch /tmp/dir_test/file1
> ls -l /tmp/dir_test/file1
-rw-r-----. 1 lartigug coria 0 Nov 5 09:04 /tmp/dir_test/file1
Now let me setgid the directory /tmp/dir_test. This is achieved with the following command:
> chmod g+s /tmp/dir_test
Now check what happend:
> ls -la /tmp/dir_test
drwxr-s---. 2 lartigug gen6880 32 Nov 5 09:05 .
The s indicates that the directory is now setgid. This means that each new file/directory will be created with ownership lartigug:gen6880
Let’s check this:
> touch /tmp/dir_test/file2
> ls -l /tmp/dir_test/file2
-rw-r-----. 1 lartigug gen6880 0 Nov 5 09:05 file2
Moreover when creating a new sub-directory, it will keep its setgid property!
> mkdir subdir
> ls -l
drwxr-s---. 2 lartigug gen6880 6 Nov 5 09:15 subdir
Technical solution #3
Some difficulties may arise when you transfer files from your computer to irene.
Use tar & scp
first tar you directories transfer the tar untar
This will keep the right ownership and setgid property in all your directories/subdirectories.
Use rsync with options
Use rsync but add the following options to modify its default behaviour:
rsync --chmod=g+s --chown=:gen6880 …….
Use scp -r
If you do so, you MUST then do a chmod -R g+s on your directory because the setgid will not propagate in the subdirectories.
