Quotas on irene

From www.coria-cfd.fr
Jump to: navigation, search

What's the problem?

Sometimes, on Irene Joliot-Curie (the last TGCC supercomputer), you will be confronted to the following error:

error writing ‘XXXX’: Disk quota exceeded

This problem is actually more complex than it seems. This goal of this page is to provide the explanation of this issue and how to solve it.


On Linux, a user is defined by a number, which is set in /etc/passwd. For exemple, on my machine, I am the user 1001:

> cat /etc/passwd | grep lartigue


As you can see, the number 1001 is printed two times on this line: the first one corresponds to my 'User ID', the second one corresponds to my 'Group ID': these numbers are not necessarily identical. This means that each user is associated to a group, known as its 'principal group'.

However, a user can be associated to many other groups (which are known as ‘secondary groups’). For exemple, on irene, my groups are:

> groups

coria avbp ra0381 gen6880 gch0011 ra1113 intelmpi gen7345 gen10161 ra3999 gch0305 cont005

The first group that appears on this line is my principal group, the others are my secondary groups.

Creating files/directories

On Linux, each file on the filesystem possesses some metadata: its size, its creation date, to which user and to which group it belongs and what are the associated permissions. When a user creates a file (via cp/scp/touch/rsync/vi/...) this file is automatically associated to this user and to its principal group.

For example when I create a file on irene:

> touch /tmp/toto

> ls -l /tmp/toto

-rw-r-----. 1 lartigug coria 0 Nov 5 08:38 /tmp/toto

This file is empty (size=0), it has been created on Nov. 5th at 8:38AM and is associated to me (user=lartigug) and to my principal group (group=coria).

Moreover, as my umask is set to 0027, its permissions are set to -rw-r-----

The real Issue

It is actually quite simple to understand: on every production machine, there are some quota associated to users and groups. This means that the amount of data that belongs to you or to one of your group can not go beyond a certain limit.

Now, if you transfer a lot of files (scp/rsync) or if you create a lot of file (during a simulation), all these files will belong to your principal group. And if there is a quota on this group, you will get the nasty error error writing ‘XXXX’: Disk quota exceeded.

How to solve this issue

Well, the idea is quite simple: you must create the files with a different group for which the quota is not exceeded…

For example on irene, the quota on group gen6880 is much larger than on the group coria.

But HOW?

Technical solution #1

If the files are already there but you want to change their group, just use the classical chgrp [-R] command (with -R to go recursive).

> mkdir /tmp/dir_test

> ls -la /tmp/dir_test

drwxr-x---. 2 lartigug coria 6 Nov 5 08:58 .

chgrp gen6880 /tmp/dir_test

drwxr-x---. 2 lartigug gen6880 6 Nov 5 08:58 .

Technical solution #2

You can change the default behaviour of directories so that new files are created with different ownership.

If I create a file in the previous directory /tmp/dir_test (group=gen6880), this file is created with the classical ownership lartigug:coria

> touch /tmp/dir_test/file1

> ls -l /tmp/dir_test/file1

-rw-r-----. 1 lartigug coria 0 Nov 5 09:04 /tmp/dir_test/file1

Now let me setgid the directory /tmp/dir_test. This is achieved with the following command:

> chmod g+s /tmp/dir_test

Now check what happend:

> ls -la /tmp/dir_test

drwxr-s---. 2 lartigug gen6880 32 Nov 5 09:05 .

The s indicates that the directory is now setgid. This means that each new file/directory will be created with ownership lartigug:gen6880

Let’s check this:

> touch /tmp/dir_test/file2

> ls -l /tmp/dir_test/file2

-rw-r-----. 1 lartigug gen6880 0 Nov 5 09:05 file2

Moreover when creating a new sub-directory, it will keep its setgid property!

> mkdir subdir

> ls -l

drwxr-s---. 2 lartigug gen6880 6 Nov 5 09:15 subdir

Technical solution #3

Some difficulties may arise when you transfer files from your computer to irene.

Use tar & scp

first tar you directories transfer the tar untar

This will keep the right ownership and setgid property in all your directories/subdirectories.

Use rsync with options

Use rsync but add the following options to modify its default behaviour:

rsync --chmod=g+s --chown=:gen6880 …….

Use scp -r

If you do so, you MUST then do a chmod -R g+s on your directory because the setgid will not propagate in the subdirectories.