Reproduce why a 1-byte file is blocked by a 30 MB quota

Goal

You set a namespace quota and a space quota on directories, and confirm from the rejection messages that the namespace quota counts the directory itself and that the space quota reserves in advance as much as block size × replication factor. You let the same file in by reducing the block size, and also see that a quota blocks even moves (mv).

Why it matters

When several teams share one HDFS, one team's mistake (a job that creates files endlessly, millions of small files) takes the NameNode's heap and disk away from everyone. A quota is a device that confines that damage to one directory. The namespace quota limits the number of files and directories, and the space quota limits the bytes including replicas. The space quota has one trap. When a file starts being written, the NameNode does not yet know how many bytes will be written, so it reserves one whole block (block size × replication factor). So if the remaining quota is less than 128MB, even a 1-byte file cannot get in. When writing finishes, it is settled at the actual size, but it is useless if it cannot even start. Most of the inquiries in operations along the lines of "there's quota left, so why can't I write" are this. A quota applies to everything that creates a new name. Moving in from another directory is also a new name from that directory's point of view.

Steps

1. Create HDFS /quota/team-a and set a namespace quota of 10 (hdfs dfsadmin -setQuota).
2. Upload small local files one at a time to team-a until it is rejected, and save the error output of the rejected put to /root/hdp/quota/quota_err.txt.
3. Create /quota/team-b and set a space quota of 30MB (hdfs dfsadmin -setSpaceQuota 30m).
4. Create a 1-byte local file /root/hdp/quota/one.txt, try uploading it to /quota/team-b/one.txt, and save the error output to /root/hdp/quota/space_err.txt.
5. Upload the same file with a block size of 1MiB (-D dfs.blocksize=1048576) to /quota/team-b/one-1m.txt.
6. Save the output of hdfs dfs -count -q -v /quota/team-a /quota/team-b to /root/hdp/quota/count.txt.
7. Create /quota/staging, upload one file (/quota/staging/f.txt), then try -mv on it to team-a and save the error output to /root/hdp/quota/mv_err.txt.
8. In /root/hdp/quota/report.md, write three sections: ## 이름 쿼터, ## 공간 쿼터 and ## 쿼터가 막는 것 (use exactly these Korean headings in this order; they mean "Namespace quota", "Space quota" and "What a quota blocks"). Put the number of files that actually went into team-a in the first section and the number of bytes that it tried to reserve (134217728) in the second.

Notes

• Setting a quota is an administrator command (hdfs dfsadmin) and only the superuser can do it. In this Pod, that is root.
• The unit of the space quota is bytes, and you can use suffixes such as 30m and 1g. Replicas are counted too: with a replication factor of 3, a 1MB file uses 3MB.
• The columns of -count -q: QUOTA · REM_QUOTA · SPACE_QUOTA · REM_SPACE_QUOTA · DIR_COUNT · FILE_COUNT · CONTENT_SIZE · PATHNAME.
• Common mistakes: thinking 10 files go in with a namespace quota of 10; and judging that you can write by looking only at the remaining space quota.
• Official documentation: HDFS Quotas Guide · HDFS Commands — dfsadmin · FileSystem Shell — count · WebHDFS — Get Content Summary

Set a namespace quota

Create HDFS /quota/team-a and set a namespace quota of 10 with hdfs dfsadmin -setQuota 10 /quota/team-a.

The namespace quota is the upper limit on the number of files and directories under that directory (including itself). You see the set value in the first column of hdfs dfs -count -q.

With 10, how many go in

Make several small local files and upload them one at a time to /quota/team-a/ with hdfs dfs -put until one is rejected, and save the error output (standard error) of the rejected put to /root/hdp/quota/quota_err.txt.

The directory itself also counts as one name. So one fewer file than the quota goes in. The rejection message prints the quota and "the count after the attempted addition" together.

Set a space quota

Create HDFS /quota/team-b and set a space quota of 30MB with hdfs dfsadmin -setSpaceQuota 30m /quota/team-b.

30m is 30 × 1024 × 1024 bytes. The space quota is an upper limit on actual disk usage counted including replicas.

One byte is blocked

Create a 1-byte local file /root/hdp/quota/one.txt (printf x > /root/hdp/quota/one.txt) and try uploading it to /quota/team-b/one.txt with hdfs dfs -put. Save the error output to /root/hdp/quota/space_err.txt.

At the moment writing starts, the NameNode reserves one block (block size × replication factor). The default block size of 128MB is larger than the 30MB quota. Compare the two numbers in the message: the quota and the space that was needed.

If you make the block small, it goes in

Upload the same /root/hdp/quota/one.txt to /quota/team-b/one-1m.txt with hdfs dfs -D dfs.blocksize=1048576 -put.

If the block size is 1MiB, the reservation is also 1MiB. When writing finishes, usage is settled at the actual size (1 byte × replication factor). See how much REM_SPACE_QUOTA in -count -q has decreased.

Read the two quotas with count -q

Save the output of hdfs dfs -count -q -v /quota/team-a /quota/team-b to /root/hdp/quota/count.txt.

The columns without a quota are printed as none/inf. For team-a, REM_QUOTA is 0, and for team-b the space is left almost untouched: the rejection was because of "the amount to reserve", not "the remaining amount".

A move is also caught by the quota

Create HDFS /quota/staging, upload one file to /quota/staging/f.txt, then try hdfs dfs -mv /quota/staging/f.txt /quota/team-a/ and save the error output to /root/hdp/quota/mv_err.txt.

A move does not shift bytes, but from team-a's point of view, one more name is added. So the namespace quota rejects it, and the file stays in staging.

So you can answer quota inquiries

In /root/hdp/quota/report.md, write three sections: ## 이름 쿼터, ## 공간 쿼터 and ## 쿼터가 막는 것 (use exactly these Korean headings in this order; they mean "Namespace quota", "Space quota" and "What a quota blocks"). Put the number of files that actually went into team-a in the first section and the number of bytes it tried to reserve in step 4 in the second, as numbers.

Write it as if you were answering the inquiries "the quota is 10, so why do only 9 go in?" and "there's 30MB left, so why can't I write even 1 byte?".