Apache Hadoop — Stand up and run HDFS and YARN in one pod

Open the finance folder to one team only and give someone else read-only access

Goal

You lock the finance folder so that only alice and the finance group can use it, see bob get rejected, and then open read-only access to bob with an ACL. You make that permission carry over to new files with a default ACL, and test the sticky bit, which keeps people from deleting other people's files in a folder everyone writes to. Finally, you sum up what simple authentication trusts.

Why it matters

HDFS's permission model is almost the same as POSIX: every file and directory has an owner, a group and permission bits, and to look inside a directory you need that directory's execute (x) permission. The difference is who decides who is who. With the default setting (simple authentication), the NameNode trusts the user name the client sends as is. With a single line, HADOOP_USER_NAME=root, anyone is the superuser. That is also why this lab can test by switching users. So permission bits and ACLs are "a fence to keep people from accidentally touching other people's data", not a security boundary. If you need a security boundary, you have to enforce authentication with Kerberos. Permissions as a fence are still important. You divide folders by team, open to other teams only as much as needed with ACLs, and set the sticky bit on the temporary folder everyone uses so that people cannot delete each other's files. If you do not set a default ACL, permissions do not carry over to newly created files, and the inquiry comes in: "it worked yesterday, but it doesn't work on today's file".

Steps

As root, create /proj/finance, and set the owner to alice:finance and the permissions to 750.
With HADOOP_USER_NAME=alice, upload /data/finance/q1.csv to /proj/finance/.
With HADOOP_USER_NAME=bob, try -cat on /proj/finance/q1.csv and save the error output to /root/hdp/perm/bob_denied.txt.
As alice, give bob ACLs: user:bob:r-x on /proj/finance and user:bob:r-- on /proj/finance/q1.csv.
As bob, read q1.csv, count the lines and write the count as an integer to /root/hdp/perm/bob_lines.txt.
As alice, set the default ACL default:user:bob:r-- on /proj/finance, and upload /data/finance/q2.csv to /proj/finance/. The new file should inherit bob's ACL.
As root, create /proj/shared with 1777 (the sticky bit), upload /data/finance/q1.csv as alice to /proj/shared/alice-q1.csv, then as bob try -rm -skipTrash on it and save the error output to /root/hdp/perm/sticky.txt.
In /root/hdp/perm/report.md, write three sections: ## 소유자와 권한, ## ACL and ## 단순 인증의 한계 (use exactly these Korean headings in this order; they mean "Owner and permissions", "ACL" and "Limits of simple authentication"). Put the line count from step 5 in the second section, and HADOOP_USER_NAME and Kerberos in the third.

Notes

The NameNode in this Pod is running as root, so root is the HDFS superuser. If you leave HADOOP_USER_NAME empty, you are root.
The NameNode looks up a user's groups in its own operating system and decides. This Pod has no alice or bob accounts, so neither has a group: the owning group (finance) is used only as a label.
ACLs are set with hdfs dfs -setfacl -m <항목> <경로> (the placeholders are the entry and the path), and checked with -getfacl. A directory's default ACL (starting with default:) carries over only to files newly created under it.
Common mistakes: giving an ACL only to the file and forgetting the directory's execute permission; thinking a default ACL applies retroactively to files that already exist; and giving the sticky bit wrongly as 777 (it should be 1777).
Official documentation: HDFS Permissions Guide · FileSystem Shell — setfacl · Hadoop in Secure Mode · WebHDFS — ACL Operations

Lock the team folder

As root, create HDFS /proj/finance and apply hdfs dfs -chown alice:finance and hdfs dfs -chmod 750.

750 is owner rwx, group r-x, others ---. Only the superuser can change the owner with chown, so you do it as root.

Upload as alice

Have alice upload the file with HADOOP_USER_NAME=alice hdfs dfs -put /data/finance/q1.csv /proj/finance/.

With simple authentication, the user changes with one environment variable. The owner of the new file is the uploading user, and the group follows the parent directory's group (the BSD rule). Check with -ls.

bob is rejected

Run HADOOP_USER_NAME=bob hdfs dfs -cat /proj/finance/q1.csv and save the standard error output to /root/hdp/perm/bob_denied.txt.

The rejection message contains who (user=), what they tried to do (access=), and at which inode they were blocked (inode=). Notice that it is blocked not at reading the file but at the directory's execute permission: if you cannot enter, you cannot see the files inside.

Open it to one person with an ACL

As alice, give the ACL user:bob:r-x on /proj/finance and user:bob:r-- on /proj/finance/q1.csv (hdfs dfs -setfacl -m).

Permission bits can express only three: owner, group and others. An ACL adds named users and groups. A directory needs execute (x) for you to reach the files inside. Changing an ACL is the job of the owner (or the superuser).

Now bob can read

With HADOOP_USER_NAME=bob, -cat /proj/finance/q1.csv, count the lines and write the count as an integer to /root/hdp/perm/bob_lines.txt.

The permission check on a file with an ACL looks at the named-user entry and the mask together. The #effective: marker of -getfacl is the permission that actually applies.

Default ACL: carry it over to new files too

As alice, set default:user:bob:r-- on /proj/finance, and with HADOOP_USER_NAME=alice, upload /data/finance/q2.csv to /proj/finance/. The new file q2.csv must have user:bob:r-- carried over.

A default ACL is set only on directories and is copied to things newly created under it. It is not applied retroactively to the existing q1.csv. See with -getfacl that once you set a default ACL, the other (others) permissions of the new file also follow the default ACL.

The sticky bit: protect other people's files in a folder everyone uses

As root, create /proj/shared and set it with hdfs dfs -chmod 1777, upload /data/finance/q1.csv as alice to /proj/shared/alice-q1.csv, and then save the error output of HADOOP_USER_NAME=bob hdfs dfs -rm -skipTrash /proj/shared/alice-q1.csv to /root/hdp/perm/sticky.txt.

In a directory with 777, anyone can delete anyone's file (because deleting depends on the directory's write permission). With the sticky bit, only the file's owner and the directory's owner can delete. This is why /tmp is set up that way.

Record the fence and the lock separately

In /root/hdp/perm/report.md, write three sections: ## 소유자와 권한, ## ACL and ## 단순 인증의 한계 (use exactly these Korean headings in this order; they mean "Owner and permissions", "ACL" and "Limits of simple authentication"). Put the line count from step 5 in the second section, and HADOOP_USER_NAME and Kerberos in the third.

The third section is the key. Write what the way you switched users in this lab means in production, and therefore what a production cluster needs.