[ Pobierz całość w formacie PDF ]
.Command-Line ToolsHadoop comes with a number of command-line tools that enable basic filesystem op-erations.Like all Hadoop tools, HDFS commands are subcommands of the hadoopcommand-line utility.Running hadoop fs will display basic usage information, asshown in Example 2-1.Example 2-1.hadoop fs help information[esammer@hadoop01 ~]$ hadoop fsUsage: java FsShell[-ls ][-lsr ][-df []][-du ][-dus ][-count[-q] ][-mv ]20 | Chapter 2: HDFS[-cp ][-rm [-skipTrash] ][-rmr [-skipTrash] ][-expunge][-put.][-copyFromLocal.][-moveFromLocal.][-get [-ignoreCrc] [-crc] ][-getmerge [addnl]][-cat ][-text ][-copyToLocal [-ignoreCrc] [-crc] ][-moveToLocal [-crc] ][-mkdir ][-setrep [-R] [-w] ][-touchz ][-test -[ezd] ][-stat [format] ][-tail [-f] ][-chmod [-R] PATH.][-chown [-R] [OWNER][:[GROUP]] PATH.][-chgrp [-R] GROUP PATH.][-help [cmd]]Most of these commands will be immediately obvious to an administrator with basicshell experience.The major difference is that, because HDFS is a user space filesystem,there s no concept of a current working directory.All paths are either absolute (rec-ommended) or relative to the user s home directory within HDFS.3 An absolute pathcan be of the form /logs/2012/01/25/, or it can include the full URL to specify the loca-tion of the namenode, such as hdfs://mynamenode.mycompany.com:8020/logs/2012/01/25/.If the full URL syntax is not used, the value is taken from the fs.default.nameparameter in the core-site.xml configuration file (see Example 2-2).Example 2-2.Listing files and directories in HDFS[esammer@hadoop01 ~]$ hadoop fs -ls /user/esammerFound 4 itemsdrwx------ - esammer supergroup 0 2012-01-11 15:06 /user/esammer/.staging-rw-r--r-- 3 esammer supergroup 27888890 2012-01-10 13:41 /user/esammer/data.txtdrwxr-xr-x - esammer supergroup 0 2012-01-11 13:08 /user/esammer/teragendrwxr-xr-x - esammer supergroup 0 2012-01-11 15:06 /user/esammer/terasortTo prove to ourselves that the HDFS namespace is entirely separate from the host OS,we can attempt to list the same path using the standard ls command (see Example 2-3).Example 2-3.Attempting to list an HDFS path on the OSesammer@hadoop01 ~]$ ls /user/esammerls: /user/esammer: No such file or directory3.User home directories in HDFS are located in /user/ by default.Access and Integration | 21In many ways, HDFS is more like a remote filesystem than a local OS filesystem.Theact of copying files to or from HDFS is more like SCP or FTP than working with anNFS mounted filesystem, for example.Files are uploaded using either -put or thesynonym -copyFromLocal and are downloaded with -get or -copyToLocal.As a conve-nience, the -moveFromLocal and -moveToLocal commands will copy a file from or toHDFS, respectively, and then remove the source file (see Example 2-4).Example 2-4.Copying files to and from HDFS[esammer@hadoop01 ~]$ hadoop fs -ls /user/esammer/Found 2 itemsdrwx------ - esammer supergroup 0 2012-01-11 15:06 /user/esammer/.staging-rw-r--r-- 3 esammer supergroup 27888890 2012-01-10 13:41 /user/esammer/data.txt[esammer@hadoop01 ~]$ hadoop fs -put /etc/passwd /user/esammer/[esammer@hadoop01 ~]$ hadoop fs -ls /user/esammer/Found 3 itemsdrwx------ - esammer supergroup 0 2012-01-11 15:06 /user/esammer/.staging-rw-r--r-- 3 esammer supergroup 27888890 2012-01-10 13:41 /user/esammer/data.txt-rw-r--r-- 3 esammer supergroup 2216 2012-01-25 21:07 /user/esammer/passwdesammer@hadoop01 ~]$ ls -al passwdls: passwd: No such file or directory[esammer@hadoop01 ~]$ hadoop fs -get /user/esammer/passwd./[esammer@hadoop01 ~]$ ls -al passwd-rw-rw-r--+ 1 esammer esammer 2216 Jan 25 21:17 passwd[esammer@hadoop01 ~]$ hadoop fs -rm /user/esammer/passwdDeleted hdfs://hadoop01.sf.cloudera.com/user/esammer/passwdAlso unique to HDFS is the ability to set the replication factor of a file.This can bedone by using the -setrep command, which takes a replication factor and an optionalflag (-R) to indicate it should operate recursively (see Example 2-5).Example 2-5.Changing the replication factor on files in HDFS[esammer@hadoop01 ~]$ hadoop fs -setrep 5 -R /user/esammer/tmp/Replication 5 set: hdfs://hadoop01.sf.cloudera.com/user/esammer/tmp/aReplication 5 set: hdfs://hadoop01.sf.cloudera.com/user/esammer/tmp/b[esammer@hadoop01 ~]$ hadoop fsck /user/esammer/tmp -files -blocks -locationsFSCK started by esammer from /10.1.1.160 for path /user/esammer/tmp atWed Jan 25 21:57:39 PST 2012/user/esammer/tmp/user/esammer/tmp/a 27888890 bytes, 1 block(s): OK0.blk_2989979708465864046_2985473 len=27888890 repl=5 [10.1.1.162:50010,10.1.1.161:50010, 10.1.1.163:50010, 10.1.1.165:50010, 10.1.1 [ Pobierz całość w formacie PDF ]