大叔问题定位分享(45)hive任务udf函数偶尔报错

大叔问题定位分享(45)hive任务udf函数偶尔报错

背景

在hive sql中执行添加临时udf的操作

add jar hdfs:///user/hive/lib/tools-1.0.jar;
create temporary function decode as 'com.test.etl.Decoder';

在定时任务重偶尔会报错,报错比较随机,报错信息如下:

INFO  : Query ID = hive_20211026010225_545899e7-7afa-4b5c-b7db-fd71565a89c6INFO  : Total jobs = 1INFO  : Launching Job 1 out of 1INFO  : Starting task [Stage-1:MAPRED] in serial modeINFO  : Number of reduce tasks not specified. Estimated from input data size: 197INFO  : In order to change the average load for a reducer (in bytes):INFO  :   set hive.exec.reducers.bytes.per.reducer=INFO  : In order to limit the maximum number of reducers:INFO  :   set hive.exec.reducers.max=INFO  : In order to set a constant number of reducers:INFO  :   set mapreduce.job.reduces=INFO  : Cleaning up the staging area /user/yarn/hive/.staging/job_1629262398375_145140ERROR : Job Submission failed with exception 'java.io.FileNotFoundException(File file:/disk1/hive/download/tools-1.0.jar does not exist)'java.io.FileNotFoundException: File file:/disk1/hive/download/tools-1.0.jar does not exist    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:378)    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329)    at org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:680)    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:313)    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:205)    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:133)    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)    at java.security.AccessController.doPrivileged(Native Method)    at javax.security.auth.Subject.doAs(Subject.java:422)    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571)    at java.security.AccessController.doPrivileged(Native Method)    at javax.security.auth.Subject.doAs(Subject.java:422)    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562)    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:441)    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:146)    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2250)    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1893)    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1613)    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1332)    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1327)    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)    at org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92)    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)    at java.security.AccessController.doPrivileged(Native Method)    at javax.security.auth.Subject.doAs(Subject.java:422)    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)    at java.util.concurrent.FutureTask.run(FutureTask.java:266)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)    at java.lang.Thread.run(Thread.java:748)ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/disk1/hive/download/tools-1.0.jar does not existINFO  : Completed executing command(queryId=hive_20211026010225_545899e7-7afa-4b5c-b7db-fd71565a89c6); Time taken: 0.133 secondsError: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/disk1/hive/download/tools-1.0.jar does not exist (state=08S01,code=1)Closing: 0: jdbc:hive2://master:10000Process exited with an error: 1 (Exit value: 1)

定位

报错是偶发的,没有规律,查看报错的目录,为hive下载目录,配置为:

hive.downloaded.resources.dir=/disk1/hive/download

检查发现集群中很多节点该目录不存在,怀疑是服务器环境初始化有问题,有的节点没有正确初始化。
进一步检查发现同一个节点这个目录,有时存在,有时不存在,排除上边的怀疑。
hive中这个配置的默认值为:

hive.downloaded.resources.dir=${system:java.io.tmpdir}/${hive.session.id}_resources

可见这个目录与session相关,每次session关闭时会做删除,如果配置的路径没有sessionid,则不同session之间会相互影响

DOWNLOADED_RESOURCES_DIR("hive.downloaded.resources.dir",    "${system:java.io.tmpdir}" + File.separator + "${hive.session.id}_resources",    "Temporary local directory for added resources in the remote file system."),

修改配置为:

hive.downloaded.resources.dir=/disk1/hive/download/${hive.session.id}_resources

问题解决。

推荐阅读

    excel怎么用乘法函数

    excel怎么用乘法函数,乘法,函数,哪个,excel乘法函数怎么用?1、首先用鼠标选中要计算的单元格。2、然后选中单元格后点击左上方工具栏的fx公

    excel中乘法函数是什么?

    excel中乘法函数是什么?,乘法,函数,什么,打开表格,在C1单元格中输入“=A1*B1”乘法公式。以此类推到多个单元。1、A1*B1=C1的Excel乘法公式

    标准差excel用什么函数?

    标准差excel用什么函数?,函数,标准,什么,在数据单元格的下方输入l标准差公式函数公式“=STDEVPA(C2:C6)”。按下回车,求出标准公差值。详细

    虾类安装经验分享!

    虾类安装经验分享!,,虾类安装经验分享: 1,机箱的电源光盘应该接近代理价格,而内存和硬盘的价格是在互联网上找到的即时价格。主板套件,视频卡

    cad怎么设置节点|CAD怎么加节点

    cad怎么设置节点|CAD怎么加节点,,1. CAD怎么加节点天正CAD软件画节点的方法:1、在打开的软件中点击左侧工具栏中的图块图案-通用图库。2、

    excel常用函数都有哪些?

    excel常用函数都有哪些?,函数,哪些,常用,1、SUM函数:SUM函数的作用是求和。函数公式为=sum()例如:统计一个单元格区域:=sum(A1:A10)  统计多个

    海龙节点快捷键|海龙工具快捷键

    海龙节点快捷键|海龙工具快捷键,,1. 海龙工具快捷键惠普笔记本电脑的优点:1、 品牌:这年头买任何东西都要涉及这样一个问题,很多人购买东西都