rsync文件到hadoop

我有6台服务器,每台都包含很多日志.我想通过rsync将这些日志放到hadoop fs上.现在我正在使用fuse和rsync直接写入熔丝安装的fs / mnt / hdfs.
但是有一个大问题.大约一天后,保险丝守护程序占用5 GB的RAM,并且无法对安装的fs做任何事情.所以我必须重新安装保险丝,一切都很好,但只是一段时间.
Rsync命令是

rsync --port=3360 -az --timeout=10 --contimeout=30 server_name::ap-rsync/archive /mnt/hdfs/logs

一段时间后,Rsync会生成错误消息：

rsync error: timeout in data send/receive (code 30) at io.c(137) [sender=3.0.7]
rsync: connection unexpectedly closed (498784 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [receiver=3.0.7]
rsync: connection unexpectedly closed (498658 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7]

解决方法

> patch fuse-hdfs,它可以在hadoop中找到.

https://issues.apache.org/jira/browse/HDFS-861
> patch rsync(版本3.0.8).

diff -r rsync-3.0.8.no_excl/syscall.c rsync-3.0.8/syscall.c

234a235,252
> #if defined HAVE_SECURE_MKSTEMP && defined HAVE_FCHMOD && (!defined HAVE_OPEN64 || defined HAVE_MKSTEMP64)
>   {
>       int fd = mkstemp(template);
>       if (fd == -1)
>           return -1;
>       if (fchmod(fd,perms) != 0 && preserve_perms) {
>           int errno_save = errno;
>           close(fd);
>           unlink(template);
>           errno = errno_save;
>           return -1;
>       }
> #if defined HAVE_SETMODE && O_BINARY
>       setmode(fd,O_BINARY);
> #endif
>       return fd;
>   }
> #else
237c255,256
<   return do_open(template,O_WRONLY|O_CREAT,perms);
---
>   return do_open(template,O_RDWR|O_EXCL|O_CREAT,perms);
> #endif

（编辑：东莞站长网）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!