技巧:Linux环境下载超大文件-i386文件下载

前言

最近在学校的远程深度学习服务器上跑数据,需要从境外服务器上下载几十G的训练数据。

刚开始尝试用curl下载,下载到一半总是失去连接,直到用上aria2和screen这两个神器。

技巧:Linux环境下载超大文件

centos下面shell环境

技巧:Linux环境下载超大文件

top命令

aria2

  • 安装

# Ubuntu
sudo apt-get install aria2
# Fredora
yum install aria2
# 手动编译
wget https://github.com/aria2/aria2/releases/download/release-1.31.0/aria2-1.31.0.tar.gz
tar xzvf aria2-1.31.0.tar.gz
cd aria2-1.31.0
./configure
make
make install
  • 下载命令:

aria2c -s 5 -x 5 -c http://xxxxx
# [-s 5] 文件分割为5段下载
# [-x 5] 最大连接数5
# [-c] 支持断点续传
# 选项
-s, --split=<N>
Download a file using N connections. If more than N URIs are given, first N URIs are used and
remaining URIs are used for backup. If less than N URIs are given, those URIs are used more
than once so that N connections total are made simultaneously. The number of connections to
the same host is restricted by the --max-connection-per-server option. See also the
--min-split-size option. Default: 5
NOTE:
Some Metalinks regulate the number of servers to connect. aria2 strictly respects them.
This means that if Metalink defines the maxconnections attribute lower than N, then aria2
uses the value of this lower value instead of N.
-c, --continue[=true|false]
Continue downloading a partially downloaded file. Use this option to resume a download
started by a web browser or another program which downloads files sequentially from the begin-
ning. Currently this option is only applicable to HTTP(S)/FTP downloads.
-x, --max-connection-per-server=<NUM>
The maximum number of connections to one server for each download. Default: 1

screen

通过 ssh 登录服务器下载文件,但如果断开登录连接,aria2也会停止运行。

下面给出两种方法解决这个问题:

  • 方法1 nohup

nohup aria2c -s 5 -x 5 -c http://xxxxx > nohup.txt &
# [nohup] 忽略 SIGHUP 信号
# & 后台运行程序
  • 方法2 screen

screen 可以理解为一个虚拟屏幕;

在同一个终端下拥有多块“屏幕”——即“会话”。

#创建一个离线会话
screen -dmS test
#查看会话列表
screen -ls
There is a screen on:
19393.test (Detached)
1 Socket in /var/run/screen/S-root.
#进入会话
screen -r test
# 或者
screen -r 19393
# 执行下载命令
aria2c -s 5 -x 10 -c http://xxxxx
#退出会话
C-a d
#删除会话
screen -d test
#下次进入会话重复上面的步骤即可
#退出会话后,里面运行的程序不会停止

到此,linux环境下面下载大文件就再也不用愁了,哈哈。

推荐阅读