https://zhuanlan.zhihu.com/p/588208157
nerdctl是一个较新的containerd工具,兼容Docker命令行工具,比ctr覆盖更全面,另外还支持docker-compose(不包括swarm)以及一些可选的高级特性
参考 https://github.com/containerd/nerdctl
下载并安装
mkdir -p nerdctl
cd nerdctl
wget https://github.com/containerd/nerdctl/releases/download/v1.0.0/nerdctl-1.0.0-linux-amd64.tar.gz
tar -xf nerdctl-1.0.0-linux-amd64.tar.gz
ls
--------------------------------------
containerd-rootless-setuptool.sh containerd-rootless.sh nerdctl nerdctl-1.0.0-linux-amd64.tar.gz
这里看到有安装containerd的脚本,我们不需要
拷贝nerdctl到系统二进制命令路径下
cp nerdctl /usr/local/bin/
验证版本,为1.0.0,提示buildctl未找到,buildkit安装参见 How:用buildkit和containerd构建镜像
nerdctl version
------------------
WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH
Client:
Version: v1.0.0
OS/Arch: linux/amd64
Git commit: c00780a1f5b905b09812722459c54936c9e070e6
buildctl:
Version:
Server:
containerd:
Version: 1.6.10
GitCommit: 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
看看有什么功能
nerdctl --help
---------------------------------------------------------
nerdctl is a command line interface for containerd
Config file ($NERDCTL_TOML): /etc/nerdctl/nerdctl.toml
Usage: nerdctl [flags]
Management commands:
apparmor Manage AppArmor profiles
builder Manage builds
container Manage containers
image Manage images
ipfs Distributing images on IPFS
namespace Manage containerd namespaces
network Manage networks
system Manage containerd
volume Manage volumes
Commands:
build Build an image from a Dockerfile. Needs buildkitd to be running.
commit Create a new image from a container's changes
completion Generate the autocompletion script for the specified shell
compose Compose
cp Copy files/folders between a running container and the local filesystem.
create Create a new container. Optionally specify "ipfs://" or "ipns://" scheme to pull image from IPFS.
events Get real time events from the server
exec Run a command in a running container
help Help about any command
history Show the history of an image
images List images
info Display system-wide information
inspect Return low-level information on objects.
internal DO NOT EXECUTE MANUALLY
kill Kill one or more running containers
load Load an image from a tar archive or STDIN
login Log in to a container registry
logout Log out from a container registry
logs Fetch the logs of a container. Currently, only containers created with `nerdctl run -d` are supported.
pause Pause all processes within one or more containers
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image from a registry. Optionally specify "ipfs://" or "ipns://" scheme to pull image from IPFS.
push Push an image or a repository to a registry. Optionally specify "ipfs://" or "ipns://" scheme to push image to IPFS.
rename rename a container
restart Restart one or more running containers
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container. Optionally specify "ipfs://" or "ipns://" scheme to pull image from IPFS.
save Save one or more images to a tar archive (streamed to STDOUT by default)
start Start one or more running containers
stats Display a live stream of container(s) resource usage statistics.
stop Stop one or more running containers
tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
top Display the running processes of a container
unpause Unpause all processes within one or more containers
update Update one or more running containers
version Show the nerdctl version information
wait Block until one or more containers stop, then print their exit codes.
Flags:
-H, --H string Alias of --address (default "/run/containerd/containerd.sock")
-a, --a string Alias of --address (default "/run/containerd/containerd.sock")
--address string containerd address, optionally with "unix://" prefix [$CONTAINERD_ADDRESS] (default "/run/containerd/containerd.sock")
--cgroup-manager string Cgroup manager to use ("cgroupfs"|"systemd") (default "systemd")
--cni-netconfpath string cni config directory [$NETCONFPATH] (default "/etc/cni/net.d")
--cni-path string cni plugins binary directory [$CNI_PATH] (default "/opt/cni/bin")
--data-root string Root directory of persistent nerdctl state (managed by nerdctl, not by containerd) (default "/var/lib/nerdctl")
--debug debug mode
--debug-full debug mode (with full output)
--experimental Control experimental: https://github.com/containerd/nerdctl/blob/master/docs/experimental.md [$NERDCTL_EXPERIMENTAL] (default true)
-h, --help help for nerdctl
--host string Alias of --address (default "/run/containerd/containerd.sock")
--hosts-dir strings A directory that contains <HOST:PORT>/hosts.toml (containerd style) or <HOST:PORT>/{ca.cert, cert.pem, key.pem} (docker style) (default [/etc/containerd/certs.d,/etc/docker/certs.d])
--insecure-registry skips verifying HTTPS certs, and allows falling back to plain HTTP
-n, --n string Alias of --namespace (default "default")
--namespace string containerd namespace, such as "moby" for Docker, "k8s.io" for Kubernetes [$CONTAINERD_NAMESPACE] (default "default")
--snapshotter string containerd snapshotter [$CONTAINERD_SNAPSHOTTER] (default "overlayfs")
--storage-driver string Alias of --snapshotter (default "overlayfs")
-v, --version version for nerdctl
Run 'nerdctl COMMAND --help' for more information on a command.
这个和K8s的名字空间不是一回事,其中default就是containerd的默认名字空间,http://k8s.io是K8s的名字空间
nerdctl ns ls
---------------
NAME CONTAINERS IMAGES VOLUMES LABELS
default 0 1 0
k8s.io 43 64 0
moby 90 0 0
在查看http://k8s.io名字空间的镜像时出现了很多空标签的镜像,和ctr查看的结果类似,显示方式更易懂一些
用crictl查看镜像则不会有空标签的镜像
# 查看默认名字空间的镜像,暂时还没有
nerdctl image ls
--------------------
nerdctl image ls
--------------------
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
# 查看k8s.io名字空间的镜像,很多空标签的镜像
nerdctl -n k8s.io image ls
---------------------------------------------------------------------------------------------------------------------------------------
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
calico/apiserver v3.24.5 aabc97c06afa 8 days ago linux/amd64 79.2 MiB 33.9 MiB
calico/apiserver <none> aabc97c06afa 8 days ago linux/amd64 79.2 MiB 33.9 MiB
calico/cni v3.24.5 e282ea2914c8 8 days ago linux/amd64 188.5 MiB 83.4 MiB
calico/cni <none> e282ea2914c8 8 days ago linux/amd64 188.5 MiB 83.4 MiB
calico/kube-controllers v3.24.5 2b6acd7f677f 8 days ago linux/amd64 68.2 MiB 29.7 MiB
calico/kube-controllers <none> 2b6acd7f677f 8 days ago linux/amd64 68.2 MiB 29.7 MiB
......
<none> <none> f6902791fb9a 42 hours ago linux/amd64 125.5 MiB 32.6 MiB
<none> <none> 8e352a029d30 10 days ago linux/amd64 47.0 MiB 14.2 MiB
<none> <none> 5972ad2bcbdc 8 days ago linux/amd64 224.4 MiB 77.8 MiB
<none> <none> 1b1f3456bb19 42 hours ago linux/amd64 63.1 MiB 19.3 MiB
<none> <none> 3d380ca88645 9 days ago linux/amd64 672.0 KiB 294.7 KiB
<none> <none> e282ea2914c8 8 days ago linux/amd64 188.5 MiB 83.4 MiB
<none> <none> 2526315b1c01 10 days ago linux/amd64 115.4 MiB 29.8 MiB
<none> <none> c01fc12b477e 41 hours ago linux/amd64 68.6 MiB 20.1 MiB
<none> <none> 78780437eefc 41 hours ago linux/amd64 63.1 MiB 27.1 MiB
<none> <none> 9330c53feca7 42 hours ago linux/amd64 51.9 MiB 15.1 MiB
<none> <none> 840d5b9fc29f 10 days ago linux/amd64 51.9 MiB 15.1 MiB
可以不写镜像全名,自动补全名称去dockerhub官网下载
nerdctl pull nginx:alpine
--------------------------------------------------------------
docker.io/library/nginx:alpine: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:455c39afebd4d98ef26dd70284aa86e6810b0485af5f4f222b19b89758cabf1e: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:0f2ab24c6aba5d96fcf6e7a736333f26dca1acf5fa8def4c276f6efc7d56251f: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:19dd4d73108a1feefc29d299f3727467ac02486c83474fc3979e4a7637291fe6: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ca7dd9ec2225f2385955c43b2379305acd51543c28cf1d4e94522b3d94cce3ce: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2f12a0e7c01d607251a4040fa41518fd2542f3ebab83a6f7817867d0de111c96: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4342b1ab302e894161372b32fe2976899a978bf8ff2241fb1655dc25e6645a34: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:76a48b0f58980a64d28bc3575ae4733eb337f7b82403559122b13d5e2ced3921: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1a7b9b9bbef6853211515e42f58be7763749950c244a0c485bb4afd1946e06d7: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b704883c57afcf77f6bc48709943bcf808c9e9945d7e04926be41226fa415d33: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 23.8s
# abc.com/debian
FROM debian
RUN apt-get install -y --force-yes locales
RUN echo "LC_ALL=\"zh_CN.UTF-8\"" >> /etc/default/locale
RUN locale-gen "zh_CN.UTF-8"
执行以下指令,我们可以在本地default名字空间下得到镜像http://abc.com/debian
nerdctl build -t abc.com/debian .
2. 假设我们要基于前面生成的本地镜像http://abc.com/debian,构建一个新镜像http://abc.com/nginx,Dockerfile文件如下
# abc.com/nginx
FROM abc.com/debian
RUN apt-get install -y --force-yes nginx
执行以下指令构建新镜像http://abc.com/nginx
nerdctl build -t abc.com/nginx .
我们会得到一个错误。因为nerdctl会根据Dockerfie里FROM参数指定的镜像的域名去网上找这个镜像http://abc.com/debian,而并不会像Docker一样在本地看到已经有这个镜像就直接使用它。
如果nerdctl找到了网络上的镜像http://abc.com/debian,和本地同名镜像校验无误之后,它才会直接使用本地的镜像来构建新镜像。
这样安全性要高一点,因为构建镜像时引用的其他镜像名都必须具有唯一性,这不就是当年发明DNS的意义吗?
3. 上传镜像
所以我们有必要先把http://abc.com/debian镜像上传了
nerdctl push abc.com/debian
4. 再次构建镜像http://abc.com/nginx
nerdctl build -t abc.com/nginx .
成功了
和Docker自己处理网络不一样,containerd是依赖cni插件来处理网络的,如果cni有问题,那么启动的容器用到了网络就会报错。
前文我们下载了最新的cni插件,覆盖了之前可能是安装K8s时带的cni插件。
nerdctl network ls
---------------------------------------------------------------------------
NETWORK ID NAME FILE
k8s-pod-network /etc/cni/net.d/10-calico.conflist
17f29b073143 bridge /etc/cni/net.d/nerdctl-bridge.conflist
host
none
我们看到nerdctl不知何时自动创建了一个网络10.4.0.0/24
cat /etc/cni/net.d/nerdctl-bridge.conflist
----------------------------------------------
{
"cniVersion": "1.0.0",
"name": "bridge",
"nerdctlID": "17f29b073143d8cd97b5bbe492bdeffec1c5fee55cc1fe2112c8b9335f8b6121",
"nerdctlLabels": {},
"plugins": [
{
"type": "bridge",
"bridge": "nerdctl0",
"isGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"ranges": [
[
{
"gateway": "10.4.0.1",
"subnet": "10.4.0.0/24"
}
]
],
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"type": "host-local"
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
},
{
"type": "firewall",
"ingressPolicy": "same-bridge"
},
{
"type": "tuning"
}
]
nerdctl run -d -p 80:80 --name web nginx:alpine
------------------------------------------------------------------
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2022-11-30T22:49:05+08:00" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"
Failed to write to log, write /var/lib/nerdctl/1935db59/containers/default/678d8ce549913b4af2301babb46a18da9afc874d76a9c9d62c363e92e6c94dac/oci-hook.createRuntime.log: file already closed: unknown
报错了,可能是之前K8s安装时带的cni插件bridge版本太低了,下载一个最新的插件来覆盖掉现在的
参考 https://github.com/containernetworking/plugins ,下载cni插件,把原来的cni执行备份,把新的cni覆盖过去
mkdir -p /tmp/cni
cd /tmp/cni
wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
tar -xf cni-plugins-linux-amd64-v1.1.1.tgz
cp -r /opt/cni/bin /opt/cni/bin_bak
cp * /opt/cni/bin
rm /opt/cni/bin/cni-plugins-linux-amd64-v1.1.1.tgz
删除旧容器再次运行容器
nerdctl container rm web
----------------------------
web
nerdctl run -d -p 80:80 --name web nginx:alpine
---------------------------------------------------------------------
8da324c443b2271a3a1d1bf6afc782e387bc455df59ba7f779c423c5da3727ee
访问 http://127.0.0.1 成功看到nginx首页
上文面我们成功的启动成功并在本机上正常访问,但是我们发现从别的地方无法访问到这个地址,确切地说,Docker的容器都还是好好的,Containerd的容器都出不了本机。原因是Docker和containerd/nerdctl的cni都创建了一些iptables rule,好像它们之间是不能完美共存的。
汉贼不两立,王业不偏安,所以我们就把Docker删了,有nerdctl,还要什么自行车。
查看容器列表,-a参数表示把已经停止的容器也列出来,这里我们看到了一个之前ctr创建现已停止的容器
nerdctl container ls -a
---------------------------------------------------------------------
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8da324c443b2 docker.io/library/nginx:alpine "/docker-entrypoint.…" 31 seconds ago Up 0.0.0.0:80->80/tcp web
停止容器
nerdctl container stop web
-----------------------------
web
启动容器,把之前停止的容器启动
nerdctl container start web
------------------------------------------------------------------------------------------------
web
删除容器,必须先停止才能删除
nerdctl container stop web
nerdctl container rm web
杀死容器,如果容器启动时带了restart=always参数,则需要强制杀掉容器
nerdctl container kill web
查看系统信息
nerdctl system info
---------------------------
Client:
Namespace: default
Debug Mode: false
Server:
Server Version: 1.6.10
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.10.0-19-amd64
Operating System: Debian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.784GiB
Name: test1
ID: 1470b95e-69e8-4d88-9319-72209545ab31
清理无用数据,这个指令十分可怕,它不是和Docker那样只是把标签为"none"的镜像清理掉,而是把所有没有"正在使用"的镜像清理了,比如我用来编译的mave和node镜像。至于清理volumes这个指令,我暂时还没有勇气执行它。
nerdctl system prune -h
--------------------------
Remove unused data
Usage: nerdctl system prune [flags]
Flags:
-a, --all Remove all unused images, not just dangling ones
-f, --force Do not prompt for confirmation
-h, --help help for prune
--volumes Prune volumes
See also 'nerdctl --help' for the global flags such as '--namespace', '--snapshotter', and '--cgroup-manager'.
因为用了docker好多年,所以我们要做的是把docker-compose服务迁移到nerdctl上来运行
找一个之前运行的docker-compose服务jenkins,为了便于理解,先亮出docker-compose.yml文件
# docker-compose.yml
version: '3.1'
services:
jenkins:
build:
context: .
dockerfile: Dockerfile
image: local/jenkins
ports:
- 83:8080
- 50000:50000
volumes:
- jenkins_home:/var/jenkins_home
- /root/.ssh:/var/jenkins_home/.ssh
environment:
JAVA_OPTS: '-Xmx512m -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=60'
restart: always
volumes:
jenkins_home:
因为这个docker-compose服务里用到的镜像是要在启动之前构建的,所以还需要一个Dockerfile文件
# Dockerfile
# local/jenkins
FROM jenkins/jenkins:2.286
USER root
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& echo 'Asia/Shanghai' >/etc/timezone
USER jenkins
因为启动过程中要构建镜像,所以需要事先安装buildkit工具,参见 How:用buildkit和containerd构建镜像
现在用nerdctl启动docker-compose服务
cd jenkins
docker-compose down
nerdctl compose -f docker-compose.yml up -d
然后查看服务,启动正常
nerdctl compose -f docker-compose.yml ps
----------------------------------------
NAME COMMAND SERVICE STATUS PORTS
jenkins_jenkins_1 "/sbin/tini -- /usr/…" jenkins running 0.0.0.0:83->8080/tcp, 0.0.0.0:50000->50000/tcp
用浏览器访问jenkins时我们发现出现了初次安装jenkins的提示,这是因为docker-compose里创建的volume位置和nerdctl创建的volume位置不一样,我们还需要迁移一下volume数据
# 停止jenkins服务
nerdctl compose -f docker-compose.yml down
# 查看nerdctl为jenkins服务创建的volume位置
nerdctl volume ls
-----------------------------------------------
VOLUME NAME DIRECTORY
jenkins_jenkins_home /var/lib/nerdctl/1234db59/volumes/default/jenkins_jenkins_home/_data
# 查看docker-compose为jenkins服务创建的volume位置
docker volume inspect jenkins_jenkins_home
--------------------------------------------
[
{
"CreatedAt": "2022-11-30T23:04:19+08:00",
"Driver": "local",
"Labels": {
"com.docker.compose.project": "jenkins",
"com.docker.compose.version": "1.27.4",
"com.docker.compose.volume": "jenkins_home"
},
"Mountpoint": "/var/lib/docker/volumes/jenkins_jenkins_home/_data",
"Name": "jenkins_jenkins_home",
"Options": null,
"Scope": "local"
}
]
# 完全删除nerdctl创建的volume的目录里的文件,包含隐藏文件
cd /var/lib/nerdctl/1234db59/volumes/default/jenkins_jenkins_home/_data
rm -rf *
rm .*
ls -al
# 拷贝docker创建的jenkins的volume到nerdctl创建的volume
rsync -avz --progress --delete /var/lib/docker/volumes/jenkins_jenkins_home/_data/ /var/lib/nerdctl/1234db59/volumes/default/jenkins_jenkins_home/_data
再次启动jenkins服务,然后浏览器访问,成功了
nerdctl compose -f docker-compose.yml up -d
发现一个和Docker不一样的地方,jenkins服务在运行时构建的镜像local/jenkins本来是基于jenkins/jenkins:2.286镜像的,Docker的做法是把jenkins/jenkins:2.286镜像也保存下来了,而nerdctl默认并不保存jenkins/jenkins:2.286镜像。
升级到nerdctl 1.1.0,首先下载并解压
cd ~/nerdctl
wget https://github.com/containerd/nerdctl/releases/download/v1.0.0/nerdctl-1.0.0-linux-amd64.tar.gz
tar -xf nerdctl-1.0.0-linux-amd64.tar.gz
然后去把正在执行的nerdctl进程都停下来,类似以下操作
# 查看正在运行的容器
nerdctl container ls
# 停止jenkins服务以及其他服务
cd jenkins
nerdctl compose -f docker-compose.yml down
。。。。。。
拷贝nerdctl到系统二进制命令路径下
cp nerdctl /usr/local/bin/
重新启动nerdctl控制的服务
# 启动jenkins服务以及其他服务
cd jenkins
nerdctl compose -f docker-compose.yml up -d
。。。。。。
已发现nerdctl 1.0.0不支持docker-compose.yml里面的shm_size 配置,重启后shm仍是系统默认的64M大小。
暂时的解决办法是,使用 nerdctl run --shm-size 来运行容器。
升级到nerdctl 1.1.0之后,此问题仍未解决。