构建dagu和benthos作为调度平台+数据处理框架的镜像并运行

dagu.yaml。

一点见解

644人浏览 · 2024-04-19 00:58:25

一点见解 · 2024-04-19 00:58:25 发布

构建镜像

准备文件

dagu和benthos下载地址(根据版本自行修改)
dagu: https://github.com/dagu-dev/dagu/releases/download/v1.12.11/dagu_1.12.11_linux_amd64.tar.gz
benthos: https://github.com/benthosdev/benthos/releases/download/v4.27.0/benthos_4.27.0_linux_amd64.tar.gz

上传文件到服务器的任意目录

我用$HOME/daguDocker目录

解压并将可执行程序放到指定的目录

tar -xf dagu_1.12.11_linux_amd64.tar.gz
tar -xf benthos_4.27.0_linux_amd64.tar.gz
mv dagu $HOME/daguDocker/
mv benthos $HOME/daguDocker/

创建Dockerfile

FROM debian:12-slim

ENV TZ=Asia/Shanghai
ENV LANG=C.utf8

ENV DAGU_BASICAUTH_USERNAME="admin"
ENV DAGU_BASICAUTH_PASSWORD="admin"
ENV DAGU_AUTHTOKEN="daguApiToken"

ENV BENTHOS_HOME=/opt/etlDir/benthos
ENV DAGU_HOME=/opt/etlDir/dagu
ENV DAGU_HOST=0.0.0.0
ENV DAGU_PORT=8080
ENV DAGU_IS_BASICAUTH=1
ENV DAGU_IS_AUTHTOKEN=1
ENV DAGU_NAVBAR_COLOR="#a0aeba"
ENV DAGU_NAVBAR_TITLE="Dagu"

RUN set -eux; \
        mkdir -p ${DAGU_HOME}; \
        mkdir -p ${BENTHOS_HOME}; \
        apt-get update; \
        apt-get install -y --no-install-recommends ca-certificates; \
        rm -rf /var/lib/apt/lists/*

VOLUME ${DAGU_HOME}/dags ${DAGU_HOME}/data ${DAGU_HOME}/logs ${BENTHOS_HOME}

COPY dagu /usr/local/bin/
COPY benthos /usr/local/bin/

ENTRYPOINT ["dagu"]

EXPOSE 8080
CMD ["start-all"]

构建镜像

docker build -t ctrimages.iisp.net/eusp/dagu:1.12.11 .

运行

在docker上测试运行

docker run --rm -it --privileged=true --name=dagu \
           -p 8088:8080 \
           -v /tmp/etlDir/benthos:/opt/etlDir/benthos \
           -v /tmp/etlDir/dagu/dags:/opt/etlDir/dagu/dags \
           -v /tmp/etlDir/dagu/data:/opt/etlDir/dagu/data \
           -v /tmp/etlDir/dagu/logs:/opt/etlDir/dagu/logs \
           ctrimages.iisp.net/eusp/dagu:1.12.11

注意：生产环境运行的时候应该是后台运行，且不应该把数据卷放在临时目录

使用浏览器访问dagu，需要注意防火墙开放端口

http://192.168.10.216:8088

用户名和密码在Dockerfile中由环境变量指定了

ENV DAGU_BASICAUTH_USERNAME="admin"
ENV DAGU_BASICAUTH_PASSWORD="admin"

在Rancher平台运行

定义`dagu.yaml`文件

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dagu12-pvc
  namespace: eusp-product
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: nfs-client-storageclass
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dagu12-server
  namespace: eusp-product
spec:
  selector:
    matchLabels:
      app: dagu12-server
  template:
    metadata:
      labels:
        app: dagu12-server
    spec:
      containers:
        - name: dagu12-server
          image: ctrimages.iisp.net/eusp/dagu:1.12.11
          imagePullPolicy: Always
          env:
            - name: DAGU_BASICAUTH_PASSWORD
              value: HzjcDagu.com!2
          ports:
            - containerPort: 8080
              name: dagu
              protocol: TCP
          volumeMounts:
            - mountPath: /opt/etlDir/benthos
              name: dagu12-vm
              subPath: benthos
            - mountPath: /opt/etlDir/dagu/dags
              name: dagu12-vm
              subPath: dags
            - mountPath: /opt/etlDir/dagu/data
              name: dagu12-vm
              subPath: data
            - mountPath: /opt/etlDir/dagu/logs
              name: dagu12-vm
              subPath: logs
      volumes:
        - name: dagu12-vm
          persistentVolumeClaim:
            claimName: dagu12-pvc
      imagePullSecrets:
        - name: harbor-secret
---
apiVersion: v1
kind: Service
metadata:
  name: dagu12-server-nodeport
  namespace: eusp-product
spec:
  ports:
    - name: dagu
      nodePort: 31622
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: dagu12-server
  type: NodePort