☰

企业官网建设流程全解析

首页 / 建站日记 / 企业官网建设流程全解析
速进！SeaTunnel 2.3.11 用 Docker 部署，实现 Kafka 同步 Hive/ES
2026/7/2 10:25:41 网站建设 项目流程

本文档详细介绍如何使用 Docker 部署 SeaTunnel 2.3.11，并配置 Kafka 虚拟表、数据源以及 Kafka 同步到 Hive 和 Elasticsearch 的完整实战案例。

安装准备

目录结构

seatunnel-docker/├── docker-compose.yml# 主编排文件├── hive/# Hive 配置│ ├── hive-site.xml │ └── lib/# 依赖 jar 包│ └── postgresql-42.5.1.jar ├── init-sql/# 初始化 SQL│ └── seatunnel_server_mysql.sql ├── seatunnel/# SeaTunnel 服务端配置│ ├── Dockerfile │ └── apache-seatunnel-2.3.11/# 解压后的二进制包│ └── lib/# 依赖 jar 包│ ├── hive-exec-3.1.3.jar │ ├── hive-metastore-3.1.3.jar │ ├── libfb303-0.9.3.jar │ ├── mysql-connector-java-8.0.28.jar │ └── seatunnel-hadoop3-3.1.4-uber.jar └── seatunnel-web/# SeaTunnel Web 配置├── Dockerfile └── apache-seatunnel-web-1.0.3-bin/# 解压后的二进制包└── libs/# 依赖 jar 包└── mysql-connector-java-8.0.28.jar

下载 seatunnel

# seatunnel-2.3.11https://dlcdn.apache.org/seatunnel/2.3.11/apache-seatunnel-2.3.11-bin.tar.gz# 源码构建seatunnel-web-1.0.3gitclone https://github.com/apache/seatunnel-web.gitcdseatunnel-webshbuild.sh code

下载依赖包

#hive-metastore容器需要（PostgreSQL为Hive 元数据库）https://jdbc.postgresql.org/download/postgresql-42.5.1.jar#hive-metastore同步报错缺少依赖包（实际验证加前3个包即可）https://repo1.maven.org/maven2/org/apache/hive/hive-exec/3.1.3/hive-exec-3.1.3.jar https://repo1.maven.org/maven2/org/apache/hive/hive-metastore/3.1.3/hive-metastore-3.1.3.jar https://repo.maven.apache.org/maven2/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar https://repo1.maven.org/maven2/org/apache/thrift/libthrift/0.12.0/libthrift-0.12.0.jar https://repo1.maven.org/maven2/org/apache/hive/hive-common/3.1.3/hive-common-3.1.3.jar

创建项目目录

将准备好的相关文件存放 seatunnel-docker 目录

mkdirseatunnel-dockercdseatunnel-docker

Docker 部署

docker-compose.yml 配置

version:'3.9'networks:seatunnel-network:driver:bridgeipam:config:-subnet:172.16.0.0/24services:# ===== Hive 相关服务 =====hive-metastore-db:image:postgres:15container_name:hive-metastore-dbhostname:hive-metastore-dbenvironment:POSTGRES_DB:metastore_dbPOSTGRES_USER:hivePOSTGRES_PASSWORD:hive123456ports:-"5432:5432"volumes:-./hive-metastore-db-data:/var/lib/postgresql/datanetworks:seatunnel-network:ipv4_address:172.16.0.2healthcheck:# 添加健康检查test:["CMD-SHELL","pg_isready -U hive -d metastore_db"]interval:5stimeout:5sretries:10start_period:10shive-metastore:image:apache/hive:4.0.0container_name:hive-metastorehostname:hive-metastoredepends_on:hive-metastore-db:condition:service_healthy# 等待数据库健康后才启动environment:SERVICE_NAME:metastoreDB_DRIVER:postgresSERVICE_OPTS:>--Djavax.jdo.option.ConnectionDriverName=org.postgresql.Driver-Djavax.jdo.option.ConnectionURL=jdbc:postgresql://hive-metastore-db:5432/metastore_db-Djavax.jdo.option.ConnectionUserName=hive-Djavax.jdo.option.ConnectionPassword=hive123456ports:-"9083:9083"volumes:-./hive/lib/postgresql-42.5.1.jar:/opt/hive/lib/postgresql-42.5.1.jar-./hive/hive-site.xml:/opt/hive/conf/hive-site.xml-./hive-warehouse:/opt/hive/data/warehousenetworks:seatunnel-network:ipv4_address:172.16.0.3hive-server2:image:apache/hive:4.0.0container_name:hive-server2hostname:hive-server2depends_on:-hive-metastoreenvironment:HIVE_SERVER2_THRIFT_PORT:10000SERVICE_NAME:hiveserver2IS_RESUME:"true"SERVICE_OPTS:"-Dhive.metastore.uris=thrift://hive-metastore:9083"ports:-"10000:10000"-"10002:10002"volumes:-./hive-warehouse:/opt/hive/data/warehousenetworks:seatunnel-network:ipv4_address:172.16.0.4# ===== MySQL =====mysql-seatunnel:image:mysql:8.0.42container_name:mysql-seatunnelhostname:mysql-seatunnelenvironment:MYSQL_ROOT_PASSWORD:root123456MYSQL_DATABASE:seatunnelMYSQL_ROOT_HOST:'%'ports:-"3806:3306"volumes:-./mysql_data:/var/lib/mysql-./init-sql:/docker-entrypoint-initdb.dnetworks:seatunnel-network:ipv4_address:172.16.0.5command:--default-authentication-plugin=mysql_native_passwordhealthcheck:test:["CMD","mysqladmin","ping","-h","localhost"]interval:10stimeout:5sretries:5# ===== SeaTunnel =====seatunnel-master:build:context:./seatunneldockerfile:Dockerfileimage:seatunnel:2.3.11container_name:seatunnel-masterhostname:seatunnel-masterextra_hosts:-"hive-metastore:172.16.0.3"-"hive-metastore-db:172.16.0.2"environment:-SEATUNNEL_HOME=/opt/seatunnelcommand:>sh -c " cd /opt/seatunnel && exec bin/seatunnel-cluster.sh -r master "ports:-"5801:5801"volumes:-./seatunnel/apache-seatunnel-2.3.11/:/opt/seatunnel/-./logs/master:/opt/seatunnel/logs# [修改点] 挂载 Hive 仓库目录，确保数据写入宿主机共享目录-./hive-warehouse:/opt/hive/data/warehousenetworks:seatunnel-network:ipv4_address:172.16.0.10seatunnel-worker1:image:seatunnel:2.3.11container_name:seatunnel-worker1hostname:seatunnel-worker1extra_hosts:-"hive-metastore:172.16.0.3"-"hive-metastore-db:172.16.0.2"environment:-SEATUNNEL_HOME=/opt/seatunnelcommand:>sh -c " cd /opt/seatunnel && exec bin/seatunnel-cluster.sh -r worker "volumes:-./seatunnel/apache-seatunnel-2.3.11/:/opt/seatunnel/-./logs/worker1:/opt/seatunnel/logs# [修改点] 挂载 Hive 仓库目录，确保数据写入宿主机共享目录-./hive-warehouse:/opt/hive/data/warehousedepends_on:-seatunnel-masternetworks:seatunnel-network:ipv4_address:172.16.0.11seatunnel-worker2:image:seatunnel:2.3.11container_name:seatunnel-worker2hostname:seatunnel-worker2extra_hosts:-"hive-metastore:172.16.0.3"-"hive-metastore-db:172.16.0.2"environment:-SEATUNNEL_HOME=/opt/seatunnelcommand:>sh -c " cd /opt/seatunnel && exec bin/seatunnel-cluster.sh -r worker "volumes:-./seatunnel/apache-seatunnel-2.3.11/:/opt/seatunnel/-./logs/worker2:/opt/seatunnel/logs# [修改点] 挂载 Hive 仓库目录，确保数据写入宿主机共享目录-./hive-warehouse:/opt/hive/data/warehousedepends_on:-seatunnel-masternetworks:seatunnel-network:ipv4_address:172.16.0.12seatunnel-web:build:context:./seatunnel-webdockerfile:Dockerfileimage:seatunnel-web:1.0.3container_name:seatunnel-webhostname:seatunnel-webextra_hosts:-"hive-metastore:172.16.0.3"-"hive-metastore-db:172.16.0.2"environment:-SEATUNNEL_HOME=/opt/seatunnel-SEATUNNEL_WEB_HOME=/opt/seatunnel-webports:-"8801:8801"volumes:-./seatunnel/apache-seatunnel-2.3.11/:/opt/seatunnel/-./seatunnel-web/apache-seatunnel-web-1.0.3-bin/:/opt/seatunnel-web/-./logs/web:/opt/seatunnel-web/logs# [修改点] 挂载 Hive 仓库目录，保持环境一致性-./hive-warehouse:/opt/hive/data/warehousedepends_on:-seatunnel-masternetworks:seatunnel-network:ipv4_address:172.16.0.13

SeaTunnel 配置

Dockerfile

FROM eclipse-temurin:8-jdk-ubi9-minimal WORKDIR /opt/seatunnel/ # 设置环境变量 ENV SEATUNNEL_HOME=/opt/seatunnel ENV PATH=$PATH:$SEATUNNEL_HOME/bin # 暴露端口 EXPOSE 5801 # 启动命令 CMD ["sh", "bin/seatunnel-cluster.sh", "-r", "master"]

hazelcast-client.yaml 客户端配置

编辑seatunnel/apache-seatunnel-2.3.11/config/hazelcast-client.yaml：

hazelcast-client:cluster-name:seatunnelproperties:hazelcast.logging.type:log4j2connection-strategy:connection-retry:cluster-connect-timeout-millis:3000network:cluster-members:-seatunnel-master:5801

hazelcast-master.yaml 配置

编辑seatunnel/apache-seatunnel-2.3.11/config/hazelcast-master.yaml：

hazelcast:cluster-name:seatunnelnetwork:rest-api:enabled:falseendpoint-groups:CLUSTER_WRITE:enabled:trueDATA:enabled:truejoin:tcp-ip:enabled:truemember-list:-seatunnel-master:5801-seatunnel-worker1:5802-seatunnel-worker2:5802port:auto-increment:falseport:5801properties:hazelcast.invocation.max.retry.count:20hazelcast.tcp.join.port.try.count:30hazelcast.logging.type:log4j2hazelcast.operation.generic.thread.count:50hazelcast.heartbeat.failuredetector.type:phi-accrualhazelcast.heartbeat.interval.seconds:2hazelcast.max.no.heartbeat.seconds:180hazelcast.heartbeat.phiaccrual.failuredetector.threshold:10hazelcast.heartbeat.phiaccrual.failuredetector.sample.size:200hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis:100

hazelcast-worker.yaml 配置

编辑seatunnel/apache-seatunnel-2.3.11/config/hazelcast-worker.yaml：

hazelcast:cluster-name:seatunnelnetwork:join:tcp-ip:enabled:truemember-list:-seatunnel-master:5801-seatunnel-worker1:5802-seatunnel-worker2:5802port:auto-increment:falseport:5802properties:hazelcast.invocation.max.retry.count:20hazelcast.tcp.join.port.try.count:30hazelcast.logging.type:log4j2hazelcast.operation.generic.thread.count:50hazelcast.heartbeat.failuredetector.type:phi-accrualhazelcast.heartbeat.interval.seconds:2hazelcast.max.no.heartbeat.seconds:180hazelcast.heartbeat.phiaccrual.failuredetector.threshold:10hazelcast.heartbeat.phiaccrual.failuredetector.sample.size:200hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis:100

安装连接器依赖包

配置同步任务，点击 Source 组件，源名称下拉框没有数据，需要安装依赖才可以显示。

cdseatunnel/apache-seatunnel-2.3.11/shbin/install-plugin.sh

SeaTunnel Web 配置

Dockerfile 配置

FROM eclipse-temurin:8-jdk-ubi9-minimal WORKDIR /opt/seatunnel-web/ # 设置环境变量 ENV SEATUNNEL_WEB_HOME=/opt/seatunnel-web ENV SEATUNNEL_HOME=/opt/seatunnel # 暴露端口 EXPOSE 8801 # 启动命令 CMD ["sh", "bin/seatunnel-backend-daemon.sh", "start"]

application.yml 配置

编辑seatunnel-web/apache-seatunnel-web-1.0.3-bin/conf/application.yml：

server:port:8801spring:main:allow-circular-references:trueapplication:name:seatunneljackson:date-format:yyyy-MM-dd HH:mm:ssdatasource:driver-class-name:com.mysql.cj.jdbc.Driverurl:jdbc:mysql://mysql-seatunnel:3306/seatunnel?useSSL=false&useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&allowPublicKeyRetrieval=trueusername:rootpassword:root123456jwt:expireTime:86400# please add key when deploy 要配置下token 32位secretKey:a3f5c8d2e1b4098765432109abcdef1234567890abcdefalgorithm:HS256

hazelcast-client.yaml 客户端配置

编辑seatunnel-web/apache-seatunnel-web-1.0.3-bin/conf/hazelcast-client.yaml：

hazelcast-client:cluster-name:seatunnelproperties:hazelcast.logging.type:log4j2connection-strategy:connection-retry:cluster-connect-timeout-millis:3000network:cluster-members:-seatunnel-master:5801

seatunnel-backend-daemon.sh

编辑seatunnel-web/apache-seatunnel-web-1.0.3-bin/bin/seatunnel-backend-daemon.sh：
去除后台模式 去掉 nohup 和最后的 &

$JAVA_HOME/bin/java$JAVA_OPTS\-cp"$CLASSPATH"$SPRING_OPTS\org.apache.seatunnel.app.SeatunnelApplication>>"${LOGDIR}/seatunnel.out"2>&1echo"seatunnel-web started"

plugin-mapping.properties 配置

实际验证此步骤可省略。
拷贝 seatunnel/apache-seatunnel-2.3.11/connectors/plugin-mapping.properties 到
seatunnel-web/apache-seatunnel-web-1.0.3-bin/conf/plugin-mapping.properties

cdseatunnel-dockercpseatunnel/apache-seatunnel-2.3.11/connectors/plugin-mapping.properties seatunnel-web/apache-seatunnel-web-1.0.3-bin/conf/plugin-mapping.properties

Hive 配置

hive-site.xml 配置

<?xml version="1.0" encoding="UTF-8"?><configuration><property><name>hive.metastore.uris</name><value>thrift://hive-metastore:9083</value></property><property><name>hive.metastore.warehouse.dir</name><value>/opt/hive/data/warehouse</value></property><property><name>metastore.metastore.event.db.notification.api.auth</name><value>false</value></property></configuration>

lib 目录 依赖包

postgresql-42.5.1.jar

Mysql 配置

init-sql 目录 初始 SQL 脚本

拷贝 seatunnel-web/apache-seatunnel-web-1.0.3-bin/script/seatunnel_server_mysql.sql 到
init-sql/seatunnel_server_mysql.sql

cdseatunnel-dockercpseatunnel-web/apache-seatunnel-web-1.0.3-bin/script/seatunnel_server_mysql.sql init-sql/seatunnel_server_mysql.sql

docker 启动

# 启动所有服务dockercompose up-d--build# 访问web ui页面 默认登录账号：admin / adminopenhttp://localhost:8801

运行示例

登录配置语言

登录页面

设置

配置语言

配置数据源

kafka 数据源

ES 数据源

Hive-metastore 本地数据源

配置为 thrift://hive-metastore:9083 也可以。

配置虚拟表

虚拟表列表

创建虚拟表流程

  1. 进入「虚拟表」菜单，点击「创建」按钮
  2. 选择数据源，配置虚拟表信息
  3. 点击「下一步」配置字段映射
  4. 点击「下一步」确认信息并保存

配置同步任务

kafka-hive 同步任务

任务组件配置

Source 组件配置

FieldMapper 组件配置（模型视图）

Sink 组件配置

Kafka-Elasticsearch 同步任务

任务组件配置

Source 组件配置

FieldMapper 组件配置（模型视图）

Sink 组件配置

创建同步任务通用流程

  1. 进入「任务」→「同步任务定义」，点击「创建」按钮
  2. 拖拽或选择 Source、FieldMapper、Sink 组件构建任务流程
  3. 双击 Source 组件，配置数据源信息（选择已配置的 Kafka 数据源）
  4. 双击 FieldMapper 组件，点击「模型」按钮配置字段映射关系
  5. 双击 Sink 组件，配置目标数据源信息（Hive 或 Elasticsearch）
  6. 保存并启动任务
    需要配置 job mode ，不然保存不了，报错job env can't be empty, please change config

hive 相关操作

创建表

# 进入 HiveServer2 容器dockerexec-ithive-server2 beeline-ujdbc:hive2://localhost:10000-e" CREATE TABLE IF NOT EXISTS default.test_user_data3 ( user_id STRING, type STRING, content STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE; "# 或创建 Parquet 格式（推荐）dockerexec-ithive-server2 beeline-ujdbc:hive2://localhost:10000-e" CREATE TABLE IF NOT EXISTS default.test_user_data3 ( user_id STRING, type STRING, content STRING ) STORED AS PARQUET; "

查看表结构

dockerexec-ithive-server2 beeline-ujdbc:hive2://localhost:10000-e" SHOW TABLES IN default; DESCRIBE default.test_user_data3; "

查询表数据

dockerexec-ithive-server2 beeline-ujdbc:hive2://localhost:10000-e" SELECT * FROM default.test_user_data3 LIMIT 10; "

备注

seatunnel-web 容器启动就退出

排查是否配置

seatunnel-backend-daemon.sh

编辑seatunnel-web/apache-seatunnel-web-1.0.3-bin/bin/seatunnel-backend-daemon.sh：
去除后台模式 去掉 nohup 和最后的 &

$JAVA_HOME/bin/java$JAVA_OPTS\-cp"$CLASSPATH"$SPRING_OPTS\org.apache.seatunnel.app.SeatunnelApplication>>"${LOGDIR}/seatunnel.out"2>&1echo"seatunnel-web started"

seatunnel-web 启动后访问页面报错 Unknown exception. secret key byte array cannot be null or empty

排查 application.yml 是否配置

jwt:expireTime:86400# please add key when deploysecretKey:a3f5c8d2e1b4098765432109abcdef1234567890abcdefalgorithm:HS256

hive 地址解析异常

seatunnel seatunnel-web ERROR[qtp2135089262-20][MetaStoreUtils.logAndThrowMetaException():166]- Got exception: java.net.URISyntaxException Illegal characterinhostnameat index44: thrift://hive-metastore.seatunnel-docker_seatunnel-network:9083

docker-compose.yml 对应容器加上 ip 绑定

extra_hosts:-"hive-metastore:172.16.0.3"-"hive-metastore-db:172.16.0.2"

Hive 同步报错 error java.lang.NoClassDefFoundError

seatunnel/apache-seatunnel-2.3.11/lib 存放依赖包

hive-exec-3.1.3.jar hive-metastore-3.1.3.jar libfb303-0.9.3.jar

hive 同步任务显示成功，实际没有数据写入

docker-compose.yml 对应容器加上 hive 写入本地目录的配置

volumes:# [修改点] 挂载 Hive 仓库目录，确保数据写入宿主机共享目录-./hive-warehouse:/opt/hive/data/warehouse

查看任务执行日志 will be executed on worker

./logs/master/seatunnel-engine-master.log

Task[TaskGroupLocation{jobId=1080750681855361026,pipelineId=1,taskGroupId=2}]will be executed on worker[[seatunnel-worker2]:5801], slotID[2], resourceProfile[ResourceProfile{cpu=CPU{core=0},heapMemory=Memory{bytes=0}}], sequence[db6b679c-67cc-43b8-b64a-acaa85c2a4c0], assigned[1080750681855361026]

作者 | 云婷
原文链接：https://www.cnblogs.com/thao/p/19666609

标签： 网站建设 企业官网 项目流程 UI设计 前端开发

热门文章

文章分类

标签云

网站建设 响应式设计 用户体验 SEO优化 前端开发 小程序 电商平台 性能优化

相关文章

您可能感兴趣的其他内容

3D 裸眼动画制作需要提供哪些素材？大屏项目全流程准备指南
2026/7/2 10:22:28 网站建设

3D 裸眼动画制作需要提供哪些素材？大屏项目全流程准备指南

随着商圈户外 L 型大屏、商场中庭异形 LED 屏、企业展厅、品牌发布会的视觉营销升级&#xff0c;3D 裸眼动画凭借超强的立体视觉冲击力&#xff0c;成为线下引流、品牌曝光的热门宣传形式。 不少市场策划、品牌运营、商场企划负责人在第一次启动裸眼 3D 项目时&#xff0c;常常…...

阅读更多 →
VMware挂起vs恢复：99%的管理员都混淆的7个底层机制差异及性能影响实测数据
2026/7/2 10:21:54 网站建设

VMware挂起vs恢复：99%的管理员都混淆的7个底层机制差异及性能影响实测数据

更多请点击&#xff1a; https://codechina.net 第一章&#xff1a;挂起与恢复的本质定义与核心场景辨析 挂起&#xff08;Suspend&#xff09;与恢复&#xff08;Resume&#xff09;是操作系统内核调度与资源管理中一对互逆的运行时状态迁移操作&#xff0c;其本质并非简单的…...

阅读更多 →
终极Nintendo Switch破解指南：TegraRcmGUI图形化工具完全教程
2026/7/2 10:21:03 网站建设

终极Nintendo Switch破解指南：TegraRcmGUI图形化工具完全教程

终极Nintendo Switch破解指南&#xff1a;TegraRcmGUI图形化工具完全教程 【免费下载链接】TegraRcmGUI C GUI for TegraRcmSmash (Fuse Gele exploit for Nintendo Switch) 项目地址: https://gitcode.com/gh_mirrors/te/TegraRcmGUI 在Nintendo Switch的破解世界中&am…...

阅读更多 →

需要专业的网站建设服务？

联系我们获取免费的网站建设咨询和方案报价，让我们帮助您实现业务目标

立即咨询