@cloudcanal-万少 好的，谢谢👍

有上 promethus 监控么？canal server 的监控指标还是能看出写名堂来的。

@Alex 感谢楼主的分享，期待更多canal好文

@navyaijm2017 也就是你手动创建了其他分区就正常了么？

需要不同服务器安装adapter

问题已经解决：原因是集群的配置里面没有指定zk地址，需要注意的是zk地址只能写ip:port，后面不支持写目录

前面server的过滤正则决定了哪些数据被过滤，新增表的数据符合正则的话后面也会收到，理论上不用重启。如果发现ES没收到数据，可以在最近的点打个断点试下

需要在公共配置中加入zk集群地址配置，如：

canal.zkServers = 127.0.0.1:2181,127.0.0.2:2181,127.0.0.3:2181

不设置timestamp只设置了position，canal是根据这个position在binlog文件里面找到最近的一个事务开始同步的，所以实际开始的offset比你自己设置的position靠前是正常的

c98a5a92-36ff-4449-b220-05f8dd360706-image.png

您好，默认的canal是不支持只过滤delete事件这种功能的

@野人 我看你另外开了个提问帖，那边我贴了代码 你可以断点调试下，或者加日志调试下

@野人 可以看看这个类，打下断点，看看数据有没有过来。这个是数据类型做转换前的数据。
21133fa1-c4cb-43ab-a1db-8c71c8d077e4-image.png

@hhxxttxs0826 如果确认配置没问题，看看是不是配置有加载错。对应代码可以参考ESConfigMonitor的init逻辑

能否再提供更加详细一些的信息。源端和目标端是什么，canal版本是什么，怎样的情况产生了这样的报错？

@魑魅 目录叫做es6或者es7，es的yaml 扔到conf/es6/下

@楚天乐 这里发起的请求是dump binlog，可以看看订阅的DB实例的本地binlog file是否正常。show binary logs, show master status都可以查看

根据堆栈信息，应该是某些表的类型解析报错。可以关注下A库和B库订阅的表的表结构上的差异。你的另外个问题我已经回复了如何debug canal，可以参考下：请问如何源码调试canal

@楚天乐 先下载和你使用的版本像对应的源码版本，这样堆栈中的行号才会和你源码中的行号互相匹配。

阅读堆栈，定位DEBUG的类和方法

如果看到异常堆栈，可以确定异常的方法和类。例如如下堆栈，可以看到出问题的方法主要是LogBuffer,RowsLogBuffer和LogEventConvert这几个类。选择哪个类和方法debug来诊断问题是有技巧的，主要选择哪些本地变量有一些语义的地方打印日志是比较好的。针对下面这个堆栈，我们查看相关类和方法的源码之后，可以直接在fetchValue这个方法上打断点，因为这边入参有columnName等信息，可以方便我们确定是哪个列的内容解析时产生了这个问题。可以直接加个try-catch，这样方便有异常的时候才进入断点，或者使用IDEA的条件断点也是可以的。

Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed. Caused by: java.lang.IllegalArgumentException: limit excceed: 258 at com.taobao.tddl.dbsync.binlog.LogBuffer.getInt8(LogBuffer.java:253) at com.taobao.tddl.dbsync.binlog.event.RowsLogBuffer.fetchValue(RowsLogBuffer.java:286) at com.taobao.tddl.dbsync.binlog.event.RowsLogBuffer.nextValue(RowsLogBuffer.java:95) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseOneRow(LogEventConvert.java:409) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEvent(LogEventConvert.java:331) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parse(LogEventConvert.java:101) at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parse(LogEventConvert.java:61) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser.parseAndProfilingIfNecessary(AbstractEventParser.java:322) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3$1.sink(AbstractEventParser.java:175) at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:124) at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:208) at java.lang.Thread.run(Thread.java:748)

d2b67497-f739-4787-9e13-112ee2d521dd-image.png

配置和启动程序

源码中相关的几个properties文件也需要配置好canal.properties和instance.properties等。Server可以通过CanalLauncher的main方法启动，然后等待进入断点

@navyaijm2017 在 canal无法写入kafka多分区 中说：

已解决，正确的配置

mq config

canal.mq.dynamicTopic=db_chj_test.sbtest2
#canal.mq.partition=3
canal.mq.partitionsNum=3
canal.mq.partitionHash=db_chj_test.sbtest2:id

补充下。

canal.mq.partition 这个参数实际上表示的是在 canal.mq.partitionsNum 和 canal.mq.partitionHash 未设置的情况下，默认写入的partition。以下是参考源码

for (FlatMessage flatMessage : flatMessages) { /**canal.mq.partitionsNum和canal.mq.partitionHash **/ if (mqDestination.getPartitionHash() != null && !mqDestination.getPartitionHash().isEmpty()) { FlatMessage[] partitionFlatMessage = MQMessageUtils.messagePartition(flatMessage, mqDestination.getPartitionsNum(), mqDestination.getPartitionHash(), this.mqProperties.isDatabaseHash()); int length = partitionFlatMessage.length; for (int i = 0; i < length; i++) { FlatMessage flatMessagePart = partitionFlatMessage[i]; if (flatMessagePart != null) { records.add(new ProducerRecord<>(topicName, i, null, JSON.toJSONBytes(flatMessagePart, SerializerFeature.WriteMapNullValue))); } } } else { /**canal.mq.partition**/ final int partition = mqDestination.getPartition() != null ? mqDestination.getPartition() : 0; records.add(new ProducerRecord<>(topicName, partition, null, JSON.toJSONBytes(flatMessage, SerializerFeature.WriteMapNullValue))); } }

@junyu-cloudcanal 大佬麻烦看下
2021-03-10 10:05:41.922 [Thread-6] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## stop the canal server
2021-03-10 10:05:41.922 [Thread-7] INFO c.a.o.canal.connector.kafka.producer.CanalKafkaProducer - ## stop the kafka producer
2021-03-10 10:05:41.927 [Thread-7] INFO c.a.o.canal.connector.kafka.producer.CanalKafkaProducer - ## kafka producer is down.
2021-03-10 10:05:43.106 [Thread-6] INFO com.alibaba.otter.canal.deployer.CanalController - ## stop the canal server[172.16..(172.16..**):11111]
2021-03-10 10:05:43.109 [Thread-6] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## canal server is down.
2021-03-10 10:10:42.901 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2021-03-10 10:10:42.947 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2021-03-10 10:10:43.404 [main] WARN org.apache.kafka.clients.producer.ProducerConfig - The configuration ‘kerberos.enable’ was supplied but isn’t a known config.
2021-03-10 10:10:43.406 [main] WARN org.apache.kafka.clients.producer.ProducerConfig - The configuration ‘kerberos.krb5.file’ was supplied but isn’t a known config.
2021-03-10 10:10:43.406 [main] WARN org.apache.kafka.clients.producer.ProducerConfig - The configuration ‘kerberos.jaas.file’ was supplied but isn’t a known config.
2021-03-10 10:10:43.408 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2021-03-10 10:10:43.505 [main-SendThread(jt-xw-test01:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.lang.NoClassDefFoundError: org/apache/log4j/Logger
at org.apache.zookeeper.Login.<init>(Login.java:44) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:198) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:104) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:943) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_141]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_141]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[na:1.8.0_141]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_141]
… 5 common frames omitted
2021-03-10 10:10:44.607 [main-SendThread(jt-xw-test01:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.lang.NoClassDefFoundError: org/apache/log4j/Logger
at org.apache.zookeeper.Login.<init>(Login.java:44) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:198) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:104) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:943) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
2021-03-10 10:10:45.708 [main-SendThread(jt-xw-test01:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.lang.NoClassDefFoundError: org/apache/log4j/Logger
at org.apache.zookeeper.Login.<init>(Login.java:44) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:198) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:104) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:943) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993) ~[zookeeper-3.4.5.jar:3.4.5-1392090]