Cannot start ( activating) etcd.service

[jetchan88]

etcd in alidbsdev01 and 02 is working fine. but alidbsdev03 etcd process exited with error code. Please help
root@alidbsdev03:/var/lib/etcd/member/snap# systemctl start etcd.service
Job for etcd.service failed because the control process exited with error code.
See "systemctl status etcd.service" and "journalctl -xeu etcd.service" for details.
root@alidbsdev03:/var/lib/etcd/member/snap# systemctl status etcd.service
● etcd.service - Etcd Server
Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
Active: activating (start) since Fri 2025-01-10 17:36:07 HKT; 570ms ago
Main PID: 1960757 (etcd)
Tasks: 11 (limit: 75372)
Memory: 10.1M
CPU: 773ms
CGroup: /system.slice/etcd.service
└─1960757 /usr/local/bin/etcd

Jan 10 17:36:07 alidbsdev03 bash[1960757]: {"level":"info","ts":"2025-01-10T17:36:07.904337+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"alidbsdev03"}
Jan 10 17:36:07 alidbsdev03 bash[1960757]: {"level":"warn","ts":"2025-01-10T17:36:07.904593+0800","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}
Jan 10 17:36:07 alidbsdev03 bash[1960757]: {"level":"info","ts":"2025-01-10T17:36:07.90465+0800","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/usr/local/bin/etcd"]}
Jan 10 17:36:07 alidbsdev03 bash[1960757]: {"level":"warn","ts":"2025-01-10T17:36:07.9048+0800","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"membernew","data-dir":"/var/lib/etcd"}
Jan 10 17:36:07 alidbsdev03 bash[1960757]: {"level":"info","ts":"2025-01-10T17:36:07.905171+0800","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"}
Jan 10 17:36:07 alidbsdev03 bash[1960757]: {"level":"warn","ts":"2025-01-10T17:36:07.905208+0800","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}
Jan 10 17:36:07 alidbsdev03 bash[1960757]: {"level":"info","ts":"2025-01-10T17:36:07.905226+0800","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://192.168.172.70:2380"]}
Jan 10 17:36:07 alidbsdev03 bash[1960757]: {"level":"info","ts":"2025-01-10T17:36:07.905441+0800","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2379","http://192.168.172.70:237>
Jan 10 17:36:07 alidbsdev03 bash[1960757]: {"level":"info","ts":"2025-01-10T17:36:07.905574+0800","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha":"bdbbde998","go-version":"go1.19.9","go-os":>
Jan 10 17:36:08 alidbsdev03 bash[1960757]: {"level":"info","ts":"2025-01-10T17:36:08.227484+0800","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"321.696358ms"}

[jetchan88]

[Jan 09 11:36:19 alidbsdev03 bash[193477]: {"level":"info","ts":"2025-01-](panic: tocommit(1979841) is out of range [lastIndex(1920809)]. Was the raft log corrupted, truncated, or lost?)

09T11:36:19.833889+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"443a781955c3c91c became follower at term 35"}
Jan 09 11:36:19 alidbsdev03 bash[193477]: {"level":"info","ts":"2025-01-09T11:36:19.834096+0800","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"443a781955c3c91c","to":"8b660369ed6ac371","stream-type":"stream MsgAp>
Jan 09 11:36:19 alidbsdev03 bash[193477]: {"level":"info","ts":"2025-01-09T11:36:19.834125+0800","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","lo>
Jan 09 11:36:19 alidbsdev03 bash[193477]: {"level":"panic","ts":"2025-01-09T11:36:19.834082+0800","logger":"raft","caller":"etcdserver/zap_raft.go:101","msg":"tocommit(1979841) is out of range [lastIndex(1920809)]. Was the raft log corr>
Jan 09 11:36:19 alidbsdev03 bash[193477]: panic: tocommit(1979841) is out of range [lastIndex(1920809)]. Was the raft log corrupted, truncated, or lost?
Jan 09 11:36:19 alidbsdev03 bash[193477]: goroutine 124 [running]:
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000142780, {0x0, 0x0, 0x0})
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.uber.org/[email protected]/zapcore/entry.go:234 +0x49b
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.uber.org/zap.(*SugaredLogger).log(0xc00068e020, 0x4, {0x10dbaab?, 0x40b3c5?}, {0xc001acbf80?, 0xee3dc0?, 0xc000112a00?}, {0x0, 0x0, 0x0})
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.uber.org/[email protected]/sugar.go:227 +0xee
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.uber.org/zap.(*SugaredLogger).Panicf(...)
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.uber.org/[email protected]/sugar.go:159
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/server/v3/etcdserver.(*zapRaftLogger).Panicf(0x1d4f29?, {0x10dbaab?, 0xc00012c5c0?}, {0xc001acbf80?, 0x555492?, 0x47677e?})
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/server/v3/etcdserver/zap_raft.go:101 +0x49
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/v3.(*raftLog).commitTo(0xc00025f5e0, 0x1e35c1)
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/[email protected]/log.go:237 +0x103
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/v3.(*raft).handleHeartbeat(, {0x8, 0x443a781955c3c91c, 0x8b660369ed6ac371, 0x23, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/[email protected]/raft.go:1508 +0x45
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/v3.stepFollower(, {0x8, 0x443a781955c3c91c, 0x8b660369ed6ac371, 0x23, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/[email protected]/raft.go:1434 +0x3d8
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/v3.(*raft).Step(_, {0x8, 0x443a781955c3c91c, 0x8b660369ed6ac371, 0x23, 0x0, 0x0, {0x0, 0x0, 0x0}, ...})
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/[email protected]/raft.go:975 +0x137a
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/v3.(*node).run(0xc000501b60)
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/[email protected]/node.go:356 +0x965
Jan 09 11:36:19 alidbsdev03 bash[193477]: created by go.etcd.io/etcd/raft/v3.RestartNode
Jan 09 11:36:19 alidbsdev03 bash[193477]: go.etcd.io/etcd/raft/[email protected]/node.go:244 +0x24a
Jan 09 11:36:19 alidbsdev03 systemd[1]: etcd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 09 11:36:19 alidbsdev03 systemd[1]: etcd.service: Failed with result 'exit-code'.
Jan 09 11:36:19 alidbsdev03 systemd[1]: Failed to start Etcd Server.
Jan 09 11:36:19 alidbsdev03 systemd[1]: etcd.service: Consumed 4.353s CPU time.
Jan 09 11:36:20 alidbsdev03 systemd[1]: etcd.service: Scheduled restart job, restart counter is at 1.
Jan 09 11:36:20 alidbsdev03 systemd[1]: Stopped Etcd Server.
Jan 09 11:36:20 alidbsdev03 systemd[1]: etcd.service: Consumed 4.353s CPU time.
Jan 09 11:36:20 alidbsdev03 systemd[1]: Starting Etcd Server...
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.186914+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value">
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187371+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/var/lib/et>
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.18739+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ELECTION_TIMEOUT","variable-value":"5000>
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187426+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_HEARTBEAT_INTERVAL","variable-value":"1>
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187445+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable->
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187452+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"alid>
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187461+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value">
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.18747+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":>
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187479+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ELECTION_TICK_ADVANCE","variabl>
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187495+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"h>
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187504+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"htt>
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187515+0800","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"alidbsdev03"}
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"warn","ts":"2025-01-09T11:36:20.187606+0800","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187644+0800","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/usr/local/bin/etcd"]}
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"warn","ts":"2025-01-09T11:36:20.187706+0800","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":".bash_logout","data-dir":"/var/lib/etcd"}
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"warn","ts":"2025-01-09T11:36:20.187714+0800","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":".bashrc","data-dir":"/var/lib/etcd"}
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"warn","ts":"2025-01-09T11:36:20.187719+0800","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":".profile","data-dir":"/var/lib/etcd"}
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187725+0800","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"}
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"warn","ts":"2025-01-09T11:36:20.187737+0800","caller":"embed/config.go:673","msg":"Running http and grpc server on single port. This is not recommended for production."}
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187745+0800","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://192.168.172.70:2380"]}
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187896+0800","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2379","http://192.168.172.70:2379>
Jan 09 11:36:20 alidbsdev03 bash[193580]: {"level":"info","ts":"2025-01-09T11:36:20.187978+0800","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha":"bdbbde998","go-version":"go1.19.9","go-os":">