MongoDB | 2634916

Info

I have 5 node replicaset. where host-0 is the primary, host 1 to 4 are secondary. I made the host-4 & host-4 down for some maintenance work. The rs.status() was look like : ``` host-0 -> primary host-1, host-2 -> secondary host-3, host-4 -> not reachable/healthy ``` Then my rs-2 pod gets down for a connection issue, & after that I see there is no primary in the replicaset. Even the previous primary host-0 converts to secondary !! I have no idea why. Here are the member found from `rs.status()` output members: [ { _id: 0, name: 'mg-rs-0.mg-rs-pods.demo.svc.cluster.local:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 1281, optime: { ts: Timestamp( { t: 1712581101, i: 1 } ), t: Long('3') }, optimeDate: ISODate('2024-04-08T12:58:21.000Z'), lastAppliedWallTime: ISODate('2024-04-08T12:58:21.498Z'), lastDurableWallTime: ISODate('2024-04-08T12:58:21.498Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: '', configVersion: 9, configTerm: 3, self: true, lastHeartbeatMessage: '' }, { _id: 1, name: 'mg-rs-1.mg-rs-pods.demo.svc.cluster.local:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 1219, optime: { ts: Timestamp( { t: 1712581101, i: 1 } ), t: Long('3') }, optimeDurable: { ts: Timestamp( { t: 1712581101, i: 1 } ), t: Long('3') }, optimeDate: ISODate('2024-04-08T12:58:21.000Z'), optimeDurableDate: ISODate('2024-04-08T12:58:21.000Z'), lastAppliedWallTime: ISODate('2024-04-08T12:58:21.498Z'), lastDurableWallTime: ISODate('2024-04-08T12:58:21.498Z'), lastHeartbeat: ISODate('2024-04-08T12:58:38.514Z'), lastHeartbeatRecv: ISODate('2024-04-08T12:58:38.507Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: '', syncSourceId: -1, infoMessage: '', configVersion: 9, configTerm: 3 }, { _id: 2, name: 'mg-rs-2.mg-rs-pods.demo.svc.cluster.local:27017', health: 0, state: 8, stateStr: '(not reachable/healthy)', uptime: 0, optime: { ts: Timestamp( { t: 0, i: 0 } ), t: Long('-1') }, optimeDurable: { ts: Timestamp( { t: 0, i: 0 } ), t: Long('-1') }, optimeDate: ISODate('1970-01-01T00:00:00.000Z'), optimeDurableDate: ISODate('1970-01-01T00:00:00.000Z'), lastAppliedWallTime: ISODate('2024-04-08T12:58:04.600Z'), lastDurableWallTime: ISODate('2024-04-08T12:58:04.600Z'), lastHeartbeat: ISODate('2024-04-08T12:58:38.163Z'), lastHeartbeatRecv: ISODate('2024-04-08T12:58:10.511Z'), pingMs: Long('0'), lastHeartbeatMessage: 'Error connecting to mg-rs-2.mg-rs-pods.demo.svc.cluster.local:27017 :: caused by :: Could not find address for mg-rs-2.mg-rs-pods.demo.svc.cluster.local:27017: SocketException: Host not found (authoritative)', syncSourceHost: '', syncSourceId: -1, infoMessage: '', configVersion: 9, configTerm: 3 }, { _id: 3, name: 'mg-rs-3.mg-rs-pods.demo.svc.cluster.local:27017', health: 0, state: 8, stateStr: '(not reachable/healthy)', uptime: 0, optime: { ts: Timestamp( { t: 0, i: 0 } ), t: Long('-1') }, optimeDurable: { ts: Timestamp( { t: 0, i: 0 } ), t: Long('-1') }, optimeDate: ISODate('1970-01-01T00:00:00.000Z'), optimeDurableDate: ISODate('1970-01-01T00:00:00.000Z'), lastAppliedWallTime: ISODate('2024-04-08T12:45:44.788Z'), lastDurableWallTime: ISODate('2024-04-08T12:45:44.788Z'), lastHeartbeat: ISODate('2024-04-08T12:58:38.629Z'), lastHeartbeatRecv: ISODate('2024-04-08T12:45:49.039Z'), pingMs: Long('0'), lastHeartbeatMessage: 'Error connecting to mg-rs-3.mg-rs-pods.demo.svc.cluster.local:27017 :: caused by :: Could not find address for mg-rs-3.mg-rs-pods.demo.svc.cluster.local:27017: SocketException: Host not found (authoritative)', syncSourceHost: '', syncSourceId: -1, infoMessage: '', configVersion: 9, configTerm: 1 }, { _id: 4, name: 'mg-rs-4.mg-rs-pods.demo.svc.cluster.local:27017', health: 0, state: 8, stateStr: '(not reachable/healthy)', uptime: 0, optime: { ts: Timestamp( { t: 0, i: 0 } ), t: Long('-1') }, optimeDurable: { ts: Timestamp( { t: 0, i: 0 } ), t: Long('-1') }, optimeDate: ISODate('1970-01-01T00:00:00.000Z'), optimeDurableDate: ISODate('1970-01-01T00:00:00.000Z'), lastAppliedWallTime: ISODate('2024-04-08T12:45:24.787Z'), lastDurableWallTime: ISODate('2024-04-08T12:45:24.787Z'), lastHeartbeat: ISODate('2024-04-08T12:58:38.120Z'), lastHeartbeatRecv: ISODate('2024-04-08T12:45:33.529Z'), pingMs: Long('0'), lastHeartbeatMessage: 'Error connecting to mg-rs-4.mg-rs-pods.demo.svc.cluster.local:27017 :: caused by :: Could not find address for mg-rs-4.mg-rs-pods.demo.svc.cluster.local:27017: SocketException: Host not found (authoritative)', syncSourceHost: '', syncSourceId: -1, infoMessage: '', configVersion: 9, configTerm: 1 }

Top User Comments

Steps to Reproduce

Easy. Just make a 5 node replicaset. And make 3 secondaries down.