[Pgcluster-general] ClusterDB will not recover

[Leif Madsen]

Morning all,

I've been trying to get my DBs to recover this morning after they lost
connection to pgreplicate over night (not sure why yet... but this is
just development thus far).

My topology includes 3 physical boxes: db1, db2, db3.

db1, db2: clusterDB
db3: clusterDB, pgreplicate

Two questions/problems:
1) When I was trying to bring everything back up, I had pgreplicate
running, then started the DB on db3. pgreplicate did not seem to setup
a MasterDB until I ran BEGIN;END; on db3, at which point I could run:
pg_ctl start -o '-R'   on db2, and then it would recover. I thought
with Life Check this wasn't necessary anymore?

2) When starting up db1 with: pg_ctl start -o '-R'      I get the
following messages on pgreplicate. It would get through all its 1st
recovery step, but then it would just fail, not attempting the 2nd
recovery step. pgreplicate then goes into a loop... ? The hostnames
have been changes to db1, db2, db3 to protect the innocent :)

2007-02-28 09:21:27 [8707] DEBUG:hostName = db1
2007-02-28 09:21:27 [8707] DEBUG:pg_data = /var/lib/pgsql/data
2007-02-28 09:21:27 [8707] DEBUG:pgrecovery_loop():receive packet no:1
2007-02-28 09:21:27 [8707] DEBUG:pgrecovery_loop():1st master db3 - 5432
2007-02-28 09:21:27 [8707] DEBUG:pgrecovery_loop():1st target db2 - 5432
2007-02-28 09:21:27 [8707] DEBUG:first_setup_recovery():1st setup target db1
2007-02-28 09:21:27 [8707] DEBUG:first_setup_recovery():1st setup port 5432
2007-02-28 09:21:27 [8707] DEBUG:first_setup_recovery():add recovery
target to host table
2007-02-28 09:21:27 [8707] DEBUG:first_setup_recovery():set
RECOVERY_PGDATA_REQ packet data
2007-02-28 09:21:28 [8707]
DEBUG:PGRsend_replicate_packet_to_server():connect db:template1
port:5432 user:postgres host:db3-ip_address query:VACUUM
2007-02-28 09:21:28 [8707]
DEBUG:send_replicate_packet_to_server():sync_command(SELECT
PGR_SYSTEM_COMMAND_FUNCTION(3,0,0,0,1,23) )
2007-02-28 09:21:28 [8707]
DEBUG:send_replicate_packet_to_server():PQexec send :VACUUM
2007-02-28 09:21:28 [8707]
DEBUG:send_replicate_packet_to_server():PQexec returns :VACUUM
2007-02-28 09:21:28 [8707] DEBUG:first_setup_recovery():send packet to
master db3 recoveryPort 7001
2007-02-28 09:21:28 [8707] DEBUG:first_setup_recovery():wait answer
from master server
2007-02-28 09:21:28 [8707] DEBUG:read_packet():receive packet
2007-02-28 09:21:28 [8707] DEBUG:no = 3
2007-02-28 09:21:28 [8707] DEBUG:max_connect = 100
2007-02-28 09:21:28 [8707] DEBUG:port = 5432
2007-02-28 09:21:28 [8707] DEBUG:recoveryPort = 7001
2007-02-28 09:21:28 [8707] DEBUG:hostName = db3
2007-02-28 09:21:28 [8707] DEBUG:pg_data = /var/lib/pgsql/data
2007-02-28 09:21:28 [8707] DEBUG:first_setup_recovery():get answer
from master:no[3]
2007-02-28 09:21:28 [8707] DEBUG:pgrecovery_loop():first_setup_recovery end:0
2007-02-28 09:21:32 [8707] DEBUG:read_packet():receive packet
2007-02-28 09:21:32 [8707] DEBUG:no = 5
2007-02-28 09:21:32 [8707] DEBUG:max_connect = 100
2007-02-28 09:21:32 [8707] DEBUG:port = 5432
2007-02-28 09:21:32 [8707] DEBUG:recoveryPort = 7001
2007-02-28 09:21:32 [8707] DEBUG:hostName = db3
2007-02-28 09:21:32 [8707] DEBUG:pg_data = /var/lib/pgsql/data
2007-02-28 09:21:32 [8707] DEBUG:pgrecovery_loop():receive packet no:5
2007-02-28 09:21:32 [8707] DEBUG:pgrecovery_loop():2nd master db3 - 5432
2007-02-28 09:21:32 [8707] DEBUG:pgrecovery_loop():2nd target db1 - 5432
2007-02-28 09:21:33 [8707] DEBUG:now, waiting clear every transaction
for recovery
2007-02-28 09:21:34 [8707] DEBUG:now, waiting clear every transaction
for recovery
2007-02-28 09:21:35 [8707] DEBUG:now, waiting clear every transaction
for recovery
2007-02-28 09:21:36 [8707] DEBUG:now, waiting clear every transaction
for recovery

-- 
Leif Madsen.
http://www.leifmadsen.com
http://www.oreilly.com/catalog/asterisk