MySQL MHA-Go Deployment Guide

This article explains the support boundaries and basic usage of mha_go.yml. mha_go.yml deploys the Go rewrite of MHA (mha-go) without the Perl MHA toolchain. It runs as one static binary plus one YAML config on the manager node.

The current role of mha_go.yml is: deploy the mha-go manager on top of an existing GTID primary-replica topology created by dbbot master_slave.yml. If the MySQL topology does not exist yet, run master_slave.yml first.

1. How it works

mha-go provides failure detection, online switchover, and failover for GTID single-primary replication:

• GTID-only: binlog file/position replication is not supported.
• Topology discovery: SQL probes discover roles, GTID, read-only state, replication threads, lag, and semi-sync state.
• Candidate selection: the order of slave_ips maps to candidate_priority; the first replica has the highest priority.
• Failure recovery policy: dbbot defaults to availability-first, allowing promotion of the most advanced survivor when the old primary host is unreachable; transactions not replicated from the old primary may be lost.
• Writer endpoint abstraction: default writer_endpoint.kind: none; VIP mode calls /usr/local/bin/mha_ip_failover.sh.
• Runtime shape: the manager runs as mysql under systemd and writes JSON logs to the journal.

2. Comparison with legacy MHA

For new MySQL 8.4/9.7 GTID primary-replica clusters, prefer mha_go.yml. Keep legacy mha.yml only for MySQL 5.7 historical topologies.

3. Support boundaries

• Target architecture: one primary, one or more replicas, and one MHA-Go manager.
• MySQL versions: 8.4.x or 9.7.0 ER/EA.
• Replication: gtid_mode=ON and enforce_gtid_consistency=ON are mandatory.
• Default deployment: pure async replication, semi_sync.policy: disabled, and salvage.policy: availability-first.
• Semi-sync: use preferred or required only after MySQL semi-sync plugins are installed and enabled.
• Unsupported: MySQL 5.7, 8.0, 9.6, and non-GTID replication.

MySQL 9.7.x uses glibc2.28 packages and is not supported on the CentOS/RHEL 7 family. Use MySQL 8.4.x as the prerequisite primary-replica cluster on those systems.

4. Topology conventions

mha_go.yml reuses the [dbbot_mysql] host group and uses master_ip, slave_ips, and manager_ip for roles:

• master_ip: current primary, rendered as db1 in cluster.yaml.
• slave_ips: replica list, rendered as db2, db3, and so on.
• manager_ip: replica that runs mha-manager. It must be listed in slave_ips and cannot equal master_ip. The default is the last replica, which keeps the controller outside the primary host failure domain while preserving the first replica as the highest-priority promotion candidate.

dbbot default test environment:

192.168.161.11  primary
192.168.161.12  replica / preferred failover candidate
192.168.161.13  replica / manager

5. Key variables

Edit mysql_ansible/playbooks/vars/var_mha_go.yml:

master_ip: 192.168.161.11
slave_ips:
  - 192.168.161.12
  - 192.168.161.13
sub_nets: "192.168.161.%"

manager_ip: "{{ slave_ips | last }}"
mha_go_cluster_name: app1

mha_go_semi_sync_policy: disabled
mha_go_semi_sync_wait_for_replica_count: 0
mha_go_semi_sync_timeout: 5s
mha_go_salvage_policy: availability-first
mha_go_salvage_timeout: 30s

Common variables:

6. Prerequisites

make_mha_go checks these conditions before running:

• Every node has master_slave_finish.flag under datadir, meaning master_slave.yml completed.
• Every node has gtid_mode=ON and enforce_gtid_consistency=ON.
• master_ip, slave_ips, and manager_ip can all be found in the inventory, and manager_ip belongs to slave_ips.

Current mha_go.yml role chain:

pre_check_and_set -> make_mha_go

It does not reinstall MySQL or rebuild replication.

7. Entry point

cd /usr/local/dbbot/mysql_ansible/playbooks
python3 /usr/local/dbbot/portable-ansible/ansible-playbook \
  -i ../inventory/hosts.ini \
  mha_go.yml \
  -e dbbot_confirmation_input=confirm

When testing with dbbot public default passwords, explicitly allow them:

-e '{"fcs_allow_dbbot_default_passwd": true}'

8. Artifacts

On the manager node:

• /usr/local/bin/mha: the static mha-go binary bundled with dbbot.
• /etc/mha/cluster.yaml: cluster config with topology.kind: mysql-replication-single-primary and replication.mode: gtid.
• /etc/systemd/system/mha-manager.service: runs mha manager --log-format json as mysql:mysql.
• /var/log/mha/: log directory.

The manager node gets mha_go_finish.flag under datadir.

VIP mode also creates:

• /usr/local/bin/mha_ip_failover.sh: the root:mysql 0750 VIP switch script.
• /home/mysql/.ssh/id_rsa_dbbot_mha_go: a dedicated RSA-3072 private key used only by the manager VIP hook.
• /home/mysql/.ssh/known_hosts_dbbot_mha_go: node host keys collected and pinned during deployment.
• /usr/local/libexec/dbbot-mha-go-vip-ssh: a root-managed forced-command allowlist that prevents the dedicated key from opening a general mysql shell.
• /etc/sudoers.d/dbbot-mha-go-vip: exact ip addr add/del and arping commands for the configured VIP, mask, and interface.
• /var/lib/dbbot/mha-go-vip.yml: managed state used to remove the previous address and permissions when VIP mode is disabled.

9. Common commands

Run on the manager node:

mha version
mha check-repl --config /etc/mha/cluster.yaml
mha switch --config /etc/mha/cluster.yaml --new-primary db2 --dry-run
mha failover-plan --config /etc/mha/cluster.yaml
systemctl status mha-manager
journalctl -u mha-manager -f

failover-plan and failover-execute are expected to block while the primary is still alive.

9.1 Replica read-only state after restart

MHA-Go reuses the asynchronous replication base configuration created by master_slave.yml, so my.cnf keeps super_read_only=OFF. master_slave.yml sets super_read_only=ON on replicas at runtime when replication is first created, but the base template does not persist that state. The current MHA-Go manager detects and reports an unexpected read-only state; it does not modify the database variables automatically after mysqld restarts.

After every database process or host restart, discover the live topology again before restoring application traffic. Run the following only on a confirmed replica:

SHOW REPLICA STATUS\G
SELECT @@GLOBAL.read_only, @@GLOBAL.super_read_only;
SET GLOBAL super_read_only = ON;
SELECT @@GLOBAL.read_only, @@GLOBAL.super_read_only;

Validation must show both read_only and super_read_only as 0 on the current primary and as 1 on every replica, with healthy replication threads. Then run:

mha check-repl --config /etc/mha/cluster.yaml

Do not infer the actual primary from an old master_ip value or the role recorded in cluster.yaml, and never run SET GLOBAL super_read_only=ON on the live primary. The asynchronous topology deliberately avoids persisting this value as ON, because a crashed primary could restart in read-only mode before a failover completes. Operations must therefore maintain replica read-only state after restarts.

9.2 Host failure, salvage, and manager lifecycle

The dbbot-generated cluster.yaml has no per-node ssh section. MHA-Go SQL discovery, candidate selection, promotion, and replication repointing do not require OS SSH, and SSH binlog salvage is not enabled by default. Only the external VIP hook requires the manager’s mysql user to SSH to every node; the playbook creates and validates that dedicated path when VIP mode is enabled. A successful deployment test with VIP disabled therefore proves nothing about VIP SSH readiness.

The default availability-first policy promotes the most advanced survivor when the old primary host is unreachable and transactions unique to that host cannot be proven or recovered. This matches traditional MHA availability semantics, but pure async replication can lose transactions committed only on the old primary and not yet received by any replica. Use salvage-if-possible or strict when avoiding that risk is more important; with the old host fully unreachable and no per-node SSH/agent salvage, those policies block automatic promotion and require manual intervention.

After direct primary probes reach their threshold, the manager also waits until replica IO threads no longer confirm connectivity to the old primary. This suppresses false failover during a manager-only network partition. Total detection time therefore also depends on MySQL replica_net_timeout, replication heartbeats, and operating-system TCP timeouts; it is not bounded only by adding monitor.interval, failure_threshold, and reconfirm_timeout.

After a successful failover, the manager exits normally by design because the roles in the old cluster.yaml are stale. Recover or rebuild the old primary, update master_ip / slave_ips to the live topology, and rerun mha_go.yml. A blocked or failed execution now returns nonzero so Restart=on-failure restarts the service; inspect the first blocking or failed step in the journal instead of treating a restart loop as recovery.

10. Uninstall

mha_go_unsafe_uninstall.yml stops and disables mha-manager.service, removes mha-go manager files and the configured VIP, and then removes the MySQL instance data, log, and runtime directories for the current mysql_port:

cd /usr/local/dbbot/mysql_ansible/playbooks
python3 /usr/local/dbbot/portable-ansible/ansible-playbook \
  -i ../inventory/hosts.ini \
  mha_go_unsafe_uninstall.yml \
  -e dbbot_confirmation_input=confirm

Do not run this entry if you want to keep the MySQL primary-replica topology; it also removes the MySQL instance for the current mysql_port.

11. Enabling the VIP writer endpoint

VIP is disabled by default. To expose a stable writer endpoint:

mha_go_writer_endpoint_enabled: true
vip: 192.168.161.10
vip_netmask: "32"
net_work_interface: enp1s0

The playbook then:

• Rejects a VIP already present on a non-primary node and initially assigns it to master_ip.
• Generates a dedicated manager SSH key, installs it with forced-command restrictions on every MySQL node, and verifies every connection with BatchMode=yes and strict host-key checking. Commands outside the allowlist are rejected.
• Installs argument-level sudo rules; it does not grant unrestricted /usr/sbin/ip access or any scp privilege.
• Deploys the VIP hook, which removes the VIP from the old primary, adds it to the new primary, and sends gratuitous ARP.

vip, vip_netmask, and net_work_interface must match the real network. Sample environments using the dbbot 192.168.161.* test inventory usually use enp1s0; in other environments, confirm the interface with ip route or ip addr first. When changing the VIP, mask, or interface, update var_mha_go.yml and rerun the playbook so the hook, sudo policy, and current-primary assignment change together. Do not edit only one generated file. The playbook stops on a stale VIP at a non-primary node to prevent a duplicate address.

When mha_go_writer_endpoint_enabled is changed back to false and the playbook is rerun, dbbot uses /var/lib/dbbot/mha-go-vip.yml to remove the old VIP, sudo policy, dedicated SSH authorization, and hook. Do not manually delete that state file before disabling the feature. When manager_ip changes, rerunning the playbook first stops and removes stale mha-manager.service units from non-manager nodes and removes their old binary, config, and finish flag. This prevents dual managers with independent in-memory leases; existing journal logs are preserved.

12. Notes

• mha_go.yml is an incremental deployment entry point, not a MySQL primary-replica initialization entry point.
• manager_ip must be a replica listed in slave_ips. Do not colocate the manager with the current primary because a host-level failure would remove both the database and its failover controller.
• The default availability-first policy prioritizes restored writer availability, not zero data loss. Pure async RPO depends on whether each transaction reached a surviving replica before the failure.
• MySQL 9.7.0 renders version_series: "9.7" in cluster.yaml; MySQL 8.4.x renders "8.4".
• dbbot defaults semi_sync.policy to disabled so pure async topologies do not produce misleading semi-sync warnings.
• cluster.yaml is mode 0640 and owned by mysql:mysql because the systemd service reads it as the mysql user.