When I started developping pg_back, I wanted to keep it a simple as possible, by limiting its features. It also limited contributions, and at some point by accepting some useful contributions, the script grew and became not so simple. We could go on with the shell script and try to carefully control its growth, but it will sooner or later reach some limits of shell scripting.
After some time without making it evolve, I got interested in the Go programming language. It appeared that pg_back could be a good candidate projet to let me learn Go. This is why pg_back version 2 is written in Go.
Go can do better than shell scripting, like many languages, but one helpful
feature of Go is that it produces static binaries. This make it even easier to
install than the shell script, because pg_back
no longer has any external
dependencies, apart from pg_dump
and pg_dumpall
. When one does not have
root privileges, it make it possible be deploy the binary file by dropping it
somewhere like one would do for a shell script.
About the rewrite, getting rid of dependencies on external command is
great. For example, the syntax of Go being close to C, it was quite easy to
port pg_dumpacl
and integrate it directly in pg_back. It is now just two
function dumpCreateDBAndACL()
and makeACLCommands()
in
sql.go.
Continuing on external commands, pg_back now connects to PostgreSQL without
using psql
, with the native database/sql
API. The standard library is
pretty rich, it allowed the use standard Go APIs for checksums, file locking,
logging, etc. Only 4 external modules are required, plus one for unit tests:
gopkg.in/ini.v1
for the configuration filegithub.com/spf13/pflag
for POSIX command line optionsgithub.com/lib/pq
to access PostgreSQLgithub.com/anmitsu/go-shlex
to parse shell commands (mostly hooks)
The other module is github.com/google/go-cmp
, with is useful to compare
structs in unit tests. Speaking of which, unit tests are native to Go, and it
was easy to add tests while developping. The difficult part of unit testing is
related to the replication control functions. Unit testing the replication
control fonctions is difficult because it requires a more complex setup. We
need a PostgreSQL instance with a replica.
This feature is interesting, because I could use some features of Go to implement pausing the replication.
When dumping from a replica, we need to ensure pg_dump won’t be stuck on an exclusive lock, we must pause the replication replay to avoid pg_dump being cancelled and there must not be any exclusive lock granted on an object when the replication is paused. Otherwise the exclusive lock is only released when replication replay is resumed, making the dump of the object impossible.
I used a go routine with a time.Ticker
to try to pause the replication every
10 seconds:
func pauseReplicationWithTimeout(db *pg, timeOut int) error {
// ...
ticker := time.NewTicker(time.Duration(10) * time.Second)
done := make(chan bool)
stop := make(chan bool)
fail := make(chan error)
l.Infoln("pausing replication")
// We want to retry pausing replication at a defined interval
// but not forever. We cannot put the timeout in the same
// select as the ticker since the ticker would always win
go func() {
var rerr *pgReplicaHasLocks
defer ticker.Stop()
for {
if err := pauseReplication(db); err != nil {
if errors.As(err, &rerr) {
l.Warnln(err)
} else {
fail <- err
return
}
} else {
done <- true
return
}
select {
case <-stop:
return
case <-ticker.C:
break
}
}
}()
// ...
}
When there is an exclusive lock, a custom error is used to inform the goroutine that it must retry:
type pgReplicaHasLocks struct{}
func (*pgReplicaHasLocks) Error() string {
return "replication not paused because of AccessExclusiveLock"
}
func pauseReplication(db *pg) error {
// ..
if void == "failed" {
return &pgReplicaHasLocks{}
}
return nil
}
The main goroutine just wait on some channel until there is a timeout using
time.After()
, replication has been paused or an error occured:
// ... (After lauching the goroutine)
// Return as soon as the replication is paused or stop the
// goroutine if we hit the timeout
select {
case <-done:
l.Infoln("replication paused")
case <-time.After(time.Duration(timeOut) * time.Second):
stop <- true
return fmt.Errorf("replication not paused after %v", time.Duration(timeOut)*time.Second)
case err := <-fail:
return fmt.Errorf("%s", err)
}
// ...
See the complete function in sql.go#L509-L629.
Another use of goroutines was to run pg_dump
commands concurrently, in order to dump many
database at the same time. It uses the worker example from
gobyexample.com, a great ressource for
beginners, by the way!
The other problems with shell scripting were about parsing strings, like
keyword=value
connection strings of PostgreSQL
(connstring.go). Shell
scripting is notoriously difficult when it comes to handling strings properly,
and Go helped a lot. It was happily surprised that the Go implementation of pg_back
could dump databases with weird characters in their names, like the equal sign,
spaces, quotes, etc. without a lot of extra work.
Finally, the embed feature of Go 1.16 is so easy to use that embeding the default configuration file, including examples and comments, and adding a command line option to print it took a just few lines of code:
import (
// ...
_ "embed"
// ...
)
//go:embed pg_back.conf
var defaultCfg string
func parseCli(args []string) (options, []string, error) {
// ...
pflag.BoolVar(&pce.ShowConfig, "print-default-config", false, "print the default configuration\n")
if pce.ShowConfig {
fmt.Print(defaultCfg)
// ...
}
// ...
}
This will make the feature easier to maintain in the future because it avoids
duplication, as I would have had to copy the file in a string in the source
code without embed
.
In conclusion, rewriting pg_back in a proper programming language with a rich standard library and ecosystem was great. I enjoyed doing it with Go and it convinced me that Go is programming language worth learning.