3

I'm experiencing constant high memory usage from Duplicity on all servers running it as a backup tool to S3.

Isn't Duplicity supposed to run its backup task and kill its job afterwards, or am I missing something here?

duply -v
duply version 2.0.1
(http://duply.net)

Using installed duplicity version 0.7.11, python 2.7.6, gpg 1.4.16 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1', grep 'grep (GNU grep) 2.16', bash '4.3.11(1)-release (x86_64-pc-linux-gnu)'.

I'm using Duply to manage profiles on each server, here's one of them:

GPG_KEY='FOO'
GPG_PW='FOO'

TARGET='s3://s3-eu-central-1.amazonaws.com/foo-bucket/bar-location'

export AWS_ACCESS_KEY_ID='FOO'
export AWS_SECRET_ACCESS_KEY='FOO'

# base directory to backup
SOURCE='/'

# exclude folders containing exclusion file (since duplicity 0.5.14)
# Uncomment the following two lines to enable this setting.
FILENAME='.duplicity-ignore'
DUPL_PARAMS="$DUPL_PARAMS --exclude-if-present '$FILENAME'"

# Time frame for old backups to keep, Used for the "purge" command.
# see duplicity man page, chapter TIME_FORMATS)
MAX_AGE=2M

# Number of full backups to keep. Used for the "purge-full" command.
# See duplicity man page, action "remove-all-but-n-full".
MAX_FULL_BACKUPS=2

# Number of full backups for which incrementals will be kept for.
# Used for the "purge-incr" command.
# See duplicity man page, action "remove-all-inc-of-but-n-full".
MAX_FULLS_WITH_INCRS=1

# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
# forces a full backup if last full backup reaches a specified age, for the
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
# Uncomment the following two lines to enable this setting.
MAX_FULLBKP_AGE=1M
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "

# sets duplicity --volsize option (available since v0.4.3.RC7)
# set the size of backup chunks to VOLSIZE MB instead of the default 25MB.
# VOLSIZE must be number of MB's to set the volume size to.
# Uncomment the following two lines to enable this setting.
VOLSIZE=100
DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "

# more duplicity command line options can be added in the following way
# don't forget to leave a separating space char at the end
#DUPL_PARAMS="$DUPL_PARAMS --put_your_options_here "

Here's the cronjob to run the backups:

12 3 * * * nice -n19 ionice -c2 -n7 duply database backup_verify_purge --force --name foo_database >> /var/log/duplicity.log 2>&1

26 3 * * * nice -n19 ionice -c2 -n7 duply websites backup_verify_purge --force --name foo_websites >> /var/log/duplicity.log 2>&1

53 4 * * * nice -n19 ionice -c2 -n7 duply system backup_verify_purge --force --name foo_system >> /var/log/duplicity.log 2>&1

Here's a 24 hour graph of the memory usage:

enter image description here

Pierre.Vriens
  • 1,159
  • 34
  • 15
  • 19
Jasper
  • 51
  • 4

4 Answers4

2

I'm using a nearly identical configuration as you and seeing very similar memory usage. My python and gpg are slightly newer revs, and I'm not using duply. duplicity is 0.7.11.

One of the machines(el6) I've tested on was backing about 3.5 million files to s3 and maxed at about 3G of res memory. Restoring a single file from this backup maxed at about 3.9G of RAM:

29067 root     /opt/duplicity/bin/python2         0  3893924  3893957  3894704

Another machine(el5) that is backing up about 7.5 million files is currently using 1.9G res while running.

Wondering two things that are relevant to your post:

  1. Is this memory usage normal? Thinking this is probably normal when there are a large number of files in the backup volume. If not, how much should I expect to use for a backup of 3.5M files of about 50G to s3 or swift(if it matters).
  2. Will changing the --volsize parameter to be less than 200M help? I noticed in the changelog that .7.11 increased default volsize quite a bit.

Anything else that can reduce memory usage?

Sorry I can't answer your question, but I feel this post is relevant since we appear to be dealing with a similar issue. I've looked around quite a bit and don't see a lot about normal duplicity memory usage except that related to the bug from ~ 2012.

Thanks!

iwaseatenbyagrue
  • 3,588
  • 12
  • 22
newt
  • 21
  • 1
2

It looks like my problem was caused by the version of Duplicity installed through apt-get.

Installing the tarball version resolved the memory issues.

Here's more info: https://answers.launchpad.net/duplicity/+question/577611

Jasper
  • 51
  • 4
0

Also saw this from Kenneth which may be relevant with huge files:

Kenneth Loafman (kenneth-loafman) wrote on 2016-02-26: On huge files a great number of sigs are generated and held until the file is complete. To mitigate this problem use "--max-blocksize=20480" or higher to avoid this issue.

FFQ
  • 1
0

I have the same problem, has anybody had any luck resolving this issue?

I have updated duplicity, python-boto, tried with high --max-blocksize and no luck. It backups without any problem to the FTP location, if I try to backup to S3 in fails with OOM kill...

It is weird that I have production and staging servers, that are the same, and staging backs up normally to S3 with 600mb RAM usage while doing it. Production server eats up 3GB+ and then fails.

b4d
  • 125
  • 6