bug-coreutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

bug#39273: unwanted behavior in the combination of an scenario regarding


From: Paul Eggert
Subject: bug#39273: unwanted behavior in the combination of an scenario regarding btrfs, ssh, borg, and 'df' from the core utils
Date: Tue, 4 Feb 2020 11:35:49 -0800
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1

[cc'ing address@hidden; please keep them in the loop.]

In response to the email quoted below, the key issue seems to be here:

statfs("/", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414566026]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0

That is, the statfs system call says that "/" has no blocks available. To fix this, you'll need to see why the kernel (presumably in its btrfs filesystem code) is mistakenly saying that f_bavail=0. This stuff is inside the kernel so df and coreutils are not the problem here.

You might ask the Linux-BTRFS mailing list about this. It appears to be related to a known issue in their code; see:

https://lore.kernel.org/linux-btrfs/address@hidden/T/



On 2/4/20 6:56 AM, Wismerhill wrote:
Hi Paul,


suddenly i ran again into the same Error, it was now during a
installation, i wanted to install a Package an the process ended with a
  "No Space left"

... but I was able then to run the commands for the output you
required.

df -h gave me:

Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
devtmpfs         12G       0   12G    0% /dev
tmpfs            12G    120M   12G    1% /dev/shm
tmpfs            12G    9,7M   12G    1% /run
tmpfs            12G       0   12G    0% /sys/fs/cgroup
/dev/sda1       384G     56G     0  100% /
/dev/sda1       384G     56G     0  100% /boot/grub2/x86_64-efi
/dev/sda1       384G     56G     0  100% /srv
/dev/sda1       384G     56G     0  100% /var
/dev/sda1       384G     56G     0  100% /boot/grub2/i386-pc
/dev/sda1       384G     56G     0  100% /.snapshots
/dev/sda1       384G     56G     0  100% /root
/dev/sda1       384G     56G     0  100% /usr/local
/dev/sda1       384G     56G     0  100% /opt
/dev/sda1       384G     56G     0  100% /tmp
/dev/sdb1       932G    699G  230G   76% /home
tmpfs           2,4G     96K  2,4G    1% /run/user/1000



wismerhill:/home/lux # btrfs fi usage /
Overall:
     Device size:                 383.98GiB
     Device allocated:             61.03GiB
     Device unallocated:          322.95GiB
     Device missing:                  0.00B
     Used:                         55.61GiB
     Free (estimated):            328.22GiB      (min: 328.22GiB)
     Data ratio:                       1.00
     Metadata ratio:                   1.00
     Global reserve:              141.11MiB      (used: 0.00B)

Data,single: Size:59.00GiB, Used:53.72GiB (91.06%)
    /dev/sda1      59.00GiB

Metadata,single: Size:2.00GiB, Used:1.88GiB (94.17%)
    /dev/sda1       2.00GiB

System,single: Size:32.00MiB, Used:16.00KiB (0.05%)
    /dev/sda1      32.00MiB

Unallocated:
    /dev/sda1     322.95GiB
wismerhill:/home/lux #

.... btrfs says there is plenty of space ....

STRACE OUTPUT OF THE 2 COMMANDS:


wismerhill:/home/lux # strace df -h >/home/lux/df_trace
execve("/bin/df", ["df", "-h"], 0x7ffde69ce648 /* 95 vars */) = 0
brk(NULL)                               = 0x55f0e2c56000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=287090, ...}) = 0
mmap(NULL, 287090, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff42000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\20\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2034848, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f554ff40000
mmap(NULL, 3906144, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f554f9aa000
mprotect(0x7f554fb5b000, 2093056, PROT_NONE) = 0
mmap(0x7f554fd5a000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b0000) = 0x7f554fd5a000
mmap(0x7f554fd60000, 14944, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f554fd60000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7f554ff41500) = 0
mprotect(0x7f554fd5a000, 16384, PROT_READ) = 0
mprotect(0x55f0e23f8000, 4096, PROT_READ) = 0
mprotect(0x7f554ff89000, 4096, PROT_READ) = 0
munmap(0x7f554ff42000, 287090)          = 0
brk(NULL)                               = 0x55f0e2c56000
brk(0x55f0e2c77000)                     = 0x55f0e2c77000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = -1 ENOENT
(No such file or directory)
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2939, ...}) = 0
read(3, "# Locale name alias data base.\n#"..., 4096) = 2939
read(3, "", 4096)                       = 0
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_IDENTIFICATION",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_IDENTIFICATION",
O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=351, ...}) = 0
mmap(NULL, 351, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff88000
close(3)                                = 0
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=26244, ...}) = 0
mmap(NULL, 26244, PROT_READ, MAP_SHARED, 3, 0) = 0x7f554ff81000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_MEASUREMENT", O_RDONLY|O_CLOEXEC)
= -1 ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_MEASUREMENT", O_RDONLY|O_CLOEXEC) =
3
fstat(3, {st_mode=S_IFREG|0644, st_size=23, ...}) = 0
mmap(NULL, 23, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff80000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_TELEPHONE", O_RDONLY|O_CLOEXEC) =
-1 ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_TELEPHONE", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=56, ...}) = 0
mmap(NULL, 56, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff7f000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_ADDRESS", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_ADDRESS", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=155, ...}) = 0
mmap(NULL, 155, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff7e000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_NAME", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_NAME", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=83, ...}) = 0
mmap(NULL, 83, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff7d000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_PAPER", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_PAPER", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=34, ...}) = 0
mmap(NULL, 34, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff7c000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_MESSAGES", O_RDONLY|O_CLOEXEC) =
-1 ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_MESSAGES", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=30, ...}) = 0
close(3)                                = 0
open("/usr/lib/locale/de_DE.utf8/LC_MESSAGES/SYS_LC_MESSAGES",
O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=60, ...}) = 0
mmap(NULL, 60, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff7b000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_MONETARY", O_RDONLY|O_CLOEXEC) =
-1 ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_MONETARY", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=294, ...}) = 0
mmap(NULL, 294, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff7a000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_COLLATE", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_COLLATE", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1244054, ...}) = 0
mmap(NULL, 1244054, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554fe10000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2354, ...}) = 0
mmap(NULL, 2354, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff79000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=54, ...}) = 0
mmap(NULL, 54, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554ff78000
close(3)                                = 0
open("/usr/lib/locale/de_DE.UTF-8/LC_CTYPE", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
open("/usr/lib/locale/de_DE.utf8/LC_CTYPE", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=330764, ...}) = 0
mmap(NULL, 330764, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554fdbf000
close(3)                                = 0
open("/usr/share/locale-langpack/de/LC_MESSAGES/coreutils.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de/LC_MESSAGES/coreutils.mo", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=348263, ...}) = 0
mmap(NULL, 348263, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f554f954000
close(3)                                = 0
openat(AT_FDCWD, "/proc/self/mountinfo", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "22 93 0:20 / /sys rw,nosuid,node"..., 1024) = 1024
read(3, "nosuid,nodev,noexec,relatime sha"..., 1024) = 1024
read(3, " 29 0:38 / /sys/fs/cgroup/pids r"..., 1024) = 1024
read(3, "rub2/i386-pc /boot/grub2/i386-pc"..., 1024) = 1024
read(3, "-fuse gvfsd-fuse rw,user_id=1000"..., 1024) = 328
read(3, "", 1024)                       = 0
lseek(3, 0, SEEK_CUR)                   = 4424
close(3)                                = 0
stat("/dev", {st_mode=S_IFDIR|0755, st_size=4460, ...}) = 0
stat("/sys/kernel/security", {st_mode=S_IFDIR|0755, st_size=0, ...}) =
0
stat("/dev/shm", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=40, ...}) = 0
stat("/run", {st_mode=S_IFDIR|0755, st_size=1180, ...}) = 0
stat("/sys/fs/cgroup", {st_mode=S_IFDIR|0755, st_size=380, ...}) = 0
stat("/sys/fs/cgroup/unified", {st_mode=S_IFDIR|0555, st_size=0, ...})
= 0
stat("/sys/fs/cgroup/systemd", {st_mode=S_IFDIR|0555, st_size=0, ...})
= 0
stat("/sys/fs/pstore", {st_mode=S_IFDIR|0750, st_size=0, ...}) = 0
stat("/sys/fs/cgroup/cpu,cpuacct", {st_mode=S_IFDIR|0555, st_size=0,
...}) = 0
stat("/sys/fs/cgroup/rdma", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("/sys/fs/cgroup/net_cls,net_prio", {st_mode=S_IFDIR|0555,
st_size=0, ...}) = 0
stat("/sys/fs/cgroup/hugetlb", {st_mode=S_IFDIR|0555, st_size=0, ...})
= 0
stat("/sys/fs/cgroup/memory", {st_mode=S_IFDIR|0555, st_size=0, ...}) =
0
stat("/sys/fs/cgroup/freezer", {st_mode=S_IFDIR|0555, st_size=0, ...})
= 0
stat("/sys/fs/cgroup/cpuset", {st_mode=S_IFDIR|0555, st_size=0, ...}) =
0
stat("/sys/fs/cgroup/devices", {st_mode=S_IFDIR|0555, st_size=0, ...})
= 0
stat("/sys/fs/cgroup/perf_event", {st_mode=S_IFDIR|0555, st_size=0,
...}) = 0
stat("/sys/fs/cgroup/blkio", {st_mode=S_IFDIR|0555, st_size=0, ...}) =
0
stat("/sys/fs/cgroup/pids", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat("/", {st_mode=S_IFDIR|0755, st_size=188, ...}) = 0
stat("/dev/hugepages", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("/boot/grub2/x86_64-efi", {st_mode=S_IFDIR|0755, st_size=0, ...})
= 0
stat("/srv", {st_mode=S_IFDIR|0755, st_size=44, ...}) = 0
stat("/var", {st_mode=S_IFDIR|0755, st_size=140, ...}) = 0
stat("/boot/grub2/i386-pc", {st_mode=S_IFDIR|0755, st_size=6186, ...})
= 0
stat("/.snapshots", {st_mode=S_IFDIR|0750, st_size=226, ...}) = 0
stat("/root", {st_mode=S_IFDIR|0700, st_size=496, ...}) = 0
stat("/usr/local", {st_mode=S_IFDIR|0755, st_size=76, ...}) = 0
stat("/opt", {st_mode=S_IFDIR|0755, st_size=156, ...}) = 0
stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=19504, ...}) = 0
stat("/home", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
stat("/run/user/1000", {st_mode=S_IFDIR|0700, st_size=440, ...}) = 0
stat("/run/user/1000/gvfs", 0x7fff58c09890) = -1 EACCES (Permission
denied)
stat("/run/user/1000/doc", 0x7fff58c09890) = -1 EACCES (Permission
denied)
stat("/sys/kernel/debug/tracing", {st_mode=S_IFDIR|0700, st_size=0,
...}) = 0
uname({sysname="Linux", nodename="wismerhill.fritz.box", ...}) = 0
statfs("/dev", {f_type=TMPFS_MAGIC, f_bsize=4096, f_blocks=3061960,
f_bfree=3061960, f_bavail=3061960, f_files=3061960, f_ffree=3061355,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID}) = 0
statfs("/sys/kernel/security", {f_type=SECURITYFS_MAGIC, f_bsize=4096,
f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/dev/shm", {f_type=TMPFS_MAGIC, f_bsize=4096, f_blocks=3064570,
f_bfree=3034100, f_bavail=3034100, f_files=3064570, f_ffree=3064495,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV}) = 0
statfs("/run", {f_type=TMPFS_MAGIC, f_bsize=4096, f_blocks=3064570,
f_bfree=3062090, f_bavail=3062090, f_files=3064570, f_ffree=3063691,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV}) = 0
statfs("/sys/fs/cgroup", {f_type=TMPFS_MAGIC, f_bsize=4096,
f_blocks=3064570, f_bfree=3064570, f_bavail=3064570, f_files=3064570,
f_ffree=3064552, f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RDONLY|ST_NOSUID|ST_NODEV|ST_NOEXEC}) = 0
statfs("/sys/fs/cgroup/unified", {f_type=CGROUP2_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/systemd", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/pstore", {f_type=PSTOREFS_MAGIC, f_bsize=4096,
f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/cpu,cpuacct", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/rdma", {f_type=CGROUP_SUPER_MAGIC, f_bsize=4096,
f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/net_cls,net_prio", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/hugetlb", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/memory", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/freezer", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/cpuset", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/devices", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/perf_event", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/blkio", {f_type=CGROUP_SUPER_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/sys/fs/cgroup/pids", {f_type=CGROUP_SUPER_MAGIC, f_bsize=4096,
f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0
statfs("/", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414566026]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/dev/hugepages", {f_type=HUGETLBFS_MAGIC, f_bsize=2097152,
f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=2097152,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/boot/grub2/x86_64-efi", {f_type=BTRFS_SUPER_MAGIC,
f_bsize=4096, f_blocks=100657920, f_bfree=86044517, f_bavail=0,
f_files=0, f_ffree=0, f_fsid={val=[517186897, 2414563814]},
f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/srv", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414563819]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/var", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414563820]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/boot/grub2/i386-pc", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414563815]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/.snapshots", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414563812]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/root", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414563816]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/usr/local", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414563821]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/opt", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414563817]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/tmp", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044517, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414563818]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/home", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=244190208, f_bfree=61029605, f_bavail=60164413, f_files=0,
f_ffree=0, f_fsid={val=[956236011, 2590044172]}, f_namelen=255,
f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/run/user/1000", {f_type=TMPFS_MAGIC, f_bsize=4096,
f_blocks=612914, f_bfree=612890, f_bavail=612890, f_files=3064570,
f_ffree=3064507, f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_RELATIME}) = 0
statfs("/run/user/1000/gvfs", {f_type=FUSE_SUPER_MAGIC, f_bsize=0,
f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=0, f_frsize=0,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_RELATIME}) = 0
statfs("/run/user/1000/doc", {f_type=FUSE_SUPER_MAGIC, f_bsize=0,
f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=0, f_frsize=0,
f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_RELATIME}) = 0
statfs("/sys/kernel/debug/tracing", {f_type=TRACEFS_MAGIC,
f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
fstat(1, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
write(1, "Dateisystem    Gr\303\266\303\237e Benutzt V"..., 864) = 864
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++



wismerhill:/home/lux #



-----------------------------------------------------------------------


wismerhill:/home/lux # strace btrfs fi usage /
execve("/sbin/btrfs", ["btrfs", "fi", "usage", "/"], 0x7ffcc2b1b578 /*
95 vars */) = 0
brk(NULL)                               = 0x562f9a93a000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=287090, ...}) = 0
mmap(NULL, 287090, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f92a61d6000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\31\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=30848, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f92a61d4000
mmap(NULL, 2125848, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f92a5df0000
mprotect(0x7f92a5df6000, 2097152, PROT_NONE) = 0
mmap(0x7f92a5ff6000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f92a5ff6000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib64/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\260\0\0\0\0\0\0"...
, 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=338976, ...}) = 0
mmap(NULL, 2434936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f92a5b9d000
mprotect(0x7f92a5bea000, 2097152, PROT_NONE) = 0
mmap(0x7f92a5dea000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4d000) = 0x7f92a5dea000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@'\0\0\0\0\0\0"..., 832)
= 832
fstat(3, {st_mode=S_IFREG|0755, st_size=92216, ...}) = 0
mmap(NULL, 2187280, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f92a5986000
mprotect(0x7f92a599c000, 2093056, PROT_NONE) = 0
mmap(0x7f92a5b9b000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f92a5b9b000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib64/liblzo2.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0
!\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=137424, ...}) = 0
mmap(NULL, 2232496, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f92a5764000
mprotect(0x7f92a5785000, 2093056, PROT_NONE) = 0
mmap(0x7f92a5984000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x20000) = 0x7f92a5984000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360a\0\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=140688, ...}) = 0
mmap(NULL, 2217064, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f92a5546000
mprotect(0x7f92a555f000, 2093056, PROT_NONE) = 0
mmap(0x7f92a575e000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f92a575e000
mmap(0x7f92a5760000, 13416, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f92a5760000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\20\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2034848, ...}) = 0
mmap(NULL, 3906144, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f92a518c000
mprotect(0x7f92a533d000, 2093056, PROT_NONE) = 0
mmap(0x7f92a553c000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b0000) = 0x7f92a553c000
mmap(0x7f92a5542000, 14944, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f92a5542000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f92a61d2000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f92a61cf000
arch_prctl(ARCH_SET_FS, 0x7f92a61cf8c0) = 0
mprotect(0x7f92a553c000, 16384, PROT_READ) = 0
mprotect(0x7f92a575e000, 4096, PROT_READ) = 0
mprotect(0x7f92a5984000, 4096, PROT_READ) = 0
mprotect(0x7f92a5b9b000, 4096, PROT_READ) = 0
mprotect(0x7f92a5ff6000, 4096, PROT_READ) = 0
mprotect(0x7f92a5dea000, 20480, PROT_READ) = 0
mprotect(0x562f99149000, 20480, PROT_READ) = 0
mprotect(0x7f92a621d000, 4096, PROT_READ) = 0
munmap(0x7f92a61d6000, 287090)          = 0
set_tid_address(0x7f92a61cfb90)         = 19502
set_robust_list(0x7f92a61cfba0, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f92a554bc60, sa_mask=[],
sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f92a5558300}, NULL, 8)
= 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f92a554bd00, sa_mask=[],
sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO,
sa_restorer=0x7f92a5558300}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024,
rlim_max=16384*1024}) = 0
statfs("/", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
f_blocks=100657920, f_bfree=86044382, f_bavail=0, f_files=0, f_ffree=0,
f_fsid={val=[517186897, 2414566026]}, f_namelen=255, f_frsize=4096,
f_flags=ST_VALID|ST_RELATIME}) = 0
stat("/", {st_mode=S_IFDIR|0755, st_size=188, ...}) = 0
stat("/", {st_mode=S_IFDIR|0755, st_size=188, ...}) = 0
open("/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=188, ...}) = 0
brk(NULL)                               = 0x562f9a93a000
brk(0x562f9a95b000)                     = 0x562f9a95b000
ioctl(3, BTRFS_IOC_TREE_SEARCH, {key={tree_id=3 /*
BTRFS_CHUNK_TREE_OBJECTID */, min_objectid=0,
max_objectid=18446744073709551615 /* UINT64_MAX */,
max_offset=18446744073709551615 /* UINT64_MAX */,
max_transid=18446744073709551615 /* UINT64_MAX */, min_type=255,
max_type=0, nr_items=4096}} => {key={nr_items=35}, buf=...}) = 0
ioctl(3, BTRFS_IOC_TREE_SEARCH, {key={tree_id=3 /*
BTRFS_CHUNK_TREE_OBJECTID */, max_objectid=18446744073709551615 /*
UINT64_MAX */, min_offset=285536681985, max_offset=18446744073709551615
/* UINT64_MAX */, max_transid=18446744073709551615 /* UINT64_MAX */,
min_type=228 /* BTRFS_CHUNK_ITEM_KEY */, max_type=0, nr_items=35}} =>
{key={nr_items=28}, buf=...}) = 0
ioctl(3, BTRFS_IOC_TREE_SEARCH, {key={tree_id=3 /*
BTRFS_CHUNK_TREE_OBJECTID */, max_objectid=18446744073709551615 /*
UINT64_MAX */, min_offset=413882384385, max_offset=18446744073709551615
/* UINT64_MAX */, max_transid=18446744073709551615 /* UINT64_MAX */,
min_type=228 /* BTRFS_CHUNK_ITEM_KEY */, max_type=0, nr_items=28}} =>
{key={nr_items=0}, buf=...}) = 0
ioctl(3, BTRFS_IOC_FS_INFO, {max_id=1, num_devices=1, fsid=aaa1b342-
dccb-4010-b472-1613532010fe, nodesize=16384, sectorsize=4096,
clone_alignment=4096}) = 0
ioctl(3, BTRFS_IOC_DEV_INFO, {devid=0}) = -1 ENODEV (No such device)
ioctl(3, BTRFS_IOC_DEV_INFO, {devid=1} => {uuid=d68b71fe-c964-4ddf-
a10c-d76bf7085f25, bytes_used=65531805696, total_bytes=412294840320,
path="/dev/sda1"}) = 0
openat(AT_FDCWD, "/dev/sda1", O_RDONLY) = 4
pread64(4,
"\334\301u\237\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
..., 4096, 65536) = 4096
close(4)                                = 0
openat(AT_FDCWD, "/dev/sda1", O_RDONLY) = 4
ioctl(4, BLKGETSIZE64, [412294840320])  = 0
close(4)                                = 0
sysinfo({uptime=27223, loads=[16736, 24192, 20704],
totalram=25104957440, freeram=17284509696, sharedram=173166592,
bufferram=2154496, totalswap=0, freeswap=0, procs=704, totalhigh=0,
freehigh=0, mem_unit=1}) = 0
ioctl(3, BTRFS_IOC_SPACE_INFO, {space_slots=0} => {total_spaces=4}) = 0
ioctl(3, BTRFS_IOC_SPACE_INFO, {space_slots=4} => {total_spaces=4,
spaces=...}) = 0
fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 0), ...}) = 0
write(1, "Overall:\n", 9Overall:
)               = 9
write(1, "    Device size:\t\t 383.98GiB\n", 29    Device
size:          383.98GiB
) = 29
write(1, "    Device allocated:\t\t  61.03Gi"..., 34    Device
allocated:                 61.03GiB
) = 34
write(1, "    Device unallocated:\t\t 322.95"..., 36    Device
unallocated:              322.95GiB
) = 36
write(1, "    Device missing:\t\t     0.00B\n", 32    Device
missing:                0.00B
) = 32
write(1, "    Used:\t\t\t  55.61GiB\n",
23    Used:                       55.61GiB
) = 23
write(1, "    Free (estimated):\t\t 328.22Gi"..., 51    Free
(estimated):                328.22GiB      (min: 328.22GiB)
) = 51
write(1, "    Data ratio:\t\t\t      1.00\n", 29    Data
ratio:                       1.00
) = 29
write(1, "    Metadata ratio:\t\t      1.00\n", 32    Metadata
ratio:                 1.00
) = 32
write(1, "    Global reserve:\t\t 141.11MiB\t"..., 46    Global
reserve:                 141.11MiB      (used: 0.00B)
) = 46
write(1, "\n", 1
)                       = 1
ioctl(3, BTRFS_IOC_SPACE_INFO, {space_slots=0} => {total_spaces=4}) = 0
ioctl(3, BTRFS_IOC_SPACE_INFO, {space_slots=4} => {total_spaces=4,
spaces=...}) = 0
write(1, "Data,single: Size:59.00GiB, Used"..., 51Data,single:
Size:59.00GiB, Used:53.72GiB (91.06%)
) = 51
write(1, "   /dev/sda1\t  59.00GiB\n", 24   /dev/sda1     59.00GiB
) = 24
write(1, "\n", 1
)                       = 1
write(1, "Metadata,single: Size:2.00GiB, U"..., 53Metadata,single:
Size:2.00GiB, Used:1.88GiB (94.17%)
) = 53
write(1, "   /dev/sda1\t   2.00GiB\n", 24   /dev/sda1      2.00GiB
) = 24
write(1, "\n", 1
)                       = 1
write(1, "System,single: Size:32.00MiB, Us"..., 52System,single:
Size:32.00MiB, Used:16.00KiB (0.05%)
) = 52
write(1, "   /dev/sda1\t  32.00MiB\n", 24   /dev/sda1     32.00MiB
) = 24
write(1, "\n", 1
)                       = 1
write(1, "Unallocated:\n", 13Unallocated:
)          = 13
write(1, "   /dev/sda1\t 322.95GiB\n", 24   /dev/sda1    322.95GiB
) = 24
close(3)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++
wismerhill:/home/lux #




wismerhill:/home/lux # btrfs fi df  /
Data, single: total=59.00GiB, used=53.72GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=2.00GiB, used=1.88GiB
GlobalReserve, single: total=141.11MiB, used=0.00B



-----------------------------------------------------------------------


I hope the output helps to track down the reason so this is solveable.
One observation is that the Error appears on filesystems whitch are
pretty empty, less than 60% filled.

Phil :-)





Am Mittwoch, den 29.01.2020, 11:18 -0800 schrieb Paul Eggert:
On 1/29/20 5:42 AM, Wismerhill wrote:

i tried to replicate the error, but i couldn´t do it exact the same
procedure,  the Disk  was all ready filled up to 70%, and was not
empty
like when the error appeared.
I couldn´t move the files away due a space problem.
I tried with my Musicarchiv (ca. 750 GB) but the error didn´t
appear.

As a workaround for my problem (how i filled up the Disk) i created
a
borg backup local on a USB 1 TB Disk (Btrfs Filesystem) without
problems and used Rsync to copy that backup to the server (happens
befor i got your mail).
On the Server i got the same confusing freespace by then, df, and
KDE
Plasma widgets show me 0 Byte left, but Rsync finished without
error,
and the borg repositorie is  working troubleless remote.

As soon as i run again into that error i will do the procedure you
described me, and send in the requested datas.


Thanks for the heads-up. Please cc address@hidden with any
further info that you may provide.






reply via email to

[Prev in Thread] Current Thread [Next in Thread]