Re: Discussion about file format for the future

[Derek Atkins]

I do wonder...  Removing year-old incrementals from backups with lots of
small files seems to take a very long time.  I don't know if that time
is being spent in executing 'rm' or spent updating metadata..

To that end, if it IS spent in metadata, I wonder if something like a
SQLite DB would make sense to speed that up?

-derek

Patrik Dufresne <ikus060@gmail.com> writes:

> As mentioned by Robert searching for metadata is complex because you need
> to scan multiple file to actually find the right value. instead of having a
> query if we were using a database.
>
> Obviously performance-wise it's not great either because we need to scan
> multiple file.
>
> The only thing I hate about that  is lake of visibility as a compromise
> maybe we can find the most common database and add layer on top using
> command line to search in this database? To let users be autonomous.
> SQLite is probably one of those very popular and simple database.
>
> On Thu., Jun. 4, 2020, 11:03 p.m. Robert Nichols, <
> rnicholsNOSPAM@comcast.net> wrote:
>
>> On 6/4/20 11:43 AM, Patrik Dufresne wrote:
>> > But two cent on the subject is, should we really keep this filebase ? For
>> > rdiffweb, scanning the metadata files is a nightmare. When I just need a
>> > subset of the data to be displayed to the user. I always thought a
>> database
>> > could be better fit for the job. Something like a key store or similar.
>>
>> +1 from me
>>
>> The way rdiff-backup stores metadata is its worst feature, in my opinion.
>> Keeping the metadata in various text files makes analysis unnecessarily
>> complex and searches very inefficient. Inode data for hard-linked files
>> is replicated in the mirror_metadata file, except for the checksum, which
>> is stored just on the first entry for that inode, so you have to go
>> hunting for it, and make sure it is always in the right place when
>> that linking changes. That sort of thing just screams to be stored in
>> a database.
>>
>> --
>> Bob Nichols     "NOSPAM" is really part of my email address.
>>                  Do NOT delete it.
>>
>>
>>
>
>

-- 
       Derek Atkins                 617-623-3745
       derek@ihtfp.com             www.ihtfp.com
       Computer and Internet Security Consultant