The Libdar library provides a complete abstraction layer for handling Dar archives. The general operations provided are:
Note that Disk ARchive and libdar have been released under the Gnu General Public License (GPL). All code linked to libdar (statically or dynamically), must also be covered by the GPL or compatible license. Commercial use is prohibited unless otherwise explicitely agreeded with libdar's author.
This tutorial will show you how to
use the libdar API. Since release 2.0.0 the
dar command-line executable also
relies on this API, looking at it's code may provide a good
illustration on the way to use libdar,
the file src/dar_suite/dar.cpp
is the primary consumer of the libdar API.
However we will see here, step by step how
you can leverage the API for your own use.
In the following sample codes will be provided, they
are solely illustrative and is not guaranteed to compile.
Reference documentation for this API
is contained in the source code as doxygen comment which
can be extracted and "compiled" in the
the doc/html directory when compiling dar/libdar.
This API reference document is also available
online
and will be referred below as the API reference documentation.
Dar and libdar are written in
C++, and so is the libdar fundamental API. Though you can find
a Python binding in the src/python subdirectory,
which strongly follows the class hierarchy that will see here with
C++ sample code. The libdar_test.py
document rawly follows with python code what we will see
below with C++.
Feel free to contribute if you want bindings for other languages.
Only one include file is required in your program to have access to libdar:
#include <dar/libdar.hpp>
All libdar symbols are defined within
the libdar namespace. You
can either add the using namespace libdar;
statement at the beginning of your source files...
using namespace libdar;
get_version();
...
...or, as shown below, you can explicitly use the namespace in front of libdar symbols, we will use this notation in the following, which has the advantage of avoiding name conflicts and clarify origin of the symbols used, but lead to a more heavier code, less easy to read:
libdar::get_version();
...
The library makes use of exception to report unexpected conditions. These contain the reason and context the error occurred in, so you can be catch these in your code to display details of the error and its cause. All exceptions used within libdar inherit from the pure virtual class libdar::Egeneric
Most of the time you will use only one of the following two methods:
std::string & get_message() conststd::string & dump_str() constNow, messages are for human, you may need to provide different behaviors depending on the type of the error libdar has met and which triggers such exception. This is can be done by checking the type of the exception.
We will only focus on most common exception type read the API reference documentation for an exhaustive list of the exception type used by libdar:
get_message() method in that
situation would not provide all necessary details to understand
and fix the bug, so it is advised to always use dump_str()
for that specific type of exception and abort further execution regarding
libdar.
get_message()
libdar::infinint
object, which is a internal integer type that can handle arbitrary
large numbers. Today it is only used if libdar has been compiled
with --enable-mode=infinint so you are not likely to meet
it.
Some other exist and are well described in the API reference documentation. You can thus wrap the whole libdar interaction with a statement like in the following example:
try
{
// calls to libdar
...
//
}
catch(libdar::Ebug & e)
{
std::string msg = e.dump_str();
// do something with msg like for example:
std::cerr << msg << std::endl;
}
catch(libdar::Euser_abort & e)
{
// some specific treatment for this
// type of exception
...
}
catch(libdar::Egeneric & e)
{
// a global treatment for all other libdar exceptions
std::string msg = e.get_message(); << std::endl;
...
}
the libdar initialization is
performed by calling the libdar::get_version()
function.
This function can be called several time though only once is necessary, but this call has to complete before any other call to libdar.
In a multi-thread context libthreadar
initialization is not re-entrant. In other word the first
call to libdar::get_version() must complete
before any other call to libdar can take place in another
thread of the running process. Once libdar has
been initialized, you can call libdar::get_version()
concurrently from several threads at the same time without
problem.
libdar::get_version();
Libdar used some
data-structures (mutex, secured memory, etc.) that need to be released
properly before ending the process. It is important to invoke the
close_and_clean() function before exiting your program if you
had called get_version() previously. After that,
the only allowed call to libdar is get_version().
libdar::close_and_clean()
closes_and_clean()
makes the necessary for memory to be released in the
proper order.
Not calling close_and_clean() at the
end of your program may result in
uncaught exception message from libdar at the end of
the execution.
This depends on the compiler, libc and option
activated in libdar at compilation time.
All in one, at the highest level, you code should look like the following:
libdar::get_version();
try
{
try
{
// calls to libdar
// thing we will see in next
...
...
}
catch(libdar::Ebug & e)
{
std::string msg = e.dump_str();
// do something with msg like for example:
std::cerr << msg << std::endl;
}
catch(libdar::Egeneric & e)
{
std::string msg = e.get_message();
// do something with msg like for example
std::cerr << msg << std::endl;
}
}
catch(...)
{
libdar::close_and_clean();
throw;
}
libdar::close_and_clean();
libdar by itself does not make use of any signal (see signal(2) and
kill(2)). However, gpgme library with which libdar may be linked with
in order to support asymmetrical strong encryption (i.e. encryption using
public/private keys) may trigger the PIPE signal. Your application
shall thus either ignore it (signal(SIGPIPE, SIG_IGN))
or provide an adhoc handle.
By default the PIPE signal leads the receiving process to terminate.
The main components of libdar are four classes:
In the following we will first see class libdar::archive which will take most of our effort as other classes which we will see at the end are very trivial.
Except when explicitely mentioned, a given libdar object can only be manipulated by a single thread. You can however perform several operations concurrently from different thread, each having its own set of libdar objects. Though, if one thread is creating an archive by mean of an first object and at the same time another thread by mean of a second object is trying to read the same archive under construction, things might not work as expected. But this is obvious considerations we will not dig any further assuming you know what you are doing.
Creating a libdar::archive object depending on
the constructor used, leads to either:
For archive creation the constructor format is the following one:
archive::archive(const std::shared_ptr<user_interaction> & dialog,
const path & fs_root,
const path & sauv_path,
const std::string & filename,
const std::string & extension,
const archive_options_create & options,
statistics * progressive_report);
For now we will left beside some parameters, that we will see in detail later:
Once the object has been created (the constructor has returned), the archive operation has completed and a new backup has been completely written on disk.
libdar::archive my_first_backup(nullptr,
"/home/me",
"/tmp",
"first_backup",
"dar",
libdar::archive_options_create(),
std::nullptr);
The previous example has create a single sliced archive first_backup.1.dar located under /tmp. This backup contains the content of the directory /home/me and its sub-directories, without compression and without ciphering. You have guessed: compression, slicing, ciphering can be set by playing with passing an adhoc archive_option_create object to this archive constructor, something we will see in details a bit further.
Once the object my_first_backup
has been created there is only little thing we can do
with it: only archive listing or archive isolation. For archive
extraction, testing or diffing, we needs creating a new object with a
"read" constructor.
If we had allocated the archive on the heap (using new),
we would have just added the delete invocation right
after the construction of the my_first_backup
object:
libdar::archive* my_first_backup = new libdar::archive(nullptr,
"/home/me",
"/tmp",
"first_backup",
"dar",
libdar::archive_options_create(),
nullptr);
// we assume std::bad_alloc would be thrown if an allocation
// problem had occurred
// same thing if libdar throws an exception at constructor time,
// the object would not be created and would not have to be deleted.
// So now we can delete the created object:
delete my_first_backup;
During the operation we get nothing shown unless an error occurs.
To provide more visibility on the process we will use an
libdar::statistics
object passed as last argument of this constructor. Then we will
use some interesting method of class
If you have a doubt about the meaning and use of a particular counter
in a particular operation, please refer to API reference documentation
for class libdar::statistics, the private fields
corresponding to these counter are explicitly defined there.
libdar::statistics stats ;
libdar::archive my_first_backup(nullptr,
"/home/me",
"/tmp",
"first_backup",
"dar",
libdar::archive_options_create(),
& stats );
// in another thread we can see the progression:
std::cout << stats .get_treated_str()
<< " file(s) saved" << std::endl;
std::cout << stats .get_errored_str()
<< " file(s) failed to backup" << std::endl;
std::cout << stats .get_ea_treated_str()
<< " Extended Attributes saved" < std::endl;
in the previous example, we have created an temporary object of class libdar::archive_options_create and passed it on-fly to the archive constructor without modifying it. Thus we used the default options for this operations. But a lot of options are available, each one can be modified by a specific method of this class Follow is a subset of the available options. We won't details them all, but you can refer the doxygen documentation for class libdar::archive_options_create for additional information.
void set_reference(std::shared_ptr<libdar::archive> ref_arch)void set_selection(const libdar::mask & selection)void set_subtree(const libdar::mask & subtree)void set_allow_over(bool allow_over)void set_warn_over(bool warn_over)void set_info_details(bool info_details)void set_display_treated(bool display_treated, bool only_dir)void set_display_skipped(bool display_skipped)void set_display_finished(bool display_finished)void set_pause(const libdar::infinint & pause)void set_empty_dir(bool empty_dir)void set_compression(libdar::compression compr_algo)void set_compression_level(libdar::U_Icompression_level)void set_slicing(const libdar::infinint & file_size, const libdar::infinint & first_file_size)void set_ea_mask(const libdar::mask & ea_mask)void set_execute(const std::string & execute)void set_crypto_algo(libdar::crypto_algocrypto)void set_crypto_pass(const libdar::secu_string & pass)void set_compr_mask(const libdar::mask & compr_mask);void set_min_compr_size(const libdar::infinint & min_compr_size)void set_nodump(bool nodump)void set_exclude_by_ea(const std::string & ea_name)void set_what_to_check(libdar::comparison_fields what_to_check)void set_hourshift(const libdar::infinint & hourshift)void set_empty(bool empty)void set_alter_atime(bool alter_atime)void set_furtive_read_mode(bool furtive_read)void set_same_fs(bool same_fs)void set_snapshot(bool snapshot)void set_cache_directory_tagging(bool cache_directory_tagging)void set_fixed_date(const libdar::infinint & fixed_date)void set_slice_permission(const std::string & slice_permission)void set_slice_user_ownership(const std::string & slice_user_ownership)void set_slice_group_ownership(const std::string & slice_group_ownership)void set_retry_on_change(const libdar::infinint & count_max_per_file, const libdar::infinint & global_max_byte_overhead)void set_security_check(bool check)void set_user_comment(const std::string & comment)void set_hash_algo(libdar::hash_algo hash)void set_slice_min_digits(libdar::infinint val)void set_backup_hook(const std::string & execute, const mask & which_files);void set_delta_diff(bool val)void set_delta_signature(bool val)void set_delta_mask(const libdar::mask & delta_mask)First you may have find some new libdar types (=classes) in arguments, we will briefly explain how to set them:
C++11 shared smart-pointer to an existing libdar::archive object. We will see how to use it next when performing differential backup
can be set from a classical unsigned int, unsigned long or other unsigned integer type, so for you this is like another unsigned integer type
This class is a top of a class hierarchy containing several classes provided with libdar. It allows you to define a filtering mecanism for the feature wher it is used. We will see how to use it in a further paragraph of this tutorial.
It is an enumeration which values are (among others not listed here):
libdar::compression::gziplibdar::compression::bzip2libdar::compression::xzlibdar::compression::lzoA the opposite of the libdar::infinint this is not a class but an alias to the system classical unsigned integer. Depending on the operating system and CPU this might point to unsigned long or unsigned long long or other equivalent type.
This is an enumeration with values like:
libdar::crypto_algo::scramblinglibdar::crypto_algo::blowfishlibdar::crypto_algo::aes256libdar::crypto_algo::twofish256libdar::crypto_algo::serpent256libdar::crypto_algo::camellia256This is a class used to securely storing password and sensible cryptographic information. It can be setup from a char* or better, from a filedescriptor. its main constructor is:
This is an enumeration with values like:
libdar::comparison_fields::alllibdar::comparison_fields::ignore_ownerlibdar::comparison_fields::mtimelibdar::comparison_fields::inode_typeFollows a variant of the previous backup creation example, here we set some options to a given value:
libdar::archive_options_create opt;
opt.set_allow_over(false);
// forbids slice overwriting
opt.set_display_finished(true);
// show a summary after each completed directory
opt.set_slicing(1024000,
2000);
// slices of 1000 kiB initial slice of 2000 bytes
opt.set_pause(2);
// pause every two slices
opt.set_execute("echo slice %N completed");
// command executed after each slice
opt.set_crypto_algo(libdar::crypto_algo::aes256);
// providing an empty secu_string leads dar
// interactively ask the passphrase in a secure manner
opt.set_crypto_pass(secu_string());
// But this previous call is useless as en empty secu_string is the default
// though one could have setup a secu_string from a std::string
// this way:
std::string my_pass("hello world!");
libdar::secu_string my_secupass(my_pass.c_str(), my_pass.size());
opt.set_crypto_pass(my_secupass);
opt.set_compression(libdar::compression::xz);
opt.set_compression_level(6);
opt.set_min_compr_size(10240);
// not trying compressing file smaller than 10 kiB
// now we have the opt option object ready
// we can proceed to the archive creation using it:
libdar::archive my_first_backup(nullptr,
"/home/me",
"/tmp",
"first_backup",
"dar",
opt,
nullptr);
And of course, you can use both libdar::statistics and libdar::archive_options_create at the same time, when creating a backup
Maybe you have guessed? Compared to the previous operation (full backup) doing an differential or incremental backup will only ask to open, in read mode, an existing archive and pass this object as argument of class archive_options_create::set_reference() seen just above.
The read-only constructor for class archive is the following:
archive::archive(const std::shared_ptr<user_interaction> & dialog,
const path & chem,
const std::string & basename,
const std::string & extension,
const archive_options_read & options);
same as before:
user_interaction class
// first we open the previously created archive in read mode:
std::shared_ptr<libdar::archive> ref_archive ;
ref_archive = std::make_shared<libdar::archive>(nullptr,
"/home/me",
"first_backup",
"dar",
archive_create_options_read());
// for clarity we separated the creation of the
// archive object used as archive of reference from
// the archive object which creation will perform
// differential backup (see below). We could have made
// in one step using an anonymous temporary object
// (in place of ref_archive)
// invoking directly std::make_shared
// when calling set_reference() below:
libdar::archive_options_create opt;
opt.set_reference(ref_archive );
libdar::archive my_second_backup(nullptr,
"/home/me",
"/tmp",
"diff_backup",
"dar",
opt,
nullptr);
creating a incremental backup is exactly the same, the difference is the nature of the archive of reference. We used to describe a differential backup one that has a full backup as reference, while an incremental backup has another incremental or differential backup as reference (not a full backup).
Archive listing operation consist of the creation of an archive object
in read mode as we just did above for ref_archive and
invoking a method on that newly object to see all or a sub-directory
content of the archive. Before
looking at the listing method let's zoom on the class
libdar::archive_create_options_read which
we just skip over previously.
The same as the class archive_options_create detailed above, the
class archive_options_read has a constructor without argument that sets
the different options to their default value. You can change them one
by one by mean of specific methods. The most usual ones are:
set_execute() runs a command before reading a new slice of the
archive. See API reference documentation for details. You will meet
class archive_options_create in order to test an archive,
compare an archive with filesystem, isolate an archive and repair an archive.
There is several ways to read an given archive contents:
making use of a callback function that will be called in turn for each entry of the archive even special entries that flag the end of a directory and the next entry will be located in the parent directory:
void op_listing(libdar::archive_listing_callback callback,
void *context,
const libdar::archive_options_listing & options) const;
using the same callback but only for the different entries of a given directory, directory that has to exist in the archive of course. It returns false when the end of the directory has been reached:
bool get_children_of(libdar::archive_listing_callback callback,
void *context,
const std::string & dir,
bool fetch_ea = false);
like previous listing a given directory content but returning a vector of objects libdar::list_entry that provide detailed information about each entry, no callback is used here:
const std::vector<libdar::list_entry> get_children_in_table(const std::string & dir,
bool fetch_ea = false) const;
For the two first methods you have to define a callback function of the following form:
void (*)(const std::string & the_path,
const list_entry & entry,
void *context);
This callback will receive as argument the full path of the object, a libdar::list_entry object providing much details on it and the "context" value passed as argument of archive::op_listing() or archive::get_children_of()
In the following example we will use only a few methods of class libdar::list_entry that are available to get detail of a given entry of an archive, feel free to explore this class's documentation for to get all details:
// we first create a read-mode archive object that will be used
// in the three following examples,
libdar::archive_options_read opt;
opt.set_info_details(true);
opt.set_execute("echo 'about to read slice %p/%b.%N.%e with context %c'");
libdar::archive my_backup(nullptr, // this is user_interaction we will see further
"/home/me",
"diff_backup",
"dar",
opt);
// we will also illustrate here the use of libdar::archive_options_read
// inside the callback function we will define here:
void my_listing_callback(const std::string & the_path,
const libdar::list_entry & entry,
void *context)
{
std::cout << the_path;
if(entry.is_dir())
std::cout << " is a directory";
std::cout << " with permission " << entry.get_perm();
std::cout << " file located in slices " << entry.get_slices().display();
std::cout << std::endl;
// yep, we do not need context, this
// is available if you need it though
if(entry.is_eod())
{
// only op_listing() provides such type of object
// which occurs when we reached the End Of Directory
// next entry will be located in the parent directory.
//
// Note for op_listing: when reading a directory we recurs in it,
// meaning that the next entry this callback will be
// invoked for will be located in that directory
//
// for get_children_of() no recursion or eod object is
// performed about directory. The next entry following
// a directory is still located in the same parent directory
// which when fully read stops the get_children_of() routine
// at the difference of op_listing() which parse the whole
// directory tree.
//
// For example, reading a empty directory will provide
// that directory info, then an eod object a the next
// callback invocation.
}
}
These two objects my_backup and my_listing_callback()
we just defined will be used in the following examples.
First possibility: we can pass nullptr as callback function to archive::op_listing, all will be displayed in stdout
my_backup.op_listing(nullptr, // no callback function
nullptr, // we don't use the context here
archive_options_listing()) // and use default listing options
Second possibility: we use the callback defined previously:
my_backup.op_listing(my_listing_callback,
nullptr, // we still don't use the context here
archive_options_listing()) // and still the default listing options
In complement of both previous variant we can of course set non default listing options
libdar::archive_options_listing opt;
opt.set_filter_unsaved(true);
// skip entry that have not been saved since the archive of reference
opt.set_slice_location(true);
// necessary if we want to have slicing information available in the callback function
opt.set_fetch_ea(false);
// this is the default. Set it to true if you
// want to use list_entry::get_ea_reset_read()/get_ea_next_read()
my_backup.op_listing(my_listing_callback,
nullptr, // we still don't care of context here
opt); // and still the default listing options
// With this method we only list one directory
my_backup.get_children_of(my_listing_callback,
nullptr, // we still don't care of context here
"", // we read the root directory of the archive
true); // and ask for EA retrieval, but as we do not
// use list_entry::get_ea_read_next() in the
// callback this is just wasting CPU and memory
// or course if you have a sub-directory /home/me/.gnupg/private-keys-v1.d
// in your home directory and you want to check how it is saved in the
// archive, as we defined the root of the backup as /home/me and as you
// always have to pass a relative path (no leading /) you could do that by
// calling the following:
my_backup.get_children_of(my_listing_callback,
nullptr,
".gnupg/private-keys-v1.d");
// still listing a single directory but this time without callback function:
my_backup.init_catalogue();
// necessary to fill read the whole catalogue in memory
// in particular if archive has been opened in sequential read mode
std::vector<libdar::list_entry> result = my_backup.get_children_in_table(".gnupg/private-keys-v1.d");
// now reading the std::vector
std::vector<libdar::list_entry>::iterator it = result.begin();
while(it != result.end())
{
if(it->is_dir())
std::cout << " is a directory";
std::cout << " with permission " << it->get_perm();
std::cout << " located in slices " << it->get_slices().display();
std::cout << std::endl;
}
As seen for listing operation we assume a archive object has been create in read mode. Testing the coherence of the relative archive files on disk is done by calling the libdar::op_test method:
libdar::statistics op_test(const libdar::archive_options_test & options,
libdar::statistics * progressive_report);
You may have recognized the
libdar::statistics type we saw for archive creation. It is present as
argument and the provided libdar::statistics object can be read during
the whole testing operation by another thread. But if you just want the
to know the result, you'd better just use the returned value as it
makes the operation quicker due to the absence of multithread
management.
// for the exercise, we will change some default options:
archive_options_test opt;
opt.set_info_details(true); // to have a verbose output
libdar::statistics stats;
stats = my_backup.op_test(nullptr, // still the user_interaction we will see further
opt; // the non default options set above
nullptr); // we will just use the returned value
std::cout << stats.get_treated_str() << "file(s) tested" << std::endl;
std::cout << stats.get_errored_str() << " file(s) with errors" << std::endl;
std::cout << stats.get_ea_treated_str() << " Extended Attributes tested" << std::endl;
As simple as previously, but using the archive::op_diff method:
statistics op_diff(const path & fs_root,
const archive_options_diff & options,
statistics * progressive_report);
Over the type of the option field, you see the fs_root argument which define which directory of the filesystem to compare the archive to
// for the exercise, we will change some default options:
archive_options_diff opt;
opt.set_info_details(true); // to have a verbose output
opt.set_what_to_check(libdar::comparison_fields::ignore_owner);
// this option above will consider equal two files which
// only change due to user or group ownership difference
// by default any difference will be considered a difference
(void)my_backup.op_diff("/home/me",
opt; // the non default options set above
nullptr); // not using it for this example
Still as simple as previously, but using the archive::op_isolate method:
void op_isolate(const path &sauv_path,
const std::string & filename,
const std::string & extension,
const archive_options_isolate & options);
You will find similitude with the archive creation though here this is not a constructor
// for the exercise, we will change some default options:
archive_options_isolate opt;
opt.set_warn_over(false);
// by default overwriting is allowed by a warning is issued first
// here overwriting will take place without warning
opt.set_compression(libdar::compression::gzip);
opt.set_compression_level(9);
// this is the default
opt.set_min_compr_size(10240);
// not trying compressing file smaller than 10 kiB
my_backup.op_isolate("/tmp",
"CAT_diff_backup",
"dar",
opt); // the non default options set above
// have you noted? There is no libdar statistics field returned nor as argument.
Quite as simple as previously, here we use the archive::op_extractmethod:
statistics op_extract(const path & fs_root,
const archive_options_extract & options,
statistics *progressive_report);
// as we still do not have seen masks, we will restore all files contained in the backup
// such mask would be provided to the
// archive_options_extract::set_selection() and/or
// to the archive_options_extract::set_subtree() methods
// to precisely define what files to restore
archive_options_extract opt;
opt.set_dirty_behavior(false, false); // dirty files are not restored
(void) my_backup.op_extract("/home/me/my_home_copy",
opt,
nullptr); // we have seen previously how to use statistics
Here we will need two archive objects open in read-mode and we will invoke a specific archive constructor passing these two objects as argument, once the constructor will have completed the merging operation will be done.
archive(const std::shared_ptr<user_interaction> & dialog,
const path & sauv_path,
std::shared_ptr<archive> ref_arch1,
const std::string & filename,
const std::string & extension,
const archive_options_merge & options,
statistics * progressive_report);
// assuming you have two backups:
// the first is /tmp/home_backup.*.dar
// the second is /var/tmp/system_backup.*.dar
// we will create /tmp/merged.*.dar as result of the merging
// of these two backups
// 1 - first things first: opening the first backup
libdar::archive_options_read opt;
opt.set_info_details(true);
opt.set_execute("echo 'about to read slice %p/%b.%N.%e with context %c'");
std::shared_ptr<libdar::archive> home_backup(new libdar::archive(nullptr, // this is user_interaction we'll see further
"/tmp",
"home_backup",
"dar",
opt));
// 2 - opening the second backup
std::shared_ptr<libdar::archive> system_backup(new libdar::archive(nullptr,
"/var/tmp",
"system_backup",
"dar",
opt);
// 3 - setting up the options for merging operation
libdar::archive_options_merge opt_merge;
opt_merge.set_auxiliary_ref(system_backup);
// while merging the second backup is optional, where from the use of option for it
opt_merge.set_slicing(1048576, 0); // all slice would have 1 MiB at most
opt_merge.set_compression(libdar::compression::bzip2);
opt_merge.set_keep_compressed(true);
opt_merge.set_user_comment("archive resulting of the merging of home_backup and system_backup");
opt_merge.set_hash_algo(libdar::hash_algo::sha512); // will generate on-fly hash file for each slice
// 4 - now performing the merging operation
libdar::archive merged(nullptr, // still the user_interaction we will see further
"/tmp",
home_backup, // first back is mandatory, not part of options
"merged",
"dar",
opt_merge,
nullptr); // progressive_report, we don't use here
Decremental backup is an operation that, from two full backups, an old and a recent one, creates a backward differential backup corresponding to the old full backup, which difference is based on the new full backup. In other words, instead of keeping two full backups, you can keep the latest and replace the oldest by its decremental counterpart. This will save you space while letting you restore as if you had the old full backup by restoring first the recent (full) backup then the decremental backup.
Creating a decremental backup is exactly the same as creating a merging
backup, you need just to set the
archive_options_merge::set_decremental_mode() before proceeding to the
merging. To avoid duplication we will just illustrate the last step of
the previous operation modified for decremental backup:
// creating the two read-only backup as for merging operation
// the only difference is that here both backups are mandatory
std::shared_ptr<libdar::archive> old_full_backup(...); // not detailing this part
std::shared_ptr<libdar::archive> new_full_backup(...); // not detailing this part
// setting up the options for a decremental operation
libdar::archive_options_merge opt_merge;
opt_merge.set_decremental_mode(true);
opt_merge.set_auxiliary_ref(new_full_backup);
// now performing the merging operation (here decremental backup)
libdar::archive merged(nullptr, // still the user_interaction we will see further
"/tmp",
old_full_backup,
"decremental_backup",
"dar",
opt_merge,
nullptr); // progressive_report we don't use here
If an archive has been truncated due to lack of disk space and if sequential marks (aka tape marks) had not been disable, it is possible to rebuild sane archive beside this truncated one.
We just need to invoke a specific libdar::archive constructor which form follows:
archive(const std::shared_ptr<user_interaction> & dialog,
const path & chem_src,
const std::string & basename_src,
const std::string & extension_src,
const archive_options_read & options_read,
const path & chem_dst,
const std::string & basename_dst,
const std::string & extension_dst,
const archive_options_repair & options_repair);
You should now be familiarized with the different types and variable uses. As you can note, this constructor takes in charge the work to read the damaged archive because if the archive is corrupted a normal constructor would fail, so you won't have to do it first. As always, this constructor will end only once the operation will have completed, that's to say at the end of the reparing.
// assuming the archive /tmp/home_backup.*.dar is damaged
// and you want to have repaired archive as /tmp/home_backup_repaired.*.dar
libdar::archive repaired(nullptr, // still the user_interaction we have not yet seen
"/tmp"
"home_backup",
"dar",
archive_options_read(),
"/tmp",
"home_backup_repaired",
"dar",
archive_options_repair());
// we have not done fancy things with the two option classes, but we did above
// enough time for you get all the necessary information from the API reference
// documentation
we have covered the different operations the class libdar::archive
can be used for, still remains some concepts to view:
Then we will see the three other more simple classes:
This will be the subject of the following chapters, but for now,
maybe you remember that we had to initialize libdar before use,
by calling libdar::get_version()?
This routine also exists with arguments that will provide as
its name suggests the libdar version:
void get_version(U_I & major, U_I & medium, U_I & minor, bool init_libgcrypt = true);
It is advised to use this form to fetch the libdar version major, medium and minor numbers to be sure the library you've dynamically linked with is compatible with the features you will be using:
If you use libgcrypt beside libdar in your application you should initialize libgcrypt and not let it be done by libdar the latest argument of this form should be set to false in that case, according to the libgcrypt documentation which indicates that libgcrypt should normally (always) be initialized directly from the application not from an intermediate library.
Follows an example of test that can be performed while initializing libdar:
U_I major, medium, minor;
libdar::get_version(major, medium, minor);
if(maj != libdar::LIBDAR_COMPILE_TIME_MAJOR ||
med < libdar::LIBDAR_COMPILE_TIME_MEDIUM)
{
std::cout << "libdar version we link with is too old for this code" << std::endl;
// throw an exception or anything else appropriate to that condition:
throw "something";
}
Once we have called one of the get_version*
function it is possible to access the list of features activated at
compilation time thanks to a set of function located in the
compile_time nested namespace inside libdar:
void my_sample_function()
{
bool ea = libdar::compile_time::ea();
bool largefile = libdar::compile_time::largefile();
bool nodump = libdar::compile_time::nodump();
bool special_alloc = libdar::compile_time::special_alloc();
U_I bits = libdar::compile_time::bits();
// bits is equal to zero for infinint,
// else it is equal to 32 or 64 depending on
// the compilation mode used.
bool thread = libdar::compile_time::thread_safe();
bool libz = libdar::compile_time::libz();
bool libbz2 = libdar::compile_time::libbz2();
bool liblzo = libdar::compile_time::liblzo();
bool libxz = libdar::compile_time::libxz();
bool libcrypto = libdar::compile_time::libgcrypt();
bool furtive_read = libdar::compile_time::furtive_read();
// for details see the compile_time namespace in the API reference documentation
}
we have seen std::shared_pointer on class libdar::user_interaction
previously but did not used this feature.
class libdar::user_interaction defines the way libdar interact with the user during an operation, like an archive creation, restoration, testing and so on. Only four types of interaction are used by libdar:
void message(const std::string & message);
void pause(const std::string & message);
std::string get_string(const std::string & message, bool echo);
secu_string get_secu_string(const std::string & message, bool echo);
By default an inherited class of libdar::user_interaction, called
libdar::shell_interaction, is used and implements these
four type of exchange by mean of text terminal:
For a GUI you will probably not want stdin and stdout to be used. Instead of that you have the possibility to implement your own inherited class from user_interaction. This one should look like the following:
class my_user_interaction: public libdar::user_interaction
{
protected:
// display of informational message
virtual void inherited_message(const std::string & message) override;
// display of a question and returns the answer from user as true/false
virtual bool inherited_pause(const std::string & message) override;
// display the message and returns a string from the user,
// with or without display what the user typed (echo)
virtual std::string inherited_get_string(const std::string & message, bool echo) override;
// same as the previous be the user provided string is returned as secu_string
virtual libdar::secu_string inherited_get_secu_string(const std::string & message, bool echo) override;
};
As an alternative to defining
your own inherited class from libdar::user_interaction,
libdar provides a class called user_interaction_callback
which is an implementation of the user interaction, based on callback
functions.
You will need to implement four callback functions:
using message_callback = void (*)(const std::string &x, void *context);
using pause_callback = bool (*)(const std::string &x, void *context);
using get_string_callback = std::string (*)(const std::string &x, bool echo, void *context);
using get_secu_string_callback = secu_string (*)(const std::string & x, bool echo, void *context);
Then you can create an libdar::user_interaction_callback object using this constructor:
user_interaction_callback(message_callback x_message_callback,
pause_callback x_answer_callback,
get_string_callback x_string_callback,
get_secu_string_callback x_secu_string_callback,
void *context_value);
Here follows an example of use:
void my_message_cb(const std::string & x, void *context)
{
std::cout << x << std::endl;
}
bool void my_pause_cb(const std::string & x, void *context)
{
char a;
std::cout << x << endl;
std::cin >> a;
return a == 'y';
}
std::string my_string_cb(const std::string & x, bool echo, void *context)
{
// to be defined
}
libdar::secu_string my_secu_string_cb(const std::string & x, bool echo, void *context)
{
// to be defined
}
// eventually using a context_value that will be passed to the callback of the object
void *context_value = (void *)(& some_datastructure);
std::shared_ptr<libdar::user_interaction> my_user_interaction(new libdar::user_interaction_callback(my_message_cb,
my_pause_cb,
my_string_cb,
my_secu_string_cb,
context_value));
You can also find the predefined classes libdar::user_interaction_blind which always says no in name of the user displays nothing and provide empty strings, as well as libdar::shell_interaction_emulator which given a user_interaction object send to it the formatted information as if it was a shell_interaction object, leading one to emulate libdar default behavior under any type of "terminal".
libdar::user_interaction inherited classes
provided by libdar are not designed to be manipulated by more
than one thread at a time. The use of std::shared_ptr
is only here to let the caller not have to manage
such object and let libdar release it when no more needed or
to let the caller reuse the same user_interaction object for a
subsequent call to libdar which would not be possible if a
std::unique_ptr was used instead.
Now if you design your own user_interaction inherited class and provide them mecanism (mutex, ...) that allow them to be used simultaneously by several thread there is no issue to pass a such object as argument to different libdar object used by different threads running at the same time.
Mask are used to define which string
will be considered and which will not. Libdar implements masks as
several classes that all inherit from a virtual class named
libdar::mask that defines the way masks are used
(interface class). This class defines the
bool mask::is_covered(const std::string & expression) const method
which libdar uses to determine whether a given string matches or
not a mask and thus whether the corresponding entry (filename, EA, directory path
depending on the context), is eligible or not for an operation.
Strings applied to masks may correspond to filename only, to full path or maybe to other things (like Extended Attributes). That's in the context where the mask is used that the string meaning take place, thing we will see further.
There is several different basic masks classes you can use to build fairly complex masks, while it is possible you should not need to define you own mask classes, if the need arises, please contact libdar developer if you thing an additional class should take place beside the following ones:
| class name | behavior |
|---|---|
| class libdar::bool_mask | boolean mask, either always true or false (depending on the boolean passed at its constructor), it matches either all or none of the submitted strings |
| class libdar::simple_mask | matches strings as done by the shell on the command lines (see "man 7 glob") |
| class libdar::regular_mask | matches regular expressions (see "man 7 regex") |
| class libdar::not_mask | negation of another mask (the mask given at construction time) |
| class libdar::et_mask | makes an *AND* operator between two or more masks |
| class libdar::ou_mask | makes the *OR* operator between two or more masks |
| class lbdar::simple_path_mask | matches whether the string to evaluate is subdirectory of, or is the directory itself that has been given at construction time. |
| class libdar::same_path_mask | matches if the string is exactly the given mask (no wild card expression) |
| class libdar::exclude_dir_mask | matches if string is the given string or a sub directory of it |
| class libdar::mask_list | matches a list of files defined in a given file |
Let's play with some masks:
// all files will be elected by this mask
libdar::bool_mask m1(true);
// all string that match the glob expression "A*~" will match.
// the second argument of the constructor tell whether the match is case sensitive so here,
// any file beginning by 'A' or by 'a' and ending by '~' will be selected by this mask:
libdar::simple_mask m2(std::string("A*~"), false);
// m3 is the negation if m2. This mask will thus match
// any string that does not begin by 'A' or 'a' or does not finishing by '~'
libdar::not_mask m3(m2);
// this mask matches any string that is a subdirectory of "/home/joe"
// and any directory that contains /home/joe, meaning
// "/", "/home", "/jome/joe" and any subdirectory are matched.
// here, the second argument is also case sensitivity (so
// "/HoMe" will not be selected by this mask as we set it to "true".
libdar::simple_path_mask m4 = simple_path_mask("/home/joe",
true);
// now let's do some more complex things:
// m5 will now match only strings that are selected by both m2 AND m4
libdar::et_mask m5;
m5.add_mask(m2);
m5.add_mask(m4);
// we can make more interesting things like this, where m5 will select files
// that match m2 AND m4 AND m3. But as m3 is not(m2), m5 will never
// match any file... but the idea here is to see the flexibility of use:
m5.add_mask(m3);
// but we could do the same with an "ou_mask" and would get a silly
// counterpart of m1 (a mask that matches any files)
libdar::ou_mask m6;
m6.add_mask(m2);
m6.add_mask(m4);
m6.add_mask(m3);
// lastly, the NOT, AND and OR operation can be used recursively.
// Frankly, it's even possible to have masks referring each other!
libdar::not_mask m7(m6);
m6.add_mask(m7);
// now that's to you to build something that makes sense...
The idea here is not to create object manually, but to link their creation to the action and choices the user makes from the user interface (Graphical User Interface of your application, for example)
Now that you've seen the power of these masks, you should know that in libdar masks are used at several places:
A first place is to select files against their
names (without path information) this the argument of the set_selection()
method of libdar::archive_options_* classes.
The mask here does not apply to directories.
A second place is to select files against their
path+name and it applies here to all type of files including
directories, this is the argument of the set_subtree()
method of libdar::archive_options_* classes.
So with it, you can prune directories, or in any other way restrict the
operation to a particular subdirectory, as well as to a particular
plain file for example.
Important note: about this second mask, what
your own mask will be compared to by libdar is the absolute path of
the file under consideration. If you want to exclude /usr/local/bin
from the operation whatever is the fs_root
value (which correspond the -R option of dar) using here a
libdar::simple_mask("/usr/local/bin")
as argument of libdar::archive_options_*::get_subtree()
will do the trick.
An exception is the archive testing operation, which has no
fs_root argument (because the operation is not
relative to an existing filesystem), however the
subtree argument exist to receive a mask for
comparing the path of file to include or exclude from the
testing operation. In this case the situation is as if the
fs_root was set to the value "<ROOT>".
For example, masks will be compared to "<ROOT>/some/file"
when performing an archive test operation.
Instead of using explicit string "<ROOT>" you can use libdar::PSEUDO_ROOT predifined std::string variable
libdar::archive_options_* classes.
It is applied to the full EA name in the form
<domain>.<name> where <domain>
is any string value like but not limited to the usual "user"
or "system" domains.
libdar::archive_options_*
classes. it is works the same as
set_selection() methods seen above,
based only to filename without any path
consideration.
libdar::archive_option_create
class. I has to be used the same as
set_subtree(). For more about this feature see
the backup-hook feature in dar man page (-<, -> and -= options).
If the POSIX thread support is available, libdar will be built in a thread-safe manner, giving you the possibility to have several threads using libdar at the same time (but on different objects except concerning the libdar::statistics which can be shared between threads). You may then wish to interrupt a given thread. But aborting a thread form the outside (like sending it a KILL signal) will most of the time let some memory allocated or even worse can lead to dead-lock situation, when the killed thread was inside a critical section and had not got the opportunity to release a mutex. For that reason, libdar proposes a set of calls to abort any processing libdar call which is ran by a given thread.
// next is the thread ID in which we want to have lidbar call canceled
// here for simplicity we don't describe the way the ID has been obtained
// but it could be for example the result of a call to pthread_self() as
// defined in <pthread.h> system header file
pthread_t thread_id = 161720;
// the most simple call is:
libdar::cancel_thread(thread_id);
// this will make any libdar call in this thread be canceled immediately
// but you can use something a bit more interesting:
libdar::cancel_thread(thread_id, false);
// this second argument is true for immediate cancellation and can be ommited in
// that case. But using false instead leads to a delayed cancellation,
// in which case libdar aborts the operation
// but produces something usable, espetially if you were performing a backup.
// You then get a real usable archive which only contains files saved so far, in place
// of having a broken archive which misses a catalogue at the end. Note that this
// delayed cancellation needs a bit more time to complete, depending on the
// size of the archive under process.
As seen above, cancellation can be very simple. What now succeeds when
you ask for a cancellation? Well, an exception of type Ethread_cancel
is thrown. All along his path, memory is released and mutex are freed.
Last, the exception appears to the libdar caller. So, you can catch it
to define a specific comportment. And if you don't want to use
exceptions a special returned code is used.
try
{
libdar::archive my_arch(...);
...
}
catch(libdar::Ethread_cancel & e)
{
... do something specific when thread has been canceled;
}
Some helper routines are available to know the cancellation status for a particular thread or to abort a cancellation process if it has not yet been engaged.
thread_t tid;
// how to know if the thread tid is under cancellation process?
if(libdar::cancel_status(tid))
std::cout << "thread cancellation is under progress for thread : "
<< tid << std::endl;
else
std::cout << "no thread cancellation is under progress for thread : "
<< std::endl;
// how to cancel a pending thread cancellation ?
if(libdar::cancel_clear(tid))
std::cout << "pending thread cancellation has been reset, thread "
<< tid << " has not been canceled"
<< std::endl;
else
std::cout << "too late, could not avoid thread cancellation for thread "
<< tid
<< std::endl;
Last point, back to the libdar::Ethread_cancel
exception, this class has two methods you may find useful, when you
catch it:
try
{
... some libdar calls
}
catch(libdar::Ethread_cancel & e)
{
if(e.immediate_cancel())
std::cout << "cancel_thread() has been called with \"true\" as second argument"
<< std::endl;
else
std::cout << "cancel_thread() has been called with \"false\" as second argument"
<< std::endl;
U64 flag = e.get_flag();
... do something with the flag variable...
}
// what is this flag stored in this exception?
// You must consider that the complete definition of cancel_thread() is the following:
// void cancel_thread(pthread_t tid, bool immediate = true, U_64 flag = 0);
// thus, any argument given in third is passed to the thrown Ethread_cancel exception,
// value which can be retrieved thanks to its get_flag() method. The value given to this
// flag is not used by libdar itself, it is a facility for user program to have the possibility
// to include additional information about the thread cancellation.
// supposing the thread cancellation has been invoked by:
libdar::cancel_thread(thread_id, true, 19);
// then the flag variable in the catch() statement above would have received
// the value 19.
For more about dar_manager, please read the man page where are described in detail its available features.
To get dar_manager features you need to use the class database. Most of the methods of the database class make use options. The same as what has been seen with class archive a auxiliary class is used to carry these options.
Two constructor are available. The first creates a brand-new but empty database in memory:
database(const std::shared_ptr<user_interaction> & dialog);
As seen for libdar::archive dialog can be set to a null pointer if the
default interaction mode (stdin/stdout/stderr) suits your need.
The second constructor opens an existing database from filesystem and stores its contents into memory ready for further use and actions:
database(const std::shared_ptr<user_interaction> & dialog,
const std::string & base,
const database_open_options & opt);
libdar::archive we can use an default temporary object to use default
option
database:
std::shared_ptr<libdar::user_interaction> ui_ptr; // points to null
libdar::database base(ui_ptr);
// we have created an empty database (no archive in it) called "base"
libdar::database other(ui_ptr, // we can reuse it as it points to nullptr
"/tmp/existing_base.dmd",
libdar::database_open_options());
// we have created a database object called "other" which contains
// (in RAM) all information that were contained in the
// database file "/tmp/existing_base.dmd"
libdar::database_open_option opt;
opt.set_partial(true);
opt.set_warn_order(false);
libdar::database other2(ui_ptr,
"/tmp/existing_base.dmd",
opt);
// we have created yet another database object called "other2" which differs
// from "other" by the option we used. While "other" is a fully loaded
// database, "other2" is a partial database. This notion is explained
// below
In the following we will indicate whether a database operation can be applied to a partially loaded database or not. All operation can be applied to a fully loaded databse.
A database can be open in read-write mode partially loaded (still read-write) mode and last in partially loaded read-only mode. All operations are available in the first mode, but some are not in the second and even less in the third mode. We will detail which one are available in each mode:
Well, you might now say that as description this is a bit light for a tutorial, yes. In fact these call are really very simple to use, you can find a complete description in the API reference documentation. This documentation is built if doxygen is available and is put under doc/html after calling make in the source package. It is also available from dar's homepage.
dar_slave role is to read an archive while interacting with a dar process through a pair of pipes. Dar asks portion of the archive or information about the archive in the first pipe from dar to dar_slave. And dar_slave sends the requested information into the other pipe toward dar (embedded into an expected format).
Since API 6.0.x, dar_slave has an API. It is implemented by the class libdar::libdar_slave. You need firs to create an object using the following constructor:
libdar_slave(std::shared_ptr<user_interaction> & dialog,
const std::string & folder,
const std::string & basename,
const std::string & extension,
bool input_pipe_is_fd,
const std::string & input_pipe,
bool output_pipe_is_fd,
const std::string & output_pipe,
const std::string & execute,
const infinint & min_digits);
Once the object is created, you will need to call the libdar_slave::run() method which will end when the dar process at the other end will no more need of this slave:
libdar::libdar_slave slave(nullptr,
"/tmp",
"first_backup",
"dar",
false,
"/tmp/toslave", // assuming this is an existing named pipe
false,
"/tmp/todar", // assuming this is also an existing named pipe
"echo 'reading slice %p/%b.%N.%e in context %c'",
0);
slave.run();
// once run() has returned, you can launch it again for another process
// it will continue to provide access to the /tmp/first_backup.*.dar archive
dar_xform creates a copy of a given archive modifying its slicing. it does not require decompressing nor deciphering the archive to do so. There is different constructor depending whether the archive is read from filesystem, from a named pipe of from a provided file descriptor
libdar::libdar_xform(const std::shared_ptr<user_interaction> & ui,
const std::string & chem,
const std::string & basename,
const std::string & extension,
const infinint & min_digits,
const std::string & execute);
libdar_xform(const std::shared_ptr<user_interaction> & dialog,
const std::string & pipename);
libdar_xform(const std::shared_ptr<user_interaction> & dialog,
int filedescriptor);
Once the libdar::libdar_xform object is created it can copy the
referred archive to another location in another form thanks to one of
the two libdar_xform::xform_to methods. There is not link between
the constructor used and the libdar_xform::xform_to flavor used,
any combination is possible.
void xform_to(const std::string & path,
const std::string & basename,
const std::string & extension,
bool allow_over,
bool warn_over,
const infinint & pause,
const infinint & first_slice_size,
const infinint & slice_size,
const std::string & slice_perm,
const std::string & slice_user,
const std::string & slice_group,
libdar::hash_algo hash,
const libdar::infinint & min_digits,
const std::string & execute);
void xform_to(int filedescriptor,
const std::string & execute);
Here follows an example of use. We will convert a possibly multi-sliced archive to a single slice one, generating a sha512 hash file on-fly.
std::shared_ptr<libdar::user_interaction> ui_ptr; // points to null
libdar::libdar_xform transform(ui_ptr,
"/tmp",
"my_first_archive",
"dar",
0,
"echo 'reading slice %p/%b.%N.%e context is %c'");
transform.xform_to("/tmp",
"my_other_first_archive",
"dar",
false, // no overwriting allowed
true, // does not matter whether we warn or not as we do not allow overwriting
0, // no pause between slices
0, // no specific first slice
0, // no slicing at all (previous argument is thus not used anyway in that case)
"", // using default permission for created slices
"", // using default user ownership for created slices
"", // using default group ownership for created slices
libdar::hash_algo::sha512, // the hash algo to use (for no hashing use hash_none instead)
0, // min_digits ... not using this feature here where from we use "0"
"echo 'Slice %p/%b.%N.%e has been written. Context is %c'");
All the symbols found in the libdar API defined from <dar/libdar.h>. So you should only need to include this header. If the header file is not located in a standard directory, in order to compile your code, you may need some extra flags to pass to the compiler (like -I/opt/...). The pkg-config tool can help here to avoid system dependent invocation:
shell prompt > cat my_prog.cpp
#include <dar/libdar.h>
main()
{
libdar::get_version(...);
...
}
shell prompt > gcc `pkg-config --cflags libdar` -c my_prog.cpp
Of course, you need to link your program with libdar. This is done by adding -ldar plus other library libdar can rely on like libz, libbzip2, liblzo or libgcrypt, depending on the feature activated at compilation time. Here too, pkg-config can provide a great help to avoid having system dependent invocation:
shell prompt > gcc pkg-confg --libs libdar` my_prog.o -o my_prog
The compilation and linking steps described above assume you have a "full" libdar library. but beside the full (alias infinint) libdar flavor, libdar also comes in 32 and 64 bits versions. In these last ones, in place of internally relying on a special type (which is a C++ class called infinint) to handle arbitrary large integers, libdar32 relies on 32 bits integers and libdar64 relies on 64 bits integers (there are limitations which are described in doc/LIMITATIONS). But all these libdar version (infinint, 32bits, 64bits) have the same interface and must be used the same way, except for compilation and linking:
These different libdar versions can
coexist on the same system, they
share the same include files. But the LIBDAR_MODE macro must be set to
32 or 64 when compiling or linking with libdar32 or libdar64
respectively, this macro changes the way the libdar headers
files are interpreted by the compiler.
pkg-config --cflags will set the
correct LIBDAR_MODE, so you should only bother calling it with either
libdar, libdar32 or libdar64 depending on your need:
"pkg-confg --cflags libdar for example.
shell prompt > cat my_prog.cpp
#include <dar/libdar.h>
main()
{
libdar::get_version(...);
...
}
shell prompt > gcc -c `pkg-config --cflags libdar64 ` my_prog.cpp
shell prompt > gcc `pkg-config --libs libdar64 ` my_prog.o -o my_prog>
and replace 64 by 32 to link with libdar32.