It was created in 1996 by Andrew Tridgell and Paul Mackerras.
It is currently maintained by Wayne Davison.

rsync is freely available under the GNU General Public License.
rsync source code is here

The rsync algorithm is a type of delta encoding, and is used for minimizing network usage.
It efficiently computes/identify which parts (splitted blocks by fragmentation) of a source file match some part of an existing destination file (those parts don’t need to be sent across the communication link), thus minimizing the amount of data to transfer by only moving the portions of files that have changed.

For further speed improvements, the data sent to the receiver can be compressed using any of the supported algorithms.

ssh is the default remote shell for rsync since version 2.6.0 (January 1st 2004)

Install rsync (Debian)
# apt install rsync

rsync version
$ rsync -V

Usage

Local SRC > Local DST
rsync [OPTIONS] SRC [DST]

Push (Local SRC > Remote DST) rsync [OPTIONS] SRC [USER@]HOST:DST
Pull (Local DST < Remote SRC) rsync [OPTIONS] [USER@]HOST:SRC [DST]

Usages with just one SRC arg and no DST arg will list the source files instead of copying:
<type><perms_rwx> <size_bytes> <mtime YYYY/MM/DD> <mtime hh:mm:ss> <relative_path>

IMPORTANT: rsync must be installed on both the source and destination machines

If you still has this error:
rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream ...

It means Local rsync cannot find the remote rsync executable.
In this case you need to know the path of the remote host’s rsync binary and make it part of the command with --rsync-path=/path/to/remote/rsync

$ which rsync
/usr/bin/rsync  (Debian)

Options

If --delete option is specified, rsync will identify the files NOT present on the sender and delete them on the receiver. This option can be dangerous if used incorrectly! It is recommended to do a simulation run before, using the --dry-run option (-n) to find out which files are going to be deleted.

Each file from the list generated by rsync will be checked to see if it can be skipped.
In the most common mode of operation, files are not skipped if the modification time or size differs.

rsync performs a slower but comprehensive check if invoked with --checksum option.
This forces a full checksum comparison on every file present on both systems.

--checksum, -c
Skip files based on checksum, not mtime AND size.
This changes the way rsync checks if the files have been changed and are in need of a transfer.
Without this option, rsync uses a “quick check” that (by default) checks if each file’ size and time of last modification match between the sender and receiver.
This option changes this to compare a 128-bit checksum for each file that has a matching size.
Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer, so this can slow things down significantly (and this is prior to any reading that will be done to transfer changed files)

--human-readable, -h
Output numbers in a more human-readable format.
Unit letters: K (Kilo), M (Mega), G (Giga), T (Tera), or P (Peta).

--dry-run, -n
Simulation run (no changes made)

--verbose, -v
Increases the amount of information you are given during the transfer.
By default, rsync works silently.
A single -v will give you information about what files are being transferred and a brief summary at the end.
Two -v options will give you information on what files are being skipped and slightly more information at the end.
More than two -v options should only be used if you are debugging rsync.

--quiet, -q
Decreases the amount of information you are given during the transfer, notably suppressing information messages from the remote server. This option is useful when invoking rsync from cron.

--info=FLAGS
Choose the information output
An individual flag name may be followed by a level number, with 0 meaning to silence that output, 1 being the default output level, and higher numbers increasing the output of that flag (for those that support higher levels).
$ rsync --info=help
$ rsync -av --info=progress2 SRC/ DST/

--progress
Print information showing the progress of the transfer.
This is the same as specifying '--info=flist2,name,progress' but any user-supplied settings for those info flags takes precedence (e.g. --info=flist0 --progress).

While rsync is transferring a regular file, it updates a progress line that looks like this:
<reconstructed_bytes> <%_current_file> <throughput/sec> <remaining_time>

When the file transfer is done, rsync replaces the progress line with a summary line that looks like this:
<filesize_bytes> 100% <throughput/sec> <elapsed_time> (xfr#?, to-chk=???/N)
where ? is the nth transfer, ??? is the remaining files for the receiver to check (to see if they are uptodate or not)

In an incremental recursion scan (--recursive), rsync doesn’t know the total number of files in the files list until it reaches the end of the scan. Since it starts transfering files during the scan, it displays a line with the text “ir-chk” (for incremental recursion check) instead of “to-chk” until it knows the full size of the list, at which point it switches to “to-chk”. “ir-chk” lets you know that the number of files in the files list is still going to increase.

--archive, -a
It is equivalent to -rlptgoD
This is a quick way of saying you want recursion and want to preserve almost everything.
Be aware that it does not include preserving ACLs (-A), xattrs (-X), atimes (-U), crtimes (-N), nor the finding and preserving of hardlinks (-H).
The only exception to the above equivalence is when --files-from is specified, in which case -r is not implied.

--recursive, -r
This tells rsync to copy directories recursively. See also --dirs (-d).
Beginning with rsync 3.0.0, the recursive algorithm used is now an incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed.
It is only possible when both ends of the transfer are at least version 3.0.0.

Some options require rsync to know the full files list, these options disable the incremental recursion mode.
These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates.

Because of this, the default delete mode when you specify --delete is now --delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during to request this improved deletion mode explicitly).
See also the –delete-delay option that is a better choice than using –delete-after.

Incremental recursion can be disabled using the --no-inc-recursive option or its shorter --no-i-r alias.

--delete-during, --del
Request that the file deletions on the receiving side be done incrementally as the transfer happens.
The per-directory delete scan is done right before each directory is checked for updates, so it behaves like a more efficient --delete-before. This option was first added in rsync version 2.6.4. See --delete (which is implied) for more details on file deletion.

--delete-before
Request that the file deletions on the receiving side be done before the transfer starts.
It does imply a delay before the start of the transfer, and this delay might cause the transfer to timeout (if --timeout was specified). It also forces rsync to use the old, non-incremental recursion algorithm that requires rsync to scan all the files in the transfer into memory at once (see --recursive).

--delete-after
Request that the file deletions on the receiving side be done after the transfer has completed.
Important: this option forces rsync to use the old, non-incremental recursion algorithm that requires rsync to scan all the files in the transfer into memory at once (see --recursive). Use --delete-delay instead.

--delete-delay
Request that the file deletions on the receiving side be computed during the transfer (like –delete-during), but removed after the transfer completes. This is more efficient than using --delete-after.
If the number of removed files overflows an internal buffer, a temporary file will be created on the receiving side to hold the names. If the creation of the temporary file fails, rsync will try to fall back to using --delete-after (which it cannot do if --recursive is doing an incremental scan).

--links, -l
By default, symbolic links are not transferred at all.
A message "skipping non-regular" file is emitted for any symlinks that exist.
If --links is specified, then symlinks are recreated with the same target on the destination.
Note that --archive implies --links.

--perms, -p
Preserve permissions
This option causes the receiving rsync to set the destination permissions to be the same as the source permissions.
(See also the --chmod option for a way to modify what rsync considers to be the source permissions)

When this option is off, permissions are set as follows:

– Existing files (including updated files) retain their existing permissions, though the --executability option might change just the execute permission for the file.

– New files get their "normal" permission bits set to the source file’s permissions masked with the receiving directory’s default permissions (either the receiving umask, or the permissions specified via the destination directory’s default ACL), AND their special permission bits disabled except in the case where a new directory inherits a setgid bit from its parent directory.

Thus, when --perms and --executability are both disabled, rsync’s behavior is the same as that of other file copy utilities, such as cp(1) and tar(1).

In summary:
To give destination files (both existing and new) the source permissions, use --perms.
To give new files the destination default permissions (while leaving existing files unchanged), make sure that the --perms option is off and use --chmod=ugo=rwX (which ensures that all non-masked bits get enabled).

The preservation of the destination’s setgid bit on newly-created directories when –perms is off was added in rsync 2.6.7.

--times, -t
Preserve modification times
This tells rsync to transfer modification times along with the files and update them on the remote system.
Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective.
In other words, a missing -t or -a will cause the transfer to behave as if it used --ignore-times, causing all files to be updated (though rsync’s delta-transfer algorithm will make the update fairly efficient if the files haven’t actually changed, you’re much better off using -t).

--ignore-times, -I
Normally rsync will skip any files that are already the same size and have the same modification timestamp.
This option turns off this "quick check" behavior, causing all files to be updated.

--atimes, -U
Preserve access times
This tells rsync to set the access (use) times of the destination files to the same value as the source files.
nanoseconds are not preserved (set to .000000000), command cp -a does.

IMPORTANT:
There is no option to preserve ctime "status time"
(the timestamp used to record when the inode changed, it is specific to a filesystem)
An inode changes if any of its attributes are updated:
– at creation time (new file)
– file name
– mode/permissions
– owner/group
– hard link count
etc.

The creation of a file is one of the conditions listed above (creation of inode/file).
ctime cannot be preserved when files are brought into a new filesystem.

--open-noatime
Avoid changing the atime on opened file
This tells rsync to open files with the O_NOATIME flag (on systems that support it) to avoid changing the access time of the files that are being transferred. If your OS does not support the O_NOATIME flag then rsync will silently ignore this option. Note also that some filesystems are mounted to avoid updating the atime on read access even without the O_NOATIME flag being set.

--crtimes, -N
MAY NOT BE SUPPORTED, DEPENDS ON THE FILESYSTEM.
This tells rsync to set the create times (newness) of the destination files to the same value as the source files.

--group, -g
Preserve group
This option causes rsync to set the group of the destination file to be the same as the source file.
If the receiving program is not running as the super-user (or if --no-super was specified), only groups that the invoking user on the receiving side is a member of will be preserved. Without this option, the group is set to the default group of the invoking user on the receiving side.

--owner, -o
This option causes rsync to set the owner of the destination file to be the same as the source file, but only if the receiving rsync is being run as the super-user (see also the --super and --fake-super options).
Without this option, the owner of new and/or transferred files are set to the invoking user on the receiving side.

--acls, -A
This option causes rsync to update the destination ACLs to be the same as the source ACLs.
The option also implies --perms.
The source and destination systems must have compatible ACL entries for this option to work properly.
See the --fake-super option for a way to backup and restore ACLs that are not compatible.

--xattrs, -X
This option causes rsync to update the destination extended attributes to be the same as the source ones.

--hard-links, -H
This tells rsync to look for hard-linked files in the source and link together the corresponding files on the destination. Without this option, hard-linked files in the source are treated as though they were separate files.

This option does NOT necessarily ensure that the pattern of hard links on the destination exactly matches that on the source.

Usual Usage

Local SRC_DIR > Local DST_DIR
NOTE: By default, if Local DST_DIR does not exist it is created

Copy SRC_DIR inside /path/to/local/DST_DIR/ : /path/to/local/DST_DIR/SRC_DIR
$ rsync -av --info=progress2 /path/to/local/SRC_DIR /path/to/local/DST_DIR

Copy <src_path>'s content inside <dst_path>/
$ rsync -av --info=progress2 <src_path>/ <dst_path>/
$ rsync -av --info=progress2 <src_path>/* <dst_path>/

Local SRC_FILE > Local DST

$ rsync -av --info=progress2 /path/to/local/SRC_FILE /path/to/local/DST

IF DST is a directory, it copies SRC_FILE inside DST (DIR)

IF DST is a file, its content is replaced with the content of SRC_FILE
(with -a option ONLY mtime is the same, atime, ctime are different, you may use -U + -N options)

If DST does not exist :
– if there is a trailing slash ‘/’ it creates the directory DST (only for a direct subdirectory of an existing path “mkdir -p does not work” ) AND copies SRC_FILE inside DST

– otherwise it creates file DST (copy of SRC_FILE)

Portable Document Format

Adobe created the PDF in 1992 by Dr. John Warnock, offering an easy, reliable way to present and exchange documents regardless of the software, hardware, or operating systems being used.
Today, it is one the most trusted file formats around the world, it can be easily viewed on any operating system.

PDF was standardized as ISO 32000 in 2008 as an open standard.
The PDF format is now maintained by the International Organization for Standardization (ISO).
ISO 32000-2:2020 edition was published in December 2020, it does not include any proprietary technologies.

The PDF specification also provides for encryption (in which case a password is needed to view or edit the contents), digital signatures (to provide secure authentication), file attachments, and metadata.
PDF 2.0 defines 256-bit AES encryption as standard for PDF 2.0 files.

The standard security provided by PDF consists of two different passwords:

– user password, which encrypts the file and prevents opening

– owner password, which specifies operations that should be restricted even when the document is decrypted, which can include modifying, printing, or copying text and graphics out of the document, or adding or modifying text notes.

The user password encrypts the file, the owner password does not, instead relying on client software to respect content restrictions.
An owner password can easily be removed by software.
Thus, the used restrictions that an author places on a PDF document are not secure, and cannot be assured once the file is distributed.

Metadata includes information about the document and its content, such as the author’s name, document title, description, creation/modification dates, application used to create the file, keywords, copyright information, etc.

Install qpdf (Debian)

# apt install qpdf

Usage

--linearize
Create linearized (web-optimized) output file.
Linearized files are formatted in a way that allows compliant readers to begin displaying a PDF file before it is fully downloaded.
Ordinarily, the entire file must be present before it can be rendered because important cross-reference information typically appears at the end of the file.

$ qpdf --linearize infile.pdf  outfile.pdf

Merge PDF files with pages selection

qpdf allows you to use the --pages option to select pages from one or more input files.

$ qpdf primary_input_file.pdf --pages . [--password=password] [page-range] [ ... ] -- outputfile.pdf

Within [ ... ] you may repeat the following:  inputfile_N.pdf [--password=password] [page-range]

The special input file '.' can be used as an alias for the primary input file.
Multiple input files may be specified and you can select specific pages from it.
For each inputfile that pages should be extracted from, specify the filename, a password (if needed) to open the file, and a page range.
Note that '--' terminates parsing of page selection flags.

--password=password specifies a password for accessing encrypted files
The password option is only needed for password-protected files

The page range may be omitted. In this case, all pages are included.

Document-level information (metadata, outline, etc.) is taken from the primary input file (in the above example, primary_input_file.pdf) and is preserved in outputfile.pdf
You can use --empty in place of the primary input file to start from an empty file (without any metadata, outline, etc.) and just merge selected pages from input files.

In most cases you will most likely use this following syntax

$ qpdf --empty --pages inputfile_1.pdf [page-range] inputfile_2.pdf [page-range] inputfile_3.pdf [page-range] [ ... ] -- outputfile.pdf

The page-range is a set of numbers separated by commas, ranges of numbers separated dashes, or combinations of those.
The character 'z' represents the last page.
A number preceded by an 'r' indicates to count from the end, so r3-r1 would be the last three pages of the document.
Pages can be specified in any order (selection of any pages).
Ranges can be specified in any order (ascending or descending): a high number followed by a low number causes the pages to appear in reverse.
Numbers may be repeated in a page range.
A page range may be optionally appended with :even or :odd to indicate only the even or odd pages in the given range.
Note that even and odd refer to the positions within the specified, range, not whether the original number is even or odd.

Example page ranges:

1,3,5-9,15-12
Pages 1, 3, 5, 6, 7, 8, 9, 15, 14, 13, and 12 in that order

z-1
All pages in the document in reverse

r3-r1
The last three pages of the document

r1-r3
The last three pages of the document in reverse order

1-20:even
Even pages from 2 to 20

5,7-9,12:odd
Pages 5, 8 and 12, which are the pages in odd positions from among the original range (pages 5, 7, 8, 9, and 12)

Example, to extract pages 1 through 5 from infile.pdf while preserving all metadata associated with that file in outfile.pdf
$ qpdf infile.pdf --pages . 1-5 -- outfile.pdf

If you want pages 1 through 5 from infile.pdf without any metadata, use
$ qpdf --empty --pages infile.pdf 1-5 -- outfile.pdf

Merge all .pdf files
$ qpdf --empty  --pages *.pdf -- outfile.pdf

Split a PDF into separate PDF files

--split-pages[=n]
Write each group of n pages to a separate output file.
If n is not specified, create single pages.

Output file names are generated as follows:
If the string %d appears in the output file name, it is replaced with a range of zero-padded page numbers starting from 1.
Otherwise, if the output file name ends in .pdf (case insensitive), a zero-padded page range, preceded by a dash, is inserted before the file extension.
Otherwise, the file name is appended with a zero-padded page range preceded by a dash.

Zero padding is added to all page numbers in file names so that all the numbers are the same length, which causes the output filenames to sort lexically in numerical order.

Page ranges are a single number in the case of single-page groups or two numbers separated by a dash otherwise.

Here are some examples. In these examples, infile.pdf has 20 pages

Output files are 01-outfile through 20-outfile with no extension
$ qpdf --split-pages infile.pdf %d-outfile

Output files are outfile-01.pdf through outfile-20.pdf
$ qpdf --split-pages infile.pdf outfile.pdf

Output files are outfile-01-04.pdf, outfile-05-08.pdf, outfile-09-12.pdf, outfile-13-16.pdf, outfile-17-20.pdf
$ qpdf --split-pages=4 infile.pdf outfile.pdf

Output files are outfile.notpdf-01 through outfile.notpdf-20
The extension .notpdf is not treated in any special way regarding the placement of the number
$ qpdf --split-pages infile.pdf outfile.notpdf

Note that metadata, outline, etc, and other document-level features of the original PDF file are not preserved.
For each page of output, this option creates an empty PDF and copies a single page from the output into it.
If you require the document-level data, you will have to run qpdf with the --pages option once for each page.
Using --split-pages is much faster if you don’t require the document-level data.

If you don’t want to split out every page, use page ranges to select the pages you only want to extract.
The page range is used to specify the pages or ranges you want, but each extracted page is still stored in a single PDF.

$ qpdf --split-pages infile.pdf outfile.pdf --pages infile.pdf 4-5,8,9-13 --

cURL is an Open Source project consisting of voluntary members from all over the world.
The cURL project is completely independent and free.

It is a client-side program (the ‘c’), a URL (Uniform Resource Locator) client, and shows the data (by default).
So ‘c’ for Client and URL: cURL

Everything curl is a detailed and free book that explains basically everything there is to know about curl, libcurl and the associated project.

cURL is a project and its primary purpose and focus is to make two products:
– curl, the command-line tool
– libcurl the transfer library with a C API

Both the tool and the library do Internet transfers for resources specified as URLs using Internet protocols.
Everything and anything that is related to Internet protocol transfers can be considered curl’s business.

The protocol describes exactly how to ask the server for data, or to tell the server that there is data coming.
Protocols are typically defined by the IETF (Internet Engineering Task Force ),
which hosts RFC documents that describe exactly how each protocol works: how clients and servers are supposed to act and what to send and so on.

curl and libcurl are distributed under an Open Source license known as a MIT license derivative.
A key thing to remember is, that libcurl is the library and that this library is the biggest component of the curl command-line tool.

Where’s the code ?
The curl git tree can be browsed with a web browser at https://github.com/curl/curl
To check out the curl source code from git, you can clone it like this:

$ git clone https://github.com/curl/curl.git

curl started out as a command-line tool and it has been invoked from shell prompts and from within scripts by an uncountable number of users over the years.
curl has established itself as one of those trusty tools that is there for you to help you get your work done.

Here you can find why curl is your first choice.

curl vs. wget

https://daniel.haxx.se/docs/curl-vs-wget.html

wget has (recursive) downloading powers that curl does not feature and it also handle download retries over unreliable connections possibly slightly more effective.
For just about everything else, curl is probably the more suitable tool.

curl operates on URLs. URI (Uniform Resource Identifier) is actually the correct name for them.
The syntax is defined in RFC 3986 (2005): https://datatracker.ietf.org/doc/html/rfc3986

Install curl (Debian)

# apt install curl

curl options basics

-V, --version
Displays information about curl and the libcurl version it uses.
The output from that command line is typically four lines, out of which some will be rather long and might wrap in your terminal window

Line 1: curl
The first line includes the full version of curl, libcurl and other 3rd party libraries linked with the executable.
The first line starts with ‘curl’ and first shows the main version number of the tool.
Then follows the “platform” the tool was built for within parentheses and the libcurl version.
Those three fields are common for all curl builds.

If the curl version number has -DEV appended to it, it means the version is built straight from a in-development source code and it is not an officially released and “blessed” version.
The rest of this line contains names of third party components this build of curl uses, often with their individual version number next to it with a slash separator.

Line 2: Release-Date
This line shows the date this curl version was released by the curl project,
and it can also show a secondary “Patch date” if it has been updated somehow after it was originally released.

Line 3: Protocols
The third line (starts with “Protocols:”) is a list of all transfer protocols (URL schemes really) in alphabetical order that this curl build supports.
All names are shown in lowercase letters.

Line 4: Features
The fourth line (starts with “Features:”) is the list of features this build of curl supports.
If the name is present in the list, that feature is enabled. If the name is not present, that feature is not enabled.

-v, --verbose
Makes curl verbose during the operation.
Useful for debugging and seeing what’s going on “under the hood”.
When verbose mode is enabled, curl gets more talkative and will explain and show a lot more of its doings.
It will add informational tests and prefix them with ‘*’.

A line starting with ‘>’ means “header data” sent by curl,
A line starting with ‘<‘ means “header data” received by curl that is hidden in normal cases,
A line starting with ‘*’ means additional info provided by curl.

If you only want HTTP headers in the output, -i, --include might be the option you’re looking for.
If you think this option still doesn’t give you enough details, consider using --trace or --trace-ascii instead.

See also -i, --include. This option overrides --trace and --trace-ascii.

-s, --silent
Silent or quiet mode.
Don’t show progress meter or error messages when errors occur. Makes Curl mute.
It will still output the downloaded data you ask for, potentially even to the terminal/stdout unless you redirect it.
Use -S, --show-error in addition to this option to only disable progress meter but still show error messages.

-S, --show-error
When used with -s, --silent, it makes curl show an error message if it fails.

-I, --head
(HTTP, FTP, FILE) Fetch the headers only.
HTTP-servers feature the command HEAD which this uses to get nothing but the header of a document.
When used on an FTP or FILE file, curl displays the file size and last modification time only.

--ssl
Try to use SSL/TLS for the connection.
Reverts to a non-secure connection if the server doesn’t support SSL/TLS.

--ssl-reqd
Require SSL/TLS for the connection. Terminates the connection if the server doesn’t support SSL/TLS.

-L, --location (HTTP 3NN Redirection)
(HTTP) If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3NN response code), this option will make curl redo the request automatically on the new place.

If used together with -i, –include or -I, –head, headers from all requested pages will be shown.

--stderr
Redirect all writes to stderr to the specified file instead.
If the file name is a plain ‘-‘, it is instead written to stdout.
If this option is used several times, the last one will be used.

OUPUT
If not told otherwise, curl writes the received data to stdout.
It can be instructed to instead save that data into a local file, using the -o, --output OR -O, --remote-name options.
If curl is given multiple URLs to transfer on the command line, it similarly needs multiple options for where to save them.
curl does not parse or otherwise “understand” the content it gets or writes as output.
It does no encoding or decoding, unless explicitly asked so with dedicated command line options.

-o, --output <path>
Write output to <path> instead of stdout.

-O, --remote-name
Write output to a local file named like the remote file you get.
(Only the file part of the remote file is used, the path is cut off)

The file will be saved in the current working directory.
If you want the file saved in a different directory, make sure you change the current working directory before invoking curl with this option.

The remote file name to use for saving is extracted from the given URL, nothing else, and if it already exists it will be overwritten.
There is no URL decoding done on the file name. If it has %20 or other URL encoded parts of the name, they will end up as-is as file name.

You may use this option as many times as the number of URLs you have.

-C, --continue-at
Continue/Resume a previous file transfer at the given offset.
The given offset is the exact number of bytes that will be skipped, counting from the beginning of the source file before it is transferred to the destination.

Use '-C -' to tell curl to automatically find out where/how to resume the transfer.
It then uses the given output/input files to figure that out.
If this option is used several times, the last one will be used.

Download a file
$ curl URL -o /path/to/dst_filename

Download a file to the current directory
$ curl -O URL

Resume an interrupted download
$ curl -O -C - URL

PROGRESS METER
curl normally displays a progress meter during operations, indicating the amount of transferred data, transfer speeds and estimated time left, etc.
The progress meter displays number of bytes and the speeds are in bytes per second.
The suffixes (K, M, G, T, P) are 1024 based. For example 1K is 1024 bytes. 1M is 1048576 bytes (1024^2).

curl displays this data to the terminal by default, so if you invoke curl to do an operation and it is about to write data to the terminal, it disables the progress meter as otherwise it would mess up the output mixing progress meter and response data.

If you want a progress meter for HTTP POST or PUT requests, you need to redirect the response output to a file, using shell redirect (>), -o, --output or similar.

It is not the same case for FTP upload as that operation does not spit out any response data to the terminal.

If you prefer a progress “bar” instead of the regular meter, -#, --progress-bar is your friend.
You can also disable the progress meter completely with the -s, --silent option.

-#, --progress-bar
Make curl display transfer progress as a simple progress bar instead of the standard, more informational, meter.
This progress bar draws a single line of ‘#’ characters across the screen and shows a percentage if the transfer size is known.
For transfers without a known size, there will be space ship (-=o=-) that moves back and forth but only while data is being transferred, with a set of flying hash sign symbols on top.

--no-progress-meter
Option to switch off the progress meter output without muting or otherwise affecting warning and informational messages like -s, --silent does.
Note that this is the negated option name documented.
You can thus use --progress-meter to enable the progress meter again.
Added in version 7.67.0.

Connection Test: Get Server Information

$ curl -I -v --silent --ssl  [protocol://]ip_or_hostname | grep '^[><*]'

If you specify URL without protocol:// prefix, curl will attempt to guess what protocol you might want.
It will then default to HTTP but try other protocols based on often-used host name prefixes.
For example, for host names starting with “ftp.” curl will assume you want to speak FTP.

curl will do its best to use what you pass to it as a URL.

https://everything.curl.dev/usingcurl/tls/enable

Using --ssl means that curl will attempt to upgrade the connection to TLS but if that fails, it will still continue with the transfer using the plain-text version of the protocol.
To make the --ssl option require TLS to continue, there is instead the --ssl-reqd option which will make the transfer fail if curl cannot successfully negotiate TLS.

Mozilla (stylized as moz://a) is a free software community founded in 1998 by members of Netscape.

Jamie Zawinski from Netscape registered mozilla.org
He was a founder of Mozilla.org, personally registering its domain name on the day of Netscape’s open source announcement and helping design and run the organization through its first year.

He developed the Unix release of Netscape Navigator 1.0 and later, Netscape Mail, the first mail reader (or Usenet reader) to natively support HTML.
The project took its name “Mozilla”, after the original code name of the Netscape Navigator browser.

Mozilla’s most known products include the Firefox web browser and Thunderbird e-mail client.

How does Thunderbird emails storage work ?

There are several ways to store emails on disk. Two of the most common are mbox files and Maildir directories.

Unless you explicitly use optional support for Maildir which allows a single unique file per email (using the EML data format), Thunderbird stores emails content in two distinct files.

The first one is an mbox file (a text file without extension), which is a standard format to store email content.
All messages from a mailbox folder are concatenated and stored as plain text in a mbox file.
It was first implemented in Fifth Edition Unix.

mbox database files sometimes have an .mbox extension, but this is not required nor expected.
Each message starts with the four characters “From” followed by a space (the so-called “From_ line”).

RFC 4155 (2005) The application/mbox Media Type : https://datatracker.ietf.org/doc/html/rfc4155.html

The second is an .msf file, which stands for Mail Summary File.
This is an index file, which contains only headers and summary of emails.

See Mail Summary Files writeup explaining Mozilla mbox and .msf file design choices.

Jamie Zawinski

"In this modern world, the only sensible thing for someone implementing a mail reader to do is use BSD mbox files to store mail messages, because that is the de-facto standard used by almost every popular mail reader, on Unix, Windows, and Mac.
…
The mbox file format is basically one big text file containing every message of the folder sequentially, with separator lines between them. This has obvious efficiency problems: you don't want to spend a lot of time doing linear searches through the file, and you don't want to ever pull the whole file into memory at once, and deletions are expensive.

So, you have to summarize it, and be lazy about updates.
My approach, when designing this for Netscape Mail in the 2.0 and 3.0 timeframe, was to identify the pieces of information that needed to be gotten at quickly and repeatedly, and store those in a cache.

Most other information was only needed on a per-message basis (not when operating on many messages at once) so that latter sort was simply parsed from the message on demand, as the message was being displayed.
…
So, we have these summary files, one summary file for each folder, which provide the necessary info to generate that message list quickly.

Each summary file is a true cache, in that if it is deleted (accidentally or on purpose) it will be regenerated when it is needed again (which might take a little while, but which only needs to be done once.)
…"

Mail Summary Files (.msf) use the Mork format.
Mork is a database file format invented by David McCusker for the Mozilla code since the original Netscape database information was proprietary and could not be released open source.
It is not very human-readable, it has been phased out in favor of SQLite, a more widely-supported file format.
As you may notice it is still used by Thunderbird application.

The first line in the .msf file is the header:

// <!-- <mdb:mork:z v="1.4"/> -->

It is structured as a comment so it can be safely ignored by the parser.

NOTES
EML (Electronic Mail Format) files typically store each message as a single file (unlike MBOX which concatenates all the messages from a folder into one file), and attachments may either be included as MIME content in the message or written off as a separate file, referenced from a marker in the EML file.

RFC 5322 (2008) Internet Message Format : https://datatracker.ietf.org/doc/html/rfc5322

Christopher J.Prom (2011)

"The IMF serves as the basis for two of the most common storage and exchange formats: MBOX and EML.

In reality, neither of these methods constitutes a storage format per se, since each server or client implements them somewhat differently.

MBOX (sometimes known as Berkeley format) is a set of four slightly different storage formats, developed originally for Unix systems.

Generally, a single file with the extension .mbox or .mbx contains the contents of an entire folder, with MIME content stored directly in the file.

Files can and do grow to astronomical sizes, and even slight file corruption may affect the ability of certain clients to access individual messages or even the entire folder.

MBOX files also include the attachments in their MIME format, meaning that action will likely need to be taken to migrate them, if they are to remain accessible in the future.

EML files typically store each message as a single file, and attachments may either be included as MIME content in the message or written off as a separate file, referenced from a marker in the EML file.

In spite of these issues, MBOX and EML have achieved a certain status as de facto standards.
Most modern email clients and servers can import and export one or both of the formats"

Where does Thunderbird store data ?

Thunderbird saves all the data such as emails, passwords, user preferences and settings in a set of files called a profile.

The profile is stored in a separate location from the Thunderbird program files.
The location of the profile folder varies according to the Operating System.

With Debian it is located in /home/<username>/.thunderbird/

It is possible to have several profiles, each profile is stored in a dedicated folder.

A profile folder is identified using the following convention: <random_string>.<profile_name>
where <random_string> is eight alphanumeric characters randomly generated by Thunderbird and <profile_name> is the name you assigned to the profile.

Thunderbird uses the /home/<username>/.thunderbird/profiles.ini file to store profiles information

The default profile is set via

[Install<ID>]
Default=<profile_folder>

[General]
StartWithLastProfile=<boolean_value>

0=Thunderbird will ask you to select a profile (and make the one selected the default profile)
1=Thunderbird won't ask you to select a profile, it will use the default profile

This option corresponds to Profile Manager 'Use the selected profile without asking at startup'

Get Thunderbird’s Default Profile (Debian)

$ sed -n '/^\[Install/,/^Default=/p' $HOME/.thunderbird/profiles.ini | sed -n 's;^Default=;;p'

NOTES
When you open Thunderbird for the first time, it creates a default profile. This profile is used automatically.
While it is possible to have multiple profiles, most users just use the default one which may contain several email accounts.

The Profile Manager is used to create/rename/delete profiles and to select the profile to use for a Thunderbird session.
It is not displayed by default, you have to start Thunderbird with the option -ProfileManager

$ thunderbird -ProfileManager

All non-default settings: prefs.js file

/home/<username>/.thunderbird/<random_string>.<profile_name>/prefs.js

Thunderbird stores any settings you change or create using the GUI via ‘Account Settings’ and/or ‘Settings’ in the prefs.js file

The recommended way to add or modify a setting which is not available via the User Interface is to use ‘Config Editor‘, equivalent to Firefox’s about:config.

OR

You can add/modify a setting by editing prefs.js directly using a text editor.
However, prefs.js does not contain all of the settings, it only contains any settings that have been modified (not set with their default value).
In fact, if you add a setting with a default value Thunderbird will remove it from the file the next time you run it.
Be careful when you edit prefs.js while Thunderbird is running since it may overwrite some of the settings when it exits, losing your edits.

user.js file

/home/<username>/.thunderbird/<random_string>.<profile_name>/user.js

It does not exist by default, you have to create it. This is a user-set overriding preferences (prefs.js)
It is mainly used by administrators to set the same settings in several profiles.
Unless you know what you are doing, it is recommended that you don’t use it since any settings you add to it will overwrite prefs.js when Thunderbird starts, preventing permanent changes using the Config Editor.
Whenever Thunderbird starts it will overwrite any settings in prefs.js with the corresponding settings from user.js

ImapMail: IMAP Folders containing mbox .msf .sbd files

/home/<username>/.thunderbird/<random_string>.<profile_name>/ImapMail/

When you set up an IMAP account Thunderbird creates a folder named after the IMAP server name.
So if your IMAP server is imap.mozilla.org Thunderbird will create a directory
/home/<username>/.thunderbird/<random_string>.<profile_name>/ImapMail/imap.mozilla.org/

You will find inside: the mbox and .msf files according to the standard folders (Inbox, Draft, Sent, Junk, Trash, etc.) and the specific ones you have created on the IMAP server.

The .sbd subdirectories are used to store the folders in a hierarchy
Subfolders are obtained with the .sbd extension
If you created subfolders within your Inbox folder on the IMAP server, you will find an extra file Inbox.sbd which will also contain mbox and .msf files (and eventually other .sbd files for each subfolder)

Be aware though that you may get a different folder name from what is shown in the application and the actual matching mbox .msf files
This renaming option is set within .msf files

Local Folders

/home/<username>/.thunderbird/<random_string>.<profile_name>/Mail/Local Folders

The name says it all : it is local folders (on your computer) meaning there is no direct interaction with any mail server.
Local Folders provides a convenient place to store messages locally when desired, ideal for backups.
The only space limit is the amount of space available on your disk.

By default you have a ‘Trash’ and ‘Outbox’ empty folder

Trash folder:
/home/<username>/.thunderbird/<random_string>.<profile_name>/Mail/Local Folders/Trash
/home/<username>/.thunderbird/<random_string>.<profile_name>/Mail/Local Folders/Trash.msf

Outbox folder:
/home/<username>/.thunderbird/<random_string>.<profile_name>/Mail/Local Folders/Unsent Messages
/home/<username>/.thunderbird/<random_string>.<profile_name>/Mail/Local Folders/Unsent Messages.msf

Outbox is a special folder used to queue messages when you use ‘Send Later’ rather than ‘Send’
When you compose a message and save it to be sent later (File > Send Later), it is saved in the ‘Unsent Messages’ mbox file in ‘Local Folders’.

That folder is also used to send a message in the background if you set mailnews.sendInBackground true

NOTES
emails can be stored on the client, on the server side, or in both places.
Standard formats for mailboxes include Maildir and mbox.

msgFilterRules.dat (Message filters)

Message filters allow you to set up Thunderbird to organize your messages automatically.
Each account has its own set of filters.

Message filters are useful if you routinely want to perform certain actions on messages:
moving messages to folders, deleting them, forwarding them to other email addresses, etc.

Filters can be applied automatically to incoming mail, or you can run them manually when desired.

The message filters for each account are stored in a msgFilterRules.dat file

IMAP Backup Solution

If you want to back up/archive your emails from an IMAP account you may use Local Folders

Keep in mind that this simple solution exists and is appropriate in several use cases:
– You are running out of space on your email account and need to free up server space
– You want to create a snapshot of your emails (copy/backup)

In order to proceed, create a new folder yyyymmdd-backup-emailaccount into ‘Local Folders’
via the GUI: right-click on Local Folders > New Folder… > Create as a subfolder of Local Folders

It will create 2 files (mbox and .msf) :
/home/<username>/.thunderbird/<profile_folder>/Mail/Local Folders/yyyymmdd-backup-emailaccount
/home/<username>/.thunderbird/<profile_folder>/Mail/Local Folders/yyyymmdd-backup-emailaccount.msf

IMPORTANT
Make sure Thunderbird has fully synchronized all your IMAP folders and subfolders (downloaded all emails)

Set mail.server.default.check_all_folders_for_new

File > Offline > Download/Sync Now

Download and/or sync the following : Select ‘Mail messages’ only and click on ‘Select’
Tick any folder you want to sync and click on ‘OK’ twice

Depending on the email account and the network capacity, it may take a few minutes…

Your IMAP folders should be synchronized now

Via the GUI select the IMAP folders you want to backup (Ctrl+Left Click)
Drag and Drop to the folder yyyymmdd-backup-emailaccount in ‘Local Folders’

Here too, it may take a few minutes to complete…

You have now a local copy of your emails (snapshot):
/home/<username>/.thunderbird/<profile_folder>/Mail/Local Folders/yyyymmdd-backup-emailaccount
/home/<username>/.thunderbird/<profile_folder>/Mail/Local Folders/yyyymmdd-backup-emailaccount.msf
/home/<username>/.thunderbird/<profile_folder>/Mail/Local Folders/yyyymmdd-backup-emailaccount.sbd

The good thing is that you get exactly the same IMAP folder<>file names match.

It is recommended that you save/move the mbox .msf and .sbd files to a directory ‘yyyymmdd-backup-emailaccount’ on a separate backup device

Do NOT create this directory into /home/<username>/.thunderbird/<profile_folder>/Mail/Local Folders/

Close and reopen Thunderbird application

If you want to restore your backup:
– Copy the three files back into /home/<username>/.thunderbird/<profile_folder>/Mail/Local Folders/
– Close and reopen Thunderbird application

Thank You Mozilla

Menu Edit > Settings > General > Config Editor…

Sort By Date
   Advanced Preferences   mailnews.default_sort_type   18

Sort Order   1 (ascending)   2 (descending)
   Advanced Preferences   mailnews.default_sort_order   2

Enable Threaded View   0 (unthreaded)   1 (threaded)
   Advanced Preferences   mailnews.default_view_flags    1

Menu Edit > Settings > General > Config Editor…

Advanced Preferences   mail.server.default.check_all_folders_for_new    true