Public-key cryptography, or asymmetric cryptography, is the field of cryptographic systems that use pairs of related keys.
Each key pair consists of a private key and a corresponding public key.
Key pairs are generated with cryptographic algorithms.

The main uses of asymmetric cryptography are :

– Encryption (public key)
Anyone with a public key can encrypt a message, but only those who own the corresponding private key can decrypt the encrypted message to obtain the original message.

– Digital Signatures (private key)
This process proves the Authenticity of the sender and the Integrity of the message.

In many scenarios, the sender and receiver of a message must be sure that a message has not been altered during transmission.
Although encryption hides the contents of a message, it may be possible to alter an encrypted message.
If a message is digitally signed, any change in the message after signature invalidates the signature.
A message is signed with the sender’s private key and can be verified by anyone who has access to the sender’s public key.

Given a message and a private key, a signing algorithm produces a signature.
Given the message, the public key and the signature, a signature verifying algorithm either accepts or rejects the message’s claim to authenticity.

Digital signatures may be attached to the message or detached.
A detached signature is transmitted separately from the message it signs.

Security of public key cryptography depends on keeping the private key secret, while the public key can be openly distributed.

Transport Layer Security (TLS)

Transport Layer Security (TLS) is a protocol that encrypts data sent over the Internet, the most familiar usage is with HTTPS.

TLS uses a combination of symmetric and asymmetric cryptography, as this provides a good compromise between performance and security when transmitting data securely.

With symmetric cryptography, data is encrypted and decrypted with a secret key known to both sender and recipient, typically 128 but preferably 256 bits in length (anything less than 80 bits is now considered insecure).
Symmetric cryptography is efficient in terms of computation, but having a common secret key means it needs to be shared in a secure manner.

Asymmetric cryptography uses key pairs: a public key, and a private key.
The public key is mathematically related to the private key, but given sufficient key length, it is computationally impractical to derive the private key from the public key.
This allows the public key of the recipient to be used by the sender to encrypt the data they wish to send to them, but that data can only be decrypted with the private key of the recipient.

The advantage of asymmetric cryptography is that the process of sharing encryption keys does not have to be secure, but the mathematical relationship between public and private keys means that much larger key sizes are required.
The recommended minimum key length is 1024 bits, with 2048 bits preferred, but this is up to a thousand times more computationally intensive than symmetric keys of equivalent strength (e.g. a 2048-bit asymmetric key is approximately equivalent to a 112-bit symmetric key) and makes asymmetric encryption too slow for many purposes.

For this reason, TLS uses asymmetric cryptography for securely generating and exchanging a session key.
The session key is then used for encrypting the data transmitted by one party, and for decrypting the data received at the other end.

Once the session is over, the session key is discarded.

X.509 Certificates

ITU-T X.509 | ISO/IEC 9594-8 is an International Telecommunication Union (ITU) standard defining the format of public key certificates.

In cryptography, a public key certificate, also known as a digital certificate or X.509 certificate is an electronic document used to prove the validity of a public key.

An X.509 certificate binds an identity to a public key using a digital signature.

The certificate includes a public key, information about the identity of its owner (called the subject), and the digital signature of an entity that has verified the certificate’s contents (called the issuer).

If the signature is valid, and the software examining the certificate trusts the issuer, then it can use that public key to communicate securely with the subject.

The X.509 standard was first issued in 1988 and is described in several RFCs.
X.509 Version 3 Certificate is defined in RFC 5280 (2008)

A digital certificate is normally issued by an authority, referred to as a Certification Authority (CA).
That authority guarantees the validity of the information in the certificate which is valid for a limited period of time.

A Certification Authority (CA) is an entity that stores, signs, and issues digital certificates.
A digital certificate certifies the ownership of a public key by the named subject of the certificate.

One particularly common use for Certificate Authorities is to sign certificates used for HTTPS, the secure browsing protocol for Internet.

The CA that issues public-key certificates also has the responsibility to indicate the validity of the public-key certificates that it issues. Generally, public-key certificates are subject to possible subsequent revocation. This revocation and a notification of the revocation may be done directly by the same CA that issued the public-key certificate, or indirectly by another authority duly authorized by the CA that issued the public-key certificate.

If, for any reason, a CA revokes a previously issued public-key certificate, entities need to be able to learn that revocation has occurred so they do not use an untrustworthy public-key certificate.
Revocation lists are one scheme that can be used to notify entities of revocations.

CAs that do revoke public-key certificates are required to state what mechanism(s) can be used by relying parties to obtain revocation status information about public-key certificates issued by that CA.
This includes a Certificate Revocation List (CRL) mechanism and Authorization and Validation List (AVL) mechanism, but does not preclude the use of alternative mechanisms.

One such alternative mechanism is the Online Certificate Status Protocol (OCSP) specified in IETF RFC 6960 (2013). Using this protocol, a relying party (client) requests the revocation status of a public-key certificate from an OCSP server. The server may use CRLs, or other mechanisms to check the status of the public-key certificate and respond to the client accordingly.

Steps to get an SSL/TLS certificate from a Certification Authority

1) Create a private key and a Certificate Signing Request (CSR)
2) Send the CSR to the trusted CA
3) The CA sends you the certificate signed by its root certificate and private key
4) Install the certificate on the web server

Certificate Signing Request (CSR)

A Certificate Signing Request (CSR) is the first step to get an SSL/TLS certificate from a Certification Authority.
You normally generate it on the same server you plan to use the certificate (the private key must be kept secret).
The CSR contains identification information (Country, State/Province, Locality, Organization, Organizational Unit, Common Name, Email) AND the public key that the Certificate Authority (CA) will use to create the certificate.
Not all of these fields may be required and will vary depending on the assurance level of your certificate.

Assurance levels for SSL/TLS certificates

Class 1 Certificates are considered to be low assurance, as the verification method simply confirms that the subscriber controls the domain.
They can do this by adding a DNS record associated with the domain, the process is automated.
No verification checks of the subscriber’s identity are performed.
This level of validation is referred to as Domain Validation (DV) and is the cheapest.

Class 2 Certificates are considered to be medium assurance.
They provide a greater level of assurance over Class 1 Certificates, because in addition to domain control, basic verification steps are performed to confirm the identity of the Subscriber.
This level of validation is referred to as Organization Validation (OV).
Organization Validation SSL/TLS certificates will contain the organization’s name and address, making them more trustworthy for users than Domain Validation certificates.

Class 3 Certificates provide a high level of assurance.
They are issued only after rigorous validation of the identity of the Subscriber.
This level of validation is referred to as Extended Validation (EV).

Extended Validation involves a full background check of the organization.
The CA will make sure that the organization exists and is legally registered as a business, that they actually are present at the address they list, and so on.
This validation level takes the longest and costs the most, but Extended Validation SSL certificates are more trustworthy than other types of SSL certificates.
Consequently, these certificates are necessary for a website’s address to turn the browser URL bar green, the visual representation for users of a trustworthy TLS-encrypted site.

PEM Format

A CSR is usually in a PKCS#10 format, by default a CSR is created in a Base-64 PEM format, and isn’t designed to be read by a human.

PEM (originally ‘Privacy Enhanced Mail’) is a standard and the most common format for X.509 certificates, CSRs, and cryptographic keys.

A PEM file is a text file containing one or more items (defined by <type>) in Base64 ASCII encoding, each with plain-text headers and footers :

starting with
-----BEGIN <type>-----

and ending with:
-----END <type>-----

Everything in between is base64 encoded (uppercase and lowercase letters, digits, +, / and =).
Base64 is a binary-to-text encoding scheme. It represents binary data in a printable ASCII string format by translating it into a radix-64 representation.
It is defined in RFC 4648 (2006)

The PEM file will tell you what it is used for in the header (if it is a private key, a CSR, a certificate)

-----BEGIN PRIVATE KEY-----
…
-----END PRIVATE KEY-----

-----BEGIN CERTIFICATE REQUEST-----
…
-----END CERTIFICATE REQUEST-----

-----BEGIN CERTIFICATE-----
…
-----END CERTIFICATE-----

Thus, a Certificate Signing Request (CSR) is a request sent to a Certification Authority to sign your public key and associated informations.

The CSR is signed by the applicant’s private key, this proves to the CA that the applicant has possession and control of the private key that corresponds to the public key included in the CSR.

The CA first verifies the PKCS#10 signature with the public key placed in the PKCS#10.
Once the signature is successfully verified, the requested information in the CSR passes a vetting process and domain control is established, the CA signs the applicant’s public key so that it can be publicly trusted.

It becomes a trusted signed certificate that internet browsers will be able to verify when communication is made to your server.

Most internet browsers and operating systems hold a copy of root CA certificates of all the trusted certified Certificated Authorities.

On Debian it is located in /usr/share/ca-certificates/mozilla/ (package ca-certificates)
It contains the certificate authorities shipped with Mozilla’s browser to allow SSL-based applications to check for the authenticity of SSL connections.

That’s the reason internet browsers won’t show any security messages when you visit websites using https with a certificate issued from a trusted and well-known commercial Certificate Authority.

You can also sign the CSR yourself but this will cause browsers to display warning messages to users attempting to reach your server until they explicitly grant their browsers permission to communicate with it.
If you use a self-signed certificate, your browser will throw a security warning.
The reason is that internet browsers only trust certificates from a trusted Certificate Authority.

For this reason it is recommended only to self-sign a certificate for testing purposes or only if your server needs to be reached internally.

Install OpenSSL (Debian)

# apt install openssl

The /etc/ssl/certs directory is used to store the public certificates, it should already exists on the server.
You may need to create an /etc/ssl/private directory as well, to store the private keys.
Since the secrecy of those private keys is essential for security, it’s important to lock down the permissions to prevent unauthorized access.

# mkdir /etc/ssl/private

$ ls -l /etc/ssl
drwxr-xr-x   root root      certs
-rw-r--r--   root root      openssl.cnf
drwx--x---   root ssl-cert  private

Generate a private key + Certificate Signing Request

Create a specific directory to store CSRs

# mkdir /etc/ssl/csr

Create a private key and generate a Certificate Signing Request from it

# openssl genrsa -out /etc/ssl/private/domain-name.key 2048
# openssl req -new -key /etc/ssl/private/domain-name.key -out /etc/ssl/csr/domain-name.csr

OR 

# openssl req -newkey rsa:2048 -nodes -keyout /etc/ssl/private/domain-name.key -out /etc/ssl/csr/domain-name.csr

You may be prompted to enter extra attributes including an optional challenge password.
Just skip this step by hitting the enter button.

Examine and verify Certificate Signing Request

$ openssl req -in /etc/ssl/csr/domain-name.csr -text -verify -noout

What is a Self Signed Certificate?

A self-signed certificate is an SSL/TSL certificate not signed by a public or private Certification Authority.
Instead, it is signed by the creator’s own private key or root CA certificate.

Many organizations use self-signed certificated for their internal applications that are not internet-facing.
These certificates are generated using the organization’s internal PKI infrastructure.

A self-signed certificate will encrypt communication between your server and any clients.
However, because it is not signed by any of the trusted certificate authorities included with web browsers, users cannot use the certificate to validate the identity of your server automatically.

Create a private key AND a self-signed public certificate with OpenSSL – Method 1

# openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 -keyout /etc/ssl/private/hostname.key -out /etc/ssl/certs/selfsigned-hostname.crt

openssl req
OpenSSL command line tool, PKCS#10 X.509 Certificate Signing Request (CSR) Management.

-x509
This option tells openssl you want to make a self-signed certificate instead of generating a certificate signing request.

-sha256
This option tells openssl to sign with a sha256 message digest

-nodes
This option tells openssl not to encrypt the private key with a passphrase.
The web server needs to be able to read the file without user intervention, when the server starts up.
A passphrase would prevent this from happening because you would have to enter it after every restart.

-days 3650
This option sets how many days the certificate will be considered valid (3650 = 10 years)

-newkey rsa:2048
This option specifies that you want to generate a new private key at the same time.
(RSA key that is 2048 bits long)

-keyout
This option tells openssl where to place the generated private key file

-out
This option tells openssl where to place the self-signed certificate

Note: It is important that you enter your domain/host name or your server’s IP address when you’re prompted for the Common Name.

Display the contents of the certificate
$ openssl x509 -in /etc/ssl/certs/selfsigned-hostname.crt -text -noout

Create a private key AND a self-signed public certificate with OpenSSL – Method 2

Step 1 - Create a private key
# openssl genrsa -out /etc/ssl/private/hostname.key 2048

Step 2 - Create a CSR
# openssl req -new -key /etc/ssl/private/hostname.key -out /etc/ssl/csr/hostname.csr

NOTE: Step 1 and 2 can be combined in one line
# openssl req -new --nodes newkey rsa:2048 -keyout /etc/ssl/private/hostname.key -out /etc/ssl/csr/hostname.csr

Step 3 - Check the CSR
$ openssl req -in /etc/ssl/csr/hostname.csr -text -noout

Step 4 - Create Self-signed Certificate
# openssl x509 -req -days 3650 -in /etc/ssl/csr/hostname.csr -signkey /etc/ssl/private/hostname.key -out /etc/ssl/certs/selfsigned-hostname.crt

Create a private key AND a self-signed public certificate with OpenSSL – Method 3: own Certification Authority (CA)

IMPORTANT
For security reasons, Certification Authority's private key and certificate SHOULD BE safeguarded on a separate machine.

# mkdir /etc/ssl/MyOwnCA


Step 1 - Create Certification Authority rootCA.key and rootCA.crt
We will use the rootCA.key and rootCA.crt to sign the SSL/TLS certificate.

# openssl req -x509 -nodes  -newkey rsa:2048 -keyout /etc/ssl/MyOwnCA/rootCA.key -days 3650 -sha256 -out /etc/ssl/MyOwnCA/rootCA.crt

Country Name (2 letter code) []: FR
State or Province Name (full name) []: *
Locality Name (eg, city) []:
Organization Name (eg, company) []: COMPANY  
Organizational Unit Name (eg, section) []: PKI ADMINISTRATION
Common Name (e.g. server FQDN or YOUR name) []: COMPANY-ROOT-CA
Email Address []:

Step 2 - Create the Host Private Key and Certificate Signing Request
# openssl req -newkey rsa:2048 -nodes -keyout /etc/ssl/private/hostname.key -out /etc/ssl/csr/hostname.csr

Step 3 - Generate X.509 certificate signed with own CA
# openssl x509 -req -in /etc/ssl/csr/hostname.csr -CA /etc/ssl/MyOwnCA/rootCA.crt -CAkey /etc/ssl/MyOwnCA/rootCA.key -CAcreateserial -days 3650 -sha256 -out /etc/ssl/certs/ownCAsigned-hostname.crt

Install X.509 Certificate on Web Server (Apache – Debian)

In order to set up and use a (self-signed) certificate, you first have to be sure that Apache ssl module which provides support for SSL encryption, is installed and enabled on the server.

# a2enmod ssl

Create an Apache directory for the hosts certificates and private keys

# mkdir /etc/apache2/certificates

Set up the virtual host to use the certificate

# vi /etc/apache2/sites-available/website.conf

<VirtualHost *:443>
...
        SSLEngine on
        SSLCertificateFile /etc/apache2/certificates/website.crt
        SSLCertificateKeyFile /etc/apache2/certificates/website.key
</VirtualHost>


Optionally, you may want to automatically redirect HTTP traffic to HTTPS

<VirtualHost *:80>
...
        Redirect "/" "https://website"
</VirtualHost>

It was created in 1996 by Andrew Tridgell and Paul Mackerras.
It is currently maintained by Wayne Davison.

rsync is freely available under the GNU General Public License.
rsync source code is here

The rsync algorithm is a type of delta encoding, and is used for minimizing network usage.
It efficiently computes/identify which parts (splitted blocks by fragmentation) of a source file match some part of an existing destination file (those parts don’t need to be sent across the communication link), thus minimizing the amount of data to transfer by only moving the portions of files that have changed.

For further speed improvements, the data sent to the receiver can be compressed using any of the supported algorithms.

ssh is the default remote shell for rsync since version 2.6.0 (January 1st 2004)

Install rsync (Debian)
# apt install rsync

rsync version
$ rsync -V

Usage

Local SRC > Local DST
rsync [OPTIONS] SRC [DST]

Push (Local SRC > Remote DST) rsync [OPTIONS] SRC [USER@]HOST:DST
Pull (Local DST < Remote SRC) rsync [OPTIONS] [USER@]HOST:SRC [DST]

Usages with just one SRC arg and no DST arg will list the source files instead of copying:
<type><perms_rwx> <size_bytes> <mtime YYYY/MM/DD> <mtime hh:mm:ss> <relative_path>

IMPORTANT: rsync must be installed on both the source and destination machines

If you still has this error:
rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream ...

It means Local rsync cannot find the remote rsync executable.
In this case you need to know the path of the remote host’s rsync binary and make it part of the command with --rsync-path=/path/to/remote/rsync

$ which rsync
/usr/bin/rsync  (Debian)

Options

If --delete option is specified, rsync will identify the files NOT present on the sender and delete them on the receiver. This option can be dangerous if used incorrectly! It is recommended to do a simulation run before, using the --dry-run option (-n) to find out which files are going to be deleted.

Each file from the list generated by rsync will be checked to see if it can be skipped.
In the most common mode of operation, files are not skipped if the modification time or size differs.

rsync performs a slower but comprehensive check if invoked with --checksum option.
This forces a full checksum comparison on every file present on both systems.

--checksum, -c
Skip files based on checksum, not mtime AND size.
This changes the way rsync checks if the files have been changed and are in need of a transfer.
Without this option, rsync uses a “quick check” that (by default) checks if each file’ size and time of last modification match between the sender and receiver.
This option changes this to compare a 128-bit checksum for each file that has a matching size.
Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer, so this can slow things down significantly (and this is prior to any reading that will be done to transfer changed files)

--human-readable, -h
Output numbers in a more human-readable format.
Unit letters: K (Kilo), M (Mega), G (Giga), T (Tera), or P (Peta).

--dry-run, -n
Simulation run (no changes made)

--verbose, -v
Increases the amount of information you are given during the transfer.
By default, rsync works silently.
A single -v will give you information about what files are being transferred and a brief summary at the end.
Two -v options will give you information on what files are being skipped and slightly more information at the end.
More than two -v options should only be used if you are debugging rsync.

--quiet, -q
Decreases the amount of information you are given during the transfer, notably suppressing information messages from the remote server. This option is useful when invoking rsync from cron.

--info=FLAGS
Choose the information output
An individual flag name may be followed by a level number, with 0 meaning to silence that output, 1 being the default output level, and higher numbers increasing the output of that flag (for those that support higher levels).
$ rsync --info=help
$ rsync -av --info=progress2 SRC/ DST/

--progress
Print information showing the progress of the transfer.
This is the same as specifying '--info=flist2,name,progress' but any user-supplied settings for those info flags takes precedence (e.g. --info=flist0 --progress).

While rsync is transferring a regular file, it updates a progress line that looks like this:
<reconstructed_bytes> <%_current_file> <throughput/sec> <remaining_time>

When the file transfer is done, rsync replaces the progress line with a summary line that looks like this:
<filesize_bytes> 100% <throughput/sec> <elapsed_time> (xfr#?, to-chk=???/N)
where ? is the nth transfer, ??? is the remaining files for the receiver to check (to see if they are uptodate or not)

In an incremental recursion scan (--recursive), rsync doesn’t know the total number of files in the files list until it reaches the end of the scan. Since it starts transfering files during the scan, it displays a line with the text “ir-chk” (for incremental recursion check) instead of “to-chk” until it knows the full size of the list, at which point it switches to “to-chk”. “ir-chk” lets you know that the number of files in the files list is still going to increase.

--archive, -a
It is equivalent to -rlptgoD
This is a quick way of saying you want recursion and want to preserve almost everything.
Be aware that it does not include preserving ACLs (-A), xattrs (-X), atimes (-U), crtimes (-N), nor the finding and preserving of hardlinks (-H).
The only exception to the above equivalence is when --files-from is specified, in which case -r is not implied.

--recursive, -r
This tells rsync to copy directories recursively. See also --dirs (-d).
Beginning with rsync 3.0.0, the recursive algorithm used is now an incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed.
It is only possible when both ends of the transfer are at least version 3.0.0.

Some options require rsync to know the full files list, these options disable the incremental recursion mode.
These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates.

Because of this, the default delete mode when you specify --delete is now --delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during to request this improved deletion mode explicitly).
See also the –delete-delay option that is a better choice than using –delete-after.

Incremental recursion can be disabled using the --no-inc-recursive option or its shorter --no-i-r alias.

--delete-during, --del
Request that the file deletions on the receiving side be done incrementally as the transfer happens.
The per-directory delete scan is done right before each directory is checked for updates, so it behaves like a more efficient --delete-before. This option was first added in rsync version 2.6.4. See --delete (which is implied) for more details on file deletion.

--delete-before
Request that the file deletions on the receiving side be done before the transfer starts.
It does imply a delay before the start of the transfer, and this delay might cause the transfer to timeout (if --timeout was specified). It also forces rsync to use the old, non-incremental recursion algorithm that requires rsync to scan all the files in the transfer into memory at once (see --recursive).

--delete-after
Request that the file deletions on the receiving side be done after the transfer has completed.
Important: this option forces rsync to use the old, non-incremental recursion algorithm that requires rsync to scan all the files in the transfer into memory at once (see --recursive). Use --delete-delay instead.

--delete-delay
Request that the file deletions on the receiving side be computed during the transfer (like –delete-during), but removed after the transfer completes. This is more efficient than using --delete-after.
If the number of removed files overflows an internal buffer, a temporary file will be created on the receiving side to hold the names. If the creation of the temporary file fails, rsync will try to fall back to using --delete-after (which it cannot do if --recursive is doing an incremental scan).

--links, -l
By default, symbolic links are not transferred at all.
A message "skipping non-regular" file is emitted for any symlinks that exist.
If --links is specified, then symlinks are recreated with the same target on the destination.
Note that --archive implies --links.

--perms, -p
Preserve permissions
This option causes the receiving rsync to set the destination permissions to be the same as the source permissions.
(See also the --chmod option for a way to modify what rsync considers to be the source permissions)

When this option is off, permissions are set as follows:

– Existing files (including updated files) retain their existing permissions, though the --executability option might change just the execute permission for the file.

– New files get their "normal" permission bits set to the source file’s permissions masked with the receiving directory’s default permissions (either the receiving umask, or the permissions specified via the destination directory’s default ACL), AND their special permission bits disabled except in the case where a new directory inherits a setgid bit from its parent directory.

Thus, when --perms and --executability are both disabled, rsync’s behavior is the same as that of other file copy utilities, such as cp(1) and tar(1).

In summary:
To give destination files (both existing and new) the source permissions, use --perms.
To give new files the destination default permissions (while leaving existing files unchanged), make sure that the --perms option is off and use --chmod=ugo=rwX (which ensures that all non-masked bits get enabled).

The preservation of the destination’s setgid bit on newly-created directories when –perms is off was added in rsync 2.6.7.

--times, -t
Preserve modification times
This tells rsync to transfer modification times along with the files and update them on the remote system.
Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective.
In other words, a missing -t or -a will cause the transfer to behave as if it used --ignore-times, causing all files to be updated (though rsync’s delta-transfer algorithm will make the update fairly efficient if the files haven’t actually changed, you’re much better off using -t).

--ignore-times, -I
Normally rsync will skip any files that are already the same size and have the same modification timestamp.
This option turns off this "quick check" behavior, causing all files to be updated.

--atimes, -U
Preserve access times
This tells rsync to set the access (use) times of the destination files to the same value as the source files.
nanoseconds are not preserved (set to .000000000), command cp -a does.

IMPORTANT:
There is no option to preserve ctime "status time"
(the timestamp used to record when the inode changed, it is specific to a filesystem)
An inode changes if any of its attributes are updated:
– at creation time (new file)
– file name
– mode/permissions
– owner/group
– hard link count
etc.

The creation of a file is one of the conditions listed above (creation of inode/file).
ctime cannot be preserved when files are brought into a new filesystem.

--open-noatime
Avoid changing the atime on opened file
This tells rsync to open files with the O_NOATIME flag (on systems that support it) to avoid changing the access time of the files that are being transferred. If your OS does not support the O_NOATIME flag then rsync will silently ignore this option. Note also that some filesystems are mounted to avoid updating the atime on read access even without the O_NOATIME flag being set.

--crtimes, -N
MAY NOT BE SUPPORTED, DEPENDS ON THE FILESYSTEM.
This tells rsync to set the create times (newness) of the destination files to the same value as the source files.

--group, -g
Preserve group
This option causes rsync to set the group of the destination file to be the same as the source file.
If the receiving program is not running as the super-user (or if --no-super was specified), only groups that the invoking user on the receiving side is a member of will be preserved. Without this option, the group is set to the default group of the invoking user on the receiving side.

--owner, -o
This option causes rsync to set the owner of the destination file to be the same as the source file, but only if the receiving rsync is being run as the super-user (see also the --super and --fake-super options).
Without this option, the owner of new and/or transferred files are set to the invoking user on the receiving side.

--acls, -A
This option causes rsync to update the destination ACLs to be the same as the source ACLs.
The option also implies --perms.
The source and destination systems must have compatible ACL entries for this option to work properly.
See the --fake-super option for a way to backup and restore ACLs that are not compatible.

--xattrs, -X
This option causes rsync to update the destination extended attributes to be the same as the source ones.

--hard-links, -H
This tells rsync to look for hard-linked files in the source and link together the corresponding files on the destination. Without this option, hard-linked files in the source are treated as though they were separate files.

This option does NOT necessarily ensure that the pattern of hard links on the destination exactly matches that on the source.

Usual Usage

Local SRC_DIR > Local DST_DIR
NOTE: By default, if Local DST_DIR does not exist it is created

Copy SRC_DIR inside /path/to/local/DST_DIR/ : /path/to/local/DST_DIR/SRC_DIR
$ rsync -av --info=progress2 /path/to/local/SRC_DIR /path/to/local/DST_DIR

Copy <src_path>'s content inside <dst_path>/
$ rsync -av --info=progress2 <src_path>/ <dst_path>/
$ rsync -av --info=progress2 <src_path>/* <dst_path>/

Local SRC_FILE > Local DST

$ rsync -av --info=progress2 /path/to/local/SRC_FILE /path/to/local/DST

IF DST is a directory, it copies SRC_FILE inside DST (DIR)

IF DST is a file, its content is replaced with the content of SRC_FILE
(with -a option ONLY mtime is the same, atime, ctime are different, you may use -U + -N options)

If DST does not exist :
– if there is a trailing slash ‘/’ it creates the directory DST (only for a direct subdirectory of an existing path “mkdir -p does not work” ) AND copies SRC_FILE inside DST

– otherwise it creates file DST (copy of SRC_FILE)

Portable Document Format

Adobe created the PDF in 1992 by Dr. John Warnock, offering an easy, reliable way to present and exchange documents regardless of the software, hardware, or operating systems being used.
Today, it is one the most trusted file formats around the world, it can be easily viewed on any operating system.

PDF was standardized as ISO 32000 in 2008 as an open standard.
The PDF format is now maintained by the International Organization for Standardization (ISO).
ISO 32000-2:2020 edition was published in December 2020, it does not include any proprietary technologies.

The PDF specification also provides for encryption (in which case a password is needed to view or edit the contents), digital signatures (to provide secure authentication), file attachments, and metadata.
PDF 2.0 defines 256-bit AES encryption as standard for PDF 2.0 files.

The standard security provided by PDF consists of two different passwords:

– user password, which encrypts the file and prevents opening

– owner password, which specifies operations that should be restricted even when the document is decrypted, which can include modifying, printing, or copying text and graphics out of the document, or adding or modifying text notes.

The user password encrypts the file, the owner password does not, instead relying on client software to respect content restrictions.
An owner password can easily be removed by software.
Thus, the used restrictions that an author places on a PDF document are not secure, and cannot be assured once the file is distributed.

Metadata includes information about the document and its content, such as the author’s name, document title, description, creation/modification dates, application used to create the file, keywords, copyright information, etc.

Install qpdf (Debian)

# apt install qpdf

Usage

--linearize
Create linearized (web-optimized) output file.
Linearized files are formatted in a way that allows compliant readers to begin displaying a PDF file before it is fully downloaded.
Ordinarily, the entire file must be present before it can be rendered because important cross-reference information typically appears at the end of the file.

$ qpdf --linearize infile.pdf  outfile.pdf

Merge PDF files with pages selection

qpdf allows you to use the --pages option to select pages from one or more input files.

$ qpdf primary_input_file.pdf --pages . [--password=password] [page-range] [ ... ] -- outputfile.pdf

Within [ ... ] you may repeat the following:  inputfile_N.pdf [--password=password] [page-range]

The special input file '.' can be used as an alias for the primary input file.
Multiple input files may be specified and you can select specific pages from it.
For each inputfile that pages should be extracted from, specify the filename, a password (if needed) to open the file, and a page range.
Note that '--' terminates parsing of page selection flags.

--password=password specifies a password for accessing encrypted files
The password option is only needed for password-protected files

The page range may be omitted. In this case, all pages are included.

Document-level information (metadata, outline, etc.) is taken from the primary input file (in the above example, primary_input_file.pdf) and is preserved in outputfile.pdf
You can use --empty in place of the primary input file to start from an empty file (without any metadata, outline, etc.) and just merge selected pages from input files.

In most cases you will most likely use this following syntax

$ qpdf --empty --pages inputfile_1.pdf [page-range] inputfile_2.pdf [page-range] inputfile_3.pdf [page-range] [ ... ] -- outputfile.pdf

The page-range is a set of numbers separated by commas, ranges of numbers separated dashes, or combinations of those.
The character 'z' represents the last page.
A number preceded by an 'r' indicates to count from the end, so r3-r1 would be the last three pages of the document.
Pages can be specified in any order (selection of any pages).
Ranges can be specified in any order (ascending or descending): a high number followed by a low number causes the pages to appear in reverse.
Numbers may be repeated in a page range.
A page range may be optionally appended with :even or :odd to indicate only the even or odd pages in the given range.
Note that even and odd refer to the positions within the specified, range, not whether the original number is even or odd.

Example page ranges:

1,3,5-9,15-12
Pages 1, 3, 5, 6, 7, 8, 9, 15, 14, 13, and 12 in that order

z-1
All pages in the document in reverse

r3-r1
The last three pages of the document

r1-r3
The last three pages of the document in reverse order

1-20:even
Even pages from 2 to 20

5,7-9,12:odd
Pages 5, 8 and 12, which are the pages in odd positions from among the original range (pages 5, 7, 8, 9, and 12)

Example, to extract pages 1 through 5 from infile.pdf while preserving all metadata associated with that file in outfile.pdf
$ qpdf infile.pdf --pages . 1-5 -- outfile.pdf

If you want pages 1 through 5 from infile.pdf without any metadata, use
$ qpdf --empty --pages infile.pdf 1-5 -- outfile.pdf

Merge all .pdf files
$ qpdf --empty  --pages *.pdf -- outfile.pdf

Split a PDF into separate PDF files

--split-pages[=n]
Write each group of n pages to a separate output file.
If n is not specified, create single pages.

Output file names are generated as follows:
If the string %d appears in the output file name, it is replaced with a range of zero-padded page numbers starting from 1.
Otherwise, if the output file name ends in .pdf (case insensitive), a zero-padded page range, preceded by a dash, is inserted before the file extension.
Otherwise, the file name is appended with a zero-padded page range preceded by a dash.

Zero padding is added to all page numbers in file names so that all the numbers are the same length, which causes the output filenames to sort lexically in numerical order.

Page ranges are a single number in the case of single-page groups or two numbers separated by a dash otherwise.

Here are some examples. In these examples, infile.pdf has 20 pages

Output files are 01-outfile through 20-outfile with no extension
$ qpdf --split-pages infile.pdf %d-outfile

Output files are outfile-01.pdf through outfile-20.pdf
$ qpdf --split-pages infile.pdf outfile.pdf

Output files are outfile-01-04.pdf, outfile-05-08.pdf, outfile-09-12.pdf, outfile-13-16.pdf, outfile-17-20.pdf
$ qpdf --split-pages=4 infile.pdf outfile.pdf

Output files are outfile.notpdf-01 through outfile.notpdf-20
The extension .notpdf is not treated in any special way regarding the placement of the number
$ qpdf --split-pages infile.pdf outfile.notpdf

Note that metadata, outline, etc, and other document-level features of the original PDF file are not preserved.
For each page of output, this option creates an empty PDF and copies a single page from the output into it.
If you require the document-level data, you will have to run qpdf with the --pages option once for each page.
Using --split-pages is much faster if you don’t require the document-level data.

If you don’t want to split out every page, use page ranges to select the pages you only want to extract.
The page range is used to specify the pages or ranges you want, but each extracted page is still stored in a single PDF.

$ qpdf --split-pages infile.pdf outfile.pdf --pages infile.pdf 4-5,8,9-13 --

https://github.com/ITEC4B/ASCII-random-string-generator

Hard Links

Every file on the Linux filesystem starts with a single hard link.
The link is between the filename and the actual data stored on the filesystem (directory entry > inode > data blocks).

When you create a hard link you create a file that gets the same inode as the target file.
You have different file names/paths for a unique physical file on a partition (pointing to the same inode)

Hard links can only be created for regular files (not directories or special files) and ONLY within the same filesystem.
A hard link cannot span multiple filesystems.

If you delete the "original file", you can still access it via any remaining hard link having the same inode.
Apart from the filename/filepath, you cannot tell which one is the hard link since they share the same inode.

$ ln /path/to/target_file /path/to/hardlink

List inodes files from <dir>, Recursive, No Sort, date (mtime)
CMD_LS_RECURSIVE_INODE_MTIME_NOSORT
OUTPUT:
<inode> <mode?rwx> <links> <uname> <gname> <size_bytes> <date YYYY-MM-DD> <time hh:mm:ss> <filepath>

$ LC_ALL=C ls -ilR --time-style='+%F %T' <dir> 2>/dev/null | sed -e '/:$/,/^total [0-9]\{1,\}/d' -n -e '/^[0-9]\{1,\} -/p' | tr -s '\n'


List inodes with hard link(s) from <dir>, Recursive, Natural Sort (inode first), date (mtime)
OUTPUT:
<inode> <mode?rwx> <links> <uname> <gname> <size_bytes> <date YYYY-MM-DD> <time hh:mm:ss> <filepath>

$ <CMD_LS_RECURSIVE_INODE_MTIME_NOSORT> | awk '$3 > 1 {print $0}' | sort


Find inodes with hard link(s)
OUTPUT: <inode>
$ <CMD_LS_RECURSIVE_INODE_MTIME_NOSORT> | awk '$3 > 1 {print $1}' | sort -u

Soft/Symbolic Links

When you create a soft link, you create a new file with a new inode, which points to the target path.
It doesn’t reference the target inode. If the target’s path/name changes or is deleted, the reference breaks (pointing to a nonexistent file path).

Symbolic links can link together non-regular and regular files.
They also can span multiple filesystems.

A symbolic link is identified by the mode lrwxrwxrwx, you cannot change it, so it is easy to identify them.
The symbolic link’ size is the target’s path length.

Changing owner, group, permissions of a symbolic link only has effect on the target file, in this case target file’s ctime is updated.

A change to a symlink name, updates its access time (atime) and status time (ctime).
That is the only thing you can change for the symbolic file itself.

$ ln -s /path/to/target_file /path/to/symlink

NOTE:
'stat' command provides 'Size:' and 'Blocks:' informations
'Size:' is the data file' size in bytes (logical size)
'Blocks:' is the real disk usage in blocks of 512 bytes (physical size)

List size files from DIR
OUTPUT: <logical_size_bytes> <physical_size_bytes> <filepath>

$ LC_ALL=C find DIR -type f -exec stat -c '%s %b %B %n' {} + 2>/dev/null | awk '{ fname=""; for (i=4; i <= NF; i++) fname=fname $i " "; print $1" "($2*$3)" "fname }'

Linux ext filesystem uses a default 4096 bytes for a block size because that’s the default pagesize of CPUs, so there’s an easy mapping between memory-mapped files and disk blocks.
The hardware (specifically, the Memory Management Unit, which is part of the CPU) determines what page sizes are possible. It is the smallest unit of data for memory management in a virtual memory operating system. Almost all architectures support a 4kB page size. Modern architectures support larger pages (and a few also support smaller pages), but 4kB is a very widespread default.

Get the filesystem block size in bytes
(size used internally by kernel, it may be modified by filesystem driver on mount)
# blockdev --getbsz /dev/<device>

Get the system's page size
(number of bytes in a memory page, where "page" is a fixed-length block, the unit for memory allocation and file mapping)
$ getconf PAGE_SIZE
$ getconf PAGESIZE

Print inode's metadata for a specific file/dir using stat command
$ LC_ALL=C stat /path/to/file_or_dir
$ LC_ALL=C stat -c '%i %y %A %U %s %N' /path/to/file_or_dir | sed -e 's;[.][0-9]\{9\} +[0-9]\{4\};;g'

Get inode number(s) with ls -i
$ ls -i1 /path/to/file_or_dir

Get the number of blocks a file uses on disk, so you can calculate disk space really used per file (physical file size).
IMPORTANT:
By default 'ls', 'du' and 'df' commands use 1block=1024bytes which may differ from the filesystem unit. You can use --block-size option or set environment variables:
Display values are in units of the first available SIZE from --block-size, DF_BLOCK_SIZE, BLOCK_SIZE and BLOCKSIZE environment variables. Otherwise, units default to 1024 bytes (or 512 if POSIXLY_CORRECT is set).

$ du --block-size=4096 /path/to/file
$ ls -s --block-size=4096 /path/to/file

ls -l prints the data size in bytes (logical file size), which is less than the actual used space on disk.

Inodes Metadata

$ man inode

– Inode number
Each file in a filesystem has a unique inode number (except for hard links).
Inode numbers are guaranteed to be unique only within a filesystem (i.e. the same inode numbers may be used by different filesystems, which is the reason that hard links may not cross filesystem boundaries).

– Device where inode resides
Each inode (as well as the associated file) resides in a filesystem that is hosted on a device.
That device is identified by the combination of its major ID (which identifies the general class of device) and minor ID (which identifies a specific instance in the general class).

– Device represented by this inode
If the current file (inode) represents a device, then the inode records the major and minor ID of that device.

– Links count (number of hard links to the file)

– User ID (of the owner of the file)

– Group ID (of the file)

– Mode: File Type + Permissions (read, write and execute permissions of the file for the owner, group and others)
The standard Unix file types are regular, directory, symbolic link, FIFO (named pipe), block device, character device, and socket as defined by POSIX.

– File size (in bytes)
This field gives the size of the file (if it is a regular file) in bytes.
The size of a symbolic link is the length of the pathname it contains, without a terminating null byte.
Default size for a directory is usually one block size (4096 bytes on most ext4 filesystems).

– Preferred block size for I/O operations (in bytes)
This field gives the "preferred" block size for efficient filesystem I/O operations.
(Writing to a file in smaller chunks may cause an inefficient read-modify-rewrite)

– Number of blocks allocated to the file
This field indicates the number of blocks allocated to the file in 512-byte units

– File creation (birth) timestamp (btime)
This is set on file creation and not changed subsequently.
The btime timestamp was not historically present on UNIX systems and is not currently supported by most Linux filesystems.

– Last modification timestamp (mtime)
This is the file’s last modification timestamp. It is changed by file modifications (file’s content: data).
Moreover, the mtime timestamp of a directory is changed by the creation or deletion of files in that directory.
The mtime timestamp is not changed for changes in file’s name, owner, group, hard link count, or mode.

– Last access timestamp (atime)
It is changed by file accesses.

– Last status change timestamp (ctime)
It is changed by modifying file’s metadata information (i.e. file’s name, owner, group, link count, mode, etc.).

According to The POSIX standard an inode is a "file serial number", defined as a per-filesystem unique identifier for a file.
Combined together with the device ID of the device containing the file, they uniquely identify the file within the whole system.

Two files can have the same inode, but only if they are part of different partitions (except for hard links).
Inodes are only unique on a partition level, not on the whole system.

Directory Entry

You may have noticed that inodes do not contain the file’s name.
The file’s name is not stored in the inode metadata but in its directory structure.
UNIX systems use a directory stream mapping system: directory entries contain the filenames and their inodes number.

From a user perspective a directory contains files, technically a directory is a structure used to locate other files/directories.
In most Unix filesystems, a directory is a mapping from filenames to inode numbers.
There’s a separate table mapping inode numbers to inode data.

The header file dirent.h describes the format of a directory entry.

Format of a Directory Entry
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html
https://www.gnu.org/software/libc/manual/html_node/Directory-Entries.html

In the glibc implementation, the dirent structure is defined as follows:

struct dirent {
   ino_t   d_ino;              /* Inode number */
   off_t   d_off;              /* Not an offset */
   unsigned short   d_reclen;  /* Length of this record */
   unsigned char   d_type;     /* Type of file; not supported by all filesystem types */
   char d_name[256];           /* Null-terminated filename */
};

The only fields in the dirent structure that are mandated by POSIX.1 are d_name and d_ino.
The other fields are unstandardized, and not present on all systems.

/* This is the data type of directory stream objects. */
typedef struct __dirstream DIR;

The DIR data type represents a directory stream.
You shouldn’t ever allocate objects of the struct dirent or DIR data types, since the directory access functions do that for you. Instead, you refer to these objects using the pointers returned by the functions.
Directory streams are a high-level interface.

The design of data block pointers is actually more complex than the schema illustrates below, it also depends on the filesystem. For ext filesystem an inode pointer structure is used to list the addresses of a file’s data blocks (around 15 direct/indirect data blocks pointers).

Filesystem

Linux uses filesystems to manage data stored on storage devices.
The filesystem manages a map (inodes table) to locate each file placed in the storage device.
The filesystem divides the partition into blocks: small contiguous areas.
The size of these blocks is defined during creation of the filesystem.

Before you can mount a drive partition, you must format it using a filesystem.

The default filesystem used by most Linux distributions is ext4.
ext4 filesystem provides journaling, which is a method of tracking data not yet written to the drive in a log file, called the journal. If the system fails before the data can be written to the drive, the journal data can be recovered and stored upon the next system boot.

After creating a partition, you need to create a filesystem (mkfs program is specifically dedicated for that)

#  LC_ALL=C mkfs -t ext4 /dev/<partition_id>

Some filesystems (ext4 included), allocate a limited number of inodes when created.
If the filesystem runs out of inode entries in the table, you cannot create any more files, even if there is still space available on the drive: that may happen with a multitude of very small files.
When a file is created on the partition or volume, a new entry in the inode table is created.
Using the -i option with the df command will show you the percentage of inodes used.
It is theoretically possible (although uncommon) that you could run out of available inodes while still having actual space left on your partition, so it’s worth keeping that in mind.

Report file system disk space usage

By blocks (most important)
$ LC_ALL=C df -Th --block-size=4096 -x tmpfs -x devtmpfs -x squashfs 2>/dev/null

By inodes
$ LC_ALL=C df -Ti -x tmpfs -x devtmpfs -x squashfs 2>/dev/null

Linux uses e2fsprogs package to provide utilities for working with ext filesystems

Package: build-essential
This package is normally required for building Debian packages, it is OK to install it since it includes all what is necessary to get a C/C++ Development Environment.

# apt install build-essential

Package: gcc-doc
Documentation for the GNU compilers (gcc, gobjc, g++)

# apt install gcc-doc

Virtual Package: libc-dev
Package: libc<version>-dev
GNU C Library: Development Libraries and Header Files
Contains the symlinks, headers, and object files needed to compile and link programs which use the standard C library

Package: libstdc++-<version>-dev
GNU Standard C++ Library v3 (development files)
This package contains the headers and static library files necessary for building C++ programs which use libstdc++.

Package: libstdc++-<version>-doc
GNU Standard C++ Library v3 (documentation files)
This package contains documentation files for the GNU stdc++ library.

Package: gcc
GNU C compiler
This is the GNU C compiler, a fairly portable optimizing compiler for C.

Package: g++
GNU C++ compiler
This is the GNU C++ compiler, a fairly portable optimizing compiler for C++.

Package: make
Utility for directing compilation
GNU make is a utility which controls the generation of executables and other target files of a program from the program’s source files.
It determines automatically which pieces of a large program need to be (re)created, and issues the commands to (re)create them.
make can be used to organize any task in which targets (files) are to be automatically updated based on input files whenever the corresponding input is newer. It is not limited to building computer programs. Indeed, make is a general purpose dependency solver.

Package: manpages-dev
Manual pages about using GNU/Linux for development
These man pages describe the Linux programming interface, including these two sections:
2 = Linux system calls.
3 = Library calls (note that a more comprehensive source of information may be found in the glibc-doc and glibc-doc-reference packages).

Package: ccache
Compiler cache for fast recompilation of C/C++ code
It speeds up recompilation by caching previous compilations and detecting when the same compilation is being done again.

$ LC_ALL=C dpkg-query -f '${Package} ${Version} ${Status}\n' -W | awk '{print $1" "$2" "$4" "$5}' | column -t

Look For Specific Package(s)

Multiple package names and glob patterns works

$ LC_ALL=C dpkg-query -f '${Package} ${Version} ${Status}\n' -W <pkg_name> | awk '{print $1" "$2" "$4" "$5}' | column -t

The process of translating source code into an executable program is called “compiling the program” or just “compiling”.
We usually view the compilation process as a single action and generally refer to it as such.
Nevertheless, a modern compiler actually consists of 4 separate programs:

- Preprocessor
  Expand macros and included header files

- Compiler
  Convert source code to assembly language

- Assembler
  Convert assembly language to machine code

- Linker
  Link object files and binary libraries, Create the final executable

So here is the process :

Source Code > Preprocessor > Compiler > Assembler > Linker > Executable Program

A single program usually consist of multiple source code files.
It is both awkward and inconvenient to deal with large programs in a single source code file, and spreading them over multiple files has many advantages:

1. It breaks large, complex programs into smaller, independent conceptual units
Easier to understand, follow and maintain.

2. It allows multiple programmers to work on a single program at the same time
Each programmer works on a separate set of files.

3. It may speed up compilation (depending on the compiler system options used)
The compiler system stores the generated machine code in an object file, one object file for each source code file. The compiler system may not delete the object files, so if the source code file is unchanged, the linker uses the existing object code file.

4. It permits related programs to share files
For example, office suites often include a word processor, a slide show editor, and a spreadsheet.
By maintaining the User Interface code in one shared file, they can present a consistent User Interface.

5. Although less important, it allows software developers to market software as object code organized as (binary black box) libraries, which is useful when supplying code that interfaces with applications.

Preprocessor

The Preprocessor takes the source code, removes the comments, includes headers, and replaces macros.

The preprocessor handles statements or lines of code that begin with the “#” character, which are called “preprocessor directives“.

Note that directives are not C/C++ statements (and therefore do not end with a semicolon) but rather instruct the preprocessor to carry out some action.

For each .c/.cpp file, the preprocessor handles directives that begin with the # character and creates a temporary file to store its output.
The preprocessor reads and processes each file one at a time from top to bottom.
It does not change the content of any of the source files it processes.

The results are files which contains the source code merged with headers files and with all macros expanded.
By convention, preprocessed files are given the file extension .i for C programs and .ii for C++ programs.
In practice, the preprocessed file is not saved to disk unless the -save-temps option is used.

Two of the most common directives, and the first that we will use, are #include and #define.

The #include Directive

When the preprocessor encounters the #include directive, it opens the header file and adds its contents into the temporary file.
The symbols surrounding the name of the header file are important and determine where the preprocessor looks for the file.

#include <name>
The angle brackets denote a system include file that is part of the compiler itself (think of it as “library” code)
and directs the preprocessor to search for the file where the system header files are located (which varies from one compiler to another and from one Operating System to another).

#include "name.h"
The double quotation marks identify a header file that is written as a part of a program.
The quotation marks instruct the preprocessor to look for the header file in the current directory (i.e., in the same directory as the source code).
Header files that a programmer writes as part of an application program typically end with a .h extension.

You might see two kinds of system header files in a C++ program :
Older system header files end with a “.h” extension: <name.h>.
These header files were originally created for C programs, but may also be used with C++.
Newer system header files do not end with an extension: <name>, may only be used with C++.

File names appearing between < and > refer to system header files
File names appearing between an opening and closing ” refer to header files written by the programmer as a part of the program.

Note:
The include directive does not end with a semicolon and there must be at least one space between the directive and the file name.

The #define Directive and Symbolic Constants

The #define directive introduces a programming construct called a macro.
A simple macro only replaces one string of characters with another string.

The #define directive is one (old) way of creating a symbolic constant (also known as a named or manifest constant).
The const and enum keywords are newer techniques for creating constants.
It is a well-accepted naming practice to write the names of symbolic constants with all upper-case characters (this provides a visual clue that the name represents a constant).

Note:
The define directive does not end with a semicolon and there must be at least one space between the directive and the identifier, and between the identifier and the defined value; the defined value (the third part of the directive) is optional.

Stop after the Preprocessing stage. 
The output is in the form of preprocessed source code, which is sent to the standard output.
Input files that don't require preprocessing are ignored.

$ gcc -E <program_file_1>.c <program_file_2>.c ... <program_file_n>.c

$ g++ -E <program_file_1>.cpp <program_file_2>.cpp ... <program_file_n>.cpp

Compiler

The Compiler translates source code into assembly code for a specific processor.

As the Preprocessor processes each source code file one at a time and produces a single temporary file (for each source code file).
Similarly, the Compiler processes each temporary file one at a time and produces one assembly code file for each temporary file.

The Compiler also detects syntax errors and provides the diagnostic output programmers use to find and correct those errors.
Despite all that the compiler does, its operation is transparent to programmers for the most part.

Stop after the stage of Compilation, do not Assemble. 
The output is in the form of an assembler code file for each non-assembler input file specified.
By default, the assembler file name for a source file is made by replacing the suffix .c, .cpp, .i, .ii, etc., with .s
Input files that don't require compilation are ignored.

$ gcc -S <program_file_1>.c <program_file_2>.c ... <program_file_n>.c

$ g++ -S <program_file_1>.cpp <program_file_2>.cpp ... <program_file_n>.cpp

Assembler

The Assembler translates assembly code into machine code the processor understands and can execute.

The purpose of the Assembler is to convert assembly language into machine code and generate an object file.

When there are calls to external functions in the assembly source file, the Assembler leaves the addresses of the external functions undefined, to be filled in later by the linker.

Compile AND Assemble the source files, but do not Link.
The output is in the form of an object file for each source file.
By default, the object file name for a source file is made by replacing the suffix .c, .cpp, .i, .ii, .s, etc., with .o
Unrecognized input files, not requiring compilation or assembly, are ignored.

$ gcc -c <program_file_1>.c <program_file_2>.c ... <program_file_n>.c

$ g++ -c <program_file_1>.cpp <program_file_2>.cpp ... <program_file_n>.cpp

Linker

The final stage of compilation is the linking of object files to create an executable program.

Object files contain machine code and information that the Linker uses to complete its tasks.
(Note that “object” in this context has nothing to do with the objects involved in Object-Oriented Programming)

This is where all of the object files and any additional binary libraries are linked together to make the final program.

It takes each object files created by the Assembler and links them together, along with system and runtime libraries, to form a complete, executable program.

An executable requires many external functions from system and runtime libraries.
They contain functions that are necessary to run a program on a given architecture
(linux-vdso.so.n, libc.so.n, ld-linux-x86-64.so.n (amd64), ld-linux.so.n (i386), etc.)

A library is a binary file (usually not directly executable) containing compiled functions/programming code that may be used/called by other programs/applications.

As a convention, a library name starts with ‘lib’, and the extension determines the type of the library:
.a stands for archive (static library)
.so stands for shared object (dynamic library)

Static Linking :
The linker adds all the libraries the program needs inside the final executable file (content is included).
Static linking may simplify the process of distributing a program to multiple similar environments, since it already has everything it needs to run. But any update to the libraries dependencies won’t be effective until you perform a whole compilation and linking process again.

Dynamic Linking :
The linker only places a reference to the required libraries in the final program (content is not included).
The actual linking happens when the program is executed (loaded at runtime).
You don’t need to recompile the program if any update occurs to the libraries dependencies, but they all need to be present/installed on the system for the program to work.

Libraries (binaries) Location

GNU C Library: Shared libraries   (package: libc<n>)
Contains the standard libraries that are used by nearly all programs on the system.

GNU Standard C++ Library v3       (package: libstdc++<n>)
Contains an additional runtime library for C++ programs built with the GNU compiler. 

Symbolic link /lib -> /usr/lib
On Debian 64-bits amd64 architecture:   /lib/x86_64-linux-gnu/
On Debian 32-bits i386 architecture:    /lib/i386-linux-gnu/

List of paths that ld (the linker) will search for libraries
The directories are searched in the order in which they are specified
$ ld --verbose | grep SEARCH_DIR | sed 's/; /\n/g'

The name of the executable file depends on the hosting Operating System:
On Linux, Unix, and macOS systems, the linker produces a file named ‘a.out’ by default.
On a Windows computer, the linker produces a file whose name ends with a .exe extension.

Users may also specify a name that overrides the default.

For example, if you want gcc to generate an executable with a specific name, use the -o option followed with the desired name:

$ gcc -o <program_name> <program_file_1>.c <program_file_2>.c ... <program_file_n>.c

$ g++ -o <program_name> <program_file_1>.cpp <program_file_2>.cpp ... <program_file_n>.cpp

When the compiling finishes, temporary/intermediate files are removed.

This command shows all shared library dependencies (what libraries the executable requires)

$ ldd <program_name>

readelf displays information about ELF format object files. 
The options control what particular information to display.
This program performs a similar function to objdump but it goes into more detail

$ readelf -a <program_name>

Loader

This stage happens when the program starts up.
The program is scanned for references to shared libraries.
Any references found are resolved and the libraries are mapped into the program.

The dynamic linker/loader programs ld.so (or ld.so.n) and ld-linux.so (or ld-linux.so.n) find and load the shared objects (shared libraries) needed/used by a program, prepare the program to run, and then run it.

In Debian:
$ ls -l /lib/$( arch )-linux-gnu/ld-linux*

$ <loader_program> <program_name>

To command a computer you need to speak its language.
Not all the computers “speak the same way”, there are different technical implementations and representation of instructions.

The instructions that a machine can understand is called the instruction set (range of instructions that a CPU can execute).

The Central Processing Unit (CPU), also called processor, is the electronic component that executes instructions.
It is one of the most important parts of any computer.
Every CPU has a set of built-in commands (the instruction set), these “basic” operations are hardwired into the CPU.
CPUs only understand those operations encoded in binary code, the low-level machine code language (native code).
Instructions are sequentially mixed together to make what is known a program.

In computer science, an Instruction Set Architecture (ISA) is an abstract model of a computer.
A device that executes instructions described by an ISA, such as a CPU, is called an implementation.

The only way you can interact with the hardware is through the instruction set of the processor.
The ISA specifies what the processor is capable of doing.

It is basically the interface between the hardware and the software.
It defines the supported data types, the registers, how the hardware manages main memory, key features (such as the memory consistency, addressing modes, virtual memory), which instructions a microprocessor can execute, and the input/output model of a family of implementations of the ISA.

It can be viewed as a “programmer’s manual”, the technical description of how it works and what you can do with it.

Each operation to perform from an instruction set is identified by a binary code known as an opcode (Operation Code).
The opcode is the first part of an instruction (the first bits).
It’s a unique code that identifies a specific operation.

On traditional architectures, an instruction includes an opcode that specifies the operation to perform AND zero or more operand specifiers, which may be registers, memory addresses, or literal data the operation will use or manipulate.

In Very Long Instruction Word (VLIW) architectures, multiple simultaneous opcodes and operands are specified in a single instruction.

The number of operands is one of the factors that may give an indication about the performance of the instruction set.

A word is the fixed-sized piece of data handled as a unit by the processor.

The number of bits in a word (word size) is an important characteristic of any specific processor design or computer architecture, it implies how many operations the computer is capable in a single word.

Computer Architecture

The von Neumann architecture is a computer architecture based on a 1945 description by John von Neumann, and by others, in the First Draft of a Report on the EDVAC (Electronic Discrete Variable Automatic Computer) one of the earliest electronic computers.

The report is an incomplete 101-page document written by hand by John von Neumann.
It contains the first published description of the logical design of a computer using the stored-program concept, which has controversially come to be known as the von Neumann architecture.

The document describes a design architecture for an electronic digital computer with these components:
– A Processing Unit with both an Arithmetic Logic Unit and processor registers
– A Control Unit that includes an Instruction Register and a Program Counter
– Memory that stores data and instructions
– External mass storage
– Input and output mechanisms

The von Neumann architecture is not perfect, an instruction fetch and a data operation cannot occur at the same time since they share a common bus. This is referred to as the von Neumann bottleneck, which limits the performance of the corresponding system.

A stored-program digital computer keeps both program instructions and data in read–write, random-access memory (RAM)

The vast majority of modern computers use the same memory for both data and program instructions, but have caches between the CPU and memory, and, for the caches closest to the CPU, have separate caches for instructions and data, so that most instruction and data fetches use separate buses (split cache architecture)

If based on the von Neumann architecture, processors contain at least a Control Unit (CU), an Arithmetic Logic Unit (ALU), and processor registers.

Every modern processor includes very small super-fast memory banks, called registers.
The registers are the fastest accessible memory location for the CPU and sit on the top of the memory hierarchy.
They can be read and written at high speed since they are internal to the CPU.
They are much smaller in size than local memory (size of a word: usually 64 or 32 bits) and are used to store machine instructions, memory addresses, and certain other values.

Data is loaded from the main memory to the registers (via the CPU cache) after which it undergoes various arithmetic operations.

The manipulated data is then written back to the memory via the CPU cache.

CPU’s cache memory is dedicated to hold (inside or close to the CPU) the most commonly used memory words, in order to avoid slower accesses to main memory (RAM).

Most CPUs have a hierarchy of multiple cache levels, with specific instruction and data caches at Level 1.
The L1 cache or first-level cache is the closest to the CPU, making it the type of cache with the highest speed and lowest latency of the entire cache hierarchy.

Instruction cache: used to speed up executable instruction fetch
Data cache: used to speed up data fetch and store

Instruction Cycle

A program is a sequence of instructions in memory.

The CPU executes operations through a cycle known as “Fetch, Decode, and Execute”.

The most important registers (Control Unit) are :
– Program Counter (PC), which points (holds the memory address) to the next instruction to be fetched for execution
– Instruction Register (IR), which holds the instruction currently being executed

1. Fetch the instruction from memory into the Instruction Register
2. Change the Program Counter register to point to the next instruction
3. Decode the instruction
      Determine the type of instruction (opcode)
      If the instruction operand is a word in memory: 
         Determine where it is located (memory address)
         Retrieve the data from memory into a register
4. Execute the instruction (ALU)
5. Go to step 1 to begin executing the next instruction

The operation code tells the ALU what operation to perform, the operands are used in the operation

Technology Evolution

Since the invention of the transistor (electronic switch) in 1947 by John Bardeen, Walter Brattain, William Shockley
AND the Silicon Integrated Circuit in 1958 by Jack Kilby and Robert Noyce
The computer industry development has never stopped, advances in technology has revolutionized computers, leading to smaller, faster, better products at lower prices.

Manufacturers have packed more and more transistors per chip every year, meaning larger memories and more powerful processors.

Latest processors contains billions of transistors.

Moore’s law is the observation that the number of transistors in an Integrated Circuit doubles about every two years.
It is an observation and projection of a historical trend since 1965.

While Moore’s law will probably continue to be proven for some years, it has a limit:
First, you cannot shrink a transistor size more than you can.
Second, you have problems of power consumption and heat dissipation.

Smaller transistors make it possible to run at higher clock frequencies, but also requires using a higher voltage.
That is, going faster (clock speed) means having more heat to get rid of.

The solution is the multi-core processor architecture: two identical CPUs on a chip consume less power than one CPU at twice the speed.
That is one of the reasons why processors have more and more cores and larger caches rather than higher clock speeds.

Taking advantage of these multiprocessors poses great challenges to programmers, it requires knowledge to explicitly control/manage parallel execution.

CPU Core

Before multi-core processor architecture, computers only had one CPU: the processor could only perform one instruction at a time.
A CPU core is a physical hardware processor with all the architecture that comes with it.
We now have multiple processors grouped inside one Integrated Circuit (single chip), running independently: This is real hardware parallelism (as long as the Operating System uses it).
The design is far more advanced, it requires a different architecture to orchestrate all this (controllers, buses, memory access, etc.).

This technology has allowed Machine Virtualization (standard practice in enterprise IT architecture), which is the foundation of Cloud Computing.

It allows the hardware elements of a single computer (processors, memory, storage, and more) to be divided into multiple virtual computers, commonly called Virtual Machines (VM). Each VM runs its own Operating System (OS) and behaves like an independent computer, even though it is running on just a portion of the actual underlying computer hardware.

The more cores there are in a CPU, the more efficient it is and the more you can do.

CPU Thread

Simultaneous MultiThreading (SMT) is a technique for improving the overall efficiency of CPUs with hardware multithreading.
SMT allows to better use the resources provided by modern processor architectures.

When SMT is operational, the Operating System sees the processor as having “double the cores” (Logical Processors).
Two logical cores can work through tasks more efficiently than a native single-threaded core, by taking advantage of idle time when the core would formerly be waiting for other tasks to complete.
It improves CPU throughput (usage optimization).

CPU Clock Speed

The clock speed measures the number of cycles your CPU executes per second, measured in GHz (gigahertz).
A cycle is the basic unit that measures a CPU’s speed.
During each cycle, billions of transistors within the processor open and close.

A CPU with a clock speed of 3.4 GHz executes 3.4 billion cycles per second. (Older CPUs had speeds measured in MegaHertz, or millions of cycles per second)

Sometimes, multiple instructions are completed in a single clock cycle.
In other cases, one instruction might be handled over multiple clock cycles.

How Do We Communicate With The Processor ?

Unless you are a supernatural alien coming from another galaxy, we use programming languages
(created by skillful and talented people)

Programming languages are often categorized as low-level, mid-level or high-level depending on
“how close you are from the hardware”.

Low-Level Programming Languages

low-level programming languages are hardware-dependent and machine-centered (tied to the hardware, providing operations matching the hardware’s capabilities).

low-level programs execute faster than high-level programs, with a small memory footprint.

Assembly language (asm), is any low-level programming language with a very strong correspondence between the instructions in the language and the processor’s instruction set.

Assembly is very close to machine code but is “more readable” and uses mnemonics.
You need to have a strong technical knowledge to use it (interaction with the hardware), Assembly is not easy.

The statements are made up of opcodes and operands (processor registers, memory addresses, etc.), which are translated into machine code (instructions that the processor understands).

One line of assembly equals one line of machine code.

Assembly code is converted into executable machine code by a utility program referred to as an assembler.

Each assembly language is specific to a particular computer architecture, it is not portable to a different type of architecture.

Mid-Level, High-Level Programming Languages

Most programming is done using high-level compiled or interpreted languages, which are easier for humans to understand, write, debug and do not require knowledge of the the system (hardware) running the program.

These languages need to be compiled (translated into system-specific machine code) by a compiler, or run through other system-specific compiled programs.

High-level programming languages are generally hardware-independent and problem-centered (providing operations supporting general problem-solving).
Programmers can move hardware-independent code from one computer to another fairly easily.

Delroy A. Brinkerhoff, Ph.D :

The C programming language is deemed a mid-level language because it allows programmers more access to the hardware than other higher-level languages.

We can locate C++ at two different places in this spectrum.

First, it represents a mid-level language because it retains C’s access to the hardware.
But second, it also represents a high-level language because it supports object-orientation, a problem-centered approach to programming.

The combination of high- and mid-level features makes C++ a popular choice for writing Operating Systems, games and large industrial applications.

Computers can’t directly execute programs written in high-level languages,
so there must be some way of translating a program written in a high-level language into machine language.

Two kinds of computer programs perform the necessary translation: compilers and interpreters.

A compiler is a program that translates other programs written in a high-level programming language like C or C++ into machine code or machine language.

Some languages such as Java and C# take a different route.
Compilers for these languages translate the high-level source code into an intermediate form (a representation that lies somewhere between the high-level and true machine code) called virtual machine code.

The virtual machine code then becomes the input to another program called an interpreter or Virtual Machine (VM), a program that simulates a hardware CPU. Note here that VM is a software component dedicated to run virtual machine code (runtime environment for applications), it is different from Hardware Virtualization.

Other languages, such as Javascript and Perl, are completely interpreted.
These languages don’t use compilers at all.
The interpreter reads the source code, written in the high-level language, and interprets the instructions one at a time.
That is, the interpreter itself carries out each instruction in the program.

Compiling and running a program written in a language that produces machine code
The compiler reads the C/C++ source code from a file that ends with .c or .cpp and produces a machine code file that is executable.
See C/C++ Compiler Operations

Compiling and running a program written in a language that produces virtual machine code
Languages like Java and C# are hybrid languages because they use both a compiler and a Virtual Machine.
They first compile the source code to virtual machine code, that is, to machine code for a virtual computer (a computer that doesn’t exist but is simulated by another computer).
After compiling the source code, a Virtual Machine (VM) executes the code by simulating the actions of a real computer.
The Operating System loads the VM into main memory and runs it.
It is the VM that reads and runs the virtual machine code.

Running a program written in a purely interpreted language
Languages like Javascript and Perl do not compile the source code at all.
Like the hybrid languages (Java and C#), the Operating System run the interpreter or VM.
The interpreter reads the source code file and executes the program one statement at a time without translating the whole program to any other language.
Web browsers incorporate interpreters for some languages (like Javascript) while the Operating System runs the interpreters for other languages (like Perl) as application programs.

High-Level Programming Languages Advantages and Disadvantages

Each approach to running a program written in a high-level programming language has advantages and disadvantages.

Programs written in fully compiled languages (e.g., C and C++) execute faster than programs written in partially compiled languages (e.g., Java and C#) and run much faster than programs written in fully interpreted languages (e.g., Javascript and Perl).

To give some idea of the difference in performance, let’s say that a C++ program, once compiled, executes in time 1.
A program in a hybrid language (compiled and interpreted) will generally run in time 3 to 10.
In a purely interpreted language, the same program runs in a time of about 100.

Contemporary versions of the Java and C# VMs use a Just In Time (JIT) interpreter that compiles some of the virtual code to machine code while processing it.
JIT processors reduce run time to about 1.5 times that of purely compiled language systems.

“How does Java compare in terms of speed to C or C++ or C# or Python? The answer depends greatly on the type of application you’re running. No benchmark is perfect, but The Computer Language Benchmarks Game is a good starting point.”

On the other hand, once we compile a program written in purely compiled languages, we can’t easily move the resulting executable machine code to a different platform (e.g., you can’t run a Windows program to an Apple computer).

In contrast, we can easily move programs we write in interpreted languages between different computers.

Interpreted programs are portable because they run on a VM or interpreter.
From the hardware and Operating System’s perspective, the interpreter is the running program.
Interpreters and VMs are written in purely compiled languages, so they are not portable, but the programs that they run are.
Once we install the interpreter on a system, we can move interpretable programs to the system and run them without further processing.

Execution speed is not the only criteria to take into consideration, there is also the speed/ease of development.

here is an article about Python speed.