Source code
Revision control
Copy as Markdown
Other Tools
.. _mozilla_projects_nss_an_overview_of_nss_internals:
An overview of NSS Internals
============================
.. container::
| A High-Level Overview to the Internals of `Network Security Services
| Software developed by the Mozilla.org projects traditionally used its own implementation of
security protocols and cryptographic algorithms, originally called Netscape Security Services,
nowadays called Network Security Services (NSS). NSS is a library written in the C programming
language. It's free and open source software, and many other software projects have decided to
use it. In order to support multiple operating systems (OS), it is based on a cross platform
portability layer, called the Netscape Portable Runtime (NSPR), which provides cross platform
application programming interfaces (APIs) for OS specific APIs like file system access, memory
management, network communication, and multithreaded programming.
| NSS offers lots of functionality; we'll walk through the list of modules, design principles,
and important relevant standards.
| In order to allow interoperability between software and devices that perform cryptographic
operations, NSS conforms to a standard called PKCS#11. (Note that it's important to look at the
number 11, as there are other PKCS standards with different numbers that define quite different
topics.)
| A software or hardware module conforming to the PKCS#11 standard implements an interface of C
calls, which allow querying the characteristics and offered services of the module. Multiple
elements of NSS's own modules have been implemented with this interface, and NSS makes use of
this interface when talking to those modules. This strategy allows NSS to work with many
hardware devices (e.g., to speed up the calculations required for cryptographic operations, or
to access smartcards that securely protect a secret key) and software modules (e.g., to allow
to load such modules as a plugin that provides additional algorithms or stores key or trust
information) that implement the PKCS#11 interface.
| A core element of NSS is FreeBL, a base library providing hash functions, big number
calculations, and cryptographic algorithms.
| Softoken is an NSS module that exposes most FreeBL functionality as a PKCS#11 module.
| Some cryptography uses the same secret key for both encrypting and decrypting, for example
password based encryption (PBE). This is often sufficient if you encrypt data for yourself, but
as soon as you need to exchange signed/encrypted data with communication partners, using public
key encryption simplifies the key management. The environment that describes how to use public
key encryption is called Public Key Infrastructure (PKI). The public keys that are exchanged
between parties are transported using a container; the container is called a certificate,
following standard X.509 version 3. A certificate contains lots of other details; for example,
it contains a signature by a third party that expresses trust in the ownership relationship for
the certificate. The trust assigned by the third party might be restricted to certain uses,
which are listed in certificate extensions that are contained in the certificate.
| Many (if not most) of the operations performed by NSS involve the use of X.509 certificates
(often abbreviated as “cert”, unfortunately making it easy to confuse with the term “computer
emergency response team“).
| When checking whether a certificate is trusted or not, it's necessary to find a relevant trust
anchor (root certificate) that represents the signing capability of a trusted third party,
usually called a Certificate Authority (CA). A trust anchor is just another X.509 certificate
that is already known and has been deliberately marked as trusted by a software vendor,
administrators inside an organizational infrastructure, or the software user. NSS ships a
predefined set of CA certificates. This set, including their trust assignments, is provided by
NSS as a software module, called CKBI (“built-in root certificates”), which also implements the
PKCS#11 interface. On an organizational level the contents of the set are managed according to
the Mozilla CA policy. On a technical level the set is a binary software module.
| A cryptographic transaction, such as encryption or decryption related to a data exchange,
usually involves working with the X.509 certs of your communication partners (peer). It's also
required that you safely keep your own secret keys that belong to your own certificates. You
might want to protect the storage of your secret keys with PBE. You might decide to modify the
default trust provided by NSS. All of this requires storing, looking up, and retrieving data.
NSS simplifies performing these operations by offering storage and management APIs. NSS doesn't
require the programmer to manage individual files containing individual certificates or keys.
Instead, NSS offers to use its own database(s). Once you have imported certificates and keys
into the NSS database, you can easily look them up and use them again.
| Because of NSS's expectation to operate with an NSS database, it's mandatory that you perform
an initialization call, where you tell NSS which database you will be using. In the most simple
scenario, the programmer will provide a directory on your filesystem as a parameter to the init
function, and NSS is designed to do the rest. It will detect and open an existing database, or
it can create a new one. Alternatively, should you decide that you don't want to work with any
persistent recording of certificates, you may initialize NSS in a no-database mode. Usually,
NSS will flush all data to disk as soon as new data has been added to permanent storage.
Storage consists of multiple files: a key database file, which contains your secret keys, and a
certificate database file which contains the public portion of your own certificates, the
certificates of peers or CAs, and a list of trust decisions (such as to not trust a built-in
CA, or to explicitly trust other CAs). Examples for the database files are key3.db and
cert8.db, where the numbers are file version numbers. A third file contains the list of
external PKCS#11 modules that have been registered to be used by NSS. The file could be named
secmod.db, but in newer database generations a file named pkcs11.txt is used.
| Only NSS is allowed to access and manipulate these database files directly; a programmer using
NSS must go through the APIs offered by NSS to manipulate the data stored in these files. The
programmer's task is to initialize NSS with the required parameters (such as a database), and
NSS will then transparently manage the database files.
| Most of the time certificates and keys are supposed to be stored in the NSS database.
Therefore, after initial import or creation, the programmer usually doesn't deal with their raw
bytes. Instead, the programmer will use lookup functions, and NSS will provide an access handle
that will be subsequently used by the application's code. Those handles are reference counted.
NSS will usually create an in-memory (RAM) presentation of certificates, once a certificate has
been received from the network, read from disk, or looked up from the database, and prepare
in-memory data structures that contain the certificate's properties, as well as providing a
handle for the programmer to use. Once the application is done with a handle, it should be
released, allowing NSS to free the associated resources. When working with handles to private
keys it's usually difficult (and undesired) that an application gets access to the raw key
data; therefore it may be difficult to extract such data from NSS. The usual minimum
requirement is that private keys must be wrapped using a protective layer (such as
password-based encryption). The intention is to make it easier to review code for security. The
less code that has access to raw secret keys, the less code that must be reviewed.
| NSS has only limited functionality to look up raw keys. The preferred approach is to use
certificates, and to look up certificates by properties such as the contained subject name
(information that describes the owner of the certificate). For example, while NSS supports
random calculation (creation) of a new public/private key pair, it's difficult to work with
such a raw key pair. The usual approach is to create a certificate signing request (CSR) as
soon as an application is done with the creation step, which will have created a handle to the
key pair, and which can be used for the necessary related operations, like producing a
proof-of-ownership of the private key, which is usually required when submitting the public key
with a CSR to a CA. The usual follow up action is receiving a signed certificate from a CA.
(However, it's also possible to use NSS functionality to create a self-signed certificate,
which, however, usually won't be trusted by other parties.) Once received, it's sufficient to
tell NSS to import such a new certificate into the NSS database, and NSS will automatically
perform a lookup of the embedded public key, be able to find the associated private key, and
subsequently be able to treat it as a personal certificate. (A personal certificate is a
certificate for which the private key is in possession, and which could be used for signing
data or for decrypting data.) A unique nickname can/should be assigned to the certificate at
the time of import, which can later be used to easily identify and retrieve it.
| It's important to note that NSS requires strict cleanup for all handles returned by NSS. The
application should always call the appropriate dereference (destroy) functions once a handle is
no longer needed. This is particularly important for applications that might need to close a
database and reinitialize NSS using a different one, without restarting. Such an operation
might fail at runtime if data elements are still being referenced.
| In addition to the FreeBL, Softoken, and CKBI modules, there is an utility library for general
operations (e.g., encoding/decoding between data formats, a list of standardized object
identifiers (OID)). NSS has an SSL/TLS module that implements the Secure Sockets
Layer/Transport Layer Security network protocols, an S/MIME module that implements CMS
messaging used by secure email and some instant messaging implementations, a DBM library that
implements the classic database storage, and finally a core NSS library for the big set of
“everything else”. Newer generations of the database use the SQLite database to allow
concurrent access by multiple applications.
| All of the above are provided as shared libraries. The CRMF library, which is used to produce
certain kinds of certificate requests, is available as a library for static linking only.
| When dealing with certificates (X.509), file formats such as PKCS#12 (certificates and keys),
PKCS#7 (signed data), and message formats as CMS, we should mention ASN.1, which is a syntax
for storing structured data in a very efficient (small sized) presentation. It was originally
developed for telecommunication systems at times where it was critical to minimize data as much
as possible (although it still makes sense to use that principle today for good performance).
In order to process data available in the ASN.1 format, the usual approach is to parse it and
transfer it to a presentation that requires more space but is easier to work with, such as
(nested) C data structures. Over the time NSS has received three different ASN.1 parser
implementations, each having their own specific properties, advantages and disadvantages, which
is why all of them are still being used (nobody has yet dared to replace the older with the
newer ones because of risks for side effects). When using the ASN.1 parser(s), a template
definition is passed to the parser, which will analyze the ASN.1 data stream accordingly. The
templates are usually closely aligned to definitions found in RFC documents.
| A data block described as DER is usually in ASN.1 format. You must know which data you are
expecting, and use the correct template for parsing, based on the context of your software's
interaction. Data described as PEM is a base64 encoded presentation of DER, usually wrapped
between human readable BEGIN/END lines. NSS prefers the binary presentation, but is often
capable to use base64 or ASCII presentations, especially when importing data from files. A
recent development adds support for loading external PEM files that contain private keys, in a
software library called nss-pem, which is separately available, but should eventually become a
core part of NSS.
| Looking at the code level, NSS deals with blocks of raw data all the time. The common structure
to store such an untyped block is SECItem, which contains a size and an untyped C pointer
variable.
| When dealing with memory, NSS makes use of arenas, which are an attempt to simplify management
with the limited offerings of C (because there are no destructors). The idea is to group
multiple memory allocations in order to simplify cleanup. Performing an operation often
involves allocating many individual data items, and the code might be required to abort a task
at many positions in the logic. An arena is requested once processing of a task starts, and all
memory allocations that are logically associated to that task are requested from the associated
arena. The implementation of arenas makes sure that all individual memory blocks are tracked.
Once a task is done, regardless whether it completed or was aborted, the programmer simply
needs to release the arena, and all individually allocated blocks will be released
automatically. Often freeing is combined with immediately erasing (zeroing, zfree) the memory
associated to the arena, in order to make it more difficult for attackers to extract keys from
a memory dump.
| NSS uses many C data structures. Often NSS has multiple implementations for the same or similar
concepts. For example, there are multiple presentations of certificates, and the NSS internals
(and sometimes even the application using NSS) might have to convert between them.
| Key responsibilites of NSS are verification of signatures and certificates. In order to verify
a digital signature, we have to look at the application data (e.g., a document that was
signed), the signature data block (the digital signature), and a public key (as found in a
certificate that is believed to be the signer, e.g., identified by metadata received together
with the signature). The signature is verified if it can be shown that the signature data block
must have been produced by the owner of the public key (because only that owner has the
associated private key).
| Verifying a certificate (A) requires some additional steps. First, you must identify the
potential signer (B) of a certificate (A). This is done by reading the “issuer name” attribute
of a certificate (A), and trying to find that issuer certificate (B) (by looking for a
certificate that uses that name as its “subject name”). Then you attempt to verify the
signature found in (A) using the public key found in (B). It might be necessary to try multiple
certificates (B1, B2, ...) each having the same subject name.
| After succeeding, it might be necessary to repeat this procedure recursively. The goal is to
eventually find a certificate B (or C or ...) that has an appropriate trust assigned (e.g.,
because it can be found in the CKBI module and the user hasn't made any overriding trust
decisions, or it can be found in a NSS database file managed by the user or by the local
environment).
| After having successfully verified the signatures in a (chain of) issuer certificate(s), we're
still not done with verifying the certificate A. In a PKI it's suggested/required to perform
additional checks. For example: Certificates were valid at the time the signature was made,
name in certificates matches the expected signer (check subject name, common name, email, based
on application), the trust restrictions recorded inside the certificate (extensions) permit the
use (e.g., encryption might be allowed, but not signing), and based on environment/application
policy it might be required to perform a revocation check (OCSP or CRL), that asks the
issuer(s) of the certificates whether there have been events that made it necessary to revoke
the trust (revoke the validity of the cert).
| Trust anchors contained in the CKBI module are usually self signed, which is defined as having
identical subject name and issuer name fields. If a self-signed certificate is marked as
explicitly trusted, NSS will skip checking the self-signature for validity.
| NSS has multiple APIs to perform verification of certificates. There is a classic engine that
is very stable and works fine in all simple scenarios, for example if all (B) candidate issuer
certificates have the same subject and issuer names and differ by validity period; however, it
works only in a limited amount of more advanced scenarios. Unfortunately, the world of
certificates has become more complex in the recent past. New Certificate Authorities enter the
global PKI market, and in order to get started with their business, they might make deals with
established CAs and receive so-called cross-signing-certificates. As a result, when searching
for a trust path from (A) to a trusted anchor (root) certificate (Z), the set of candidate
issuer certificates might have different issuer names (referring to the second or higher issuer
level). As a consequence, it will be necessary to try multiple different alternative routes
while searching for (Z), in a recursive manner. Only the newer verification engine (internally
named libPKIX) is capable of doing that properly.
| It's worth mentioning the Extended Validation (EV) principle, which is an effort by software
vendors and CAs to define a stricter set of rules for issuing certificates for web site
certificates. Instead of simply verifying that the requester of a certificate is in control of
an administrative email address at the desired web site's domain, it's required that the CA
performs a verification of real world identity documents (such as a company registration
document with the country's authority), and it's also required that a browser software performs
a revocation check with the CA, prior to granting validity to the certificate. In order to
distinguish an EV certificate, CAs will embed a policy OID in the certificate, and the browser
is expected to verify that a trust chain permits the end entity (EE) certificate to make use of
the policy. Only the APIs of the newer libPKIX engine are capable of performing a policy
verification.
| That's a good opportunity to talk about SSL/TLS connections to servers in general (not just EV,
not just websites). Whenever this document mentions SSL, it refers to either SSL or TLS. (TLS
is a newer version of SSL with enhanced features.)
| When establishing an SSL connection to a server, (at least) a server certificate (and its trust
chain) is exchanged from the server to the client (e.g., the browser), and the client verifies
that the certificate can be verified (including matching the name of the expected destination
server). Another part of the handshake between both parties is a key exchange. Because public
key encryption is more expensive (more calculations required) than symmetric encryption (where
both parties use the same key), a key agreement protocol will be executed, where the public and
private keys are used to proof and verify the exchanged initial information. Once the key
agreement is done, a symmetric encryption will be used (until a potential re-handshake on an
existing channel). The combination of the hash and encryption algorithms used for a SSL
connection is called a cipher suite.
| NSS ships with a set of cipher suites that it supports at a technical level. In addition, NSS
ships with a default policy that defines which cipher suites are enabled by default. An
application is able to modify the policy used at program runtime, by using function calls to
modify the set of enabled cipher suites.
| If a programmer wants to influence how NSS verifies certificates or how NSS verifies the data
presented in a SSL connection handshake, it is possible to register application-defined
callback functions which will be called by NSS at the appropriate point of time, and which can
be used to override the decisions made by NSS.
| If you would like to use NSS as a toolkit that implements SSL, remember that you must init NSS
first. But if you don't care about modifying the default trust permanently (recorded on disk),
you can use the no-database init calls. When creating the network socket for data exchange,
note that you must use the operating system independent APIs provided by NSPR and NSS. It might
be interesting to mention a property of the NSPR file descriptors, which are stacked in layers.
This means you can define multiple layers that are involved in data processing. A file
descriptor has a pointer to the first layer handling the data. That layer has a pointer to a
potential second layer, which might have another pointer to a third layer, etc. Each layer
defines its own functions for the open/close/read/write/poll/select (etc.) functions. When
using an SSL network connection, you'll already have two layers, the basic NSPR layer and an
SSL library layer. The Mozilla applications define a third layer where application specific
processing is performed. You can find more details in the NSPR reference documents.
| NSS occassionally has to create outbound network connections, in addition to the connections
requested by the application. Examples are retrieving OCSP (Online Certificate Status Protocol)
information or downloading a CRL (Certificate Revocation List). However, NSS doesn't have an
implementation to work with network proxies. If you must support proxies in your application,
you are able to register your own implementation of an http request callback interface, and NSS
can use your application code that supports proxies.
| When using hashing, encryption, and decryption functions, it is possible to stream data (as
opposed to operating on a large buffer). Create a context handle while providing all the
parameters required for the operation, then call an “update” function multiple times to pass
subsets of the input to NSS. The data will be processed and either returned directly or sent to
a callback function registered in the context. When done, you call a finalization function that
will flush out any pending data and free the resources.
| This line is a placeholder for future sections that should explain how libpkix works and is
designed.
| If you want to work with NSS, it's often helpful to use the command line utilities that are
provided by the NSS developers. There are tools for managing NSS databases, for dumping or
verifying certificates, for registering PKCS#11 modules with a database, for processing CMS
encrypted/signed messages, etc.
| For example, if you wanted to create your own pair of keys and request a new certificate from a
CA, you could use certutil to create an empty database, then use certutil to operate on your
database and create a certificate request (which involves creating the desired key pair) and
export it to a file, submit the request file to the CA, receive the file from the CA, and
import the certificate into your database. You should assign a good nickname to a certificate
when importing it, making it easier for you to refer to it later.
| It should be noted that the first database format that can be accessed simultaneously by
multiple applications is key4.db/cert9.db – database files with lower numbers will most likely
experience unrecoverable corruption if you access them with multiple applications at the same
time. In other words, if your browser or your server operates on an older NSS database format,
don't use the NSS tools to operate on it while the other software is executing. At the time of
writing NSS and the Mozilla applications still use the older database file format by default,
where each application has its own NSS database.
| If you require a copy of a certificate stored in an NSS database, including its private key,
you can use pk12util to export it to the PKCS#12 file format. If you require it in PEM format,
you could use the openssl pkcs12 command (that's not NSS) to convert the PKCS#12 file to PEM.
| This line is a placeholder for how to prepare a database, how to dump a cert, and how to
convert data.
| You might have been motivated to work with NSS because it is used by the Mozilla applications
such as Firefox, Thunderbird, etc. If you build the Mozilla application, it will automatically
build the NSS library, too. However, if you want to work with the NSS command line tools, you
will have to follow the standalone NSS build instructions, and build NSS outside of the Mozilla
application sources.
| The key database file will contain at least one symmetric key, which NSS will automatically
create on demand, and which will be used to protect your secret (private) keys. The symmetric
key can be protected with PBE by setting a master password on the database. As soon as you set
a master password, an attacker stealing your key database will no longer be able to get access
to your private key, unless the attacker would also succeed in stealing the master password.
| Now you might be interest in how to get the
:ref:`mozilla_projects_nss_nss_sources_building_testing`