Datagram based protocols can be susceptible to Denial of Service attacks. A
DTLS attacker could, for example, submit a series of handshake initiation
requests that cause the server to allocate state (and possibly perform
cryptographic operations) thus consuming server resources. The attacker could
also (with UDP) quite simply forge the source IP address in such an attack.
As a counter measure to that DTLS includes a stateless cookie mechanism. The
idea is that when a client attempts to connect to a server it sends a
ClientHello message. The server responds with a HelloVerifyRequest which
contains a unique cookie. The client then resends the ClientHello, but this time
includes the cookie in the message thus proving that the client is capable of
receiving messages sent to that address. All of this can be done by the server
without allocating any state, and thus without consuming expensive resources.
OpenSSL implements this capability via the DTLSv1_listen() function. The ssl
parameter should be a newly allocated SSL object with its read and write BIOs
set, in the same way as might be done for a call to SSL_accept(). Typically the
read BIO will be in an unconnected state and thus capable of receiving
messages from any peer.
When a ClientHello is received that contains a cookie that has been verified,
then DTLSv1_listen() will return with the ssl parameter updated into a state
where the handshake can be continued by a call to (for example) SSL_accept().
Additionally the BIO_ADDR pointed to by peer will be filled in with
details of the peer that sent the ClientHello. If the underlying BIO is unable
to obtain the BIO_ADDR of the peer (for example because the BIO does not
support this), then *peer will be cleared and the family set to AF_UNSPEC.
Typically user code is expected to connect the underlying socket to the peer
and continue the handshake in a connected state.
Prior to calling DTLSv1_listen() user code must ensure that cookie generation
and verification callbacks have been set up using
SSL_CTX_set_cookie_generate_cb() and SSL_CTX_set_cookie_verify_cb()
Since DTLSv1_listen() operates entirely statelessly whilst processing incoming
ClientHellos it is unable to process fragmented messages (since this would
require the allocation of state). An implication of this is that DTLSv1_listen()
only supports ClientHellos that fit inside a single datagram.
From OpenSSL 1.1.0 a return value of >= 1 indicates success. In this instance
the peer value will be filled in and the ssl object set up ready to
continue the handshake.
A return value of 0 indicates a non-fatal error. This could (for
example) be because of non-blocking IO, or some invalid message having been
received from a peer. Errors may be placed on the OpenSSL error queue with
further information if appropriate. Typically user code is expected to retry the
call to DTLSv1_listen() in the event of a non-fatal error. Any old errors on the
error queue will be cleared in the subsequent call.
A return value of <0 indicates a fatal error. This could (for example) be
because of a failure to allocate sufficient memory for the operation.
Prior to OpenSSL 1.1.0 fatal and non-fatal errors both produce return codes
<= 0 (in typical implementations user code treats all errors as non-fatal),
whilst return codes >0 indicate success.