Why encryption causes DDoS defence headaches

Encryption is one of the best methods to protect security and privacy online.

It enables individuals to ensure their privacy when online when making mobile calls or using instant messaging and it enables their personal information to be stored securely.

Encryption enables people to exchange data confidentially and even authenticate who or what we are exchanging data with.

Encryption has helped users 'trust' the connected world, as it has infiltrated many aspects of peoples' everyday lives.

The problem is that encryption is not a solution to all security challenges as it is used in a lot of ransomware.

Some forms of encryption technology, like the type used in the latest version of Transport Layer Security (TLS 1.3), can make identifying and blocking some threats more difficult.

Many network-based threat and fraud detection solutions have historically relied upon transparent, passive decryption of encrypted sessions via access to the server private key(s).

With the introduction of TLS 1.3, this is not as simple, as all the additional information needed to decrypt a session cannot be sniffed from the line.

TLS 1.3 dictates that Perfect Forward Secrecy (PFS) must be used, enhancing the confidentiality of communications but forcing a rethinking of the mechanisms for dealing with another set of problems.

One area which does need to be reconsidered is the mechanism for detecting and mitigating some forms of DDoS attack.

The latest Netscout Worldwide Infrastructure Security Report (WISR) confirms attacks targeting encrypted web services have become increasingly common in recent years.

Specifically, in 2017, 53% of enterprise, government and education (EGE) organisations detected attacks on encrypted services at the application layer.

Application layer attacks use traffic that is very difficult to distinguish from genuine user traffic, often requiring analysis of the actual application layer transaction to identify the patterns of activity involved in an attack.

The approach to this process must change as TLS 1.3 is adopted.

The sharing of keys

One approach is to use a Content Delivery Network (CDN) service, as these types of service can be effective against application layer attacks.

Where encrypted services are being protected, this can mean the service owner handing over or generating private keys for use by the third-party provider.

Whether this occurs or not, the CDN provider will terminate and decrypt customer communication within their environment for inspection.

This can allow them to mitigate application-layer DDoS attacks, but there are other risks around confidentiality. Sometimes these risks are acceptable to end-customers and service owners, and sometimes not, leading to the second option of using an on-network reverse-proxy to do the job.

Using an organisation's own reverse-proxies is common for load-balancing, as they inherently allow traffic to be inspected.

In an ideal world, the proxy would provide telemetry to a DDoS protection solution so that attacking hosts could be identified and blocked, preventing resources being consumed on the proxy, as proxies are susceptible to state-exhaustion DDoS attacks.

State-exhaustion attacks target the ability of the proxy to manage sessions and are very common.

This problem can be overcome by front-ending the reverse proxy with a DDoS protection solution that can identify and block both state-exhaustion attacks and those that target TLS negotiation.

However, there is a third option: transparent, passive decryption.

Passive decryption is still possible with TLS 1.3 when using ephemeral Diffie-Helman ciphers (as used in TLS 1.3), but only if static keys are re-used across sessions, shared with on-network security solutions (using a key management platform) and then periodically cycled.

This mechanism allows transparent decryption of traffic, for threat identification and blocking, in a similar manner to existing pre-TLS 1.3 mechanisms.

As with all things in security, different solutions will appeal to different organisations based on their needs, those of their customers and prevailing regulatory requirements.

However, with application layer DDoS attacks becoming ever more prevalent, an appropriate solution must be put in place.

Encryption is essential and PFS undoubtedly improves the overall security of the interactions with the connected world, but overcoming its impact is essential to other elements of the defensive stack.

This requires organisations to work across the IT, network and security teams within their organisations, to ensure they adopt the most appropriate approach for their business.