MACsec, IPsec, or optical layer encryption? A comparison for carrier and enterprise networks

All three layers encrypt with the same algorithm, AES-256. They differ in what exactly they hide, how much throughput they take, and what latency they introduce. The spread is wider than the apparent similarity suggests, because it ranges from nanoseconds and zero overhead in the optical layer to hundreds of microseconds and a clear loss of throughput with IPsec.
The choice is not a matter of preference or of a level of security. It is a consequence of which route you are protecting, what your latency requirements are, and where the trust boundary runs in your network. Below is a comparison of the three layers together with three deployment scenarios and an assessment of the post-quantum readiness of each.
The scope of protection on a single route
IPsec protects the whole route, but the data is in the clear inside both edge routers. MACsec protects each hop separately, so at every node the frame is decrypted and encrypted again. Layer one encryption covers only the optical path, but with no overhead and without exposing metadata.
Each layer protects a different section of the route
IPsec protects the whole route, but the data is in the clear inside both edge routers. MACsec protects each hop separately, so at every node the frame is decrypted and encrypted again. Layer one covers only the optical path, but with no overhead and without exposing metadata.
Three layers, three different scopes of protection
Layer one: optical encryption, OTNsec
Physical-layer encryption operates at the level of the OTN frame, before the signal is sent into the fibre. The frame payload is encrypted one to one, with no bytes added. The authentication data fits into reserved fields of the OTN header, and the key exchange uses an auxiliary channel that is part of the frame anyway. The consequence is full protocol transparency. Everything that passes through the path is encrypted, regardless of whether it is Ethernet, Fibre Channel, OTN, or something else. To an observer tapping the fibre, neither addresses nor traffic patterns are visible. In industry materials this is referred to as black fibre. The limitation is just as clear, because the solution requires dedicated hardware on both sides of the link and works point to point. It cannot be stretched across a route with another vendor's device in the middle.
Layer two: MACsec
MACsec is the IEEE 802.1AE standard, encrypting Ethernet frames between adjacent devices. The overhead is about 32 bytes per frame, made up of the SecTAG header and the ICV integrity check value. Encryption is done in hardware, hence the low latency. The scope of protection is a single link section, that is a hop. When data passes through several layer-two devices, each of them decrypts the frame and encrypts it again, so inside every device along the way the data exists in the clear. The MAC headers remain visible, because they are needed for frame switching.
Layer three: IPsec
IPsec protects IP packets end to end. Intermediate devices have no way to decrypt them, which makes it the foundation of VPN solutions and the only sensible option when traffic passes through infrastructure you do not control. The price is complexity and performance. Processing is often done in software, the header overhead is markedly larger than with MACsec, and the latency rises by one or two orders of magnitude. In tunnel mode the outer IP headers remain visible.
The comparison
The encryption algorithm is the same in all three cases, usually AES-256. The difference in security therefore does not stem from the strength of the cipher, but from how large a portion of the communication a given layer covers and over what section.
| Criterion | Layer 1OTNsec | Layer 2MACsec | Layer 3IPsec |
|---|---|---|---|
| Per-frame overhead | zero, 1:1 mapping | about 32 bytes | tens of bytes, depending on the mode |
| Effective throughput | full | a small loss | a significant loss, dependent on packet size |
| Latency | on the order of nanoseconds | single microseconds | hundreds of microseconds |
| Scope of protection | the whole payload including metadata | Ethernet frame, MAC headers visible | IP packet, outer headers visible |
| Reach of protection | point-to-point in the optical path | hop-by-hop | end to end |
| Protocol transparency | full, any protocol | Ethernet only | IP only |
| Hardware requirements | dedicated hardware on both sides | a switch or router with 802.1AE | broad, including software |
| Operation over foreign infrastructure | no | limited | yes |
| Key management complexity | low, auxiliary channel in the frame | low, MKA | high, IKEv2 and policies |
Two rows deserve a comment. The scope of metadata protection tends to be overlooked in comparisons, yet in risk analysis it often weighs more than the confidentiality of the content itself. Layer one encryption also hides traffic patterns, because an observer does not know who is communicating with whom or when the intensity rises. MACsec and IPsec leave the MAC addresses and the outer IP addresses respectively, from which the topology and activity can be reconstructed. The reach of protection decides where data exists in the clear. With MACsec every intermediate layer-two device decrypts the frame and encrypts it again, so if the threat model includes those devices, MACsec is not the answer.
Three deployment scenarios
An ISP with a backbone network
Links of 100G and above, routes through areas without full physical control. Every percent of throughput is commercial capacity, the traffic of many customers in different protocols, the threat being physical access to the fibre.
OTNsec on the backbone, MACsec at the interfaces with business customers
Synchronous replication
Two sites, distance a dozen to a few dozen km, leased dark fibre. Every microsecond lengthens the write-acknowledgement time, the auditor asks whether the data is in the clear anywhere between the sites.
OTNsec on the route, IPsec as a supplement for selected applications
A municipal network
The office, schools, subordinate units, monitoring. Mixed traffic, distances of a few to a dozen km, a limited budget, infrastructure often owned, traffic entirely Ethernet.
the base layer, OTNsec on selected critical sections
Scenario one: a network operator
An ISP with its own backbone network, links with a throughput of 100G and above, routes running through areas over which the operator does not have full physical control. The recommendation is layer one. The justification has three pillars. First, at a throughput of 100G and above, every percent of lost throughput translates directly into the commercial capacity of the link, so zero overhead stops being a technical detail and becomes a revenue item. Second, the operator carries the traffic of many customers, often in different protocols, and protocol-transparent encryption handles everything without distinction, whereas MACsec would require all the traffic to be Ethernet. Third, the main threat vector here is physical access to the fibre outside the nodes. The methods of tapping include both inserting a splitter and intercepting the signal through an appropriately bent fibre, where a fraction of a percent of the optical power is enough to read the transmission. Layer one encryption protects exactly that section. MACsec remains a sensible supplement at the interfaces with business customers, where the traffic is Ethernet and the requirements concern a single access link.
Scenario two: a bank and two data centres
A financial institution with two processing sites, synchronous replication between them, a distance of a dozen to a few dozen kilometres, leased dark fibre. The recommendation is layer one, with IPsec as a supplementary layer for selected applications. Synchronous replication is the deciding factor here. Every microsecond of latency on the route lengthens the write-acknowledgement time, and with synchronous replication that time translates directly into the performance of the transaction systems. Latency on the order of hundreds of microseconds, characteristic of IPsec, is unacceptable in this scenario as the sole protection for the route. The second factor is throughput, because replicating production environments generates traffic for which a throughput loss on the order of a dozen or so percent means having to buy additional capacity. The third is the regulatory requirement. Requirements for protecting data in transit are usually met by AES-256 encryption regardless of the layer, but auditors also ask whether the data exists anywhere in the clear between the sites. With layer one encryption the answer is no, without reservation. IPsec remains useful for individual applications with heightened requirements, where end-to-end protection must also cover the interior of both sites. It is a supplement, not an alternative.
Scenario three: a local-government unit
A municipal network connecting the office, schools, subordinate units, and monitoring. Mixed traffic, distances of a few to a dozen kilometres, a limited budget, infrastructure often owned. The recommendation is MACsec as the base layer, layer one on selected critical sections. It is the only one of the three scenarios in which MACsec comes out best, and it is worth understanding why. The entry cost is markedly lower, because MACsec is supported by many access and aggregation switches as standard, without the need to buy dedicated encryption devices. In a network with a few dozen nodes the difference in capital outlay is significant. The traffic is entirely Ethernet, so limiting MACsec to one protocol is not a problem. Latency requirements are lenient, because a few microseconds on a section do not matter for line-of-business systems, mail, or video monitoring. There remains the question of sections where traffic passes through areas outside the unit's control, for example a route leased from an operator or running across private land. There it is worth considering layer one encryption on that specific section, not across the whole network. A separate note concerns compliance. The requirements arising from the data-protection regulations and from the national cybersecurity system speak of adequate technical measures, without indicating a specific layer. MACsec deployed consistently on all links meets these requirements, provided a risk analysis is documented that justifies the choice.
Post-quantum readiness
This question comes up today in most procurement proceedings, and it is worth answering precisely, because a fair amount of misunderstanding circulates around it. The symmetric cipher itself is not the problem. AES-256, used in all three layers, is regarded as resistant to known quantum algorithms. Grover's algorithm halves the effective key strength, which at 256 bits still leaves a security level corresponding to 128 bits, and that is a sufficient margin. The problem is key exchange. Mechanisms based on asymmetric cryptography, such as Diffie-Hellman or RSA, are vulnerable to Shor's algorithm. It is they, not AES itself, that constitute the weak point from the perspective of quantum cryptography. This concerns both IKEv2 in IPsec and the MKA protocol in MACsec.
Has the most advanced standardisation work on hybrid key exchange, combining a classical mechanism with a post-quantum one. Commercial solutions are already available.
The subject of active research into post-quantum key agreement. Publications show working prototypes, but production availability remains limited.
Key exchange via the auxiliary channel in the frame opens the way to feeding a key from an external source, which can be QKD. Interception of the key is detectable from the laws of physics, not from computational assumptions.
The harvest-now-decrypt-later scenario is why the matter is urgent already today. An adversary can intercept and archive encrypted traffic now, to decrypt it in ten or fifteen years. For data with a long sensitivity window, such as medical records, personal data, or trade secrets, the risk is real now. Layer one has a special property here, because the key exchange takes place via an auxiliary channel in the frame, which opens the way to feeding it a key from an external source. That source can be quantum key distribution, that is QKD. The key is then generated by a separate device using a quantum stream, and its interception by an attacker is detectable from the laws of physics, not from computational assumptions.
Determine which routes carry data with a sensitivity window longer than ten years, because those are the target of the harvest-now-decrypt-later scenario.
Cover those routes with layer one encryption, which costs neither throughput nor latency, and also hides metadata.
Introduce a key resistant to quantum attack, for example from QKD, fed via the auxiliary channel in the frame.
How to test before deployment
Manufacturers' declarations about latency and throughput are measured in laboratory conditions. In your network the results may be different, so a pilot on a single link before a decision on scale is an investment that pays back with the first problem detected.
The pilot is worth running for at least two weeks, to cover a full weekly traffic cycle including the weekend and periods of increased load.
What happens to the traffic when the key agreement process is interrupted. Whether the link rebuilds itself and after how long.
Whether encryption is preserved after the switchover and how long the session takes to rebuild.
Whether the configuration and key material survive a restart, or whether intervention is required.
Whether the state of the encrypted session is reported to monitoring and whether a failure generates an alarm, or only a silent degradation.
How quickly encryption can be disabled if the pilot turns out badly, and whether it requires a maintenance window.
A summary in three sentences
If you are protecting a route between two locations that you control, and you care about throughput or latency, choose layer one. If you are building an Ethernet network with many nodes on a limited budget, choose MACsec. If the traffic passes through infrastructure you do not control, you have to use IPsec, regardless of the performance cost. In most real deployments these layers complement rather than exclude one another. Layer one encryption on the backbone routes plus IPsec for selected applications is the arrangement we see most often with customers that have regulatory requirements.
Sources:
Microchip, In-flight Encryption in Service Provider Networks, a comparison of OTN, MACsec, and IPsec overhead, Cho, Sergeev, Post-quantum MACsec in Ethernet Networks, 32-byte overhead, 1 to 3 µs latency, throughput measurements, Automatic Intent-Based Secure Service Creation, arXiv, properties of physical-layer encryption, MACsec and IPsec, Chaotic Encryption for 10-Gb Ethernet Optical Links, arXiv, the effect of IPsec on throughput, Multi-Layer Encryption in Optical Fiber Networks, a comparison of the latencies and scope of protection of the three layers
FAQ:
FAQ, encryption in the network layers
Rozpocznij z nami nowy projekt!

How to Deploy a Coherent QSFP-DD Module in an Existing DWDM Network?

Your fiber can carry 40 times more data. You probably haven't been told that.
