What we think.

Reading time: 6 minutes

Knowing how and when to use logs in embedded software is one of the hardest things a developer has to do. From isolating a runtime error to capturing performance data, we spend a lot of time and money on debugging our code – simply because it’s hard to know where to look.

Whether it’s printf() statements or a sophisticated CAN bus logging framework, the effort spent on logging and log tools seems to be worth it. It’s much cheaper to fix an error before production, and much faster to resolve a field issue if the device logs point the way.

Now let’s add cybersecurity to the mix. Most automotive vulnerabilities can be traced back to software issues, amplifying the need for logging. Tesla and Mercedes have felt this pain, and the financial impact of a security breach can be just as significant as the reputational damage. Data regulation and other standards are also getting stricter, from the European General Data Protection Regulation (GDPR) to the ISO/SAE 21434 cybersecurity engineering standard.

Cybersecurity plus logging introduces a volatile combination. Is logging safe to use? Are the logs themselves secure?

Not only do logs tend to proliferate throughout code and have access to critical functions, their usage will only grow. More than 125 million passenger cars with embedded connectivity will be on the road by next year and researchers have quantified the rate of hacking attempts on computers with internet access (including connected cars!) at every 39 seconds. These numbers indicate that it’s incumbent for your business to understand and strengthen the software security environment from the bottom up.

The basics of automotive logging

Logs and loggers are tools that developers use to investigate and address unexpected behaviors in function or performance. They come in several flavors but the most common types for automotive embedded systems are:

• Trace statements – basic function calls that take input data and output it to a memory location or filestream, like printf() or cout.
• Open-source loggers – packages that offer varying degrees of logging features beyond outputting messages, such as timestamps, structured severity levels, retention windows, thread safety, and more. There are many examples out there but the Automotive Grade Linux logger provides a good overview of usage.
• Commercial loggers – sophisticated applications like Percepio or QNX System Profiler that tend to instrument runtime code with calls that feed information back to a development host for recording, monitoring, and visualization. While the user-facing software isn’t deployed to vehicles, it’s not uncommon for instrumented code to be released, either by design or by accident.

A good logging strategy includes sufficient coverage, meaningful messages, consistent severity levels and thresholds, and low overhead. Cybersecurity is almost always an afterthought, whether it’s the logger itself or the downstream log data that’s onboard the vehicle or transferred to the cloud.

One of the most common types of code vulnerabilities is the buffer overflow, where an attacker gains access to areas of memory outside the expected boundaries — especially relevant for loggers reading and writing to memory.

Consider this example (overly simplified for illustration):

This code is off by one: it will copy one more entry than is likely intended, meaning that additional log entries may be unintentionally exposed to other areas of the application. Worse, if the destination isn’t properly protected, the copied data will overwrite adjacent memory locations. If an attacker guesses or knows this behavior, they could exploit it to extract the information or cause unpredictable behavior.

Do insecure logs matter if a threat actor isn’t inside the car and plugged into the OBD-II port? The Nissan Leaf vulnerability, where a researcher gained access to battery status records and control of the climate system through unsecured API calls, demonstrates the potential exposures that come with the connected car. Security experts are already thinking ahead in their research:

The standards behind automotive cybersecurity

If the threat of lost revenue and reputation isn’t enough, governments and standard bodies are close to requiring OEMs have cybersecurity measures in place:

• NHTSA has the Cybersecurity Best Practices for Modern Vehicles report, which is currently undergoing revision.
• The ISO/SAE 21434 Road vehicles cybersecurity engineering standard covers cybersecurity risk management across the engineering lifecycle, from concept through production to decommissioning.
• In 2020, the United Nations adopted the UNECE WP.29 Cybersecurity and Cybersecurity Management Systems regulation to support the placement of cybersecurity processes in the automotive sector.
• The European Union has mandated compliance to the UNECE WP.29 regulation by July 2024 and other countries are following suit.

A push towards cybersecurity logging

To better prevent and remediate cybersecurity incidents, there is a growing desire among regulators to improve investigative tools. President Biden’s “Executive Order on Improving the Nation’s Cybersecurity” defines specific steps required by organizations to do business with the US government, including logs:

The common thread between these initiatives is best described by this quote from the NHTSA report: “it [is] important for the automotive industry to make vehicle cybersecurity an organizational priority. This includes proactively adopting and using available guidance such as this document and existing standards and best practices.”

Four steps towards better logging cybersecurity

Bringing a cybersecurity mindset to logging strategy boils down to one thing: Accepting that logs and logging tools are a threat source. To mitigate this threat, in security terms, is the concept of “Zero Trust” and the following principles.

Adopt a zero trust mindset

A zero trust strategy for logging requires not accepting any user, interaction, or software package without security validation in place. It’s too easy for leadership and teams to assume that their code is not compromised or that vehicles will never be exposed to malicious threat actors. Zero trust promotes the reality that cybersecurity challenges are growing with the connected car and that most breaches are caused by human error and cybercriminals taking advantage of people, whether developers or drivers.

Understand the threats

Knowing how a logger or log data may be compromised (the attack vectors) is the first step towards building a threat model of the overall system. Typically, security engineers start from the top of the system architecture and work down into the components, but effective cybersecurity should also start from the bottom. Educating developers and putting tools in place at the module level only helps overall security in the long run.

The OWASP top 10 list is a good place to start as it identifies the most common vulnerabilities and their remediation steps at the code level. All the OWASP items are relevant for different aspects of a logging workflow but here are three to consider for now:

1. A01:2021-Broken Access Control – the exposure of sensitive components or information due to users having access outside their intended permissions. An example would be the use of unverified data in an SQL call to extract data from a log database
2. A06:2021-Vulnerable and Outdated Components – the potential for unsupported or out-of-date components, like open source or commercial logging tools, to be vulnerable to attack
3. A09:2021-Security Logging and Monitoring Failures – the detection, escalation, and response to active breaches by implementing logs that are relevant, secure, and usable for audit paper trails

Embrace cybersecurity standards

The NIST special publication on zero trust architecture (ZTA) includes seven principles for zero trust that enforce a granular, component-based level of security rather than the traditional system-based perimeter. Interpreting these principles within the context of logging, either onboard the vehicle or within connected infrastructure, your organization should ensure:

1. All data sources and logging services are considered resources.
2. All logging communications are secured regardless of location.
3. Access to individual resources is granted on a per-session basis.
4. Access to resources is determined by dynamic policy that may include other behavioral and environmental attributes.
5. The integrity and security posture of all owned and associated assets (such as cloud-based log aggregation and databases) are monitored and measured.
6. All resource authentication and authorization are dynamic and strictly enforced before access is allowed.
7. The collection of data is done on as much information as possible about the current state of assets, network infrastructure, and communications and used it to improve its security posture.

Validate supply chain security

Like functional safety, your business should recognize the security implications of procured software and open source code (the Tesla drone hack offers an example). It’s often easier for hackers to find out about and exploit vulnerabilities in the software supply chain rather than attack an OEM directly and they’ll continue to do so as more vehicles rely on more software.

A supply chain security strategy focuses on understanding the source of code and software packages and ensuring awareness of vulnerabilities and security patches. Typically, this includes:

• Policies around supply chain security that are communicated to all vendors and internal development teams
• Tools and processes to validate new software and audit existing packages
• Tracking and maintaining a database of all software packages and versions in use by the organization

Conclusion

It’s much cheaper to plug a security hole before production, and much faster to remediate an exposure if logs point the way. As the connected car industry grows, loggers and the data they collect offer unprecedented opportunities for threat actors to exploit. From leveraging insecure code to extracting sensitive log data from unprotected systems, it’s incumbent on OEMs to apply the same level of discipline to automotive logging cybersecurity as they do functional safety.

---