Most platform evaluations reward feature lists and vendor-data demos, then fail on variants, re-imports and audit trails in production. Score candidates instead on the questions your stack cannot answer today, proven on your own files, weighted for your role as OEM or Tier-1, and gated by deployment and identity requirements.
Automotive cybersecurity platform evaluation has become a recurring task for OEM and Tier-1 teams, and many run it badly for an understandable reason: the request for proposal is a feature list, the demo runs on vendor sample data, and the score sheet rewards the longest column of checkmarks. Months later the team finds that the TARA cannot say which variants a risk applies to, or that the audit trail stops at "record modified".
This checklist is for the people who make the call: CSMS owners, PSIRT and VSOC leads, engineering managers, and the IT security and procurement staff who review deployment and identity requirements. Each of seven areas has the same shape: questions, demo proof, red flags, and how an OEM and a Tier-1 should weight it. It names no vendors and ranks nothing.
Start from the questions your current stack cannot answer
List five to ten questions the organization answers slowly or not at all: which programs and variants carry a given component version; which controls treat a risk and which tests verified them on the shipped baseline; which markets have vehicles carrying a part number an active CVE reaches. Those questions become the demo script, and a platform that answers them from connected records on your own files is doing something a spreadsheet cannot.
Keep generic tooling in its place: ticketing, the GRC platform and the SIEM stay in their roles, and the gap they leave is described in why generic GRC and ITSM platforms lose automotive cybersecurity context. The evaluation is about the product model as linked, versioned records.
Weighting differs by role. An OEM buys mainly for the program layer: many projects and suppliers, variants, fleet exposure and incident command. A Tier-1 buys for TARA depth and for evidence an OEM can consume without re-typing it or buying seats to read it.
Score the platform on the questions you cannot answer today, not on the features you already have.
Architecture modeling: the system as engineered, not as drawn
Questions: can the platform import the architecture from the engineering source of record, in UML/XMI and AUTOSAR ARXML, rather than requiring a redraw? Does it carry component types, connection properties such as protocol and encryption, and trust boundaries? Can each component of a vehicle architecture link to its own security project, so component analysis rolls up into the program? Can an ECU hold software units, each with its own architecture and SBOM?
Proof to request: import a real model. A sound import shows a conformance summary before saving, lists quality warnings such as inheritance cycles or ports whose interface cannot be found, and records undrawn elements in a downgrade list so nothing is lost silently. For production projects, ask for a staged import approved by a second person, each step in the audit trail.
Red flags: no warnings on a messy model; components that are drawings without properties; a program that is a folder of projects rather than the vehicle architecture described in programs and vehicle architecture.
Weighting: OEM high, because the program layer and variants depend on it; Tier-1 medium, rising as customers ask for architecture-level evidence per deliverable.
TARA: generated risks, governed scoring and traceable treatment
Questions: are risks generated from a threat scenario linked to an attack path, one per asset and cybersecurity property, or typed by hand? Is impact taken from the linked damage scenarios across safety, financial, operational and privacy ratings, likelihood from attack feasibility, and a Cybersecurity Assurance Level from impact and attack vector? Does a policy define the risk method, accepted levels and permitted treatment strategies, and refuse what it does not permit? Does treatment record strategy (reduce, retain, share or avoid), justification, assignee and residual risk, linking reduced risks to goals, requirements and controls and retained risks to claims with evidence? The method is covered in the ISO/SAE 21434 TARA guide.
Proof to request: import one of your own TARAs, from a spreadsheet or OpenXSAM, and follow one risk from asset to test case. Delete the asset; the risk should be flagged with a removed reference, not vanish. Try to retain a high risk under a policy that accepts only medium and below. Where AI recommends treatments, ask how a suggestion is accepted or rejected and whether AI-enriched records carry a marker a reviewer must acknowledge.
Red flags: editable risk scores; a compliance panel that is a filled template; no residual risk; a TARA that cannot be versioned per baseline, the problem TARA version control addresses.
Weighting: Tier-1 high, because TARA depth is the deliverable; OEM high, with the emphasis on roll-up across, say, 14 component projects.
SBOM and CVE impact analysis: from component to part number
Questions: which formats (CycloneDX and SPDX at minimum) and which re-import options (merge, replace, update only), so a supplier's re-delivery is diffed rather than overwritten? Which vulnerability sources are matched, including regional feeds such as CNVD where GB 44495 applies, and does policy control which are trusted? Is there a status workflow with accepted-with-rationale and suppressed-as-false-positive, so a "not affected" statement is a record rather than an email? Above all, are architecture modules mapped to SBOM components, automatically on package URL or unique name and held for review when ambiguous, and can a vulnerability be traced from software component to software unit, system component, part number, vehicle and market?
Proof to request: bring your own SBOM for a software unit whose architecture you can also export. Read the three counts after matching: matched, unmatched components, unmatched modules. Pick a component with a known vulnerability, ask for its blast radius through the architecture, then for the markets whose vehicles carry a part number it reaches. If the chain is incomplete, the platform should say so, as automotive CVE blast radius across vehicle variants explains.
Red flags: vulnerability counts without a status workflow; zero ambiguities on a real SBOM; "supplier management" that is a contact list. On formats, see CycloneDX vs. SPDX for automotive.
Weighting: OEM high for the part-number and fleet chain; Tier-1 high for per-deliverable hygiene and defensible not-affected statements.
Incident handling: from fleet signal to realized risk
An incident must link to the affected components, the risks that modeled the attack, the vulnerable software and the controls meant to detect it, and feed back into the TARA. Questions: which sources open an incident (manual report, automated monitoring, threat intelligence, vulnerability scan, external party)? When a confirmed incident shows a modeled attack happening, is the risk marked realized? When the attack was never modeled, is a TARA gap raised for engineering to fill? Are response times per severity set in policy and enforced?
Proof to request: open an incident from a threat intelligence alert, link it to a component and a risk, add an update and an attachment, then check the risk's status. Ask how the program owner sees it: open incidents by market, beside vulnerabilities and risks in one view, with a traceability map anchored on the incident. Ask for webhook notifications and for metrics beyond mean time to resolve (incident response metrics beyond MTTR).
Red flags: incidents that cannot link to risks or components; no path back to the TARA; SLA fields with no enforcement. The closure evidence an assessor expects is in the incident closure evidence checklist.
Weighting: OEM very high, since post-production monitoring and fleet-wide response are the OEM's UN R155 obligation and the EU CRA adds reporting duties; Tier-1 medium, with emphasis on escalation to and from the OEM.
Evidence and traceability: baselines, approvals and the audit trail
Everything above holds only if the platform can reproduce the records as they stood when a decision was made. Questions: are there project baselines, with a released baseline read-only and branchable into a new draft? Does a design freeze protect the TARA while the assurance chain continues? Are variants modeled so shared entities cannot be changed from inside one variant? Is a program-level traceability map anchored on one finding shareable as a link that restores the exact view? Are policies versioned automatically, with a weakening change waiting for a second approver? Can a claim carry uploaded evidence with a status such as pending evidence or validated?
Proof to request: release a baseline, try to edit a risk inside it, then branch a draft and make the change there. Ask for the audit trail of a staged import and of a policy change. Ask for every record touching one risk, then the same for a vulnerability at program level. The chain is described in automotive cybersecurity traceability: a practical guide and the assessor's view in CSMS audit preparation.
Red flags: "versioning" that means exporting a PDF; an audit log without field, actor and reason; policies that are documents rather than enforced configuration.
Weighting: high for both. An OEM needs it for type approval under UN R155 and R156; a Tier-1 because every handover is a baseline and customers audit against ISO/SAE 21434 and increasingly the CRA.
Integrations: staged imports, open formats and status that flows back
Evaluate integrations by category and direction, not logo count. Across the pipeline: communication matrix inputs (ARXML, DBC, VSS); system modeling (UML/XMI, SysML, AUTOSAR); asset identification from diagnostic catalogs (ODX, CDD); TARA import (spreadsheets, OpenXSAM); requirements exchange (ReqIF); software architecture and SBOM (CycloneDX, SPDX); code and binary analysis findings (SARIF); vulnerability databases; bench testing returned as verification evidence; data exchange (STIX, PDF, CSV, Excel); the VSOC and threat intelligence (event streaming, STIX ingest); and identity (SAML 2.0, LDAP). Connecting the SIEM and VSOC to engineering tools covers operations.
Questions: is every import staged and reviewed before it lands on the graph? When a treatment creates a task in the ticketing system, does status flow back? Are identifiers preserved across round trips: CWE in SARIF, package URL in SBOMs, requirement IDs in ReqIF? Is the API included in your tier? Proof to request: one round trip on your files, architecture in, requirements out as ReqIF, SBOM in, a finding in via SARIF, a task out and its status back. Red flags: one-directional exports, connector lists without format versions, imports that cannot be previewed.
Weighting: OEM high for VSOC, threat intelligence and ticketing round trips; Tier-1 high for requirements-tool interchange and OEM-readable exports.
Deployment and security requirements for OEMs and Tier-1s
Deployment is often decided after the engineering evaluation, which is backwards: a platform that cannot run where your data is allowed to live is not a candidate. A hosted subscription suits a department whose data classification permits it; private cloud uses the organization's own tenancy; on-premise and air-gapped deployments are the norm for classified vehicle data and markets with residency rules. For an air-gapped instance the question above all others is how vulnerability feeds arrive: the answer must be an offline feed sync with a documented cadence.
| Requirement | Why it matters | Question to ask | Proof to request |
|---|---|---|---|
| SSO / SAML | Joiners and leavers handled once by the identity provider | Which protocols (SAML 2.0, LDAP), at which tier? | Login via your identity provider; a removed user loses access |
| RBAC | Suppliers, reviewers and approvers see different things | Permissions per module and action (read, write, treat, approve, export)? | A supplier account that sees shared evidence only |
| MFA | Approvals need more than a password | Enforceable for all users, stronger factor for production changes? | A production import commit demanding a fresh passkey |
| Audit trail | Assessors ask who changed what, when and why | Actor, field, timestamp and reason on every change, including policy and imports? | Exported trail of a staged import and a policy edit |
| Data residency | Vehicle and supplier data may not leave a jurisdiction | Where is data stored per model; in-region hosting for markets such as China? | Data-flow description per model; region options in the contract |
| Air-gap feed sync | Matching is only as current as the feeds | How are NVD, OSV and regional feeds delivered offline, how often, with what integrity check? | Offline feed import on a disconnected instance, feed date visible |
| API access | Automation needs a stable interface | Documented, versioned, in the tier, respecting RBAC? | A read of your own project via the API with a scoped token |
An OEM should treat SSO, RBAC, audit trail and residency as pass or fail. A Tier-1 should add one requirement the table omits: sharing evidence with a customer who holds no license in the supplier's instance.
The printable automotive cybersecurity platform evaluation checklist
The table consolidates the seven areas into scorable rows. Weights are high (H), medium (M) or pass/fail (P), for an OEM and then a Tier-1.
| Area | Question | Evidence to request | Weight (OEM / Tier-1) |
|---|---|---|---|
| Architecture | Import with conformance summary, warnings, downgrade list and staged approval; components link to their own projects? | Import of your model; audit trail of analyze, approve, commit; program-to-project navigation | H / M |
| TARA | Risks generated from threat scenario and attack path; impact, likelihood and CAL derived? | One risk followed asset to control; asset deleted, flag shown | H / H |
| TARA | Policy sets accepted levels and strategies and refuses violations; treatment records residual risk and the chain? | Retain a high risk under a strict policy; treatment wizard; assurance graph | H / H |
| SBOM | CycloneDX and SPDX import; merge, replace, update-only; status workflow? | Re-import of a changed supplier SBOM, diff shown | M / H |
| SBOM | Modules matched to components, ambiguous held for review; CVE traced to part number and market? | Matched and unmatched counts; a CVE followed to markets, or an honest "chain incomplete" | H / H |
| Incidents | Links to components, risks and software; confirmed attack marks risk realized; unmodeled attack raises a TARA gap; SLAs enforced? | Incident from a threat intelligence alert through to risk status; webhook to a test endpoint | H / M |
| Evidence | Released baselines read-only and branchable; design freeze; variant protection; map shareable by link? | Release a baseline, attempt an edit, branch a draft; reopen a shared map link | H / H |
| Governance | Policies auto-versioned; weakening change needs a second approver; ReqIF export? | Policy edit with pending-approval banner; the ReqIF file | H / H |
| Integrations | Round trip: ReqIF out, SARIF in, ticket out, status back, identifiers preserved? | The round trip on your files | H / H |
| Deployment | Identity, RBAC, MFA, audit, residency, air-gap feeds, API per the table above? | Each proof in the deployment table | P / P |
Three rules keep scoring honest. Send your files a week ahead and insist the demo runs on them; a demo on vendor data scores at most one point on any row. Score each row from zero to three: not shown, shown on vendor data, shown on your data, shown on your data with the audit trail. Finish with a time-boxed pilot on one program and one baseline. The supplier cybersecurity questionnaire applies the same discipline to the other side of the handover.
How ThreatZ answers this checklist
ThreatZ is built around the connected record this checklist keeps asking for. Its architecture model imports UML/XMI and AUTOSAR and links each vehicle component to its own security project inside a program. Risks are generated from threat scenarios and attack paths, scored from damage scenarios and attack feasibility, given a CAL from impact and attack vector, and governed by a policy that refuses what it does not allow. Treatment links each risk to goals, requirements and controls or to a claim with evidence, with an ISO/SAE 21434 compliance panel flagging gaps. Software Composition matches architecture modules to SBOM components and traces a vulnerability through software unit, system component and part number to VINs and markets on the Traceability Map and Fleet and Risk Map. Incidents mark a modeled risk realized when confirmed and raise a TARA gap when the attack was never modeled. This supports the evidence for ISO/SAE 21434, UN R155 and R156, EU CRA and GB 44495 work; it does not replace the assessment.
The Team and Professional tiers run as a hosted subscription on AWS Marketplace, with MFA and RBAC from Professional upward. Enterprise adds the program layer, fleet map and incident command, the supplier workspace, SSO/SAML and API access, and the deployment options in the table: Private cloud or on-premise — air-gapped supported, with offline vulnerability-feed sync and in-region hosting for GB 44495 work. Licensing is per named user, per tier, billed annually; supplier-share guests who only view shared evidence are free. Every evaluation can begin with a four-week free trial on sample vehicle data.
Frequently asked questions
How long should an automotive cybersecurity platform evaluation take?
Plan in stages rather than by calendar: one to two weeks to agree the unanswered questions and collect files, one scripted demo per candidate on those files, and a time-boxed pilot on one program and baseline for the shortlisted platform. A four-week trial fits the pilot; evaluations that run longer usually lack a script, not time.
Should an OEM and its Tier-1 suppliers use the same platform?
Not necessarily; the checklist lets both score independently. What matters at the interface is that evidence crosses it in open formats, ReqIF for requirements, CycloneDX or SPDX for SBOMs, OpenXSAM for TARA content, and that a supplier can share records with a customer without the customer buying seats in the supplier's instance.
Is a hosted deployment acceptable for vehicle cybersecurity data?
It depends on data classification and jurisdiction, which is why the deployment table is a gate rather than a scored row. Many departments run a hosted subscription with SSO, MFA and RBAC. Programs with classified vehicle data or strict residency rules typically require private cloud, on-premise or air-gapped operation, with offline feed synchronization.
How should AI features be evaluated in a TARA platform?
Treat AI as a reviewer-assisted drafting aid and test the controls around it: can a recommendation be accepted or rejected explicitly, do the resulting records carry a visible marker until a person acknowledges them, and do risk scores remain derived from the threat model and the policy? A suggestion that lands silently in the assurance chain is a governance problem.
What is the single most revealing demo request?
Delete an asset in the threat model and watch what happens to its risk, then re-import a changed supplier SBOM and watch what happens to the previous one. Both tests show whether the platform keeps relationships as managed data. A flagged removed reference and a diffed re-import pass; a vanished risk and a silent overwrite fail.
- Script before you shop: write the questions your stack cannot answer and run every demo against them on your own files.
- Weight by role: OEMs weight the program layer, fleet exposure and incident command; Tier-1s weight TARA depth, per-deliverable SBOMs and inexpensive handover.
- Imports must be staged: conformance summaries, warnings, downgrade lists and second-person approval separate a managed import from a paste.
- Risks are generated, not typed: from threat scenario and attack path, scored from damage scenarios and feasibility, governed by an enforced policy.
- Follow the CVE to the part number: SBOM value is the chain from component to software unit, system component, part number, vehicle and market.
- Incidents feed the TARA: a confirmed attack marks its risk realized; an unmodeled attack raises a TARA gap.
- Deployment is a gate: SSO, RBAC, MFA, audit trail, residency, air-gap feed sync and API access are pass or fail, decided before feature scoring.
Related reading on VxLabs Security
- Compare ThreatZ with your current setup
- ThreatZ platform
- ThreatZ for OEM CSMS teams
- ThreatZ for Tier-1 suppliers
- Automotive cybersecurity maturity model
- Building a unified CSMS operating model
Authoritative references
- ISO/SAE 21434 Road vehicles — Cybersecurity engineering — ISO and SAE International, ISO/SAE 21434
- UN Regulation No. 155 Cyber security and cyber security management system — UNECE, R155
- UN Regulation No. 156 Software update and software update management system — UNECE, R156
- Cyber Resilience Act — European Parliament and Council, Regulation (EU) 2024/2847
- GB 44495-2024 Technical requirements for vehicle cybersecurity — SAMR/SAC, GB 44495-2024
- ISO/PAS 5112 Road vehicles — Guidelines for auditing cybersecurity engineering — ISO, ISO/PAS 5112