Adding memory safety related checks

[balteravishay]

Is your feature request related to a problem? Please describe.
Memory safety comes up quite frequently these days in regards to developing secure and safe software. Yet there are hardly any automation tools that try to validate the "memory safety" of a software project. below are some ideas around that notion that were discussed in the recent OpenSSF Memory Safety SIG meetings.

Describe the solution you'd like
Adding platform/language specific checks for memory safety to scorecard, bundled under "Memory Safety" category.

This can be built using existing OpenSSF knowledge and content such as the C/C++ binary hardening guide from the best practices WG, the memory safety SIG's best practice for memory safe by default languages. It and can include checks such as mentioned in issue #200, or the use of automation tools that detect memory safety hardening of the application. It may look at python ctypes dependencies, or at Java UseBoundChecks or older version's Djava.security.manager=unrestricted.
These are just preliminary ideas to raise the discussion on memory safety related checks in Scorecard.

Describe alternatives you've considered
There are no alternatives at the moment for checking or scoring repositories based on memory safety practices.

[ccpalmer]

Are there examples of automation capable of making such an assessment?

[pnacht]

Can't SAST tools detect many common memory safety issues? Not all of them, of course (i.e. global variables used everywhere, concurrency, etc), but a fair share of common "classes" of mistakes.

[ccpalmer]

The OWASP SAST page says that finding buffer overflows is a strength of SAST tools, but it also says "High numbers of false positives." and "Difficult to ‘prove’ that an identified security issue is an actual vulnerability". Appears to be a case of "it's good at some things, but so bad at others it may prove less useful."

[pnacht]

Oh yeah, SAST is unfortunately riddled with false positives. But to me, that seems like something that should be fixed in the SAST tools, not necessarily something that Scorecard should implement itself.

[evverx]

This can be built using existing OpenSSF knowledge and content such as the C/C++ binary hardening guide from the best practices WG

While that guide is helpful I don't think it's generally applicable to upstream projects scorecard is supposed to assess because usually upstream projects don't compile their own compilers and all their defaults are overridden downstream with custom CFLAGS, CXXFLAGS and so on anyway. If production binaries aren't hardened it should be reported to vendors (or whoever produces and ships those binaries).

(if upstream projects aren't compatible with certain flags it should be reported upstream of course but it goes far beyond what socrecard can do).

checks such as mentioned in issue #200

As far as I can tell the idea behind that check is to figure out whether projects (and their dependencies) are written in Rust (or any other memory-safe languages) and how much code is unsafe. In other words a hypothetical DNS server written in Rust would get more points than a hypothetical DNS server written in C. That sort of check makes sense upstream but it would have to rely on external tools like cargo-geiger. Then again cargo-geiger alone can't be used to scan projects written in both C and Rust.

the use of automation tools that detect memory safety hardening of the application

I'm not 100% what it means but since binskim was mentioned my guess would be that it isn't applicable to upstream projects either.

Can't SAST tools detect many common memory safety issues?

I think the idea behind the SAST, Fuzzing, CI and Binary-Artifacts checks was to figure out whether projects follow practices that are supposed to catch those kinds of issues (and prevent projects from shipping binaries at all). If for example projects written in memory-unsafe languages are continuously fuzzed it kind of implies that they are built and run under ASan/UBSan/MSan with all sorts of weird things passed to them. It doesn't guarantee that they have no bugs of course (nothing can guarantee that) but it at least shows that they care about corner cases (where all those buffer overflows, use-after-frees and so on with funny names come from).

All in all I think #200 kind of makes sense upstream. The other things probably make sense downstream.

[balteravishay]

Thanks for the discussion!

Agree that some of the checks may be included in SAST. for instance the Java samples above would presumamly be caught with SAST tools. The question I guess is should scorecard make special case for memory safety checks that may or may not be included in specific SAST tools. for instance, is a project written in go uses unsafe package or leverages the go data race detector during CI, as suggested here. I'm not sure if a SAST tool that is ran on the code would catch that, or any such "build time" checks, that are mandated by scorecard in some cases.

regarding @evverx's comment about the binary hardening guide. scorecard today does have some checks on how binaries are built or shipped like pinned-dependency, packaging, signed-releases, etc. if we check for packaging or binaries with SC, why not include checking the flags that are used in those?

Lastly, memory safety is not only that we use rust. almost all languages, including rust, may tap into the "unsafe" zone. so the idea behind this thread is to see if we can somehow surface memory safety practices through scorecard.

[evverx]

scorecard today does have some checks on how binaries are built or shipped like pinned-dependency, packaging, signed-releases, etc.

Those checks aren't applicable to a lot of projects either. For example low-level projects responsible for the underlying infrastructure are never built upstream. They release tarballs consumed by distributions, vendors producing various devices and so on. Those upstream projects have no control over how they are packaged and built there.

(It makes sense to release signed tarballs of course but there are no binaries, dependencies or packages there. they are all downstream)

The problem is that one-size-fits-all approach with a lot of different ecosystems doesn't work well.

Lastly, memory safety is not only that we use rust. almost all languages, including rust, may tap into the "unsafe" zone

Agreed but I was talking about #200 specifically and it boils down to whether projects are written in memory-safe languages and how far they tap into the unsafe zone (that's what cargo-geiger does).

[zmanion]

I'll offer a very coarse (possiby too coarse) check, "How many LoC are written in memory-unsafe langauges?" Since this is so coarse, maybe treat it as a low risk level?

Data can be pulled from GitHub, via API, or using cloc or linguist (what GitHub uses).

1. Decide what langauge count as "memory unsafe," e.g., (in cloc terms): "C" "C++" "C/C++ Header" "Assembly" "D" "Cython"
2. Possibly discount "non-executable languages," e.g., markdown maybe shouldn't count against the total LoC since markdown can't create a vulnerability?
3. unsafe_lang_use = unsafe_loc / ( total_loc - comments - blank lines - non-exec_lines )

IIUC GitHub/linguist measures file size not LoC. Not claiming one is better than the other, I've recently used cloc to count LoC.

cc @AevaOnline

[evverx]

"How many LoC are written in memory-unsafe languages?"

scorecard should recommend something actionable to maintainers based on that and I'm not sure what it can do without using ecosystem-specific tools (if projects are already written in memory-safe languages). If projects are written in memory-unsafe languages it would probably imply that they should be rewritten and that could be kind of controversial.

[pnacht]

scorecard should recommend something actionable to maintainers [...] If projects are written in memory-unsafe languages it would probably imply that they should be rewritten and that could be kind of controversial.

Yeah, Scorecard must always be actionable:

scorecard/checks/write.md

Lines 20 to 21 in 2ef20f1

	- Any criteria in the scorecard must be actionable. It should be possible,
	with help, for any project to "check all the boxes".

It's not explicit, but I also take that to imply "and feasible/reasonable". That is, the remediation can't be "rewrite your C project in Rust".

And if that's not acceptable, then the check has to be something that a C project can get a 10/10 in... which basically takes any sort of LoC-per-language metric out of contention.

So, as far as I can tell, we can basically just evaluate the use of tooling... but tooling that detects memory safety issues usually falls under the SAST category (or DAST, but that's less common).

(I considered metrics such as "use of raw pointers instead of smart pointers", but that just sounds like a SAST check)

[alpire]

If projects are written in memory-unsafe languages it would probably imply that they should be rewritten and that could be kind of controversial.

That would be controversial, but perhaps warranted nonetheless?

There is now a lot of data that memory-unsafe languages lead to bad security outcomes. Memory unsafety commonly accounts for 70%+ of severe vulnerabilities in C and C++ codebases. I think scorecards would more accurately evaluate security risks by having a more opinionated stance against inherently insecure technologies. Right now, it doesn't seem quite right that, say, a C media parser would be scored as "safe" as an equivalent memory-safe parser (all else being equal, e.g. SAST/DAST usage).

If a stronger stance against insecure technologies is not an option, may I suggest:

• Increasing the requirements on those insecure technologies. For instance, to achieve similar security outcomes to a Rust project with few unsafe blocks, it's likely a C or C++ project would need fuzzing with high coverage. So perhaps fuzzing and its coverage should be weighed more heavily for memory unsafe languages (even though fuzzing is not sufficient to make C or C++ sufficiently safe).
• Code quality metrics indicating usage of secure-by-design practice. For instance, adherence to C++ Buffer hardening would prevent most spatial safety vulnerabilities.

[evverx]

perhaps warranted nonetheless?

Who is going to do that? Here's what the curl maintainer wrote recently for example: https://daniel.haxx.se/blog/2023/12/13/making-it-harder-to-do-wrong/

Why no rewrite
Because I’m not an expert on rust. Someone else would be a much more suitable person to lead such a rewrite. In fact, we could suspect that the entire curl maintainer team would need to be replaced since we are all old C developers maybe not the most suitable to lead and take care of a twin project written in rust. Dedicated long-term maintainer internet transfer library teams do not grow on trees.

I think scorecards would more accurately evaluate security risks

Can't argue with that. From a consumer point of view it's totally reasonable. The problem is that scorecard supposedly helps maintainers too and "just rewrite your thing in Rust" is unhelpful (and annoying) and that's what I was trying to say. I'm not arguing against memory safe languages here.

[evverx]

fuzzing with high coverage

I don't know if scorecard is planning to go down that route but just in case In its current form the tooling scorecard can use to measure that isn't good enough (the Fuzz-Introspector folks are aware of its limitations) but other than that high coverage in a vacuum isn't what matters in terms of accessing projects because it can't be used to figure out whether code paths triggered on receiving weird packets (or interacting with hostile environments in general) are actually covered and how high that coverage is. Other than that high coverage can annoy folks doing their best but having low coverage just because their projects do more than just parsing things: google/oss-fuzz#10924 (comment)

fuzzing and its coverage should be weighed more heavily

That's reasonable but it would be nice if the tooling actually made it easier to do that and was in line with how software gets developed in 2024: google/oss-fuzz#7190 and google/oss-fuzz#11398. Without that it's going to be unhelpful "guidance" and "best practices" all over again.

[jduck]

Can't SAST tools detect many common memory safety issues? Not all of them, of course (i.e. global variables used everywhere, concurrency, etc), but a fair share of common "classes" of mistakes.

The tools out there are very noisy, often presenting a majority of false positives. Developers don't have the security expertise to proper interpret the results and thus sometimes mark true positives as false positives. There are a lot of potential issues that they cannot identify at all. It's not to say such tools cannot improve, but so far it has been quite hard to realize the benefits from them.

EDIT: I see this was already stated, oops.