Weak Hashing findings vanished from 1.1.11 ruleset?

[davewichers]

I maintain the OWASP Benchmark project:

• https://owasp.org/www-project-benchmark/
• https://github.com/OWASP-Benchmark/BenchmarkJava

Which is a test suite for testing the effectiveness of software security analysis tools. I've had codeQL scripts for scanning OWASP Benchmark for a while.

They can be found in the scripts folder here: https://github.com/OWASP-Benchmark/BenchmarkJava/tree/master/scripts

• translateCodeQL.sh
• runCodeQL.sh

You have to follow the codeQL install instructions listed in the translateCodeQL.sh script, then run translateCodeQL.sh and then runCodeQL.sh.

After completing, it puts the codeQL SARIF results file into the results/ folder and you can then score the tool against Benchmark by running createScoreards.sh.

I noticed that when using the latest version of CodeQL 2.20.1, with the 1.1.9 ruleset, it properly detected 69% of the Weak Hashing test cases in Benchmark (and had zero false positives). But when I upgraded to ruleset 1.1.11, it now detects none of them.

Is this on purpose? Or was a bug introduced. or mistake made, to cause those rules to go away in 1.1.11?

[redsun82]

Hey @davewichers, thanks a lot for reaching out with this! I'll circle this back to our internal team responsible for Java analysis 👍

[redsun82]

Hey @davewichers, as reported in this change note, we recently removed reporting of MD5 and SHA1 hashing from the java/weak-cryptographic-algorithm to the less precise java/potentially-weak-cryptographic-algorithm, as the former was alerting on too many cases of legitimate non cryptographic usages of those hashes. Maybe you can switch to using that query instead in your benchmarking?

[davewichers]

@redsun82 - the codeQL documentation related to finding out which rulepacks exist and how to use them is super confusing. Where is the list of ALL the published codeql rulepacks? I only found: codeql\java-queries and codeql\java-all, but when I try to use java-all, it says 'this is a library and does not contain queries to run'. I want to use ALL the codeQL java rules in my run. How do I do that?

[redsun82]

Hi @davewichers, the concept of a set of queries is captured by "query suites":

• docs for running with the CLI (see in particular the explanation of the <packs,queries> positional argument). What I also found indeed a bit confusing here, is that we don't stress enough that this documentation relates to full codeQL bundles published as releases of codeql-action. Please find more instructions below.
• docs for codescanning
• files defining query suites for java here)

I heard back from the team, and they advise to use the java-security-extended suite for the benchmark. To do so, you should provide java-security-extended or java-security-extended.qls as second positional argument to codeql database analyze. If you download the latest release for the full codeql bundle, that will be included together with precompiled queries which will make the command run faster. You can however also run the queries from a checkout of codeql, provided you give the right --search-path.

Where is the list of ALL the published codeql rulepacks?

Probably the best way to get that list is to run

codeql resolve queries

This will list all query suites with their .qls extension. The query suite argument to codeql database analyze can be provided either with or without that extension.

I'm sorry if you found the documentation lacking, we will try to make it better! 🙌

[davewichers]

Well, These clarifications didn't help either. I'm doing these steps: codeql pack download codeql/java-queries (works fine) codeql database create owasp-benchmark --language=java --overwrite --command="mvn clean package" (works fine) codeql database analyze owasp-benchmark codeql/java-queries --format-sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif (works fine) So trying to use a different 'stronger' java rule pack I tried this: codeql pack download codeql/java-all (works fine) codeql database create owasp-benchmark --language=java --overwrite --command="mvn clean package" (works fine) codeql database analyze owasp-benchmark codeql/java-all --format-sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif ERROR: CodeQL pack 'codeql/java-all' is a library and does not contain queries. So, I try to use the java-security-extended rulepack you suggest with: codeql pack download codeql/java-security-extended ERROR: blah blah Does this CodeQL pack exist on the public GitHub Container registry blah blah And this doesn't work either: codeql database analyze owasp-benchmark codeql/java-security-extended --format-sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif ERROR: blah blah that rulepack doesn't exist You state: "If you download the latest release <https://github.com/github/codeql-action/releases/latest> for the full codeql bundle," I finally figured out how to do that, but there is no java-security-extended query pack in that bundle. There are only java-queries and java-examples So, 1) How do you download the rule pack that contains java-security-extended, and then 2) How do you invoke it properly with: codeql database analyze owasp-benchmark WHATRULEPACKNAMEHERE?

…

On Mon, Jan 20, 2025 at 3:49 AM Paolo Tranquilli ***@***.***> wrote: Hi @davewichers <https://github.com/davewichers>, the concept of a set of queries is captured by "query suites": - docs for running with the CLI <https://docs.github.com/en/code-security/codeql-cli/getting-started-with-the-codeql-cli/analyzing-your-code-with-codeql-queries> (see in particular the explanation of the <packs,queries> positional argument). What I also found indeed a bit confusing here, is that we don't stress enough that this documentation relates to full codeQL bundles published as releases of codeql-action <https://github.com/github/codeql-action>. Please find more instructions below. - docs for codescanning <https://docs.github.com/en/code-security/code-scanning/managing-your-code-scanning-configuration/codeql-query-suites> - files defining query suites for java here <https://github.com/github/codeql/tree/main/java/ql/src/codeql-suites>) I heard back from the team, and they advise to use the java-security-extended suite for the benchmark. To do so, you should provide java-security-extended or java-security-extended.qls as second positional argument to codeql database analyze. If you download the latest release <https://github.com/github/codeql-action/releases/latest> for the full codeql bundle, that will be included together with precompiled queries which will make the command run faster. You can however also run the queries from a checkout of codeql, provided you give the right --search-path. Where is the list of ALL the published codeql rulepacks? Probably the best way to get that list is to run codeql resolve queries This will list all query suites with their .qls extension. The query suite argument to codeql database analyze can be provided either with or without that extension. I'm sorry if you found the documentation lacking, we will try to make it better! 🙌 — Reply to this email directly, view it on GitHub <#18518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGFWBN6WC4RC4CHUH5LOST2LS2DDAVCNFSM6AAAAABVKJBOTOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBRG44DEOBUGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>