ROX-17374: Change dependency discovery method

[mtodor]

We have a problem that downstream builds are failing because Cachito is not storing the information required for ossls to work.

We are using go list -m command to discover golang dependencies. This command requires *.info files, which are no longer persisted in Cachito.

This PR is adding change to the method of how dependencies are discovered and fixing two additional problems.

Changes

Change 1 - filter packages without directory

This is a bug discovered with the previous functionality. We have a lot of modules like this:

{
   "Path": "sigs.k8s.io/kustomize/kustomize/v4",
   "Version": "v4.5.4",
   "Time": "2022-03-28T22:46:23Z",
   "Indirect": true
}

Because of a bug in reading JSON, such records would keep Dir from the last module with Dir defined. The outcome of such an operation is that we reported a license from package_1 under the package_2 name.

Example:

Proj: "github.com/steveyen/gtreap" - Dir: "/Users/mtodorov/go/pkg/mod/github.com/steveyen/gtreap@v0.1.0"
Proj: "github.com/stoewer/go-strcase" - Dir: "/Users/mtodorov/go/pkg/mod/github.com/steveyen/gtreap@v0.1.0"
Proj: "github.com/streadway/amqp" - Dir: "/Users/mtodorov/go/pkg/mod/github.com/steveyen/gtreap@v0.1.0"
Proj: "github.com/streadway/handy" - Dir: "/Users/mtodorov/go/pkg/mod/github.com/steveyen/gtreap@v0.1.0"

You can see that github.com/streadway/amqp will have a license from github.com/steveyen/gtreap.

Change 2 - use go mod download to get modules

Based on the investigation made here: https://github.com/chmeliik/godeps/tree/main#module-identification-methods-vs-corner-cases - there are several methods how to get dependencies. go mod download is problematic for replacements that are using local directories. I have checked stackrox and scanner, and we don't have such cases in these repositories.

We don't have information about Replace from go mod. We will simply get the package used as a replacement. And that is correct.

Change 3 - report licenses for github.com/stackrox/ repos

This is mostly relevant for replacements. We had a problem in the previous solution. We would report the license from a stackrox repository as it is a license from a replaced repository. This is incorrect. If we want to handle licenses properly - we should simply report them (also from our packages). And inside our packages, we should define the license accordingly.

Now we will have at least licenses from stackrox repos because they are actually used to build products. And not replaced libraries.

Testing

Test on stackrox/stackrox:

• build ossls - go build -o ossls
• use to generate report ossls audit --export ./THIRD_PARTY_NOTICES_report
• check results

Also, test PR for downstream is open:
https://gitlab.cee.redhat.com/stackrox/rhacs-midstream/-/merge_requests/1399

Made comparisons

Cmp 1 - list -json -m all vs mod download -json

Get list -json -m all results.
Get mod download -json results.

Step 1:
Remove all wrongly reported packages (packages that are not available - no Dir for them) from list -json -m all results. There is still a difference between list -json -m all and mod download -json.

Step 2:
Get diff between step 1 version of list -json -m all and mod download -json. All diff packages are dependencies used in tests of our dependency packages. They are not direct test dependencies. We are not shipping these libraries, and there is no need to report their licenses.

Cmp 2 - go-licenses vs mod download -json

I used https://github.com/google/go-licenses - to get results. There have been some differences. Some libraries were not reported by go-licenses. But some additional licenses have been reported.

An example of a reported license by go-licenses: sigs.k8s.io/kustomize/kyaml/internal/forked/github.com/go-yaml/yaml - this is a case where project has multiple licenses depending on used packages.

We didn't report these with the previous solution, and it's not a regression introduced with this functionality.

An example of not reported license by go-licenses: cloud.google.com/go/compute - this is used by some tools we are using in Makefile. So it does not hurt to list it if it's used during the build of the project.

[RTann]

@mtodor any plans to cut a new release with these latest changes? I think the current CI images are using a version of this repo without these changes