feat(nvd): use go to upload NVD conversion to gcs upon conversion#5099
feat(nvd): use go to upload NVD conversion to gcs upon conversion#5099jess-lowe wants to merge 17 commits intogoogle:masterfrom
Conversation
There was a problem hiding this comment.
We should make the gcs-tools repo generic to only uploading to GCS, but we shouldn't put CVE specific logic into here.
There was a problem hiding this comment.
If uploading to GCS is going to take a while, I would even put the multithreading / concurrency logic in here.
E.g. provide a function that will spin up X number of works, and a "gcs client" that just contains a channel.
Other code can pass the client to their code to upload.
Probably for a separate PR though.
There was a problem hiding this comment.
We should make the gcs-tools repo generic to only uploading to GCS, but we shouldn't put CVE specific logic into here.
Moved these into their own thing in conversion/writer
There was a problem hiding this comment.
If uploading to GCS is going to take a while, I would even put the multithreading / concurrency logic in here. E.g. provide a function that will spin up X number of works, and a "gcs client" that just contains a channel.
Other code can pass the client to their code to upload.
Probably for a separate PR though.
For uploading vulnerability records, this is too nuanced, hence it has its own thing in writer.VulnWorker, but with the NVD data this will be happening in the same thread that converts the record
…sv.dev into refactor/nvd-use-gcs
This PR introduces support to immediately upload records to GCS instead of saving locally and then syncing - using the helper functions defined in https://github.com/google/osv.dev/pull/4984/changes.
The decision making for when/whether data is saved/uploaded/downloaded is now done at a much higher level, also allowing for the CVEToOSV function to focus only on converting the record and returning the vulnerability.