Embeddings: add progress updates during indexing #53202
Conversation
camdencheek
force-pushed
the
cc/embeddings-progress
branch
from
5dcab3b to
0ea0281
Compare
| // JSONMessage wraps a value that can be encoded/decoded as JSON so that | ||
| // it implements db.Scanner and db.Valuer. |
There was a problem hiding this comment.
This is somewhat unrelated to the PR, but it makes working with JSON a little less painful. Normally, we'd scan to a []byte, then unmarshal separately and do the opposite during insert, but this wrapper allows you to just use a JSON-serializable type directly.
camdencheek
changed the title
| """ | ||
| The number of files scheduled to be embedded. | ||
| """ | ||
| filesScheduled: Int! | ||
|
|
||
| """ | ||
| The number of files we generated embeddings for. | ||
| This will be updated periodically while the embeddings job is processing. | ||
| """ | ||
| filesEmbedded: Int! | ||
|
|
||
| """ | ||
| The number of files skipped. | ||
| This will be updated periodically while the embeddings job is processing. | ||
| """ | ||
| filesSkipped: Int! |
There was a problem hiding this comment.
It's possible to expose more granular information, but I'd like to keep it simple for now.
|
Codenotify: Notifying subscribers in CODENOTIFY files for diff 26de97c...a25bd51.
|
client/web/src/enterprise/site-admin/cody/RepoEmbeddingJobNode.tsx
Outdated
Show resolved
Hide resolved
jtibshirani
left a comment
jtibshirani
left a comment
There was a problem hiding this comment.
This looks good to me. Just one remaining comment: the new table will still grow at the same rate as the main job table (and we don't prune old records), so it'd still be good to reduce the number of columns it has.
This adds incremental stats reporting to our embedding indexing jobs and displays it in the UI while the job is processing.
This adds incremental stats reporting to our embedding indexing jobs and displays it in the UI while the job is processing.
Closes https://github.com/sourcegraph/sourcegraph/issues/52358
Test plan
Manually tested clarity and usability. Added database test. Added unit test that ensures we update periodically.