fix: Do not add a request that is already in progress to MemoryRequestQueueClient

[Mantisus]

Description

• Do not add a request that is already in progress for MemoryRequestQueueClient.

Issues

• Closes: MemoryStorageClient issue with deduplication #1383

[vdusek]

Old version:

┌───────────────────────────────┬────────────────┐
│ requests_finished             │ 9024           │
│ requests_failed               │ 0              │
│ retry_histogram               │ [9024]         │
│ request_avg_failed_duration   │ None           │
│ request_avg_finished_duration │ 482.4ms        │
│ requests_finished_per_minute  │ 541            │
│ requests_failed_per_minute    │ 0              │
│ request_total_duration        │ 1h 12min 33.1s │
│ requests_total                │ 9024           │
│ crawler_runtime               │ 16min 40.5s    │
└───────────────────────────────┴────────────────┘

New version:

┌───────────────────────────────┬─────────────┐
│ requests_finished             │ 4512        │
│ requests_failed               │ 0           │
│ retry_histogram               │ [4512]      │
│ request_avg_failed_duration   │ None        │
│ request_avg_finished_duration │ 444.8ms     │
│ requests_finished_per_minute  │ 579         │
│ requests_failed_per_minute    │ 0           │
│ request_total_duration        │ 33min 26.9s │
│ requests_total                │ 4512        │
│ crawler_runtime               │ 7min 47.7s  │
└───────────────────────────────┴─────────────┘

With the new version, we only process about half as many requests.

However, based on the SDK's deduplication, there should only be 2362 unique pages, so there still seems to be an issue.

cc @Pijukatel

[Mantisus]

However, based on the SDK's deduplication, there should only be 2362 unique pages, so there still seems to be an issue.

2363 if the start URL is https://crawlee.dev/
4512 if the start URL is http://crawlee.dev/

This is a non-deduplication error.

In the JS version, similarly

[vdusek]

Oh, thanks for the explanation

[vdusek]

LGTM, let's wait for @Pijukatel as well, but I believe this is fine