feat: Add always_enqueue option to bypass URL deduplication
#621
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
@vdusek I have reviewed this issue and your suggestions and implemented them in this PR. Please review and suggest changes, if any. Thanks. cc; @janbuchar |
vdusek
requested changes
Oct 29, 2024
Collaborator
There was a problem hiding this comment.
This looks quite good, but make sure the CI passes (lint check, type check, unit tests... all are failing), see the CONTRIBUTING.md for more information.
Co-authored-by: Vlada Dusek <[email protected]>
Contributor
Author
|
@vdusek Please run the CI checks again. I have pushed a commit to fix them all. Thanks. cc: @souravjain540 |
janbuchar
reviewed
Oct 31, 2024
janbuchar
approved these changes
Oct 31, 2024
Collaborator
janbuchar
left a comment
janbuchar
left a comment
There was a problem hiding this comment.
LGTM, thank you for your contribution!
1 task
1 task
Pijukatel
added a commit
to apify/apify-sdk-python
that referenced
this pull request
Nov 28, 2025
#677) ### Description - Make sure that storage from `ApifyFileSystemStorageClient` does not get purged twice due to storage from `FileSystemStorageClient` pointing to the same location. (Those storage clients will have the same cache key and thus there can be only one.) - Ensure that `Actor` will open input containing KVS on initialization to ensure that an aware storage client is used. - Support any possible pre-existing input key and file that is defined through `Configuration.input_key`. Different input files will have different handling based on their suffix: - ".json" is parsed as json. - ".txt" is opened as plain text - everything else is opened as bytes - without extension is tried to be parsed as json first, but falls back to bytes - Create a metadata file for the valid pre-existing input file, without modifying the input file (otherwise, cli might detect the change to the input, which would be a false positive) - Raise an error if two valid pre-existing input files exist in the expected storage directory. - CLI does not respect env variables with the input key so far. TODO: apify/apify-cli#960 ### Issues Closes: apify/crawlee-python#621 Related to: [#INPUT.json Automatically Deleted on Each Run (Python SDK Local Storage Issue)](#686) ### Testing - Added unit tests. - Manually tested with [[email protected]](https://www.npmjs.com/package/apify-cli/v/1.1.2-beta.20) - npx [email protected] run -i {\"a\":\"c\"} with pre-existing input file or without input and multiple times in a row - npx [email protected] run with pre-existing input file or without input and multiple times in a row
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR implements a new input parameter, always_enqueue, in the Request.from_url constructor. This feature allows users to bypass the request deduplication process, ensuring that each request is always enqueued and processed.
Key Changes:
With this addition, users can seamlessly manage their requests while maintaining flexibility in their queuing strategy.
Issues
Fixes #547
Testing
The modifications can be verified by making requests to the same URL. The operations should execute each time this is done without treating it as a duplicate request when
always_enqueueis set to true.Checklist