Fix/processors fit process

[jhnwu3]

This pull request introduces several improvements and fixes across documentation, benchmarking scripts, dataset processing, and processor design. The most significant update is the addition of comprehensive documentation for writing custom feature processors, clarifying the use of fit() and process() methods. There are also bug fixes and enhancements to the dataset loading logic, as well as updates to benchmarking scripts for more consistent development and memory settings.

Documentation Improvements:

• Added a detailed section to the processors documentation explaining how to write custom FeatureProcessor classes, including guidance on when to use fit() vs. process(), example implementations, and processor registration instructions.

Benchmarking Script Updates:

• Changed the default development and memory limit settings in benchmark_workers_1.py and benchmark_workers_4.py to ensure benchmarks run with production-like configurations by default. [1] [2]
• Updated cache directory usage and naming in benchmark_workers_4.py for clarity and to avoid overwriting previous results. [1] [2]
• Added a new example script, timeseries_mimic4.py, demonstrating dataset creation, task assignment, and dataset splitting for time series data.

Dataset Loading and Processing Enhancements:

• Improved timestamp handling in BaseDataset.load_table() to avoid errors with missing or invalid values, using errors="coerce" and ensuring proper datetime conversion.
• Refactored code for readability and maintainability, including better formatting of warning messages, merging logic, and limiting patient selection in dev mode. [1] [2] [3] [4] [5] [6]
• Added a fit() method to TimeseriesProcessor to automatically determine the number of features (n_features) from input data, improving processor usability and reliability.

Versioning:

• Bumped the project version from 2.0a11 to 2.0a12 in pyproject.toml.

[Logiquo]

process is always only called once during the cache phase. But it should be stateless as whatever mutation done in the process

1. may or may not share with another sample running process call
2. is not saved at the final SampleDataset.

[jhnwu3]

Yeah I think that's why the timeseries processor kept erroring as self.n_features is never set unless in fit(). I will update the docs to correct this info though as it's still being pretransformed.

[Logiquo]

This would work for the else case, though i think the if case is also problematic is some of the columns are NA. But we can worry about it in a later PR.

[Logiquo]

I think this may hide errors? e.g. if someone specify the timestamp_format wrong, they don't see any errors but see lots of null timestamp, which may lead to confussion.

[jhnwu3]

Will change this in next commit.

[jhnwu3]

Will merge once some jobs finish running, and then update pypi to have a 2.0a13