🚀 Features
<details>
<summary>Split up `som_utils.py` into separate modules for pixel and cell clustering alex-l-kong (662)</summary>
**What is the purpose of this PR?**
Closes 544. It's confusing to have both pixel and cell clustering helper functions reside in the same module, so we'll split them up.
**How did you implement your changes**
Reorganize the functions to `pixel_cluster_utils.py` and `cell_cluster_utils.py`, with the test modules also being moved accordingly.
</details>
<details>
<summary>Remove Rplots.pdf camisowers (638)</summary>
**If you haven't already, please read through our [contributing guidelines](https://ark-analysis.readthedocs.io/en/latest/_rtd/contributing.html) before opening your PR**
**What is the purpose of this PR?**
Closes 519. Removes the pdf that is produced by R scripts in the example_pixel_clustering notebook.
</details>
<details>
<summary>Docker Tags in line with ark-analysis Version srivarra (636)</summary>
**If you haven't already, please read through our [contributing guidelines](https://ark-analysis.readthedocs.io/en/latest/_rtd/contributing.html) before opening your PR**
**What is the purpose of this PR?**
Closes 634, 659.
Currently `latest` always points to the most recent build of the docker image, which may not be the current version of the repository. As a best practice we should tag the docker image to the same version of ark-analysis.
**How did you implement your changes**
Adjusted the docker build section of the `.travis.yml` to include the tag, and pushing it properly.
**Remaining issues**
None atm.
</details>
<details>
<summary>Add nuclear filtering functionality to pixel clustering pipeline alex-l-kong (653)</summary>
**What is the purpose of this PR?**
Closes 619. Prior to pixel clustering, users may need to filter out non-nuclear marker signal from the nucleus and vice-versa for nuclear marker signal.
**How did you implement your changes**
Add a function `filter_with_nuclear_mask` to `som_utils` which runs this process. The code in 619 provides most of the functionality, however we'll need to make two modifications:
- Ensure `seg_dir is not None` prior to running. Users may run pixel clustering without segmentation labels, however they will not be able to run this function if that's the case because it relies on the `_feature_1.tiff` files.
- Ensure `channel` defaults to `None` in the notebook and function. This is the easiest guard against errors caused by running all the cells at once in the notebook.
The cell that calls this function in `example_pixel_clustering` will be placed after the smoothing process.
This PR also solidifies some of the notebook testing, as it was not using the renamed channels after smoothing. This should also apply for the nuclear filtering tests too.
</details>
<details>
<summary>Added Batching Functionality to Pixel Mask Generation and FOV Saving srivarra (643)</summary>
**If you haven't already, please read through our [contributing guidelines](https://ark-analysis.readthedocs.io/en/latest/_rtd/contributing.html) before opening your PR**
**What is the purpose of this PR?**
Closes 552
**How did you implement your changes**
Added two batching functions for the pixel and cell clustering notebooks. In addition added testbook tests.
**Remaining issues**
None ATM.
</details>
<details>
<summary>Create Mantis Project Function srivarra (641)</summary>
- added mantis function
- mantis_project made
**If you haven't already, please read through our [contributing guidelines](https://ark-analysis.readthedocs.io/en/latest/_rtd/contributing.html) before opening your PR**
**What is the purpose of this PR?**
Closes 563.
**How did you implement your changes**
Added the function which creates a mantis project directory. Added tests to the mantis project function.
**Remaining issues**
Need to encapsulate pixel clustering output as well. Potential reorganization of the output if necessary. Also need to add an argument which only runs over a subset of fovs supplied (so the user won't copy over their entire run contents into a mantis project).
</details>
<details>
<summary>Save fov images as int16 camisowers (644)</summary>
**If you haven't already, please read through our [contributing guidelines](https://ark-analysis.readthedocs.io/en/latest/_rtd/contributing.html) before opening your PR**
**What is the purpose of this PR?**
Closes 521.
Edits the `save_fov_images` function so the pixel clustering notebook saves overlays images that can be loaded into photoshop.
**Remaining issues**
Could instead add the saving dtype as a default arg to the function, if that's something you think we'd use.
</details>
<details>
<summary>Ensure the meta cluster rename drop down menu for remapping doesn't get cut off alex-l-kong (642)</summary>
**What is the purpose of this PR?**
Closes 630. The way the toolbar is set up, the meta cluster renaming textbox/drop-down menu can get cut off on the far right. This needs to be pushed further to the center.
**How did you implement your changes**
Adjust the toolbar's `layout.justify_content` setting to `'center'`.
</details>
<details>
<summary>Add option to restart pixel clustering notebook from crash alex-l-kong (620)</summary>
**What is the purpose of this PR?**
Closes 575. `example_pixel_clustering.ipynb` includes very expensive processes such as preprocessing, SOM training, and SOM cluster assignment. This PR provides a way to prevent these from re-running (at least in full) for items that have already been checked off on a previous run that crashed midway through.
**How did you implement your changes**
Add options to check for files, or specific columns in files, indicating that certain processes have finished or not. This ensures we only need to process FOVs that haven't been completed.
</details>
<details>
<summary>Removal of `force_ints` in all load functions. srivarra (609)</summary>
**If you haven't already, please read through our [contributing guidelines](https://ark-analysis.readthedocs.io/en/latest/_rtd/contributing.html) before opening your PR**
**What is the purpose of this PR?**
Closes 606.
`load_imgs_from_tree` and `load_imgs_from_dir` will no longer allow any type conversions. There are significantly fewer user accessible type conversion options throughout ark.
**How did you implement your changes**
Removed the majority of front facing instances `dtype` in function arguments. In `deepcell_service_utils` the raw bytes output from `Deepcell` will now be converted from floating point to integer values.
**Remaining issues**
The JSON returned by Deepcell has the following metadata when running the `Segment_Image_Data.ipynb` notebook:
`{"shape": [1, 1024, 1024]}`. This is the same for this branch, master (after 605 was merged in), and with the last commit in 624. Seems like it was just something I got confused with when I was adjusting the reshaping earlier in the PR process.
</details>
<details>
<summary>Add `dtype` argument to channel and pixel norm functions alex-l-kong (623)</summary>
**What is the purpose of this PR?**
Closes 612. The `dtype` argument currently isn't propagated through to the channel and pixel normalization functions, causing the `load_imgs_from_tree` calls to attempt to load the images using the default `int16` type. This will throw a warning if the images were of type `float32` regardless of if the user passed `float32` to `create_pixel_matrix`.
**How did you implement your changes**
Add `dtype` with a default argument of `int16` to both `calculate_channel_percentiles` and `calculate_pixel_intensity_percentile`. Explicitly pass the `dtype` argument received by `create_pixel_matrix` to these functions.
</details>
🐛 Bug Fixes
<details>
<summary>Docker Tags in line with ark-analysis Version srivarra (636)</summary>
**If you haven't already, please read through our [contributing guidelines](https://ark-analysis.readthedocs.io/en/latest/_rtd/contributing.html) before opening your PR**
**What is the purpose of this PR?**
Closes 634, 659.
Currently `latest` always points to the most recent build of the docker image, which may not be the current version of the repository. As a best practice we should tag the docker image to the same version of ark-analysis.
**How did you implement your changes**
Adjusted the docker build section of the `.travis.yml` to include the tag, and pushing it properly.
**Remaining issues**
None atm.
</details>
<details>
<summary>Ensure channel norm and pixel norm files get re-computed if a new set of channels are provided alex-l-kong (654)</summary>
**What is the purpose of this PR?**
Closes 599. The underlying issue is if the user provides a different set of channels on a `pixel_output_dir` that has an existing `channel_norm.feather` computed on a different set of channels.
**How did you implement your changes**
We need to ensure that the set of channels provided when loading in an existing `channel_norm.feather` match the `channels` argument set-wise. If they don't, then we simply re-compute and re-save.
The same needs to be done for `pixel_norm.feather`. We store a checkpoint during the `channel_norm.feather` check to ensure this can happen seamlessly for the pixel norm too.
</details>
<details>
<summary>Pin matplotlib at 3.4.3 in requirements.txt to prevent colorbar scaling issue alex-l-kong (639)</summary>
**What is the purpose of this PR?**
Closes 614. Currently, the Python 3.7 image is built using the latest `matplotlib` version (3.5.x), which introduced several bugs to colorbar scaling. Several issues related to this are still open on the `matplotlib` repo, so it's best that we don't use this version for our purposes.
**How did you implement your changes**
Pin `requirements.txt` at version `3.4.2`.
**Remaining issues**
The Python 3.7 Docker image is still problematic because we haven't rebuilt it with `matplotlib==3.4.2` yet. Before this is done, users should continue using the Python 3.6/latest image.
Additionally, we should keep close watch of `matplotlib` updates on colorbar usage. Once they resolve the issue, we can matplotlib back to the most recent version.
</details>
<details>
<summary>add .json and .bin file compatibility to remove\_file\_extensions() camisowers (632)</summary>
**If you haven't already, please read through our [contributing guidelines](https://ark-analysis.readthedocs.io/en/latest/_rtd/contributing.html) before opening your PR**
**What is the purpose of this PR?**
Closes 631
</details>
<details>
<summary>Retry Deepcell in case of bad gateway (or similar) response errors alex-l-kong (605)</summary>
**What is the purpose of this PR?**
Closes 602, which indicates an error in decoding the JSON in case the Deepcell API returns a response indicating an error.
**How did you implement your changes**
Utilize the `Retry` module in `requests` and mount it to a `requests.Session`, which we use to invoke the `post` method to Deepcell.
We also add a `try`/`except` block around this statement in case none of the retries work and the resulting `json` cannot be decoded anyways.
**Remaining issues**
I'm a bit in the dark here because I've not encountered this error personally. ngreenwald let me know if this resolves the issue, or if it will need additional work.
</details>
<details>
<summary>PKG\_FOLDER variable in `setup.py` cannot find `requirements.txt` due to filepath issue. srivarra (624)</summary>
**If you haven't already, please read through our [contributing guidelines](https://ark-analysis.readthedocs.io/en/latest/_rtd/contributing.html) before opening your PR**
**What is the purpose of this PR?**
`ark-analysis` will not install properly via PyPI, the issue seems to reside with the `PKG_FOLDER` variable.
**How did you implement your changes**
Set `PKG_FOLDER` the same as it is in `toffy`.
**Remaining issues**
Need to adjust ark-analysis requirement in angelolab/toffy132, and make sure it can install properly.
</details>
alex-l-kong, camisowers and srivarra