Lightly

DINO tricks, Imagenette benchmarks, API improvements

DINO tricks
DINOHead now allows to freeze the last layer which stabilizes the model performance.
DINOHead now also allows to normalise the last layer.
This was implemented by Atharva-Phatak. Thank you very much!

Imagenette benchmarks
We now include benchmarks of all models on Imagenette.

API Improvements
Better documentation of custom metadata
The CLI command to upload custom metadata is now included in the command line tool examples.
Better dataset upsizing
Upsizing a dataset in the Lightly Platform by adding more samples to it now cannot happen accidentally anymore, instead you have to specify `append=True`. Furthermore, bugs regarding appending new custom metadata have been fixed.
Create ApiWorkflowClient with token from env
When creating an `ApiWorkflowClient`, you can now pass the token as environment variable `LIGHTLY_TOKEN` instead of as argument.
Bugfixes in `check_embeddings()`
When checking embedding files, now columns like `masked` and `selected` are accounted for properly.

Models
- [Bootstrap your own latent: A new approach to self-supervised Learning, 2020](https://arxiv.org/abs/2006.07733)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021](https://arxiv.org/abs/2103.03230)
- [DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021](https://arxiv.org/abs/2104.14294)
- [SimSiam: Exploring Simple Siamese Representation Learning, 2020](https://arxiv.org/abs/2011.10566)
- [MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019](https://arxiv.org/abs/1911.05722)
- [SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020](https://arxiv.org/abs/2002.05709)
- [NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021](https://arxiv.org/pdf/2104.14548.pdf)
- [SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020](https://arxiv.org/abs/2006.09882)

Active Learning for Keypoint Detection and New API Workflows

Active Learning: Keypoint Detection
707 added an active learning scorer for keypoint detection. This means that you can now do active learning for keypoint detection with Lightly!

New API Workflows
With 712 and 714 we added all the necessities to [configure and run the compute worker](https://docs.lightly.ai/docker/integration/docker_with_datasource.html) from your Python code.

Access Shared Datasets
722 makes it possible to access datasets which were shared with you in the Lightly web-app from your Python code as if they were your own.

Documentation
We made many smaller changes and fixes in the docs 🙂

Models
- [Bootstrap your own latent: A new approach to self-supervised Learning, 2020](https://arxiv.org/abs/2006.07733)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021](https://arxiv.org/abs/2103.03230)
- [DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021](https://arxiv.org/abs/2104.14294)
- [SimSiam: Exploring Simple Siamese Representation Learning, 2020](https://arxiv.org/abs/2011.10566)
- [MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019](https://arxiv.org/abs/1911.05722)
- [SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020](https://arxiv.org/abs/2002.05709)
- [NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021](https://arxiv.org/pdf/2104.14548.pdf)
- [SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020](https://arxiv.org/abs/2006.09882)

Bugfixes and Documentation Improvements

Bugfix in train_model_and_embed_images()
Fixed the bug that the `train_model_and_embed_images()` function did not return the embeddings, filenames and labels. Thanks for finding this bug opassos!

Bugfix in video loader
When using the video_loader backend of torchvision, it could fail when training for multiple epochs and had a linearly growing demand for memory and file handlers. This was fixed.

Documentation update in Readme.md
The benchmark table for CIFAR10 was fixed and the DINO model added to the list of supported models.

Models
- [Bootstrap your own latent: A new approach to self-supervised Learning, 2020](https://arxiv.org/abs/2006.07733)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021](https://arxiv.org/abs/2103.03230)
- [DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021](https://arxiv.org/abs/2104.14294)
- [SimSiam: Exploring Simple Siamese Representation Learning, 2020](https://arxiv.org/abs/2011.10566)
- [MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019](https://arxiv.org/abs/1911.05722)
- [SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020](https://arxiv.org/abs/2002.05709)
- [NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021](https://arxiv.org/pdf/2104.14548.pdf)
- [SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020](https://arxiv.org/abs/2006.09882)

DINO:
We added support for the [DINO](https://arxiv.org/abs/2104.14294) model which works with ResNet and Transformer backbones! To get started head over to our [examples section](https://docs.lightly.ai/examples/dino.html) in the docs which contains example code for DINO using PyTorch and PyTorch Lightning.

Update Cifar10 Benchmarks
We updated the Cifar10 benchmarks with results for all models provided by Lightly. The benchmark does now only use low level blocks which we introduced in a [previous release](https://github.com/lightly-ai/lightly/releases/tag/v1.2.0). You can find the updated benchmark results and code in our [docs](https://docs.lightly.ai/getting_started/benchmarks.html#cifar10).

Models
- [Bootstrap your own latent: A new approach to self-supervised Learning, 2020](https://arxiv.org/abs/2006.07733)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021](https://arxiv.org/abs/2103.03230)
- [DINO: Emerging Properties in Self-Supervised Vision Transformers, 2021](https://arxiv.org/abs/2104.14294)
- [SimSiam: Exploring Simple Siamese Representation Learning, 2020](https://arxiv.org/abs/2011.10566)
- [MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019](https://arxiv.org/abs/1911.05722)
- [SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020](https://arxiv.org/abs/2002.05709)
- [NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021](https://arxiv.org/pdf/2104.14548.pdf)
- [SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020](https://arxiv.org/abs/2006.09882)

Faster Image Upload
When uploading images through the CLI interface we now by default use 8-32 threads to speed up the process. This setting can be controlled with using the `loader.num_workers` argument.

Create Tags from Filenames
The `ApiWorkflowClient` now has a `create_tag_from_filenames` method which allows you easily to create a new tag on the Lightly Platform by passing a list of filenames.

Updated Visibility of Environment Variables in Docs
All environment variables in the docs should now be properly indicated using braces: `{ENV_VARIABLE}`

Models
- [Bootstrap your own latent: A new approach to self-supervised Learning, 2020](https://arxiv.org/abs/2006.07733)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021](https://arxiv.org/abs/2103.03230)
- [SimSiam: Exploring Simple Siamese Representation Learning, 2020](https://arxiv.org/abs/2011.10566)
- [MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019](https://arxiv.org/abs/1911.05722)
- [SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020](https://arxiv.org/abs/2002.05709)
- [NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021](https://arxiv.org/pdf/2104.14548.pdf)
- [SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020](https://arxiv.org/abs/2006.09882)

Bug fixes for Video datasets

Make LightlyDataset threadsafe for videos
The VideoLoader now uses threading.local which allows the loader to be used from multiple threads.

LightlySubset of VideoDataset returns correct filenames
Resolved the bug when calling `subset.get_filenames()` on the subset of a video dataset would return the filenames of the full dataset instead of the subset.

Download of videos allows restarting
We added retries to handle errors when downloading videos.

Getting video meta information allows ignoring metadata
When trying to find out meta information about a video like the number of frames or timestamps there are two ways:
- Try to read it from the metadata. However, sometimes this metadata is missing or false.
- Find it out by reading the video itself. This is computationally expensive

We now allow ignoring metadata and choosing the 2nd option, which is slower but guaranteed to give the correct data.

Models
- [Bootstrap your own latent: A new approach to self-supervised Learning, 2020](https://arxiv.org/abs/2006.07733)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction, 2021](https://arxiv.org/abs/2103.03230)
- [SimSiam: Exploring Simple Siamese Representation Learning, 2020](https://arxiv.org/abs/2011.10566)
- [MoCo: Momentum Contrast for Unsupervised Visual Representation Learning, 2019](https://arxiv.org/abs/1911.05722)
- [SimCLR: A Simple Framework for Contrastive Learning of Visual Representations, 2020](https://arxiv.org/abs/2002.05709)
- [NNCLR: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021](https://arxiv.org/pdf/2104.14548.pdf)
- [SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, M. Caron, 2020](https://arxiv.org/abs/2006.09882)