Concepts
This page is an overview of all concepts used in panoptic. It can be a good reference to understand quickly a term or to get a global view of what's inside the software. For more information on a specific concept go to the dedicated page.
Projects are in panoptic a way to store your work. You could for instance create different projects on a same corpus if you aim to make different annotations or import different data along the images, or even mix different image corpuses in the same project.
For instance, it could be interesting to work on an historical photographs dataset to make annotations and explore the reccurent themes, but it could also be interesting to mix the same corpus with an internet scrapped corpus to see if some of these historical images can be found around websites and if yes in which context they appear.
When you create a project you must select a folder in which the project data will be stored. Each project stores its data separately in a sqlite database (if you want to see the detailled schema).
A project can be configured with parameters allowing to configure each plugin default values per project, but also for instance to store a copy of the images directly in the database, allowing easy share of the panoptic project by just sending a single file to someone.
One of the hardest concept to understand in panoptic in the difference between images and instances. To understand the necessity of such difference, we must consider the case of an image that was found on twitter, but posted by two different users, at different times, and with very different messages. It means that it's technically the same image, but since the contexts they appear in are so dissimilar, we might argue they are not semantically the same.
Which means that if we want to observe images in panoptic with their contexts stored in properties, a same image can appear multiple times in panoptic if it was found several times in the corpus with different properties.
So long story short, an image is only the group of pixels, and an image instance is the occurence of this group of pixels in a specific context. Therefore an image can have several instances, but an instance can only be bound to one image only.
Properties are what's used in panoptic to add additionnal data that is not images. They can be imported to panoptic or manually added to create annotations on the images.
Properties can be of different types depending on the kind of annotation that needs to be added. Different types will have different behaviour. For instance, date properties can allow to group images dynamically by year, month week etc...
| Property type | Description |
|---|---|
| Tag | Property can only contain a single tag |
| MultiTags | Property can contain multiple tags |
| URL | Clickable web url |
| Date | A date |
| Color | A color that can be chosen among 12 in panoptic |
| Checkbox | Simple checkbox |
| Number | Number that can be integer or float |
| Text | Textual data |
When created / imported, properties can be linked either to images or to instances. A image property will be shared among all instances of this image and is therefore often used to describe visual features of the image. An instance property is linked to a single instance of an image. To continue the example used earlier to describe instances, if we consider the same image posted two times on twitter: the authors, the date of publication of each time the image was posted or the comment associated with the text, all of those should be instances related and not image related:
The most used property types in panoptic are "tag" and "multi_tags" since they are pretty easy to create, manipulate and have autocomplete to facilitate annotations of images. Though it can be sometimes tricky to organize tags when there are a lot of them, especially since in panoptic, tags can be hierarchical AND have multiple parents. Therefore we also created a dedicated modal for tag management, which is often refered as "tags modal" and can allow to reorganize or rename your tags visually.
Properties can be imported through the import modal and by using a csv file.
Properties can be exported through the export modal to a csv file, a subselection of images can also be exported along the csv file.
Panoptic allows to create dynamic views of the images based on filters, sorts and groups.
Filters allow to display only the images matching the condition set in a filter or a combination of filters. Multiple filters can be created on multiple properties and can be combined using AND or OR keywords.
Sorts change the order in which images are displayed, they can be ascending or descending and can be applied over multiple properties, e.g : "sort by date ascending, then by text_description descending"
Groups regroup images by a selected property, subgroups can be created by selecting another property. Groups can then be sorted by the value of the property or by the number of elements inside the groups.
You can perform a full text search across all textual properties using the search bar.
When displaying images you can chose how they should be presented. There are curently three different "views":
- The default view is the grid view and is focused on the images.
- The table view allow to visualize more and / or longer properties
- The graph view creates graphs if you group images with a numerical or a date value. You can then hover the graph to visialize a sample of the images at each datapoint.
One main goal of panoptic is to give differents way of viewing images and to search inside the corpus at once. Therefore, we use tabs to create different workspace, wich each their own filters, sorts, groups, views and displayed properties.
To allow the use of machine learning algorithms first images are transformed in so called "vectors" (or embeddings) when imported in panoptic. Basically vectors are a list of numbers each representing "features" of the images. Features are not super easy to define but they are parts of the images that the deep learning model learnt to recognize as "important" during it's training.
For instance, with super simplified vectors we could have these vectors for the following images:
It's not easy to infer what each number represent on an image. But what we can do, is see that both cats have more similar vectors than the dog. With this in mind, we could then represent the images in a 2D space (for simplicity here we'll take only the first and last numbers of the vector but there are actually method that allows to compute relevant coordinates from vectors of any size, see for instance this article on PCA ):
We can see that cats are considered "more similar" based on these vectors. Panoptic vectors are the same but with vectors of size 452 instead of 3.
Once vectors are computed, images can be regrouped algorithmically by computing clusters. Unlike groups created with the properties, clusters can be parametrized, can use several algorithm to obtain different results, and are not deterministic, running a clustering twice can lead to different results (even if most of the time it's only small changes). This also means that depending on the number of images you have, the parameters you use, and the algorithm used to compute the clusters, running a clustering can last from a few milliseconds to several minutes for the worst cases. On the previous image, clusters could be represented by the doted blue circles.
Once vectors are computed, the similarity between images can be obtained. Just like clusters, different algorithms can be used to compute the similarity between two images. The one used by default in panoptic is called the cosine similarity, with values being caped between 0 and 1.
Computations that are not one of the ones listed above can be run by using the "Execute" button, which regroups all differents execute actions provided by the plugins. The default plugin offers to search images by using text-image similarity. Other examples of computations could be:
- find main colors of images and store them in properties
- run OCR on the images
- find the main topics of text properties associated to images
- extract objects from images
- ...
Plugins are a way to add new functionnalities to panoptic, and by enabling them on the project level, allowing to select only what you need, when you need it. Even the main deep learning functionnalities of panoptic are embedded in a plugin and panoptic can be actually used without it. (Even if the usage will of course be limited, it allows users that just want to manipulate images through their properties to be able to do so with a lightweight tool.
Plugins can add new ways of:
- creating clusters
- computing similarities
- add any specific action on the images through the execute button.
Here is the doc to create your own plugin
And feel free to check our own official plugins: