--- tags: - Instance Segmentation - Vision Transformers - CNN - Optical Space Missions - Artefact Mapping - ESA pretty_name: XAMI-model license: mit datasets: - iulia-elisa/XAMI-dataset ---

XAMI-model: XMM-Newton optical Artefact Mapping for astronomical Instance segmentation

This repository contains the weights of the **[XAMI model](https://github.com/ESA-Datalabs/XAMI-model)**. The model is trained on images from the XAMI dataset (available on Github and HuggingFace). The images are astronomical observations from the Optical Monitor (XMM-OM) onboard the XMM-Newton X-ray mission. Information about the XMM-OM can be found here: - [The ESA website](https://www.cosmos.esa.int/web/xmm-newton/technical-details-om) - [The article *The XMM-Newton optical/UV monitor telescope*](https://ui.adsabs.harvard.edu/abs/2001A%26A...365L..36M/abstract).

Figure 1: The XAMI model combining a detector and segmentor, while freezing the detector model previously trained on the XAMI dataset.

## Cloning the repository ```bash git clone https://github.com/ESA-Datalabs/XAMI-model.git cd XAMI-model # creating the environment conda env create -f environment.yaml conda activate xami_model_env # Install the package in editable mode pip install -e . ``` ## Downloading the dataset and model checkpoints from HuggingFace The dataset is splited into train and validation categories and contains annotated artefacts in COCO format for Instance Segmentation. We use multilabel Stratified K-fold (k=4) to balance class distributions across splits. We choose to work with a single dataset splits version (out of 4) but also provide means to work with all 4 versions. The [Dataset-Structure.md](https://github.com/ESA-Datalabs/XAMI-dataset/blob/main/Datasets-Structure.md) offers more details about the dataset structure. We provide the following dataset formats: COCO format for Instance Segmentation (commonly used by [Detectron2](https://github.com/facebookresearch/detectron2) models) and YOLOv8-Seg format used by [ultralytics](https://github.com/ultralytics/ultralytics). Check the [dataset_and_model.ipynb](https://github.com/ESA-Datalabs/XAMI-model/blob/main/dataset_and_model.ipynb) for downloading the dataset and model weights. ## Model Inference After cloning the repository and setting up the environment, use the following code for model loading and inference: ```python from xami_model.inference.xami_inference import InferXami det_type = 'rtdetr' # 'rtdetr' 'yolov8' detr_checkpoint = f'./xami_model/train/weights/{det_type}_sam_weights/{det_type}_detect_300e_best.pt' sam_checkpoint = f'./xami_model/train/weights/{det_type}_sam_weights/{det_type}_sam.pth' detr_sam_pipeline = InferXami( device='cuda:0', detr_checkpoint=detr_checkpoint, sam_checkpoint=sam_checkpoint, model_type='vit_t', # the SAM checkpoint and model_type (vit_h, vit_t, etc.) must be compatible use_detr_masks=True, detr_type=det_type) masks = detr_sam_pipeline.run_predict('./example_images/S0893811101_M.png', show_masks=True) ``` For training the model, check the training [README.md](https://github.com/ESA-Datalabs/XAMI-model/blob/main/xami_model/train/README.md). ## Performance metrics

Figure 2: Cumulative distribution of IoUs between predicted and true masks using RT-DETR as detector.

Category	Precision		Recall
Category	YOLO-v8	RT-DETR	YOLO-v8	RT-DETR
Overall	84.3	62.7	72.1	78.3
Central-Ring	89.3	89.1	94.0	97.0
Read-out-Streak	71.1	68.3	73.3	95.3
Smoke-Ring	80.6	78.1	85.6	93.8
Star-Loop	80.5	71.6	74.1	83.3
Other	100.0	6.2	33.3	22.2

Table 1: Metrics per object detector used.