Inference on a single video

[luca-zanella-dvl]

Inference on a single video

[Ryan7180]

Just like Fig. 3 in the paper, thanks

[benedettaliberatori]

Hi , thank you for your interest in our work!

At inference time the evaluated videos are loaded based on the annotation_file_test txt file (you can check in configs/data/ the file yaml of your dataset of interest), which for example for the UCF-Crime dataset is the file

AnomalyCLIP/configs/data/ucfcrime.yaml

Line 20 in ffb0aa6

annotation_file_test: "/usr/src/datasets/UCFCrime/Annotations/Anomaly_Test.txt"

structured as:

Abuse/Abuse028_x264 0 1411 0
Abuse/Abuse030_x264 0 1543 0
Arrest/Arrest001_x264 0 2373 1
Arrest/Arrest007_x264 0 3143 1
...

One possibility is to create a new_txt file with the same format containing only the video, or subset of videos, you want to evaluate. Then you can run the evaluation as:

python src/eval.py model=anomaly_clip_<dataset_name> data=<dataset_name> ckpt_path=/path/to/checkpoints/<dataset_name>/last.ckpt data.annotation_file_test=/path/to/new_txt

This will run the evaluation on the selected videos only, and you can access the frame-wise abnormal scores and anomalous classes predicted in the function test_step. As for the code used to produce the plots with the curves of the anomaly scores, you can check this function:

def plot_abnormal_scores(abnormal_scores, labels, path, normal_id, save_dir):
    abnormal_scores_np = abnormal_scores.detach().cpu().numpy()

    fig, ax = plt.subplots(figsize=(18, 4))
    fig.subplots_adjust(top=0.95, bottom=0.15, left=0.06, right=0.99)

    x = np.arange(abnormal_scores.size(0))
    ax.plot(x, abnormal_scores_np, color="#4e79a7", linewidth=1)
    ymin, ymax = 0, 1
    xmin, xmax = 0, abnormal_scores.size(0)
    ax.set_xlim([xmin, xmax])
    ax.set_ylim([ymin, ymax])

    title = Path(path[0].strip().split()[0]).stem

    start_idx = None
    for i, label in enumerate(labels):
        if start_idx is None:
            if label != normal_id:
                start_idx = i
            else:
                rect = plt.Rectangle(
                    (start_idx, 0),
                    i - start_idx,
                    ymax - ymin,
                    color="#e15759",
                    alpha=0.5,
                )
                ax.add_patch(rect)
                start_idx = None
    if start_idx is not None:
        rect = plt.Rectangle(
            (start_idx, 0),
            len(labels) - start_idx,
            ymax - ymin,
            color="#e15759",
            alpha=0.5,
        )
        ax.add_patch(rect)

    ax.text(0.02, 0.90, title, fontsize=22, transform=ax.transAxes)
    for yline in [0.25, 0.5, 0.75]:
        ax.axhline(y=yline, color="grey", linestyle="--", linewidth=0.8)
    ax.set_yticks([0.25, 0.5, 0.75])
    ax.tick_params(axis="y", labelsize=16)

    ax.set_ylabel("Anomaly Probability", fontsize=19)
    ax.set_xlabel("Frame Number", fontsize=19)

    fig_file = save_dir / f"{Path(path[0]).name.replace('.npy', '')}_abnscores.png"
    plt.savefig

taking in input the frame-wise abnormal_scores, as obtained in

AnomalyCLIP/src/models/anomaly_clip_module.py

Line 473 in ffb0aa6

"abnormal_scores": abnormal_scores,

Alternatively, someone else successfully ran the inference of AnomalyCLIP by bypassing the hydra-based template. For details, please check issue #8.

If you want to perform inference on an arbitrary video outside the evaluated datasets, you could consider one of the two options above, adjusting the data loading files in order to handle the new video. However, take into consideration that the method is not meant for zero-shot anomaly detection, therefore using the set of labels and the normal centroid from one of the provided dataset may be sub-optimal.