The Distant TV Toolkit is a Python package designed to facilitate the computational analysis of visual culture. It contains low-level architecture for applying state-of-the-art computer vision algorithms to still and moving images. The higher-level functionality of the toolkit allows users to quickly extract semantic metadata from digitized collections. Extracted information can be visualized for search and discovery as well as aggregated and analyzed to find patterns across a corpus.
Both the high-level command-line interface and the low-level object-oriented API are introduced below. More information about the toolkit and project is available on the following pages:
- Search and discovery interface example: DVT Video Visualization
- Example analysis using aggregated metadata: "Visual Style in Two Network Era Sitcoms"
- Theory of the project: "Distant Viewing: Analyzing Large Visual Corpora."
- Project homepage: The Distant Viewing Lab
- Software Whitepaper: A Python Package for the Analysis of Visual Culture
- Documentation
- PyPI repository
If you have any trouble using the toolkit, please open a GitHub issue. If you have additional questions or are interested in collaborating, please contact us at [email protected] and [email protected].
The Distant Viewing Toolkit is supported by the National Endowment for the Humanities through a Digital Humanities Advancement Grant.
The distant viewing toolkit requires Python 3.7. It is primarily built and tested using Anaconda Python, which we suggest installing from here: Anaconda Distribution. The package can then be installed through PyPI:
pip install dvt
Additional Python requirements should be installed automatically through PyPI. It should also be possible to build from a clean virtual environment under Python 3.7.
The following code assumes that you have installed the dvt toolkit and have the video file video-clip.mp4 in your working directory. Run the following command to run the default pipeline of annotators from dvt:
python3 -m dvt video-viz video-clip.mp4
This may take several minutes to complete. Some minimal logging information
should display the annotators progress in your terminal. Once finished,
you should have a new directory dvt-output-data
that contains extracted
metadata and frames from the source material. You can view the extracted
information by starting a local http server:
python3 -m http.server --directory dvt-output-data
And opening the following: http://0.0.0.0:8000/.
You can repeat the same process with your own video inputs, though keep in mind that it may take some time (often several times the length of the input video file) to finish. You can see an example of the toolkit's output on several video files here.
The command line tools provide a fast way to get started with the toolkit, and there is much more functionality available when using the full Python API provided by the module. The following code assumes that you have installed the dvt toolkit and have the video file video-clip.mp4, audio file video-clip.wav, and subtitle file video-clip.srt in your working directory.
Using the distant viewing toolkit starts by constructing a DataExtraction
object that is associated with input data (either a video file or a
collection of still images). Algorithms are then applied to the
DataExtraction
; the results are stored as Pandas DataFrames and can be
exported as CSV or JSON files. There are two distinct types of algorithms:
- annotators: algorithms that work directly with the visual data source but are able to only work with a small subset of frames or still images
- aggregators: algorithms that have access to information extracted from previously run annotators across across the entire input, but cannot direclty access the visual data
The separation of algorithms into these two parts makes it easier to write straightforward, error-free code. It closely mirrors our theory of distant viewing:
Distant viewing is distinguished from other approaches by making explicit the interpretive nature of extracting semantic metadata from images. In other words, one must 'view' visual materials before studying them. Viewing, which we define as an interpretive action taken by either a person or a model, is necessitated by the way in which information is transferred in visual materials. Therefore, in order to view images computationally, a representation of elements contained within the visual material—a code system in semiotics or, similarly, a metadata schema in informatics—must be constructed. Algorithms capable of automatically converting raw images into the established representation are then needed to apply the approach at scale.
The annotator algorithms conduct the process of 'viewing' the material whereas the aggregator algorithms perform a 'distant' (e.g., separated from the raw materials) analysis of the visual inputs.
Here is an example showing the usage of these elements to detect shot breaks in a video input. We start by running an annotator that detects the differences between subsequent shots and then apply the cut aggregator to determine where the changes indicate a pattern consistent with a shot break. As in the Minimal Demo, the code assumes that the video file video-clip.mp4 is in your working directory:
from dvt.core import DataExtraction, FrameInput
from dvt.annotate.diff import DiffAnnotator
from dvt.aggregate.cut import CutAggregator
dextra = DataExtraction(FrameInput(input_path="video-clip.mp4"))
dextra.run_annotators([DiffAnnotator(quantiles=[40])])
dextra.run_aggregator(CutAggregator(cut_vals={'q40': 3}))
Looking at the output data, we see that there are four detected shots in the video file:
dextra.get_data()['cut']
And its output:
frame_start frame_end
0 0 74
1 75 154
2 155 299
3 300 511
There are many annotators and aggregators currently available in the toolkit. Pipelines as well as pre-bundled sequences of annotators and aggregators are also included in the package. Currently available implementations in the toolkit are:
Annotators | Aggregators | Pipelines |
---|---|---|
CIElabAnnotator | CutAggregator | ImageVizPipeline |
DiffAnnotator | DisplayAggregator | VideoCsvPipeline |
EmbedAnnotator | ShotLengthAggregator | VideoVizPipeline |
FaceAnnotator | PeopleAggregator | |
HOFMAnnotator | ||
ObjectAnnotator | ||
OpticalFlowAnnotator | ||
PngAnnotator | ||
AudioAnnotator* | ||
SubtitleAnnotator* |
Details of these implementations can be found in the full API documentation. Additionally, it is possible to construct your own Annotator and Aggregator objects. Details are available in this tutorial. If you develop an object that you think may be useful to others, consider contributing your code to the toolkit.
(*) Special Annotators: The audio and subtitle annotators have a special format because they require additional inputs (metadata about the video file and access to the raw audio and subtitle data, respectively). To use these annotators, specify the location of the audio and/or subtitle inputs when creating the DataExtraction object. Next, run the corresponding DataExtraction methods, as follows:
from dvt.core import DataExtraction, FrameInput
dextra = DataExtraction(
vinput=FrameInput(input_path="video-clip.mp4"),
ainput="video-clip.wav",
sinput="video-clip.srt"
)
dextra.run_audio_annotator()
dextra.run_subtitle_annotator()
The audio data will be stored in an annotation named "audio":
dextra.get_data()['audio'].head()
data data_left data_right
0 -1108 -1113 -1103
1 -1546 -1552 -1539
2 -1123 -1121 -1125
3 -1007 -1008 -1006
4 -1258 -1262 -1253
And subtitle information will be stored in an annotation named "subtitle":
dextra.get_data()['subtitle']
time_start time_stop caption \
0 0.585 2.588 You guy's are messing with me, right?
1 5.145 7.100 Yeah
2 8.224 9.789 That was a good one. For a second there, I was...
frame_start frame_stop
0 17 78
1 154 213
2 246 294
Aggregators can be written that further analyze the audio and subtitle data. These aggregators are written no differently than aggregators that work with just the video data; in fact, we can build aggregators that put together audio, textual, and visual information. The pipeline does include several common audio-based annotators. For example, the PowerToneAggregator calculates the RMS of the audio power across pre-defined blocks of the audio source (indexed by frame):
from dvt.aggregate.audio import PowerToneAggregator
dextra.run_aggregator(PowerToneAggregator(breaks=[0, 50, 200, 220]))
dextra.get_data()['power']
frame_start frame_end rms
0 0 50 917.916747
1 50 200 1208.046528
2 200 220 997.259411
If you make use of the toolkit in your work, please cite the relevant papers describing the tool and its application to the study of visual culture:
@article{,
title = "Distant Viewing: Analyzing Large Visual Corpora",
author = "Arnold, Taylor B and Tilton, Lauren",
journal = "Digital Scholarship in the Humanities",
year = "2019",
doi = "10.1093/digitalsh/fqz013",
url = "http://dx.doi.org/10.1093/digitalsh/fqz013"
}
@article{,
title = "Visual Style in Two Network Era Sitcoms",
author = "Arnold, Taylor B and Tilton, Lauren and Berke, Annie",
journal = "Cultural Analytics",
year = "2019",
doi = "10.22148/16.043",
url = "http://dx.doi.org/10.22148/16.043"
}
The distant viewing toolkit includes a full test suite, based on the python modules pytest. To run the tests clone the repository and run the following from a terminal. These commands install pytest and other optional dependencies, run the pytest unit tests, and display the coverage statistics:
pip install -U pytest
pip install .\[tests,optional\]
cd tests; pytest --disable-warnings --cov=dvt --cov-report term-missing .
To properly test the entire toolkit, we need to load and run several configurations of large neural network] models. Therefore, testing can take upwards of 20 minutes to complete.
Contributions, including bug fixes and new features, to the toolkit are welcome. When contributing to this repository, please first discuss the change you wish to make via a GitHub issue or email with the maintainers of this repository before making a change. Small bug fixes can be given directly as pull requests.
Please note that the project has a code of conduct. Contributors are expected to follow the guidelines.