Finding shared moments in concert clips through audio.

1. The problem

Fans record footage that artists may want to use. An artist who wants that footage faces two separate problems: getting permission to use it, and making sense of it once they have it.

Timestamp metadata has several limitations. Consumer device clocks drift on the order of tens of milliseconds per minute [13], capture time is frequently stripped or rewritten in the path from camera roll to social platform, and the granularity that survives is coarser than the thing being measured. Two clips stamped the same minute may be ninety seconds apart; two clips stamped differently may overlap exactly.

Everyone in the venue recorded the same sound source. If a clip’s audio can be located inside a reference recording of the same show, that clip has a position on the show’s timeline, and comparing positions becomes arithmetic. Acoustic self-localisation also underpins early work on organising crowd-sourced concert video [9][10].

2. System architecture

CrowdClip is a NestJS API over PostgreSQL, with media in secure storage and the signal processing isolated in a Python child process. The pipeline saves its progress at each stage, so interrupted work can resume from its saved state.

Trust boundaries

• CrowdClip follows Meta’s platform rules.
• CrowdClip gives the fan a secure upload link that expires. It does not store the working link in a database.

3. Alignment method

CrowdClip follows Concert Stitch [1], which addresses this exact problem: organising and synchronising crowd-sourced recordings. Concert Stitch in turn builds on the Haitsma–Kalker fingerprint [2].

3.1 Fingerprinting

Audio is downmixed to mono and resampled to 5 kHz, then framed and reduced to 32 bits per frame. The parameters are taken directly from the paper:

Each bit is the sign of the energy difference taken along the frequency axis and then along the time axis, which is what makes the representation survive a phone microphone in a loud room: it encodes changes in energy, so overall level, EQ, and compression largely fall out.

The bits are stored as ±1 floats. This means an FFT cross-correlation of two fingerprint sequences directly counts matching bits. Normalised similarity is 1 − the bit error rate, and the entire offset search uses a single transform.

3.2 Affine scale search

The paper’s main pipeline runs subsequence dynamic time warping over the fingerprint distance matrix and trains an SVM to pick the true overlap path, using cross-correlation only for sample-accurate refinement. CrowdClip uses affine scale search to account for clock-rate error in smartphone recordings.

DTW is there to absorb playback speed that varies within a recording, specifically the analog wow-and-flutter of the tape-sourced Grateful Dead collection evaluated in the paper. In smartphone recordings, the dominant distortion is a small constant clock-rate error, represented by a single scalar. The analyser searches time-scale factors from 0.92 to 1.08 in steps of 0.001, resampling the query fingerprint sequence at each scale and cross-correlating against the reference. The best (scale, offset) pair gives the estimated alignment.

3.3 Confidence statistics

The paper’s cross-correlation baseline accepts a match below a fixed 0.35 bit error rate. Its own authors flag this threshold as fragile on unseen data. CrowdClip’s analyser reports the shape of the correlation peak and lets the caller decide:

Acceptance uses configurable thresholds. Defaults live in the process contract: similarity ≥ 0.59, peak_margin ≥ 0.02, and a time scale inside 0.95–1.05. A caller can tighten or relax them per catalogue without touching the DSP. Gating on margin as well as absolute similarity is the standard false-match suppressor in the fingerprinting literature [4], and the implausible-scale rejection catches sped-up re-uploads that would otherwise align at incorrect offsets.

3.4 Grouping

Two clips are the same moment when they matched the same reference track in the same catalogue and their reference offsets fall within a five-second window. Everything else, including a different catalogue, a different track, an unmatched analysis, an offset outside the window, returns an explicit reason to help tune thresholds when a group looks wrong.

4. Evaluation

Two evaluations exist. The first was a feasibility spike: clips cut from one audience recording of a 1977 show, placed against a different recording of the same performance. It is the figure below, playable with the audio used in the test.

The second is the one the thresholds come from. Using a 2019 concert from the Live Music Archive, the matrix source served as the reference and a separate audience source supplied the queries: eight fifteen-second clips, plus eight clips taken from a different song in the same show as deliberate false candidates.

All eight true matches resolved to the same constant offset within a 145 ms spread. Consistency across independent clips supports the alignment estimates and can be checked without ground-truth annotation. The minimum true-match margin was 0.066, and the maximum false-candidate margin was 0.002. The similarity gap was about 0.09. These results support using margin to distinguish matches.

Degradation behaviour

Fans singing along are one source of degradation, especially when someone close to the phone is louder than the PA. I simulated that by mixing a detuned, 150 ms-late copy of the song over the clip. Similarity falls to 0.64 when the singer is as loud as the music, and 0.59 when the singer is 6 dB louder. Both still recover the exact offset. This costs slightly more similarity than uncorrelated noise at equal loudness because correlated interference flips fingerprint bits in a biased direction.

Under ±3% clock drift plus 10 dB SNR noise, the correct offset is still recovered while raw similarity falls below the 0.59 floor. The peak_z_score values are (5–6 for true matches against roughly 4.7 for unrelated audio). If real uploads prove noisier than archive audience recordings, z-score could provide an additional acceptance criterion.

5. Use cases

• Cleared recap edits. Use original, full-quality fan footage of the show, with permission recorded per clip.
• Multi-angle stitching. Align footage of the same moment from several vantage points so editors can choose between angles.
• Moment ranking. Overlap density is itself a popularity signal. The parts of the show the most people chose to film are the parts worth cutting to [9].
• Audio repair. With clips aligned, the best-sounding source in a group can supply the audio bed for the best-looking one.
• Setlist reconstruction. Matched reference tracks plus offsets produce a timestamped record of what was played and when, derived from fan recordings.

6. References

1. V. Subramanian and A. Lerch. Concert Stitch: Organization and Synchronization of Crowd-Sourced Recordings. ISMIR, Paris, 2018. The method CrowdClip implements.
2. J. Haitsma and T. Kalker. A Highly Robust Audio Fingerprinting System. ISMIR, 2002. The fingerprint itself.
3. A. Wang. An Industrial-Strength Audio Search Algorithm. ISMIR, 2003. The Shazam algorithm; assumes the same signal, so it does not cross from studio to live.
4. J. Six and M. Leman. Panako: A Scalable Acoustic Fingerprinting System Handling Time-Scale and Pitch Modification. ISMIR, 2014. Closest open-source analogue to this output tuple.
5. R. Sonnleitner and G. Widmer. Quad-based Audio Fingerprinting Robust to Time and Frequency Scaling. DAFx, 2014.
6. S. Chang et al. Neural Audio Fingerprint for High-Specific Audio Retrieval Based on Contrastive Learning. ICASSP, 2021. A method to evaluate if peak-based matching fails on crowd noise.
7. Z. Rafii, B. Coover and J. Han. An Audio Fingerprinting System for Live Version Identification. ICASSP, 2014.
8. T. Tsai, T. Prätzlich and M. Müller. Known-Artist Live Song Identification Using Audio Hashprints. ISMIR, 2016. The reference for the studio-catalogue regime, where constraining search to a known artist is the highest-leverage move.
9. L. Kennedy and M. Naaman. Less Talk, More Rock: Automated Organization of Community-Contributed Collections of Concert Videos. WWW, 2009. The founding paper for this problem.
10. C. Cotton and D. Ellis. Audio Fingerprinting to Identify Multiple Videos of an Event. ICASSP, 2010. On matching degraded recordings against each other.
11. S. Bano and A. Cavallaro. Discovery and Organization of Multi-Camera User-Generated Videos of the Same Event. Information Sciences, 2015. The closest published system to this whole flow.
12. J. Liang et al. Temporal Localization of Audio Events for Conflict Monitoring in Social Media. AAAI, 2017. Pairwise sync followed by a global consistency solve.
13. M. Guggenberger, M. Lux and L. Böszörmenyi. A Synchronization Ground Truth for the Jiku Mobile Video Dataset. MMM, 2015. Ground truth for this exact task, plus the companion device clock-drift study.