Article | Selected papers from the CLARIN Annual Conference 2018, Pisa, 8-10 October 2018 | Human-human, human-machine communication: on the HuComTech multimodal corpus Linköping University Electronic Press Conference Proceedings
Göm menyn

Title:
Human-human, human-machine communication: on the HuComTech multimodal corpus
Author:
Laszlo Hunyadi: Department of General and Applied Linguistics, University of Debrecen, Debrecen, Hungary Tamás Várad: MTA Institute of Linguistics, Research Group on Language Technology, Budapest, Hungary György Kovács: MTA SzTE Reserach Group on Artificial Ingelligence, Szeged, Hungary / Embedded Internet Systems Lag, Luleå Univeristy of Technology, Luleå, Sweden István Szekrényes: Institute of Philosophy, University of Debrecen, Hungary Hermina Kiss: Department of General and Applied Linguistics, University of Debrecen, Debrecen, Hungary Karolina Takács: Department of Phonetics Eötvös Loránd University, Budapest, Hungary
Download:
Full text (pdf)
Year:
2018
Conference:
Selected papers from the CLARIN Annual Conference 2018, Pisa, 8-10 October 2018
Issue:
159
Article no.:
006
Pages:
56-65
No. of pages:
10
Publication type:
Abstract and Fulltext
Published:
2019-05-28
ISBN:
978-91-7685-034-3
Series:
Linköping Electronic Conference Proceedings
ISSN (print):
1650-3686
ISSN (online):
1650-3740
Publisher:
Linköping University Electronic Press, Linköpings universitet


Export in BibTex, RIS or text

The present paper describes HuComTech, a multimodal corpus featuring over 50 hours of video taped interviews with 112 informants. The interviews were carried out in a lab equipped with multiple cameras and microphones able to record posture, hand gestures, facial expressions, gaze etc. as well as the acoustic and linguistic features of what was said. As a result of large-scale manual and semi-automatic annotation, the HuComTech corpus offers a rich dataset on 47 annotation levels. The paper presents the objectives, the workflow, the annotation work, focusing on two aspects in particular i.e. time alignment made with the Leipzig tool WEBMaus and the automatic detection of intonation contours developed by the HuComTech team. Early exploitation of the corpus included analysis of hidden patterns with the use of sophisticated multivariate analysis of temporal relations within the data points. The HuComTech corpus is one of the flagship language resources available through the HunCLARIN repository.

Keywords: Multimodality, Multimodal corpus, Hidden patterns of communication, Human-machine communication

Selected papers from the CLARIN Annual Conference 2018, Pisa, 8-10 October 2018

Author:
Laszlo Hunyadi, Tamás Várad, György Kovács, István Szekrényes, Hermina Kiss, Karolina Takács
Title:
Human-human, human-machine communication: on the HuComTech multimodal corpus
References:

[Boersma &Weenink, 2016] Boersma, D., Paul & Weenink. 2016. Praat : doing phonetics by computer [computer program]. version 6.0.22. http://www.praat.org/. (retrieved 15 November 2016)

[Beck & Russel, 2006] Berck, P. and Russel, A. 2006. ANNEX – a web-based Framework for Exploiting Annotated Media Resources. In: Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006). Genoa: European Language Resources Association, 2006.

[Cho et al., 2014] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014

[Glorot et al] Glorot, X., Bordes, A., Bengio, Y. 2011. Deep Sparse Rectifier Neural Networks. In: Gordon, G. J., Dunson, D., B. Dudík, M. (eds): AISTATS JMLR Proceedings 15. JMLR.org. 315-323.

[Hochreiter and Schmidhuber, 1997] S. Hochreiter and J. Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (November 1997), 1735-1780.

[Holz and Teresniak, 2010] Holz and Teresnai, 2010: F. Holz and S. Teresniak, “Towards automatic detection and tracking of topic change”, in Proc. CICLing, 2010, pp. 327-339

[Hunyadi 2017] Hunyadi, L. 2017. A multimodális kommunikáció grammatikája felé: szekvenciális események rekurzív hierarchikus struktúrája. In: Bánréti, Z. (ed.) Általános Nyelvészeti Tanulmányok XXIX (2017), pp. 155-182.

[Kovacs et al] Kovács, G., Grósz, T., Váradi, T. 2016. Topical unit classification using deep neural nets and probabilistic sampling. In: Proc. CogInfoCom, (pp. 199–204)

[Kovács et al. 2017] Kovács, Gy., Váradi, T. 2017. A különbözo modalitások hozzájárulásának vizsgálata a témairányítás eseteinek osztályozásához a HuComTech korpuszon, in: Vincze, Veronika (ed.) XIII. Magyar Számítógépes Nyelvészeti Konferencia (MSZNY2017) Szeged, Szegedi Tudományegyetem Informatikai Tanszékcsoport, (2017) pp. 193-204. , 12 p.

[Magnusson, 2000] Magnusson, M. S. 2000. Discovering hidden time patterns in behavior: Tpatterns and their detection behaviour research methods. Behavior Research Methods, Instruments, & Computers, 32:93–110.

[Mertens, 2004] Mertens, P. 2004. The prosogram : Semi-automatic transcription of prosody based on a tonal perception model. In Proceedings of speech prosody.

[Pápay et al, 2011] Pápay, K., Szeghalmy, S., and Szekrényes, I. 2011. Hucomtech multimodal corpus annotation. Argumentum 7:330–347.

[Rosenberg, 2012] A. Rosenberg, “Classifying skewed data: Importance weighting to optimize average recall” in Proc. Interspeech, 2012, pp. 2242-2245

[Shriberg et al., 2000] E. Shriberg, A. Stolcke. D. Hakkani-Tür, G. Tür, “Prosody-based automatic segmentation of speech into sentences and topics”, Speech Commun. Vol 32, no. 1-2, pp 127-154, 2000

[Szekrényes 2014] Szekrényes, I. 2014. Annotation and interpretation of prosodic data in the hucomtech corpus for multimodal user interfaces. Journal on Multimodal User Interfaces 8:(2):143–150.

[Tóth and Kocsor, 2005] L. Tóth and A. Kocsor, “Training HMM/ANN” hybrid speech recognizers by probabilistic sampling Wittenburg et al 2006] Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. 2006. Elan : a professional framework for multimodality research. In Proceedings of LREC 2006 (pp. 213–269)

[Zellers and Post, 2009] M. Zellers, B. Post, “Fundamental frequency and other prosodic cues to topic structure”, in Workshop on the Discourse-Prosody Interface, 2009. Pp. 377-386

[Zsibrita et al] Zsibrita, János; Vincze, Veronika; Farkas, Richárd 2013: magyarlanc: A Toolkit for Morphological and Dependency Parsing of Hungarian. In: Proceedings of RANLP 2013, pp. 763-771.

Selected papers from the CLARIN Annual Conference 2018, Pisa, 8-10 October 2018

Author:
Laszlo Hunyadi, Tamás Várad, György Kovács, István Szekrényes, Hermina Kiss, Karolina Takács
Title:
Human-human, human-machine communication: on the HuComTech multimodal corpus
Note: the following are taken directly from CrossRef
Citations:
No citations available at the moment


Responsible for this page: Peter Berkesand
Last updated: 2019-06-04