Article | Proceedings of the 10<sup>th</sup> International Modelica Conference; March 10-12; 2014; Lund; Sweden | recon -- Web and network friendly simulation data formats
Göm menyn

Title:
recon -- Web and network friendly simulation data formats
Author:
Michael Tiller: Xogeny Inc., USA Peter Harman: CyDesign Ltd., UK
DOI:
10.3384/ecp140961081
Download:
Full text (pdf)
Year:
2014
Conference:
Proceedings of the 10th International Modelica Conference; March 10-12; 2014; Lund; Sweden
Issue:
096
Article no.:
113
Pages:
1081-1093
No. of pages:
13
Publication type:
Abstract and Fulltext
Published:
2014-03-10
ISBN:
978-91-7519-380-9
Series:
Linköping Electronic Conference Proceedings
ISSN (print):
1650-3686
ISSN (online):
1650-3740
Publisher:
Linköping University Electronic Press; Linköpings universitet


Export in BibTex, RIS or text

There are many different commonly used file formats for storing time series data. Most of these file formats are designed with the assumption that the file itself will be locally available to the software that will be reading or writing the data stored in them. While this assumption is an excellent one for desktop based tools with direct access to disk drives capable of moving virtually instantaneously around from sector to sector; there are a growing number of applications for which local access is not necessarily available. For these applications; we’ve initiated the recon project to develop more suitable formats.

With the emergence of web and cloud based modeling and simulation technologies; the time has come to explore file formats specifically optimized for nondesktop applications. In this paper; we present a new set of file formats that are specifically designed for web and cloud based approaches. This paper reviews the key requirements for web and cloud enabled applications and then presents a specification for two file formats that address those requirements.

When considering the various use cases that drove our requirements; we recognized that two different file formats were actually required. The first format; the wall format; is optimized for writing. The other format; the meld format; is optimized for reading over a network (i.e.; minimizing the number of reads and bytes read). We will discuss the layout of each of these formats and describe the use cases for which they are most appropriate.

In the open tradition of the Modelica Association; the authors have made specifications and implementations for these formats available as open source libraries with the hope that they will benefit the community as a whole.

Keywords: Modelica; FMI; simulation results; cloud; web; open source

Proceedings of the 10th International Modelica Conference; March 10-12; 2014; Lund; Sweden

Author:
Michael Tiller, Peter Harman
Title:
recon -- Web and network friendly simulation data formats
DOI:
http://dx.doi.org/10.3384/ecp140961081
References:

[1] A. Pfeiffer, I. Bausch-Gall, and M. Otter. ‚ÄúProposal for a Standard Time Series File Format in HDF5‚ÄĚ. In: Proceedings of the 9th International Modelica Conference (2012). URL: http://www.bausch-gall.de/ecp12076495_ PfeifferBausch-GallOtter.pdf.


[2] J√∂rg R√§dler. ‚ÄúDyMat - HDF5 export and other Modelica/Python projects‚ÄĚ. In: (2013). URL: http://www.j-raedler.de/2013/01/dymat-hdf5-export-and-comparableprojects/.


[3] The HDF Group. HDF-Java 2.10 release. 2013. URL: http://www.hdfgroup.org/products/java/hdf-java-html/.


[4] Google. Google Protocol Buffers. 2012. URL: https://developers.google.com/protocol-buffers/.


[5] Apache Avro Project Members. Apache Avro 1.7.6. 2013. URL: https://avro.apache.org/.


[6] Apache Thrift Project Members. Apache Thrift 0.9.1. 2013. URL: https://thrift.apache.org/.


[7] Apache Hadoop Project Members. Apache Hadoop 2.2.0. 2013. URL: https://hadoop.apache.org/.


[8] Storm Development Team. STORM: Distributed and fault-tolerant realtime computation. 2013. URL: http://storm-project.net/.


[9] Sadayuki Furuhash. MessagePack - It’s like JSON but fast and small. 2013. URL: http://msgpack.org/.


[10] R. Fielding et al. Hypertext Transfer Protocol - HTTP/1.1. 1999. URL: http://www.w3.org/Protocols/rfc2616/rfc2616.html.


[11] JSON. JavaScript Object Notation. 2013. URL: http://www.json.org/.


[12] MongoDB Development Team. BSON Specification. 2013. URL: http://bsonspec.org.


[13] Inc. MongoDB. MongoDB Manual, Version 2.4. 2014. URL: http://docs.mongodb.org/manual/.


[14] Michael Tiller. recon ‚Äď Web and network friendly simulation data formats, Python Interface. 2013. URL: https://github.com/xogeny/recon.


[15] Jackson JSON processor project team. Smile Format Specification. 2013. URL: http://wiki.fasterxml.com/SmileFormatSpec.


[16] Pietrzak Roman. Binary-JSON (BJSON) format specification. 2013. URL: http://bjson.org/.


[17] Riyad Kalla. Universal Binary JSON Specification. 2013. URL: http://ubjson.org.


[18] Julian Seward. Bzip2 Homepage. 2010. URL: http://www.bzip.org/.


[19] Peter Harman. recon ‚Äď Web and network friendly simulation data formats, C Interface. 2014. URL: https://github.com/harmanpa/crecon.


[20] Peter Harman. jrecon ‚Äď Web and network friendly simulation data formats, Java Interface. 2014. URL: https://github.com/harmanpa/jrecon.

Proceedings of the 10th International Modelica Conference; March 10-12; 2014; Lund; Sweden

Author:
Michael Tiller, Peter Harman
Title:
recon -- Web and network friendly simulation data formats
DOI:
http://dx.doi.org/10.3384/ecp140961081
Note: the following are taken directly from CrossRef
Citations:
No citations available at the moment


Responsible for this page: Peter Berkesand
Last updated: 2017-02-21