Data Products

SHARE THIS:

Data collected during a dive is delivered to the science party via collaboration between the Alvin operations team and Atlantis’s Shipboard Scientific Support Group (SSSG).  Alvin data handling is independent of the Rolling Deck to Repository (R2R) program, which collects Atlantis underway data.

Unless otherwise stated below, Alvin data is delivered as originally collected during the dive, without post-dive treatment beyond organization into a cruise data package.  Alvin operations are not funded for data post-processing.  If you require post-dive or post-cruise treatment of data, or have data related questions, please contact Alvin Manager Bruce Strickrott (strickrott@whoi.edu) or the National Deep Submergence Facility (NDSF) data manager Scott McCue (smccue@whoi.edu).

Video

Video from all cameras on Alvin is either native 1080i/60 or is upconverted to that standard (1920x1080 pixels). The two science observers select which video streams are routed to the two in-sphere hard drive recorders.

At the time of recording the sources are compressed by the Apple ProRes422LT codec and are captured to computer file (.mov) as the raw delivered product. A proxy product (50% scaling, h.264 codec, 5% filesize) is created in post-processing to provide a version for viewing on personal devices. To view these proxies, which are not playable by Apple Quicktime Player or Windows Media Player, users should install an open source player such as VLC or mplayer on their personal computers. Atlantis’s SSSG technicians offer a VLC installer.

Video clip file sizes are determined by the in-sphere observers and their camera selection behavior. The choice to view and record uninterrupted from one source can lead to very large files (>100GB), and this must be accommodated in your transport and long-term storage planning.

Still Images

Alvin’s primary still image source is the externally-mounted SubC 1Cam Alpha, which captures images on a timed interval (default 30 secs) throughout the dive.  Resolution of these images is 6544 x 3680 x 8 bit and file output is .jpg.

The observers and pilot are provided with digital still cameras for both in-sphere and through viewport documentation.

The Framegrabber logging system captures 1920 x 1080 .jpg stills from the currently-viewed video stream on a timed interval.

Navigation

Alvin’s primary navigation sensors are a Doppler velocity log (DVL) paired with a fiber optic gyroscope (FOG). These and other navigation sensors are logged by the in-sphere navigation computer to a single hourly ASCII text file.  This aggregate file is parsed by sensor type during the data offload process and files for individual sensor types are created. Among these is a simple .csv file that can be viewed and processed using programs like a text editor, Matlab or Excel.

At the conclusion of every dive, Alvin-specific navigational and sensor data is transferred from the submersible to the support ship’s network for further processing. Technicians on board the ship execute a dedicated post-processing script that performs an initial ‘grooming’ of the various data and generates an output file that merges navigation and sensor data. Real-time position estimates are improved as a result of the post-processing. Maps and data reports can then be generated using the post-processed data.

The basic characteristics of the post-processed file (.scc) are as follows:

  • 1Hz ASCII file containing post-processed navigation data and a selection of other sensor data collected during the dive
  • Typical data:  date, time, latitude, longitude, depth, pressure, heading, and altitude. Other sensor data that may be made available includes magnetometer data, CTD data (conductivity, temperature, depth) and data from other sensors that may be resident on the submersible.
  • SCC file names contain both the dive number and the date of the dive

Complementary navigation datasets are logged by Alvin’s internal navigation system and equipment in Atlantis’s Top Lab.  Georeferenced information is usually produced by the Sonardyne Ranger ultra-short baseline (USBL) system, which is based in Top Lab. The cruise package includes the records from both of these logging stations.

More detailed mapping requirements associated with specialized equipment (e.g. multibeam sonar, etc.) may require processing expertise that is not routinely provided by the Alvin team. Specific needs for additional technical assistance should be communicated as part of the pre-cruise planning process.

Additional Sensors

When possible, all sensor records are produced and collected in text format, organized hourly. Alvin’s Command and Control computer collects sensor records into a single large file, which is delivered with the data package. This aggregate file is parsed by sensor type during the data offload process, creating files for individual sensor types.

Data Availability on Atlantis

Raw video and sensor data will be retrieved from the submersible and made available on the ship within six hours of a dive.  Some portions of the video package may require more time to transfer.  Video will be staged on an independently networked computer cluster, i.e., with no connections to other ship’s networks.  Video products may be accessed via the work area in Atlantis’s Main Lab, which offers Apple Mac Pro workstations for viewing, copying, and editing video.  The default setting is to provide non-passworded access to all video products.  If this is not suitable to the investigator, please raise the issue in pre-cruise discussions.  All other data types will be available from servers on Atlantis’s main network.

The Apple Mac Pro workstations provided on the ship offer Firewire800 and USB 3.0 connections. They do not offer Thunderbolt.  Users may also download via Ethernet.

Data Delivery

Alvin will produce significant volumes of data, primarily due to its video products. Volume estimates are 700GB of video and 20GB of non-video data per dive. All data will be presented on external USB hard drives, which are loaned by Alvin operations to the chief scientist so that she/he can carry the data products back to a home institution. Video data will be separate from the other data types, and will be on drives formatted with the ExFAT filesystem. You are asked to promptly return all loaned drives to WHOI after copying content.

A copy of all cruise data will also be transferred to WHOI’s Data Library and Archives. This copy will be transported and stored on standard LTO (Generation 5) archival tape media.

When your data package is delivered at the end of the cruise you will be asked to review a form that describes the package inventory and provides details for returning the drives to WHOI. You will be asked to acknowledge delivery of the package. Because NDSF does not have a default posture on data protection, you will also be asked to assign embargo durations on the various data components. You will be asked to name institutions for whom acknowledgement is due when cruise data is used for outreach or commercial purposes. Please remember that while all video and still image data is yours for academic use, UNOLS/DeSSC policy is that copyright for all imagery is retained by WHOI. For additional information please contact media@whoi.edu.

Media Recommendations

Questions often arise concerning compatibility between shipboard data systems and science party equipment.  Quite often these questions are about the file system used to format the transport disks given to the science party, e.g., FAT32, NTFS, HFS, ExFAT.  Another frequent question concerns the connection types offered on the shipboard equipment.

Alvin video drives are formatted with ExFAT because of its capabilities and compatibility with recent versions of MacOSX, linux, and Windows.  Unfortunately, ExFAT may be incompatible with older operating systems.  We recommend that users plan to bring drives (to be) formatted in ExFAT, and make sure your favorite operating system understands this format.