Data Management & Communication

A coherent strategy that enables the integration of marine data streams across disciplines, institutions, time scales, and geographic regions is central to the success of IOOS and other regional, national, and international ocean and coastal observing systems. The system that must be developed, while challenging, is within the scope of current information technology (IT). It can be developed by building upon existing capabilities through relatively straightforward software engineering. The greatest challenge to enhancing marine data integration is one of coordination and cooperation among the members of IOOS and its user communities.[1]

 

IOOS DMAC Frequently Asked Questions: 

The IOOS DMAC Frequently Asked Questions guide developed by the U.S. IOOS Program Office addresses the following key Questions about the Data Management and Communications (DMAC) Implementation Plan (September 2011):

Frequently Asked Questions

  1. What is DMAC’s architecture/configuration?
  2. What is the IOOS Catalog?
  3. What are IOOS Data Assembly Centers?
  4. What is expected of the 3 types of DACs?
  5. Why do we need functional DACs when we have Federal DACs and Regional Portals?
  6. What is the Global Telecommunications System (GTS), who uses it and how do I get data?
  7. Do we distribute IOOS data in other ways?
  8. If RA data is going through service, what else needs to be done?
  9. Has the IOOS PO mandated that all products should be required to be served via a WMS service?
  10. How are IOOS data quality controlled?
  11. How does DMAC connect to data.gov?
  12. What is data “brokering” and how will it be implemented by IOOS DMAC?
  13. What is IOOS DMAC’s relationship to Federal ocean observing programs?
  14. What is US IOOS DMAC’s relationship to other ocean observing data management efforts?
  15. How can I publish my data in DMAC?
  16. What are DMAC Services?
  17. How do we archive data in IOOS?
  18. How do vocabularies intersect with DMAC?

 

hkjhlj

hkjhlj

1. What is DMAC’s architecture/configuration?

Back to Top

DMAC uses a Federated Architecture with Distributed Implementation.  A federated architecture integrates data, services and products from multiple autonomous contributing systems and presents them as a larger integrated system. We do this by using data exchange standards to deliver a unified outward appearance, enabling user access to a wide range of content with minimal effort.   Participating organizations’ underlying architectures can remain autonomous, although changes to metadata may be needed to allow users to make full use of the data.

Distributed implementation maximizes the use of existing components, procedures, and resources to create the core of the system and only adds elements as needed to create the appearance of a unified outward appearance. While distributed implementation is less expensive than traditional system development approaches (e.g. centralized data warehouse), it relies heavily on the willing support of the owners of the contributing systems.

Another term applied to this approach is “service-oriented architecture” (SOA).  “Service,” in this context, is a way for two computers to communicate over the internet.  It allows computers to ask questions and receive answers from other computers. Service-oriented architectures use metadata to process messages between systems.   This allows easy data exchange between programs without making changes to the participating systems. For this reason, metadata management is a key factor in the success of IOOS DMAC.

The DMAC architecture has three related functional components that follow a “Register-Discover-Use” model:

  • Register data or services are used by data providers to publish information about their data and data access services to a directory (such as the IOOS DMAC Catalog) so that it can be discovered by interested users.  Registration typically uses metadata to describe the capabilities and network address (location) of the data or data access services.
  • Discover data or services are used by clients (users) to find specific data and data services.  Users “discover” the kinds of data services they’re looking for by querying the directory.
  • Use data or services are used when a client uses a data access service, based on the metadata provided by the directory, to obtain the desired data from data provider.

DMAC functional Diagram

While the functional description above outlines “what DMAC does”, a deployment architecture description outlines “where DMAC is”.   For example, organizations will be responsible for installing and maintaining the hardware and software components that deliver the functions described above.  The following sections will provide more detail on the functionality of DMAC components and who will deploy them.

hkjhlj

hkjhlj

2. What is the IOOS Catalog?

Back to Top

The IOOS Catalog is a collection of web based services that address three data discovery functions (Service Registry,Data Catalog Service and System Viewer) stated in the U.S. IOOS® “A Blueprint for Full Capability” (Version 1.0) November 2010.

The Service Registry provides the master list of all IOOS data sets available via DMAC web services. Users can search for specific data sets or browse the data holdings of IOOS data providers.   Users access the registry manually through a web based user interface or through software clients using defined web service interfaces. Currently the Open Geospatial Consortium (OGC) Catalog Service for the Web is the primary web service interface to the IOOS Catalog.  Every data set published through a DMAC service must be registered in the Service Registry.

The Data Catalog provides a more detailed view of the data sets that are available via the web services registered in the Service Registry.  Information on observing platforms, variables, and space and time bounds is extracted from the data services and summarized to provide different views of the information collected and disseminated by IOOS. The results are displayed in either map or tabular form, and include links to the actual data and metadata. The data itself is not served by the Data Catalog; the catalog refers the user back to the original source of data at a Data Assembly Center (DAC).

The System Viewer provides a map based view of the data set information present in the data catalog.  This viewer enables users to browse the data holdings of IOOS data providers and visualize the breadth of the IOOS enterprise.

The IOOS Catalog, comprised of these three components, is being developed incrementally and does not yet deliver all of the features described here.  The current version can be found at http://catalog.ioos.us/ and the next release is scheduled for mid-2014.  The existing System Viewer on the IOOS Website (http://www.ioos.noaa.gov/catalog/welcome.html) is still valid to use as a graphical representation of the IOOS catalog in presentations, but should not be scrutinized heavily.  It will be replaced in 2014 by the next generation Catalog.

hkjhlj

hkjhlj

3. What are IOOS Data Assembly Centers?

Back to Top

IOOS Data Assembly Centers (DACs) are computer facilities that aggregate and distribute ocean, coastal and Great Lakes (hereafter marine) data derived from in situ observations, remote observations, or computer models.

A DAC is where:

  1. Marine data and relevant metadata are collected and assembled into accessible files and commonly used formats
  2. Quality control is applied
  3. DMAC discovery and access services are applied.

Customers interact with DACs through web-based data services and other tools or applications that enable discovery, access and use of the data via human to machine or machine to machine interaction. DACs can be deployed via locally operated computer servers or in a cloud configuration.

IOOS has three types of DACs, Federal, Regional and Functional.  The two Federal DACs that contribute to IOOS are National Oceanic and Atmospheric Administration’s (NOAA) National Data Buoy Center (NDBC) and Center for Operational Oceanographic Products and Services (CO-OPS).  The 11 IOOS Regional Associations (RAs) operate the regional DACs.  The U.S. IOOS Program Office supports three functional DACs that provide a single type of marine observing data.  These are: ocean surface currents from high frequency radar (HF Radar), ocean profiling gliders (glider) and animal telemetry (tags).

hkjhlj

hkjhlj

4. What is expected of the 3 types of DACs?

Back to Top

Federal DAC: Provide data according to its agency’s mission and collaborate with IOOS to influence the standards and interfaces Federal agencies use.  Within NOAA, CO-OPS and NDBC are using the IOOS standards.  Integration of other NOAA programs/offices and Federal data providers outside of NOAA is still a goal but is challenging and progress is slow.

Regional DAC:  Provide data assembly, quality control, discovery and access services for marine data collected by State, Local, Tribal governments, academia and industry.  Inclusion of an observing asset in the Regional DAC is not limited to assets funded through the IOOS RAs cooperative agreements.  The Regional DAC should serve up as much of the non-Federal assets in their Region as feasible and the IOOS RAs should actively seek to include those data. Re-serve Federal data as a service if there is a stakeholder requirement. All DMAC services in use and all the data sets they publish must be registered in the Service Registry.

Functional DAC:

  • HF Radar DAC: Consists of developmental servers at Scripps which also serves as the backup operational server, operational servers at NDBC, and a third installation at Rutgers for fail-over security. NDBC is the public access site (sdf.ndbc.noaa.gov/thredds/dodsC/hfradar/)
  • Glider DAC: In 2013 the U.S. IOOS Program Office established a prototype national glider DAC,   developed by MARACOOS as a central access point for this rapidly emerging technology. The glider DAC’s initial rapid deployment was possible because IOOS has focused on data standards, the large number of glider missions flown by IOOS and academic partners, and leveraging of other platforms data standards, e.g. ARGO.  (http://gliders.ioos.us/)
  • Animal Telemetry Network (ATN) DAC:  In 2013 the U.S. IOOS Program Office began work on an ATN DAC at Stanford University for data from marine animal tags.  The prototype will be demonstrated in 2014.

Such specialized DACs simplify the consistent and centralized application of standards and quality control for observing technologies that may be widely used by investigators but the resultant data are not necessarily widely available. NDBC’s Mission Control Center represents a multi-purpose DAC that publishes mission-critical ocean observations for use by NOAA and many other organizations. NDBC is also where regional IOOS data are processed and quality controlled before release in real-time to the global science community via the Global Telecommunications System (GTS).

hkjhlj

hkjhlj

5. Why do we need functional DACs when we have Federal DACs and Regional Portals?

Back to Top

There are both technical and programmatic reasons for creating the three functional DACs we have.    The U.S. IOOS Program Office does not plan to add any additional functional DACs.

HF Radar DAC:  Work on HF Radar DAC occurred simultaneously with the standup of the IOOS RAs and was augmented by the California Currents Ocean Mapping Program and earmark dollars in NOAA.  NOAA’s funding focused on ensuring a consistent quality control and transition of this data to operations.  It has been beneficial to have this data distributed nationally from NDBC, so this unique observing platform can be seen by NOAA as demonstrating national impact.  A single data distribution point is preferable to ease integration of HF Radar data into NOAA’s oil spill modeling, systems for the weather forecast offices, and for United States Coast Guard’s (USGS) search and rescue.  The servers at Scripps and Rutgers are back up servers to NDBC.  Beginning in 2014 the HF Radar DAC will work with National Oceanographic Data Center (NODC) to archive radar data.

Glider DAC:  The decision to create a glider DAC follows rationale similar to that for the HF Radar DAC and other observing system centric DACs such as the Argo DAC (Coriolis, US GODAE Server at NRL Monterey, CA), and the OceanSITES DAC (Coriolis, NDBC).  In each case the observing systems are supported by numerous organizations working collaboratively but the primary product is an integrated data set available at a single access point.  This single access provides a cohesive view to NOAA and potentially other funding agencies.  The single access point illustrates that the national glider fleet is a capable and mature observing system and that the expertise to operate the fleet lies in the IOOS RAs.  Finally, the technical development of an integrated glider data system was more efficiently implemented by a small team than could be accomplished across 11 different teams.  The existence of a national DAC doesn’t obviate the need for data management activities at the regional level.  The glider DAC is only an access point; most processing and derivation of useful data sets is done by the IOOS RAs.

ATN:  From a policy perspective, addressing the need to integrate data from tagged animals  is a decision that the U.S. IOOS Program Office made based on: (1) our mandate to integrate data from physical, biological and chemical domains, (2) a growing recognition that data from tagged animals is important to oceanographic models (both Navy and NOAA are interested in the data) and marine ecology, and (3) the IOOS Summit recommendation to  increase the amount of biological data available to IOOS users.

IOOS initiated our collaboration with the animal telemetry community in 2011 with a workshop at the National Marine Fisheries Service (NMFS) Lab in Santa Cruz, California to discuss and understand community needs. A second workshop was held in 2012. Both meetings were conducted in partnership with NMFS, Office of Naval Research (ONR) and Bureau of Ocean Energy Management (BOEM). The U.S. IOOS Program Office is responding to three outcomes of those workshops: (1) progress towards standards for animal telemetry data via projects completed in FY11-12 with Naval Oceanographic Office (NAVO), NOAA National Center for Environmental Prediction (NCEP) and Ocean Tracking Network (OTN); (2) improved infrastructure for organizing and distributing data from tags via development of an ATN DAC and (3) clarifying broad goals and objectives for animal telemetry by writing a strategic plan.  The strategic plan will also satisfy a National Ocean Policy milestone.

The U.S. IOOS Program Office has not, nor plans to buy tags for marine animals or fish, though some IOOS Regional Associations support tagging based on regional needs.  NMFS funds tagging efforts through their Regional Science Centers but does not plan to move these efforts into a national observing system.  NAVO purchased tags in FY13 and FY14 to tag elephant seals and sharks in the Pacific.  NAVO is assimilating data from these tags for their Pacific models.  Planned FY14 U.S. IOOS Program Office support for the ATN DAC will be matched with ONR funds to support DAC enhancements.

From a technical perspective the ATN DAC was set up at the Stanford facility because they have the most advanced capability in data management practices for tagging data through their experience with the Tagging of Pacific Predators (TOPP) program which began in 2000 as one of 17 projects of the Census of Marine Life. Additionally, the IOOS ATN DAC is building on investments made by NOAA’s Office of Response and Restoration (OR&R) to Stanford under the National Resources Damage Assessment (NRDA) following the Deepwater Horizon oil spill.

Finally, the IOOS ATN DAC will provide international linkages.  For example, Zdenka Willis (Director, U.S. IOOS Program Office) is an international advisor to the Ocean Tracking Network (OTN).  OTN is primarily focused on acoustic tags and is working with or has worked with GLOS’ Great Lakes Animal Telemetry Observing System and NANOOS.  OTN was part of Gliderpalooza 2013 and through that involvement has opened up new relationships with US scientists.  OTN has developed a DAC for the acoustic information and we will continue to discuss with them the best way for IOOS to collaborate.

hkjhlj

hkjhlj

6. What is the Global Telecommunications System (GTS), who uses it and how do I get data?

Back to Top

The World Meteorological Organization (WMO) administers the GTS. The GTS is a coordinated global system of telecommunication facilities configured for the rapid collection, exchange and distribution of information within the World Weather Watch framework.  Only WMO designated “National Meteorological Hydrological Services” (NMHS) may post data to the GTS.

In the United States, only the National Weather Service and the Navy are designated to post to the GTS.    For IOOS RA data, NDBC provides marine data to the GTS.  Currently this marine data includes temperature, salinity, currents, wind speed/direction, waves, and other primarily physical observations.  The ARGO community is working to distribute oxygen, fluorometry, and optical measurements and NCEP is eager to begin receiving this type of data.

The primary purpose of the GTS is to support weather forecasting, therefore, the modeling and prediction community are the primary users of GTS data.  The GTS is not accessible to the public, so there is a need for alternative ways for others to get the data.  There are projects that serve select portions of data sets originally circulated on the GTS.  The NOAA Observing System Monitoring Center (http://osmc.noaa.gov/erddap/index.html) has begun making select GTS observations available through an ERDDAP interface and John Wilken (MARACOOS) has successfully integrated this data access service into his operational data assimilation modeling workflow.

hkjhlj

hkjhlj

7. Do we distribute IOOS data in other ways?

Back to Top

IOOS’ key principle is to make data available from the physical, chemical, biological disciplines regardless of who collected the data.  We want the data to be easy to find, easy to access and in a format that makes datasets easy to use together to solve a range of problems.

To achieve this IOOS uses open web service standards to distribute data.    The web services IOOS uses are:

  • Web services supporting the Open-source Project for a Network Data Access Protocol (OPeNDAP) such as the THREDDS Data Server (Thematic Real-time Environmental Data Distributed Services), Hyrax, and ERDDAP (the Environmental Research Divisions Data Access Program).
  • the Sensor Observation Service (SOS) and Web Map Services (WMS) which were standardized by the OGC

Use of web services alone is not sufficient to achieve the level of interoperability IOOS requires; these web services must be configured to output specific file formats adhering to specific information models and conventions.  See http://www.ioos.noaa.gov/data/contribute_data.html for more information

OPeNDAP based services originated in communities generating and serving large volumes of gridded data (model output, satellite grids etc).  These services, along with related data model and format conventions (netCDF, Climate and Forecast Conventions, Attribute Conventions for Dataset Discovery) have greatly enhanced the interoperability of these data types.  Other services, like the Web Map Service have emerged to fill different niches, in this case, to serve images of data as opposed to the data itself.  All gridded data must, at a minimum, be served by an OPeNDAP compliant web service and an OGC Web Map Service.

The SOS standard has evolved to address data collected by sensor systems such as buoys, gliders, tide gauges among others.  These in situ data types present different challenges to gridded data services.  During the past several years the grid centric focus of OPeNDAP services has lessened and they have evolved into viable standards for serving in situ data in addition to gridded data.  Still, IOOS maintains a commitment to SOS as a primary method for serving in situ data because of some unique features SOS provides.  SOS, for example, offers a way to load a database via web services in addition to just querying a database.  Similarly, the information models that SOS is built on are more naturally aligned with the information users need to know about sensors.  The metadata model also allows for more information on the processing of data (i.e. QC).  All in situ data must be served by an SOS service.

hkjhlj

hkjhlj

8. If RA data is going through service, what else needs to be done?

Back to Top

In short, nothing.  If the service is standards compliant, well-populated with metadata, and is registered with the IOOS Service Registry, then the datasets published via the service should be discoverable via the Service Registry search mechanisms and should be visible on the IOOS Catalog.  As part of routing operations and maintenance of the network the U.S. IOOS Program Office will monitor the services for reliability and availability and will assess the data sets for standards compliance.  The data services hosted by the data assembly centers will evolve over time, for example by including enhanced metadata, and the U.S. IOOS  Program Office will work with the DACs to incorporate change rationally.

hkjhlj

hkjhlj

9. Has the IOOS PO mandated that all products should be required to be served via a WMS service?

Back to Top

Not all data is appropriate for WMS, currently.  Sensor and other in situ data types don’t easily lend themselves to visualizations via a Web Map Service.  However, WMS is an excellent tool for visualizing all types of gridded or raster data sets and any data sets served by a regional association via an OPeNDAP service should also be served via a Web Map Service.

hkjhlj

hkjhlj

10. How are IOOS data quality controlled?

Back to Top

For data to be valuable to users, it must be valid. To meet this need we must provide sound practices for quality assurance (QA) and quality control (QC). Quality Assurance employs a process, typically focused on hardware and sampling protocols, to support the generation of high-quality data that are accurate, precise, and reliable when combined with manufacturer recommended procedures such as sensor calibration, in-situ verification, proper deployment, and adequate maintenance intervals. Quality Control employs automation and human intervention to support delivery of high quality data. QC practices include such things as formats, checksum, timely arrival of data, threshold checks (minimum/maximum rate of change), neighbor checks, climatology checks, model comparisons, signal/noise ratios, verification of user satisfaction, generation of data flags and other standard tests widely used by the ocean community.  Quality control is typically, but not exclusively, performed by data providers with expertise in a particular instrument or sampling protocol.  IOOS DACs then document the quality control procedures and publish QC results along with the data.

We publish our authoritative procedures for quality control of real-time ocean sensor data through the Quality Assurance of Real-Time Oceanographic Data (QARTOD) project. The long-term objective is to publish manuals for all 26 of the IOOS core variables. As of Feb 2014, four QARTOD manuals have been published. The QARTOD Board of Advisors meets quarterly to discuss topics for future manuals.  The next manual will address quality control standards for ocean optic variables.

IOOS recently published procedures for non-Federal organizations to become certified IOOS data providers. Adoption of the published QARTOD procedures is required for that certification when they exist.  In cases when they do not exist, the RA seeking certification will document the quality procedures that were applied.  All data served by an RA must have undergone some level of quality control.  See the IOOS website for more information on certification (http://www.ioos.noaa.gov/certification/welcome.html).

For more information about QARTOD go to: http://www.ioos.noaa.gov/qartod/welcome.html.

hkjhlj

hkjhlj

11. How does DMAC connect to data.gov?

Back to Top

Because IOOS is implementing discoverable web services, we will be a source for data.gov. It is not clear yet if IOOS data will be fed directly to data.gov or through an emerging NOAA service registry.  In either case, the program office will ensure publication in data.gov for each regional association that uses DMAC standards and registers their services in the IOOS Catalog.

Data.gov was established in 2009 to simplify discovery of the vast data resources of the Federal government.   The initial prototype for an ocean component of data.gov was released in 2011 as one of 13 “communities.” This effort was coordinated by the National Ocean Council with input from IOOS (Josie Quintrell, Ru Morrison, Eoin Howlett) and others. Further coordination with IOOS in 2012 included a briefing on ocean.data.gov to the Regional Associations by NOAA Coastal Services Center’s Tony Lavoi at a Regional Data Managers workshop in Charleston, SC attended by DMAC representatives from AOOS, SECOORA and PacIOOS.

Since 2009, data.gov has gone through several iterations of development addressing look and feel, discoverability, and access. This process, led by the National Ocean Council, continues to evolve and new data and descriptions are being added daily. For example, the ocean topic is evolving from what is currently a mostly geospatial catalog for coastal marine spatial planning, to a more diverse set of offerings including ocean observations from government and non-government sectors. This effort will be dependent on resources and time available from federal agencies and the incremental deployment of discoverable web offerings such as the IOOS data access services deployed in the regional associations.

hkjhlj

hkjhlj

12. What is data “brokering” and how will it be implemented by IOOS DMAC?

Back to Top

Data brokering is a potentially transformative technology that can increase interoperability by centralizing the access point and interface to disparate data without centralizing the data.

A broker is a service that sits between data providers and customers, improving the information exchange by mediating or reconciling commonalities based on the metadata descriptions. The improvements may be in efficiency, number of services available, or standardization of custom data streams.  Brokering is attractive because it places no additional burdens on providers or customers and can be implemented by third parties. The National Science Foundation has made progress in developing brokering through their Earth Cube project (http://earthcube.org/).

IOOS uses brokering approaches for both data access and data discovery services.  For example, services such as ERDDAP and the THREDDS Data Server can read data from a wide range of different services and file formats and translate them into common data models which enable access through IOOS-approved data services.   The IOOS Catalog uses Geoportal Server as its broker to read metadata from different web services and translate them into a common data model which then enables powerful and rapid search and discovery of data sets.

In summer 2014, we will initiate an SOS brokering project to evaluate centralizing some DMAC data access services, especially those of Federal data providers. The purpose will be to develop efficiencies that can lead to centralized access to select Federal data thereby reducing the burden on IOOS RAs to serve Federal data in an IOOS compliant manner. This project will be done by Axiom and will not require active support from IOOS DACs.

hkjhlj

hkjhlj

13. What is IOOS DMAC’s relationship to Federal ocean observing programs?

Back to Top

In accordance with the ICOOS Act, the U.S. IOOS Program Office has the responsibility to coordinate Federal capabilities into IOOS and lead the Regional component.

With respect to DMAC, Federal ocean observing programs fill the roles of:

  • Data provider. An entity that operates a DAC or data archive for ocean observations that is IOOS DMAC compliant and supplies data required by users for operational, applied, or research purposes.
  • Data or services customer. An entity that accesses data or data products through IOOS DMAC services.

The U.S. IOOS Program Office, within given resources, is working with the larger Federal DACs to use the IOOS data services.

  • NDBC and CO-OPS use IOOS services to provide their data.
  • NMFS:  Working towards serving their data through the IOOS DMAC services.  This effort is complicated by the fact that NMFS’ data holdings are at individual science centers and they are evaluating the feasibility of various possible NMFS-wide data standards.  In the interim, if IOOS Regions and NMFS science centers find it mutually beneficial for the IOOS Region to serve up the NMFS data, the U.S.IOOS Program Office recommends individual arrangements be made until a national solution can be adopted.
  • United States Geological Service’s (USGS) Ocean Biogeographic Information System (OBIS)-USA: Working to enhance marine biodiversity data standards with the intent of publishing their data with DMAC services and in the IOOS catalog.

IOOS DMAC enables ocean observing programs to publish data via a consistent set of tools and standards.  As more programs use DMAC, the power of a consistent, distributed network of data providers grows.  Federal programs adopting DMAC can increase the reach of their data and the value of taxpayer’s investment.  Further, by becoming DMAC users, Federal programs become more efficient customers of other IOOS data thereby increasing their access to data at potentially lower cost.

hkjhlj

hkjhlj

14. What is US IOOS DMAC’s relationship to other ocean observing data management efforts?

Back to Top

IOOS DMAC collaborates closely with many national and international data management partners. A partial list includes:

  • National Oceanic and Atmospheric Administration (NOAA)
  • Environmental Data Management Committee (EDMC): IOOS is subject to the procedural directives published by the NOAA EDMC.  Voting members of the EDMC are selected from each line office and while the U.S. IOOS Program Office is not a voting member, our views are represented by the National Ocean Service representative. The U.S. IOOS Program Office participates in drafting of the Procedural Directives (PD).   The Regional Association certification criteria are aligned with EDMC procedural directives and the EDM Framework is a guiding policy document.   The existing EDM technical and policy directives include a Data Management Planning PD, a Procedure for Scientific Records Appraisal and Archive Approval, a Data Documentation PD, and a Data Sharing PD for
    • NOAA Grants. Several other directives are in preparation. All materials are accessible at: www.nosc.noaa.gov/EDMC/index.php.
    • NOAA Data Management Integration Team &Unified Access Framework (NOAA DMIT & UAF): US IOOS Program Office is represented in the DMIT by Derrick Snowden.  This is NOAA’s community of practice for data management and by participating we influence how the UAF is being developed to meet IOOS needs.  For example, IOOS adopted UAF solutions for creating virtual THREDDS aggregations for the IOOS regions.
  • Other Federal:
    • Interagency Ocean Observation Committee and its DMAC Steering Team, a cross sector forum for DMAC planning and execution.
  • Non-Government Organizations
    • Open Geospatial Consortium (OGC): IOOS is a user of OGC standards and participates in the community dialog to evolve these standards.  An OGC employee participates with the IOOC’s DMAC Steering Team as one of many invited stakeholders to help communicate emerging standards and technologies and identify opportunities for collaboration.
    • Unidata: Unidata sponsors the development of software on which we depend. Specifically the java-netCDF library and the THREDDS Data Server are two of the foundational software packages DMAC is built upon.  Several IOOS DMAC developers have leadership roles within Unidata technical forums.
  • International
    • Global Earth Observation System of Systems (GEOSS): IOOS is registered in GEOSS Common Infrastructure (Discovery and Access Broker) which provides another way to discover IOOS resources using tools designed and deployed by an international community.  The GEOSS DAB is similar to the IOOS Service Registry.
    • Australia’s Integrated Marine Observing System (IMOS):  IOOS and IMOS are very similar systems. We are increasing our collaboration with them to leverage their DMAC expertise.
    • European Union’s Infrastructure for Spatial Information in Europe (EU INSPIRE): U.S. IOOS Program Office maintains awareness of activities and regularly exchanges ideas. IOOS seeks collaboration for operational data dissemination and provides input to their efforts when requested.
    •  The World Meteorological Organization’s Joint Technical Commission for Oceanography and Marine Meteorology (JCOMM): U.S. IOOS Program Office maintains awareness of activities and regularly exchanges ideas. IOOS seeks collaboration for operational data dissemination and provides input to their efforts when requested.

hkjhlj

hkjhlj

15. How can I publish my data in DMAC?

Back to Top

Information on participating in DMAC as a data provider can be found at: http://www.ioos.noaa.gov/data/contribute_data.html

hkjhlj

hkjhlj

16. What are DMAC Services?

Back to Top

“Services,” as used in this context, refers to computer software that allows computers to communicate over the internet.  This communication takes the form of questions and answers and is transparent to the computer user.  “DMAC Services” refers to the specific set of standards and services that have been approved for IOOS use.

The use of common standards and services creates an environment where marine data can be discover, accessed and used in uniform ways.  This simplifies getting data in support of modeling and decision making.

IOOS has selected three standards:

  • Open-source Project for a Network Data Access Protocol (OPeNDAP)

o   This standard can used  with services (i.e. software) such as the THREDDS Data Server (Thematic Real-time Environmental Data Distributed Services), Hyrax, and ERDDAP (the Environmental Research Divisions Data Access Program)

o   OPeNDAP based services are useful for serving large volumes of gridded data (model output, satellite grids, etc.)

o   These services, enhanced with data models and format conventions (e.g. netCDF, Climate and Forecast Conventions, Attribute Conventions for Dataset Discovery) have greatly enhanced the interoperability of these data types.

o   NetCDF files is one of the most common ways to get data into or from an OPeNDAP complaint server

  • Sensor Observation Service (SOS) OGC (OpenGeospatial Consortium) Standard

o   The SOS standard addresses in situ data types such as those collected by buoys, gliders, tide gauges, etc.

o   SOS is the primary IOOS method for serving in situ data because of the unique features SOS provides; such as a way to load a database via web services in addition to querying the database

o   The SOS metadata model also allows for more information on the processing of data such as Quality Control information.

o   52N is software that implements the SOS standard

o   ncSOS is another software based on the SOS standard that acts like a bridge that takes in situ data stored in OPeNDAP compliant files and delivers it as SOS compliant data

  • Web Map Services (WMS) OGC (OpenGeospatial Consortium) Standard

o   Web Map Service emerged to fill the need to serve images of data as opposed to the data itself

o   IOOS current efforts are focused on OPeNDAP and SOS standards based service implementations.  While WMS will certainly have a place in IOOS’s future, significant work lays ahead on how this will be implemented.

The use of web services alone is not sufficient to achieve the level of interoperability IOOS requires; these web services must be configured to output specific file formats adhering to specific information models and conventions.  See http://www.ioos.noaa.gov/data/contribute_data.html for more information.

 

17. How do we archive data in IOOS?

Back to Top

It is an ongoing goal for IOOS to ensure the long term preservation of regional ocean observations at a federal archive facility.  The NOAA NODC is the archive facility for the majority of IOOS regional association data.  For certain data sets, NODC may recommend using the Geophysical (NGDC) or Climatic (NCDC) Data Centers but in either case, the process of archiving data is similar.  The archiving process begins with a formal agreement between a data provider (typically a regional association) and the archive (NODC).  The details of this agreement are documented in a Submission Information Form (SIF).  Once the SIF is complete, the development of the technical procedures to perform the automated archiving begins.  These procedures sometimes involve reformatting data, augmenting metadata, documenting controlled vocabularies, and setting up synchronization steps.  The U.S. IOOS Program Office is working with NODC to ensure that the data formats published by DMAC services are acceptable to NODC and as a result we will minimize the reformatting step.  Even so, we expect some effort will be required of DMAC staff in the regions to archive data.

 

18. How do vocabularies intersect with DMAC?

Back to Top

Controlled vocabularies enable interoperability because they illuminate the meaning of common concepts that are used in the machine readable files that are shared between data systems.  The U.S. IOOS Program Office with help from the IOOS RAs has created several controlled vocabularies or adopted others.  The vocabularies encompass terms to support observed properties (also known as parameters, variables or phenomena), data descriptor, organization definitions, etc.  Although they primarily target SOS service requests, the terms have general applicability in other services and IOOS DMAC areas as well.

All controlled vocabularies and vocabulary mappings developed or adopted by IOOS (except for theDarwin Core biological terms) are currently hosted on the [http://mmisw.org/orr/#b Marine Metadata Interoperability (MMI) Ontology Registry and Repository (ORR)].  The advantage of using the MMI ORR to the data manager is that the repository makes a machine referenceable URL for terms that can be used in the IOOS Web service templates.  The referenceable URL is a link to enabling machine-to-machine interoperability between data systems.

 

 


[1] National Office for Integrated and Sustained Ocean Observations, An Integrated and Sustained Ocean Observing System (IOOS) for the United States: Design and Implementation, Ocean.US Publication, May 2002

 

QR Code Business Card