Bill Introduced in Minnesota to Require Use of "Open Data Formats"

The inevitable is happening more and more...

Wednesday, April 05 2006 @ 06:46 AM EDT
Views: 1,315
OpenDocument

I received an email yesterday pointing me to a bill, introduced on March 27, that would require all Executive branch agencies in the state of Minnesota to "use open standards in situations where the other requirements of a project do not make it technically impossible to do this." The text of the bill is focused specifically on "open data formats," and would amend the existing statute that establishes the authority of the Office of Enterprise Technology (OET), and the duties of the states Chief Information Officer. While the amendment does not refer to open source software, the definition of "open standards" that it contains would be conducive to open source implementations of open standards. The text of the affected sections of Minnesota Statutes Chapter 16E, showing the amendments proposed, can be found here.

The fact that such a bill has been introduced is significant in a number of respects. First, the debate over open formats will now be ongoing in two U.S. states rather than one. Second, if the bill is successful, the Minnesota CIO will be required to enforce a law requiring the use of open formats, rather than be forced to justify his or her authority to do so. Third, the size of the market share that can be won (or lost) depending upon a vendor's compliance with open standards will increase. And finally, if two states successfully adopt and implement open data format policies, other states will be more inclined to follow.


The amendment, and the statutory framework in which it exists, provides an interesting contrast to the open standards policy adopted by the Massachusetts Information Technology Division (ITD) in many respects. For example, it appears from the existing text of the statute that the Minnesota OET already has clear authority to mandate the use of open data formats, reading in part as follows:

When state agencies have need for the same or similar public data, the chief information officer, in coordination with the affected agencies, shall manage the most efficient and cost-effective method of producing and storing data for or sharing data between those agencies. The development of this information architecture must include the establishment of standards and guidelines to be followed by state agencies. The office shall ensure compliance with the architecture.

Second, the proposed amendment contains an extremely detailed (and rather eclectic) definition of "open standards," including not only the traditional concepts of availability to all on reasonable and nondiscriminatory ("RAND") terms, but a good deal more besides. For example, it requires that all permitted standards must permit royalty free implementation, and also includes a number of requirements that are far more detailed than would normally be found in the description of an open standard, but which might be agreed upon as necessary by the members of a working group developing a standard. The following is one example taken from the proposed definition of an "open standard" found in the amendment: "[An open standard] is documented, so that anyone can write software that can read and interpret the complete semantics of any data file stored in the data format." (I've included the full text of the proposed definition at the end of this blog entry.)

In addition to the very restrictive definition of open standards and open data formats, the amendment is also intolerant of making exceptions, providing:

For a particular project involving the access, storage, or transfer of data, a restricted data format may be chosen when satisfaction of essential project requirements precludes the use of an open data format. Neither the current storage format of previously collected data, nor current utilization of specific software products, is a sufficient reason, in absence of other specific overriding functional requirements, to use a restricted format;

Moreover, the amendment would require periodic review of all "existing data stored in a restricted format, to which the state of Minnesota does not own the rights, every four years to determine if the format has become open and, if not, whether an appropriate open standard exists;" The amendment, if enacted, would therefore impose a very tight collar on what types of software could be purchased and used.

On its face, the amendment is vendor neutral. It does, however, include one provision that may have been directed at Microsoft, which has at times been criticized for adding proprietary extensions to otherwise standards-compliant product features. That provision is found in the definition of an "open standard," and requires that if a standard, "allows extensions, ensures that all extensions of the data format are themselves documented and have the other characteristics of an open data format;"

There are a number of other interesting points that I note in reading the amendment at the Minnesota site, one being that the state legislation portal is set up to provide data on a bill's sponsors and current status - something that is sadly lacking in Massachusetts. There also appear to be a number of differences between the responsibilities and authority of the CIO in Minnesota, as compared to Massachusetts, that Peter Quinn might have appreciated. I may look more deeply into those differences and return to them in a future entry.

Curiously, I have been able to find out almost nothing on the Web about the amendment, indicating that thus far it has received little public notice. The news item submission (written by the submitter) that pointed me to the bill, however, reads in part as follows:

A consortium of Minnesota businesses and citizens has moved to put forward legislation that promises to assist the State in overcoming the negative effects of time, innovation and the market. Referred to as the Open Data Formats Bill, House File 3971 defines the means by which Minnesota could take control over how it stores information so as to not be bound, technically or legally, to anything other than its own technical objectives.

The Bill is an attempt to counter the fact that searches on the State website display information that is primarily stored in formats owned by vendors... that could go out of business, get acquired, or turn into the next Enron. It is also a response to what some refer to as the 'eight track tape effect' where information is stored in a way that is both popular and looks permanent, but then is quickly replaced by newer technologies. In some instances the information has been lost forever.

The Bill is not biased towards any one technology and advances a policy where at all times and in all instances the State has the ability and legal right to review, fix or improve the information it uses to conduct business. The Bill is not expected to increase State spending on technology. It is, nonetheless, expected to receive stiff opposition despite improving competition for State contracts, enhancing the ability of Minnesotans to access State services and data, and improving communication between State systems.

I will keep an eye on this new bill and report further as additional information becomes available. For now, however, it is significant to note that the debate over open data formats has now begun in a second state. It will be interesting to watch how the forces align, and the discussion becomes focused as the process moves forward.

The full text of the two new statute sections appears below.

For further blog entries on ODF, click here.

Language to be added to Minnesota Statutes 2005 Supplement, section 16E.03, subdivision 1:

(f) "Open standards" means specifications for the encoding and transfer of computer data that:

(1) is free for all to implement and use in perpetuity, with no royalty or fee;

(2) has no restrictions on the use of data stored in the format;

(3) has no restrictions on the creation of software that stores, transmits, receives, or accesses data codified in such way;

(4) has a specification available for all to read, in a human-readable format, written in commonly accepted technical language;

(5) is documented, so that anyone can write software that can read and interpret the complete semantics of any data file stored in the data format;

(6) if it allows extensions, ensures that all extensions of the data format are themselves documented and have the other characteristics of an open data format;

(7) allows any file written in that format to be identified as adhering or not adhering to the format;

(8) if it includes any use of encryption, provides that the encryption algorithm is usable on a royalty-free, nondiscriminatory manner in perpetuity, and is documented so that anyone in possession of the appropriate encryption key or keys is able to write 2.20 software to unencrypt the data.

(g) "Restricted format" means any data format that is accessed, stored, or transferred 2.22 and is not open standards compliant.

Language to be added to Minnesota Statutes 2005 Supplement, section 16E.04, subdivision 2:

(g) The office shall assist state agencies to avoid the purchase or creation of data processing devices or systems that do not comply with open standards for the accessing, storing, or transferring of data. The office shall:

(1) ensure any new data standards which the state of Minnesota defines and to which it owns all rights are open standards compliant;

(2) use open standards in situations where the other requirements of a project do not make it technically impossible to do this. For a particular project involving the access, storage, or transfer of data, a restricted data format may be chosen when satisfaction of essential project requirements precludes the use of an open data format. Neither the current storage format of previously collected data, nor current utilization of specific software products, is a sufficient reason, in absence of other specific overriding functional requirements, to use a restricted format;

(3) reexamine existing data stored in a restricted format, to which the state of Minnesota does not own the rights, every four years to determine if the format has become open and, if not, whether an appropriate open standard exists;

(4) make readily accessible, from a central location on the Internet, documentation on open data formats used by the state of Minnesota. When data in open format is made available through the state's Web site, a link shall be provided to the corresponding data format documentation.