Lesson 5: Obsolescence and Sustainability

5.1 Defining Obsolescence and Sustainability

Technology changes quickly. With that change, certain previously preferred file formats and media will stop being used in favor of other, more advantageous ones—this is called obsolescence.

This is a concern when working with research data because we know that, as researchers, our current methods for creating, storing, and sharing data could potentially become obsolete and be replaced by new formats.

Photo of a stack of black computer floppy disks.
Floppy disks, an outdated storage option

To help thwart obsolescence as best we can, we want to try to ensure that we pick the most sustainable options for our data over time. Sustainability in this case means formats or media that are less susceptible to changes in technology. Below you’ll find some tips for this.

Best practices for reducing the impact of changing technology:

In general, pick file formats that:

  • Are non-proprietary: the format is not owned by a company or manufacturer. If a file can only be opened in a certain software, it’s likely that it’s a proprietary format. The .DOCX file type you may be familiar with, produced by Microsoft Word, is a proprietary format. Common proprietary formats often have new versions and software upgrades required to use the files. Over time, new versions stop accounting for the ability to open old versions of that format, and you can lose file information or the ability to read the file entirely.
  • Have seen wide adoption in your discipline: Sometimes there isn’t a non-proprietary option for you to use and that’s okay. If a file format is widely used in your field, it’s likely that it will continued to be supported and openable into the future.
  • Have a history of backward compatibility: new versions of the software are still able to support old file versions.

Physical devices and media:

Physical devices and media have a lifespan. While devices we use for storing data have improved exponentially over the last twenty years, expect external hardware devices to have a lifespan of 3 to 5 years. Plan on migrating your files every few years if you use an external hard drive to prevent data loss!


5.2 File Formats

Type of Data Recommended Formats Acceptable Formats
Tabular data with extensive metadata
variable labels, code labels, and defined missing values
SPSS portable format (.por)
delimited text and command (‘setup’) file (SPSS, Stata, SAS, etc.)
structured text or mark-up file of metadata information, e.g. DDI XML file
proprietary formats of statistical packages: SPSS (.sav), Stata (.dta), MS Access (.mdb/.accdb)
Tabular data with minimal metadata
column headings, variable names
comma-separated values (.csv)
tab-delimited file (.tab)
delimited text with SQL data definition statements
delimited text (.txt) with characters not present in data used as delimiters
widely-used formats: MS Excel (.xls/.xlsx), MS Access (.mdb/.accdb), dBase (.dbf), OpenDocument Spreadsheet (.ods)
Geospatial data
vector and raster data
ESRI Shapefile (.shp, .shx, .dbf, .prj, .sbx, .sbn optional)
geo-referenced TIFF (.tif, .tfw)
CAD data (.dwg)
tabular GIS attribute data
Geography Markup Language (.gml)
ESRI Geodatabase format (.mdb)
MapInfo Interchange Format (.mif) for vector data
Keyhole Mark-up Language (.kml)
Adobe Illustrator (.ai), CAD data (.dxf or .svg)
binary formats of GIS and CAD packages
Textual data Rich Text Format (.rtf)
plain text, ASCII (.txt)
eXtensible Mark-up Language (.xml) text according to an appropriate Document Type Definition (DTD) or schema
Hypertext Mark-up Language (.html)
widely-used formats: MS Word (.doc/.docx)
some software-specific formats: NUD*IST, NVivo and ATLAS.ti
Image data TIFF 6.0 uncompressed (.tif) JPEG (.jpeg, .jpg, .jp2) if original created in this format
GIF (.gif)
TIFF other versions (.tif, .tiff)
RAW image format (.raw)
Photoshop files (.psd)
BMP (.bmp)
PNG (.png)
Adobe Portable Document Format (PDF/A, PDF) (.pdf)
Audio data Free Lossless Audio Codec (FLAC) (.flac) MPEG-1 Audio Layer 3 (.mp3) if original created in this format
Audio Interchange File Format (.aif)
Waveform Audio Format (.wav)
Video data MPEG-4 (.mp4)
OGG video (.ogv, .ogg)
motion JPEG 2000 (.mj2)
AVCHD video (.avchd)
Documentation and scripts Rich Text Format (.rtf)
PDF/UA, PDF/A or PDF (.pdf)
XHTML or HTML (.xhtml, .htm)
OpenDocument Text (.odt)
plain text (.txt)
widely-used formats: MS Word (.doc/.docx), MS Excel (.xls/.xlsx)
XML marked-up text (.xml) according to an appropriate DTD or schema, e.g. XHMTL 1.0
UK Data Service [1]

Check Your Understanding

Open, non-proprietary file formats are in less danger from obsolescence. Which of these four file formats would you use for the following types of research data?

  • .csv
  • .flac
  • .txt
  • .tif

Images

Correct! .tif or TIFF is a more sustainable format for images. Some image formats can compress the image and lose some of the image information. The TIFF format helps avoid that.
Incorrect. The correct answer is D) .tif (or TIFF), which is a more sustainable format for images. Some image formats can compress the image and lose some of the image information. The TIFF format helps avoid that.

Spreadsheets

Correct! .csv or Comma Separated Value is a plain text, more sustainable format for spreadsheets or tabular data. Similarly to the text formats mentioned above, other tabular data formats like .xlsx from Excel are proprietary and rely on specific software to read them.
Incorrect. The correct answer is A) .csv, or Comma Separated Value, which is is a plain text, more sustainable format for spreadsheets or tabular data. Similarly to the text formats mentioned above, other tabular data formats like .xlsx from Excel are proprietary and rely on specific software to read them.

Text

Correct! .txt or plain text is much more sustainable for text based files. Other textual file formats like .docx from Microsoft Word include extra proprietary information and rely on specific software to read them.
Incorrect. The correct answer is C) .txt, or plain text, which is much more sustainable for text based files. Other textual file formats like .docx from Microsoft Word include extra proprietary information and rely on specific software to read them.

Audio

Correct! .flac or Free Lossless Audio Codec is a more sustainable format for audio files, though you may often see .wav and .mp3 also recommended.
Incorrect. The correct answer is B) .flac, or Free Lossless Audio Codec, which is a more sustainable format for audio files. You may often see .wav and .mp3 also recommended.

References

[1] UK Data Service, “Recommended formats,” University of Essex, University of Manchester and Jisc. Retrieved from https://www.ukdataservice.ac.uk/manage-data/format/recommended-formats.