ulmo

Ulmo is a python library for clean, simple and fast access to public hydrology and climatology data.

Installation

Ulmo depends on a lot of libraries from the scientific python stack (namely: numpy, pytables and pandas) and lxml. There are a couple of ways to get these dependencies installed but it can be tricky if doing it by hand. The simplest way to get things up and running is to use a scientific python distribution that will install everything together. A full list is available on the scipy website but Anaconda / Miniconda is recommended as it is the easiest to set up.

If you are using Anaconda/Miniconda then you can install ulmo from the conda-forge channel with the following command:

conda install -c conda-forge ulmo

Otherwise, follow the instructions below:

Once the requisite scientific python libraries are installed, the most recent release of ulmo can be installed from PyPI using pip:

pip install ulmo

To install the bleeding edge development version, grab a copy of the source code and run setup.py from the root directory:

To setup a development environment using conda:

conda env create -n myenv –file conda_environment.yml

# use ‘activate myenv’ on windows

source activate myenv

pip install -e .

Quick Start

Check out this brief but awesome tutorial by Nikolay Koldunov to get a sense of what Ulmo is about.

Getting help

If your stuck, have questions, ideas, or just want to say hi feel free to drop us a line on the mailing list.

Contributing

If you’d like to contribute code or documentation, visit our github repository. Bug reports, issues and especially pull Requests are welcome.

Changelog

0.8.8 (released 2021-09-02)

Patch release. Added python-dateutil dependency back to requirements.txt. It’s only used in the LCRA reader, which seems to be broken. But adding it just in case and to enable conda-forge package build tests to complete cleanly. It’s otherwise identical to 0.8.7. 0.8.7 PyPI package will be “yanked”.

0.8.7 (released 2021-09-01, but “yanked” from PyPI 2021-09-02)

  • This release focused primarily on two improvements:
    1. Added new NOAA GOES reader for GOES DSC/DCP data ingest. This new reader replaces the USGS EDDN reader, which is no longer functional (web service is inaccessible) and was removed in this release.
    2. Overhauled the dependencies (requirements files) and test and continuous integration (CI) infrastructure to enable support for more recent Python 3 versions (3.7, 3.8 and 3.9) and reestablish the CI tests. Previously only 3.6 was supported, due to dependency pinning that was no longer necessary.
  • ulmo readers
    • New NOAA GOES reader
    • Fixed GHCN Daily get_data compatibility bug with newer (1.1 and higher) Pandas versions
    • Update to https in documentation and test references to http://hydroportal.cuahsi.org
  • Builds, dependencies, tests and continuous integration
    • Removed broken Travis CI and AppVeyor integration, and replaced them with new GitHub Action
    • Split requirements.txt file into requirements.txt and requirements-dev.txt
    • Removed pytables dependencies that’s no longer used and removed all dependency version pinning
    • Removed hdf5 and pytables tests (no longer used)
    • Fixed some failing tests

0.8.6 (released 2020-08-20)

  • The largest change was to the ReadTheDocs documentation system. It had not been updated in a long time, and the auto-update CI was broken. Many sections were updated, the auto-build was fixed (now using a .readthedocs.yml file), a separate top-level section for “readers” (often referred to as “plugins”) was implemented, and many docstrings were updated and enhanced to ensure the formatting works well. Additional documentation was added for CUAHSI readers.

  • Bug fixes and small enhancements for multiple readers: cpc, cuahsi.his_central, cuahsi.wof, usgs.eddn, usgs.nwis

    • cuahsi.wof and cuahsi.his_central: added optional user_cache argument to service requests, to direct WSDL caching files to user app directory. Avoids problems with the user not having permission to write cache files to the system tmp space (the default)
    • usgs.eddn: generalized twdb_fts parser to accept batter_voltage and water_level data in any order
    • usgs.nwis:
      • get_site_data can now accept a list of parameter codes as an argument instead of a string of comma separated values
      • enabled retrieval of iv data prior to 2007
    • Fixed or updated several tests

0.8.5 (released 2019-03-22)

  • Bug fixes and some small enhancements for multiple plugins: CDEC, CPC, CUAHSI WaterML/WOF, NASA ORNL Daymet, NCDC GSOD and GHCN, USACE and USGS EDDN.
  • Removed USGS EROS plugin (service is no longer available)
  • LCRA plugins are experiencing problems. Removed from CI testing, until fixes are applied.
  • Pandas API updates and error handling
  • Continuous Integration and test infrastructure, including Python 3.6 and 3.7 builds

0.8.4 (released 2017-02-06)

  • added stevens dual well parser for eddn service
  • bugfixes for twc-kbdi service
  • optimize dict to dataframe conversion. This speeds up ncdc-ghcn by 10x

0.8.3.2 (released 2016-08-12)

  • Fixed USGS NED services. Server changed spatial query format.

0.8.3.1 (release 2016-07-18)

  • corrected install documentation

0.8.3 (released 2016-07-18)

  • added Nasa Daymet weather data module, thanks to ColinTalbert
  • multiple bug fixes
  • improvements to installation and continuous integration testing

0.8.2 (released 2016-02-02)

  • added modules for getting LCRA hydromet and water quality data
  • fixed waterml parser bug that was affecting USGS NWIS data when parameter data collected using multiple methods exist (e.g. water level measured with RADAR and pressure transducer) through use of optional ‘method’ kwarg
  • fixed ncdc.ghcn.get_stations() failing tests with pandas==0.17.0

0.8.1 (released 2015-09-16)

  • packaging fixes. 0.8.0 does not install correctly from pypi.

0.8.0 (released 2015-09-15)

  • python 3 compatibility. ulmo is now compatible py python 2.7 and python 3.4.
  • ulmo is now being tested on Windows/Linux/MacOSX using AppVeyor and Travis CI.
  • the ioos conda channel is now the supported way to get ulmo.
  • added conda environment yaml files to make it easier to set up a development environment.

0.7.8 (released 2015-07-02)

  • no changes, had to bump version number because of a corrupt file uploaded to pypi.

0.7.7 (released 2015-07-02)

  • dropped python 2.6 support
  • fixed bug in extracting raster file from zip file on windows
  • fixes for cdec service
  • fix url_params kwarg typo for nwis service

0.7.6 (released 2015-04-27)

  • fixed ncdc.gsod services to use new station list over depreciated list (thanks Victor)
  • added USGS National Elevation Dataset (NED) raster service
  • added USGS Earth Resources Observation Systems (EROS) raster services
  • allow passthrough of extra kwargs for USGS NWIS services

0.7.5 (released 2015-01-26)

  • add support for pandas 0.14 and 0.15
  • add a function to selectively remove data from nwis hdf5 cache

0.7.4 (released 2014-09-09)

  • in cuahsi.wof module: allow optional configuration of caching behavior for underlying suds SOAP library
  • in cuahsi.wof.get_values(), variable_code is no longer a keyword argument. It didn’t make sense for it to be a kwarg since it is not optional.

0.7.3 (released 2014-07-16)

  • bugfix: using parameters argument in ulmo.ncdc.gsod.get_data() was not working

0.7.2 (released 2014-06-17)

  • add parser for TWDB DOT dataloggers to usgs.eddn

0.7.1 (released 2014-06-05)

  • bugfix: in cpc.drought dataset calculate year number externally rather than using year number from data files, as they are sometimes incorrect (see github issue #66)

0.7.0 (released 2014-04-01)

  • this release includes some backwards incompatible changes
  • update ncdc.cirs to use new nClimDiv dataset (see ftp://ftp.ncdc.noaa.gov/pub/data/cirs/climdiv/div-dataset-transition-readme.txt) for details about the new dataset
  • some ncdc.cirs element names have changed:
    • cdd -> cddc
    • hdd -> hddc
    • pcp -> pcpn
    • tmp -> tmpc
  • fix flakey cpc.drought url determination causing data for the 2011 agricultural year to be unavailable

0.6.7 (released 2014-03-28)

  • fix usgs.cpc year/week number reckoning bug

0.6.6 (released 2014-03-24)

  • fix usgs.eddn parsers: values should be interpretted as reversed for sutron and texuni dataloggers

0.6.5 (released 2014-03-14)

  • usgs.eddn hdf5 files will no longer grow larger than they have to with each update
  • fix incorrect ghcn_daily wm_oid when using pandas 0.13.1
  • fix incorrect pytables required version

0.6.4 (released 2014-01-22)

  • a bunch of minor fixes for working with pandas 0.13
  • this release drops support for pandas 0.10.1; the minimum required version of pandas is now 0.11
  • usgs.eddn.get_data will now make multiple requests if it encounters a ‘Maximum data limit reached’ message

0.6.3 (released 2013-12-12)

  • add cdec.historical for accessing California Department of Water Resources California Data Exchange Center
  • fix last_modified dates being incorrectly updated when using nwis hdf5 backend on some platforms

0.6.2 (released 2013-11-19)

  • add usgs.eddn module for accessing USGS Emergency Data Distribution Network

0.6.1 (released 2013-07-31)

  • raise a friendlier error message for usace.swtwc when no data is found
  • bugfix: waterml files with empty metadata elements (e.g. method) were breaking the parsing logic

0.6.0 (released 2013-07-30)

  • this release includes some backwards incompatible changes
  • simplified labels used for KBDI values: kbdi_avg -> avg, kbdi_min -> min…
  • added parameter_code kwarg to usgs.nwis.hdf5.get_site_data
  • added start and end dates to cuahsi.wof.get_values()
  • added support for NCDC’s CIRS drought index dataset
  • fixed filesystem timestamp related bug that could cause file-based datesets to be unnecessarily re-downloaded
  • version number is now available at ulmo.__version__
  • misc documentation fixes

0.5.0 (released 2013-05-14)

  • nwis.pytables has been replaced with nwis.hdf5 which uses the pandas.io.pytables backend; things should generally run a bit faster now and the code is much cleaner. There is one backwards-incompatible change: nwis.hdf5.get_site() is now thread-safe. There was an undocumented behavior where if a site could not be found in the hdf5 file during a nwis.pytables.get_site() call, then a request to the NWIS service would be made and the site table would be updated but no longer happens. You must explicitly run nwis.hdf5.update_site_list() to update the site list first.
  • the nwis.pytables namespace is deprecated in favor of nwis.hdf5 because it is shorter and more recognizable but pytables is still powering things under the hood (via the pandas HDFStore)
  • parsed timeseries data dicts from waterml-based datasets (cuahsi.wof and usgs.nwis) now contain parsed site info mapped to the ‘site’ key
  • add support for pre-1997 data from CPC’s weekly drought monitor dataset
  • bugfix: some related to change of the “current” agricultural year for CPC weekly drought data
  • add support for Texas Weather Connection’s Daily Keetch-Byram Drought Index (KBDI) dataset

0.4.1 (released 2013-03-04)

  • bugfix: interpret missing data (’—-’) correctly in usace.swtwc.get_stations

0.4.0 (released 2013-03-04)

  • this release includes some backwards incompatible changes
  • usgs.nwis date_range parameter has been replaced with more explicit start, end and period parameters
  • ncdc.gsod.get_data() parameters renamed: start_date -> start and end_date -> end to be consistent with other dataset APIs
  • ncdc.gsod.get_stations() now allows limiting stations to matching parameters (e.g. country=’US’)
  • all date and datetime parameters are now consistently parsed
  • added support for CUAHSI’s HIS Central service: ulmo.cuahsi.his_central
  • added preliminary support for CPC’s weekly drought monitor dataset (>=1997)
  • added preliminary support for data from USACE Tulsa Water District Control

0.3.0 (released 2013-01-14)

  • this release includes some backwards incompatible changes
  • moved module for USGS NWIS from ulmo.usgs -> ulmo.nwis.usgs
  • moved module for WaterOneFlow from ulmo.wof -> ulmo.cuahsi.wof

0.2.4 (released 2013-01-14)

  • core API functions can be called from main dataset modules
  • support python 2.6

0.2.3 (released 2013-01-13)

  • add ghcn daily dataset
  • add preliminary support for CUAHSI WOF
  • add sphinx documentation
  • clean up tests
  • make pytables an optional dependency; pytables-related functionality is just disabled if pytables can’t be found

0.2.2 (released 2012-11-05)

  • bugfix: broken usgs.pytables last_refresh handling

0.2.1 (released 2012-11-04)

  • turn on compression for pytables cache files
  • minor bugfix and test cleanup

0.2.0 (released 2012-10-12)

  • initial public release as ulmo (formerly pyhis)