### Install ncdata with Pip Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/installation.rst Installs the ncdata library from the Python Package Index (PyPI) using the pip package installer, a common method for Python packages. ```bash pip install ncdata ``` -------------------------------- ### Verify ncdata Installation Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/installation.rst Checks if the ncdata library has been successfully installed by executing a Python command that imports the NcData class and prints its default representation. ```bash $ python -c "from ncdata import NcData; print(NcData())" > ``` -------------------------------- ### Install ncdata with Conda Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/installation.rst Installs the ncdata library from the conda-forge channel using the conda package manager, suitable for environments managed by Anaconda or Miniconda. ```bash $ conda install -c conda-forge ncdata ``` -------------------------------- ### Python: Create ncdata Variables Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Shows how to instantiate new `ncdata.NcVariable` objects using its constructor. Examples include creating a minimal variable with just a name and dimensions, and a more complete example with initial data and attributes. ```python >>> var = NcVariable("data", ("x_axis",)) >>> print(var) ): data(x_axis)> >>> print(var.data) None ``` ```python >>> var = NcVariable("vyx", ("y", "x"), ... data=[[1, 2, 3], [0, 1, 1]], ... attributes=[NcAttribute('a', 1), NcAttribute('b', 'setting=off')] ... ) >>> print(var) >>> print(var.data) [[1 2 3] [0 1 1]] ``` -------------------------------- ### Test ncdata Package Installation from TestPyPI Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/developer_notes.rst Commands to create a temporary Conda environment, install the `ncdata` package from TestPyPI, and prepare for running tests. This verifies the package's installability and basic functionality before a production release. ```Shell conda create -n ncdtmp python=3.11 iris xarray filelock requests pytest pip pip install --index-url https://test.pypi.org/simple/ ncdata ``` -------------------------------- ### Create New NcData Dataset with Initial Contents Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This example demonstrates how to instantiate a new `NcData` object by providing its dimensions, variables, and global attributes directly during construction. It shows how to define variables with data, dimensions, and specific attributes like `long_name` and `units`. ```python data = NcData( dimensions=[NcDimension("y", 2), NcDimension("x", 3)], variables=[ NcVariable("y", ("y",), data=[0, 1]), NcVariable("x", ("x",), data=[0, 1, 2]), NcVariable( "vyx", ("y", "x"), data=np.zeros((2, 3)), attributes=[ NcAttribute("long_name", "rate"), NcAttribute("units", "m s-1") ] )], attributes=[NcAttribute("history", "imaginary")] ) print(data) ``` -------------------------------- ### Verify ncdata Package Installation from Official PyPI Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/developer_notes.rst Command to verify that the newly uploaded `ncdata` package is correctly available on the official PyPI by attempting a standard `pip install` without specifying an index URL. ```Shell pip install ncdata ``` -------------------------------- ### Python: Convert between ncdata and Iris Cubes Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Provides examples of converting between `ncdata` objects and `iris.cube.CubeList` using `ncdata.iris.from_iris` and `ncdata.iris.to_iris`. Notes that extra keyword arguments are passed to underlying Iris load/save routines. ```python >>> from ncdata.iris import from_iris, to_iris >>> cubes = iris.load(file) >>> ncdata = from_iris(cubes) >>> >>> cubes2 = to_iris(ncdata) ``` -------------------------------- ### Load NetCDF File into NcData Object Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst This example shows how to load data from a NetCDF file into an NcData object using the `from_nc4` function from `ncdata.netcdf`. It takes a file path as input and returns an NcData instance. ```python >>> from ncdata.netcdf import from_nc4 >>> ncdata = from_nc4("datapath.nc") ``` -------------------------------- ### Convert Iris Cube to Xarray Dataset and Save to Zarr Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst This example illustrates a chained conversion: first converting an Iris cube back to an NcData object using `from_iris`, then converting that NcData object to an Xarray Dataset using `to_xarray`. Finally, it shows how to save the resulting Xarray Dataset to a Zarr store. ```python >>> from ncdata.xarray import to_xarray >>> xrds = to_xarray(from_iris(vv)) >>> xrds.to_zarr(out_path) ``` -------------------------------- ### Use Zarr data in Iris Source: https://github.com/pp-mo/ncdata/blob/main/README.md Shows how to read Zarr data into an xarray dataset, convert it to Iris cubes for processing, and then save the results back to Zarr. This example utilizes `ncdata`'s threadlock sharing for efficient data handling. ```python from ncdata.threadlock_sharing import enable_lockshare enable_lockshare(iris=True, xarray=True) import xarray as xr dataset = xr.open_dataset(input_zarr_path, engine="zarr", chunks="auto") input_cubes = cubes_from_xarray(dataset) output_cubes = my_process(input_cubes) dataset2 = cubes_to_xarray(output_cubes) dataset2.to_zarr(output_zarr_path) ``` -------------------------------- ### Adjust NetCDF File Content Before Loading into Xarray: Correct Special Attributes Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This example demonstrates pre-processing NetCDF file content using `ncdata` before converting it to an Xarray Dataset. It shows how to correct a mis-used special attribute, such as renaming `_fillvalue` to `_FillValue`, to ensure proper interpretation by Xarray. ```python >>> from ncdata.netcdf4 import from_nc4 >>> from ncdata.xarray import to_xarray >>> ncdata = from_nc4(input_filepath) >>> for var in ncdata.variables: >>> if "_fillvalue" in var.attributes: >>> var.attributes.rename("_fillvalue", "_FillValue") ... >>> cubes = to_iris(ncdata) ``` -------------------------------- ### Mix Xarray and Iris operations with ncdata Source: https://github.com/pp-mo/ncdata/blob/main/README.md This example demonstrates how to seamlessly integrate Xarray and Iris operations using `ncdata.iris_xarray`. It shows opening an xarray dataset, performing a rolling mean, converting to Iris cubes, extracting a specific cube, and then plotting it with Iris. ```python import xarray import ncdata.iris_xarray as nci import iris.quickplot as qplt ds = xarray.open_dataset(filepath) ds_resample = ds.rolling(time=3).mean() cubes = nci.cubes_from_xarray(ds_resample) temp_cube = cubes.extract_cube("air_temperature") qplt.contourf(temp_cube[0]) ``` -------------------------------- ### Shallow Copying NcData Objects and Variable Data Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/common_operations.rst Explains the behavior of the `.copy()` method for `ncdata` core objects. It highlights that while the objects themselves are copied, the underlying variable data arrays are shallow-copied, meaning changes to one copy's data will affect the other. The example demonstrates this shared data reference. ```Python # Construct a simple test dataset ds = NcData( dimensions=[NcDimension('x', 12)], variables=[NcVariable('vx', ['x'], np.ones(12))] ) # Make a copy ds_copy = ds.copy() # The new dataset has a new matching variable with a matching data array # The variables are different .. ds_copy.variables['vx'] is ds.variables['vx'] # ... but the arrays are THE SAME ARRAY ds_copy.variables['vx'].data is ds.variables['vx'].data # So changing one actually CHANGES THE OTHER ... ds.variables['vx'].data[6:] = 777 ds_copy.variables['vx'].data ``` -------------------------------- ### Adjust NetCDF File Content Before Loading into Iris: Rename Coordinates Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This example illustrates how to pre-process NetCDF file content using `ncdata` before converting it to an Iris CubeList. It specifically shows how to identify and replace an invalid coordinate name within variable attributes to prevent loading issues in Iris. ```python >>> from ncdata.netcdf4 import from_nc4 >>> from ncdata.iris import to_iris >>> ncdata = from_nc4(input_filepath) >>> for var in ncdata.variables: >>> coords = var.attributes.get('coordinates', "") >>> if "old_varname" in coords: >>> coords.replace("old_varname", "new_varname") >>> var.set_attrval("coordinates", coords) ... >>> cubes = to_iris(ncdata) ``` -------------------------------- ### Adjust Xarray Dataset Output Before Saving to NetCDF: Convert to Masked Integers Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This example illustrates how to transform an Xarray Dataset's variable data into masked integers using `ncdata` before writing to a NetCDF file. It shows how to handle NaN values by converting them to a specific fill value and updating the variable's attributes accordingly. ```python >>> from numpy import ma >>> from ncdata.iris import from_xarray >>> from ncdata.netcdf4 import to_nc4 >>> ncdata = from_xarray(dataset) >>> var = ncdata.variables['experiment'] >>> mask = var.data.isnan() >>> data = var.data.astype(np.int16) >>> data[mask] = -9999 >>> var.data = data >>> var.set_attrval("_FillValue", -9999) >>> to_nc4(ncdata, "output.nc") ``` -------------------------------- ### Modify Global and Variable Attributes in NcData Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This snippet demonstrates how to load an NcData object from a NetCDF file, then modify, add, or remove global and variable attributes. It includes examples of updating the history attribute, deleting specific global attributes, and renaming/correcting variable attributes like 'coords' and 'units' to ensure CF-compliance. ```python from ncdata.netcdf4 import from_nc4, to_nc4 ds = from_nc4('input.nc4') history = ds.get_attrval("history") if "history" in ds.attributes else "" ds.set_attrval("history", history + ": modified to SPEC-FIX.A") removes = ("grid_x", "review") for name in removes: if name in ds.attributes: del ds.attributes.[name] for var in ds.variables.values(): if "coords" in var.attributes: var.attributes.rename("coords", "coordinates") # common non-CF problem units = var.get_attrval("units") if units and units == "ppm": var.set_attrval("units", "1.e-6") # another common non-CF problem to_nc(ds, "output_fixed.nc") ``` -------------------------------- ### Creating Basic NcData Objects Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst Demonstrates the creation of a basic `NcData` object, which serves as the primary container for datasets or groups. It shows how to initialize an `NcData` instance, add an `NcDimension`, and print the object's structure, illustrating the fundamental building blocks of NcData. ```python from ncdata import NcData, NcDimension, NcVariable data = NcData("myname") data print(data) dim = NcDimension("x", 3) data.dimensions.add(dim) print(data) data.dimensions['x'] is dim ``` -------------------------------- ### Build ncdata Project Documentation Locally Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/developer_notes.rst Instructions for building the project's documentation locally using `make html`. This command triggers Sphinx to rebuild the API documentation and generate HTML output, useful for local testing. ```Shell make html ``` -------------------------------- ### Create New NcData Dataset Iteratively Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This snippet shows an alternative method for building an `NcData` object by first creating an empty instance and then iteratively adding dimensions, variables, and attributes. It demonstrates how to set data for variables and assign attributes using `set_attrval`. ```python data = NcData() ny, nx = 2, 3 data.dimensions.add(NcDimension("y", ny)) data.dimensions.add(NcDimension("x", nx)) data.variables.add(NcVariable("y", ("y",))) data.variables.add(NcVariable("x", ("x",))) data.variables.add(NcVariable("vyx", ("y", "x"))) vx, vy, vyx = [data.variables[k] for k in ("x", "y", "vyx")] vx.data = np.arange(nx) vy.data = np.arange(ny) vyx.data = np.zeros((ny, nx)) vyx.set_attrval("long_name", "rate"), vyx.set_attrval("units", "m s-1") data.set_attrval("history", "imaginary") ``` -------------------------------- ### Reading and Writing NcData to NetCDF Files Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst Illustrates how to save an `NcData` object to a NetCDF file and then load it back using the `ncdata.netcdf4` module. It includes a system command to verify the file's contents, showcasing the basic file I/O capabilities. ```python from ncdata.netcdf4 import to_nc4, from_nc4 filepath = "./tmp.nc" to_nc4(data, filepath) from subprocess import check_output print(check_output('ncdump -h tmp.nc', shell=True).decode()) data2 = from_nc4(filepath) print(data2) ``` -------------------------------- ### NcData Xarray Integration Overview Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This section describes the fundamental integration points between `ncdata` and `xarray`. It details how `ncdata.xarray.to_xarray` and `ncdata.xarray.from_xarray` leverage `xarray.Dataset`'s load and save mechanisms, and how keyword arguments are propagated. ```APIDOC ncdata.xarray.to_xarray: Calls: xarray.Dataset.load_store Description: Converts an NcData object to an xarray.Dataset by loading from a store. Kwargs: Any additional keyword arguments are passed to xarray.Dataset.load_store. ncdata.xarray.from_xarray: Calls: xarray.Dataset.dump_to_store Description: Converts an xarray.Dataset to an NcData object by dumping to a store. Kwargs: Any additional keyword arguments are passed to xarray.Dataset.dump_to_store. General: An NcData object can be written or read as an xarray.Dataset. ``` -------------------------------- ### ncdata Library API Reference Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Reference for key classes, methods, and functions within the `ncdata` library and its integrations. ```APIDOC ncdata.NcData.set_attrval(name: str, value: Any) Purpose: Set the value of an attribute on a dataset or group. Parameters: name (str): The name of the attribute. value (Any): The value to assign to the attribute. Type can be freely changed. Notes: Attributes are writeable; assigned value is cast with numpy.asarray if setting attribute.value directly. ``` ```APIDOC ncdata.NcVariable.set_attrval(name: str, value: Any) Purpose: Set the value of an attribute on a variable. Parameters: name (str): The name of the attribute. value (Any): The value to assign to the attribute. Type can be freely changed. Notes: Attributes are writeable; assigned value is cast with numpy.asarray if setting attribute.value directly. ``` ```APIDOC ncdata.NcVariable.__init__(name: str, dimensions: Tuple[str, ...], data: Optional[Any] = None, attributes: Optional[List[NcAttribute]] = None) Purpose: Constructor for creating a new NcVariable object. Parameters: name (str): The name of the variable. dimensions (Tuple[str, ...]): A tuple of dimension names. data (Optional[Any]): Optional initial data array (e.g., numpy.ndarray, dask.array.Array). attributes (Optional[List[NcAttribute]]): Optional list of NcAttribute objects. ``` ```APIDOC ncdata.NcVariable.data: Union[numpy.ndarray, dask.array.Array, None] Purpose: Property holding the data array for the variable. Type: numpy.ndarray or dask.array.Array (real or lazy). Notes: Can be freely overwritten. If not None, should always be an array of the correct shape. ``` ```APIDOC ncdata.netcdf4.from_nc4(filepath: str, dim_chunks: Optional[Dict[str, int]] = None) -> NcData Purpose: Load a dataset from a NetCDF file. Parameters: filepath (str): Path to the NetCDF file. dim_chunks (Optional[Dict[str, int]]): Dictionary to control chunking ``` -------------------------------- ### Managing Attributes for NcData Variables Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst Explains how to set and retrieve attributes on an `NcVariable` using `set_attrval`. It demonstrates how attributes are stored and displayed, noting that they are `NcAttribute` objects rather than simple key-value pairs, and how they integrate into the NcData structure. ```python var.set_attrval('a', 1) NcAttribute('a', 1) var.set_attrval('b', 'this') NcAttribute('b', 'this') print(var) print(var.attributes) {'a': NcAttribute('a', 1), 'b': NcAttribute('b', 'this')} print(data) > ``` -------------------------------- ### Build Python Distribution for ncdata Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/developer_notes.rst Command to build the Python distribution package (wheel and sdist) for `ncdata` using the `build` module. This step is a prerequisite for uploading the package to PyPI. ```Python python -m build ``` -------------------------------- ### Python: Convert between ncdata and Xarray Datasets Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Illustrates how to convert between `ncdata` objects and `xarray.Dataset` using `ncdata.xarray.from_xarray` and `ncdata.xarray.to_xarray`. ```python >>> from ncdata.xarray import from_xarray, to_xarray >>> dataset = xarray.open_dataset(filepath) >>> ncdata = from_xarray(dataset) >>> >>> ds2 = to_xarray(ncdata) ``` -------------------------------- ### Initializing NcData with Variables from a List Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/data_objects.rst Shows a common and useful way to initialize an NcData object by providing a list of NcVariable objects to its 'variables' property. This leverages the NameMap.from_items utility for flexible object construction. ```python ds2 = NcData( variables=[ NcVariable('v1', ('x',), data=[1,2]), NcVariable('v2', ('x',), data=[2,3]) ] ) print(ds2) variables: > ``` -------------------------------- ### xarray.Dataset.to_netcdf for Advanced File Creation Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/interface_support.rst This `xarray` method offers comprehensive options for controlling NetCDF file creation, including compression and chunking strategies. It is recommended as an alternative to `ncdata.netcdf4.to_nc4` when specific storage controls are required. ```APIDOC xarray.Dataset.to_netcdf ``` -------------------------------- ### Adding and Accessing Variables in NcData Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst Shows how to create an `NcVariable` and add it to an `NcData` object's `variables` attribute, which behaves like a dictionary. It demonstrates accessing the variable and printing the updated `NcData` structure, highlighting variable management. ```python var = NcVariable("vx", dimensions=["x"], dtype=float) data.variables.add(var) data.variables {'vx': } data.variables['vx'] is var True print(data) > ``` -------------------------------- ### Load, Add Elements, and Save NetCDF Data with ncdata Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This snippet demonstrates how to load an existing NetCDF dataset, create a new empty `NcData` object, and then selectively add dimensions and variables from the input dataset to the new one before saving it. This is useful for creating subsets or restructuring data. ```python >>> ds_in = from_nc4(input_filepath) >>> ds_out = NcData() >>> for varname in ('data1', 'data2', 'dimx', 'dimy'): >>> var = ds_in.variables[varname] >>> ds_out.variables.add(var) >>> for name in var.dimensions if name not in ds_out.dimensions: >>> ds_out.dimensions.add(ds_in.dimensions[dimname]) ... >>> to_nc4(ds_out, output_filepath) ``` -------------------------------- ### Python: Save ncdata to NetCDF File Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Demonstrates how to write an `NcData` object to a new NetCDF file using the `ncdata.netcdf4.to_nc4` function. ```python >>> from ncdata.netcdf4 import to_nc4 >>> to_nc4(data, filepath) ``` -------------------------------- ### Upload ncdata Package to TestPyPI Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/developer_notes.rst Command to upload the built `ncdata` Python package to TestPyPI for pre-release testing. This uses `twine` to push the distribution files to the specified test repository. ```Python python -m twine upload --repository testpypi dist/* ``` -------------------------------- ### Iris Cube Conversion Functions Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst API documentation for functions within the `ncdata.iris` module, enabling conversion of data to and from Iris Cube objects. ```APIDOC Module: ncdata.iris from_iris(iris_cube: iris.cube.Cube) -> NcData Description: Converts an Iris Cube object to an NcData object. Parameters: iris_cube (iris.cube.Cube): The Iris Cube to convert. Returns: NcData: An NcData object representing the data. to_iris(ncdata: NcData, variables: Optional[List[str]] = None) -> Union[iris.cube.Cube, Tuple[iris.cube.Cube, ...]] Description: Converts an NcData object to one or more Iris Cube objects. Parameters: ncdata (NcData): The NcData object to convert. variables (Optional[List[str]]): A list of variable names to convert. If None, all variables are converted. Returns: Union[iris.cube.Cube, Tuple[iris.cube.Cube, ...]]: One or more Iris Cube objects. ``` -------------------------------- ### Initializing NcData with Nested Groups Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/data_objects.rst Illustrates how to construct an NcData object and populate its 'groups' property using a dictionary of NcData objects. This demonstrates the implicit use of NameMap.from_items for container initialization. ```python ds1 = NcData(groups={ 'x':NcData('x'), 'y':NcData('y') }) print(ds1) groups: > ``` -------------------------------- ### Upload ncdata Package to Official PyPI Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/developer_notes.rst Command to upload the `ncdata` Python package to the official PyPI repository. This step should only be performed after successful testing on TestPyPI. ```Python python -m twine upload dist/* ``` -------------------------------- ### Python: Read NetCDF Data with ncdata Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Illustrates how to load a dataset from a NetCDF file into an `NcData` object using the `ncdata.netcdf4.from_nc4` function. ```python >>> from ncdata.netcdf4 from_nc4 >>> ds = from_nc4(filepath) >>> print(ds) ``` -------------------------------- ### Xarray Dataset Conversion Functions Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst API documentation for functions within the `ncdata.xarray` module, facilitating conversion of data to and from Xarray Dataset objects. ```APIDOC Module: ncdata.xarray from_xarray(xarray_dataset: xarray.Dataset) -> NcData Description: Converts an Xarray Dataset object to an NcData object. Parameters: xarray_dataset (xarray.Dataset): The Xarray Dataset to convert. Returns: NcData: An NcData object representing the data. to_xarray(ncdata: NcData) -> xarray.Dataset Description: Converts an NcData object to an Xarray Dataset object. Parameters: ncdata (NcData): The NcData object to convert. Returns: xarray.Dataset: An Xarray Dataset object. ``` -------------------------------- ### NetCDF File Conversion Functions Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst API documentation for functions within the `ncdata.netcdf4` module, providing capabilities to convert data to and from NetCDF files. ```APIDOC Module: ncdata.netcdf4 from_nc4(filepath: str) -> NcData Description: Converts data from a NetCDF file to an NcData object. Parameters: filepath (str): The path to the NetCDF file. Returns: NcData: An NcData object representing the data. to_nc4(ncdata: NcData, filepath: str) -> None Description: Converts an NcData object to a NetCDF file. Parameters: ncdata (NcData): The NcData object to convert. filepath (str): The path where the NetCDF file will be saved. Returns: None ``` -------------------------------- ### Direct Conversion from Xarray to Iris Cubes Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst This snippet demonstrates the direct conversion of an Xarray Dataset to Iris cubes using `cubes_from_xarray` from the `ncdata.iris_xarray` module. It also includes an assertion to verify the equivalence of the converted Iris cube with a previously derived one. ```python >>> from ncdata.iris_xarray import cubes_from_xarray >>> vv2 = cubes_from_xarray(xrds) >>> assert vv2 == vv ``` -------------------------------- ### netCDF4.Dataset.createVariable Storage Options Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/interface_support.rst This method from the `netCDF4` library allows fine-grained control over data compression and storage settings when creating variables within a NetCDF file. It is mentioned as the source of options not exposed by `ncdata.netcdf4.to_nc4`. ```APIDOC netCDF4.Dataset.createVariable ``` -------------------------------- ### Convert NcData to Iris Cubes and Perform Calculation Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst This snippet demonstrates converting an NcData object to Iris cubes using `to_iris`, extracting specific variables ('x_wind', 'y_wind'), and then performing a simple calculation to derive a new Iris cube. It also shows how to assign units to the resulting cube. ```python >>> from ncdata.iris import to_iris, from_iris >>> xx, yy = to_iris(ncdata, ['x_wind', 'y_wind']) >>> vv = (xx * xx + yy * yy) ** 0.5 >>> vv.units = xx.units ``` -------------------------------- ### Iterative Creation of NcData Objects Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/common_operations.rst Shows an alternative approach to creating `NcData` objects by first initializing an empty instance and then iteratively adding dimensions and variables. This method is useful for building data structures dynamically. ```Python data = NcData() dims = [("y", 2), ("x", 3)] data.variables.addall([ NcVariable(nn, dimensions=[nn], data=np.arange(ll)) for ll, nn in dims ]) data.variables.add( NcVariable("dd", dimensions=["y", "x"], data=np.arange(6).reshape(2,3)) ) ``` -------------------------------- ### Inline Creation of NcData Objects Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/common_operations.rst Demonstrates how to construct `NcData` objects with dimensions and variables in a single, readable inline declaration. This method allows for immediate population of the data structure upon instantiation. ```Python data = NcData( dimensions=[ NcDimension("y", 2), NcDimension("x", 3) ], variables=[ NcVariable("y", dimensions=["y"], data=[10, 11]), NcVariable("x", dimensions=["y"], data=[20, 21, 22]), NcVariable("dd", dimensions=["y", "x"], data=[[0, 1, 2], [3, 4, 5]]) ] ) ``` -------------------------------- ### Combine NcData from Multiple Files Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This snippet illustrates how to merge variables from two separate NcData objects, loaded from different NetCDF files, into a single output NcData object. It demonstrates iterating through desired variables from a source dataset and adding them to a target dataset before saving. ```python from ncdata.netcdf4 import from_nc4, to_nc4 data = from_nc4('input1.nc') data2 = from_nc4('input2.nc') # Add some known variables from file2 into file1 wanted = ('x1', 'x2', 'x3') for name in wanted: data.variables.add(data2.variables[name]) to_nc4(data, 'output.nc') ``` -------------------------------- ### Convert Iris Cubes to Xarray Dataset and Vice Versa Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This snippet demonstrates how to convert data between Iris cubes and Xarray datasets using `ncdata.iris_xarray.cubes_to_xarray` and `ncdata.iris_xarray.cubes_from_xarray`. These functions act as convenient wrappers for combined `ncdata.xarray` and `ncdata.iris` operations. ```python from ncdata.iris_xarray import cubes_from_xarray, cubes_to_xarray cubes = iris.load(filepath) dataset = cubes_to_xarray(cubes) cubes2 = cubes_from_xarray(dataset) ``` ```python cubes = cubes_from_xarray( dataset, iris_load_kwargs={'constraints': 'air_temperature'}, xr_save_kwargs={'unlimited_dims': ('time',)}, ) ``` -------------------------------- ### Load NetCDF Files with Variable-Width String Variables Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This snippet addresses the specific challenge of loading NetCDF files containing variable-width string variables using `ncdata.netcdf4.from_nc4`. It highlights the necessity of providing a `dim_chunks` keyword argument to avoid Dask auto-chunking errors. ```python >>> from ncdata.netcdf4 import from_nc4 >>> # This file has a netcdf "string" type variable, with dimensions ('date',). >>> # : don't chunk that dimension. >>> dataset = from_nc4(filepath, dim_chunks={"date": -1}) ``` -------------------------------- ### Python: Control NetCDF Read Chunking Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Shows how to specify chunking for dimensions when reading a NetCDF file using the `dim_chunks` argument of the `ncdata.netcdf4.from_nc4` function. ```python >>> from ncdata.netcdf4 from_nc4 >>> ds = from_nc4(filepath, dim_chunks={"time": 3}) >>> print(ds.variables["x"].data.chunksize) (3,) ``` -------------------------------- ### Add New Components and Attributes to NcData Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/common_operations.rst Illustrates how to add new content components using the `add` method. It also shows how to add or update attribute values on existing components using the specialized `set_attrval` method. ```Python dataset.variables.add(NcVariable("x", dimensions=["x"], data=my_data)) dataset.variables["x"].set_attrval("units", "m s-1") ``` -------------------------------- ### Manage NetCDF Thread Lock Sharing with Context Manager Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/threadlock_sharing.rst This snippet illustrates an alternative approach to managing thread lock sharing using the `lockshare_context` context manager. This method ensures that thread safety is automatically handled within the `with` block, simplifying resource management and guaranteeing that locks are properly released upon exiting the context. ```python with lockshare_context(iris=True): ncdata = NcData(source_filepath) ncdata.variables['x'].attributes['units'] = 'K' cubes = ncdata.iris.to_iris(ncdata) iris.save(cubes, output_filepath) ``` -------------------------------- ### Initializing NcVariable with Attributes Dictionary Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/data_objects.rst Demonstrates a special case where an NcVariable can be initialized with its 'attributes' property directly from a regular Python dictionary. The dictionary keys become attribute names, and values are automatically converted to NcAttribute objects. ```python var = NcVariable( 'v3', attributes={'x':'this', 'b':1.4, 'arr': [1, 2, 3]} ) print(var) ): v3() v3:x = 'this' v3:b = 1.4, v3:arr = array([1, 2, 3]) > ``` -------------------------------- ### Direct Iris-Xarray Conversion Functions Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst API documentation for functions within the `ncdata.iris_xarray` module, providing direct conversion capabilities between Iris Cube and Xarray Dataset objects. ```APIDOC Module: ncdata.iris_xarray cubes_to_xarray(iris_cubes: Union[iris.cube.Cube, Iterable[iris.cube.Cube]]) -> xarray.Dataset Description: Converts one or more Iris Cube objects directly to an Xarray Dataset. Parameters: iris_cubes (Union[iris.cube.Cube, Iterable[iris.cube.Cube]]): One or more Iris Cube objects to convert. Returns: xarray.Dataset: An Xarray Dataset object. cubes_from_xarray(xarray_dataset: xarray.Dataset) -> Union[iris.cube.Cube, Tuple[iris.cube.Cube, ...]] Description: Converts an Xarray Dataset directly to one or more Iris Cube objects. Parameters: xarray_dataset (xarray.Dataset): The Xarray Dataset to convert. Returns: Union[iris.cube.Cube, Tuple[iris.cube.Cube, ...]]: One or more Iris Cube objects. ``` -------------------------------- ### NcData Core Object Methods and Utilities API Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/common_operations.rst This section documents key methods available on `ncdata.NameMap` and `ncdata.NcData` core component types, along with utility functions from `ncdata.utils`. These APIs facilitate common operations like deletion, addition, renaming, copying, and comprehensive equality checking. ```APIDOC ncdata.NameMap: __delitem__(key): Removes an item by key. pop(key, default=None): Removes and returns an item by key. add(component): Adds a new component under its own name. set_attrval(name, value): Adds or sets an attribute value. rename(old_name, new_name): Renames a component in the container and its own name. ncdata.NcData: copy(): Creates a shallow copy of the NcData object, referencing the same variable data arrays. ncdata.utils: ncdata_copy(obj): Utility function to create a shallow copy of an NcData object. dataset_differences(ds1, ds2, **kwargs): Compares two NcData objects comprehensively. variable_differences(var1, var2, **kwargs): Compares two NcVariable objects comprehensively. ``` -------------------------------- ### ncdata.iris.from_iris Unlimited Dimensions Exception Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/interface_support.rst When converting Iris cubes to `ncdata` using this function, attempting to use an `unlimited_dims` key will currently raise an exception. This is a known issue that users should be aware of. ```APIDOC ncdata.iris.from_iris ``` -------------------------------- ### Check ncdata Version Availability on Conda-Forge Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/developer_notes.rst Command to search for the `ncdata` package on Conda and verify that the new version has been successfully propagated after updating the Conda-Forge feedstock. ```Shell conda search ncdata ``` -------------------------------- ### ncdata.NcAttribute Class and String Handling Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/character_handling.rst This API documentation describes how character data is stored within `ncdata.NcAttribute` instances and how it can be converted to standard Python strings. It details the internal storage format for attribute values and the method used for conversion. ```APIDOC class ncdata.NcAttribute: .value: Type: character array (numpy dtype ">> variable.set_attr("x", 3.) ``` -------------------------------- ### Python Data Assignment Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Assigns the value from the 'data' variable to the 'data' attribute of the 'var' object. This is a fundamental operation for populating object properties or structures with data. ```Python var.data = data ``` -------------------------------- ### Load, Delete Elements, and Save NetCDF Data with ncdata Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This snippet shows an alternative approach to modifying a NetCDF dataset. It involves loading the entire dataset, then deleting unwanted variables and dimensions directly from the loaded `NcData` object, and finally saving the modified dataset. This is safe as the original file remains untouched. ```python >>> data = from_nc4(input_filepath) >>> for name in ('extra1', 'extra2', 'unwanted'): >>> del data.variables[varname] ... >>> del data.dimensions['pressure'] >>> to_nc4(data, output_filepath) ``` -------------------------------- ### Enable Thread Lock Sharing for Iris and Xarray Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst This snippet demonstrates how to enable thread lock sharing between Iris and Xarray using `enable_lockshare` from `ndata.threadlock_sharing`. This is often necessary when working with data from NetCDF files in conjunction with Iris or Xarray to prevent potential errors during computation or saving of lazy data. ```python >>> from ndata.threadlock_sharing import enable_lockshare >>> enable_lockshare(iris=True, xarray=True) ``` -------------------------------- ### Deleting and Renaming NcData Attributes and Elements Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/getting_started/introduction.rst Illustrates how to remove attributes using Python's `del` keyword and how to rename attributes, variables, or groups using the `rename` method. A warning is included regarding dimension renaming, highlighting potential side effects. ```python del var.attributes['a'] print(var) var.attributes.rename("b", "qq") print(var) print(data) > ``` -------------------------------- ### ncdata.netcdf4.to_nc4 Function Limitations Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/interface_support.rst This function, used for writing `ncdata` objects to NetCDF files, does not provide control over advanced compression or storage options. For such controls, users are advised to use `xarray.Dataset.to_netcdf` or `iris.save`. ```APIDOC ncdata.netcdf4.to_nc4 ``` -------------------------------- ### ncdata.netcdf4.from_nc4 Dask Chunking Control Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/interface_support.rst When loading data from NetCDF files into `ncdata` using this function, variables are generated as Dask lazy arrays. This function offers a simple per-dimension control over Dask chunking during the loading process. ```APIDOC ncdata.netcdf4.from_nc4 ``` -------------------------------- ### Setting NcData Variable Attributes Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/data_objects.rst Shows two methods for setting attribute values on an NcVariable: using set_attrval() for direct assignment or creating an NcAttribute object and assigning it to the 'attributes' NameMap. It also points out an incorrect approach for setting attribute values. ```python dataset.variables['var1'].set_attrval('units', "K") dataset.variables['var1'].attributes['units'] = NcAttribute("units", K) ``` -------------------------------- ### APIDOC: ncdata.NcAttribute Class Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/data_objects.rst Documentation for the NcAttribute class, representing an attribute with a name and value. ```APIDOC NcAttribute: Description: Represents an attribute, with a name and value. The value is always either a scalar or a 1-D numpy array. Properties: .name: str The name of the attribute. .value: numpy.generic or numpy.ndarray The value of the attribute. Must be a numpy scalar or 1-dimensional array. When assigned, the value is cast with numpy.asanyarray; an error is raised if this fails or yields a multidimensional array. Methods: as_python_value(): Any Returns the attribute's value as a Python scalar, numpy scalar/array, or string. This method is intended to provide values equivalent to what would be obtained from storing in an actual file and reading back, including re-interpreting length-one vectors as scalar values. ``` -------------------------------- ### Adding Dimensions to ncdata Object Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/common_operations.rst This snippet demonstrates how to add multiple dimensions to a 'data' object using the 'addall' method of its 'dimensions' attribute. It utilizes a list comprehension to create 'NcDimension' objects from 'nn' (name) and 'll' (length) pairs. A key prerequisite is that the variables associated with these dimensions must have been created before this operation. ```Python data.dimensions.addall([NcDimension(nn, ll) for nn, ll in dims]) ``` -------------------------------- ### ncdata.xarray.to_xarray Conversion Issues Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/interface_support.rst Converting `ncdata` datasets to `xarray` using this function results in the loss of dataset encodings, and critically, the `unlimited_dims` control is not preserved. This behavior is noted as a bug that may be addressed in future versions. ```APIDOC ncdata.xarray.to_xarray ``` -------------------------------- ### APIDOC: ncdata.NcVariable Class Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/data_objects.rst Documentation for the NcVariable class, representing a data variable with dimensions, optional data, and attributes. ```APIDOC NcVariable: Description: Represents a data variable, with dimensions and, optionally, data and attributes. Properties: .dimensions: list[str] A list of names (strings) of the dimensions. These are not linked to NcDimension objects. .dtype: numpy.dtype or None The data type of the variable. Can be set if creating with no data. .data: numpy.ndarray or dask.array.Array or None The array data for the variable. Assumed to have 'shape', 'dtype', and '__getitem__' properties. Methods: get_attrval(attr_name: str): Any Returns the value of a named attribute, handling type conversions for consistent results. (Equivalent to ncdata.NcAttribute.as_python_value for attributes of this variable). ``` -------------------------------- ### Convert Variable-Width String Data to Fixed-Width Character Array Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Following the loading of variable-width string data into a numpy object array, this snippet provides a method to convert that data into a fixed-width character array. It calculates the maximum string length, pads shorter strings, and updates the `ncdata` object's dimensions and variable's dimensions to reflect the new fixed-width structure. ```python >>> var = dataset.variables['name'] >>> data = var.data.compute() >>> maxlen = max(len(s) for s in var.data) >>> # convert to fixed-width character array >>> data = np.array([[s.ljust(maxlen, "\0") for s in var.data]]) >>> print(data.shape, data.dtype) (1010, 12) >> dataset.dimensions.add(NcDimension('name_strlen', maxlen)) >>> var.dimensions = var.dimensions + ("name_strlen",) ``` -------------------------------- ### APIDOC: ncdata.NcData Class Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/data_objects.rst Documentation for the NcData class, which represents a dataset or a group within the ncdata data model. ```APIDOC NcData: Description: Represents a dataset containing variables, dimensions, attributes and groups. It is also used to represent groups. Properties: .name: str or None The name of the data object. This can be None when the object is not contained in a parent. ``` -------------------------------- ### Adding ncdata components (variables, dimensions, attributes, groups) in Python Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst Insert new items into an ncdata object using the `add` method of the relevant component-container property (e.g., data.dimensions.add()). The item must be of the correct type (e.g., NcDimension) to avoid errors. While direct assignment like `data.dimensions["y"] = NcDimension("y", 4)` might work, using `add` is safer and simpler as it ensures the key matches the component's name. ```python >>> data.dimensions.add(NcDimension("y", 4)) >>> data.dimensions {'x': NcDimension('x', 3) 'y': NcDimension('y', 3)} ``` ```python data.dimensions["y"] = NcDimension("y", 4) ``` -------------------------------- ### Extract and Remove Components from NcData Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/common_operations.rst Demonstrates how to remove elements from `ncdata` objects using the `pop` method to retrieve and remove, or the `del` keyword for direct deletion. These operations behave similarly to standard Python dictionary methods. ```Python var_x = dataset.variables.pop("x") del data.variables["x"] ``` -------------------------------- ### Adjust Iris Cube Output Before Saving to NetCDF: Force Unlimited Dimension Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/howtos.rst This snippet shows how to modify an Iris CubeList's structure via `ncdata` before saving it to a NetCDF file. It specifically demonstrates how to force an additional dimension, like 'timestep', to be unlimited, which might be difficult to achieve directly when saving from Iris. ```python >>> from ncdata.iris import from_iris >>> from ncdata.netcdf4 import to_nc4 >>> ncdata = from_iris(cubes) >>> ncdata.dimensions['timestep'].unlimited = True >>> to_nc4(ncdata, "output.nc") ``` -------------------------------- ### APIDOC: ncdata.NcDimension Class Source: https://github.com/pp-mo/ncdata/blob/main/docs/userdocs/user_guide/data_objects.rst Documentation for the NcDimension class, representing a dimension within the ncdata data model. ```APIDOC NcDimension: Description: Represents a dimension, defined by its name, length, and whether it is "unlimited". Properties: .name: str The name of the dimension. .length: int The length of the dimension. .unlimited: bool Indicates if the dimension is "unlimited". ``` -------------------------------- ### Apply Iris regrid to xarray data Source: https://github.com/pp-mo/ncdata/blob/main/README.md Demonstrates how to apply an Iris regridding operation to an xarray dataset by converting between the two formats using `ncdata.iris_xarray`. This allows leveraging Iris's advanced regridding capabilities on xarray data structures. ```python from ncdata.iris_xarray import cubes_to_xarray, cubes_from_xarray dataset = xarray.open_dataset("file1.nc", chunks="auto") (cube,) = cubes_from_xarray(dataset) cube2 = cube.regrid(grid_cube, iris.analysis.PointInCell) dataset2 = cubes_to_xarray(cube2) ``` -------------------------------- ### Enable and Disable NetCDF Thread Lock Sharing with ncdata Source: https://github.com/pp-mo/ncdata/blob/main/docs/details/threadlock_sharing.rst This snippet demonstrates how to explicitly enable and disable thread lock sharing using `enable_lockshare` and `disable_lockshare` functions. It illustrates a common use case where data loaded from Xarray and Iris are combined, processed, and then saved to a NetCDF file, ensuring thread safety throughout the operation. ```python from ncdata.threadlock_sharing import enable_lockshare, disable_lockshare from ncdata.xarray import from_xarray from ncdata.iris import from_iris from ncdata.netcdf4 import to_nc4 enable_lockshare(iris=True, xarray=True) ds = from_xarray(xarray.open_dataset(file1)) ds2 = from_iris(iris.load(file2)) ds.variables['x'].data /= ds2.variables['acell'].data to_nc4(ds, output_filepath) disable_lockshare() ```