{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "In [previous](https://the-fonz.gitlab.io/posts/array-databases/) [posts](https://the-fonz.gitlab.io/posts/interpolation/) we explored how to work efficiently with [xarray](xarray.pydata.org/) and geospatial meteorological data, and while it would be possible to throw in extra adjectives to make that sound even more impressive, let's focus on the storage of this data.\n", "\n", "\n", "\n", "[Zarr](xarray.pydata.org/) is a relatively new storage format for multidimensional array data that puts every \"array chunk\" in a separate file/blob, allowing many different encoding/compression schemes within that blob while giving up random reads, which is fine as for (cloud) blob storage the latency overhead of getting a blob is relatively high, but bandwidth high as well so better just download the whole blob in one go. For google's cloud storage, these prices can be expected:\n", "\n", "| Storage type | Price [$/GB/month] | Retrieval cost [GB] |\n", "|---|---|---|\n", "| [Standard](https://cloud.google.com/storage/docs/storage-classes#standard) | .02 | 0 |\n", "| [Nearline](https://cloud.google.com/storage/docs/storage-classes#nearline) | .01 | 0.01 |\n", "| [Coldline](https://cloud.google.com/storage/docs/storage-classes#coldline) | .004 | 0.02 |\n", "| [Archive](https://cloud.google.com/storage/docs/storage-classes#archive) | .0012 | 0.05 |\n", "\n", "Please note that these prices are approximate and may vary by region and may become cheaper over time (I remember standard storage costing .026 \\\\$/GB/month not too long ago, can still see that price in some examples). Minimum storage duration is 30 days for nearline and 90 days for coldline, and there is a small cost for operations (delete/move etc) but that is usually negligible. Availability SLAs are also best for standard storage and go down a bit for the cold storage classes. Also, I assume the data is used mostly in gcloud datacenters, as egress cost of ~.10 \\\\$/GB will quickly eclipse any storage cost.\n", "\n", "From the table above, it's easy to deduct that nearline storage is cheaper than standard when downloaded less than once a month, while coldline is cheaper when downloaded at most once every 1.5 months, and archive at most once every 3 months. It can be a nice trick to set a lifecycle policy on a cloud storage bucket that moves objects to nearline after 1 month.\n", "\n", "This compares to about .05 \\\\$/GB/month for block storage on HDDs, up to .17 \\\\$/GB/month for block storage on SSDs, so blob storage is at least 2.5x cheaper. Also, it's difficult to mount block storage for concurrent reading/writing as one would need compute on top of the storage that runs something like NFS which allows connection over the network. This shows why it's so important to have good support for blob storage in file formats for storing huge amounts of numeric array data.\n", "\n", "I want to know what the best preprocessing steps (quantization?) are, what the compresson algorithm is with the best tradeoff of speed/compression, and if byte shuffling yields significant benefits.\n", "\n", "## The data\n", "We use ERA5 again as it's a representative set of meteo data, but this time downloaded from the awesome Pangeo cloud storage buckets. Make sure you use a gcloud account with billing enabled, as it's requester-pays on these buckets. See [the instructions here](https://catalog.pangeo.io/browse/master/atmosphere/era5_hourly_reanalysis_single_levels_sa/)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "!pip install -q intake intake-xarray gcsfs" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import os\n", "import tempfile\n", "from datetime import datetime\n", "from time import time\n", "from pathlib import Path\n", "\n", "import numpy as np\n", "from intake import open_catalog\n", "from itertools import permutations\n", "import matplotlib.pyplot as plt\n", "from numcodecs import blosc\n", "from zarr.codecs import Blosc\n", "# Use nice seaborn mpl theme\n", "import seaborn as sns\n", "sns.set_theme()\n", "\n", "# Make fair comparisons with compression algorithms later (some use multiple threads by default)\n", "blosc.set_nthreads(1);" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
<xarray.Dataset>\n", "Dimensions: (latitude: 721, longitude: 1440, time: 350640)\n", "Coordinates:\n", " * latitude (latitude) float32 90.0 89.75 89.5 89.25 ... -89.5 -89.75 -90.0\n", " * longitude (longitude) float32 0.0 0.25 0.5 0.75 ... 359.25 359.5 359.75\n", " * time (time) datetime64[ns] 1979-01-01 ... 2018-12-31T23:00:00\n", "Data variables:\n", " asn (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " d2m (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " e (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " mn2t (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " mx2t (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " ptype (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " ro (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " sd (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " sro (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " ssr (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " t2m (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " tcc (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " tcrw (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " tp (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " tsn (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " u10 (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", " v10 (time, latitude, longitude) float32 dask.array<chunksize=(31, 721, 1440), meta=np.ndarray>\n", "Attributes:\n", " Conventions: CF-1.6\n", " history: 2019-09-20 05:15:01 GMT by grib_to_netcdf-2.10.0: /opt/ecmw...
array([ 90. , 89.75, 89.5 , ..., -89.5 , -89.75, -90. ], dtype=float32)
array([0.0000e+00, 2.5000e-01, 5.0000e-01, ..., 3.5925e+02, 3.5950e+02,\n", " 3.5975e+02], dtype=float32)
array(['1979-01-01T00:00:00.000000000', '1979-01-01T01:00:00.000000000',\n", " '1979-01-01T02:00:00.000000000', ..., '2018-12-31T21:00:00.000000000',\n", " '2018-12-31T22:00:00.000000000', '2018-12-31T23:00:00.000000000'],\n", " dtype='datetime64[ns]')
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "
<xarray.Dataset>\n", "Dimensions: (latitude: 721, longitude: 1440, time: 31)\n", "Coordinates:\n", " * latitude (latitude) float32 90.0 89.75 89.5 89.25 ... -89.5 -89.75 -90.0\n", " * longitude (longitude) float32 0.0 0.25 0.5 0.75 ... 359.25 359.5 359.75\n", " * time (time) datetime64[ns] 1979-01-01 ... 1979-01-02T06:00:00\n", "Data variables:\n", " asn (time, latitude, longitude) float32 0.88000053 ... 0.850001\n", " d2m (time, latitude, longitude) float32 240.97255 ... 244.83789\n", " e (time, latitude, longitude) float32 nan nan ... 4.9493974e-07\n", " mn2t (time, latitude, longitude) float32 nan nan ... 248.4101 248.4101\n", " mx2t (time, latitude, longitude) float32 nan nan ... 248.46516\n", " ptype (time, latitude, longitude) float32 nan nan nan ... 0.0 0.0 0.0\n", " ro (time, latitude, longitude) float32 nan nan nan ... 0.0 0.0 0.0\n", " sd (time, latitude, longitude) float32 0.0 0.0 0.0 ... 10.0 10.0\n", " sro (time, latitude, longitude) float32 nan nan nan ... 0.0 0.0 0.0\n", " ssr (time, latitude, longitude) float32 nan nan ... 268708.75\n", " t2m (time, latitude, longitude) float32 244.07776 ... 248.47702\n", " tcc (time, latitude, longitude) float32 1.0 1.0 1.0 ... 0.0 0.0 0.0\n", " tcrw (time, latitude, longitude) float32 0.0 0.0 0.0 ... 0.0 0.0 0.0\n", " tp (time, latitude, longitude) float32 nan nan nan ... 0.0 0.0 0.0\n", " tsn (time, latitude, longitude) float32 245.32692 ... 246.29123\n", " u10 (time, latitude, longitude) float32 0.01948136 ... 0.52920276\n", " v10 (time, latitude, longitude) float32 -0.024540722 ... 0.22564238\n", "Attributes:\n", " Conventions: CF-1.6\n", " history: 2019-09-20 05:15:01 GMT by grib_to_netcdf-2.10.0: /opt/ecmw...
array([ 90. , 89.75, 89.5 , ..., -89.5 , -89.75, -90. ], dtype=float32)
array([0.0000e+00, 2.5000e-01, 5.0000e-01, ..., 3.5925e+02, 3.5950e+02,\n", " 3.5975e+02], dtype=float32)
array(['1979-01-01T00:00:00.000000000', '1979-01-01T01:00:00.000000000',\n", " '1979-01-01T02:00:00.000000000', '1979-01-01T03:00:00.000000000',\n", " '1979-01-01T04:00:00.000000000', '1979-01-01T05:00:00.000000000',\n", " '1979-01-01T06:00:00.000000000', '1979-01-01T07:00:00.000000000',\n", " '1979-01-01T08:00:00.000000000', '1979-01-01T09:00:00.000000000',\n", " '1979-01-01T10:00:00.000000000', '1979-01-01T11:00:00.000000000',\n", " '1979-01-01T12:00:00.000000000', '1979-01-01T13:00:00.000000000',\n", " '1979-01-01T14:00:00.000000000', '1979-01-01T15:00:00.000000000',\n", " '1979-01-01T16:00:00.000000000', '1979-01-01T17:00:00.000000000',\n", " '1979-01-01T18:00:00.000000000', '1979-01-01T19:00:00.000000000',\n", " '1979-01-01T20:00:00.000000000', '1979-01-01T21:00:00.000000000',\n", " '1979-01-01T22:00:00.000000000', '1979-01-01T23:00:00.000000000',\n", " '1979-01-02T00:00:00.000000000', '1979-01-02T01:00:00.000000000',\n", " '1979-01-02T02:00:00.000000000', '1979-01-02T03:00:00.000000000',\n", " '1979-01-02T04:00:00.000000000', '1979-01-02T05:00:00.000000000',\n", " '1979-01-02T06:00:00.000000000'], dtype='datetime64[ns]')
array([[[0.88000053, 0.88000053, 0.88000053, ..., 0.88000053,\n", " 0.88000053, 0.88000053],\n", " [0.88000053, 0.88000053, 0.88000053, ..., 0.88000053,\n", " 0.88000053, 0.88000053],\n", " [0.88000053, 0.88000053, 0.88000053, ..., 0.88000053,\n", " 0.88000053, 0.88000053],\n", " ...,\n", " [0.850001 , 0.850001 , 0.850001 , ..., 0.850001 ,\n", " 0.850001 , 0.850001 ],\n", " [0.850001 , 0.850001 , 0.850001 , ..., 0.850001 ,\n", " 0.850001 , 0.850001 ],\n", " [0.850001 , 0.850001 , 0.850001 , ..., 0.850001 ,\n", " 0.850001 , 0.850001 ]],\n", "\n", " [[0.88000053, 0.88000053, 0.88000053, ..., 0.88000053,\n", " 0.88000053, 0.88000053],\n", " [0.88000053, 0.88000053, 0.88000053, ..., 0.88000053,\n", " 0.88000053, 0.88000053],\n", " [0.88000053, 0.88000053, 0.88000053, ..., 0.88000053,\n", " 0.88000053, 0.88000053],\n", "...\n", " [0.850001 , 0.850001 , 0.850001 , ..., 0.850001 ,\n", " 0.850001 , 0.850001 ],\n", " [0.850001 , 0.850001 , 0.850001 , ..., 0.850001 ,\n", " 0.850001 , 0.850001 ],\n", " [0.850001 , 0.850001 , 0.850001 , ..., 0.850001 ,\n", " 0.850001 , 0.850001 ]],\n", "\n", " [[0.88000053, 0.88000053, 0.88000053, ..., 0.88000053,\n", " 0.88000053, 0.88000053],\n", " [0.88000053, 0.88000053, 0.88000053, ..., 0.88000053,\n", " 0.88000053, 0.88000053],\n", " [0.88000053, 0.88000053, 0.88000053, ..., 0.88000053,\n", " 0.88000053, 0.88000053],\n", " ...,\n", " [0.850001 , 0.850001 , 0.850001 , ..., 0.850001 ,\n", " 0.850001 , 0.850001 ],\n", " [0.850001 , 0.850001 , 0.850001 , ..., 0.850001 ,\n", " 0.850001 , 0.850001 ],\n", " [0.850001 , 0.850001 , 0.850001 , ..., 0.850001 ,\n", " 0.850001 , 0.850001 ]]], dtype=float32)
array([[[240.97255, 240.97255, 240.97255, ..., 240.97255, 240.97255,\n", " 240.97255],\n", " [240.39493, 240.39493, 240.39201, ..., 240.3964 , 240.3964 ,\n", " 240.3964 ],\n", " [239.899 , 239.89462, 239.89317, ..., 239.90483, 239.90483,\n", " 239.90192],\n", " ...,\n", " [246.43361, 246.43361, 246.43361, ..., 246.43507, 246.43507,\n", " 246.43507],\n", " [246.41464, 246.41464, 246.4161 , ..., 246.41464, 246.41464,\n", " 246.41464],\n", " [246.19002, 246.19002, 246.19002, ..., 246.19002, 246.19002,\n", " 246.19002]],\n", "\n", " [[241.26282, 241.26282, 241.26282, ..., 241.26282, 241.26282,\n", " 241.26282],\n", " [240.86461, 240.86461, 240.86461, ..., 240.86606, 240.86606,\n", " 240.86606],\n", " [240.53787, 240.53351, 240.53204, ..., 240.54225, 240.54225,\n", " 240.53934],\n", "...\n", " [245.31485, 245.31485, 245.31485, ..., 245.31339, 245.31339,\n", " 245.31339],\n", " [245.1442 , 245.1442 , 245.1442 , ..., 245.14128, 245.14128,\n", " 245.14128],\n", " [244.7489 , 244.7489 , 244.7489 , ..., 244.7489 , 244.7489 ,\n", " 244.7489 ]],\n", "\n", " [[253.66399, 253.66399, 253.66399, ..., 253.66399, 253.66399,\n", " 253.66399],\n", " [253.62752, 253.62897, 253.62897, ..., 253.62314, 253.6246 ,\n", " 253.62752],\n", " [253.36934, 253.3708 , 253.3708 , ..., 253.35913, 253.36205,\n", " 253.36496],\n", " ...,\n", " [245.29443, 245.29443, 245.29443, ..., 245.29297, 245.29297,\n", " 245.29443],\n", " [245.17773, 245.17773, 245.17773, ..., 245.17773, 245.17773,\n", " 245.17773],\n", " [244.83789, 244.83789, 244.83789, ..., 244.83789, 244.83789,\n", " 244.83789]]], dtype=float32)
array([[[ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " ...,\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan]],\n", "\n", " [[ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", "...\n", " [ 3.7974678e-07, 3.7974678e-07, 3.7974678e-07, ...,\n", " 3.7974678e-07, 3.7974678e-07, 3.7974678e-07],\n", " [ 4.1816384e-07, 4.1816384e-07, 4.1816384e-07, ...,\n", " 4.1816384e-07, 4.1816384e-07, 4.1816384e-07],\n", " [ 4.1816384e-07, 4.1816384e-07, 4.1816384e-07, ...,\n", " 4.1816384e-07, 4.1816384e-07, 4.1816384e-07]],\n", "\n", " [[-1.2330129e-06, -1.2330129e-06, -1.2330129e-06, ...,\n", " -1.2330129e-06, -1.2330129e-06, -1.2330129e-06],\n", " [-7.9527381e-06, -7.9527381e-06, -7.9527381e-06, ...,\n", " -7.9527381e-06, -7.9527381e-06, -7.9527381e-06],\n", " [-1.0064687e-05, -1.0026270e-05, -1.0026270e-05, ...,\n", " -1.0026270e-05, -1.0064687e-05, -1.0064687e-05],\n", " ...,\n", " [ 3.7974678e-07, 3.7974678e-07, 3.7974678e-07, ...,\n", " 3.7974678e-07, 3.7974678e-07, 3.7974678e-07],\n", " [ 4.1816384e-07, 4.1816384e-07, 4.1816384e-07, ...,\n", " 4.1816384e-07, 4.1816384e-07, 4.1816384e-07],\n", " [ 4.9493974e-07, 4.9493974e-07, 4.9493974e-07, ...,\n", " 4.9493974e-07, 4.9493974e-07, 4.9493974e-07]]], dtype=float32)
array([[[ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " ...,\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan]],\n", "\n", " [[ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", "...\n", " [248.91258, 248.91258, 248.91428, ..., 248.9092 , 248.91089,\n", " 248.91089],\n", " [248.76659, 248.76659, 248.76828, ..., 248.7649 , 248.76659,\n", " 248.76659],\n", " [248.3371 , 248.3371 , 248.3371 , ..., 248.3371 , 248.3371 ,\n", " 248.3371 ]],\n", "\n", " [[255.48749, 255.48749, 255.48749, ..., 255.48749, 255.48749,\n", " 255.48749],\n", " [255.23964, 255.23964, 255.24133, ..., 255.23795, 255.23795,\n", " 255.23964],\n", " [254.86955, 254.86955, 254.86955, ..., 254.86447, 254.86617,\n", " 254.86955],\n", " ...,\n", " [248.85316, 248.85486, 248.85486, ..., 248.85147, 248.85147,\n", " 248.85316],\n", " [248.80054, 248.80054, 248.80054, ..., 248.79884, 248.79884,\n", " 248.79884],\n", " [248.4101 , 248.4101 , 248.4101 , ..., 248.4101 , 248.4101 ,\n", " 248.4101 ]]], dtype=float32)
array([[[ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " ...,\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan]],\n", "\n", " [[ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", "...\n", " [248.93002, 248.93002, 248.93172, ..., 248.92493, 248.92833,\n", " 248.92833],\n", " [248.80617, 248.80786, 248.80786, ..., 248.80617, 248.80617,\n", " 248.80617],\n", " [248.399 , 248.399 , 248.399 , ..., 248.399 , 248.399 ,\n", " 248.399 ]],\n", "\n", " [[256.02167, 256.02167, 256.02167, ..., 256.02167, 256.02167,\n", " 256.02167],\n", " [255.89441, 255.89441, 255.8961 , ..., 255.89102, 255.89272,\n", " 255.89272],\n", " [255.61618, 255.61787, 255.61787, ..., 255.60939, 255.61108,\n", " 255.61447],\n", " ...,\n", " [248.90457, 248.90457, 248.90627, ..., 248.90118, 248.90288,\n", " 248.90288],\n", " [248.80786, 248.80957, 248.80957, ..., 248.80786, 248.80786,\n", " 248.80786],\n", " [248.46516, 248.46516, 248.46516, ..., 248.46516, 248.46516,\n", " 248.46516]]], dtype=float32)
array([[[ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " ...,\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan]],\n", "\n", " [[ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", "...\n", " [0. , 0. , 0. , ..., 0. , 0. ,\n", " 0. ],\n", " [0. , 0. , 0. , ..., 0. , 0. ,\n", " 0. ],\n", " [0. , 0. , 0. , ..., 0. , 0. ,\n", " 0. ]],\n", "\n", " [[4.9999847, 4.9999847, 4.9999847, ..., 4.9999847, 4.9999847,\n", " 4.9999847],\n", " [4.9999847, 4.9999847, 4.9999847, ..., 4.9999847, 4.9999847,\n", " 4.9999847],\n", " [4.9999847, 4.9999847, 4.9999847, ..., 4.9999847, 4.9999847,\n", " 4.9999847],\n", " ...,\n", " [0. , 0. , 0. , ..., 0. , 0. ,\n", " 0. ],\n", " [0. , 0. , 0. , ..., 0. , 0. ,\n", " 0. ],\n", " [0. , 0. , 0. , ..., 0. , 0. ,\n", " 0. ]]], dtype=float32)
array([[[nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " ...,\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan]],\n", "\n", " [[nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " ...,\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan]],\n", "\n", " [[nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " ...,\n", "...\n", " ...,\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.]],\n", "\n", " [[ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.]],\n", "\n", " [[ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.]]], dtype=float32)
array([[[ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.]],\n", "\n", " [[ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.]],\n", "\n", " [[ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", "...\n", " ...,\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.]],\n", "\n", " [[ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.]],\n", "\n", " [[ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.],\n", " [10., 10., 10., ..., 10., 10., 10.]]], dtype=float32)
array([[[nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " ...,\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan]],\n", "\n", " [[nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " ...,\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan]],\n", "\n", " [[nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " [nan, nan, nan, ..., nan, nan, nan],\n", " ...,\n", "...\n", " ...,\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.]],\n", "\n", " [[ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.]],\n", "\n", " [[ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.],\n", " [ 0., 0., 0., ..., 0., 0., 0.]]], dtype=float32)
array([[[ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " ...,\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan]],\n", "\n", " [[ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", " [ nan, nan, nan, ..., nan, nan,\n", " nan],\n", "...\n", " [265663.88, 265727.25, 265727.25, ..., 265600.38, 265663.88,\n", " 265663.88],\n", " [267249.75, 267249.75, 267249.75, ..., 267186.25, 267186.25,\n", " 267186.25],\n", " [269025.88, 269025.88, 269025.88, ..., 269025.88, 269025.88,\n", " 269025.88]],\n", "\n", " [[ 0. , 0. , 0. , ..., 0. , 0. ,\n", " 0. ],\n", " [ 0. , 0. , 0. , ..., 0. , 0. ,\n", " 0. ],\n", " [ 0. , 0. , 0. , ..., 0. , 0. ,\n", " 0. ],\n", " ...,\n", " [267186.25, 267186.25, 267249.75, ..., 267059.38, 267122.88,\n", " 267122.88],\n", " [267757.25, 267757.25, 267757.25, ..., 267693.75, 267693.75,\n", " 267757.25],\n", " [268708.75, 268708.75, 268708.75, ..., 268708.75, 268708.75,\n", " 268708.75]]], dtype=float32)
array([[[244.07776, 244.07776, 244.07776, ..., 244.07776, 244.07776,\n", " 244.07776],\n", " [244.01335, 244.01166, 244.00996, ..., 244.01675, 244.01505,\n", " 244.01335],\n", " [243.72865, 243.72357, 243.72018, ..., 243.73544, 243.73373,\n", " 243.73206],\n", " ...,\n", " [249.687 , 249.687 , 249.68869, ..., 249.6853 , 249.6853 ,\n", " 249.687 ],\n", " [249.59209, 249.59209, 249.59209, ..., 249.5887 , 249.5887 ,\n", " 249.5887 ],\n", " [249.28198, 249.28198, 249.28198, ..., 249.28198, 249.28198,\n", " 249.28198]],\n", "\n", " [[244.27432, 244.27432, 244.27432, ..., 244.27432, 244.27432,\n", " 244.27432],\n", " [244.37093, 244.36923, 244.36754, ..., 244.37431, 244.37431,\n", " 244.37093],\n", " [244.13538, 244.13199, 244.12689, ..., 244.14384, 244.14046,\n", " 244.13876],\n", "...\n", " [248.90407, 248.90576, 248.90576, ..., 248.90237, 248.90237,\n", " 248.90407],\n", " [248.80917, 248.80917, 248.80917, ..., 248.80748, 248.80917,\n", " 248.80917],\n", " [248.41263, 248.41263, 248.41263, ..., 248.41263, 248.41263,\n", " 248.41263]],\n", "\n", " [[256.1622 , 256.1622 , 256.1622 , ..., 256.1622 , 256.1622 ,\n", " 256.1622 ],\n", " [256.05203, 256.05203, 256.05203, ..., 256.04697, 256.04697,\n", " 256.05035],\n", " [255.77751, 255.77751, 255.7809 , ..., 255.77074, 255.77412,\n", " 255.77582],\n", " ...,\n", " [248.84645, 248.84645, 248.84645, ..., 248.84306, 248.84476,\n", " 248.84476],\n", " [248.8007 , 248.8007 , 248.8007 , ..., 248.79901, 248.8007 ,\n", " 248.8007 ],\n", " [248.47702, 248.47702, 248.47702, ..., 248.47702, 248.47702,\n", " 248.47702]]], dtype=float32)
array([[[1. , 1. , 1. , ..., 1. ,\n", " 1. , 1. ],\n", " [0.99749744, 0.99749744, 0.9974669 , ..., 0.9974669 ,\n", " 0.9974669 , 0.99749744],\n", " [0.98043734, 0.98046786, 0.98046786, ..., 0.9804984 ,\n", " 0.98046786, 0.98046786],\n", " ...,\n", " [0. , 0. , 0. , ..., 0. ,\n", " 0. , 0. ],\n", " [0.07873896, 0.07867792, 0.0786474 , ..., 0.07861689,\n", " 0.0786474 , 0.07867792],\n", " [0.195001 , 0.195001 , 0.195001 , ..., 0.195001 ,\n", " 0.195001 , 0.195001 ]],\n", "\n", " [[0.98501515, 0.98501515, 0.98501515, ..., 0.98501515,\n", " 0.98501515, 0.98501515],\n", " [1. , 1. , 1. , ..., 1. ,\n", " 1. , 1. ],\n", " [1. , 1. , 1. , ..., 1. ,\n", " 1. , 1. ],\n", "...\n", " [0. , 0. , 0. , ..., 0. ,\n", " 0. , 0. ],\n", " [0. , 0. , 0. , ..., 0. ,\n", " 0. , 0. ],\n", " [0. , 0. , 0. , ..., 0. ,\n", " 0. , 0. ]],\n", "\n", " [[1. , 1. , 1. , ..., 1. ,\n", " 1. , 1. ],\n", " [0.9987487 , 0.9988403 , 0.9989013 , ..., 0.99899286,\n", " 0.9989013 , 0.9988403 ],\n", " [0.9902034 , 0.9902034 , 0.99032545, ..., 0.99050856,\n", " 0.99032545, 0.9902034 ],\n", " ...,\n", " [0. , 0. , 0. , ..., 0. ,\n", " 0. , 0. ],\n", " [0. , 0. , 0. , ..., 0. ,\n", " 0. , 0. ],\n", " [0. , 0. , 0. , ..., 0. ,\n", " 0. , 0. ]]], dtype=float32)
array([[[0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.]],\n", "\n", " [[0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.]],\n", "\n", " [[0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", "...\n", " ...,\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.]],\n", "\n", " [[0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.]],\n", "\n", " [[0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " ...,\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.],\n", " [0., 0., 0., ..., 0., 0., 0.]]], dtype=float32)
array([[[ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " ...,\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan]],\n", "\n", " [[ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", " [ nan, nan, nan, ...,\n", " nan, nan, nan],\n", "...\n", " [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,\n", " 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,\n", " 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,\n", " 0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],\n", "\n", " [[5.2340329e-05, 5.2340329e-05, 5.2340329e-05, ...,\n", " 5.2340329e-05, 5.2340329e-05, 5.2340329e-05],\n", " [8.1703067e-05, 8.1703067e-05, 8.1703067e-05, ...,\n", " 8.2980841e-05, 8.2980841e-05, 8.2980841e-05],\n", " [1.2765825e-04, 1.2765825e-04, 1.2765825e-04, ...,\n", " 1.2893602e-04, 1.2893602e-04, 1.2893602e-04],\n", " ...,\n", " [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,\n", " 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,\n", " 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ...,\n", " 0.0000000e+00, 0.0000000e+00, 0.0000000e+00]]], dtype=float32)
array([[[245.32692, 245.32692, 245.32692, ..., 245.32692, 245.32692,\n", " 245.32692],\n", " [245.93839, 245.9368 , 245.9368 , ..., 245.9368 , 245.93839,\n", " 245.93839],\n", " [246.04855, 246.04376, 246.04216, ..., 246.04855, 246.05016,\n", " 246.04855],\n", " ...,\n", " [247.34654, 247.34973, 247.34973, ..., 247.34334, 247.34494,\n", " 247.34654],\n", " [247.23318, 247.23318, 247.23318, ..., 247.23158, 247.23158,\n", " 247.23158],\n", " [246.87875, 246.87875, 246.87875, ..., 246.87875, 246.87875,\n", " 246.87875]],\n", "\n", " [[245.31894, 245.31894, 245.31894, ..., 245.31894, 245.31894,\n", " 245.31894],\n", " [246.01183, 246.01024, 246.01024, ..., 246.01024, 246.01024,\n", " 246.01024],\n", " [246.12039, 246.1172 , 246.1156 , ..., 246.12039, 246.12039,\n", " 246.12039],\n", "...\n", " [246.88673, 246.88834, 246.88834, ..., 246.88354, 246.88354,\n", " 246.88673],\n", " [246.68877, 246.68877, 246.68877, ..., 246.68716, 246.68716,\n", " 246.68877],\n", " [246.2976 , 246.2976 , 246.2976 , ..., 246.2976 , 246.2976 ,\n", " 246.2976 ]],\n", "\n", " [[255.04984, 255.04984, 255.04984, ..., 255.04984, 255.04984,\n", " 255.04984],\n", " [255.47292, 255.47292, 255.47452, ..., 255.46652, 255.46973,\n", " 255.47131],\n", " [255.45375, 255.45375, 255.45375, ..., 255.44098, 255.44737,\n", " 255.44896],\n", " ...,\n", " [246.87875, 246.88194, 246.88194, ..., 246.87715, 246.87875,\n", " 246.87875],\n", " [246.68398, 246.68398, 246.68558, ..., 246.68237, 246.68237,\n", " 246.68398],\n", " [246.29123, 246.29123, 246.29123, ..., 246.29123, 246.29123,\n", " 246.29123]]], dtype=float32)
array([[[ 0.01948136, 0.01948136, 0.01948136, ..., 0.01948136,\n", " 0.01948136, 0.01948136],\n", " [-5.008356 , -5.020882 , -5.0334086 , ..., -4.956324 ,\n", " -4.974632 , -4.991012 ],\n", " [-6.4324927 , -6.4498367 , -6.4700713 , ..., -6.3486633 ,\n", " -6.3833513 , -6.4112945 ],\n", " ...,\n", " [-1.6031477 , -1.589658 , -1.5674961 , ..., -1.6542163 ,\n", " -1.6320544 , -1.6147103 ],\n", " [-0.9479293 , -0.93443954, -0.92094976, ..., -0.98069024,\n", " -0.9700911 , -0.95852846],\n", " [ 0.5532916 , 0.5532916 , 0.5532916 , ..., 0.5532916 ,\n", " 0.5532916 , 0.5532916 ]],\n", "\n", " [[ 0.11969125, 0.11969125, 0.11969125, ..., 0.11969125,\n", " 0.11969125, 0.11969125],\n", " [-3.1573644 , -3.1718178 , -3.184344 , ..., -3.1072595 ,\n", " -3.1236398 , -3.1409838 ],\n", " [-4.5969176 , -4.6152253 , -4.637387 , ..., -4.513088 ,\n", " -4.547776 , -4.5757194 ],\n", "...\n", " [-4.3512106 , -4.3396482 , -4.3232675 , ..., -4.3858986 ,\n", " -4.371445 , -4.359883 ],\n", " [-3.7788582 , -3.7682593 , -3.7576602 , ..., -3.8010201 ,\n", " -3.7942753 , -3.7875302 ],\n", " [ 0.5475103 , 0.5475103 , 0.5475103 , ..., 0.5475103 ,\n", " 0.5475103 , 0.5475103 ]],\n", "\n", " [[ 0.14378011, 0.14378011, 0.14378011, ..., 0.14378011,\n", " 0.14378011, 0.14378011],\n", " [ 7.647958 , 7.6209784 , 7.593999 , ..., 7.7154074 ,\n", " 7.693245 , 7.6701202 ],\n", " [ 8.090231 , 8.057469 , 8.012182 , ..., 8.202967 ,\n", " 8.158643 , 8.121064 ],\n", " ...,\n", " [-4.363737 , -4.3512106 , -4.332903 , ..., -4.4013157 ,\n", " -4.3858986 , -4.3733726 ],\n", " [-3.7856033 , -3.7740405 , -3.7634413 , ..., -3.8096921 ,\n", " -3.8019836 , -3.7933116 ],\n", " [ 0.52920276, 0.52920276, 0.52920276, ..., 0.52920276,\n", " 0.52920276, 0.52920276]]], dtype=float32)
array([[[ -0.02454072, -0.02454072, -0.02454072, ..., -0.02454072,\n", " -0.02454072, -0.02454072],\n", " [ -6.981851 , -6.967335 , -6.951966 , ..., -7.015152 ,\n", " -7.0040517 , -6.9929514 ],\n", " [ -7.251673 , -7.232034 , -7.206418 , ..., -7.308882 ,\n", " -7.2875357 , -7.2678967 ],\n", " ...,\n", " [ -5.6284714 , -5.636156 , -5.645549 , ..., -5.591755 ,\n", " -5.606271 , -5.619079 ],\n", " [ -5.607979 , -5.613102 , -5.618225 , ..., -5.585778 ,\n", " -5.593463 , -5.6011477 ],\n", " [ 0.1744104 , 0.1744104 , 0.1744104 , ..., 0.1744104 ,\n", " 0.1744104 , 0.1744104 ]],\n", "\n", " [[ 0.03181452, 0.03181452, 0.03181452, ..., 0.03181452,\n", " 0.03181452, 0.03181452],\n", " [ -7.1090775 , -7.0988307 , -7.086877 , ..., -7.131278 ,\n", " -7.123593 , -7.116762 ],\n", " [ -7.4215927 , -7.407077 , -7.3865843 , ..., -7.4617243 ,\n", " -7.4480624 , -7.4344006 ],\n", "...\n", " [ -3.6884851 , -3.7047088 , -3.7234938 , ..., -3.6218836 ,\n", " -3.6474996 , -3.670554 ],\n", " [ -3.362308 , -3.3725543 , -3.3828008 , ..., -3.32303 ,\n", " -3.335838 , -3.3486462 ],\n", " [ 0.18807226, 0.18807226, 0.18807226, ..., 0.18807226,\n", " 0.18807226, 0.18807226]],\n", "\n", " [[ 0.15904075, 0.15904075, 0.15904075, ..., 0.15904075,\n", " 0.15904075, 0.15904075],\n", " [-11.364752 , -11.378414 , -11.392076 , ..., -11.303274 ,\n", " -11.323767 , -11.344259 ],\n", " [-11.157263 , -11.175194 , -11.199102 , ..., -11.059068 ,\n", " -11.099199 , -11.132501 ],\n", " ...,\n", " [ -4.030032 , -4.0454016 , -4.0641866 , ..., -3.9625766 ,\n", " -3.9890466 , -4.011247 ],\n", " [ -3.753379 , -3.7636256 , -3.773872 , ..., -3.7132473 ,\n", " -3.7269092 , -3.7397172 ],\n", " [ 0.22564238, 0.22564238, 0.22564238, ..., 0.22564238,\n", " 0.22564238, 0.22564238]]], dtype=float32)