Buffered iterator for big arrays. `Arrayterator` creates a buffered iterator for reading big arrays in small contiguous blocks. The class is useful for objects stored in the file system. It allows iteration over the object *without* reading everything in memory; instead, small
| 14 | |
| 15 | |
| 16 | class Arrayterator: |
| 17 | """ |
| 18 | Buffered iterator for big arrays. |
| 19 | |
| 20 | `Arrayterator` creates a buffered iterator for reading big arrays in small |
| 21 | contiguous blocks. The class is useful for objects stored in the |
| 22 | file system. It allows iteration over the object *without* reading |
| 23 | everything in memory; instead, small blocks are read and iterated over. |
| 24 | |
| 25 | `Arrayterator` can be used with any object that supports multidimensional |
| 26 | slices. This includes NumPy arrays, but also variables from |
| 27 | Scientific.IO.NetCDF or pynetcdf for example. |
| 28 | |
| 29 | Parameters |
| 30 | ---------- |
| 31 | var : array_like |
| 32 | The object to iterate over. |
| 33 | buf_size : int, optional |
| 34 | The buffer size. If `buf_size` is supplied, the maximum amount of |
| 35 | data that will be read into memory is `buf_size` elements. |
| 36 | Default is None, which will read as many element as possible |
| 37 | into memory. |
| 38 | |
| 39 | Attributes |
| 40 | ---------- |
| 41 | var |
| 42 | buf_size |
| 43 | start |
| 44 | stop |
| 45 | step |
| 46 | shape |
| 47 | flat |
| 48 | |
| 49 | See Also |
| 50 | -------- |
| 51 | ndenumerate : Multidimensional array iterator. |
| 52 | flatiter : Flat array iterator. |
| 53 | memmap : Create a memory-map to an array stored in a binary file on disk. |
| 54 | |
| 55 | Notes |
| 56 | ----- |
| 57 | The algorithm works by first finding a "running dimension", along which |
| 58 | the blocks will be extracted. Given an array of dimensions |
| 59 | ``(d1, d2, ..., dn)``, e.g. if `buf_size` is smaller than ``d1``, the |
| 60 | first dimension will be used. If, on the other hand, |
| 61 | ``d1 < buf_size < d1*d2`` the second dimension will be used, and so on. |
| 62 | Blocks are extracted along this dimension, and when the last block is |
| 63 | returned the process continues from the next dimension, until all |
| 64 | elements have been read. |
| 65 | |
| 66 | Examples |
| 67 | -------- |
| 68 | >>> a = np.arange(3 * 4 * 5 * 6).reshape(3, 4, 5, 6) |
| 69 | >>> a_itor = np.lib.Arrayterator(a, 2) |
| 70 | >>> a_itor.shape |
| 71 | (3, 4, 5, 6) |
| 72 | |
| 73 | Now we can iterate over ``a_itor``, and it will return arrays of size |