WIP - Document the way to write reader and writer plugins (#91)
* 📚 add custom plugin writer * This is an auto-commit, updating project meta data, such as changelog.rst, contributors.rst * 📚 update writer plugin * 📚 update plugin writer and reader documentation * 📚 telling the location of the example codes Co-authored-by: chfw <chfw@users.noreply.github.com>
This commit is contained in:
parent
fa808870d3
commit
9517e0bf49
|
@ -91,6 +91,7 @@ get_data(.., library='pyexcel-ods')
|
|||
csvz
|
||||
sqlalchemy
|
||||
django
|
||||
options
|
||||
extensions
|
||||
|
||||
|
||||
|
|
|
@ -26,16 +26,16 @@ project = 'pyexcel-io'
|
|||
copyright = '2015-2020 Onni Software Ltd.'
|
||||
author = 'chfw'
|
||||
# The short X.Y version
|
||||
version = '0.6.0'
|
||||
version = '0.5.20'
|
||||
# The full version, including alpha/beta/rc tags
|
||||
release = '0.5.20'
|
||||
release = '0.6.0'
|
||||
|
||||
# -- General configuration ---------------------------------------------------
|
||||
|
||||
# Add any Sphinx extension module names here, as strings. They can be
|
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
||||
# ones.
|
||||
extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.viewcode',]
|
||||
extensions = [ 'sphinx.ext.autosummary', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.viewcode', 'sphinx.ext.autodoc',]
|
||||
|
||||
# Add any paths that contain templates here, relative to this directory.
|
||||
templates_path = ['_templates']
|
||||
|
|
|
@ -1,42 +1,129 @@
|
|||
Extend pyexcel-io Tutorial
|
||||
================================================================================
|
||||
|
||||
pyexcel-io itself comes with csv support.
|
||||
|
||||
Reader
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Suppose we have a yaml file, containing a dictionary where the values are
|
||||
two dimensional array. The task is write reader plugin to pyexcel-io so that
|
||||
we can use get_data() to read it out.
|
||||
|
||||
Example yaml data::
|
||||
|
||||
.. literalinclude:: ../../examples/test.yaml
|
||||
:language: yaml
|
||||
|
||||
Example code::
|
||||
|
||||
.. literalinclude:: ../../examples/custom_yeaml_reader.py
|
||||
:language: python
|
||||
|
||||
|
||||
Writer
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
Working with xls, xlsx, and ods formats
|
||||
================================================================================
|
||||
You are welcome extend pyexcel-io to read and write more tabular formats. In
|
||||
github repo, you will find two examples in `examples` folder. This section
|
||||
explains its implementations to help you write yours.
|
||||
|
||||
.. note::
|
||||
|
||||
No longer, you will need to do explicit imports for pyexcel-io extensions.
|
||||
Instead, you install them and manage them via pip.
|
||||
|
||||
Work with physical file
|
||||
Reader
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Suppose we have a yaml file, containing a dictionary where the values are
|
||||
two dimensional array. The task is to write a reader plugin to pyexcel-io so that
|
||||
we can use get_data() to read yaml file out.
|
||||
|
||||
.. literalinclude:: ../../examples/test.yaml
|
||||
:language: yaml
|
||||
|
||||
**Implement IReader**
|
||||
|
||||
First, let's impolement reader interface as below. Three implementations are required:
|
||||
|
||||
1. `content_array` attribute, is expected to be a list of `NamedContent`
|
||||
2. `read_sheet` function, read sheet content by its index.
|
||||
3. `close` function, to clean up any file handle
|
||||
|
||||
.. literalinclude:: ../../examples/custom_yaml_reader.py
|
||||
:language: python
|
||||
:lines: 19-33
|
||||
|
||||
**Implement ISheet**
|
||||
|
||||
`YourSingleSheet` makes this simple task complex in order to show case its inner
|
||||
workings. Two abstract functions require implementation:
|
||||
|
||||
1. `row_iterator`: should return a row: either content arry or content index as long as
|
||||
`column_iterator` understands
|
||||
|
||||
2. `column_iterator`: should return cell values one by one.
|
||||
|
||||
.. literalinclude:: ../../examples/custom_yaml_reader.py
|
||||
:language: python
|
||||
:lines: 8-16
|
||||
|
||||
|
||||
**Plug in pyexcel-io**
|
||||
|
||||
Last thing is to register with pyexcel-io about your new reader. `relative_plugin_class_path`
|
||||
meant reference from current module, how to refer to `YourReader`. `locations` meant
|
||||
the physical presence of the data source: "file", "memory" or "content". "file" means
|
||||
files on physical disk. "memory" means a file stream. "content" means a string buffer.
|
||||
`stream_type` meant the type of the stream: binary for BytesIO and text for StringIO.
|
||||
|
||||
.. literalinclude:: ../../examples/custom_yaml_reader.py
|
||||
:language: python
|
||||
:lines: 36-41
|
||||
|
||||
|
||||
**Test your reader **
|
||||
|
||||
Let's run the following code and see if it works.
|
||||
|
||||
.. literalinclude:: ../../examples/custom_yaml_reader.py
|
||||
:language: python
|
||||
:lines: 43-45
|
||||
|
||||
Writer
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Now for the writer, let's write a pyexcel-io writer that write a dictionary of
|
||||
two dimentaional arrays back into a yaml file seen above.
|
||||
|
||||
** Implement IWriter **
|
||||
|
||||
Two abstract functions are required:
|
||||
|
||||
1. `create_sheet` creates a native sheet by sheet name, that understands how to code up the native sheet. Interestingly, it returns your sheet.
|
||||
2. `close` function closes file handle if any.
|
||||
|
||||
.. literalinclude:: ../../examples/custom_yaml_writer.py
|
||||
:language: python
|
||||
:lines: 18-30
|
||||
|
||||
** Implement ISheetWriter **
|
||||
|
||||
It is imagined that you will have your own sheet writer. You simply need to figure
|
||||
out how to write a row. Row by row write action was already written by `ISheetWrier`.
|
||||
|
||||
|
||||
.. literalinclude:: ../../examples/custom_yaml_writer.py
|
||||
:language: python
|
||||
:lines: 7-14
|
||||
|
||||
**Plug in pyexcel-io**
|
||||
|
||||
Like the reader plugin, we register a writer.
|
||||
|
||||
.. literalinclude:: ../../examples/custom_yaml_writer.py
|
||||
:language: python
|
||||
:lines: 33-38
|
||||
|
||||
**Test It**
|
||||
|
||||
Let's run the following code and please examine `mytest.yaml` yourself.
|
||||
|
||||
.. literalinclude:: ../../examples/custom_yaml_writer.py
|
||||
:language: python
|
||||
:lines: 40-46
|
||||
|
||||
|
||||
|
||||
Other pyexcel-io plugins
|
||||
-----------------------------------------------------------------------------
|
||||
|
||||
Get xls support
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
||||
$ pip install pyexcel-xls
|
||||
|
||||
|
||||
Here's what is needed::
|
||||
|
||||
>>> from pyexcel_io import save_data
|
||||
|
@ -71,7 +158,6 @@ And you can also get the data back::
|
|||
|
||||
The same applies to :meth:`pyexcel_io.get_data`.
|
||||
|
||||
|
||||
Other formats
|
||||
-----------------------------------------------------------------------------
|
||||
|
||||
|
|
|
@ -193,6 +193,7 @@ get_data(.., library='pyexcel-ods')
|
|||
csvz
|
||||
sqlalchemy
|
||||
django
|
||||
options
|
||||
extensions
|
||||
|
||||
|
||||
|
|
|
@ -0,0 +1,46 @@
|
|||
import yaml
|
||||
from pyexcel_io import save_data
|
||||
from pyexcel_io.plugins import IOPluginInfoChainV2
|
||||
from pyexcel_io.plugin_api import IWriter, ISheetWriter
|
||||
|
||||
|
||||
class MySheetWriter(ISheetWriter):
|
||||
def __init__(self, sheet_reference):
|
||||
self.native_sheet = sheet_reference
|
||||
|
||||
def write_row(self, data_row):
|
||||
self.native_sheet.append(data_row)
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
|
||||
class MyWriter(IWriter):
|
||||
def __init__(self, file_name, file_type, **keywords):
|
||||
self.file_name = file_name
|
||||
self.content = {}
|
||||
|
||||
def create_sheet(self, name):
|
||||
array = []
|
||||
self.content[name] = array
|
||||
return MySheetWriter(array)
|
||||
|
||||
def close(self):
|
||||
with open(self.file_name, "w") as f:
|
||||
f.write(yaml.dump(self.content, default_flow_style=False))
|
||||
|
||||
|
||||
IOPluginInfoChainV2(__name__).add_a_writer(
|
||||
relative_plugin_class_path="MyWriter",
|
||||
locations=["file"],
|
||||
file_types=["yaml"],
|
||||
stream_type="text",
|
||||
)
|
||||
|
||||
if __name__ == "__main__":
|
||||
data_dict = {
|
||||
"sheet 1": [[1, 3, 4], [2, 4, 9]],
|
||||
"sheet 2": [["B", "C", "D"]],
|
||||
}
|
||||
|
||||
save_data("mytest.yaml", data_dict)
|
|
@ -2,9 +2,9 @@ overrides: "pyexcel.yaml"
|
|||
project: "pyexcel-io"
|
||||
name: pyexcel-io
|
||||
nick_name: io
|
||||
version: 0.6.0
|
||||
version: 0.5.20
|
||||
current_version: 0.6.0
|
||||
release: 0.5.20
|
||||
release: 0.6.0
|
||||
copyright_year: 2015-2020
|
||||
moban_command: false
|
||||
is_on_conda: true
|
||||
|
@ -31,6 +31,12 @@ keywords:
|
|||
- csvz
|
||||
- django
|
||||
- sqlalchemy
|
||||
sphinx_extensions:
|
||||
- sphinx.ext.autosummary
|
||||
- sphinx.ext.doctest
|
||||
- sphinx.ext.intersphinx
|
||||
- sphinx.ext.viewcode
|
||||
- sphinx.ext.autodoc
|
||||
description: A python library to read and write structured data in csv, zipped csv format and to/from databases
|
||||
python_requires: ">=3.6"
|
||||
min_python_version: "3.6"
|
||||
|
|
|
@ -9,10 +9,13 @@ class IReader(object):
|
|||
"""
|
||||
|
||||
def read_sheet(self, sheet_index) -> ISheet:
|
||||
raise NotImplementedError("")
|
||||
raise NotImplementedError("Read the sheet by index")
|
||||
|
||||
def sheet_names(self):
|
||||
return [content.name for content in self.content_array]
|
||||
|
||||
def __len__(self):
|
||||
return len(self.content_array)
|
||||
|
||||
def close(self):
|
||||
raise NotImplementedError("Close the file")
|
||||
|
|
|
@ -1,9 +1,9 @@
|
|||
class ISheet(object):
|
||||
def row_iterator(self):
|
||||
raise NotImplementedError("")
|
||||
raise NotImplementedError("iterate each row")
|
||||
|
||||
def column_iterator(self, row):
|
||||
raise NotImplementedError("")
|
||||
raise NotImplementedError("iterate each column at a given row")
|
||||
|
||||
|
||||
class ISheetWriter(object):
|
||||
|
@ -16,3 +16,6 @@ class ISheetWriter(object):
|
|||
"""
|
||||
for row in table:
|
||||
self.write_row(row)
|
||||
|
||||
def close(self):
|
||||
raise NotImplementedError("How would you close your file")
|
||||
|
|
6
setup.py
6
setup.py
|
@ -40,7 +40,7 @@ DESCRIPTION = (
|
|||
"format and to/from databases"
|
||||
)
|
||||
URL = "https://github.com/pyexcel/pyexcel-io"
|
||||
DOWNLOAD_URL = "%s/archive/0.5.20.tar.gz" % URL
|
||||
DOWNLOAD_URL = "%s/archive/0.6.0.tar.gz" % URL
|
||||
FILES = ["README.rst", "CHANGELOG.rst"]
|
||||
KEYWORDS = [
|
||||
"python",
|
||||
|
@ -85,8 +85,8 @@ EXTRAS_REQUIRE = {
|
|||
}
|
||||
# You do not need to read beyond this line
|
||||
PUBLISH_COMMAND = "{0} setup.py sdist bdist_wheel upload -r pypi".format(sys.executable)
|
||||
GS_COMMAND = ("gs pyexcel-io v0.5.20 " +
|
||||
"Find 0.5.20 in changelog for more details")
|
||||
GS_COMMAND = ("gs pyexcel-io v0.6.0 " +
|
||||
"Find 0.6.0 in changelog for more details")
|
||||
NO_GS_MESSAGE = ("Automatic github release is disabled. " +
|
||||
"Please install gease to enable it.")
|
||||
UPLOAD_FAILED_MSG = (
|
||||
|
|
|
@ -24,6 +24,10 @@ class TestISheetWriter:
|
|||
def test_write_row(self):
|
||||
self.isheet_writer.write_row([1, 2])
|
||||
|
||||
@raises(NotImplementedError)
|
||||
def test_close(self):
|
||||
self.isheet_writer.close()
|
||||
|
||||
|
||||
class TestIReader:
|
||||
def setUp(self):
|
||||
|
@ -33,6 +37,10 @@ class TestIReader:
|
|||
def test_read_sheet(self):
|
||||
self.ireader.read_sheet(1)
|
||||
|
||||
@raises(NotImplementedError)
|
||||
def test_close(self):
|
||||
self.ireader.close()
|
||||
|
||||
|
||||
class TestIWriter:
|
||||
def setUp(self):
|
||||
|
|
Loading…
Reference in New Issue