WIP - Document the way to write reader and writer plugins (#91)
* 📚 add custom plugin writer * This is an auto-commit, updating project meta data, such as changelog.rst, contributors.rst * 📚 update writer plugin * 📚 update plugin writer and reader documentation * 📚 telling the location of the example codes Co-authored-by: chfw <chfw@users.noreply.github.com>
This commit is contained in:
parent
fa808870d3
commit
9517e0bf49
|
@ -91,6 +91,7 @@ get_data(.., library='pyexcel-ods')
|
||||||
csvz
|
csvz
|
||||||
sqlalchemy
|
sqlalchemy
|
||||||
django
|
django
|
||||||
|
options
|
||||||
extensions
|
extensions
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -26,16 +26,16 @@ project = 'pyexcel-io'
|
||||||
copyright = '2015-2020 Onni Software Ltd.'
|
copyright = '2015-2020 Onni Software Ltd.'
|
||||||
author = 'chfw'
|
author = 'chfw'
|
||||||
# The short X.Y version
|
# The short X.Y version
|
||||||
version = '0.6.0'
|
version = '0.5.20'
|
||||||
# The full version, including alpha/beta/rc tags
|
# The full version, including alpha/beta/rc tags
|
||||||
release = '0.5.20'
|
release = '0.6.0'
|
||||||
|
|
||||||
# -- General configuration ---------------------------------------------------
|
# -- General configuration ---------------------------------------------------
|
||||||
|
|
||||||
# Add any Sphinx extension module names here, as strings. They can be
|
# Add any Sphinx extension module names here, as strings. They can be
|
||||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
||||||
# ones.
|
# ones.
|
||||||
extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.viewcode',]
|
extensions = [ 'sphinx.ext.autosummary', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.viewcode', 'sphinx.ext.autodoc',]
|
||||||
|
|
||||||
# Add any paths that contain templates here, relative to this directory.
|
# Add any paths that contain templates here, relative to this directory.
|
||||||
templates_path = ['_templates']
|
templates_path = ['_templates']
|
||||||
|
|
|
@ -1,42 +1,129 @@
|
||||||
Extend pyexcel-io Tutorial
|
Extend pyexcel-io Tutorial
|
||||||
================================================================================
|
================================================================================
|
||||||
|
|
||||||
pyexcel-io itself comes with csv support.
|
You are welcome extend pyexcel-io to read and write more tabular formats. In
|
||||||
|
github repo, you will find two examples in `examples` folder. This section
|
||||||
Reader
|
explains its implementations to help you write yours.
|
||||||
--------------------------------------------------------------------------------
|
|
||||||
|
|
||||||
Suppose we have a yaml file, containing a dictionary where the values are
|
|
||||||
two dimensional array. The task is write reader plugin to pyexcel-io so that
|
|
||||||
we can use get_data() to read it out.
|
|
||||||
|
|
||||||
Example yaml data::
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/test.yaml
|
|
||||||
:language: yaml
|
|
||||||
|
|
||||||
Example code::
|
|
||||||
|
|
||||||
.. literalinclude:: ../../examples/custom_yeaml_reader.py
|
|
||||||
:language: python
|
|
||||||
|
|
||||||
|
|
||||||
Writer
|
|
||||||
--------------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Working with xls, xlsx, and ods formats
|
|
||||||
================================================================================
|
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
No longer, you will need to do explicit imports for pyexcel-io extensions.
|
No longer, you will need to do explicit imports for pyexcel-io extensions.
|
||||||
Instead, you install them and manage them via pip.
|
Instead, you install them and manage them via pip.
|
||||||
|
|
||||||
Work with physical file
|
Reader
|
||||||
|
--------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
Suppose we have a yaml file, containing a dictionary where the values are
|
||||||
|
two dimensional array. The task is to write a reader plugin to pyexcel-io so that
|
||||||
|
we can use get_data() to read yaml file out.
|
||||||
|
|
||||||
|
.. literalinclude:: ../../examples/test.yaml
|
||||||
|
:language: yaml
|
||||||
|
|
||||||
|
**Implement IReader**
|
||||||
|
|
||||||
|
First, let's impolement reader interface as below. Three implementations are required:
|
||||||
|
|
||||||
|
1. `content_array` attribute, is expected to be a list of `NamedContent`
|
||||||
|
2. `read_sheet` function, read sheet content by its index.
|
||||||
|
3. `close` function, to clean up any file handle
|
||||||
|
|
||||||
|
.. literalinclude:: ../../examples/custom_yaml_reader.py
|
||||||
|
:language: python
|
||||||
|
:lines: 19-33
|
||||||
|
|
||||||
|
**Implement ISheet**
|
||||||
|
|
||||||
|
`YourSingleSheet` makes this simple task complex in order to show case its inner
|
||||||
|
workings. Two abstract functions require implementation:
|
||||||
|
|
||||||
|
1. `row_iterator`: should return a row: either content arry or content index as long as
|
||||||
|
`column_iterator` understands
|
||||||
|
|
||||||
|
2. `column_iterator`: should return cell values one by one.
|
||||||
|
|
||||||
|
.. literalinclude:: ../../examples/custom_yaml_reader.py
|
||||||
|
:language: python
|
||||||
|
:lines: 8-16
|
||||||
|
|
||||||
|
|
||||||
|
**Plug in pyexcel-io**
|
||||||
|
|
||||||
|
Last thing is to register with pyexcel-io about your new reader. `relative_plugin_class_path`
|
||||||
|
meant reference from current module, how to refer to `YourReader`. `locations` meant
|
||||||
|
the physical presence of the data source: "file", "memory" or "content". "file" means
|
||||||
|
files on physical disk. "memory" means a file stream. "content" means a string buffer.
|
||||||
|
`stream_type` meant the type of the stream: binary for BytesIO and text for StringIO.
|
||||||
|
|
||||||
|
.. literalinclude:: ../../examples/custom_yaml_reader.py
|
||||||
|
:language: python
|
||||||
|
:lines: 36-41
|
||||||
|
|
||||||
|
|
||||||
|
**Test your reader **
|
||||||
|
|
||||||
|
Let's run the following code and see if it works.
|
||||||
|
|
||||||
|
.. literalinclude:: ../../examples/custom_yaml_reader.py
|
||||||
|
:language: python
|
||||||
|
:lines: 43-45
|
||||||
|
|
||||||
|
Writer
|
||||||
|
--------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
Now for the writer, let's write a pyexcel-io writer that write a dictionary of
|
||||||
|
two dimentaional arrays back into a yaml file seen above.
|
||||||
|
|
||||||
|
** Implement IWriter **
|
||||||
|
|
||||||
|
Two abstract functions are required:
|
||||||
|
|
||||||
|
1. `create_sheet` creates a native sheet by sheet name, that understands how to code up the native sheet. Interestingly, it returns your sheet.
|
||||||
|
2. `close` function closes file handle if any.
|
||||||
|
|
||||||
|
.. literalinclude:: ../../examples/custom_yaml_writer.py
|
||||||
|
:language: python
|
||||||
|
:lines: 18-30
|
||||||
|
|
||||||
|
** Implement ISheetWriter **
|
||||||
|
|
||||||
|
It is imagined that you will have your own sheet writer. You simply need to figure
|
||||||
|
out how to write a row. Row by row write action was already written by `ISheetWrier`.
|
||||||
|
|
||||||
|
|
||||||
|
.. literalinclude:: ../../examples/custom_yaml_writer.py
|
||||||
|
:language: python
|
||||||
|
:lines: 7-14
|
||||||
|
|
||||||
|
**Plug in pyexcel-io**
|
||||||
|
|
||||||
|
Like the reader plugin, we register a writer.
|
||||||
|
|
||||||
|
.. literalinclude:: ../../examples/custom_yaml_writer.py
|
||||||
|
:language: python
|
||||||
|
:lines: 33-38
|
||||||
|
|
||||||
|
**Test It**
|
||||||
|
|
||||||
|
Let's run the following code and please examine `mytest.yaml` yourself.
|
||||||
|
|
||||||
|
.. literalinclude:: ../../examples/custom_yaml_writer.py
|
||||||
|
:language: python
|
||||||
|
:lines: 40-46
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Other pyexcel-io plugins
|
||||||
-----------------------------------------------------------------------------
|
-----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
Get xls support
|
||||||
|
|
||||||
|
.. code-block::
|
||||||
|
|
||||||
|
|
||||||
|
$ pip install pyexcel-xls
|
||||||
|
|
||||||
|
|
||||||
Here's what is needed::
|
Here's what is needed::
|
||||||
|
|
||||||
>>> from pyexcel_io import save_data
|
>>> from pyexcel_io import save_data
|
||||||
|
@ -71,7 +158,6 @@ And you can also get the data back::
|
||||||
|
|
||||||
The same applies to :meth:`pyexcel_io.get_data`.
|
The same applies to :meth:`pyexcel_io.get_data`.
|
||||||
|
|
||||||
|
|
||||||
Other formats
|
Other formats
|
||||||
-----------------------------------------------------------------------------
|
-----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
|
@ -193,6 +193,7 @@ get_data(.., library='pyexcel-ods')
|
||||||
csvz
|
csvz
|
||||||
sqlalchemy
|
sqlalchemy
|
||||||
django
|
django
|
||||||
|
options
|
||||||
extensions
|
extensions
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -0,0 +1,46 @@
|
||||||
|
import yaml
|
||||||
|
from pyexcel_io import save_data
|
||||||
|
from pyexcel_io.plugins import IOPluginInfoChainV2
|
||||||
|
from pyexcel_io.plugin_api import IWriter, ISheetWriter
|
||||||
|
|
||||||
|
|
||||||
|
class MySheetWriter(ISheetWriter):
|
||||||
|
def __init__(self, sheet_reference):
|
||||||
|
self.native_sheet = sheet_reference
|
||||||
|
|
||||||
|
def write_row(self, data_row):
|
||||||
|
self.native_sheet.append(data_row)
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class MyWriter(IWriter):
|
||||||
|
def __init__(self, file_name, file_type, **keywords):
|
||||||
|
self.file_name = file_name
|
||||||
|
self.content = {}
|
||||||
|
|
||||||
|
def create_sheet(self, name):
|
||||||
|
array = []
|
||||||
|
self.content[name] = array
|
||||||
|
return MySheetWriter(array)
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
with open(self.file_name, "w") as f:
|
||||||
|
f.write(yaml.dump(self.content, default_flow_style=False))
|
||||||
|
|
||||||
|
|
||||||
|
IOPluginInfoChainV2(__name__).add_a_writer(
|
||||||
|
relative_plugin_class_path="MyWriter",
|
||||||
|
locations=["file"],
|
||||||
|
file_types=["yaml"],
|
||||||
|
stream_type="text",
|
||||||
|
)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
data_dict = {
|
||||||
|
"sheet 1": [[1, 3, 4], [2, 4, 9]],
|
||||||
|
"sheet 2": [["B", "C", "D"]],
|
||||||
|
}
|
||||||
|
|
||||||
|
save_data("mytest.yaml", data_dict)
|
|
@ -2,9 +2,9 @@ overrides: "pyexcel.yaml"
|
||||||
project: "pyexcel-io"
|
project: "pyexcel-io"
|
||||||
name: pyexcel-io
|
name: pyexcel-io
|
||||||
nick_name: io
|
nick_name: io
|
||||||
version: 0.6.0
|
version: 0.5.20
|
||||||
current_version: 0.6.0
|
current_version: 0.6.0
|
||||||
release: 0.5.20
|
release: 0.6.0
|
||||||
copyright_year: 2015-2020
|
copyright_year: 2015-2020
|
||||||
moban_command: false
|
moban_command: false
|
||||||
is_on_conda: true
|
is_on_conda: true
|
||||||
|
@ -31,6 +31,12 @@ keywords:
|
||||||
- csvz
|
- csvz
|
||||||
- django
|
- django
|
||||||
- sqlalchemy
|
- sqlalchemy
|
||||||
|
sphinx_extensions:
|
||||||
|
- sphinx.ext.autosummary
|
||||||
|
- sphinx.ext.doctest
|
||||||
|
- sphinx.ext.intersphinx
|
||||||
|
- sphinx.ext.viewcode
|
||||||
|
- sphinx.ext.autodoc
|
||||||
description: A python library to read and write structured data in csv, zipped csv format and to/from databases
|
description: A python library to read and write structured data in csv, zipped csv format and to/from databases
|
||||||
python_requires: ">=3.6"
|
python_requires: ">=3.6"
|
||||||
min_python_version: "3.6"
|
min_python_version: "3.6"
|
||||||
|
|
|
@ -9,10 +9,13 @@ class IReader(object):
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def read_sheet(self, sheet_index) -> ISheet:
|
def read_sheet(self, sheet_index) -> ISheet:
|
||||||
raise NotImplementedError("")
|
raise NotImplementedError("Read the sheet by index")
|
||||||
|
|
||||||
def sheet_names(self):
|
def sheet_names(self):
|
||||||
return [content.name for content in self.content_array]
|
return [content.name for content in self.content_array]
|
||||||
|
|
||||||
def __len__(self):
|
def __len__(self):
|
||||||
return len(self.content_array)
|
return len(self.content_array)
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
raise NotImplementedError("Close the file")
|
||||||
|
|
|
@ -1,9 +1,9 @@
|
||||||
class ISheet(object):
|
class ISheet(object):
|
||||||
def row_iterator(self):
|
def row_iterator(self):
|
||||||
raise NotImplementedError("")
|
raise NotImplementedError("iterate each row")
|
||||||
|
|
||||||
def column_iterator(self, row):
|
def column_iterator(self, row):
|
||||||
raise NotImplementedError("")
|
raise NotImplementedError("iterate each column at a given row")
|
||||||
|
|
||||||
|
|
||||||
class ISheetWriter(object):
|
class ISheetWriter(object):
|
||||||
|
@ -16,3 +16,6 @@ class ISheetWriter(object):
|
||||||
"""
|
"""
|
||||||
for row in table:
|
for row in table:
|
||||||
self.write_row(row)
|
self.write_row(row)
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
raise NotImplementedError("How would you close your file")
|
||||||
|
|
6
setup.py
6
setup.py
|
@ -40,7 +40,7 @@ DESCRIPTION = (
|
||||||
"format and to/from databases"
|
"format and to/from databases"
|
||||||
)
|
)
|
||||||
URL = "https://github.com/pyexcel/pyexcel-io"
|
URL = "https://github.com/pyexcel/pyexcel-io"
|
||||||
DOWNLOAD_URL = "%s/archive/0.5.20.tar.gz" % URL
|
DOWNLOAD_URL = "%s/archive/0.6.0.tar.gz" % URL
|
||||||
FILES = ["README.rst", "CHANGELOG.rst"]
|
FILES = ["README.rst", "CHANGELOG.rst"]
|
||||||
KEYWORDS = [
|
KEYWORDS = [
|
||||||
"python",
|
"python",
|
||||||
|
@ -85,8 +85,8 @@ EXTRAS_REQUIRE = {
|
||||||
}
|
}
|
||||||
# You do not need to read beyond this line
|
# You do not need to read beyond this line
|
||||||
PUBLISH_COMMAND = "{0} setup.py sdist bdist_wheel upload -r pypi".format(sys.executable)
|
PUBLISH_COMMAND = "{0} setup.py sdist bdist_wheel upload -r pypi".format(sys.executable)
|
||||||
GS_COMMAND = ("gs pyexcel-io v0.5.20 " +
|
GS_COMMAND = ("gs pyexcel-io v0.6.0 " +
|
||||||
"Find 0.5.20 in changelog for more details")
|
"Find 0.6.0 in changelog for more details")
|
||||||
NO_GS_MESSAGE = ("Automatic github release is disabled. " +
|
NO_GS_MESSAGE = ("Automatic github release is disabled. " +
|
||||||
"Please install gease to enable it.")
|
"Please install gease to enable it.")
|
||||||
UPLOAD_FAILED_MSG = (
|
UPLOAD_FAILED_MSG = (
|
||||||
|
|
|
@ -24,6 +24,10 @@ class TestISheetWriter:
|
||||||
def test_write_row(self):
|
def test_write_row(self):
|
||||||
self.isheet_writer.write_row([1, 2])
|
self.isheet_writer.write_row([1, 2])
|
||||||
|
|
||||||
|
@raises(NotImplementedError)
|
||||||
|
def test_close(self):
|
||||||
|
self.isheet_writer.close()
|
||||||
|
|
||||||
|
|
||||||
class TestIReader:
|
class TestIReader:
|
||||||
def setUp(self):
|
def setUp(self):
|
||||||
|
@ -33,6 +37,10 @@ class TestIReader:
|
||||||
def test_read_sheet(self):
|
def test_read_sheet(self):
|
||||||
self.ireader.read_sheet(1)
|
self.ireader.read_sheet(1)
|
||||||
|
|
||||||
|
@raises(NotImplementedError)
|
||||||
|
def test_close(self):
|
||||||
|
self.ireader.close()
|
||||||
|
|
||||||
|
|
||||||
class TestIWriter:
|
class TestIWriter:
|
||||||
def setUp(self):
|
def setUp(self):
|
||||||
|
|
Loading…
Reference in New Issue