Skip to content

ENH: Add calamite engine to read_excel #50581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 48 commits into from
Closed
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
30da9a4
ENH: add calamite excel reader and modify test to include engine
kostyafarber Jan 5, 2023
a47d3fb
Merge branch 'main' into issue-50395
kostyafarber Jan 5, 2023
6c1dd87
Merge branch 'main' into issue-50395
kostyafarber Jan 5, 2023
fd06ad9
fix deps for python-calamine
kostyafarber Jan 5, 2023
8b6200a
Merge branch 'main' into issue-50395
kostyafarber Jan 5, 2023
6a8d822
fix deps for python-calamine, add as pip package
kostyafarber Jan 5, 2023
efcb2fc
ENH: fix typo in engine declaration, add import_optional_dependency, …
kostyafarber Jan 7, 2023
e1105de
Merge branch 'main' into issue-50395
kostyafarber Jan 11, 2023
6b50e0c
calamite -> calamine, updated some tests for calamine
dimastbk Jan 11, 2023
0784733
calamine excel engine: skip tests with datetime
dimastbk Jan 12, 2023
5971199
Merge branch 'main' into issue-50395
kostyafarber Jan 12, 2023
655318b
Merge branch 'issue-50395' into issue-50395-dima
dimastbk Jan 18, 2023
cc049cf
Merge branch 'main' into issue-50395
kostyafarber Jan 20, 2023
038133e
ENH: change reader filename match library, fix typo in engine name in…
kostyafarber Jan 22, 2023
52c2cbd
Merge branch 'issue-50395' into issue-50395-dima
kostyafarber Jan 22, 2023
2dc5e02
Merge pull request #1 from dimastbk/issue-50395-dima
kostyafarber Jan 22, 2023
2076e11
Merge pull request #2 from dimastbk/issue-50395-dima-skip-tests
kostyafarber Jan 22, 2023
6b0a7ac
Merge branch 'main' into issue-50395
kostyafarber Jan 22, 2023
a614089
ENH: add back get_sheet_by_index
kostyafarber Jan 22, 2023
256f9f9
Merge branch 'main' into issue-50395
kostyafarber Jan 23, 2023
eee8b4e
Merge branch 'main' into issue-50395
kostyafarber Jan 23, 2023
9fc2209
ENH: fix mypy and trailing whitespace
kostyafarber Jan 24, 2023
cf1268a
Merge branch 'main' into issue-50395
kostyafarber Jan 24, 2023
bebfec5
Merge branch 'main' into issue-50395
kostyafarber Jan 24, 2023
9019904
added conversion date/time/float, support file_rows_needed, fixed sup…
dimastbk Jan 26, 2023
08a5616
Update test_readers.py
dimastbk Jan 26, 2023
677a224
Merge pull request #3 from dimastbk/issue-50395
kostyafarber Jan 26, 2023
8c55e5d
Merge branch 'main' into issue-50395
kostyafarber Jan 26, 2023
d817999
update python-calamine to 0.0.7
dimastbk Jan 27, 2023
12aaf19
Merge branch 'main' into issue-50395
kostyafarber Jan 28, 2023
255e8fb
Merge pull request #4 from dimastbk/issue-50395
kostyafarber Jan 28, 2023
500fa9f
Merge branch 'main' into issue-50395
kostyafarber Jan 29, 2023
5d94728
fix review: use CalamineReader/CalamineSheet
dimastbk Jan 31, 2023
15874c3
fixed pyright, fixed docs in __init__
dimastbk Feb 6, 2023
89ae49e
Merge pull request #5 from dimastbk/issue-50395
kostyafarber Feb 7, 2023
33e5b7e
Merge branch 'main' into issue-50395
kostyafarber Feb 7, 2023
85d31ec
Merge branch 'main' into issue-50395
kostyafarber Mar 23, 2023
a0d4193
Merge branch 'main' into issue-50395
kostyafarber Mar 25, 2023
a6b6fb2
bump python-calamine to 0.1.0
dimastbk Mar 26, 2023
0a431c5
_ValueT -> _CellValueT
dimastbk Mar 29, 2023
745cd09
Merge pull request #6 from dimastbk/issue-50395
kostyafarber Mar 29, 2023
942a16a
Merge branch 'main' into issue-50395
kostyafarber Apr 1, 2023
8803ca9
Merge branch 'main' into issue-50395
kostyafarber Apr 2, 2023
2f5ffba
added xfail to tests, small fixes
dimastbk Apr 3, 2023
b8b1a9a
Merge pull request #7 from dimastbk/issue-50395
kostyafarber Apr 8, 2023
f5ab40d
Merge branch 'main' into issue-50395
kostyafarber Apr 8, 2023
02c2e7f
bump calamine to 0.1.1, update tests (472 passed, 75 xfailed), update…
dimastbk May 1, 2023
74a3e70
Merge pull request #8 from dimastbk/issue-50395
kostyafarber May 1, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ci/deps/actions-310.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,6 @@ dependencies:
- xlrd
- xlsxwriter
- zstandard

- pip:
- python-calamine
3 changes: 3 additions & 0 deletions ci/deps/actions-38-downstream_compat.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -69,3 +69,6 @@ dependencies:
- pandas-gbq
- pyyaml
- py

- pip:
- python-calamine
1 change: 1 addition & 0 deletions ci/deps/actions-38-minimum_versions.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,4 @@ dependencies:

- pip:
- pyqt5==5.15.1
- python-calamine==0.0.5
4 changes: 4 additions & 0 deletions ci/deps/actions-38.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ dependencies:
- pymysql
- pyreadstat
- pytables
- python-calamine
- python-snappy
- pyxlsb
- s3fs>=2021.08.0
Expand All @@ -54,3 +55,6 @@ dependencies:
- xlrd
- xlsxwriter
- zstandard

- pip:
- python-calamine
3 changes: 3 additions & 0 deletions ci/deps/actions-39.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,6 @@ dependencies:
- xlrd
- xlsxwriter
- zstandard

- pip:
- python-calamine
3 changes: 3 additions & 0 deletions ci/deps/circle-38-arm64.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,6 @@ dependencies:
- xlrd
- xlsxwriter
- zstandard

- pip:
- python-calamine
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v2.0.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ Other enhancements
- Improved error message in :func:`to_datetime` for non-ISO8601 formats, informing users about the position of the first error (:issue:`50361`)
- Improved error message when trying to align :class:`DataFrame` objects (for example, in :func:`DataFrame.compare`) to clarify that "identically labelled" refers to both index and columns (:issue:`50083`)
- Performance improvement in :func:`to_datetime` when format is given or can be inferred (:issue:`50465`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are in the processing of cutting 2.0.0 now; at this point should target 2.1.0 for this PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kostyafarber, hi! Can you merge main in issue-50395?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep will do

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dimastbk merged!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is still in the v2.0.0 whatsnew

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I asked for merging main in pr branch. Now fixed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merged.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should revert any changes to this file - the v2.0.0.rst file shouldn't be touched as part of this PR

-
- Added ``calamite`` as an engine to ``read_excel`` (:issue: ``50395``)

.. ---------------------------------------------------------------------------
.. _whatsnew_200.notable_bug_fixes:
Expand Down
1 change: 1 addition & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,4 @@ dependencies:

- pip:
- sphinx-toggleprompt
- python-calamine
1 change: 1 addition & 0 deletions pandas/compat/_optional.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
"tzdata": "2022.1",
"qtpy": "2.2.0",
"pyqt5": "5.15.1",
"python-calamine": "0.0.5",
}

# A mapping from import name to package name (on PyPI) for packages where
Expand Down
2 changes: 2 additions & 0 deletions pandas/io/excel/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -1435,6 +1435,7 @@ class ExcelFile:
This is not supported, switch to using ``openpyxl`` instead.
"""

from pandas.io.excel._calamitereader import CalamineExcelReader
from pandas.io.excel._odfreader import ODFReader
from pandas.io.excel._openpyxl import OpenpyxlReader
from pandas.io.excel._pyxlsb import PyxlsbReader
Expand All @@ -1445,6 +1446,7 @@ class ExcelFile:
"openpyxl": OpenpyxlReader,
"odf": ODFReader,
"pyxlsb": PyxlsbReader,
"calamite": CalamineExcelReader,
}

def __init__(
Expand Down
69 changes: 69 additions & 0 deletions pandas/io/excel/_calamitereader.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
from __future__ import annotations

from io import (
BufferedReader,
BytesIO,
)
from pathlib import PurePath
from tempfile import NamedTemporaryFile
from typing import Any

from python_calamine import (
get_sheet_data,
get_sheet_names,
)

from pandas.io.excel._base import (
BaseExcelReader,
inspect_excel_format,
)


class __calamine__:
pass


class CalamineExcelReader(BaseExcelReader):
book: str
_sheet_names: list[str] | None = None

@property
def _workbook_class(self) -> type[__calamine__]:
return __calamine__

def load_workbook(
self, filepath_or_buffer: str | PurePath | BufferedReader | BytesIO
) -> str:
if isinstance(filepath_or_buffer, BufferedReader):
filepath_or_buffer = filepath_or_buffer.name

elif isinstance(filepath_or_buffer, BytesIO):
ext = inspect_excel_format(filepath_or_buffer)
with NamedTemporaryFile(suffix=f".{ext}", delete=False) as tmp_file:
tmp_file.write(filepath_or_buffer.getvalue())
filepath_or_buffer = tmp_file.name

elif isinstance(filepath_or_buffer, PurePath):
filepath_or_buffer = filepath_or_buffer.as_posix()

assert isinstance(filepath_or_buffer, str)

self._sheet_names = get_sheet_names(filepath_or_buffer)
return filepath_or_buffer

@property
def sheet_names(self) -> list[str]:
if self._sheet_names is None:
self._sheet_names = get_sheet_names(self.book)
return self._sheet_names

def get_sheet_by_name(self, name: str) -> int:
self.raise_if_bad_sheet_by_name(name)
return self.sheet_names.index(name)

def get_sheet_by_index(self, index: int) -> int:
self.raise_if_bad_sheet_by_index(index)
return index
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume there is more being added here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was ported over from the implementation in the library's repo here, but obvs wasn't implemented by the maintainer.

I'm unsure if this is something the maintainer of the library will add in the future but I can remove this for now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python-calamine isn't returning sheet from calamine (you can use fastexcel if you need sheet). This functions are used only in https://github.com/pandas-dev/pandas/blob/1f836f16b15adc0838afe634df5bda3977b3ad10/pandas/io/excel/_base.py#L733_L736 and working in this case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.


def get_sheet_data(self, sheet: int, convert_float: bool) -> list[list[Any]]:
return get_sheet_data(self.book, sheet)
1 change: 1 addition & 0 deletions pandas/tests/io/excel/test_readers.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
),
pytest.param("pyxlsb", marks=td.skip_if_no("pyxlsb")),
pytest.param("odf", marks=td.skip_if_no("odf")),
pytest.param("calamite"),
]


Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ all = ['beautifulsoup4>=4.9.3',
'pytest>=6.0',
'pytest-xdist>=1.31',
'pytest-asyncio>=0.17.0',
'python-calamine>=0.0.5',
'python-snappy>=0.6.0',
'pyxlsb>=1.0.8',
'qtpy>=2.2.0',
Expand Down
1 change: 1 addition & 0 deletions requirements-dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -87,4 +87,5 @@ feedparser
pyyaml
requests
sphinx-toggleprompt
python-calamine
setuptools>=61.0.0