Introduce plotting.backend configuration with Plotly support #1639

DumbMachine · 2020-07-08T20:15:39Z

Aims to fix #1626
Each backend returns the figure in their own format, allowing for further editing or customization if required.

How to use?:

import databricks.koalas as ks

kdf = ks.DataFrame([[1, 2, 3, 4], [5, 6, 7, 8]], columns=["A", "B", "C", "D"])
kdf.plot(title="Example Figure") # defaults to backend="matplotlib"

kdf.plot(backend="pandas_bokeh", title="Example Figure")
## same as:
# ks.options.plotting.backend = "pandas_bokeh"
# kdf.plot(title="Example Figure")

fig = kdf.plot(backend="plotly", title="Example Figure", height=500, width=500)
fig.show()

# further edits can be made to the figure
fig.update_layout(template="plotly_dark")
fig.show()

ueshin · 2020-07-08T20:45:57Z

@DumbMachine ~~What version of black are you using? The format seems something wrong.~~
ah, seems like now black check passed.

DumbMachine · 2020-07-08T20:49:12Z

I have the version required by requirements-dev.txt

(base) ➜  koalas git:(plotly-support) ✗ black --version                        
black, version 19.10b0

databricks/koalas/plot.py

ueshin · 2020-07-08T21:36:29Z

databricks/koalas/config.py

@@ -220,6 +220,12 @@ def validate(self, v: Any) -> None:
            "'plotting.sample_ratio' should be 1.0 >= value >= 0.0.",
        ),
    ),
+    Option(
+        key="plotting.backend",
+        doc=("Backend to use for plotting. Default is matplotlib."),


Could you add a list of currently supported backends?

Also check_func to avoid setting invalid backend?

DumbMachine · 2020-07-08T22:21:33Z

I have some trouble understanding the error here. Some tests have problems with trailing ,
and at the same time, some tests recommend to run dev/reformat which has the following effect (adds the trailing ,):

(the image is the result of :git diff plot.py, after running dev/reformat)

ueshin · 2020-07-08T22:28:48Z

Let me try to fix it.

ueshin

black seems to autodetect the python version as per each file, so the files can't include new syntax.

databricks/koalas/plot.py

databricks/koalas/config.py

ueshin · 2020-07-08T23:33:29Z

Btw, I'm wondering how the other backends, pandas_bokeh and pandas_altair?
I tried pandas_bokeh locally, but it shows the same figure as matplotlib. Am I missing something?

HyukjinKwon · 2020-07-09T00:44:47Z

Awesome! @DumbMachine

itholic · 2020-07-09T01:45:24Z

databricks/koalas/plot.py

+    """
+    # function copied from pandas.plotting._core
+
+    import pkg_resources  # Delay import for performance.


Could we move this comment to the above of import pkg_resources to keep the consistency with other comments if there is no special reason ?

itholic · 2020-07-09T01:51:35Z

databricks/koalas/plot.py

+        _backends[entry_point.name] = entry_point.load()
+
+    try:
+        return _backends[backend]


Can we check if "backend" is in "_backends" and raise an KeyError manually rather than using try ??

We use the try because the _backends initially does not have any modules. _backends gets modules assigned to it when the modules are loaded, upon the user's request of the plotting module. The use of:

return _backends[backend]

is that if the use has the module already saved in the _backends dict, there will be no need for reimporting the package. The image below will example better:

@DumbMachine Thanks for the explanation. let's keep this as it as

databricks/koalas/plot.py

itholic · 2020-07-09T02:08:29Z

databricks/koalas/plot.py

@@ -1507,6 +1615,37 @@ class KoalasFramePlotMethods(PandasObject):
    ``df.plot(kind='hist')`` is equivalent to ``df.plot.hist()``
    """

+    @staticmethod
+    def _get_args_map(backend_name, data, kind, kwargs):
+        """Appropriate call args and data preprocessing mapping for the backend


Can we have a simple example for this methods ?

itholic · 2020-07-09T02:10:48Z

@DumbMachine Thanks for the work for this !! 👍

databricks/koalas/plot.py

databricks/koalas/tests/test_series_plot.py

DumbMachine · 2020-07-09T19:33:44Z

The tests fail I believe due to plotly not being installed in the testing environment. What should be done here?

ueshin · 2020-07-09T19:42:38Z

@DumbMachine you can add the libraries you need to the Test category in requirements-dev.txt file.

DumbMachine · 2020-07-09T20:44:37Z

The tests that are failing, fail due to lack of pandas_bokeh. I did add the package to test requirements.

ueshin · 2020-07-09T21:04:32Z

I'll fix the conda builds.

ueshin · 2020-07-09T21:39:19Z

ah, pandas_bokeh is not in conda-forge. maybe we should remove the test with it.

ueshin

Otherwise, LGTM. Awesome work!
@HyukjinKwon @itholic Could you take another look?

databricks/koalas/tests/test_series_plot.py

requirements-dev.txt

databricks/koalas/plot.py

DumbMachine · 2020-07-10T09:46:30Z

Made the changes and all types of arguments are now supported. Please test it out.

databricks/koalas/tests/test_series_plot.py

ueshin · 2020-07-10T18:27:07Z

databricks/koalas/plot.py

+    # Values not being same as default implies user is explicitly passing the arguments
+    for arg, def_val in positional_args:
+        if arg in kwargs and kwargs[arg] == def_val:
+            kwargs.pop(arg, None)


I guess we should apply this before Line 1274, otherwise ploty specific arguments could have different values?

Made the changes. Now we try to map if possible and remove arguements if default.

ueshin · 2020-07-10T18:32:27Z

@DumbMachine Can I ask you another favor while we are here?
Could you update the PR description to have some simple example codes to use the plotting backend and the result figures?

e.g.,:

kdf = ...
kdf.plot(...)

figure

ks.option.plotting.backend = ...
kdf.plot(...)

figure

...

HyukjinKwon

Looks good to me too.

HyukjinKwon · 2020-07-14T10:45:39Z

I will double check by myself tomorrow and make a followup to clean up.
Thank you @DumbMachine for working on this!

itholic · 2020-07-16T05:51:54Z

Sorry for the late.
LGTM, either.
Thanks for the work on this, @DumbMachine 👍

DumbMachine · 2020-07-16T11:59:28Z

Always happy to help.
Is there a reason to explicitly have two different plotter KoalasFramePlotMethods and KoalasSeriesPlotMethods ? Most of the documentation and function calls are same for both plotters. I believe this could be simplified having a single class (similar to PlotAccessor from pandas) as diff functions to plot already exist (plot_series and plot_frame).

ueshin · 2020-07-16T18:29:51Z

@DumbMachine I'm not sure about the original motivation for having two plotter classes, but it'd be great if we could simplify the plotters. Would you like to work on it? cc @HyukjinKwon

DumbMachine · 2020-07-17T03:41:47Z

Yes, I'd like to work on it. I will open a PR for it, soon.

…

On Fri, 17 Jul, 2020, 12:00 am Takuya UESHIN, ***@***.***> wrote: @DumbMachine <https://github.com/DumbMachine> I'm not sure about the original motivation for having two plotter classes, but it'd be great if we could simplify the plotters. Would you like to work on it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1639 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFSMMCD42M6NR5XYOVC6GZ3R35BLBANCNFSM4OU5LA7A> .

HyukjinKwon · 2020-07-20T02:14:14Z

Sure, nice!

DumbMachine added 6 commits July 8, 2020 17:07

implemented plotly functions using PlotAccessor

97b6482

initial/temp implementations

be2de95

plotly plots working

bff1b6a

rephrase docs for the plotting option

2202081

ran dev/reformat

489400c

ran dev/reformat again

0320b06

minor spelling mistake correction

99eb505

ueshin reviewed Jul 8, 2020

View reviewed changes

databricks/koalas/plot.py Outdated Show resolved Hide resolved

removed use of f strings

5b13421

ueshin reviewed Jul 8, 2020

View reviewed changes

databricks/koalas/plot.py Outdated Show resolved Hide resolved

databricks/koalas/plot.py Outdated Show resolved Hide resolved

databricks/koalas/plot.py Outdated Show resolved Hide resolved

databricks/koalas/plot.py Outdated Show resolved Hide resolved

ueshin reviewed Jul 8, 2020

View reviewed changes

DumbMachine added 4 commits July 9, 2020 03:19

minor changes

34c4d69

removed dangling

c9958fa

added options details for plotting backend

5ce6361

removed another dangling

9466324

ueshin reviewed Jul 8, 2020

View reviewed changes

databricks/koalas/plot.py Outdated Show resolved Hide resolved

databricks/koalas/plot.py Outdated Show resolved Hide resolved

databricks/koalas/plot.py Outdated Show resolved Hide resolved

databricks/koalas/config.py Show resolved Hide resolved

itholic reviewed Jul 9, 2020

View reviewed changes

databricks/koalas/plot.py Outdated Show resolved Hide resolved

itholic reviewed Jul 9, 2020

View reviewed changes

databricks/koalas/plot.py Outdated Show resolved Hide resolved

itholic reviewed Jul 9, 2020

View reviewed changes

doc update, removed more , review changes

68bb4fa

ueshin reviewed Jul 9, 2020

View reviewed changes

databricks/koalas/plot.py Outdated Show resolved Hide resolved

ueshin reviewed Jul 9, 2020

View reviewed changes

databricks/koalas/tests/test_series_plot.py Outdated Show resolved Hide resolved

ueshin reviewed Jul 9, 2020

View reviewed changes

databricks/koalas/tests/test_series_plot.py Outdated Show resolved Hide resolved

DumbMachine added 3 commits July 10, 2020 01:21

minor changes

2c46820

indent changes

c9d5dde

reformat changes

fe131f1

DumbMachine added 2 commits July 10, 2020 03:15

removed pandas_bokeh from test plotting backends

2e390a3

removed check_func for plotting.backend

80d1c09

ueshin reviewed Jul 9, 2020

View reviewed changes

better arguments are support

9d307f3

ueshin reviewed Jul 10, 2020

View reviewed changes

minor changes

338b3a9

HyukjinKwon approved these changes Jul 14, 2020

View reviewed changes

HyukjinKwon changed the title ~~Plotly support~~ Introduce plotting.backend configuration with Plotly support Jul 14, 2020

HyukjinKwon merged commit 6361053 into databricks:master Jul 14, 2020

jainayush007 mentioned this pull request Jul 14, 2020

Plotly on koalas dataframes #1626

Closed

DumbMachine mentioned this pull request Jul 17, 2020

Simplify the Plot Assessors #1661

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce plotting.backend configuration with Plotly support #1639

Introduce plotting.backend configuration with Plotly support #1639

DumbMachine commented Jul 8, 2020 •

edited

Loading

ueshin commented Jul 8, 2020 •

edited

Loading

DumbMachine commented Jul 8, 2020

ueshin Jul 8, 2020

ueshin Jul 8, 2020

DumbMachine commented Jul 8, 2020

ueshin commented Jul 8, 2020

ueshin left a comment

ueshin commented Jul 8, 2020

HyukjinKwon commented Jul 9, 2020

itholic Jul 9, 2020

itholic Jul 9, 2020 •

edited

Loading

DumbMachine Jul 9, 2020

itholic Jul 10, 2020

itholic Jul 9, 2020

itholic commented Jul 9, 2020

DumbMachine commented Jul 9, 2020

ueshin commented Jul 9, 2020

DumbMachine commented Jul 9, 2020

ueshin commented Jul 9, 2020

ueshin commented Jul 9, 2020

ueshin left a comment •

edited

Loading

DumbMachine commented Jul 10, 2020

ueshin Jul 10, 2020

DumbMachine Jul 11, 2020

ueshin commented Jul 10, 2020

HyukjinKwon left a comment

HyukjinKwon commented Jul 14, 2020

itholic commented Jul 16, 2020

DumbMachine commented Jul 16, 2020

ueshin commented Jul 16, 2020 •

edited

Loading

DumbMachine commented Jul 17, 2020 via email

HyukjinKwon commented Jul 20, 2020

Introduce plotting.backend configuration with Plotly support #1639

Introduce plotting.backend configuration with Plotly support #1639

Conversation

DumbMachine commented Jul 8, 2020 • edited Loading

ueshin commented Jul 8, 2020 • edited Loading

DumbMachine commented Jul 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DumbMachine commented Jul 8, 2020

ueshin commented Jul 8, 2020

ueshin left a comment

Choose a reason for hiding this comment

ueshin commented Jul 8, 2020

HyukjinKwon commented Jul 9, 2020

Choose a reason for hiding this comment

itholic Jul 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itholic commented Jul 9, 2020

DumbMachine commented Jul 9, 2020

ueshin commented Jul 9, 2020

DumbMachine commented Jul 9, 2020

ueshin commented Jul 9, 2020

ueshin commented Jul 9, 2020

ueshin left a comment • edited Loading

Choose a reason for hiding this comment

DumbMachine commented Jul 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ueshin commented Jul 10, 2020

HyukjinKwon left a comment

Choose a reason for hiding this comment

HyukjinKwon commented Jul 14, 2020

itholic commented Jul 16, 2020

DumbMachine commented Jul 16, 2020

ueshin commented Jul 16, 2020 • edited Loading

DumbMachine commented Jul 17, 2020 via email

HyukjinKwon commented Jul 20, 2020

DumbMachine commented Jul 8, 2020 •

edited

Loading

ueshin commented Jul 8, 2020 •

edited

Loading

itholic Jul 9, 2020 •

edited

Loading

ueshin left a comment •

edited

Loading

ueshin commented Jul 16, 2020 •

edited

Loading