Comments (44)
In summary: we are OK to change this given we know what color to pick. The restriction is that:
- It has to be one color (as we cannot detect the theme the user uses).
- The intention is to dim
missing
a bit (it should not cry-out, rather it should be visually a bit less standing out, as this is the point of having a different color for it)
from dataframes.jl.
Side note: the same issue applies to the column type subheader.
from dataframes.jl.
By the way - can you please confirm how the following renders under the color schemes we discuss? Thank you!
I am asking, because maybe we can use the gray that uses standard Julia in this printout.
from dataframes.jl.
I agree that Solarized seems broken. Given that it also affects Julia (and even more seriously than DataFrames), I'd prefer to find a more general solution there, e.g. which would allow overriding the colors that correspond to each named color. This should ideally be used by Crayons and we wouldn't have anything to do here.
from dataframes.jl.
Thank you for the comment. Do you have a recommendation then?
CC @ronisbr
from dataframes.jl.
Thank for the quick reply!
I think one of the colours should be chosen, as neither background
/foreground
nor black
/dark grey
/light gray
/white
seem to work well for both light an dark themes.
This leaves red
, green
, orange
, blue
, magenta
and cyan
as options. The alternate colours originally were intended as brighter tones (light red
, light green
, yellow
, light blue
, light purple
and teal
), but some themes – for whatever reason – abuse them for completely different colours. Solarized Dark for example uses these places in the colour array to store some dark tones. I don’t know why this is the case, but it seems these cannot be used reliably in every case. So if you chose for example yellow
, then in some themes that text gets flashy, while in Solarized Dark it gets very dark.
Julia seems to mostly use cyan
as accent colour, so maybe this is a good choice? Otherwise orange
and blue
also seem to be a good fit. I’d reserve red
, green
and magenta
for signaling purposes.
This webpage might be helpful.
Here are some theme examples:
from dataframes.jl.
Hi!
Fixing this issue would be extremely simple since we just need to change the color here:
Personally, I have no idea what would be the best option.
from dataframes.jl.
The best option, I guess, would be to select light gray
for dark themes and dark gray
for light themes. But that seems complicated. 😶️
from dataframes.jl.
Hum, there is no universal way to check if the theme is dark or light AFAIK :(
from dataframes.jl.
I agree 100% with @bkamins 's proposal here.
from dataframes.jl.
In summary: we are OK to change this given we know what color to pick. The restriction is that:
1. It has to be one color (as we cannot detect the theme the user uses). 2. The intention is to dim `missing` a bit (it should not cry-out, rather it should be visually a bit less standing out, as this is the point of having a different color for it)
I searched for a bit, but it doesn’t seem possible to tick both boxes. Maybe then hardcoding a medium gray, which is suitable for both theme variants?
from dataframes.jl.
I searched for a bit, but it doesn’t seem possible to tick both boxes. Maybe then hardcoding a medium gray, which is suitable for both theme variants?
Do you mean, setting a RGB color?
from dataframes.jl.
I think it would make sense to hardcode RGB medium gray (I guess it is possible - right?)
from dataframes.jl.
In the shell it sure is, I just don’t know how Julia interfaces with that. Shells use strange color codes like \033[31m
, as you can see in my link above.
from dataframes.jl.
The problem is for terminals that do not support 24-bit colors. I need to test to check what will happen.
from dataframes.jl.
In the shell it sure is, I just don’t know how Julia interfaces with that. Shells use strange color codes like
\033[31m
, as you can see in my link above.
This code says the terminal must change the foreground color to red. In PrettyTables, we use Crayons.jl that is a conversor between a color and those escape sequences.
Maybe we can assume everyone will be using at least a terminal that supports 256 colors. In this case, we have access to a wider range of colors. IMHO, assume 24-bit color is too much restrictive.
In the case of 256 colors, there are those options:
![IMG_4384](https://private-user-images.githubusercontent.com/1068295/281568320-a737ebdf-336c-4aab-886f-697a4a83bf50.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk3MTE2NjksIm5iZiI6MTcwOTcxMTM2OSwicGF0aCI6Ii8xMDY4Mjk1LzI4MTU2ODMyMC1hNzM3ZWJkZi0zMzZjLTRhYWItODg2Zi02OTdhNGE4M2JmNTAucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDMwNiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAzMDZUMDc0OTI5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NDE0MzI1NmJiZTY5ODE0M2ZhYjE2YmM2OGRmN2Q4ZTNhOTBhZGIwYzk5NjZhN2QwZThhODY5YTg2Njk2N2YwMiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.mDhii3_5xmMQxMS8HCIv1V4pqqp7ltH5NR8e85FsGsU)
(from https://www.lihaoyi.com/post/BuildyourownCommandLinewithANSIescapecodes.html)
Thus, we can select a color between 232 and 255.
from dataframes.jl.
You can also substitute :dark_gray
by (127,127,127)
in your referenced code. I just tested this.
from dataframes.jl.
But AFAIK it is converted to a 24-bit color code, which should not work in terminals that do not support it.
from dataframes.jl.
While we’re at this: Is it possible to specify this color as an environment variable or by a function call? I guess that would be nice for circumventing corner cases due to hardcoded values, for example in the rare case anyone has a grey background.
from dataframes.jl.
Yes, it should be fine to check if a ENV variable is defined and use this value instead. Maybe it will add an overhead, but should not be noticeable. What do you think @bkamins ? Any suggestions about the variable name?
from dataframes.jl.
In the past we have had a policy not to introduce behavior depending on global state.
@nalimilan - in this case do you think it would be acceptable?
from dataframes.jl.
Isn't this the same problem as JuliaLang/julia#38730 caused by the fact the solarized theme uses same colours for background and foreground?
from dataframes.jl.
Isn't this the same problem as JuliaLang/julia#38730 caused by the fact the solarized theme uses same colours for background and foreground?
Yup, seems so. Thanks for the tip, now I see that my stack traces contain more information than what is visible. 😅️
from dataframes.jl.
In the past we have had a policy not to introduce behavior depending on global state.
@nalimilan - in this case do you think it would be acceptable?
I would find it kind of weird to have a mechanism to set the color used when printing a particular object (missing
). Isn't there a more general theming system that we could use?
Anyway, regarding this particular issue, requiring users to set an environment variable seems suboptimal. Most of them won't even know they can do that. Is there a way to check whether light_grey
gives the same color as the background, and switch to dark_grey
in that case?
from dataframes.jl.
Unfortunately no. There is no universal way to check what is the color the terminal will use given a ANSI escape sequence.
from dataframes.jl.
In the past we have had a policy not to introduce behavior depending on global state.
@nalimilan - in this case do you think it would be acceptable?I would find it kind of weird to have a mechanism to set the color used when printing a particular object (
missing
). Isn't there a more general theming system that we could use?Anyway, regarding this particular issue, requiring users to set an environment variable seems suboptimal. Most of them won't even know they can do that. Is there a way to check whether
light_grey
gives the same color as the background, and switch todark_grey
in that case?
My proposal was to use medium gray by default and only if that collides with themes (which then are neither light nor dark but rather something in the middle, which I assue to be pretty rare) users are required to manually chose a suitable color for de-highlighting. The problem is, that terminals just define highlighting-colors but no de-highlighting-colors.
from dataframes.jl.
My proposal was to use medium gray by default and only if that collides with themes (which then are neither light nor dark but rather something in the middle, which I assue to be pretty rare) users are required to manually chose a suitable color for de-highlighting. The problem is, that terminals just define highlighting-colors but no de-highlighting-colors.
Notice that this is only possible if we assume a terminal capable of outputting 256 colors since the default 8 colors only contains bright black as gray. IMHO, I think the number of people using terminals that cannot output 256 colors should be minimal. Thus, it should be fine to select one of those colors:
![Captura de Tela 2023-11-13 às 09 33 28](https://private-user-images.githubusercontent.com/1068295/282461340-e746993d-79bd-42db-b5e5-80d5de057e65.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk3MTE2NjksIm5iZiI6MTcwOTcxMTM2OSwicGF0aCI6Ii8xMDY4Mjk1LzI4MjQ2MTM0MC1lNzQ2OTkzZC03OWJkLTQyZGItYjVlNS04MGQ1ZGUwNTdlNjUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDMwNiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAzMDZUMDc0OTI5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YzJhMmRkODRkYTlmZDM2YWYzNWQ3NjhiYTI2MGY1ZTQ0NjRkNmMwZmFiYmI0ODJiY2FiZGM5MDczNTljMDA5NCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.-uOK6-GTajyHozID-hgRgE9BG5De1CUEfY_AnAdN-1M)
Assuming 24-bit and selecting the RGB values is way more restrictive.
@fbruetting can you please test in your terminal what would be a good gray selection for those themes? To do so, open a terminal with your theme paste: echo "\e[38;5;239mmissing"
, changing 239 to the color you are testing.
from dataframes.jl.
Of course, here it is for Solarized Dark and Light: (btw: echo -e
was necessary)
I think 243 provides the best readability for both variants.
from dataframes.jl.
In the "default" themes, we will have (before on top, after on bottom):
![Captura de Tela 2023-11-13 às 10 12 41](https://private-user-images.githubusercontent.com/1068295/282469852-15d85ab6-db64-4ec2-bb5e-b3a3ce2ba43e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk3MTE2NjksIm5iZiI6MTcwOTcxMTM2OSwicGF0aCI6Ii8xMDY4Mjk1LzI4MjQ2OTg1Mi0xNWQ4NWFiNi1kYjY0LTRlYzItYmI1ZS1iM2EzY2UyYmE0M2UucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDMwNiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDAzMDZUMDc0OTI5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NzExODFkYjZlMmFhMmIyOGVhMmY4MjJhMDkwNzNiOGI4ZDE2Y2Y0NmRiZDdkMTk2NDJiNWVjMzhlNzI5NWM3ZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.eaQkW7sWDMLhXlYxqCo7k2AHYRwlbinOHmKiTitDWJM)
IMHO, the highlight will lose its purpose because it almost indistinguishable from the normal text (which usually is light gray instead of pure white).
from dataframes.jl.
Not having a good contrast is much less of a problem than not seeing those values at all, so I’d still prefer hardcoding medium gray.
Here’s a comparison of common themes for the colors 240-247 (all of these are shipped with Tilix):
Solarized Light / Linux / Tango
Solarized Dark / Material / Monokai Dark / Orchis / Yaru / Base16 Twilight dark
from dataframes.jl.
That would lead to invisible text in Solarized Dark, like in the Julia-issue referenced above.
Same order as above:
from dataframes.jl.
What if we took a relatively dark grey like 238, which is not super readable in Solarized Dark but is at least discernible and better than the current situation on that theme? Anyway it's not hard to find out that these entries correspond to missing values, what matters is that you notice the cell isn't completely empty.
from dataframes.jl.
Relevant Solarized bug: altercation/solarized#220
When I read this, there are MASSIVE complications and Solarized is to blame because of intentionally violating the specification. It might be better to let everything be as it is and let Solarized be broken, because when you’re fixing the colors for Solarized, all other themes suffer, because DF then hardcodes values in an environment where all other colors are variable.
Best solution would be to indroduce a config setting, which lets Solarized users alter that color if there’s need, then every other theme would work perfectly and there won’t be compromises at all – just a manual fix for an intentional spec violation. Seems the correct approach to me. Solarized just needs to fix the palette, or needs a fork, obviously.
from dataframes.jl.
Best solution would be to indroduce a config setting, which lets Solarized users alter that color if there’s need, then every other theme would work perfectly and there won’t be compromises at all – just a manual fix for an intentional spec violation. Seems the correct approach to me.
I totally agree. I had problems with solarized themes in the past for a similar reason. Thus, the option is to define a ENV variable that will change the colors in subheaders and in those highlighters.
from dataframes.jl.
Actually, it would be best if Julia would implement that grey-color ENV/setting (if that’s not already possible) and DF to then refer to that Julia-grey-color-setting. Then we’d have a Julia-global solution.
@nalimilan ? 😄
P.S.: Even better would be if Julia could be able to modify that color and pass it on to all packages, then no package has to implement such a reference, but I don’t know if this would even be possible. Should I open up an issue in Julia for that?
from dataframes.jl.
I agree that Solarized seems broken. Given that it also affects Julia (and even more seriously than DataFrames), I'd prefer to find a more general solution there, e.g. which would allow overriding the colors that correspond to each named color. This should ideally be used by Crayons and we wouldn't have anything to do here.
I think this is not the correct approach. Crayons.jl uses the escape sequences given the ANSI definitions. It should not modify the colors. The correct way to replace a color is to set the desired one in your terminal, by changing its theme. Of course, each application (DataFrames in this case) can provide a mechanism to tune the colors it uses if necessary. Since there is no definition of a "theme" in DataFrames because we only use few colors by default, I think an ENV to address this specific problem would not be bad.
from dataframes.jl.
It’s not always possible to modify the color palette. In GNOME Builder you for example can just select a theme but not modify it.
I think we should shrink O(1 + n_pkgs)
to O(1)
and have a global fix in Crayons then, I created an issue there:
from dataframes.jl.
I think we should shrink O(1 + n_pkgs) to O(1) and have a global fix in Crayons then, I created an issue there:
No, it will not be global. There are packages, like Term.jl, (and Julia itself) that just output the escape sequences without using Crayons.jl at all. It will not have the intended behavior of a global fix. IMHO, if a theme is setting something that violates the ANSI specification (making a color equal to background when it just should not happen), the theme must be fixed.
Even inside PrettyTables.jl this will not be global. We have a text cell type called AnsiTextCell
that allows to have decorated strings inside the tables cells. It does not use Crayons.jl, it parses the escape sequences as per ANSI specification. In this case, we will fix the missing
in DataFrames.jl but those other cases will be completely inconsistent.
Allowing Crayons.jl (or anything) to manually change the color can lead to a lot of problems. Example: Package A uses :red
to highlight an error. For some reason Package B overloads :red
to be black. Package A will print invisible error messages. Hence, it would not be a simple task. It needs to be containerized so that each package has its own theme. This kind of thing requires changing Crayons.jl API or using global variables. In summary: it can be done, but it will not be simple, and I think it is really the wrong move due to an error in a theme.
from dataframes.jl.
Btw, I was reading about the fixes in all the backlinks to that issue in solarized theme. Almost all involve changing the theme color in the application.
from dataframes.jl.
Hm, I understood nalimilan so as that the core color definition happens in Crayons and everything else refers to that.
You’re correct that the theme needs a fix, but reality is that this wasn’t fixed in 10 years and yet it’s usage went through the roof – so you can be sure that that won’t happen. There needs a fork to be established, which is a process of at least a decade, so we can’t do anything here, therefore the ugly ENV fix idea.
For fixing the applications it’s the question of which color to put there. Guess that’s arbitrary then. But I also see your point in the logic chain “application ships a broken theme” ⇒ application needs to fix it.
from dataframes.jl.
Yes, this is precisely my point! However, I also agree that solarized is indeed very popular. Hence, IMHO, the optimal approach (given the circumstances) would be an ENV in DataFrames that at the end is selecting the color for specific types of cells.
from dataframes.jl.
But that is O(1 + n_pgks)
and fixes neither Julia (like stacktraces) nor the same issues in other packages. We should strive for a Julia-global fix.
from dataframes.jl.
We should strive for a Julia-global fix.
In this case, I do not agree because the only global fix will be fixing the theme, as many others are doing.
If we change the color definition in Crayons.jl, we fix this issue in DataFrames.jl, but the stack traces will continue to be unreadable.
IMHO, changing the color inside the application is like type-piracy. The terminal you are using is responsible to render the dark gray
color. The dark gray
color is defined by your theme. Hence, we can fix Julia stack traces manually (by changing the color), we can fix Crayons.jl, but any other package that prints raw escape sequences will be wrong.
from dataframes.jl.
Looking at JuliaLang/julia#38730 I realize that Julia already has a workaround via e.g. Base.text_colors[:light_black] = "\e[38;5;243m"
. We could use that value instead of :dark_grey
(Julia doesn't have that color for some reason), given that by default Base.text_colors[:light_black] == "\e[90m"
, i.e. it defaults to the value set by the theme.
I'd argue that the best place to do this would be even more upstream, i.e. in Crayons and similar packages, to ensure consistency. But anyway since this has to be set manually it won't affect users that don't want it so it won't make things worse than the current situation.
from dataframes.jl.
Related Issues (20)
- feature request: allow `skipmissing` column types HOT 3
- `combine` on grouped df return empty df when args is empty HOT 2
- Add rename!(::DataFrame, ::Pair{Regex, SubstitutionString}) method
- GroupBy then combine changes column order HOT 2
- Inconsistent Mean Calculation in Grouped DataFrame Compared to Overall DataFrame HOT 2
- What is the best way to write large DataFrames efficiently and with high performance in Julia while minimizing memory usage? HOT 4
- Segmentation Fault when reading compressed file HOT 1
- Revisit spreading for `AsTable` output` HOT 6
- Better error message when forming a DataFrame from a vector of dictionaries with missing data. HOT 2
- `describe` is slow HOT 3
- CartesianIndex error in Julia 1.11 HOT 4
- `DataFrame(x=Int[], y=Int)` HOT 3
- Add comparison function for dataframes which can handle both isapprox and isequal column types HOT 2
- unique fails with column-type FixedDecimal HOT 5
- mapcols! should modify the parent of a SubDataFrame HOT 11
- Feature request: Pairs in stack HOT 2
- Grouped DataFrame with array elements fails to combine HOT 4
- error when combining a grouped empty dataframe using `first` HOT 6
- Short circuit && on subset? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataframes.jl.