Add pre-commit to the project

[mrpowers]

Add pre-commit to the project

[robertkossendey]

IMO we should close #67 and use this issue to:

• remove the current flake8 GitHub action
• add a Ruff GitHub action
• add Ruff as a pre-commit hook
• add PyTest as a pre-commit hook

I think that should be done in one PR since it are rather minor changers but I am happy to discuss this.

[MrPowers]

Thanks @ajitkshirsagar. I assigned this one to you.

[SemyonSinchenko]

I suppose that if we are going with Ruff (as discussed in #67) then we should use ruff-pre-commit instead of flake8.

[ajitkshirsagar]

Yes, I agree with @robertkossendey.

[MrPowers]

OK, thanks @ajitkshirsagar. That sounds great. We need to get this done soon to fix the build. If it doesn't get done this week, no problem, but I will have to assign it to someone else. Thank you!

[MrPowers]

@SemyonSinchenko - can you add Ruff, so we can fix the build? The broken build isn't a great look for us.

[SemyonSinchenko]

I guess you should push into your fork instead of this repository. Make a fork; change origin (git remote set-url origin {url-of-your-fork}); push your changes into fork and open a PR from fork to this repo.

[ajitkshirsagar]

I'll update the issue to include Ruff instead of flake8.
@SemyonSinchenko then can we close #67 with the link to this issue?

[SemyonSinchenko]

I'll update the issue to include Ruff instead of flake8. @SemyonSinchenko then can we close #67 with the link to this issue?

I think it is a question to @MrPowers
For me all this things (adding Ruff and adding pre-commit hook + maybe disable some annoying rules like missing docstring in public module) can be done in one PR.

[MrPowers]

@ajitkshirsagar - let's go with @robertkossendey's proposed plan. Are you willing to create a PR to complete the four tasks he mentioned?

[MrPowers]

@ajitkshirsagar - are you going to submit a PR? It's ok if you don't have time. Just let me know so I can assign this to someone else.

[ajitkshirsagar]

Hey @MrPowers, Sorry for taking a bit longer, last weeks were busy for me. I am planning to work on it this week.

[ajitkshirsagar]

@MrPowers @SemyonSinchenko : I am getting following error while running tests with Poetry on local environment.

/root/.cache/pypoetry/virtualenvs/quinn-gGnem25e-py3.9/lib/python3.9/site-packages/py4j/protocol.py:326: in get_return_value
    raise Py4JJavaError(
E   py4j.protocol.Py4JJavaError: An error occurred while calling o0.set.
E   : java.lang.NullPointerException: null key
E   	at org.apache.spark.SparkConf.set(SparkConf.scala:88)
E   	at org.apache.spark.SparkConf.set(SparkConf.scala:83)
E   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
E   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
E   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
E   	at java.lang.reflect.Method.invoke(Method.java:498)
E   	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
E   	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
E   	at py4j.Gateway.invoke(Gateway.java:282)
E   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
E   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
E   	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
E   	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
E   	at java.lang.Thread.run(Thread.java:750)
--------------------------------------------------------------------------------------------- Captured stderr ----------------------------------------------------------------------------------------------
Warning: Ignoring non-Spark config property: None
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================================================= 1 error in 1.16 seconds ======================

Do you guys happen to know? I spent quite some time fixing it but didn't get the lead.

[ajitkshirsagar]

Also, the python version in .python-version is 3.7.5 vs 3.9 used in the CI.

[SemyonSinchenko]

@ajitkshirsagar You have to provide more information about it.

1. Which Java version are you using? (which java, java -version)
2. Is spark-shell working? Try to run poetry run pyspark --master local[1]

But for me it looks like a problem with Java. Make sure that you installed and configured JAVA_HOME correctly. Remember that you should use Java 1,8 (or Java 8).

[ajitkshirsagar]

Thanks, @SemyonSinchenko for the quick reply.
I managed to solve the issue, I think it was some nasty Java issue with my system Apple M1.

I see the following 2 test cases getting failed though I haven't changed anything in them.

describe_validate_schema.it_raises_when_struct_field_is_missing1
test_print_schema_as_code

[ajitkshirsagar]

@MrPowers @robertkossendey : Do you think we should add pytest to the pre-commit as it will make the commit slow also it is flagged by the author of pre-commit(link) that's why pytest doesn't have an inbuilt pre-commit ?

I think we are running pytest as a part of CI so having this check on commit will make it redundant.

Let me know what you think.

[ajitkshirsagar]

@MrPowers need your help with this 🙏

ERROR: Permission to MrPowers/quinn.git denied to ajitkshirsagar.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.