bhupesh-v/tutorialdb

A search 🔎 engine for programming/dev tutorials,

tutorialdb

A search engine for programming/dev tutorials.

About the Project 🔘

• tutorialdb is a small scale search engine for programming/dev tutorials, it is meant to help anyone who is getting started to learn a new technology.
• The sole purpose of tutorialdb is to help people get to resources which might help them learn new things for e.g sometimes there are tutorials on personal blogs which do not get indexed by Google easily.
• All the content (tutorials) is owned by the respective authors/sites.
• tutorialdb maintains its own database saving the links to tutorials and some meta info.

Installation 🔮

1. Create virtual environment.

   Linux/MacOS

   virtualenv -p python3 venv && cd venv && source bin/activate

   Windows (PowerShell)

   py -m venv venv; .\venv\Scripts\activate;

2. Clone the repository.

git clone https://github.com/Bhupesh-V/tutorialdb.git

3. Install dependencies.

pip install -r requirements.txt

4. Set-up virtual environment variables.

   1. Create a file named .env in the root directory & add the following contents.

   SECRET_KEY = 'my-secret-key'
   LOCAL_HOST = 'my-local-ip'
   

   2. For SECRET_KEY use Django Secret Key Generator or Djecrety.
   3. Adding LOCAL_HOST is optional.

5. Migrate tables.

python manage.py migrate

6. Run Tests.

python manage.py test

7. Run the development server.

python manage.py runserver

📝 License

This project is licensed under the MIT License. See the LICENSE.md file for details.

👋 Contributing

Please read the CONTRIBUTING file for the process of submitting pull requests to us.

✨ Contributors

Thanks goes to these wonderful people (emoji key):

MaDDogx 💻 🐛 🤔 👀 📓	Lior Shub 💻 🐛 🎨	José Gabriel Gruber 💻	Artem 💻	Viktor Strate Kløvedal 💻	Chris Shyi 💻	vikneswaran 💻
Adarsh Shetty 📖

This project follows the all-contributors specification. Contributions of any kind welcome!

tutorialdb's Issues

Implementing HIT Count

The hit can be used to query most popular tutorials for the (week, day etc).

adding tutorial count to tags

Is your feature request related to a problem? Please describe.
It would be nice to indicate no.of tutorials on each tag on the Tags Page.

Describe the solution you'd like
A nice counter attached with each tag showing tutorial count.

Issue raised by José Devezas on DEV.to

disabling tags which have no tutorials

Is your feature request related to a problem? Please describe.
Disabling tags which have no tutorials attached with it, this also corresponds to the issue #56 (showing count to 0)

Describe the solution you'd like
Just gray out tags and make them unclickable.

Issue raised by José Devezas on DEV.to

Difficult to Publish tutorials one by one.

Is your feature request related to a problem? Please describe:
It is difficult and time consuming to publish multiple tutorials by going to the specific tutorial in the admin site and publishing it.

Describe the solution you'd like
Add an admin action so that tutorials may be published in bulk.

Additional context
This is promising.

adding language option

Is your feature request related to a problem? Please describe.
Currently we only support adding English based tutorials.It would be nice to have an option to add language while contributing tutorials..

Issue raised by Juraj Ondrej H. on DEV.to

PWA fix

Describe the bug
The PWA feature had been added previously but there seems to be some issue.

Expected behavior
An Add to Home Screen option should appear on android devices.

Update CONTRIBUTING.md.

PRs are being created without specifying what issue it is that they are solving.

Updating CONTRIBUTING.md to say that PRs may be created only if issues exist prior to it may be a possible solution.

For example, PRs #44, #45, #46 were created without any existing issue that they are supposed to solve. PRs #41, #47 were created to solve issue #41.

way to handle tutorial links which are PDF

Describe the bug
Currently if a link is sumitted, the tags are generated automatically.
But the taggie app is not meant to handle PDF links

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Contribute Page'
2. Submit a tutorial which is a link to direct PDF file.
3. See error

Expected behavior
taggie should be able to handle PDF based links

Wrapping REST API into GraphQL

I guess wrapping up the existing REST api into graphql would be good Idea.
For starters we can use python-graphene or django-restql

Improve forms.

Description:
Forms have hard-coded options. I couldn't access the field choices inside the app's views so as to pass them to the context of the required templates. This question looked appropriate and thus I followed the thread and arrived here.

According to the latter, we should remove them according to the DRY principle. This will make it easy to add/remove new categories as needed.

It will also make the queries simpler and no actual strings that are specified in the TUTORIAL_CATEGORIES tuple will have to be used again (either in the shell or otherwise).

Proposed Solution:
Encapsulate the TUTORIAL_CATEGORIES tuple inside the tutorial model.
Like so:
From:

TUTORIAL_CATEGORIES = (
    ('article', 'article'),
    ('video', 'video'),
    ('course', 'course'),
    ('docs', 'docs'),
    ('book', 'book'),
    ('cheatsheet', 'cheatsheet'),
)
.
.
.
class tutorial(models.Model):
    title = models.CharField(max_length=200)
    link = models.URLField()
    tags = models.ManyToManyField(tag)
    category = models.CharField(max_length=200, choices = TUTORIAL_CATEGORIES)
    created_date = models.DateTimeField(default=timezone.now)

    def __str__(self):
        return self.title

To:

class tutorial(models.Model):
    ARTICLE = 'article'
    BOOK = 'book'
    CHEATSHEET = 'cheatsheet'
    COURSE = 'course'
    DOCS = 'docs'
    VIDEO = 'video'

    CATEGORIES = (
        (ARTICLE, 'Article'),
	(BOOK, 'Book'),
	(CHEATSHEET, 'Cheatsheet'),
	(COURSE, 'Course'),
	(DOCS, 'Documentation'),
	(VIDEO, 'Video'),
    )

    title = models.CharField(max_length=200)
    link = models.URLField()
    tags = models.ManyToManyField(tag)
    category = models.CharField(max_length=20, choices=CATEGORIES)
    created_date = models.DateTimeField(default=timezone.now)

    def __str__(self):
        return self.title

Alternatives:
Leave the code as is. Forms remain difficult to update (atleast the selection box does).

Specifying Category in List Views.

Currently when we search without using filter, there is no indication whether it's a video, article etc.

required psycopg2 version does not support python3.8 and above

Describe the bug
As descriped here, psycopg/psycopg2#854, psycopg version <2.8.4 does not support python3.8. Can we upgrade the psycopg version for the repository?
Or specify the dev environment to restrict python version 3.7 and below.

To Reproduce
Steps to reproduce the behavior:

1. Perform installation step with python version 3.8 or above.
2. I tested this in linux environment

Expected behavior
Installation proceeds smoothly. I tried installinng psycopg2.9.x. It works!

Desktop (please complete the following information):

• OS: [Windows using debian WSL]

Moving search bar to Top

Will it be good idea to move the search bar for tutorials to the top ?

README incorrect and other suggestions.

Installation is incorrect. It should be python manage.py migrate instead of python migrate
Please change.

Suggestion:
Windows-specific installation steps could also be included.

Dynamic title

Currently the title of diifferent pages remains the same - tutorialdb.
It would be nice if the title changes dynamically too for e.g tutorialdb - API, tutorialdb - Latest

The button shaped cube should be a button.

Is your feature request related to a problem? Please describe.
When I tried to click on the round green cube, it did nothing. It took a second to realize that I need to click the title.

Describe the solution you'd like
You could make it a button.

Describe alternatives you've considered
If you don't want two buttons that do the same thing, just shift the rounded cubes color to light gray. Or you make it a tag like you do with the language. You could additionally clarify where to click by underlining the title.

Additional context
See the round green rectangle? You can't click that. Minor issue, but a bit annoying.

Error with local setup

Describe the bug
When I tried to set up this application, I got errors related to psycopg2 when I executed the pip install -r requirements.txt.

Expected behavior
A smooth setup process is expected.

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu 20.04.2 LTS
• Browser: Chrome
• Chrome Version 90.0.4430.93 (Official Build) (64-bit)

Additional context
Referred related links: https://stackoverflow.com/questions/65821330/error-failed-building-wheel-for-psycopg2-ubuntu-20-04-python-3-8-5-venv but could not solve the issue.

Can't generate SECRET_KEY.

In CONTRIBUTING.md it says to generate the SECRET_KEY from the django shell, but in fact, you can't run the shell itself unless your SECRET_KEY is set-up.

That's why I suggest to remove the environment variable and put a SECRET_KEY generated prior so that people can generate it on their machine and then use the python-dotenv library to keep their virtual environment variables in a .env file.

Edit:

Also, set Debug to True since it's the development code.

Edit 2:

README.md/CONTRIBUTING.md might have to be updated accordingly.

Separate Branches for Development and Production.

Is your feature request related to a problem? Please describe:
Currently, all changes are being made to the master branch. Setting up is also an issue for new contributors.

Describe the solution you'd like:
A separate branch for development and production.

Additional context:
#52 seems related. Updates to CONTRIBUTING.md and README.md may be required.

Static Files Bug when running 'manage.py test' for setup

Describe the bug
The bug appears after going through the setup; the program is unable to locate the static files root directory. I have tried running python manage.py collectstatic while changing the static file paths which resolves most errors, but still leaves me with a final 404 error:

`FAIL: test_tutorials_page_status_code (api.tests.test_views.APITests)

Traceback (most recent call last):
File "C:\Users\temp\hacktoberfest\tutorialdb\api\tests\test_views.py", line 8, in test_tutorials_page_status_code
self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200`

To Reproduce
Steps to reproduce the behavior:

1. Go through the initial setup steps of creating venv and installing requirements
2. After running the migrate command, run the test command from setup
3. See error

Expected behavior
Status code of 200, asserting page was created correctly

tutorialdb-app.herokuapp.com shows application error. Seems like Procfile misconfigured.

Describe the bug
https://kutt.it/tutorialdb/ redirects to https://tutorialdb-app.herokuapp.com which shows application error. Seems like the Procfile is misconfigured. Required to check logs.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://kutt.it/tutorialdb/'
2. See application error.

Expected behavior
Website should work as expected instead of the server error.

newsletter maybe

In future we can ask users for signing up for a weekly tutorialdb newsletter which may feature top tutorials in the week.
This also corresponds to issue #5 .

Similar word searches

Just thought of this amazing feature:
Providing similar text searches
For e.g when someone searches "csh" we may provide with output:
Did you mean css

Just like search engines do

fix URL error when submitting long links

Describe the bug
If we Contribute a link with more than 200 char length, its throwing up error.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://tutorialdb-app.herokuapp.com/contribute/
2. Paste a legit link with more than 200 character length.
3. Press Submit.
4. See error.

Additional context
Fixing the model tag would fix the issue.

Caching the database queries

Caching the DB queries will vastly improve the performance of this kind of projects
This is a read-heavy web application where the read is pretty more when comparted to write. As data becomes huge the query performance will be very slow. So introducing the caching will be good to scale up this application.

Search Results are getting rendererd when there is no search string

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to Home Page
2. Click on Submit button
3. See error

Expected behavior
Search should not appear until unless there is a Search string

See docs buttons style in API page is not clear

When entering the API page, we can see "See Docs" text. When hovering on the text, it's not clear that it is clickable. Also, the whole row is clickable, which is bad for UX.

Remove duplicate code from app & api

Currently the implementation of updating the db is almost the same in app & api

def post(self, request):
        linkCount = tutorial.objects.filter(link = request.POST['tlink']).count()
        if linkCount == 0:
            tags, title = generateTags(request.POST['tlink'])
            if 'other' in tags:
                return render(request, 'contribute.html', {'error': "Not a Tutorial Link, Try Again"})
            else:
                tutorialObject = tutorial.objects.create(
                    title = title, 
                    link = request.POST['tlink'], 
                    category = request.POST['tcategory']
                )
                for t in tags:
                    obj, created = tag.objects.get_or_create(name=t)

                tagObjList = tag.objects.filter(name__in = tags)
                tutorialObject.tags.set(tagObjList)

A possible solution is to shift this post method in serializers.py.

Templates need a fixin'

Templates could be managed better. Need to follow DRY Principle.

Context names could be more descriptive than "object_list".

Some method-based views can be converted to class based views.

API template is hardcoded (WHY?).

Suggestions:

1. Static pages can be rendered by class based views.
2. If you're using template inheritance, use it throughout the site.
   Edit:
3. Move the search form to the top. It's a search engine FFS.

Tests required.

None of the apps have any tests written. We NEED tests ASAP.

adding version info on about page

It would be nice to add version info and future up-comings in the about page.
The new releases can be fetched through the GitHub API v3.
End-Point : https://api.github.com/repos/Bhupesh-V/tutorialdb/releases/latest

Adding theme colors

This corresponds to the color of your browser tab.
See this for reference.
Make sure the color matches with the color scheme of website header

highlight current page no.

Describe the bug
The current page is not highlighted when we search something.

Expected behavior
The current page should be highlighted/grayed out so that the user doesn't forget the current page number.

Additional context
Reading the documentation of bulma.io may help.

setting up for deployment

The SECRET_KEY which is commented by default may cause problems while deploying.
I propose we uncomment that and leave to user for generating a new key.
I found this online service and one more for creating secret keys.

Other settings will remain same, user will create a .env file & add the key and LOCAL_IP in it

error while adding Tags in tutorial object

In the file api/views.py
The line tutorialObject.tags.set(tagList) is causing the following error.

Internal Server Error: /api/tutorials/
Traceback (most recent call last):
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/rest_framework/decorators.py", line 55, in handler
    return func(*args, **kwargs)
  File "/home/bhupesh/Desktop/tutorialdb-test/tutorialdb/api/views.py", line 97, in tutorials
    tutorialObject.tags.set(tagList)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 997, in set
    self.add(*new_objs, through_defaults=through_defaults)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 938, in add
    through_defaults=through_defaults,
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 1065, in _add_items
    '%s__in' % target_field_name: new_ids,
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/query.py", line 892, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/query.py", line 910, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1290, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1318, in _add_q
    split_subq=split_subq, simple_col=simple_col,
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1251, in build_filter
    condition = self.build_lookup(lookups, col, value)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1116, in build_lookup
    lookup = lookup_class(lhs, rhs)
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/lookups.py", line 20, in __init__
    self.rhs = self.get_prep_lookup()
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/fields/related_lookups.py", line 59, in get_prep_lookup
    self.rhs = [target_field.get_prep_value(v) for v in self.rhs]
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/fields/related_lookups.py", line 59, in <listcomp>
    self.rhs = [target_field.get_prep_value(v) for v in self.rhs]
  File "/home/bhupesh/Desktop/tutorialdb-test/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 966, in get_prep_value
    return int(value)
ValueError: invalid literal for int() with base 10: 'e'

I am trying to send this json repsonse

{
"link":"https://youtu.be/DHvMXvCVQVA",
"category":"video"
}

A way to review submitted links.

Currently as soon as the link is submitted it gets live.
We need a way to not make it live unless until the admin checks it

LOCAL_HOST is not optional

Is your feature request related to a problem? Please describe.
The README.md file specifies that adding LOCAL_HOST is optional but I am facing an issue on my desktop. I can't run the code without adding localhost.

Describe the solution you'd like
I have fixed this issue. I have already sent a PR. #95

Using Pipenv over requirements.txt

Pipenv consolidates and simplifies the development process to a single command line tool.
Pipenv uses pip and virtualenv under the hood but simplifies their usage with a single command line interface.

adding last time a course was updated

Is your feature request related to a problem? Please describe.
A course may get old with its content & relevance.

Additional Context
This issue may not be fixed with the current architecture of the app.
Its will still remain open, you can provide your insights.
Make sure to discuss here before sending any PRs.

Issue raised by Derek on DEV.to

Would love to contribute to this project as a beginner

Is your feature request related to a problem? Please describe.
I have been trying to rank my tech blog on Google but found it really tough. It's great to see that you guys are doing such a amazing job to make the life easier on that regard. As a Django developer myself, I would love to contribute this project.

Describe the solution you'd like
I can help with Documentation, API, or any CRUD operations.

Additional context
Even though I am a beginner in OS projects, I am eagerly waiting to kickstart my journey with Tutorialdb.

https://tutorialdb-app.herokuapp.com/ Shows Application error.

Describe the bug
https://kutt.it/tutorialdb/ redirects to https://tutorialdb-app.herokuapp.com/ which says An error occurred in the application and your page could not be served.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://kutt.it/tutorialdb/'
2. Check your logs for more details.

Expected behavior
The website should work as intended instead of raising server errors.

Screenshots

No redirect in ContributeView.

When I tried to submit an invalid link, instead of the URL being /contribute/ as it should, it is /thankyou/.

Also, shouldn't the site redirect to /thankyou/ after a successful form submission? Instead of a render shouldn't it be an HttpResponseRedirect in the ContributeView's post method? I may be wrong about this, but there is a bug vis a vis the URL.

relevance of search results

Describe the bug
Currently if we search java tutorials, the search results are javascript tutorials.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://tutorialdb-app.herokuapp.com/search/?q=java
2. Scroll down.
3. See error.

Expected behavior
Relevant search results should appear, we have to give preference to Java tutorials instead of JavaScript.

Additional context
Maybe we need to add priorities to tags.

Issue raised by Daiyrbek Artelov on DEV.to

Context names could be more descriptive than "object_list".

AJAX autocomplete search suggestion for better search experience

Is your feature request related to a problem? Please describe.
Thinking of Adding AJAX-based search suggestion for smooth search experience like one we see in google/YouTube.

Describe the solution you'd like
For every keypress(more than 2 characters), we can create an HTTP request and query database and if the result possible we can list them as links so the user can easily navigate through.

Describe alternatives you've considered
Or else we can cache the tags and if characters match with cached tags, we can redirect to the tag page. This will save the database querying.

Additional context
Add any other context or screenshots about the feature request here.

Search by tags 📌

Currently the only way to search by tags is by going to /tags.
It would be nice to enter tags in the search queries itself (just like we add topics in github repos)

Tests failing with 'ModuleNotFoundError: No module named 'django.utils.six''

Describe the bug
A new user following the guide to run the project.
When you try to run the tests as per the setup guide you have the following error
Traceback (most recent call last): File "/root/tutorialdb/app/tests/test_views.py", line 57, in test_tags_page_name response = self.client.get('/tags/') File "/usr/local/lib/python3.7/site-packages/django/test/client.py", line 518, in get response = super().get(path, data=data, secure=secure, **extra) File "/usr/local/lib/python3.7/site-packages/django/test/client.py", line 346, in get **extra, File "/usr/local/lib/python3.7/site-packages/django/test/client.py", line 421, in generic return self.request(**r) File "/usr/local/lib/python3.7/site-packages/django/test/client.py", line 484, in request response = self.handler(environ) File "/usr/local/lib/python3.7/site-packages/django/test/client.py", line 126, in __call__ self.load_middleware() File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 35, in load_middleware middleware = import_string(middleware_path) File "/usr/local/lib/python3.7/site-packages/django/utils/module_loading.py", line 17, in import_string module = import_module(module_path) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/local/lib/python3.7/site-packages/whitenoise/middleware.py", line 5, in <module> from whitenoise.django import DjangoWhiteNoise File "/usr/local/lib/python3.7/site-packages/whitenoise/django.py", line 18, in <module> from django.utils.six.moves.urllib.parse import urlparse ModuleNotFoundError: No module named 'django.utils.six'

To Reproduce
Steps to reproduce the behavior:

1. Ran the project in a docker https://hub.docker.com/_/python
   tag 3.7

2. pip installed the requirements

3. Edited the .env file.

4. Ran python mange.py migrate

5. Ran python manage.py test

6. Got the errors above
   Expected behavior
   To see the test pass.

labeler keeps on failing, issues in workflows

Update CONTRIBUTING.md

The exact changes have been mentioned in the PR #99

Search bar with Autocomplete search suggestions.

I'm always frustrated when using the search bar for searching for something and landing on a page where it shows no results and
then again go back to the search bar for searching it through proper keyword or proper spelling.

**Solution **
The best solution what I thought off was to add up a Autocomplete search suggestion so that if the search engine has the results then it will show the autocomplete suggestion if not it will show not yet added to search engine or something like that. I think it would be very easy to use and it would be very handful for searching.....

I've also added some screen shots.
thank you 😄
Screen shots.docx