dariush-bahrami/character-tokenizer

A character tokenizer for Hugging Face Transformers

Hi there! 👋

Welcome to my GitHub profile! 🚀

About Me 😄

I'm passionate about artificial intelligence and love exploring the fascinating realms of computer vision and natural language processing. 🤖 I have a strong background in developing artificial neural networks and enjoy leveraging my skills to create innovative solutions. 💡

Skills and Expertise 🛠️

• 🧠 Artificial Neural Networks Development
• 👁️ Computer Vision
• 🌐 Natural Language Processing
• 🐍 Python Programming

Experiences 🎓

Here are some of the exciting projects I've worked on:

1. 📷 Image Recognition

2. 📷 Object Detection

3. 📷 Image Segmentation

4. 📷 Generative Adversarial Networks for Image Generation

5. 📷 Augmented Reality using Deep Learning

6. 📚 Sentiment Analysis for Text Classification

7. 📚 Semantic Text Matching

8. 📚 Text Generation

9. 🍏 Health Assessment using Deep Learning

10. 🕷️ Systematic Web Scraping of Dynamic Web Pages

Let's Connect! 🌐

I'm always open to collaborating on exciting projects and exploring new opportunities. Feel free to connect with me on the LinkedIn. Looking forward to connecting with you and making amazing things happen together! 🤝

Thank You! 🙏

Thank you for visiting my GitHub profile. Feel free to explore my repositories and don't forget to leave a ⭐️ if you find something interesting!

Keep learning, keep exploring, and keep coding! 💻✨

character-tokenizer's Issues

Confusion Regarding EOS Token

Thank you for providing this great library. I am relatively new to NLP, therefore it might be that I misunderstood something. Could please explain why [SEP] is assigned instead of [EOS] in the following line?

character-tokenizer/charactertokenizer/core.py

Line 36 in 94a5d5b

eos_token = AddedToken("[SEP]", lstrip=False, rstrip=False)

It seems like the same token is used for <sep> and <eos>. Thank you in advance!