jiansoung/issues-list

记录日常学习和开发遇到的问题。欢迎评论交流：）

issues-list's Issues

"Demo" is from an unidentified developer and differs from previously opened versions. Are you sure you want to open it?

今天把系统更新至 macOS Sonoma 后，使用 Xcode 编译运行项目的时候总会弹出如下的警告窗口：

只要修改项目代码再编译运行，就一定会再次弹出这样的窗口。这很令人烦恼！

Google 了下，发现了一篇文章《macOS Sonoma sandbox security》解释了出现这种情况的原因。

根据我对这篇文章的理解，我发现只要在 TARGETS > Signing & Capabilities 里给项目设置好 Team 和 Signing Certificate 就能解决这个的问题。

设置前：

设置后：

重要的是，你的 Apple ID 并不需要 Enroll Apple Developer Program 才可以解决这个问题。也就说你无需注册缴费成为 Apple 开发者，只要你有 Apple ID 即可。

当然，如果你是缴过费的正式 Apple 开发者，那么选择你对应的 Signing & Capabilities 选项也是可以的。
PS：正式开发者的选项具体是什么我也不知道，毕竟我还没有注册缴费成为 Apple 开发者😂

如果你也遇到这样的问题，那就赶紧试一试 ！😁

have fun :-)

Create a global .gitignore file

You can create a global .gitignore file, which is a list of rules for ignoring files in every Git repository on your computer. For example, you might create the file at ~/.gitignore_global and add some rules to it.

Open Terminal.
Run the following command in your terminal:
git config --global core.excludesFile ~/.gitignore_global

References

• Create a global ignoring files

`brew doctor` - Warning: "config" scripts exist outside your system or Homebrew directories.

Issue

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:
  /Users/songjian/.pyenv/shims/python3.6m-config
  /Users/songjian/.pyenv/shims/python-config
  /Users/songjian/.pyenv/shims/python3-config
  /Users/songjian/.pyenv/shims/python3.6-config

Fix It !

Put the following line into your shell profile (~/.bash_profile or ~/.zshrc)

alias brew="env PATH=${PATH/\/Users\/${USER}\/\.pyenv\/shims:/} brew"

References

1. Macにpyenvを入れたらbrew doctorで怒られるようになったので対応したメモ

pip + there was a problem confirming the ssl certificate

pip + there was a problem confirming the ssl certificate

Issue

  Could not fetch URL https://pypi.python.org/simple/scrapy/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:645) - skipping
  Could not find a version that satisfies the requirement Scrapy
No matching distribution found for Scrapy

Fix

Upgrade pip using:

curl https://bootstrap.pypa.io/get-pip.py | python3

References

• Problem Confirming the SSL Certificate - OSX

pyenv, pip and virtual environment

pyenv, pip and virtual environment

Notes on The Hitchhiker’s Guide to Python

pyenv and how does it work ?

pyenv is a tool to allow multiple versions of the Python interpreter to be installed at the same time.

pyenv works by filling a shims directory with fake versions of the Python interpreter (plus other tools like pip and 2to3). When the system looks for a program named python, it looks inside the shims directory first, and uses the fake version, which in turn passes the command on to pyenv. pyenv then works out which version of Python should be run based on environment variables, .python-version files, and the global default.

pip and virtual environment

In order to make sure that you install packages to your active virtual environment when you use pip install, consider adding the following lines to your ~/.bashrc (or ~/.zshrc) file:

export PIP_REQUIRE_VIRTUALENV=true

gpip() {
    PIP_REQUIRE_VIRTUALENV="" pip "$@"
} 

After saving this change and sourcing the ~/.bashrc file (or ~/.zshrc) with source ~/.bashrc (or source ~/.zshrc), pip will no longer let you install packages if you are not in a virtual environment. (But you can use gpip install packages globally. )

You can also do this configuration by editing your pip.conf or pip.ini file.

References

1. The Hitchhiker’s Guide to Python!
2. https://docs.python-guide.org/dev/pip-virtualenv/#requiring-an-active-virtual-environment-for-pip

Scrapy 简介

Scrapy 简介

Scrapy 是用来爬取网站并抽取结构化数据的应用框架，它对所有请求的调度和处理都是异步的。

基本特性

作为一款优秀的爬虫框架，Scrapy 提供了一些基本特性：

• 设置每个请求间的下载延迟( download delay )
• 限制对每个域名或 IP 的并发请求数

甚至可以使用 AutoThrottle 扩展 来自动调节这些设置。

除了基本特性，Scrapy 还提供了很多强大的特性使得爬取更加简单有效。

更多特性

• 支持用来选择和抽取数据的 CSS 选择器 和 XPath 表达式
• 内置交互式终端( scrapy shell )
• 支持多种数据导出格式(JSON, CSV, XML)和存储后端(FTP, S3, 本地文件系统)
• 内置健壮的编码支持和自动编码探测功能
• 强大的可扩展性：允许使用 signals 和预定义API( middlewares, extensions, pipelines ) 插入自定义功能
• 内置广泛的 middlewares 和 extensions 来处理：
  • cookies 和 session
  • HTTP 压缩、授权、缓存等
  • User-Agent 欺骗
  • robots.txt
  • 爬取深度限制
  • 等等
• 内置用来审查(introspect) 和调试爬虫进程的 Telnet Console
• 以及可重用的 Sitemaps 和 XML/CSV feeds 爬虫、自动下载图片或其他媒体资源的 media pipeline 、caching DNS resolver 等等

安装

推荐在虚拟环境中安装 Scrapy ，这样可以避免可能的包冲突。以 Python 3 为例：

# 安装 scrapy
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip setuptools
pip install ipython scrapy

# 测试 scrapy 是否已经正确安装
scrapy bench

注：如果在 Debian 上安装，需要按如下方式确保依赖完整：

sudo apt-get install python3-venv python3-dev

如果 Scrapy 无法在 Python 3.7 上正常工作，请参考: scrapy/scrapy#3143

总结

Scrapy 是一款集调度、请求、抽取&处理、编码解码、存储、调试&监控( Telnet Console ) 等特性于一体的应用框架。

参考

• Scrapy 1.5 documentation // FIRST STEPS // Scrapy at a glance
• Scrapy 1.5 documentation // FIRST STEPS // Installation guide
• Python 3.7 support #3143

Difference between sh and bash

Difference between sh and bash

OP: https://stackoverflow.com/questions/5725296/difference-between-sh-and-bash

sh & bash

sh is a specification, not an implementation.

bash started as an sh-compatible implementation.

Which one to use

For my own scripts, I prefer sh for the following reasons:

• it is standardized
• it is much simpler and easier to learn
• it is portable across POSIX systems — even if they happen not to have bash, they are required to have sh

References

1. Difference between sh and bash
2. Roman Cheplyaka's answer
3. Roman Cheplyaka

Quick Tutorial for Scrapy

Quick Tutorial for Scrapy

Architecture overview

Creating a project

$ scrapy startproject tutorial

$ tree
tutorial
├── scrapy.cfg
└── tutorial
    ├── __init__.py
    ├── __pycache__
    ├── items.py
    ├── middlewares.py
    ├── pipelines.py
    ├── settings.py
    └── spiders
        ├── __init__.py
        └── __pycache__

4 directories, 7 files

Writing a spider

Spiders must subclass scrapy.Spider and define the initial requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data.

Create a new spider tutorial/spiders/quotes_spider.py:

import scrapy


# Scrapy spiders must subclass `scrapy.Spider`
class QuotesSpider(scrapy.Spider):
    # `name`: identifies the Spider, and must be unique within a project.
    name = "quotes"

    # `start_requests`: must return an iterable of Requests (list or generator)
    # which the Spider will begin to crawl from. Subsequent requests will be
    # generated successively from these initial requests.
    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    # A shortcut to the start_requests method
    # start_urls = [
    #     'http://quotes.toscrape.com/page/1/',
    #     'http://quotes.toscrape.com/page/2/',
    # ]

    # `parse`: a method that will be called to handle the response downloaded
    # for each of the requests made.
    # The `response` parameter is an instance of `TextResponse` that holds the
    # page content and has further helpful methods to handle it.
    # The parse() method usually parses the response, extracting the scraped
    # data as dicts and also finding new URLs to follow and creating new
    # requests (Request) from them.
    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
                'tags': quote.css('div.tags a.tag::text').extract(),
            }

        next_page = response.css('li.next a::attr(href)').extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            # What you see here is Scrapy’s mechanism of following links: when
            # you yield a Request in a callback method, Scrapy will schedule
            # that request to be sent and register a callback method to be
            # executed when that request finishes.
            yield scrapy.Request(next_page, callback=self.parse)

        # A shortcut for creating Requests
        # next_page = response.css('li.next a::attr(href)').extract_first()
        # if next_page is not None:
        #     yield response.follow(next_page, callback=self.parse)

        # Or follow all link
        # for a in response.css('li.next a'):
        #     yield response.follow(a, callback=self.parse)

Run a spider

$ scrapy crawl quotes -o quotes.json

Play with Scrapy shell

$ scrapy shell 'http://quotes.toscrape.com/page/1/'

References

• Scrapy Tutorial

Remove MariaDB (MySQL) Databases Completely on Debian/Ubuntu

Remove MariaDB (MySQL) Databases Completely

Backup database

mysqldump -u {user} -p {database} > /home/$USER/Documents/backup.sql

Remove MariaDB

sudo service mysql stop
sudo apt-get --purge remove "mysql*"
sudo rm -rf /etc/mysql/ 

Remove unused dependencies

sudo apt-get autoremove
sudo apt-get autoclean

Check

sudo updatedb
locate mysql

References

• https://askubuntu.com/a/806441

Versioning In Software Management

Versioning In Software Management

Semantic Versioning

Given a version number MAJOR.MINOR.PATCH, increment the:

• MAJOR version when you make incompatible API changes,
• MINOR version when you add functionality in a backwards-compatible manner, and
• PATCH version when you make backwards-compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Version Specifiers

A version specifier consists of a series of version clauses, separated by commas. For example:

~= 0.9, >= 1.0, != 1.3.4.*, < 2.0

The comparison operator determines the kind of version clause:

• ~=: Compatible release clause
• ==: Version matching clause
• !=: Version exclusion clause
• <=, >=: Inclusive ordered comparison clause
• <, >: Exclusive ordered comparison clause
• ===: Arbitrary equality clause.

References

• https://semver.org
• PEP 440 -- Version Identification and Dependency Specification

Changing git committer info(author & email)

Changing git committer info

Step 0

Open termial & change directory to your repository.

Step 1

Replacing the following variables based on the information you gathered:

• OLD_EMAIL

• CORRECT_NAME

• CORRECT_EMAIL

git filter-branch --env-filter '

OLD_EMAIL="[email protected]"
CORRECT_NAME="Your Correct Name"
CORRECT_EMAIL="[email protected]"

if [ "$GIT_COMMITTER_EMAIL" = "$OLD_EMAIL" ]
then
    export GIT_COMMITTER_NAME="$CORRECT_NAME"
    export GIT_COMMITTER_EMAIL="$CORRECT_EMAIL"
fi
if [ "$GIT_AUTHOR_EMAIL" = "$OLD_EMAIL" ]
then
    export GIT_AUTHOR_NAME="$CORRECT_NAME"
    export GIT_AUTHOR_EMAIL="$CORRECT_EMAIL"
fi
' --tag-name-filter cat -- --branches --tags

Step 2

Copy & Paste the above command to your terminal, and press Enter to run it.

Step 3

If the repository has a remote repository, push the corrected history to remote repository.

git push --force --tags origin 'refs/heads/*'

References

• Chaning author info

pyenv install 3.6.6 - zipimport.ZipImportError: can't decompress data; zlib not available

pyenv install 3.6.6 - zipimport.ZipImportError: can't decompress data; zlib not available

macOS + Homebrew + Pyenv

Issue

$ pyenv install 3.6.6
python-build: use openssl from homebrew
python-build: use readline from homebrew
Installing Python-3.6.6...
python-build: use readline from homebrew

BUILD FAILED (OS X 10.14 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/9s/4dz5cwfj0dv6j_1_k8nlnd5m0000gn/T/python-build.20181028140503.9082
Results logged to /var/folders/9s/4dz5cwfj0dv6j_1_k8nlnd5m0000gn/T/python-build.20181028140503.9082.log

Last 10 log lines:
  File "/private/var/folders/9s/4dz5cwfj0dv6j_1_k8nlnd5m0000gn/T/python-build.20181028140503.9082/Python-3.6.6/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/private/var/folders/9s/4dz5cwfj0dv6j_1_k8nlnd5m0000gn/T/python-build.20181028140503.9082/Python-3.6.6/Lib/ensurepip/__init__.py", line 204, in _main
    default_pip=args.default_pip,
  File "/private/var/folders/9s/4dz5cwfj0dv6j_1_k8nlnd5m0000gn/T/python-build.20181028140503.9082/Python-3.6.6/Lib/ensurepip/__init__.py", line 117, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/private/var/folders/9s/4dz5cwfj0dv6j_1_k8nlnd5m0000gn/T/python-build.20181028140503.9082/Python-3.6.6/Lib/ensurepip/__init__.py", line 27, in _run_pip
    import pip._internal
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [install] Error 1

Fix

Install zlib

brew install zlib

Add the following to your ~/.zshrc (if you use zsh)

the following content partly from the output of brew install zlib

DO NOT FORGET TO ADD ${LDFLAGS}, ${CPPFLAGS}, ${PKG_CONFIG_PATH} !

# For compilers to find zlib you may need to set:
export LDFLAGS="${LDFLAGS} -L/usr/local/opt/zlib/lib"
export CPPFLAGS="${CPPFLAGS} -I/usr/local/opt/zlib/include"

# For pkg-config to find zlib you may need to set:
export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/zlib/lib/pkgconfig"

MySQL / MariaDB: Set Character Set and Collation to UTF8

MySQL / MariaDB: Set Character Set and Collation to UTF8

Never use utf8 in MySQL(MySQL, MariaDB, PerconaDB, etc.)! (use utf8mb4)

Append following lines to your /etc/mysql/my.cnf

[client]
default-character-set=utf8mb4

[mysql]
default-character-set=utf8mb4

[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
skip-character-set-client-handshake

Restart mysql server

service mysql restart

Check

MariaDB [(none)]> show variables like 'char%'; show variables like 'collation%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8mb4                    |
| character_set_connection | utf8mb4                    |
| character_set_database   | utf8mb4                    |
| character_set_filesystem | binary                     |
| character_set_results    | utf8mb4                    |
| character_set_server     | utf8mb4                    |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.01 sec)

+----------------------+--------------------+
| Variable_name        | Value              |
+----------------------+--------------------+
| collation_connection | utf8mb4_general_ci |
| collation_database   | utf8mb4_general_ci |
| collation_server     | utf8mb4_general_ci |
+----------------------+--------------------+
3 rows in set (0.00 sec)

References

• MySQL utf8 vs utf8mb4 – What’s the difference between utf8 and utf8mb4?

• Changing the default character set to UTF-8

• MySQL / MariaDB: Set Character Set and Collation to UTF8

Phony Targets in Makefile

Phony Targets in Makefile

.PHONY

The prerequisites of the special target .PHONY are considered to be phony targets. When it is time to consider such a target, make will run its recipe unconditionally, regardless of whether a file with that name exists or what its last-modification time is.

Phony Targets

A phony target is one that is not really the name of a file; rather it is just a name for a recipe to be executed when you make an explicit request.

There are two reasons to use a phony target: to avoid a conflict with a file of the same name, and to improve performance.

If you write a rule whose recipe will not create the target file, the recipe will be executed every time the target comes up for remaking. Here is an example:

clean:
        rm *.o temp

Because the rm command does not create a file named clean, probably no such file will ever exist. Therefore, the rm command will be executed every time you say make clean.

In this example, the clean target will not work properly if a file named clean is ever created in this directory. Since it has no prerequisites, clean would always be considered up to date and its recipe would not be executed. To avoid this problem you can explicitly declare the target to be phony by making it a prerequisite of the special target .PHONY as follows:

.PHONY: clean
clean:
        rm *.o temp

Once this is done, make clean will run the recipe regardless of whether there is a file named clean.

References

1. Phony Targets
2. What is the purpose of .PHONY in a makefile?
3. https://stackoverflow.com/a/2145605/7950607

git push 时提示输入账号密码，正确输入后依然报错？

一、问题

如下图所示，当执行 git push 推送本地代码至远程仓库时，提示输入账号密码，但按提示输入正确的账号密码后却报错。

二、问题分析

这里面存在两个问题：一是我的 SSH Key 有正确设置，为什么 git 提示输入账号密码而不使用 SSH Key？二是为什么按照提示输入了正确的账号密码后会报错？

问题一：我的 SSH Key 有正确设置，为什么 git 提示输入账号密码而不使用 SSH Key？

对这个问题，我首先检查了 SSH Key 是否有正常设置。如下图，SSH Key 测试通过，也就是说设置是没问题的。

$ ssh -T [email protected]

那为什么 git 不使用 SSH Key，而提示输入账号密码呢？ GitHub 文档(Why is Git always asking for my password?)给出了解释：

If Git prompts you for a username and password every time you try to interact with GitHub, you're probably using the HTTPS clone URL for your repository.

检查 remote URL 发现确实如文档所述当初 clone 使用的是 HTTPS clone URL：

$ git config --get remote.origin.url

至此，第一个问题算是解决了。

总结就是当初 clone 的时候使用的是 HTTPS 的 remote URL，后来 push 也就默认使用了 HTTPS 的方式，和 SSH Key 是否正常设置无关。

问题二：为什么按照提示输入了正确的账号密码后会报错？

原因很简单，GitHub 从2021年8月13日 09:00 PST 开始就不接受使用账户密码对 GitHub.com 进行 git 操作了。

ps：脱离业界太久。。。

As previously announced, starting on August 13, 2021, at 09:00 PST, we will no longer accept account passwords when authenticating Git operations on GitHub.com. Instead, token-based authentication (for example, personal access, OAuth, SSH Key, or GitHub App installation token) will be required for all authenticated Git operations.

链接在这里：Git password authentication is shutting down

三、解决办法

问题既然都已明白了，现在就来解决我的需求 —— 使用 SSH Key 进行 git 操作。

操作很简单，只有两步：

一是设置 SSH Key ，参考GitHub 官方文档 Connecting to GitHub with SSH 。

二是更改仓库的 remote URL （使用 SSH 方式的 remote URL）。

$ git remote set-url origin [your SSH remote URL]

ps: 再次使用 git config --get remote.origin.url 命令可确认已经正确更改。

四、总结

以后 git clone 只使用 SSH remote URL 。

GPG, GnuPG, PGP and OpenPGP

GPG, GnuPG, PGP and OpenPGP

Pretty Good Privacy (PGP)

Pretty Good Privacy is proprietary software and was originally written by Phil Zimmerman in 1991.

OpenPGP

OpenPGP is a protocol. (RFC 4880)

GNU Privacy Guard (GnuPG / GPG)

GNU Privacy Guard, also known as GnuPG and GPG, is the Free Software Foundation’s implementation (licensed under GNU Public Licence v3) of the OpenPGP protocol, written almost entirely by Werner Koch.

What to call my key?

“GPG Key” is a key that specific to GnuPG.
“OpenPGP Key” is the right common name.

References

• The Difference Between GPG, GnuPG, PGP and OpenPGP

String Similarity & Maximum-Weighted Matching in Graphs

String Similarity & Maximum-Weighted Matching in Graphs

The Similarity of two strings s₁, s₂

Maximum-Weighted Matching in Graphs

A matching is a subset of edges in which no node occurs more than once. The weight of a matching is the sum of the weights of its edges. A maximal matching cannot add more edges and still be a matching.

Combine Two Concepts

Graph G: strings as nodes, and similarity of two strings as their edge's weight.

The maximum-weighted matching in the Graph G is the best result of similarity strings pairs between two datasets.

References

• editdistance

• Maximum-Weighted Matching in Graphs

How to install mysqlclient on macOS (Homebrew) with MariaDB for Python ?

# Assume you are activating Python 3 venv
$ brew install mariadb pkg-config mysql-client

# For pkg-config to find mysql-client you may need to set PKG_CONFIG_PATH.
# This path can be found when you brew install mysql-client, replace below with yours.
$ export PKG_CONFIG_PATH="/usr/local/opt/mysql-client/lib/pkgconfig"

$ pip install mysqlclient

References

• mysqlclient ・ PyPI

Set name of current host system for macOS

To keep the hostname between reboots, run
scutil --set HostName name-of-host

Configuring MySQL(NOT MariaDB) Repository

Configuring MySQL(NOT MariaDB) Repository

Check the version here: Download MySQL APT Repository

wget http://repo.mysql.com/mysql-apt-config_0.8.10-1_all.deb

Install the release package

sudo dpkg -i mysql-apt-config_0.8.10-1_all.deb

sudo apt update
sudo apt install mysql-server

Issue & Fix

If you see following errors:

dpkg: error processing package mysql-community-server (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-community-server (= 8.0.13-1debian9); however:
  Package mysql-community-server is not configured yet.

dpkg: error processing package mysql-server (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Errors were encountered while processing:
 mysql-community-server
 mysql-server

Fix

sudo apt-get install -f
sudo systemctl start mysql
sudo systemctl status mysql

Securing MySQL (Optional)

sudo mysql_secure_installation

References

• A Quick Guide to Using the MySQL APT Repository

• How to Install MySQL on Debian 9

pyenv + pipenv quick tutorial

pyenv + pipenv quick tutorial

Python Development Workflow with Pyenv + Pipenv

Prerequisites

• Install pyenv and some python version (I use 3.6.6)

Install Python with pyenv instead of Hombrew (if you use macOS).

NOTE: Do all the following commands in your own project directory.

STEP ONE

Set the shell-specific Python version:

$ pyenv shell 3.6.6

STEP TWO

1. Downgrade pip to 18.0
2. Install pipenv
3. Create a new project using Python 3.6
4. Downgrade virtualenv's pip to 18.0

Why downgrade pip to 18.0 ?

jwodder's answer:

This is a bug in pipenv caused by using it alongside the newest version of pip (18.1): pypa/pipenv#2924. You need to downgrade pip — both inside and outside the pipenv environment — to version 18.0 in order for pipenv to work.

ONLY EXECUTE FOLLOWING COMMANDS AT THE CREATION OF YOUR PROJECT

$ pip install --upgrade pip==18.0  # 1
$ pip install pipenv  # 2
$ pipenv install --python 3.6  # 3
$ pipenv run pip install --upgrade pip==18.0  # 4

STEP THREE

Install packages and use the installed packages

$ pipenv install <package>

Now that package is installed, you can use it in your python script. And then you can run your python script using pipenv run:

$ pipenv run python <your script>

Using $ pipenv run ensures that your installed packages are available to your script.

It’s also possible to spawn a new shell that ensures all commands have access to your installed packages with $ pipenv shell.

More

$ pipenv -h

References

1. https://stackoverflow.com/a/52706841/7950607
2. Running pipenv gives TypeError: 'module' object is not callable
3. https://github.com/pypa/pipenv

How To Add, Delete, and Grant Sudo Privileges to Users on Debian

How To Add, Delete, and Grant Sudo Privileges to Users on Debian

Login as a root user

su - root

or via ssh

ssh root@your-server

Add a user

adduser your_new_user_name

Granting users administrative privileges

usermod -aG sudo your_new_user_name

Delete a user

deluser --remove-home your_new_user_name

References

• How To Add, Delete, and Grant Sudo Privileges to Users on a Debian VPS

Notes on JSON Web Token

Notes on JSON Web Token

JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed.

When should you use JSON Web Tokens?

• Authorization (e.g. Single Sign On)
• Information Exchange

What is the JSON Web Token structure?

<Header>.<Payload>.<Signature>

Header

The header is a JSON object, and is Base64Url encoded, typically consists of two parts:

• typ: the type of the token, which is JWT
• alg: the hashing algorithm being used

Payload

The payload (JSON object) contains the claims, which are statements about an entity (typically, the user) and additional data. And also be Base64Url encoded.

Types of claims:

• Registered: iss (issuer), exp (expiration time), etc.
• Public: These can be defined at will by those using JWTs.
• Private: created to share information between parties.

Do not put secret information in the payload or header elements of a JWT unless it is encrypted.

Signature

To create the signature part you have to take the encoded header, the encoded payload, a secret, the algorithm specified in the header, and sign that.

Create signature with the HMAC SHA256 algorithm:

HMACSHA256(
	base64UrlEncode(header) + "." +
	base64UrlEncode(payload),
	secret
)

The signature is used to verify the message wasn't changed along the way, and, in the case of tokens signed with a private key, it can also verify that the sender of the JWT is who it says it is.

How do JSON Web Tokens work?

Typically client sends the JWT in the Authorization header using the Bearer schema.

Authorization: Bearer <token>

This can be, in certain cases, a stateless authorization mechanism.

If the token is sent in the Authorization header, Cross-Origin Resource Sharing (CORS) won't be an issue as it doesn't use cookies.

References

• Introduction to JSON Web Tokens

Building Custom Comparators

Building Custom Comparators

The hybrid_property.comparator() decorator introduced in this section replaces the use of the hybrid_property.expression() decorator. They cannot be used together.

The example class below allows case-insensitive comparisons on the attribute named word_insensitive:

from sqlalchemy.ext.hybrid import Comparator, hybrid_property
from sqlalchemy import func, Column, Integer, String
from sqlalchemy.orm import Session
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class CaseInsensitiveComparator(Comparator):
    def __eq__(self, other):
        return func.lower(self.__clause_element__()) == func.lower(other)

class SearchWord(Base):
    __tablename__ = 'searchword'
    id = Column(Integer, primary_key=True)
    word = Column(String(255), nullable=False)

    @hybrid_property
    def word_insensitive(self):
        return self.word.lower()

    @word_insensitive.comparator
    def word_insensitive(cls):
        return CaseInsensitiveComparator(cls.word)

Above, SQL expressions against word_insensitive will apply the LOWER() SQL function to both sides:

>>> print Session().query(SearchWord).filter_by(word_insensitive="Trucks")
SELECT searchword.id AS searchword_id, searchword.word AS searchword_word
FROM searchword
WHERE lower(searchword.word) = lower(:lower_1)

The CaseInsensitiveComparator above implements part of the ColumnOperators interface. A “coercion” operation like lowercasing can be applied to all comparison operations (i.e. eq, lt, gt, etc.) using Operators.operate():

class CaseInsensitiveComparator(Comparator):
    def operate(self, op, other):
        return op(func.lower(self.__clause_element__()), func.lower(other))

References:

1. Building Custom Comparators

pyenv + matplotlib

pyenv + matplotlib

macOS 10.14, Homebrew 1.8.0, pyenv 1.2.7

Step by Step

Install tcl-tk before pyenv install

brew install tcl-tk

Install Python (e.g. 3.7.0) with pyenv

# VERY IMPORTANT!!!
env PYTHON_CONFIGURE_OPTS="--enable-framework CC=clang" pyenv install 3.7.0

Set local python version, create & activate virtual environment

pyenv local 3.7.0

# ONLY `venv`, DO NOT USE `virtualenv`.
python -m venv .venv

source .venv/bin/activate

If you do not install python with env PYTHON_CONFIGURE_OPTS="--enable-framework CC=clang" or not create virtual environment with python -m venv :

ImportError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.

Install matplotlib

pip install -U pip matplotlib

you can now work with matplotlib happily ! 🎊

TODO

tkinter's problem

python -m tkinter -c 'tkinter._test()'
Traceback (most recent call last):
  File "/Users/songjian/.pyenv/versions/3.7.0/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/songjian/.pyenv/versions/3.7.0/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/songjian/.pyenv/versions/3.7.0/Python.framework/Versions/3.7/lib/python3.7/tkinter/__main__.py", line 7, in <module>
    main()
  File "/Users/songjian/.pyenv/versions/3.7.0/Python.framework/Versions/3.7/lib/python3.7/tkinter/__init__.py", line 3985, in _test
    root = Tk()
  File "/Users/songjian/.pyenv/versions/3.7.0/Python.framework/Versions/3.7/lib/python3.7/tkinter/__init__.py", line 2022, in __init__
    self._loadtk()
  File "/Users/songjian/.pyenv/versions/3.7.0/Python.framework/Versions/3.7/lib/python3.7/tkinter/__init__.py", line 2037, in _loadtk
    % (_tkinter.TK_VERSION, tk_version))
RuntimeError: tk.h version (8.6) doesn't match libtk.a version (8.5)

References

• Working with Matplotlib on OSX

• Matplotlib Installation

• pyenv virtualenv not work for matplotlib #issuecomment-418515459

Using tox with pyenv

Using tox with pyenv

I want to use tox with multiple python versions, so I can test my code in multiple python versions. And I did it. 👇

$ pyenv global 2.7.15 3.4.9 3.5.6 3.6.6 3.7.0
$ pyenv versions                             
  system
* 2.7.15 (set by /Users/songjian/.pyenv/version)
* 3.4.9 (set by /Users/songjian/.pyenv/version)
* 3.5.6 (set by /Users/songjian/.pyenv/version)
* 3.6.6 (set by /Users/songjian/.pyenv/version)
* 3.7.0 (set by /Users/songjian/.pyenv/version)

$ tox

Now, everything works as I expected. :-)

References

• Using tox with pyenv
• pyenv/pyenv#92 (comment)

mysqladmin: unable to change password; error: 'Table 'mysql.role_edges' doesn't exist'

mysqladmin: unable to change password; error: 'Table 'mysql.role_edges' doesn't exist'

issue

$ mysqladmin -u root password 'your password'  # setup mysql root password
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety.
mysqladmin: unable to change password; error: 'Table 'mysql.role_edges' doesn't exist'

fix

Use mysqldump to backup your databases

Remove MySQL completely

#!/bin/sh

brew services stop mysql
brew remove mysql
brew cleanup

sudo rm -rf /usr/local/var/mysql
sudo rm -rf /usr/local/mysql*
sudo rm ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist
sudo rm -rf /Library/StartupItems/MySQLCOM
sudo rm -rf /Library/PreferencePanes/My*

launchctl unload -w ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist

# subl /etc/hostconfig`
# Remove the line MYSQLCOM=-YES-

rm -rf ~/Library/PreferencePanes/My*
sudo rm -rf /Library/Receipts/mysql*
sudo rm -rf /Library/Receipts/MySQL*
sudo rm -rf /private/var/db/receipts/*mysql*

Reinstall MySQL

$ brew install mysql

Set MySQL root password

$ mysqladmin -u root password 'your password'  # setup mysql root password

References

• Remove MySQL completely

• MySQL | Mac OS X Setup Guide

Renaming a repository

Renaming a repository

First, rename repository on the web (Github/Bitbucket/Gitlab).

Next, update an existing local clone to point to the new repository URL,

git remote set-url origin new_url

References

• Renaming a repository