安装一些模块的时候偶尔会遇到”ERROR: Could not find a version that satisfies the requirement”的提示
其实就是当前的python对应的pip安装包没有这个新版本了, 如果不想往上升, 其实是可以往下降的
以最新安装bloompy遇到问题为例, 很古老的包了, 需要两个前置组件
install_requires=[
“bitarray >= 0.8.3”,
‘mmh3 >= 2.5.1’,
],
但是pip3帮我安装的时候, 选用的mmh3 却是4.0.1版本的, 这个导致它需要setuptools 需要>= 61.0.0
因为, 可以指定版本来安装
pip3 install mmh3==2.5.1
pip3 install bitarray==0.8.3
只要跟其他组件不冲突就OK, 当然如果实在环境依赖太复杂, 那就建议直接使用conda了
旧版本的操作系统有时候需要跑python3的代码, 这里介绍下最简单的模式
从epel源安装python3
Download the EPEL repository:
wget https://archives.fedoraproject.org/pub/archive/epel/6/x86_64/epel-release-6-8.noarch.rpm
Install the EPEL repository:
rpm -Uvh epel-release-6*.rpm
yum install python34 python34-devel -y
使用GET-PIP官方脚本安装升级pip
wget https://bootstrap.pypa.io/pip/3.4/get-pip.py -O get-pip_3.4.py
python3 get-pip_3.4.py
由于RHEL6 实际上是非常古老的操作系统了, 所以这里使用了最简单的方式安装python3和升级pip让新的python代码可以在这个平台上可以运行
在我的rpmbuild SPEC里边刚好需要用到Google depot_tools的ninja
本来 正常编译pip install ninja就完事了,但是在rpm环境里发现出错了
ModuleNotFoundError: No module named ‘ninia
这个事情很奇怪, 查了下资料, 在rpmbuild 环境里边python -m site 和命令行 发现了不同
rpmbuild里边的sys.path
sys.path = [
‘/root/rpmbuild/BUILD’,
‘/usr/lib64/python39.zip’,
‘/usr/lib64/python3.9’,
‘/usr/lib64/python3.9/lib-dynload’,
‘/usr/lib64/python3.9/site-packages’,
‘/usr/lib/python3.9/site-packages’,
]
命令行里边的sys.path
sys.path = [
‘/root/nginx’,
‘/usr/lib64/python39.zip’,
‘/usr/lib64/python3.9’,
‘/usr/lib64/python3.9/lib-dynload’,
‘/usr/local/lib64/python3.9/site-packages’,
‘/usr/lib64/python3.9/site-packages’,
‘/usr/lib/python3.9/site-packages’,
]
检查python的配置文件可以看到, 识别到RPMBUILD环境, 就会去掉/local/这个路径的包
当然, 解决办法也很简单, 在rpmbuild spec里边加个pip install ninja就行了
这是 rocky linux 9 的python 3.9 以及 Fedora 36 的python 3.10开始引入的一个变化
参考文档:
https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
https://hackmd.io/@python-maint/BkqScKJW5
这里主要介绍下line-profiler
https://pypi.org/project/line-profiler/
pip install line_profiler
如果是python2,则使用3.1.0, 这是最后可用的版本
pip2 install line_profiler==3.1.0
然后在需要监测的代码函数块前边加上
@profile
然后执行
kernprof -l -v your_python_scripts.py
就能看到以行为单位的执行时间占比, 从而分析出代码的性能问题主要出在什么地方
一些古老的代码是基于python2的, 需要安装一些模块的时候发现提示出错了
#pip install requests
You are using pip version 7.1.0, however version 23.1.2 is available.
You should consider upgrading via the ‘pip install –upgrade pip’ command.
Collecting requests
/usr/lib/python2.6/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/lib/python2.6/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
看提示是SSL 协商失败了, 嗯嗯, 猜测是pip的SSL版本过于旧, 需要升级下pip,结果又失败了
#pip install –upgrade pip
You are using pip version 7.1.0, however version 23.1.2 is available.
You should consider upgrading via the ‘pip install –upgrade pip’ command.
Collecting pip
Using cached https://files.pythonhosted.org/packages/fa/ee/74ff76da0ab649eec7581233daeb43d8aa35383d8f75317b2ab3b80c922f/pip-23.1.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File “”, line 20, in
File “/tmp/pip-build-p1lQDp/pip/setup.py”, line 7
def read(rel_path: str) -> str:
^
SyntaxError: invalid syntax----------------------------------------Command “python setup.py egg_info” failed with error code 1 in /tmp/pip-build-p1lQDp/pip
其实多年以前pip的在线安装和升级已经不支持python2了, 需要手工升级下
https://bootstrap.pypa.io/pip/2.7/get-pip.py
https://bootstrap.pypa.io/pip/2.6/get-pip.py
这两个分别是python的2.6 和 2.7版本的pip手工升级包, 当前时间的版本应该是pip-20.3.4, 下载后用python执行就可以自动升级pip了, 然后安装其他模块也没有问题
upstream test_check_bin {
server 10.19.127.22:8080;
server 10.19.127.57:8080;
server 10.19.126.6:8080;
keepalive 32;
check interval=10000 rise=2 fall=3 timeout=3000 default_down=false;
}
我们使用的是一个臭名昭著的模块 , 这个模块的作者去了淘宝后便只有tengine里边的模块得到更新了
https://github.com/yaoweibin/nginx_upstream_check_module
目前发现了在当前代码存在两个问题:
第一个问题是因为这个版本的upstream check TCP代码使用的keepalive模式, 根据这个文章的解释TCP Keepalive的心跳包是7200s,两个小时,一旦建立,没有收到主动关闭请求的话在探测端会一直保留establish的状态
https://m.haicoder.net/note/tcpip-interview/tcpip-interview-tcp-keepalive.html
当upstream端突然硬件故障/交换机挂掉/网卡被拔之类的极端情况出现的 时候, nginx 基于keepalivedTCP检测是没办法捕获到这个情况的(检测模块没收到RST/或者dst unr包)
在正常情况下, 比如upstream 端web服务器 STOP了, web服务器oom被系统kill了, docker 实例被stop/kill了, 都会触发关闭连接的请求包 给到nginx 端, 能识别到服务down了
从tengine这个模块的最新代码看, 作者也意识到了这个问题, TCP检测把need_keepalive参数从1改成了0
这个参数变成0之后会影响clean_event的操作, 每次检测完毕后会close掉连接,每个检测周期都会重新发起TCP连接
第二个问题: rise counts 不增加, 这个通常都是upstream 端listen 模式有问题导致, 它并没有正确的定期回包, 通常情况下, nginx, apache都能正常的主动回包让rise counts增加
python java这些裸起的一些简单服务就没有定期主动回包, 而在这个版本的代码中, 会判断这个connection是否存在, 如果存在则return了导致计数器不会增加
不过, 划重点: 无论是否回包, rise counts 是否增加,并不影响TCP监测的存活性, TCP监测keepalive模式仅仅以能否建立连接(刚启动时)/是否收到upstream的异常包为判断依据
总结: 为了避免过于复杂的处理逻辑, tengine去掉了keepalive的TCP探测,每次请求完都销毁连接,下次探测再重新协商请求
这里给下针对这个问题的patch:
https://www.4os.org/patch/nginx_upstream_check_tcp.patch
gn gen out/release –args=”is_component_build=false is_debug=false”
ninja -C out/release nginx-1.18.0_ipdb
build/linux/debian_sid_amd64-sysroot 这个目录相当于根目录, usr/lib和usr/include 分别放了依赖库和头文件
chromium移除了这两个文件,不知道为何, 会导致nginx 编译失败,目前解决办法是从旧版本拷贝过去
build/linux/debian_sid_amd64-sysroot/usr/lib/x86_64-linux-gnu/libcrypt.so
build/linux/debian_sid_amd64-sysroot/usr/include/crypt.h
build/config/compiler/BUILD.gn, 比如nginx 可能需要把这个特性关掉
if (!is_nacl && !use_libfuzzer) {
#cflags += [ “-Wunreachable-code” ]
}
executable(“fssnginx-1.18.0_ipdb”) {
sources = [
“/root/fssnginx/nginx-1.18.0/objs/ngx_modules.c”,
“/root/fssnginx/nginx-1.18.0/src/core/nginx.c”,….
]
include_dirs = [
“/root/fssnginx/nginx-1.18.0/src/core”,
“/root/fssnginx/nginx-1.18.0/src/event”,
“/root/fssnginx/nginx-1.18.0/src/event/modules”,
“/root/fssnginx/nginx-1.18.0/src/os/unix”,
“/root/fssnginx/nginx-1.18.0/nginx_upstream_check_module-master”,
“/root/fssnginx/nginx-1.18.0/ngx_devel_kit-0.3.0/objs”,
“/root/fssnginx/nginx-1.18.0/objs/addon/ndk”,
“/root/fssnginx/nginx-1.18.0/lua-nginx-module-0.10.13/src/api”,
“/root/fssnginx/nginx-1.18.0/pcre-8.42”,
“/root/fssnginx/nginx-1.18.0/zlib-1.2.11”,
“/root/fssnginx/nginx-1.18.0/objs”,
“/root/fssnginx/nginx-1.18.0/src/http”,
“/root/fssnginx/nginx-1.18.0/src/http/modules”,
“/root/fssnginx/nginx-1.18.0/src/http/v2”,
“/root/fssnginx/nginx-1.18.0/src/http”,
“/root/fssnginx/nginx-1.18.0/ngx_devel_kit-0.3.0/src”,
“/root/fssnginx/nginx-1.18.0/ngx_devel_kit-0.3.0/src”,
“/root/fssnginx/nginx-1.18.0/ngx_devel_kit-0.3.0/objs”,
“/root/fssnginx/nginx-1.18.0/objs/addon/ndk”,
“/root/fssnginx/nginx-1.18.0/luajit/include/luajit-2.0”,
“/root/fssnginx/nginx-1.18.0/quic_module/chromium”,
]
deps = [
“:epoll_quic_tools”,
“:epoll_server”,
“:net”,
“:simple_quic_tools”,
“//base”,
“//third_party/boringssl”,
]
lib_dirs = [
“/root/fssnginx/nginx-1.18.0/json-c/lib”,
“/root/fssnginx/nginx-1.18.0/luajit/lib”,
]
libs = [
“/root/fssnginx/nginx-1.18.0/pcre-8.42/.libs/libpcre.a”,
“/root/fssnginx/nginx-1.18.0/zlib-1.2.11/libz.a”,
“luajit-5.1”,
“json-c”,
“crypt”,
]
cflags_c = [
“-D_FORTIFY_SOURCE=2”,
“-DTCP_FASTOPEN=23”,
“-DNDK_SET_VAR”,
]
}
线上有个业务需要使用到指定版本的chromium源代码, 这里记录下
$ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
$ export PATH="$PATH:/path/to/depot_tools"
Tip: /path/to/depot_tools， 为你 depot_tools 本地的路径
$ mkdir chromium && cd chromium
$ fetch chromium
//Don't use fetch --no-history chromium, 我们需要切换到历史版本
Tip: –no-history： 代表不需要历史记录， 完整仓库大约40G 源码大小大概 8G 左右，下载时间因网速而议，请耐心等待
# Make sure you are in 'src'. # This part should only need to be done once, but it won't hurt to repeat it. The first # time checking out branches and tags might take a while because it fetches an extra # 1/2 GB or so of branch commits. gclient sync --with_branch_heads --with_tags # You may have to explicitly 'git fetch origin' to pull branch-heads/ git fetch # Checkout the branch 'src' tree. git checkout -b branch_$BRANCH tags/$BRANCH # Checkout all the submodules at their branch DEPS revisions. gclient sync --with_branch_heads --with_tags
到这一步做完就可以校验下了
$cat chrome/VERSION
参考文档:
https://www.chromium.org/developers/how-tos/get-the-code/working-with-release-branches
这边有个需求, 让bind 的query_log 只记录ipv4的A记录请求
解决办法:
修改代码query.c 里边记录query_log的部分, 增加一个 客户端query.qtype的判断
--- bind-9.11.14/bin/named/query.c 2019-12-12 13:17:55.000000000 +0800
+++ bind-9.11.14.mod/bin/named/query.c 2020-03-05 16:57:31.766000000 +0800
@@ -9548,8 +9548,6 @@
return;
}
if (ns_g_server->log_queries)
log_query(client, saved_flags, saved_extflags);
/* * Check for meta-queries like IXFR and AXFR.
@@ -9559,6 +9557,10 @@
client->query.qtype = qtype = rdataset->type;
dns_rdatatypestats_increment(ns_g_server->rcvquerystats, qtype);
if ( (ns_g_server->log_queries) && (client->query.qtype == dns_rdatatype_a) ){
log_query(client, saved_flags, saved_extflags);
}
+
log_tat(client);
if (dns_rdatatype_ismeta(qtype)) {
qtype 分以下种类
enum {
dns_rdatatype_none = 0,
dns_rdatatype_a = 1,
dns_rdatatype_ns = 2,
dns_rdatatype_md = 3,
dns_rdatatype_mf = 4,
dns_rdatatype_cname = 5,
dns_rdatatype_soa = 6,
dns_rdatatype_mb = 7,
dns_rdatatype_mg = 8,
dns_rdatatype_mr = 9,
dns_rdatatype_null = 10,
dns_rdatatype_wks = 11,
dns_rdatatype_ptr = 12,
dns_rdatatype_hinfo = 13,
dns_rdatatype_minfo = 14,
dns_rdatatype_mx = 15,
dns_rdatatype_txt = 16,
dns_rdatatype_rp = 17,
dns_rdatatype_afsdb = 18,
dns_rdatatype_x25 = 19,
dns_rdatatype_isdn = 20,
dns_rdatatype_rt = 21,
dns_rdatatype_nsap = 22,
dns_rdatatype_nsap_ptr = 23,
dns_rdatatype_sig = 24,
dns_rdatatype_key = 25,
dns_rdatatype_px = 26,
dns_rdatatype_gpos = 27,
dns_rdatatype_aaaa = 28,
dns_rdatatype_loc = 29,
dns_rdatatype_nxt = 30,
dns_rdatatype_srv = 33,
dns_rdatatype_naptr = 35,
dns_rdatatype_kx = 36,
dns_rdatatype_cert = 37,
dns_rdatatype_a6 = 38,
dns_rdatatype_dname = 39,
dns_rdatatype_opt = 41,
dns_rdatatype_apl = 42,
dns_rdatatype_ds = 43,
dns_rdatatype_sshfp = 44,
dns_rdatatype_ipseckey = 45,
dns_rdatatype_rrsig = 46,
dns_rdatatype_nsec = 47,
dns_rdatatype_dnskey = 48,
dns_rdatatype_dhcid = 49,
dns_rdatatype_nsec3 = 50,
dns_rdatatype_nsec3param = 51,
dns_rdatatype_hip = 55,
dns_rdatatype_spf = 99,
dns_rdatatype_unspec = 103,
dns_rdatatype_tkey = 249,
dns_rdatatype_tsig = 250,
dns_rdatatype_dlv = 32769,
dns_rdatatype_keydata = 65533,
dns_rdatatype_ixfr = 251,
dns_rdatatype_axfr = 252,
dns_rdatatype_mailb = 253,
dns_rdatatype_maila = 254,
dns_rdatatype_any = 255
};
参考文档:
https://users.isc.org/~each/doxygen/bind9/structns__query.html
https://ri.co.cr/training/dccom/bind-9.8.2/lib/dns/include/dns/enumtype.h
最近维护一个古老代码的时候,在AS7里边编译的时候出现了不少错误,比如:
warning: comparison between pointer and integer, 这是代码 strptime(current_time, “%a%n%b%n%d%n%H:%M:%S%n%Y”, &tm) == NULL) 产生的
还有, warning: incompatible implicit declaration of built-in function ‘snprintf’, 这是类似 snprintf(path, sizeof(path), “%s/arc_%s”, archdir, com->cat) 的代码产生的
查了下,strptime的异常返回值确实是NULL, http://man7.org/linux/man-pages/man3/strptime.3.html
If strptime() fails to match all of the format string and therefore an error occurred, the function returns NULL.
一般来说, 类似的snprintf错误 添加 #include<stdlib.h>就好,但是这个古老代码里边已经有这个了
在代码的开头发现了这个东西
_XOPEN_SOURCE
Defining this macro causes header files to expose definitions as
follows:
o Defining with any value exposes definitions conforming to POSIX.1, POSIX.2, and XPG4.
o The value 500 or greater additionally exposes definitions for SUSv2 (UNIX 98).
o (Since glibc 2.2) The value 600 or greater additionally exposes definitions for SUSv3 (UNIX 03; i.e., the POSIX.1-2001 base specification plus the XSI extension) and C99 definitions.
o (Since glibc 2.10) The value 700 or greater additionally exposes definitions for SUSv4 (i.e., the POSIX.1-2008 base specification plus the XSI extension).
https://linux.cn/man7/feature_test_macros.7.html, 初略的说,大概是不同的值会影响某些函数和库的一些标准行为,而默认的_xopen_soure 不带参数会被认为是<500
#define _XOPEN_SOURCE /* or any value < 500 */
所以需要修正这个模式,添加一个标准即可,由于代码比较古老,修改为500和600都可以编译通过