问题引入

最近在鼓捣爬虫准备爬爬学校网站，用到pthon的requests库发送get请求时需要提供headers.

需要将请求头转换成json格式的数据。json格式如下所示

headers={
    "Path":"xxx",
    "User-Agent":"xxx"
}

但是从网页上f12复制下来的请求标头又不符合json格式 

但是放到记事本后可以看见是一行一行每两行就对应一个键和一个值。

:Authority:
webvpn.cn
:Method:
GET
:Path:
/https/34263054423230556a6d256d76542e4bfe905bb2df3254201d56ee3478/author/ranklist
:Scheme:
https
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding:
gzip, deflate, br
Accept-Language:
zh-CN,zh;q=0.9,en;q=0.8,ja;q=0.7,bg;q=0.6,zh-TW;q=0.5
refresh=0
Referer:
https://webvpn.zqu.edu.cn/https/34263054423230556a6d256d76542e4bfe905bb2df3254201d56ee3478/author/userdata?user_id=5158
Sec-Ch-Ua:
"Google Chrome";v="113", "Chromium";v="113", "Not-A.Brand";v="24"
Sec-Ch-Ua-Mobile:
?0
Sec-Ch-Ua-Platform:
"Windows"
Sec-Fetch-Dest:
document
Sec-Fetch-Mode:
navigate
Sec-Fetch-Site:
same-origin
Sec-Fetch-User:
?1
Upgrade-Insecure-Requests:
1
User-Agent:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36

因此这里写了一个代码读取文件中的请求头字符串生成对应的json格式数据。

用到【Python教程】删除字符串中字符的四种方法_python 字符串删除_Python热爱者的博客-CSDN博客

具体代码 

 需要注意的地方有几个

第一个:删除前置的:和后置的换行\n，不然会有报错和多余换行

第二个:在 “ 前面加上转义用的 \      ，不转义会报错

f=open("1.txt", "r", encoding="utf-8")  #读入请求头的文件
f1=open("2.txt","w",encoding="utf-8")   #打印json格式的文件

f1.write("headers={\n")  #添加一个最前面的左括号

line="1"
while line!="":
    line=f.readline()
    if(line==""):
        break
    x=line
    line=f.readline()
    y=line

    x=x.strip(':\n')   #去除前后的:和换行
    y=y.strip(':\n')    #去除前后的:和换行
    i=0
    le=len(y)
    while(i<le):
        if(y[i]=="\""):
            y=y[:i]+"\\"+y[i:]   #在“处添加一个转移符号
            i=i+1                #跳过这个”
            le=le+1              #同时将总长度增加
        i=i+1
    f1.write("\""+x+"\""+":\""+y+"\",\n") #字符串拼接成json格式

f1.write("}")

最后用上面提供的请求头运行获得如下输出，可以用于get请求时的请求头

headers={
"Authority":"webvpn.cn",
"Method":"GET",
"Path":"/https/34263054423230556a6d256d76542e4bfe905bb2df3254201d56ee3478/author/ranklist",
"Scheme":"https",
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"Accept-Encoding":"gzip, deflate, br",
"Accept-Language":"zh-CN,zh;q=0.9,en;q=0.8,ja;q=0.7,bg;q=0.6,zh-TW;q=0.5",
"Referer":"https://webvpn.zqu.edu.cn/https/34263054423230556a6d256d76542e4bfe905bb2df3254201d56ee3478/author/userdata?user_id=5158",
"Sec-Ch-Ua":"\"Google Chrome\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\"",
"Sec-Ch-Ua-Mobile":"?0",
"Sec-Ch-Ua-Platform":"\"Windows\"",
"Sec-Fetch-Dest":"document",
"Sec-Fetch-Mode":"navigate",
"Sec-Fetch-Site":"same-origin",
"Sec-Fetch-User":"?1",
"Upgrade-Insecure-Requests":"1",
"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
}