什么是Protocol Buffer
Protocol Buffers(也称protobuf)是Google公司出品的一种独立于开发语言,独立于平台的可扩展的结构化数据序列机制。通俗点来讲它跟xml和json是一类。是一种数据交互格式协议。
主要优点是它是基于二进制的,所以比起结构化的xml协议来说,它的体积很少,数据在传输过程中会更快。另外它也支持c++、java、python、php、javascript等主流开发语言。
官网地址:https://developers.google.com/protocol-buffers/
Proto3安装
下载地址 :3.x.x的版本基本都按照操作系统和语言进行了区分,系统包里只包含了protoc命令,语言包则是用于编译后使用,比如java需要生成对应的jar包。这里可以根据需要下载对应的操作系统和语言包,比如这里我下载的是protoc-3.5.1-osx-x86_64.zip(苹果系统)和protobuf-java-3.5.1.tar.gz(java语言)。
unzip protoc-3.5.1-osx-x86_64.zip
在/etc/profile中添加环境变量PROTOCTL_BUFFER_HOME(protoc-3.5.1-osx-x86_64.zip解压后目录),并在PATH中添加$PROTOCTL_BUFFER_HOME/bin
查看版本:protoc --version
:输出 libprotoc 3.5.1
以下部分只为自行编译生成对应的jar包,实际上maven中央仓库中已经存在了
tar -zxcf protobuf-java-3.5.1.tar.gz
,解压后目录名称为protobuf-3.5.1
cd protobuf-3.5.1/src
,创建软连接 ln -s $PROTOCTL_BUFFER_HOME/bin/protoc protoc
cd protobuf-3.5.1/java
,mvn package
(maven请自行安装),成功后会在protobuf-3.5.1/java/code/target下生成protobuf-java-3.5.1.jar
然后将protobuf-java-3.5.1.jar上传到maven私服或者安装到本地仓库就可以使用了
1 mvn install:install-file -Dfile=protobuf-java-3.5.1.jar -DgroupId=com.google.protobuf -DartifactId=protobuf-java -Dversion=3.5.1 -Dpackaging=jar
1 2 3 4 5 6 <dependency > <groupId > com.google.protobuf</groupId > <artifactId > protobuf-java</artifactId > <version > 3.5.1</version > </dependency >
Proto2安装
下载地址 :这里只是操作系统包,比如这里我下载的是protoc-2.6.1-osx-x86_64.exe,语言包protobuf-2.6.1.tar.gz 。
mv protoc-2.6.1-osx-x86_64.exe protoc
将上面重命名后的protoc文件所在目录加到系统环境变量PATH中
查看版本:protoc --version
:输出 libprotoc 2.6.1
以下部分只为自行编译生成对应的jar包,实际上maven中央仓库中已经存在了
tar -zxcf protobuf-2.6.1.tar.gz
,解压后目录名称为protobuf-2.6.1
cd protobuf-2.6.1/src
,创建软连接 ln -s $PROTOCTL_BUFFER_HOME/bin/protoc protoc
cd protobuf-2.6.1/java
,mvn package
(maven请自行安装),成功后会在protobuf-2.6.1/java/target下生成protobuf-java-2.6.1.jar
然后将protobuf-java-2.6.1.jar上传到maven私服或者安装到本地仓库就可以使用了
1 mvn install:install-file -Dfile=protobuf-java-2.6.1.jar -DgroupId=com.google.protobuf -DartifactId=protobuf-java -Dversion=2.6.1 -Dpackaging=jar
1 2 3 4 5 6 <dependency > <groupId > com.google.protobuf</groupId > <artifactId > protobuf-java</artifactId > <version > 2.6.1</version > </dependency >
Proto使用
先编写proto文件,具体语法请参考通信协议之Protocol buffer(Java篇)
生成java文件:protoc --java_out=. XXXX.proto
生成js文件:protoc --js_out=import_style=commonjs,binary:. XXXX.proto
『只有proto3支持该命令』
proto2与proto3语法上有一些不同,但是在使用时却没有特别的不同之处,此外proto3向下兼容proto2,所以可以只安装proto3,然后通过在proto文件中声明『syntax = “proto2”;或者syntax = “proto3”;』来指定类型
proto例子
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 package com.data.upload.proto;message BaseInfoCompany { required string CompanyId = 1 ; required string CompanyName = 2 ; required string Identifier = 3 ; required uint32 Address = 4 ; required string BusinessScope = 5 ; required string ContactAddress = 6 ; required string EconomicType = 7 ; required string RegCapital = 8 ; required string LegalName = 9 ; required string LegalID = 10 ; required string LegalPhone = 11 ; optional string LegalPhoto = 12 ; required uint32 State = 13 ; required uint32 Flag = 14 ; required uint64 UpdateTime = 15 ; optional string Reserved = 16 ; } message BaseInfoCompanyStat { required string CompanyId = 1 ; required uint32 VehicleNum = 2 ; required uint32 DriverNum = 3 ; required uint32 Flag = 4 ; required uint64 UpdateTime = 5 ; optional string Reserved = 6 ; } enum IpcType { baseInfoCompany = 0 x1001; baseInfoCompanyStat = 0 x1002; } message OTIpc { required string CompanyId = 1 ; required string Source = 2 ; required IpcType IPCType = 3 ; repeated BaseInfoCompany baseInfoCompany = 0 x1001; repeated BaseInfoCompanyStat baseInfoCompanyStat = 0 x1002; } message OTIpcList { repeated OTIpc otpic = 1 ; }
java中使用Protocol Buffer
1 2 3 4 5 6 <dependency > <groupId > com.google.protobuf</groupId > <artifactId > protobuf-java</artifactId > <version > 2.6.1</version > </dependency >
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 OTIpcDef.BaseInfoCompany.Builder baseInfoCompanyBuilder = OTIpcDef.BaseInfoCompany.newBuilder(); baseInfoCompanyBuilder.setAddress(110011 ); baseInfoCompanyBuilder.setCompanyId("companyId" ); baseInfoCompanyBuilder.setCompanyName("companyName" ); baseInfoCompanyBuilder.setIdentifier("identifier" ); baseInfoCompanyBuilder.setBusinessScope("BusinessScope" ); baseInfoCompanyBuilder.setContactAddress("ContactAddress" ); baseInfoCompanyBuilder.setEconomicType("EconomicType" ); baseInfoCompanyBuilder.setRegCapital("RegCapital" ); baseInfoCompanyBuilder.setLegalName("LegalName" ); baseInfoCompanyBuilder.setLegalID("LegalID" ); baseInfoCompanyBuilder.setLegalPhone("LegalPhone" ); baseInfoCompanyBuilder.setState(0 ); baseInfoCompanyBuilder.setFlag(1 ); baseInfoCompanyBuilder.setUpdateTime(20180226121212l ); OTIpcDef.BaseInfoCompany baseInfoCompany = baseInfoCompanyBuilder.build(); OTIpcDef.OTIpc.Builder otIpcBuilder = OTIpcDef.OTIpc.newBuilder(); otIpcBuilder.setCompanyId("companyId" ); otIpcBuilder.setSource("Source" ); otIpcBuilder.setIPCType(OTIpcDef.IpcType.baseInfoCompany); otIpcBuilder.addBaseInfoCompany(baseInfoCompany); otIpcBuilder.addBaseInfoCompany(baseInfoCompany); OTIpcDef.OTIpc otIpc = otIpcBuilder.build(); OTIpcDef.BaseInfoCompanyStat.Builder baseInfoCompanyStatBuilder = OTIpcDef.BaseInfoCompanyStat.newBuilder(); baseInfoCompanyStatBuilder.setCompanyId("companyId" ); baseInfoCompanyStatBuilder.setDriverNum(10 ); baseInfoCompanyStatBuilder.setFlag(0 ); baseInfoCompanyStatBuilder.setUpdateTime(20180226121212l ); baseInfoCompanyStatBuilder.setVehicleNum(5 ); OTIpcDef.BaseInfoCompanyStat baseInfoCompanyStat = baseInfoCompanyStatBuilder.build(); OTIpcDef.OTIpc.Builder otIpcBuilder2 = OTIpcDef.OTIpc.newBuilder(); otIpcBuilder2.setCompanyId("companyId" ); otIpcBuilder2.setSource("Source" ); otIpcBuilder2.setIPCType(OTIpcDef.IpcType.baseInfoCompanyStat); otIpcBuilder2.addBaseInfoCompanyStat(baseInfoCompanyStat); OTIpcDef.OTIpc otIpc2 = otIpcBuilder2.build(); OTIpcDef.OTIpcList.Builder oTIpcListBuilder = OTIpcDef.OTIpcList.newBuilder(); oTIpcListBuilder.addOtpic(otIpc); oTIpcListBuilder.addOtpic(otIpc2); OTIpcDef.OTIpcList otIpcList = oTIpcListBuilder.build(); byte [] array = otIpcList.toByteArray();HttpClientUtils httpClientUtils = new HttpClientUtils ();httpClientUtils.doPost4ProtocleBuffer("http://localhost:3000/demo/protoc" ,array);
HttpClientUtils.java
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 import com.yipin.entity.HttpResult;import org.apache.http.client.config.RequestConfig;import org.apache.http.client.methods.CloseableHttpResponse;import org.apache.http.client.methods.HttpPost;import org.apache.http.entity.ByteArrayEntity;import org.apache.http.impl.client.CloseableHttpClient;import org.apache.http.impl.client.HttpClients;import org.apache.http.impl.client.LaxRedirectStrategy;import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;import org.apache.http.util.EntityUtils;public class HttpClientUtils { private CloseableHttpClient httpClient; private RequestConfig requestConfig; public HttpClientUtils () { init(); } public void init () { PoolingHttpClientConnectionManager httpClientConnectionManager = new PoolingHttpClientConnectionManager (); this .requestConfig = RequestConfig.custom().build(); LaxRedirectStrategy redirectStrategy = new LaxRedirectStrategy (); this .httpClient = HttpClients.custom().setConnectionManager(httpClientConnectionManager) .setDefaultRequestConfig(requestConfig) .setRedirectStrategy(redirectStrategy) .build(); } public HttpResult doPost4ProtocleBuffer (String url, byte [] bytes) throws Exception { HttpPost httpPost = new HttpPost (url); httpPost.setConfig(this .requestConfig); httpPost.setHeader("Connection" , "keep-alive" ); httpPost.setHeader("Content-type" , "application/x-protobuf" ); httpPost.setHeader("Accept-Encoding" , "gzip" ); httpPost.setHeader("Accept-Charset" , "utf-8" ); if (bytes != null ) { ByteArrayEntity byteArrayEntity = new ByteArrayEntity (bytes); byteArrayEntity.setContentType("application/x-protobuf" ); httpPost.setEntity(byteArrayEntity); } CloseableHttpResponse response = null ; try { response = this .httpClient.execute(httpPost); return new HttpResult (response.getStatusLine().getStatusCode(), EntityUtils.toString(response.getEntity(), "UTF-8" )); } finally { if (response != null ) { response.close(); } } } }
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 InputStream in = request.getInputStream();OTIpcDef.OTIpcList otIpcList = OTIpcDef.OTIpcList.parseFrom(in); List<OTIpcDef.OTIpc> list= otIpcList.getOtpicList(); for (OTIpcDef.OTIpc otIpc : list){ String companyid = otIpc.getCompanyId(); String source = otIpc.getSource(); OTIpcDef.IpcType ipcType = otIpc.getIPCType(); if (ipcType == OTIpcDef.IpcType.baseInfoCompany){ List<OTIpcDef.BaseInfoCompany> baseInfoCompanyList = otIpc.getBaseInfoCompanyList(); for (OTIpcDef.BaseInfoCompany baseInfoCompany : baseInfoCompanyList){ String companyName = baseInfoCompany.getCompanyName(); } }else if (ipcType == OTIpcDef.IpcType.baseInfoCompanyStat){ List<OTIpcDef.BaseInfoCompanyStat> baseInfoCompanyStatList = otIpc.getBaseInfoCompanyStatList(); for (OTIpcDef.BaseInfoCompanyStat baseInfoCompanyStat : baseInfoCompanyStatList){ int driverNum = baseInfoCompanyStat.getDriverNum(); } } }
nodejs中使用Protocol Buffer
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 var OTIpcDefProto = require ('../protocbuf/OTIpcDef_pb' );var http = require ('http' );var baseInfoCompany = new OTIpcDefProto .BaseInfoCompany ();baseInfoCompany.setAddress (110011 ); baseInfoCompany.setCompanyid ("companyId" ); baseInfoCompany.setIdentifier ("identifier" ); baseInfoCompany.setCompanyname ("companyName公司名称" ); baseInfoCompany.setBusinessscope ("BusinessScope" ); baseInfoCompany.setContactaddress ("ContactAddress" ); baseInfoCompany.setEconomictype ("EconomicType" ); baseInfoCompany.setRegcapital ("RegCapital" ); baseInfoCompany.setLegalname ("LegalName" ); baseInfoCompany.setLegalid ("LegalID" ); baseInfoCompany.setLegalphone ("LegalPhone" ); baseInfoCompany.setState (0 ); baseInfoCompany.setFlag (1 ); baseInfoCompany.setUpdatetime (20180226121212 ); var otIpc = new OTIpcDefProto .OTIpc ();otIpc.setCompanyid ("companyId" ); otIpc.setSource ("Source" ); otIpc.setIpctype (OTIpcDefProto .IpcType .BASEINFOCOMPANY ); otIpc.addBaseinfocompany (baseInfoCompany); var otIpcList = new OTIpcDefProto .OTIpcList ();otIpcList.addOtpic (otIpc); var contents = otIpcList.serializeBinary ();var options = { host : 'localhost' , port : 3000 , path : '/demo2/protoc' , method : 'POST' , headers : { 'Content-Type' : 'application/x-protobuf' } }; var req = http.request (options, function (res ){ res.on ('data' , function (data ){ console .log (data); }); }); var buffer = new Buffer (contents);req.write (buffer); req.end ();
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 var express = require ('express' );var router = express.Router ();var OTIpcDefProto = require ('../protocbuf/OTIpcDef_pb' );var BufferHelper = require ('bufferhelper' );router.post ('/protoc' , function (req, res, next ) { var bufferHelper = new BufferHelper (); req.on ("data" , function (chunk ) { bufferHelper.concat (chunk); }); req.on ('end' , function ( ) { var buffer = bufferHelper.toBuffer (); var otIpcList = OTIpcDefProto .OTIpcList .deserializeBinary (new Uint8Array (buffer)); for (var i=0 ;i<otIpcList.getOtpicList ().length ;i++) { console .log (i+"========================================" ); var otIpc = otIpcList.getOtpicList ()[i]; var companyid = otIpc.getCompanyid (); var source = otIpc.getSource (); var iPCType = otIpc.getIpctype (); console .log (companyid); console .log (source); console .log (iPCType); if (iPCType == OTIpcDefProto .IpcType .BASEINFOCOMPANY ){ var baseInfoCompanyList = otIpc.getBaseinfocompanyList (); for (var j=0 ;j<baseInfoCompanyList.length ;j++){ console .log (j+"===============baseInfoCompanyList=================" ); var baseInfoCompany = baseInfoCompanyList[j]; console .log (baseInfoCompany.toObject ()); console .log (baseInfoCompany.getCompanyid ()); console .log (baseInfoCompany.getCompanyname ()); } }else if (iPCType == OTIpcDefProto .IpcType .BASEINFOCOMPANYSTAT ){ var baseInfoCompanyStatList = otIpc.getBaseinfocompanystatList (); for (var j=0 ;j<baseInfoCompanyStatList.length ;j++){ console .log (j+"===============baseInfoCompanyStatList=================" ); var baseInfoCompanyStat = baseInfoCompanyStatList[j]; console .log (baseInfoCompanyStat.toObject ()); console .log (baseInfoCompanyStat.getCompanyid ()); console .log (baseInfoCompanyStat.getDrivernum ()); } } } console .log (otIpcList.toObject ()); res.send (otIpcList.toObject ()); }); }); module .exports = router;
这里可以将protocolbuffer数据的接收过程封装到app.js中
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 app.use ('/*' ,function (req, res, next ) { var contentType = req.get ('Content-Type' ); if (contentType=='application/x-protobuf' ) { var bufferHelper = new BufferHelper (); req.on ("data" , function (chunk ) { bufferHelper.concat (chunk); }); req.on ('end' , function ( ) { var buffer = bufferHelper.toBuffer (); req.body = buffer; console .log (req.body ); next (); }); }else { next (); } });
然后在路由js中只需要按照如下方式接收数据即可
1 var otIpcList = OTIpcDefProto .OTIpcList .deserializeBinary (new Uint8Array (req.body ));
1 var otIpcList = OTIpcDefProto .OTIpcList .deserializeBinary (new Uint8Array (req.body ));
JSON与Protobuf相互转换
JAVA
1 2 3 4 5 6 <dependency > <groupId > com.googlecode.protobuf-java-format</groupId > <artifactId > protobuf-java-format</artifactId > <version > 1.4</version > </dependency >
1 2 3 4 5 6 com.googlecode.protobuf.format.JsonFormat jsonFormat = new JsonFormat (); com.google.protobuf.Message.Builder builder = OTIpcDef.BaseInfoCompany.newBuilder(); String json = com.alibaba.fastjson.JSON.toJSONString(myObject);jsonFormat.merge(new ByteArrayInputStream (json.getBytes()), builder);
1 2 3 4 OTIpcDef.OTIpcList otIpcList = oTIpcListBuilder.build(); com.googlecode.protobuf.format.JsonFormat jsonFormat = new JsonFormat (); String json = jsonFormat.printToString(otIpcList);
nodejs
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 var json2proto = function (json_str,protoObject ) { Array .prototype .contains = function ( needle ) { for (i in this ) { if (this [i] == needle) return true ; } return false ; } var p_json_str = json_str; var p_json = eval ("(" + p_json_str + ")" ); var p_json_key_array = []; var i = 0 ; for (var p in p_json){ p_json_key_array[i] = p; i++; } var s_json = protoObject.toObject (); for (var p in s_json){ if (p_json_key_array.contains (p)) { var setMethod = "set" +p.charAt (0 ).toUpperCase () + p.slice (1 ); protoObject[setMethod](p_json[p]); } } return protoObject; } module .exports = json2proto;
调用方法
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 var OTIpcDefProto = require ('../protocbuf/OTIpcDef_pb' );var json2proto = require ('../json2Proto' );var p_json_str = "{ companyid: '公司ID'," + "companyname: 'companyId'," + "identifier : 'identifier'," + "address : 111111," + "businessscope : 'businessscope'," + "contactaddress : 'contactaddress'," + "economictype : 'economictype'," + "regcapital : 'regcapital'," + "legalname : 'legalname'," + "legalid : 'legalid'," + "legalphone : 'legalphone'," + "legalphoto : 'legalphoto'," + "state : 0," + "flag : 1," + "updatetime: 20180226121212}" ; var baseInfoCompany = json2proto (p_json_str,new OTIpcDefProto .BaseInfoCompany ());console .log (baseInfoCompany.toObject ());
参考资料