Logstash 时间转换（YYYY-MM-dd HH:mm:ss转Unix时间）

Logstash 时间转换（YYYY-MM-dd HH:mm:ss转Unix时间）

适用场景 - 日志时间转Unix时间 示例日志：

2017-03-21 00:00:00,291 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.r.m.RequirementManager [RequirementManager.java:860] fetch no data from oracle2017-03-21 00:00:00,294 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.r.m.RequirementManager [RequirementManager.java:1549] begin fetch data from oracle2017-03-21 00:00:00,301 ERROR [DubboServerHandler-10.135.6.53:20885-thread-98] i.w.w.h.d.testDao [testDao.java:265] find test project failed, error code={}java.sql.SQLException: ORA-00942: table or view does not exist at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:799) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1038) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:839) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1133) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3285) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3329) ~[ojdbc14-10.2.0.1.0.jar:Oracle JDBC Driver version - "10.2.0.1.0XE"] at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4] at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) ~[commons-dbcp-1.4.jar:1.4] at io.jiankunking.demo.haier.dao.testDao.findBigResourceMaterialByStatus(testDao.java:252) ~[demo-requirement-1.0-SNAPSHOT.jar:na] at io.jiankunking.demo.requirement.manager.RequirementManager.findBigResourceMaterialByStatus(RequirementManager.java:1550) [demo-requirement-1.0-SNAPSHOT.jar:na] at io.jiankunking.demo.requirement.manager.RequirementManager$$FastClassByCGLIB$$b97dbf5c.invoke() [spring-core-3.2.6.RELEASE.jar:na] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) [spring-tx-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at io.jiankunking.demo.requirement.manager.RequirementManager$$EnhancerByCGLIB$$de554283.findBigResourceMaterialByStatus() [spring-core-3.2.6.RELEASE.jar:na] at io.jiankunking.demo.requirement.service.RequirementServiceImpl.fetchBigResourceMaterial(RequirementServiceImpl.java:4791) [demo-requirement-1.0-SNAPSHOT.jar:na] at io.jiankunking.demo.requirement.service.RequirementServiceImpl$$FastClassByCGLIB$$15e55a9c.invoke() [spring-core-3.2.6.RELEASE.jar:na] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:701) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:55) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:634) [spring-aop-3.2.6.RELEASE.jar:3.2.6.RELEASE] at io.jiankunking.demo.requirement.service.RequirementServiceImpl$$EnhancerByCGLIB$$e80e4734.fetchBigResourceMaterial() [spring-core-3.2.6.RELEASE.jar:na] at com.alibaba.dubbo.common.bytecode.Wrapper55.invokeMethod(Wrapper55.java) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:60) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:132) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.4.1-20160510.024647-3.jar:2.5.4.1-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_111] at java.lang.Thread.run(Unknown Source) [na:1.8.0_111]2017-03-21 00:00:00,302 INFO [DubboServerHandler-10.135.6.53:20885-thread-98] i.t.s.r.m.RequirementManager [RequirementManager.java:1556] fetch no data from oracle

logstash 配置文件：

input { file { path => [ "/home/jiankunking/software/testdata/test.log" ] tags => [ "test","jiankunking" ] start_position => beginning codec =>multiline { pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}" negate => true what => "previous" } }}filter { grok{ match => ["message","%{TIMESTAMP_ISO8601:time}\s{1,}%{NOTSPACE:loglevel}\s{1,}%{NOTSPACE:thread-id}\s{1,}%{NOTSPACE:packageinfo}\s{1,}%{NOTSPACE:classinfo}\s{1,}%{GREEDYDATA:msginfo}"] } date{ match=>["time","YYYY-MM-dd HH:mm:ss,SSS"] target=>"logdate" } ruby{ code => "event.set('logdateunix',event.get('logdate').to_i)" } mutate { remove_field => ["message"] }}output{ elasticsearch{ hosts => ["111.222.333.444:9200"] index => "logstash-demo"

具体效果如下：

从 Logstash-5.0 开始， LogStash::Event 改为 Java 实现，直接使用 event[“parent”][“child”] 形式获取的不是原事件的引用而是复制品。需要改用 event.get(‘[parent][child]’) 和 event.set(‘[parent][child]’, ‘value’) 的方法。

小注：本文演示logstash用的是5.0.1版本

5.0之前版本可以参考（未测试）：

作者：jiankunking ​

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。