gerrit at wikimedia
Aug 11, 2013, 6:47 PM
Post #1 of 1
(5 views)
Permalink
|
|
[MediaWiki-commits] [Gerrit] (WIP) Initial Debian version - change (operations...varnishkafka)
|
|
Faidon has uploaded a new change for review.
https://gerrit.wikimedia.org/r/78782
Change subject: (WIP) Initial Debian version
......................................................................
(WIP) Initial Debian version
These things still need work:
- The varnishkafka.conf shipped by the package. The intention is to be
a varnishncsa drop-in replacement, i.e. write to stdout, not kafka.
I've changed a few of the settings but haven't tested the output at
all.
- The init script; it's currently just varnishncsa's script with
s/ncsa/kafka/
- We need a better d/control description, the current one was written
just as a placeholder. We also need a README.md & a manpage.
- This needs to be built after I2eac5 gets merged. It doesn't
strictly need it but it'd be wrong to ship a statically linked
binary.
- This really need to be built against a tagged version instead of
just master; d/gbp.conf should be adjusted accordingly
(upstream-tree=tag)
Change-Id: Ib53267d85e7a638d34b8660557b3c62638abdd54
---
M .gitignore
A debian/changelog
A debian/compat
A debian/control
A debian/copyright
A debian/gbp.conf
A debian/rules
A debian/source/format
A debian/varnishkafka.conf
A debian/varnishkafka.default
A debian/varnishkafka.init
A debian/varnishkafka.install
A debian/varnishkafka.logrotate
13 files changed, 343 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/software/varnish/varnishkafka refs/changes/82/78782/1
diff --git a/.gitignore b/.gitignore
index 7edc56e..94bc924 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,4 @@
varnishkafka
*~
.emacs*
+.pc
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..6b5a383
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+varnishkafka (0~20130628-1) unstable; urgency=low
+
+ * Initial release.
+
+ -- Faidon Liambotis <paravoid [at] debian> Wed, 07 Aug 2013 19:55:39 +0300
diff --git a/debian/compat b/debian/compat
new file mode 100644
index 0000000..ec63514
--- /dev/null
+++ b/debian/compat
@@ -0,0 +1 @@
+9
diff --git a/debian/control b/debian/control
new file mode 100644
index 0000000..03a4470
--- /dev/null
+++ b/debian/control
@@ -0,0 +1,19 @@
+Source: varnishkafka
+Priority: optional
+Maintainer: Faidon Liambotis <paravoid [at] debian>
+Build-Depends: debhelper (>= 9), libvarnishapi-dev, librdkafka-dev
+Standards-Version: 3.9.4
+Section: web
+
+Package: varnishkafka
+Architecture: any
+Depends: varnish, ${shlibs:Depends}, ${misc:Depends}
+Enhances: varnish
+Description: Varnish to Kafka log streamer
+ varnishkafka reads Varnish shared memory logs, formats them and prints them to
+ standard output or streams them to Kafka brokers, using the Apache Kafka
+ protocol. It can format the logs into either an Apache/NCSA "combined" log
+ format or any other custom log format.
+ .
+ varnishkafka is similar to varnishncsa, as shipped by the Varnish source,
+ although it has more features and was written with high performance in mind.
diff --git a/debian/copyright b/debian/copyright
new file mode 100644
index 0000000..52a6cbb
--- /dev/null
+++ b/debian/copyright
@@ -0,0 +1,45 @@
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Upstream-Name: varnishkafka
+Source: https://github.com/wikimedia/varnishkafka
+
+License: BSD-2-clause
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+ .
+ 1. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+ .
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ POSSIBILITY OF SUCH DAMAGE.
+
+Files: *
+Copyright: 2013 Wikimedia Foundation, Inc.
+ 2013 Magnus Edenhill
+License: BSD-2-clause
+
+Files: base64.c
+License: public-domain
+ The file header indicates that it was placed into the public domain by its
+ author, Poul-Henning Kamp <phk [at] phk>
+
+Files: base64.h
+Copyright: 2006 Verdens Gang AS
+ 2006-2011 Varnish Software AS
+License: BSD-2-clause
+
+Files: debian/*
+Copyright: 2013 Faidon Liambotis <paravoid [at] debian>
+ 2013 Wikimedia Foundation, Inc.
+License: BSD-2-clause
diff --git a/debian/gbp.conf b/debian/gbp.conf
new file mode 100644
index 0000000..bda89b2
--- /dev/null
+++ b/debian/gbp.conf
@@ -0,0 +1,7 @@
+[git-buildpackage]
+upstream-tree=branch
+upstream-branch=master
+debian-branch=debian
+#no-create-orig = True
+tarball-dir = ../tarballs
+export-dir = ../build-area
diff --git a/debian/rules b/debian/rules
new file mode 100755
index 0000000..17260a4
--- /dev/null
+++ b/debian/rules
@@ -0,0 +1,15 @@
+#!/usr/bin/make -f
+
+# Uncomment this to turn on verbose mode.
+#export DH_VERBOSE=1
+
+# FIXME
+export VER=1
+
+%:
+ dh $@
+
+override_dh_auto_install:
+ # don't run "make install", use debian/*.install instead
+
+.PHONY: override_dh_auto_install
diff --git a/debian/source/format b/debian/source/format
new file mode 100644
index 0000000..163aaf8
--- /dev/null
+++ b/debian/source/format
@@ -0,0 +1 @@
+3.0 (quilt)
diff --git a/debian/varnishkafka.conf b/debian/varnishkafka.conf
new file mode 100644
index 0000000..8918eba
--- /dev/null
+++ b/debian/varnishkafka.conf
@@ -0,0 +1,121 @@
+######################################################################
+# #
+# varnishkafka configuration file #
+# #
+# #
+######################################################################
+# #
+# Format: #
+# <property-name> = <value> #
+# #
+# boolean properties: #
+# >0, "true", "yes", "on" - interpreted as true #
+# everything else - interpreted as false #
+# #
+######################################################################
+
+# Where to output varnish log lines:
+# kafka - (default) send to kafka broker
+# stdout - just print to stdout (behave like varnishncsa)
+output = stdout
+
+# Varnish log formatting
+format = %l %n %t %{Varnish:time_firstbyte}x %h %{Varnish:handling}x/%s %b %m http://%{Host}i%U%q - %{Content-Type}o %{Referer}i %{X-Forwarded-For}i %{User-agent}i %{Accept-Language}i
+
+# Start for sequence number (%n)
+# Either a number, or the string "time" which will set it to the current
+# unix time in seconds multiplied by 1,000,000.
+# Defaults to 0.
+sequence.number = time
+
+#
+# varnishkafka log messages configuration
+# Debugging, error reporting, etc, not to be confused with varnish logs.
+#
+
+# varnishkafka log level (1 = emergencies .. 7 = debug)
+log.level = 6
+
+# specify log output
+log.stderr = false
+log.syslog = true
+
+# daemonize varnishkafka (boolean)
+daemonize = true
+
+######################################################################
+# #
+# Standard varnish VSL command line arguments #
+# #
+# Format: #
+# varnish.arg.<c> = <value>, where <c> is a command line option. #
+# #
+# See varnishncsa(1) and varnishlog(1) for valid options. #
+# #
+######################################################################
+
+# -m tag:regex
+varnish.arg.m = RxRequest:^(?!PURGE$)
+
+# Examples:
+# -C: ignore case when matching regex
+# Non-value arguments need a dummy value to pass parsing, such as 'true'.
+#varnish.arg.C = true
+
+# -n: varnishd instance to get logs from.
+#varnish.arg.n = frontend
+
+
+######################################################################
+# #
+# Kafka configuration #
+# #
+# For the full range of Kafka handle and topic configuration #
+# properties, see: #
+# https://github.com/edenhill/librdkafka/blob/0.8-wip/rdkafka.h #
+# #
+# And the Apache Kafka configuration reference: #
+# http://kafka.apache.org/08/configuration.html #
+# #
+######################################################################
+
+# Initial list of kafka brokers
+metadata.broker.list = localhost:9091
+
+# Maximum number of messages allowed on the local producer queue
+queue.buffering.max.messages = 1000000
+
+# Maximum number of retries per messageset.
+message.send.max.retries = 3
+
+
+#
+# Topic configuration
+#
+
+# Topic to produce messages to
+topic = varnish
+
+# Partition (-1: random, else one of the available partitions)
+partition = -1
+
+
+# Required number of acks
+topic.request.required.acks = 1
+
+# Local message timeout (milliseconds)
+topic.message.timeout.ms = 60000
+
+
+#
+# Builtin partitioners (Future. Not implemented)
+#
+
+# partition on user name
+#partitioner = crc32(%u)
+
+# partition on client IP
+#partitioner = iphash(%{X-Forwarded-For}i)
+
+
+
diff --git a/debian/varnishkafka.default b/debian/varnishkafka.default
new file mode 100644
index 0000000..54d7a5c
--- /dev/null
+++ b/debian/varnishkafka.default
@@ -0,0 +1,6 @@
+# Configuration file for varnishkafka
+#
+# This shell script fragment is sourced by /etc/init.d/varnishkafka
+#
+# VARNISHKAFKA_ENABLED=1
+# DAEMON_OPTS="-a -w ${LOGFILE} -D -P ${PIDFILE}"
diff --git a/debian/varnishkafka.init b/debian/varnishkafka.init
new file mode 100644
index 0000000..2a340d3
--- /dev/null
+++ b/debian/varnishkafka.init
@@ -0,0 +1,107 @@
+#! /bin/sh
+
+### BEGIN INIT INFO
+# Provides: varnishkafka
+# Required-Start: $local_fs $remote_fs $network
+# Required-Stop: $local_fs $remote_fs $network
+# Default-Start: 2 3 4 5
+# Default-Stop: 0 1 6
+# Short-Description: Varnish to Kafka logging daemon
+# Description: Varnish to Kafka logging daemon
+### END INIT INFO
+
+# Source function library
+. /lib/lsb/init-functions
+
+NAME=varnishkafka
+DESC="HTTP accelerator log deamon"
+PATH=/sbin:/bin:/usr/sbin:/usr/bin
+DAEMON=/usr/bin/$NAME
+PIDFILE=/var/run/$NAME/$NAME.pid
+LOGFILE=/var/log/varnish/varnishkafka.log
+USER=varnishlog
+DAEMON_OPTS="-a -w ${LOGFILE} -D -P ${PIDFILE}"
+
+# Include defaults if available
+if [ -f /etc/default/$NAME ] ; then
+ . /etc/default/$NAME
+fi
+
+# If unset, or set to "0" or "no", exit
+if [ -z "${VARNISHKAFKA_ENABLED}" ] || \
+ [ "${VARNISHKAFKA_ENABLED}" = "0" ] || \
+ [ "${VARNISHKAFKA_ENABLED}" = "no" ]; then
+ exit 0;
+fi
+
+test -x $DAEMON || exit 0
+
+start_varnishkafka() {
+ output=$(/bin/tempfile -s.varnish)
+ log_daemon_msg "Starting $DESC" "$NAME"
+ create_pid_directory
+ if start-stop-daemon --start --quiet --pidfile ${PIDFILE} \
+ --chuid $USER --exec ${DAEMON} -- ${DAEMON_OPTS} \
+ > ${output} 2>&1; then
+ log_end_msg 0
+ else
+ log_end_msg 1
+ cat $output
+ exit 1
+ fi
+ rm $output
+}
+
+stop_varnishkafka(){
+ log_daemon_msg "Stopping $DESC" "$NAME"
+ if start-stop-daemon --stop --quiet --pidfile $PIDFILE \
+ --retry 10 --exec $DAEMON; then
+ log_end_msg 0
+ else
+ log_end_msg 1
+ fi
+}
+
+reload_varnishkafka(){
+ log_daemon_msg "Reloading $DESC" "$NAME"
+ if kill -HUP $(cat $PIDFILE) >/dev/null 2>&1; then
+ log_end_msg 0
+ else
+ log_end_msg 1
+ exit 1
+ fi
+}
+
+status_varnishkafka(){
+ status_of_proc -p "${PIDFILE}" "${DAEMON}" "${NAME}"
+ exit $?
+}
+
+create_pid_directory() {
+ install -o $USER -g $USER -d $(dirname $PIDFILE)
+}
+
+case "$1" in
+ start)
+ start_varnishkafka
+ ;;
+ stop)
+ stop_varnishkafka
+ ;;
+ reload)
+ reload_varnishkafka
+ ;;
+ status)
+ status_varnishkafka
+ ;;
+ restart|force-reload)
+ $0 stop
+ $0 start
+ ;;
+ *)
+ log_success_msg "Usage: $0 {start|stop|restart|force-reload|reload}"
+ exit 1
+ ;;
+esac
+
+exit 0
diff --git a/debian/varnishkafka.install b/debian/varnishkafka.install
new file mode 100644
index 0000000..9cc0579
--- /dev/null
+++ b/debian/varnishkafka.install
@@ -0,0 +1,2 @@
+varnishkafka usr/bin
+debian/varnishkafka.conf etc/
diff --git a/debian/varnishkafka.logrotate b/debian/varnishkafka.logrotate
new file mode 100644
index 0000000..8f3ea54
--- /dev/null
+++ b/debian/varnishkafka.logrotate
@@ -0,0 +1,13 @@
+/var/log/varnish/varnishkafka.log {
+ daily
+ rotate 7
+ missingok
+ compress
+ delaycompress
+ missingok
+ postrotate
+ if /usr/bin/pgrep -P 1 varnishkafka >/dev/null; then
+ /usr/sbin/invoke-rc.d varnishkafka reload > /dev/null
+ fi
+ endscript
+}
--
To view, visit https://gerrit.wikimedia.org/r/78782
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib53267d85e7a638d34b8660557b3c62638abdd54
Gerrit-PatchSet: 1
Gerrit-Project: operations/software/varnish/varnishkafka
Gerrit-Branch: debian
Gerrit-Owner: Faidon <faidon [at] wikimedia>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits [at] lists
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
|
