Skip to content

FLUME-3271 Update User Guide to follow log4j2 upgrade changes #272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
turcsanyip wants to merge 1 commit into
base: trunk
Choose a base branch
from
Open

FLUME-3271 Update User Guide to follow log4j2 upgrade changes #272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions conf/log4j2.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,14 @@
-->
<Configuration status="ERROR">
<Properties>
<Property name="LOG_DIR">./logs</Property>
<Property name="LOG_DIR">${sys:flume.log.dir:-./logs}</Property>
<Property name="LOG_FILE">${sys:flume.log.file:-flume.log}</Property>
</Properties>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d (%t) [%p - %l] %m%n" />
</Console>
<RollingFile name="LogFile" fileName="${LOG_DIR}/flume.log" filePattern="${LOG_DIR}/archive/flume.log.%d{yyyyMMdd}-%i">
<RollingFile name="LogFile" fileName="${LOG_DIR}/${LOG_FILE}" filePattern="${LOG_DIR}/archive/${LOG_FILE}.%d{yyyyMMdd}-%i">
<PatternLayout pattern="%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %equals{%x}{[]}{} - %m%n" />
<Policies>
<!-- Roll every night at midnight or when the file reaches 100MB -->
Expand All @@ -34,7 +35,7 @@
<DefaultRolloverStrategy min="1" max="20">
<Delete basePath="${LOG_DIR}/archive">
<!-- Nested conditions: the inner condition is only evaluated on files for which the outer conditions are true. -->
<IfFileName glob="flume.log.*">
<IfFileName glob="${LOG_FILE}.*">
<!-- Only allow 1 GB of files to accumulate -->
<IfAccumulatedFileSize exceeds="1 GB"/>
</IfFileName>
Expand All @@ -44,13 +45,13 @@
</Appenders>

<Loggers>
<Logger name="org.apache.flume.lifecycle" level="info"/>
<Logger name="org.apache.flume.lifecycle" level="INFO"/>
<Logger name="org.jboss" level="WARN"/>
<Logger name="org.apache.avro.ipc.NettyTransceiver" level="WARN"/>
<Logger name="org.apache.hadoop" level="INFO"/>
<Logger name="org.apache.hadoop.hive" level="ERROR"/>
<Root level="INFO">
<AppenderRef ref="LogFile" />
<Root level="${sys:flume.root.logger.level:-INFO">
<AppenderRef ref="${sys:flume.root.logger.appender:-LogFile}" />
</Root>
</Loggers>
</Configuration>
21 changes: 11 additions & 10 deletions flume-ng-doc/sphinx/FlumeUserGuide.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,11 +195,11 @@ several named agents; when a given Flume process is launched a flag is passed te

Given this configuration file, we can start Flume as follows::

$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger.level=INFO -Dflume.root.logger.appender=Console

Note that in a full deployment we would typically include one more option: ``--conf=<conf-dir>``.
The ``<conf-dir>`` directory would include a shell script *flume-env.sh* and potentially a log4j properties file.
In this example, we pass a Java option to force Flume to log to the console and we go without a custom environment script.
Note that in a full deployment we typically include the option: ``--conf=<conf-dir>``.
The ``<conf-dir>`` directory would include a shell script *flume-env.sh* and a *log4j2.xml* file.
In this example, we pass a Java option to force Flume to log to the console.

From a separate terminal, we can then telnet port 44444 and send Flume an event:

Expand Down Expand Up @@ -237,7 +237,8 @@ NB: it currently works for values only, not for keys. (Ie. only on the "right si
This can be enabled via Java system properties on agent invocation by setting `propertiesImplementation = org.apache.flume.node.EnvVarResolverProperties`.

For example::
$ NC_PORT=44444 bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console -DpropertiesImplementation=org.apache.flume.node.EnvVarResolverProperties

$ NC_PORT=44444 bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger.level=INFO -Dflume.root.logger.appender=Console -DpropertiesImplementation=org.apache.flume.node.EnvVarResolverProperties

Note the above is just an example, environment variables can be configured in other ways, including being set in `conf/flume-env.sh`.

Expand All @@ -256,20 +257,20 @@ connected to a `Logger Sink`_, which will output all event data to the Flume log
In some situations, however, this approach is insufficient.

In order to enable logging of event- and configuration-related data, some Java system properties
must be set in addition to log4j properties.
must be set in addition to log4j2 configuration.

To enable configuration-related logging, set the Java system property
``-Dorg.apache.flume.log.printconfig=true``. This can either be passed on the command line or by
setting this in the ``JAVA_OPTS`` variable in *flume-env.sh*.

To enable data logging, set the Java system property ``-Dorg.apache.flume.log.rawdata=true``
in the same way described above. For most components, the log4j logging level must also be set to
in the same way described above. For most components, the log4j2 logging level must also be set to
DEBUG or TRACE to make event-specific logging appear in the Flume logs.

Here is an example of enabling both configuration logging and raw data logging while also
setting the Log4j loglevel to DEBUG for console output::
setting the log4j2 logging level to DEBUG for console output::

$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=DEBUG,console -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.rawdata=true
$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger.level=DEBUG -Dflume.root.logger.appender=Console -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.rawdata=true


Zookeeper based Configuration
Expand All @@ -287,7 +288,7 @@ Following is how the Zookeeper Node tree would look like for agents a1 and a2

Once the configuration file is uploaded, start the agent with following options

$ bin/flume-ng agent --conf conf -z zkhost:2181,zkhost1:2181 -p /flume --name a1 -Dflume.root.logger=INFO,console
$ bin/flume-ng agent --conf conf -z zkhost:2181,zkhost1:2181 -p /flume --name a1 -Dflume.root.logger.level=INFO -Dflume.root.logger.appender=Console

================== ================ =========================================================================
Argument Name Default Description
Expand Down
12 changes: 8 additions & 4 deletions flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,11 @@ public class StagedInstall {
"flume.dist.tarball";

public static final String ENV_FLUME_LOG_DIR = "flume.log.dir";
public static final String ENV_FLUME_ROOT_LOGGER = "flume.root.logger";
public static final String ENV_FLUME_ROOT_LOGGER_VALUE = "DEBUG,LOGFILE";
public static final String ENV_FLUME_LOG_FILE = "flume.log.file";
public static final String ENV_FLUME_ROOT_LOGGER_LEVEL = "flume.root.logger.level";
public static final String ENV_FLUME_ROOT_LOGGER_LEVEL_VALUE = "DEBUG";
public static final String ENV_FLUME_ROOT_LOGGER_APPENDER = "flume.root.logger.appender";
public static final String ENV_FLUME_ROOT_LOGGER_APPENDER_VALUE = "LogFile";

private final File stageDir;
private final File baseDir;
Expand Down Expand Up @@ -156,9 +158,11 @@ public synchronized void startAgent(
builder.add("--conf-file", configFilePath);
builder.add("--name", agentName);
builder.add("-D" + ENV_FLUME_LOG_DIR + "=" + logDirPath);
builder.add("-D" + ENV_FLUME_ROOT_LOGGER + "="
+ ENV_FLUME_ROOT_LOGGER_VALUE);
builder.add("-D" + ENV_FLUME_LOG_FILE + "=" + logFileName);
builder.add("-D" + ENV_FLUME_ROOT_LOGGER_LEVEL + "="
+ ENV_FLUME_ROOT_LOGGER_LEVEL_VALUE);
builder.add("-D" + ENV_FLUME_ROOT_LOGGER_APPENDER + "="
+ ENV_FLUME_ROOT_LOGGER_APPENDER_VALUE);

commandOptions.forEach((key, value) -> builder.add(key, value));

Expand Down