Home / Question / How to set up Loggly on Elastic Beanstalk?

How to set up Loggly on Elastic Beanstalk?

I’d like to set up Loggly to run on AWS Elastic Beanstalk, but can’t find any information on how to do this. Is there any guide anywhere, or some general guidance on how to start?

Check Also

Nebia half spray

Nebia

Nebia leverages advances in design and thermofluids to create a better shower experience that actually saves you thousands of gallons of water a year. With those savings, Nebia pays for itself in about a year.

8 comments

  1. Follow my loggly configuration in elasticbeanstalk. For Linux + log4j

    on .ebextensions file configuration

    container_commands:
      01_configure_sudo_access:
        command: sed -i -- 's/ requiretty/ !requiretty/g' /etc/sudoers
      02_loggy_configure:
        command: sudo python .ebextensions/scripts/loggly_config.py
      03_restore_sudo_access:
        command: sed -i -- 's/ !requiretty/ requiretty/g' /etc/sudoers
    

    Loggly script in python for default AMI:

    import os
    
    rsyslog_path = '/etc/rsyslog.conf'
    loggly_file_path = '/etc/rsyslog.d/22-loggly.conf'
    
    class LogglyConfig:
    
        def __init__(self):
            self.__linux_log()
            self.__config_loggly_for_log4j()
    
        def __linux_log(self):
            #not installed on this machine
            if not os.path.exists(loggly_file_path):
                os.system('rm -f configure-linux.sh')
                os.system('wget https://www.loggly.com/install/configure-linux.sh')
                os.system('sudo bash configure-linux.sh -a DOMAIN -t TOKEN -u USER -p PASSWORD -s')
    
    
        def __config_loggly_for_log4j(self):
            f = open(rsyslog_path,'r')
            file_text = f.read()
            f.close()
            file_text = file_text.replace('#$ModLoad imudp', '$ModLoad imudp')
            file_text = file_text.replace('#$UDPServerRun 514', '$UDPServerRun 514')
            f = open(rsyslog_path,'w')
            f.write(file_text)
            f.close()
    
            os.system('service rsyslog restart')
    
    LogglyConfig()
    

    In log4j.properties on your java project

    log4j.rootLogger=INFO, SYSLOG
    
    log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
    log4j.appender.SYSLOG.SyslogHost=localhost
    log4j.appender.SYSLOG.Facility=Local3
    log4j.appender.SYSLOG.Header=true
    log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
    log4j.appender.SYSLOG.layout.ConversionPattern=java %d{ISO8601} %p %t %c{1}.%M - %m%n
    
  2. You can also use the installation script from loggly itself.
    The setup below follows the instructions for the legacy setup on https://www.loggly.com/docs/configure-syslog-script/ with minor changes (no confirmation prompts, sudo command replaced since no tty is available)

    (edit: updated link, seems to be an outdated solution now in loggly docs)

    Place the following script in .ebextensions/loggly.config

    Replace TOKEN and ACCOUNT with your own.

    #
    # Install loggly.com on AWS Elastic Beanstalk
    # Tested with node.js environment
    # Save this file as .ebextensions/loggly.config
    # Deploy per normal scripts or aws.push. To help debug the push, ssh & tail /var/log/cfn-init.log
    # See Also /var/log/eb-tools.log
    #
    
    commands:
      01_loggly_dl:
        command: wget -q -O /tmp/loggly.py https://www.loggly.com/install/configure-syslog.py
      02_loggly_config:
        command: su --session-command="python /tmp/loggly.py setup --auth TOKEN --account ACCOUNT --yes"
    
  3. This is how I do it, for papertrailapp.com (which I prefer instead of loggly). In your /ebextensions folder (see more info) you create logs.config, where specify:

    container_commands:
      01-set-correct-hostname:
        command: hostname www.example.com
      02-forward-rsyslog-to-papertrail:
        # https://papertrailapp.com/systems/setup
        command: echo "*.* @logs.papertrailapp.com:55555" >> /etc/rsyslog.conf
      03-enable-remote-logging:
        command: echo -e "$ModLoad imudpn$UDPServerRun 514n$ModLoad imtcpn$InputTCPServerRun 514n$EscapeControlCharactersOnReceive off" >> /etc/rsyslog.conf
      04-restart-syslog:
        command: service rsyslog restart
    

    55555 should be replaced with the UDP port number provided by papertrailapp.com. Every time after new instance bootstrap this config will be applied. Then, in your log4j.properties:

    log4j.rootLogger=WARN, SYSLOG
    log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
    log4j.appender.SYSLOG.facility=local1
    log4j.appender.SYSLOG.header=true
    log4j.appender.SYSLOG.syslogHost=localhost
    log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
    log4j.appender.SYSLOG.layout.ConversionPattern=[%p] %t %c: %m%n
    

    I’m not sure whether it’s an optimal solution. Read more about this mechanism in jcabi-beanstalk-maven-plugin

  4. I know this is question is fairly old but I found that the answers really didnt answer the question or just plain didnt work correctly when implemented. I found that this works (file .ebextenstions/02loggly.config):

    container_commands:
      01-transform-rsyslog.conf:
        command: sed "s/NODE_ENV/$NODE_ENV/g" scripts/22-loggly.conf.temp > scripts/22-loggly.conf
      02-setup-rsyslog.conf:
        command: cp scripts/22-loggly.conf /etc/rsyslog.d/22-loggly.conf
      03-restart:
        command: /sbin/service rsyslog restart
    

    the “01-transform-rsyslog.conf” step is optional; I use that to set a tag by NODE_ENV in the file. “22-loggly.conf.temp” is a modified version of the “22-loggly.conf” file that gets created at “/etc/rsyslog.d/” when you run the linux source setup script (https://www.loggly.com/install/configure-syslog.py). I just installed it on a ec2 instance and copied the file.

    Note I had to prepend ‘/sbin’ to my service command because it was failing for me without it. Also, this restarts syslog on every deploy, which should be fine.

    Now you just have to make sure your app logs to syslog. For Java it is going to be log4j or similar. For Node.js (which is what I’m using), rconsole works (https://github.com/tblobaum/rconsole).

  5. Typically, /etc/rsyslog.config will have a “$IncludeConfig /etc/rsyslog.d/*.conf” at the end – so you can simply introduce your own configuration file using the “files:” portion of your .ebextensions file. This works whether you are deploying to fresh servers or not.

    For a ruby production.log, you might have something like this in a .ebextensions/01loggly.config file. Note this picks up your beanstalk environment name too as a loggly tag.

    # For docs on eb configs, see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
    # This set of commands sets up loggly forwarding
    files:
       "/etc/rsyslog.d/myapp-loggly.conf" :
          mode: "000664"
          owner: root
          group: root
          content: |
             $template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [yourlogglyid@41058 tag=`{ "Ref" : "AWSEBEnvironmentName" }`] %msg%n"
             *.* @@logs-01.loggly.com:514;LogglyFormat
    
             # One time config
             $ModLoad imfile
             $InputFilePollInterval 10 
             $PrivDropToGroup adm
             $WorkDirectory /var/spool/rsyslog
    
             # Add a tag for file events
    
             # For production.log
             $InputFileName /var/app/support/logs/production.log
             $InputFileTag production-log
             $InputFileStateFile stat-production-log #this must be unique for each file being polled
             $InputFileSeverity info
             $InputFilePersistStateInterval 20000
             $InputRunFileMonitor
             # Send to Loggly then discard
             if $programname == 'myapp-production-log' then @@logs-01.loggly.com:514;LogglyFormat
             if $programname == 'myapp-production-log' then ~
    
          encoding: plain
    commands:
       00-make-work-directory:
          command: mkdir -p /var/spool/rsyslog
       01-restart-syslog:
          command: service rsyslog restart
    

    For Tomcat, you might do something like this in a .ebextesions/01logglyg.config file:

    # For docs on eb configs, see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
    # This set of commands sets up loggly forwarding
    files:
        "/etc/rsyslog.d/mytomcatapp-loggly.conf" :
            mode: "000664"
            owner: root
            group: root
            content: |
                $template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [yourlogglygidhere@41058 tag=`{ "Ref" : "AWSEBEnvironmentName" }`] %msg%n"
                *.* @@logs-01.loggly.com:514;LogglyFormat
    
                # One time config
                $ModLoad imfile
                $InputFilePollInterval 10 
                $PrivDropToGroup adm
                $WorkDirectory /var/spool/rsyslog
    
                # catalina.log
                $InputFileName /var/log/tomcat7/catalina.log
                $InputFileTag catalina-log
                $InputFileStateFile stat-catalina-log
                $InputFileSeverity info
                $InputFilePersistStateInterval 20000
                $InputRunFileMonitor
                if $programname == 'catalina-log' then @@logs-01.loggly.com:514;LogglyFormat
                if $programname == 'catalina-log' then ~
    
                # catalina.out
                $InputFileName /var/log/tomcat7/catalina.out
                $InputFileTag catalina-out
                $InputFileStateFile stat-catalina-out
                $InputFileSeverity info
                $InputFilePersistStateInterval 20000
                $InputRunFileMonitor
                if $programname == 'catalina-out' then @@logs-01.loggly.com:514;LogglyFormat
                if $programname == 'catalina-out' then ~
    
                # host-manager.log
                $InputFileName /var/log/tomcat7/host-manager.log
                $InputFileTag host-manager
                $InputFileStateFile stat-host-manager
                $InputFileSeverity info
                $InputFilePersistStateInterval 20000
                $InputRunFileMonitor
                if $programname == 'host-manager' then @@logs-01.loggly.com:514;LogglyFormat
                if $programname == 'host-manager' then ~
    
                # initd.log
                $InputFileName /var/log/tomcat7/initd.log
                $InputFileTag initd
                $InputFileStateFile stat-initd
                $InputFileSeverity info
                $InputFilePersistStateInterval 20000
                $InputRunFileMonitor
                if $programname == 'initd' then @@logs-01.loggly.com:514;LogglyFormat
                if $programname == 'initd' then ~
    
                # localhost.log
                $InputFileName /var/log/tomcat7/localhost.log
                $InputFileTag localhost-log
                $InputFileStateFile stat-localhost-log
                $InputFileSeverity info
                $InputFilePersistStateInterval 20000
                $InputRunFileMonitor
                if $programname == 'localhost-log' then @@logs-01.loggly.com:514;LogglyFormat
                if $programname == 'localhost-log' then ~
    
                # manager.log
                $InputFileName /var/log/tomcat7/manager.log
                $InputFileTag manager
                $InputFileStateFile stat-manager
                $InputFileSeverity info
                $InputFilePersistStateInterval 20000
                $InputRunFileMonitor
                if $programname == 'manager' then @@logs-01.loggly.com:514;LogglyFormat
                if $programname == 'manager' then ~
    
            encoding: plain
    commands:
        00-make-work-directory:
            command: mkdir -p /var/spool/rsyslog
        01-restart-syslog:
            command: service rsyslog restart
    

    This config is working for me – though I haven’t yet determined how to get multi-line entries coming into a single entry in Loggly yet.

  6. None of the things I tried seemed to work, and the loggly documentation is very confusing!
    I hope that this will help someone, this is how I got it to work.

    Paste the following in .ebextensions/loggly.config

    files:
      "/etc/rsyslog.conf" :
        mode: "000644"
        owner: root
        group: root
        content: |
          $ModLoad imfile
          $InputFilePollInterval 10
          $PrivDropToGroup adm
    
          # Input for FILE.LOG
          $InputFileName /var/app/current/PATH_TO_YOUR_LOG_FILE
          $InputFileTag social_php:
          $InputFileStateFile stat-social_php #this must be unique for each file being polled
          $InputFileSeverity info
          $InputRunFileMonitor
    
          #Add a tag for events from this file
          $template LogglyFormatsocial_php,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [TOKEN@41058 tag="php_log"] %msg%n"
    
          if $programname == 'social_php' then @@logs.loggly.com:37146;LogglyFormatsocial_php
          if $programname == 'social_php' then ~
          *.*    @@logs.loggly.com:37146
    
    
    commands:
      01-restart-syslog:
        command: service rsyslog restart
    
    • Replace all instances of social_php with the tag that makes sense for your application.
    • Replace /var/app/current/PATH_TO_YOUR_LOG_FILE with your log file location
  7. Here is an elasticbeanstalk config for Loggly that I’ve just started using thanks to pointers from this thread and the logging SaaS vendors setup instructions. [Loggly Config Mgmt, Papertrail rsyslog ]

    Save the file as loggly.config in the .ebextensions directory and make sure to check the YAML formatting conventions (no tabs, etc). Substitute your Loggly TCP port number, username, password and domain name into the angle brackets as required.

    Note that for AWS ruby versions of elasticbeanstalk, there may be differences in the EC2 /etc/rsyslog setup. For example, if /etc/rsyslog.d already exists, and there is already an “$IncludeConfig /etc/rsyslog.d/*.conf” directive, then command “01-forward-rsyslog-to-loggly:” can be removed.

    Deploy per normal scripts or aws.push. To help debug the push, ssh & tail /var/log/cfn-init.log

    files:
      "/etc/rsyslog.d/90-loggly.conf" :
        mode: "000664"
        owner: root
        group: root
        content: |
          # ### begin forwarding rule ###
          # The statement between the begin ... end define a SINGLE forwarding
          # rule. They belong together, do NOT split them. If you create multiple
          # forwarding rules, duplicate the whole block!
          # Remote Logging (we use TCP for reliable delivery)
          #
          # An on-disk queue is created for this action. If the remote host is
          # down, messages are spooled to disk and sent when it is up again.
          $WorkDirectory /var/lib/rsyslog # where to place spool files
          $ActionQueueFileName fwdRule1 # unique name prefix for spool files
          $ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
          $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
          $ActionQueueType LinkedList   # run asynchronously
          $ActionResumeRetryCount -1    # infinite retries if host is down
          *.* @@logs.loggly.com:<yourportnum>   # !!!Loggly supplied port number for each app!!!
          # ### end of the forwarding rule ###
        encoding: plain
      "/tmp/loggly.py" :
        mode: "000755"
        owner: root
        group: root
        content: |
          import json
          import sys
          import urllib2
          '''
          Auto-authenticate Syslog TCP inputs.
          Usage: python inputs.py -u user -p pass -s subdomain
          '''
          state = None
          params = {}
          for i in range(len(sys.argv)):
             arg = sys.argv[i]
             if state:
                 params[state] = arg
                 state = None
    
             if arg == '--username' or arg == '-u':
                 state = 'username'
    
             if arg == '--password' or arg == '-p':
                 state = 'password'
    
             if arg == '--subdomain' or arg == '-s':
                 state = 'subdomain'
          url = 'https://%s.loggly.com/api/inputs' % params['subdomain']
          password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
          password_mgr.add_password(None, url, params['username'], params['password'])
          handler = urllib2.HTTPBasicAuthHandler(password_mgr)
          opener = urllib2.build_opener(handler)
          opener.open(url)
          urllib2.install_opener(opener)
          inputs = json.loads(urllib2.urlopen(url).read())
          for input in inputs:
             if input['service']['name'] == 'syslogtcp':
                 url = 'https://%s.loggly.com/api/inputs/%d/adddevice' % 
                     (params['subdomain'], input['id'])
                 response = urllib2.urlopen(url, {}).read()
                 print response
        encoding: plain
    
    commands:
      01-forward-rsyslog-to-loggly:
        # http://loggly.com/support/sending-data/logging-from/syslog/rsyslog/cd
        command: test "$(grep -s '90-loggly.conf' /etc/rsyslog.conf)" == "" && echo -e "n# Include the loggly.conf filen$IncludeConfig /etc/rsyslog.d/90-loggly.conf" >> /etc/rsyslog.conf
      02-restart-syslog:
        command: service rsyslog restart
      03-inform_loggly:
        command: "python /tmp/loggly.py -u <Yourloginname> -p <Yourpassword> -s <Yourdomainname>"
    
  8. Here is a link to loggly support site for using syslogd with loggly:
    http://wiki.loggly.com/loggingconfiguration

    or using the loggly api with your own app:
    http://wiki.loggly.com/apidocumention

Leave a Top Ten Reply: